JingsongLi commented on code in PR #10945: URL: https://github.com/apache/paimon/pull/10945#discussion_r4478988061
########## docs/docs/flink/consumer-id.mdx: ########## @@ -97,6 +97,34 @@ does not provide end-to-end recovery of a stateful pipeline; retain Flink state ::: +## Yield within a read batch + +The native streaming Flink source checkpoints both its assigned splits and the number of +records already emitted from each split. With a consumer ID, select `consumer.mode=at-least-once` +to use this source rather than the snapshot-exact MonitorSource / ReadOperator topology. + +By default, one source poll emits a complete physical reader batch. Set +`scan.max-records-per-poll` to a positive integer to limit that emission; `1` returns control +to Flink after each emitted row. The default `0` preserves whole-batch emission. Negative +values are rejected. This option is supported only by the native streaming source, not +bounded reads, dedicated split generation or checkpoint-aligned reads. Review Comment: [P2] Distinguish static batch sources from streams with a stopping watermark The current wording excludes all bounded reads. I created a table with `scan.bounded.watermark=1234` and `scan.max-records-per-poll=1` and built the native streaming source. It accepted the budget and returned `Boundedness.BOUNDED`. The guard checks the builder's static/batch mode, while `ContinuousFileStoreSource` reports boundedness separately when it has a stopping watermark. Please describe the unsupported topology as a static/batch source and add coverage for watermark-bounded native streaming, or reject that mode explicitly if it is intended to be excluded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
