JingsongLi commented on code in PR #10945:
URL: https://github.com/apache/paimon/pull/10945#discussion_r4478988061


##########
docs/docs/flink/consumer-id.mdx:
##########
@@ -97,6 +97,34 @@ does not provide end-to-end recovery of a stateful pipeline; 
retain Flink state
 
 :::
 
+## Yield within a read batch
+
+The native streaming Flink source checkpoints both its assigned splits and the 
number of
+records already emitted from each split. With a consumer ID, select 
`consumer.mode=at-least-once`
+to use this source rather than the snapshot-exact MonitorSource / ReadOperator 
topology.
+
+By default, one source poll emits a complete physical reader batch. Set
+`scan.max-records-per-poll` to a positive integer to limit that emission; `1` 
returns control
+to Flink after each emitted row. The default `0` preserves whole-batch 
emission. Negative
+values are rejected. This option is supported only by the native streaming 
source, not
+bounded reads, dedicated split generation or checkpoint-aligned reads.

Review Comment:
   [P2] Distinguish static batch sources from streams with a stopping watermark
   
   The current wording excludes all bounded reads. I created a table with 
`scan.bounded.watermark=1234` and `scan.max-records-per-poll=1` and built the 
native streaming source. It accepted the budget and returned 
`Boundedness.BOUNDED`. The guard checks the builder's static/batch mode, while 
`ContinuousFileStoreSource` reports boundedness separately when it has a 
stopping watermark. Please describe the unsupported topology as a static/batch 
source and add coverage for watermark-bounded native streaming, or reject that 
mode explicitly if it is intended to be excluded.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to