Chanter-Machine opened a new issue, #10936:
URL: https://github.com/apache/paimon/issues/10936

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   For streaming reads with large snapshots and slow downstream enrichment or 
sinks, checkpoint responsiveness should not depend on consuming a complete 
file-reader batch in one SourceReader.pollNext call. The native 
FileStoreSourceReader already checkpoints FileStoreSourceSplitState 
(recordsToSkip), but FlinkRecordsWithSplitIds.emitRecord currently drains the 
entire RecordIterator before returning.
   
   This is separate from #9677, which bounds snapshots in flight in the 
exactly-once MonitorSource topology. That fix does not change ReadOperator 
whole-split processing. This proposal targets the native FLIP-27 source only, 
without replacing MonitorSource or changing the default consumer mode.
   
   ### Solution
   
   Add an opt-in scan.max-records-per-poll integer option. Default 0 preserves 
whole-batch emission; positive N caps rows emitted in one poll, with 1 allowing 
a mailbox/checkpoint opportunity after each emitted row. Reject negative values 
and explicitly unsupported topologies instead of silently ignoring the option.
   
   Reuse the existing split reader, row-position state, serializer and 
seek-on-restore. Expose bounded iterator views over the same physical batch, 
retaining the batch until all views are consumed or the reader is closed. Keep 
record order, projection, record limits and record metrics. Physical 
read.batch-size remains independent; no new persistent state or external store.
   
   For consumer-id streaming reads, explicitly require 
consumer.mode=at-least-once. This describes table consumer-cursor replay 
semantics, not Flink checkpoint consistency. Do not change the snapshot-exact 
consumer contract or restore state across the two topologies. A blocking single 
collect can still delay a checkpoint: this is cooperative yielding, not 
preemption or a latency guarantee.
   
   Proposed tests: per-poll emission limits, unchanged default, exact positions 
and ordered restore from a partly emitted physical batch, resource recycling, 
empty/finished batches, metrics, and validation of incompatible options. 
Validate against Flink 1.x and 2.x.
   
   ### Anything else?
   
   Related history: #1370 deliberately switched from per-record to batch 
emission. The proposed budget remains disabled by default and avoids a 
per-record batch allocation. I would like to prepare a focused proof of concept 
for review. Could maintainers confirm the scope/option and assign this issue to 
me before I open a PR, as requested by the contribution guide?
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to