Chanter-Machine opened a new pull request, #10945:
URL: https://github.com/apache/paimon/pull/10945

   ### Purpose
   
   Closes #10936. This is a draft implementation for design discussion; the 
proposal has not yet received maintainer agreement.
   
   The native Flink source currently emits an entire physical reader batch in 
one poll. With a slow downstream, this can delay the next checkpoint 
opportunity. Add optional `scan.max-records-per-poll`: `0` preserves the 
existing behavior, while a positive value bounds each emission call. `1` allows 
returning control after each emitted row without changing `read.batch-size`.
   
   The implementation reuses the native source's split position state, 
serializers and restore seek. Bounded views share a physical batch until it is 
consumed or the reader closes; there is no per-record batch allocation or new 
checkpoint state. Tests also close a previously unclosed batch/reader in the 
existing wakeup test so temporary files can be removed on Windows.
   
   This does not replace MonitorSource/ReadOperator or change the default 
consumer mode. With `consumer-id`, the option explicitly requires 
`consumer.mode=at-least-once`; unsupported topologies and negative values are 
rejected. This consumer cursor mode is distinct from Flink checkpoint 
consistency. A blocked single output or downstream checkpoint can still delay 
completion, so this is not a savepoint latency guarantee.
   
   ### Tests
   
   - Flink 1.20.1 / JDK 8: **108 tests passed**, 0 failures/errors/skips, 
across `FlinkRecordsWithSplitIdsTest`, `FileStoreSourceReaderTest`, 
`FileStoreSourceSplitReaderTest`, `FlinkSourceBuilderTest`, 
`ContinuousFileSplitEnumeratorTest`, `FileStoreSourceSplitSerializerTest`, and 
`OperatorSourceTest`.
   - Flink 2.2.0 / JDK 11: clean rebuild and the same **108 tests passed**, 0 
failures/errors/skips.
   - Coverage includes the unchanged default, emission budgets, CDC row-kind 
order, record-limit boundaries, exact positions, serialized partial-batch 
restore into a new reader, batch ownership/close, and incompatible 
configuration validation. Checkstyle, Spotless and Maven Enforcer ran without 
the fast-build profile.
   - REST documentation contract checks passed (64 catalog + 10 management 
operations). The documentation site built successfully using a temporary local 
Windows path-separator override for the existing generated-HTML loader rule; no 
documentation build configuration change is included in this PR. The unmodified 
rule fails to load generated HTML on Windows.
   - **Repository-wide RAT did not pass**: it flags two fixtures already 
present and unchanged from the base commit: 
`paimon-flink/paimon-flink-common/src/test/resources/compatibility/pending-splits-incremental-v1-chain-v2`
 and `paimon-python/pypaimon/tests/resources/btree/posting-2.bin`. All changed 
Java files have approved ASF headers. These unrelated fixture exclusions are 
not changed here.
   - `git diff --check` passed. No live Flink-cluster performance or end-to-end 
sink validation is claimed for this upstream change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to