Chanter-Machine opened a new pull request, #10945: URL: https://github.com/apache/paimon/pull/10945
### Purpose Closes #10936. This is a draft implementation for design discussion; the proposal has not yet received maintainer agreement. The native Flink source currently emits an entire physical reader batch in one poll. With a slow downstream, this can delay the next checkpoint opportunity. Add optional `scan.max-records-per-poll`: `0` preserves the existing behavior, while a positive value bounds each emission call. `1` allows returning control after each emitted row without changing `read.batch-size`. The implementation reuses the native source's split position state, serializers and restore seek. Bounded views share a physical batch until it is consumed or the reader closes; there is no per-record batch allocation or new checkpoint state. Tests also close a previously unclosed batch/reader in the existing wakeup test so temporary files can be removed on Windows. This does not replace MonitorSource/ReadOperator or change the default consumer mode. With `consumer-id`, the option explicitly requires `consumer.mode=at-least-once`; unsupported topologies and negative values are rejected. This consumer cursor mode is distinct from Flink checkpoint consistency. A blocked single output or downstream checkpoint can still delay completion, so this is not a savepoint latency guarantee. ### Tests - Flink 1.20.1 / JDK 8: **108 tests passed**, 0 failures/errors/skips, across `FlinkRecordsWithSplitIdsTest`, `FileStoreSourceReaderTest`, `FileStoreSourceSplitReaderTest`, `FlinkSourceBuilderTest`, `ContinuousFileSplitEnumeratorTest`, `FileStoreSourceSplitSerializerTest`, and `OperatorSourceTest`. - Flink 2.2.0 / JDK 11: clean rebuild and the same **108 tests passed**, 0 failures/errors/skips. - Coverage includes the unchanged default, emission budgets, CDC row-kind order, record-limit boundaries, exact positions, serialized partial-batch restore into a new reader, batch ownership/close, and incompatible configuration validation. Checkstyle, Spotless and Maven Enforcer ran without the fast-build profile. - REST documentation contract checks passed (64 catalog + 10 management operations). The documentation site built successfully using a temporary local Windows path-separator override for the existing generated-HTML loader rule; no documentation build configuration change is included in this PR. The unmodified rule fails to load generated HTML on Windows. - **Repository-wide RAT did not pass**: it flags two fixtures already present and unchanged from the base commit: `paimon-flink/paimon-flink-common/src/test/resources/compatibility/pending-splits-incremental-v1-chain-v2` and `paimon-python/pypaimon/tests/resources/btree/posting-2.bin`. All changed Java files have approved ASF headers. These unrelated fixture exclusions are not changed here. - `git diff --check` passed. No live Flink-cluster performance or end-to-end sink validation is claimed for this upstream change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
