Akash3121 opened a new pull request, #10028:
URL: https://github.com/apache/paimon/pull/10028
### Purpose
Fixes a limit-pushdown overflow in the Flink source when the requested
`LIMIT` is greater than `Integer.MAX_VALUE`.
Flink represents pushed-down limits as `long`, but `FlinkSourceBuilder`
previously narrowed the value to `int` before forwarding it to Paimon's
`ReadBuilder`. This could change the requested limit and cause Paimon to return
fewer rows than required.
Closes #9982.
### Root cause
The Flink planner passes the pushed-down limit through the source as a
`Long`. However, `FlinkSourceBuilder#createReadBuilder` previously used:
```java
readBuilder.withLimit(limit.intValue());
`Long.intValue()` keeps only the low 32 bits and does not report overflow.
For example:
Requested limit: 4294967297L
Converted value: 1
The narrowed value was then pushed into Paimon's internal scan and read
path. The internal scan could retain only enough splits or records for one row
before Flink's outer long-based limiter was applied. Once rows were removed by
the internal scan, the outer limiter could not recover them.
### Changes
#### Support long limits in the general read path
Added ReadBuilder.withLimit(long) and propagated long limits through the
general Paimon scan and read path, including:
- ReadBuilderImpl
- batch table scans
- snapshot readers
- append-table and key-value readers
- raw-file split readers
- format tables
- fallback tables
- audit-log system tables
- data-evolution scans
- final record limiting
FlinkSourceBuilder now forwards the original long value without
narrowing it to int .
#### Preserve API compatibility
The existing public method remains available:
`ReadBuilder withLimit(int limit);`
A default long overload was added instead of replacing the integer method.
This preserves source and binary compatibility for existing ReadBuilder
implementations.
For implementations that only support integer limits:
- representable long values are delegated to withLimit(int) ;
- values outside the integer range are left unpushed rather than narrowed
or rejected.
Skipping an unsupported optimization is safe because limit pushdown is an
optimization; the engine-level limit still provides the final query semantics.
Paimon's own read-builder implementations override the long overload and
support large limits throughout the internal read path.
#### Preserve Java serialization compatibility
ReadBuilderImpl and FormatReadBuilder are serializable and retain their
existing serialVersionUID values.
Changing the existing serialized limit field directly from Integer to
Long would make previously serialized builders incompatible. Therefore, this
change:
- retains the legacy Integer limit field;
- adds separate long-limit state;
- uses the long value when available;
- falls back to the legacy integer field when reading older serialized
builders.
This allows existing serialized builder state to remain readable.
#### Handle int-only local optimizations safely
Some raw-file and file-index optimizations still accept an Integer limit.
These optimizations are used only when the requested limit can be represented
safely as an integer.
For larger limits, the int-only optimization is skipped, while the
authoritative long limit is still applied by the final reader. This may avoid
an optimization for unusually large limits, but it preserves correct results
and avoids unsafe narrowing.
#### Make counters and split selection overflow-safe
The change also updates related limit handling to avoid secondary overflow
problems:
- the query-authorization limiting reader now uses a long record counter;
- split-row accumulation compares against the remaining limit before
addition, avoiding overflow near Long.MAX_VALUE ;
- the final LimitRecordReader accepts a long limit.
### Tests
Added and updated regression coverage for:
- forwarding 4294967297L unchanged from FlinkSourceBuilder to
ReadBuilder ;
- propagating a large limit from ReadBuilderImpl to both the scan and
reader;
- delegating integer-range long limits through the compatibility bridge;
- safely leaving oversized limits unpushed for legacy int-only
implementations;
- applying a limit above the integer range without narrowing it in
LimitRecordReader ;
- retaining the legacy serialized Integer limit field in both
serializable read-builder implementations.
The focused tests were run with both supported Flink profiles.
mvn -pl paimon-core -am \
-DfailIfNoTests=false \
-DwildcardSuites=none \
-Dtest=LimitRecordReaderTest,ReadBuilderImplTest,FormatReadBuilderTest
test
mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
-DfailIfNoTests=false \
-DwildcardSuites=none \
-Dtest=FlinkSourceBuilderTest#testLongLimitForwardedToReadBuilder test
mvn -pl paimon-flink/paimon-flink-common -am -Pflink2 \
-DfailIfNoTests=false \
-DwildcardSuites=none \
-Dtest=FlinkSourceBuilderTest#testLongLimitForwardedToReadBuilder test
Formatting was applied to the modified modules:
mvn -pl paimon-common,paimon-core,paimon-flink/paimon-flink-common
spotless:apply
#### Notes for reviewers
The main review points are:
1. ReadBuilder.withLimit(int) is intentionally retained because replacing
it with a long-only signature would change the JVM method descriptor and break
existing binaries.
2. The default long overload intentionally skips pushdown for an oversized
value when used by a legacy int-only implementation. It must not narrow the
value.
3. Paimon's implementations override the long overload, so normal Paimon
table reads preserve large limits end to end.
4. The legacy serialized Integer limit fields are intentionally retained
to support deserializing previously serialized read builders.
5. Top-N, vector-search, full-text-search, and hybrid-search limits remain
int ; they are bounded top-k APIs and are separate from the general relational
row-limit path.
6. Int-only file-index limit optimization is skipped for oversized values,
but the final long reader limit remains active and authoritative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
