beinan opened a new pull request, #10153:
URL: https://github.com/apache/paimon/pull/10153
### Purpose
Avoid scheduling every manifest read before an append-only LIMIT scan can
stop.
`AppendOnlyFileStoreScan.readManifestEntries` already stops consuming
entries once it has enough rows, but its normal planning path first calls
`super.readManifestEntries(..., false)`. That eagerly submits reads for all
candidate manifests, so even `LIMIT 1` can perform table-wide manifest I/O.
Use the existing sequential batched reader when the scan is eligible for
limit pruning. The batch size continues to follow `scan.manifest.parallelism`;
no new option is introduced. Scans with no positive limit, data predicates,
deletion vectors, or data evolution keep their existing path. Primary-key scans
are unchanged.
The normal ADD/DELETE merge is preserved: manifests containing DELETE
entries are still processed before selecting live ADD entries. Manifest lists
and required deletion metadata are still read, and the current batch may finish
after the limit is reached. This change bounds ADD-manifest scheduling rather
than promising exactly one physical read for every LIMIT query.
### Tests
Add `AppendOnlyLimitManifestReadTest`, which writes real Avro data and
manifests, counts manifest input-stream opens, and reads the planned data. It
covers partitioned/unpartitioned tables, limits of 1/10/larger than the table,
no limit, partition/data predicates, later deletion entries, and an empty
table. The consuming reader enforces the global limit, as scan pruning retains
whole files.
With 20 manifests, four rows per data file, and
`scan.manifest.parallelism=1`:
| Query shape | Before | After |
| --- | ---: | ---: |
| LIMIT 1 | 20 manifest reads | 1 |
| LIMIT 10 | 20 manifest reads | 3 |
| Unpartitioned LIMIT 10 | 20 manifest reads | 3 |
| Partition predicate + LIMIT 10 | 10 manifest reads | 3 |
Negative control: all four I/O regressions fail with the original
implementation, with no test errors. All nine new test cases pass with the fix.
The final focused suite passed **56 tests, zero failures/errors/skips**,
including existing table-scan, append-only, primary-key, deletion-vector, and
data-evolution LIMIT cases. The run used the normal Maven profile, so
Checkstyle, Spotless, and enforcer checks also passed.
```sh
# The codegen loader embeds classes during prepare-package; do this once on
a fresh checkout.
mvn -B -pl paimon-core -am -Pfast-build -DskipTests package
mvn -B -pl paimon-core -am \
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-DwildcardSuites=none \
'-Dtest=AppendOnlyLimitManifestReadTest,TableScanTest,AppendOnlySimpleTableTest#testLimit*,PrimaryKeySimpleTableTest#testLimitPushDownInDeletionVectorMode+testReadWithLimit*,KeyValueFileStoreScanTest#*Limit*,DataEvolutionTableTest#testLimit*,DataEvolutionDeletionVectorTest#testLimit*'
\
test
```
Validation used JDK 17 targeting Java 8, with actual local filesystem data.
It did not run a StarRocks SQL cluster or object-store performance benchmark.
Downstream engines such as StarRocks must consume a release/backport containing
this SDK change to benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
