skyyyy955 opened a new issue, #11165:
URL: https://github.com/apache/paimon/issues/11165
### Search before asking
- [x] I searched the issues and pull requests and did not find a report of
this specific combination.
### Paimon version
Master, `9b6c997198ea31584579a4bbefb239f052f40bbd` (`2.2-SNAPSHOT`).
### Compute Engine
Java API, JDK 8u451 on Windows, using a filesystem catalog and Parquet files.
### Minimal reproduce step
A Java caller can fetch selected row IDs with `withRowRanges` and limit the
result. On a Data Evolution table, this can lose rows from later files.
Create an unpartitioned table with one `id INT` column, no primary key, and
these options:
```text
bucket=-1
file.format=parquet
write-only=true
data-evolution.enabled=true
row-tracking.enabled=true
deletion-vectors.enabled=false
```
Using `newBatchWriteBuilder`, write and commit two batches separately:
values `0..255`, then `256..511`. Each batch produces one data file. The table
has 512 rows; row IDs 0 and 256 contain values 0 and 256.
Plan and read the following request:
```java
ReadBuilder read =
table.newReadBuilder()
.withRowRanges(Arrays.asList(new Range(0, 0), new Range(256,
256)))
.withLimit(2);
TableScan.Plan plan = read.newScan().plan();
List<Integer> ids = new ArrayList<>();
try (CloseableIterator<InternalRow> rows =
read.newRead().executeFilter().createReader(plan).toCloseableIterator()) {
while (ids.size() < 2 && rows.hasNext()) {
ids.add(rows.next().getInt(0));
}
}
```
This returns only `[0]`. Removing `withLimit(2)` returns both selected
values, `[0, 256]`. Passing the same row IDs through
`DataEvolutionBatchScan.withGlobalIndexResult(GlobalIndexResult.fromRanges(...))`
also loses the second value when combined with LIMIT.
TopN has the same problem with file statistics. In a fresh table, write
`0..255` in the first file and `100..355` in the second. Set
`SOURCE_SPLIT_TARGET_SIZE` to `"1 b"` and confirm that the scan has two splits.
Select row IDs 255 and 256, whose values are 255 and 100, then request
ascending Top1 on `id` with NULLS LAST. After sorting the returned candidates
and taking one row, the result is `[255]` instead of `[100]`. This fails both
with and without a separate `withLimit(1)`.
### What doesn't meet your expectations?
LIMIT should retain enough candidates to return the two selected rows. TopN
should retain the row containing the smallest selected value. Here the scan has
already discarded that row's file, so the consumer cannot recover it.
### Anything else?
In `DataEvolutionBatchScan.planEager`, LIMIT and TopN reach the underlying
batch scan before its splits are wrapped with the row-range selection.
Whole-file row counts and statistics include rows that the selection excludes,
so they are not safe for this pruning.
I have a small fix that skips these two pushdowns when `rowRangeIndex` is
present. Ordinary LIMIT and TopN pruning stays enabled when no row ranges are
supplied. Selected-row queries may need to read more candidate files; I have
not measured that cost.
The real-table regression has eight test invocations: four fail on the
unmodified commit and all eight pass with the fix. Including the existing scan
and append-only LIMIT suites, 26 tests pass with normal Maven checks. The
supplied-index-result case constructs valid row IDs; it does not test an
external index service. These results do not establish a Flink or Spark SQL
failure.
I also checked #11156: its metadata-memory changes do not address these two
pushdowns. If this scope looks right, could you assign this issue to me? I can
send the two-file fix and regression tests.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]