skyyyy955 opened a new issue, #11165:
URL: https://github.com/apache/paimon/issues/11165

   ### Search before asking
   
   - [x] I searched the issues and pull requests and did not find a report of 
this specific combination.
   
   ### Paimon version
   
   Master, `9b6c997198ea31584579a4bbefb239f052f40bbd` (`2.2-SNAPSHOT`).
   
   ### Compute Engine
   
   Java API, JDK 8u451 on Windows, using a filesystem catalog and Parquet files.
   
   ### Minimal reproduce step
   
   A Java caller can fetch selected row IDs with `withRowRanges` and limit the 
result. On a Data Evolution table, this can lose rows from later files.
   
   Create an unpartitioned table with one `id INT` column, no primary key, and 
these options:
   
   ```text
   bucket=-1
   file.format=parquet
   write-only=true
   data-evolution.enabled=true
   row-tracking.enabled=true
   deletion-vectors.enabled=false
   ```
   
   Using `newBatchWriteBuilder`, write and commit two batches separately: 
values `0..255`, then `256..511`. Each batch produces one data file. The table 
has 512 rows; row IDs 0 and 256 contain values 0 and 256.
   
   Plan and read the following request:
   
   ```java
   ReadBuilder read =
           table.newReadBuilder()
                   .withRowRanges(Arrays.asList(new Range(0, 0), new Range(256, 
256)))
                   .withLimit(2);
   TableScan.Plan plan = read.newScan().plan();
   List<Integer> ids = new ArrayList<>();
   try (CloseableIterator<InternalRow> rows =
           
read.newRead().executeFilter().createReader(plan).toCloseableIterator()) {
       while (ids.size() < 2 && rows.hasNext()) {
           ids.add(rows.next().getInt(0));
       }
   }
   ```
   
   This returns only `[0]`. Removing `withLimit(2)` returns both selected 
values, `[0, 256]`. Passing the same row IDs through 
`DataEvolutionBatchScan.withGlobalIndexResult(GlobalIndexResult.fromRanges(...))`
 also loses the second value when combined with LIMIT.
   
   TopN has the same problem with file statistics. In a fresh table, write 
`0..255` in the first file and `100..355` in the second. Set 
`SOURCE_SPLIT_TARGET_SIZE` to `"1 b"` and confirm that the scan has two splits. 
Select row IDs 255 and 256, whose values are 255 and 100, then request 
ascending Top1 on `id` with NULLS LAST. After sorting the returned candidates 
and taking one row, the result is `[255]` instead of `[100]`. This fails both 
with and without a separate `withLimit(1)`.
   
   ### What doesn't meet your expectations?
   
   LIMIT should retain enough candidates to return the two selected rows. TopN 
should retain the row containing the smallest selected value. Here the scan has 
already discarded that row's file, so the consumer cannot recover it.
   
   ### Anything else?
   
   In `DataEvolutionBatchScan.planEager`, LIMIT and TopN reach the underlying 
batch scan before its splits are wrapped with the row-range selection. 
Whole-file row counts and statistics include rows that the selection excludes, 
so they are not safe for this pruning.
   
   I have a small fix that skips these two pushdowns when `rowRangeIndex` is 
present. Ordinary LIMIT and TopN pruning stays enabled when no row ranges are 
supplied. Selected-row queries may need to read more candidate files; I have 
not measured that cost.
   
   The real-table regression has eight test invocations: four fail on the 
unmodified commit and all eight pass with the fix. Including the existing scan 
and append-only LIMIT suites, 26 tests pass with normal Maven checks. The 
supplied-index-result case constructs valid row IDs; it does not test an 
external index service. These results do not establish a Flink or Spark SQL 
failure.
   
   I also checked #11156: its metadata-memory changes do not address these two 
pushdowns. If this scope looks right, could you assign this issue to me? I can 
send the two-file fix and regression tests.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to