skyyyy955 opened a new pull request, #11167:
URL: https://github.com/apache/paimon/pull/11167

   ### Purpose
   
   Fixes #11165.
   
   Data Evolution scans can discard matching rows when a row-range selection is 
combined with LIMIT or TopN. For example, selecting one row from each of two 
256-row files and requesting LIMIT 2 returns only the first selected row. The 
lower scan counts the whole first file toward the limit before the row-range 
selection is applied. TopN can similarly discard the file containing the 
smallest selected value because its statistics include unselected rows.
   
   Skip these two file-level pushdowns in `DataEvolutionBatchScan.planEager` 
when `rowRangeIndex` is present. Keep the existing row-range pruning, 
deferred-filter checks, and global-index TopN handling. Ordinary LIMIT and TopN 
without row ranges retain their existing fast paths.
   
   The change touches one production class and adds one real-table regression 
class. It changes no public API, storage format, or write/commit behavior. 
Queries with row ranges may read more candidate files or metadata to produce 
correct results; no performance improvement is claimed.
   
   ### Tests
   
   Against master `9b6c997198ea31584579a4bbefb239f052f40bbd`, four of the eight 
new test invocations fail. The same eight pass with the fix. They cover 
explicit row ranges with LIMIT, a supplied `GlobalIndexResult` with LIMIT, TopN 
with and without a separate LIMIT, ordinary LIMIT/TopN pruning, a data 
predicate, and empty/out-of-range selections.
   
   Normal Maven validation on Windows with JDK 8u451 and Maven 3.8.9 passed 
**26 tests, with no failures, errors, or skips**:
   
   - `DataEvolutionRowRangeLimitTest`: 8.
   - `DataEvolutionBatchScanTest`: 9.
   - `AppendOnlyLimitManifestReadTest`: 9.
   
   Checkstyle, Spotless, and Enforcer passed. After building the core 
dependencies, the focused test command is:
   
   ```sh
   mvn -B -ntp -pl paimon-core -DwildcardSuites=none -Dflink.forkCount=1 
-Dtest=DataEvolutionRowRangeLimitTest,DataEvolutionBatchScanTest,AppendOnlyLimitManifestReadTest
 test
   ```
   
   This is local focused validation, not the full CI matrix. The new tests 
write and read real Parquet tables; the supplied-index case constructs row IDs 
without an external service. Linux, engine integrations, deletion vectors, and 
partial-column updates have not been validated for this patch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to