lilei1128 opened a new pull request, #11214:
URL: https://github.com/apache/paimon/pull/11214

   ### Purpose
     When a file's deletions form one hole-free contiguous span covering at
     least half of its rows, prune that span from the Parquet read as row
     ranges. Default off via data-evolution.data-file-dv-pushdown.enabled;
     scattered deletions keep the regular path, and the deletion vector is
     still applied on top, so results never change.
   
     Benchmark (Parquet files, bytes counted at FileIO):
     - 240k rows, 64KB row groups / 4KB pages: clustered 50/75/90/99%
       deletion 1.2-15.0x faster, 50-98.5% fewer bytes
     - 4M rows, 32MB row groups / 256KB pages: clustered 50/75/90/99%
       deletion 1.75-18.8x faster, 49.8-98.8% fewer bytes
     - random deletion at any ratio: 0.97-1.06x, bytes unchanged (gate
       rejects)
   
   ### Tests
    Tests:
     - Gate unit tests: contiguous prefix/suffix/middle/full, exact ratio
       boundary, holes, scattered, out-of-file positions, misaligned range
       (merged-group non-anchor), Bitmap32/Bitmap64, small-file rounding.
     - E2E test with byte counting: identical results with the pushdown on
       and off; contiguous + enabled reads fewer bytes; scattered + enabled
       reads exactly the fallback bytes; fully deleted + enabled reads zero
       (pinned with file-index read disabled).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to