q8webmaster opened a new pull request, #130:
URL: https://github.com/apache/paimon-trino/pull/130

   Fixes #129.
   
   ### Problem
   
   `TrinoPageSourceWrapper` checks each row against the file's deletion vector 
at
   `source.getCompletedPositions() + pagePosition`, i.e. the number of rows the 
page source has
   returned so far. A deletion vector is indexed by the row's position in the 
data file. When a
   filter is pushed down to the ORC reader (`TupleDomainOrcPredicate`), Trino 
skips stripes and row
   groups, the rows returned so far no longer equal the file position, and 
every later row is checked
   against another row's bit. Filtered queries on primary-key tables with
   `deletion-vectors.enabled=true` and `file.format=orc` then drop live rows 
and return deleted or
   stale (pre-update) rows. Unfiltered reads are correct.
   
   ### Change
   
   - New `TrinoOrcPageSource extends OrcPageSource`, which also exposes
     `OrcRecordReader#getFilePosition()`: the file position of the first row of 
the page the reader
     returned last. Trino's own Hive `OrcPageSource` uses the same value to 
find deleted rows of ACID
     tables.
   - `TrinoPageSourceProvider` creates a `TrinoOrcPageSource` instead of an 
`OrcPageSource` (same
     arguments).
   - `TrinoPageSourceWrapper` uses that file position as the start position of 
the page when the
     source is a `TrinoOrcPageSource`, and keeps `getCompletedPositions()` for 
other sources. The
     start position is now a `long`.
   
   ### Tests
   
   New `TrinoITCase#testDeletionVectorWithOrcFilterPushdown` with a new table 
`orc_dv_table`
   (one bucket, `deletion-vectors.enabled=true`, `file.format=orc`, row index 
stride 10,000):
   
   - Commit 1 writes ids 0..49,999, compacted into one top-level file with five 
row groups.
   - Commit 2 deletes every id divisible by 7 and updates the other ids 
divisible by 11. Its
     compaction writes the changes to a level-4 file and a deletion vector on 
the first file. The
     test asserts this layout (`$files` and `$table_indexes`), so it cannot 
silently stop covering
     deletion vectors. The values of `v` are scrambled on purpose: with `v = 
id` the first file
     compresses to ~2 KB, the change file is larger, and the compaction 
rewrites everything instead
     of writing a deletion vector.
   - It compares `count(*)`, `count(DISTINCT id)`, `sum(id)` and `sum(v)` for 
`id > 25000` (pushed
     down: row groups 0 and 1 are skipped) and `id + 0 > 25000` (not pushed 
down) against the values
     computed in the test.
   
   Without the fix the pushed-down query fails:
   
   ```
   expected: "[[21428, 21428, 803553571, 1987230923]]"
    but was: "[[21428, 19805, 803538954, -5755670142]]"
   ```
   
   With the fix it passes. `mvn clean install -P '!toolchain' -Papache-build` 
(the CI command) passes
   locally on JDK 24: checkstyle and spotless clean, 176 tests, 0 failures.
   
   ### Build note (separate commit)
   
   `main` does not build at the moment: its parent 
`org.apache.paimon:paimon-parent:1.3-SNAPSHOT` is
   no longer in the Apache snapshots repository. The first commit switches the 
parent to the released
   `1.3.1` (matching `paimon.version`), the same kind of change #125 and #127 
make (`1.4.1`,
   `1.4.2`). Happy to drop it if you prefer to fix the parent separately.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to