q8webmaster opened a new pull request, #130:
URL: https://github.com/apache/paimon-trino/pull/130
Fixes #129.
### Problem
`TrinoPageSourceWrapper` checks each row against the file's deletion vector
at
`source.getCompletedPositions() + pagePosition`, i.e. the number of rows the
page source has
returned so far. A deletion vector is indexed by the row's position in the
data file. When a
filter is pushed down to the ORC reader (`TupleDomainOrcPredicate`), Trino
skips stripes and row
groups, the rows returned so far no longer equal the file position, and
every later row is checked
against another row's bit. Filtered queries on primary-key tables with
`deletion-vectors.enabled=true` and `file.format=orc` then drop live rows
and return deleted or
stale (pre-update) rows. Unfiltered reads are correct.
### Change
- New `TrinoOrcPageSource extends OrcPageSource`, which also exposes
`OrcRecordReader#getFilePosition()`: the file position of the first row of
the page the reader
returned last. Trino's own Hive `OrcPageSource` uses the same value to
find deleted rows of ACID
tables.
- `TrinoPageSourceProvider` creates a `TrinoOrcPageSource` instead of an
`OrcPageSource` (same
arguments).
- `TrinoPageSourceWrapper` uses that file position as the start position of
the page when the
source is a `TrinoOrcPageSource`, and keeps `getCompletedPositions()` for
other sources. The
start position is now a `long`.
### Tests
New `TrinoITCase#testDeletionVectorWithOrcFilterPushdown` with a new table
`orc_dv_table`
(one bucket, `deletion-vectors.enabled=true`, `file.format=orc`, row index
stride 10,000):
- Commit 1 writes ids 0..49,999, compacted into one top-level file with five
row groups.
- Commit 2 deletes every id divisible by 7 and updates the other ids
divisible by 11. Its
compaction writes the changes to a level-4 file and a deletion vector on
the first file. The
test asserts this layout (`$files` and `$table_indexes`), so it cannot
silently stop covering
deletion vectors. The values of `v` are scrambled on purpose: with `v =
id` the first file
compresses to ~2 KB, the change file is larger, and the compaction
rewrites everything instead
of writing a deletion vector.
- It compares `count(*)`, `count(DISTINCT id)`, `sum(id)` and `sum(v)` for
`id > 25000` (pushed
down: row groups 0 and 1 are skipped) and `id + 0 > 25000` (not pushed
down) against the values
computed in the test.
Without the fix the pushed-down query fails:
```
expected: "[[21428, 21428, 803553571, 1987230923]]"
but was: "[[21428, 19805, 803538954, -5755670142]]"
```
With the fix it passes. `mvn clean install -P '!toolchain' -Papache-build`
(the CI command) passes
locally on JDK 24: checkstyle and spotless clean, 176 tests, 0 failures.
### Build note (separate commit)
`main` does not build at the moment: its parent
`org.apache.paimon:paimon-parent:1.3-SNAPSHOT` is
no longer in the Apache snapshots repository. The first commit switches the
parent to the released
`1.3.1` (matching `paimon.version`), the same kind of change #125 and #127
make (`1.4.1`,
`1.4.2`). Happy to drop it if you prefer to fix the parent separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]