kerwin-zk opened a new pull request, #8619:
URL: https://github.com/apache/paimon/pull/8619
### Purpose
When a Daft query reads a Paimon blob table but does **not** project any
BLOB column
(e.g. metadata scans / filters / aggregations), the scan currently falls
back to the
pypaimon Python reader for the whole split, only because the table *schema*
contains a
BLOB column. Blob columns are stored in separate `.blob` bunch files while
scalar columns
live in the base parquet, so those non-blob queries can be served by Daft's
native
(Rust) parquet reader.
This PR makes the reader routing **projection-aware**: for a non-PK parquet
blob table
without deletion vectors and with an explicit projection that selects no
BLOB column, the
covering parquet files are read via the native parquet reader; the skipped
`.blob` /
`.vector` files are not read.
Correctness is conservative and self-evident: the native path only fires
when the covering
parquet files (a) each fully cover the projection, (b) have disjoint row-id
ranges, and
(c) together span the split's full row-id union. Any partial coverage,
cross-file field
merge, row-range overlap, schema-evolution gap, deletion vectors, PK table,
or missing
projection falls back to pypaimon unchanged — so the native result is
provably identical
to the fallback.
Benchmark (200k rows, blob table, only scalar columns projected; native vs
forced fallback,
same query):
| query | native | pypaimon fallback | speedup |
|-------|--------|-------------------|---------|
| full scalar scan | 6.8 ms | 56.6 ms | 8.3x |
| filter `score > 900` | 7.0 ms | 68.9 ms | 9.8x |
| `group by category` agg | 6.5 ms | 24.4 ms | 3.7x |
`explain_paimon_scan(verbose=True)` is kept consistent: a blob-native split
reports only the
natively read parquet files' `file_count` / `file_size` / `file_paths` /
`row_count`
(the Paimon split row_count double-counts rows across the parquet and
`.blob` bunches).
### Tests
`paimon-python/pypaimon/tests/daft/daft_blob_native_read_test.py`:
- Pure-function coverage of `_blob_native_covering_files`: scalar projection
returns covering
parquet; blob column projected / projected column only in a `.vector` file
/ partial
parquet coverage (field merge) / overlapping row ranges / missing
`first_row_id` /
uncovered row range (schema evolution) all fall back; partition columns
excluded from
coverage.
- End-to-end: scalar projection uses the native reader and returns correct
rows; multi-append
concat (`raw_convertible=False`) native + correct; partitioned blob table
native with
correct partition values; blob projection and no-projection fall back;
schema-evolution
cases assert the Daft result equals the pypaimon reader result; verbose
split detail
excludes `.blob` files and its row_count matches the natively read rows.
### API and Format
No public API or format change. Only affects Daft reader routing for blob
tables.
### Documentation
No documentation change needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
