mixermt commented on issue #4859:
URL:
https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4913616465
Some observations after LLM iterations
`Generated by LLM, reviewed by human`
I tested the fix direction suggested here (option 3: pass the nested-pruned
`required_schema` to DataFusion as the scan's file schema, instead of full
`data_schema` + top-level projection vector) and measured native
`bytes_scanned` directly. Results with DataFusion 54, a ~2 MB file of 2000 rows
with a ~1 KB `payload` leaf next to a small `id` leaf:
| shape | bytes_scanned, pruned schema | bytes_scanned, full schema |
|---|---|---|
| `event: struct<id,payload>` → `struct<id>` | 2,045,604 | 2,045,604 |
| `events: array<struct<id,payload>>` → `array<struct<id>>` | 2,045,618 |
2,045,618 |
| top-level control: `(id, payload)` → `id` | 18,794 | 2,045,597 |
The pruned read schema fixes the scan's **output**: batches now carry
exactly Spark's pruned `ReadSchema` (this also removes a latent wrong-results
hazard, since downstream expressions bind struct fields by ordinal against the
pruned schema, which was only accidentally correct when the pruned fields were
a positional prefix of the file struct, as they are in this report). But it
does **not** reduce Parquet I/O. Unrequested nested leaves are still fetched
and decoded, then dropped in memory by the schema adapter.
Root cause: DataFusion's `ParquetOpener` derives the leaf-level
`ProjectionMask` from the projection and filter *expressions*
(`build_projection_read_plan` in
`datafusion-datasource-parquet/src/row_filter.rs`), and only `get_field(column,
...)` chains produce leaf masks. A pruned-struct-vs-full-struct schema
difference is instead handled by the `PhysicalExprAdapter` as a whole-column
cast, which the pushdown checker cannot see through, so it falls back to
`ProjectionMask::roots` and reads the entire column chunk.
The reader is not the limitation. parquet-rs `ProjectionMask::leaves`
handles nested pruning, and DataFusion already uses it for pushed-down struct
filters. The gap is purely that nothing translates a clipped logical file
schema into a leaf mask. So fully fixing this issue needs one of:
1. Upstream DataFusion: derive `ProjectionMask::leaves` from the
logical-vs-physical file schema diff (the equivalent of Spark's
`clipParquetSchema`). Benefits all DataFusion users.
2. Comet-side: a custom opener/`FileSource` that computes the leaf mask from
`required_schema` against the file's `SchemaDescriptor` by path matching. No
upstream dependency, more maintenance.
3. Partial only: DF 54's `FileSource::try_pushdown_projection` accepts
expression projections, so plain struct roots could be rewritten as
`struct(get_field(col,'x'), ...)`. But there is no expression form for pruning
through `array<struct>` or map values, which is exactly the shape in this
report, so this cannot fix it alone.
Passing the pruned schema is still worth doing for the schema-correctness
fix, just with the expectation set that stage input bytes will not improve
until the mask-level work lands.
@andygrove @mbutrovich any guide from you will be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
