jay-dee7 opened a new issue, #25499:
URL: https://github.com/apache/datafusion/issues/25499
### Is your feature request related to a problem or challenge?
Yes, while tuning a Parquet layout in our production (bloom filter sizing
per row group, page-index cost per file, verifying writer-declared sort order),
**_parquet_metadata()_** could not answer any of it.
The parquet crate already carries every value on:
1. `ColumnChunkMetaData`
- bloom_filter_offset/length
- column_index_offset/length
- offset_index_offset/length
2. `RowGroupMetaData`
- sorting_columns
- ordinal
- file_offset
3. `FileMetaData`
- created_by
- version
- num_rows
- key_value_metadata
but the CLI function projects none of them. We had to install DuckDB on the
host to read bloom filter lengths, and there is no way at all to read key-value
footer metadata from the CLI.
### Describe the solution you'd like
Three additive changes, in order of size:
1. New columns on `parquet_metadata()`
- per column chunk:
- [ ] bloom_filter_offset
- [ ] bloom_filter_length
- [ ] column_index_offset
- [ ] column_index_length
- [ ] offset_index_offset
- [ ] offset_index_length
- per row group:
- [ ] sorting_columns (list of {column_idx, descending, nulls_first} or a
string)
- [ ] row_group_ordinal
- [ ] row_group_file_offset
2. A new `parquet_file_metadata(path)` table function, one row per file:
- [ ] created_by
- [ ] version
- [ ] num_rows
- [ ] num_row_groups
- [ ] key_value_metadata (map or JSON string)
- [ ] footer length
3. A new `parquet_page_index(path)` table function, one row per page:
- [ ] row_group_id
- [ ] column_id
- [ ] page_ordinal
- [ ] first_row_index
- [ ] offset
- [ ] compressed_page_size
- [ ] column index's min, max, null_count, is_min_max_exact for that page
Item 1 is a pure projection change on the existing function. Items 2 and 3
are new functions because their row grain differs.
### Describe alternatives you've considered
- DuckDB's `parquet_metadata()` / `parquet_file_metadata()` /
`parquet_kv_metadata()`, which expose most of this. It works, but requires a
second engine on the host just to inspect files DataFusion wrote.
- A small Rust binary over the parquet crate. Trivial to write, but every
team inspecting DataFusion output ends up writing the same one.
- Reading the footer with parquet-layout / parquet-index from the parquet
crate's CLI tools. Useful, but not queryable with SQL alongside the data, which
is what makes `parquet_metadata()` valuable.
### Additional context
- All values are already decoded when the footer is read; no extra I/O for
item 1, and item 3 needs only the page index the reader can already load.
- Concrete uses that motivated this: confirming per-row-group bloom filter
sizes after changing NDV hints, measuring page-index bytes per file to decide
whether to enable it per column, and checking that a compactor preserved the
writer's sorting_columns before trusting a merge on it.
- Happy to send PRs, split as three small ones in the order above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
