bhavya-sl opened a new issue, #55325: URL: https://github.com/apache/arrow/issues/55325
### Describe the enhancement requested When a dataset has a `_metadata` file, every file's row-group metadata is already available in a single object. From Python there are only two ways to use it, and they sit at opposite extremes: 1. **`ds.parquet_dataset(metadata_path)`** — `ParquetDatasetFactory` pre-populates every fragment, so `fragment.metadata` is free and no file footer is ever read. But all of that metadata stays resident for the dataset's lifetime. 2. **Build fragments yourself** with `ParquetFileFormat.make_fragment(path, filesystem, partition_expression, row_groups)` — memory stays flat, but each fragment re-reads that file's footer on first scan, re-fetching data the caller already had. There is no middle option: a caller cannot say "here is the `FileMetaData` for this one file, don't go and read it again". Measured on a 689-file dataset over S3 (each file ~190 row groups, 13 columns, ~240 KB footer): | | setup | parent peak RSS | `fragment.metadata` | |---|---|---|---| | hand-built fragments | 2.4 s | **149 MB** | **44.6 ms per file** | | `ds.parquet_dataset()` | 3.2 s | **1850 MB** | **0.0 ms** | Over a query touching 200 files that is ~25 s of latency against ~1.7 GB of permanent RSS. For a job running under a memory cap the fast path is simply unavailable, even though the metadata had already been read once in order to build the fragment list at all. ### Why this cannot be worked around in Python today - `pq.ParquetFile(source, metadata=...)` **does** accept a pre-loaded `FileMetaData` and skips the footer read — 11.9 ms versus 66.1 ms in my measurements, i.e. no I/O at all. But that is the low-level reader, outside the dataset API, so it gives up filter pushdown and row-group selection. - `ParquetFileFragment`'s constructor and `SetMetadata()` are private, with `ParquetDatasetFactory` as the only friend (`cpp/src/arrow/dataset/file_parquet.h`). - `FileMetaData` exposes `append_row_groups`, `row_group(i)`, `set_file_path` and `write_metadata_file`, but **no way to slice one file's row groups out of a combined `_metadata`** into a new `FileMetaData`. So even if an API existed to accept per-file metadata, a caller holding `_metadata` could not construct the object to pass it. The C++ pieces already exist and are public — `ParquetFileFormat::GetReader(source, options, metadata)` takes a `FileMetaData`, and `ParquetFileFragment::EnsureCompleteMetadata(FileReader*)` takes a reader — they are simply not wired together through Python. ### Suggested shape Either of these would close the gap: ```python # a) accept metadata at fragment construction fmt.make_fragment(path, filesystem, partition_expression, row_groups, metadata=file_metadata) # b) or let an existing fragment adopt it fragment.ensure_complete_metadata(metadata=file_metadata) ``` To make either usable from a combined `_metadata`, a way to obtain a single file's `FileMetaData` from it would also be needed — for example `FileMetaData.subset(row_group_indices)`, or a helper yielding per-file `FileMetaData` keyed by `file_path`. With both, a caller could parse `_metadata` once in a short-lived subprocess, keep only the serialised per-file blobs (`FileMetaData.write_metadata_file` round-trips through a buffer and re-injects in ~14 ms), and get `parquet_dataset()`'s zero-footer-read behaviour at the hand-built path's memory profile. Related: #55324 reduces the cost of the footer read when it does happen; this issue is about not needing it at all. ### Component(s) Python, Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
