HippoBaro opened a new issue, #12364: URL: https://github.com/apache/arrow-rs/issues/12364
### Is your feature request related to a problem or challenge? I work on a database that merges files belonging to the same table using forward-only input cursors and a k-way merger. We have described this system [in a public blog post](https://www.datadoghq.com/blog/engineering/husky-storage-compaction/). Our existing fragment format places a header before each column, allowing compaction to discover columns during a single forward pass. I would like to support the same access pattern with Parquet. For the corresponding Parquet layout, each file would contain one row group, with column chunks laid out in lexicographic order. Each cursor discovers its next column's identity and presents it to the merger. The merger determines which cursors to advance, combines matching columns, and produces another lexicographically ordered sequence of columns. Each input is consumed in one forward pass. This lets us merge hundreds of input fragments without buffering their full contents, with working memory bounded by the active per-input buffers rather than the total input size. **Exactly one streaming GET request per input file is a hard requirement.** Cloud providers charge per GET request, so an additional request to fetch a footer, index, or sidecar is not acceptable for this workload. The information needed to process each column must arrive in that same forward stream. Parquet's standard page headers do not identify their columns. That information normally comes from the schema and column metadata in the footer. I would like to repeat a column's identity and application-specific decoding descriptor in its first page header, making the information available where the cursor encounters the data. The capability I am requesting is attaching application metadata within the existing Parquet structure whose meaning it enriches, and accessing it as that structure is decoded. For example, a `DOUBLE` column named `speed` does not distinguish metres per second from kilometres per hour. An application may want to annotate that column's `SchemaElement` with its measurement unit. This adds application-specific meaning to the existing type without introducing a new standard Parquet logical type. A reader that understands the annotation can validate or interpret it while reading the schema; the standard Parquet type remains `DOUBLE`. Such an annotation may contain only a few bytes, and its meaning depends on the particular schema element carrying it. Storing it as a separate binary block located through footer metadata as in https://github.com/apache/datafusion/pull/16395 requires the application to recreate that association. Attaching it directly to the schema element preserves the association through the file's existing structure. This complements the embedded-index approach demonstrated in https://github.com/apache/datafusion/pull/16395. Separately located binary regions are well suited to substantial payloads (such as indexes) that readers may fetch independently. In-structure extensions are a natural fit for small, contextual annotations on schema elements, page headers, and other metadata structures. ### Describe the solution you'd like Support application-defined metadata using the existing [Parquet Binary Protocol Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md) mechanism. Applications should be able to: - Attach opaque payloads to ordinary Parquet Thrift structs, including schema elements and page headers. - Opt in to observing extension payloads during ordinary metadata decoding. - Identify the structure and column or page to which a payload belongs. - Restrict processing to relevant locations. - Report errors when a recognized annotation is invalid or incompatible with the application. Payload meaning and encoding should remain application-defined, as per the standard. The mechanism should support different applications without requiring the crate to standardize their descriptors, annotations, or indexes. For our streaming-merge use case, the descriptor should be available on the first encountered page of a column chunk. Repeating it on a dictionary page and the first data page is acceptable. ### Describe alternatives you've considered - **Read or cache the footer first.** This is the reader model described by the [Parquet format documentation](https://github.com/apache/parquet-format/blob/master/README.md#file-format), but a separate footer fetch violates our one-GET requirement. Reading the whole object in one GET and waiting for its footer instead requires buffering or spooling data for later processing, rather than consuming it in one forward pass. - **Embed binary regions and locate them through footer metadata.** https://github.com/apache/datafusion/pull/16395 demonstrates this approach using the raw-writing support added in #7714. It is useful for independently fetched indexes, but obtaining the footer locators before consuming column data has the same upfront metadata-access requirement. It also does not attach the payload directly to a schema element or page header. - **Interleave self-framed blobs with column chunks.** A variation on https://github.com/apache/datafusion/pull/16395 would place an application-framed descriptor immediately before each column chunk. Our cursor could discover the descriptor and the following chunk in one forward GET, without footer references to those blobs. The file can remain format-compliant: column chunks stay contiguous, their offsets and sizes remain correct, and unaware metadata-directed readers skip the extra regions. This would satisfy our streaming requirements. It is less general, however: it introduces an application-specific framing and layout convention rather than annotating existing structures, and does not directly attach metadata to nested objects such as individual `SchemaElement`s. - **Existing key/value metadata or sidecars.** Footer key/value metadata is not available when the forward cursor encounters the column data, and file-level entries require an application convention to associate annotations with schema fields. A separately fetched sidecar adds a request and requires separate distribution and lifecycle management. Preexisting external metadata can help, but does not guarantee our one-GET access pattern for arbitrary inputs. [Binary Protocol Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md) are our preferred solution because the extension point and its framing are already part of Parquet. Applications can attach opaque metadata directly to existing ordinary Thrift metadata structures, from schema elements and footer metadata to page and index headers, while preserving the context in which that metadata belongs. Rather than requiring each application to define another convention for positioning, framing, and associating separate byte regions, the crate should expose this existing standardized capability. The payload's meaning remains application-defined, but the mechanism carrying it does not need reinventing. ### Additional context There is relevant prior work that this proposal should complement: - **Metadata indexing:** @etseidl's #8713 and #8714 demonstrate using Binary Protocol Extensions for a metadata index. The POC is a specific application of the mechanism this issue seeks to expose generally, rather than an alternative to it. General, user-directed support could provide shared infrastructure for that index and other application-defined payloads, while leaving index construction and interpretation out of scope. - **Embedded indexes:** @alamb's https://github.com/apache/datafusion/issues/16374, @zhuqi-lucas's https://github.com/apache/datafusion/pull/16395, and https://github.com/apache/datafusion/pull/16692 establish and explain the existing embedded-index approach. #7714 provides its raw-writing support, and that approach satisfied the custom-block request in #9860. - **Alternative metadata representations:** @rok's #9041 and #9042 explore another application of extensible metadata. This proposal does not select or standardize a replacement representation. - **Streaming readers:** #1605 addresses related I/O and decoding boundaries. Footer-independent discovery remains a separate capability from the metadata-extension support requested here. ### Willingness to contribute I plan to contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
