HippoBaro opened a new issue, #12364:
URL: https://github.com/apache/arrow-rs/issues/12364

   ### Is your feature request related to a problem or challenge?
   
   I work on a database that merges files belonging to the same table using 
forward-only input cursors and a k-way merger. We have described this system 
[in a public blog 
post](https://www.datadoghq.com/blog/engineering/husky-storage-compaction/).
   
   Our existing fragment format places a header before each column, allowing 
compaction to discover columns during a single forward pass. I would like to 
support the same access pattern with Parquet.
   
   For the corresponding Parquet layout, each file would contain one row group, 
with column chunks laid out in lexicographic order. Each cursor discovers its 
next column's identity and presents it to the merger. The merger determines 
which cursors to advance, combines matching columns, and produces another 
lexicographically ordered sequence of columns. Each input is consumed in one 
forward pass. This lets us merge hundreds of input fragments without buffering 
their full contents, with working memory bounded by the active per-input 
buffers rather than the total input size.
   
   **Exactly one streaming GET request per input file is a hard requirement.** 
Cloud providers charge per GET request, so an additional request to fetch a 
footer, index, or sidecar is not acceptable for this workload. The information 
needed to process each column must arrive in that same forward stream.
   
   Parquet's standard page headers do not identify their columns. That 
information normally comes from the schema and column metadata in the footer. I 
would like to repeat a column's identity and application-specific decoding 
descriptor in its first page header, making the information available where the 
cursor encounters the data.
   
   The capability I am requesting is attaching application metadata within the 
existing Parquet structure whose meaning it enriches, and accessing it as that 
structure is decoded.
   
   For example, a `DOUBLE` column named `speed` does not distinguish metres per 
second from kilometres per hour. An application may want to annotate that 
column's `SchemaElement` with its measurement unit. This adds 
application-specific meaning to the existing type without introducing a new 
standard Parquet logical type. A reader that understands the annotation can 
validate or interpret it while reading the schema; the standard Parquet type 
remains `DOUBLE`.
   
   Such an annotation may contain only a few bytes, and its meaning depends on 
the particular schema element carrying it. Storing it as a separate binary 
block located through footer metadata as in 
https://github.com/apache/datafusion/pull/16395 requires the application to 
recreate that association. Attaching it directly to the schema element 
preserves the association through the file's existing structure.
   
   This complements the embedded-index approach demonstrated in 
https://github.com/apache/datafusion/pull/16395. Separately located binary 
regions are well suited to substantial payloads (such as indexes) that readers 
may fetch independently. In-structure extensions are a natural fit for small, 
contextual annotations on schema elements, page headers, and other metadata 
structures.
   
   ### Describe the solution you'd like
   
   Support application-defined metadata using the existing [Parquet Binary 
Protocol 
Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md)
 mechanism.
   
   Applications should be able to:
   
   - Attach opaque payloads to ordinary Parquet Thrift structs, including 
schema elements and page headers.
   - Opt in to observing extension payloads during ordinary metadata decoding.
   - Identify the structure and column or page to which a payload belongs.
   - Restrict processing to relevant locations.
   - Report errors when a recognized annotation is invalid or incompatible with 
the application.
   
   Payload meaning and encoding should remain application-defined, as per the 
standard. The mechanism should support different applications without requiring 
the crate to standardize their descriptors, annotations, or indexes.
   
   For our streaming-merge use case, the descriptor should be available on the 
first encountered page of a column chunk. Repeating it on a dictionary page and 
the first data page is acceptable.
   
   ### Describe alternatives you've considered
   
   - **Read or cache the footer first.** This is the reader model described by 
the [Parquet format 
documentation](https://github.com/apache/parquet-format/blob/master/README.md#file-format),
 but a separate footer fetch violates our one-GET requirement. Reading the 
whole object in one GET and waiting for its footer instead requires buffering 
or spooling data for later processing, rather than consuming it in one forward 
pass.
   - **Embed binary regions and locate them through footer metadata.** 
https://github.com/apache/datafusion/pull/16395 demonstrates this approach 
using the raw-writing support added in #7714. It is useful for independently 
fetched indexes, but obtaining the footer locators before consuming column data 
has the same upfront metadata-access requirement. It also does not attach the 
payload directly to a schema element or page header.
   - **Interleave self-framed blobs with column chunks.** A variation on 
https://github.com/apache/datafusion/pull/16395 would place an 
application-framed descriptor immediately before each column chunk. Our cursor 
could discover the descriptor and the following chunk in one forward GET, 
without footer references to those blobs. The file can remain format-compliant: 
column chunks stay contiguous, their offsets and sizes remain correct, and 
unaware metadata-directed readers skip the extra regions. This would satisfy 
our streaming requirements. It is less general, however: it introduces an 
application-specific framing and layout convention rather than annotating 
existing structures, and does not directly attach metadata to nested objects 
such as individual `SchemaElement`s.
   - **Existing key/value metadata or sidecars.** Footer key/value metadata is 
not available when the forward cursor encounters the column data, and 
file-level entries require an application convention to associate annotations 
with schema fields. A separately fetched sidecar adds a request and requires 
separate distribution and lifecycle management. Preexisting external metadata 
can help, but does not guarantee our one-GET access pattern for arbitrary 
inputs.
   
   [Binary Protocol 
Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md)
 are our preferred solution because the extension point and its framing are 
already part of Parquet. Applications can attach opaque metadata directly to 
existing ordinary Thrift metadata structures, from schema elements and footer 
metadata to page and index headers, while preserving the context in which that 
metadata belongs.
   
   Rather than requiring each application to define another convention for 
positioning, framing, and associating separate byte regions, the crate should 
expose this existing standardized capability. The payload's meaning remains 
application-defined, but the mechanism carrying it does not need reinventing.
   
   ### Additional context
   
   There is relevant prior work that this proposal should complement:
   
   - **Metadata indexing:** @etseidl's #8713 and #8714 demonstrate using Binary 
Protocol Extensions for a metadata index. The POC is a specific application of 
the mechanism this issue seeks to expose generally, rather than an alternative 
to it. General, user-directed support could provide shared infrastructure for 
that index and other application-defined payloads, while leaving index 
construction and interpretation out of scope.
   - **Embedded indexes:** @alamb's 
https://github.com/apache/datafusion/issues/16374, @zhuqi-lucas's 
https://github.com/apache/datafusion/pull/16395, and 
https://github.com/apache/datafusion/pull/16692 establish and explain the 
existing embedded-index approach. #7714 provides its raw-writing support, and 
that approach satisfied the custom-block request in #9860.
   - **Alternative metadata representations:** @rok's #9041 and #9042 explore 
another application of extensible metadata. This proposal does not select or 
standardize a replacement representation.
   - **Streaming readers:** #1605 addresses related I/O and decoding 
boundaries. Footer-independent discovery remains a separate capability from the 
metadata-extension support requested here.
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to