alamb commented on issue #12364: URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7061599465
> For the corresponding Parquet layout, each file would contain one row group, ... > **Exactly one streaming GET request per input file is a hard requirement.** Cloud providers charge per GET request, so an additional request to fetch a footer, index, or sidecar is not acceptable for this workload. The information needed to process each column must arrive in that same forward stream. This is a neat use case. One potential alternative that I don't think you mentioned (though you mentioned some variations) would be to write your column headers needed for the forward pass at the start of the file in a custom header. For example you can write some opaque bytes in the file after the magic number, Something like this: ```text ┌──────────────────────────┐ │ PAR1 Magic Number │ ├──────────────────────────┤ │ New Header (ignored by │ │ other parquet readers) │ │ │ ├──────────────────────────┤ │ │ │ │ │ │ │ │ │ Standard parquet body │ │ bytes │ │ │ │ │ │ │ │ │ ├──────────────────────────┤ │ Standard parquet footer │ │ (offsets account for the │ │ new header) │ └──────────────────────────┘ ``` I think you could do that today with approach described in [this blog](https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/) ( which you reference in related work). However, this may be tough depending on what is in your column headers (eg. if it is column chunk offsets, the writer would need to buffer the entire row group before it knew what to write in the header) > This complements the embedded-index approach demonstrated in https://github.com/apache/datafusion/pull/16395. Separately located binary regions are well suited to substantial payloads (such as indexes) that readers may fetch independently. In-structure extensions are a natural fit for small, contextual annotations on schema elements, page headers, and other metadata structures. While I agree adding such metadata annotations close to where they are used makes the most logical sense, I also think it adds non trivial complexity to the writer -- in exchange for a simpler reader. In this case I am not sure the complexity in the writer is a good tradeoff (as it seems the complexity should be in the system that needs the more advanced features), but that is a judgement call for sure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
