alamb commented on issue #12364:
URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7061599465

   > For the corresponding Parquet layout, each file would contain one row 
group, ...
   
   > **Exactly one streaming GET request per input file is a hard 
requirement.** Cloud providers charge per GET request, so an additional request 
to fetch a footer, index, or sidecar is not acceptable for this workload. The 
information needed to process each column must arrive in that same forward 
stream.
   
   This is a neat use case. One potential alternative that I don't think you 
mentioned (though you mentioned some variations) would be to write your column 
headers needed for the forward pass at the start of the file in a custom header.
   
   For example you can write some opaque bytes in the file after the magic 
number, Something like this:
   
   ```text
   ┌──────────────────────────┐
   │    PAR1 Magic Number     │
   ├──────────────────────────┤
   │  New Header (ignored by  │
   │  other parquet readers)  │
   │                          │
   ├──────────────────────────┤
   │                          │
   │                          │
   │                          │
   │                          │
   │  Standard parquet body   │
   │          bytes           │
   │                          │
   │                          │
   │                          │
   │                          │
   ├──────────────────────────┤
   │ Standard parquet footer  │
   │ (offsets account for the │
   │       new header)        │
   └──────────────────────────┘
   ```
   
   I think you could do that today with approach described in [this 
blog](https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/)
 ( which you reference in related work).
   
   However, this may be tough depending on what is in your column headers (eg. 
if it is column chunk offsets, the writer would need to buffer the entire row 
group before it knew what to write in the header)
   
   
   > This complements the embedded-index approach demonstrated in 
https://github.com/apache/datafusion/pull/16395. Separately located binary 
regions are well suited to substantial payloads (such as indexes) that readers 
may fetch independently. In-structure extensions are a natural fit for small, 
contextual annotations on schema elements, page headers, and other metadata 
structures.
   
   While I agree adding such metadata annotations close to where they are used 
makes the most logical sense, I also think it adds non trivial complexity to 
the writer -- in exchange for a simpler reader. In this case I am not sure the 
complexity in the writer is a good tradeoff (as it seems the complexity should 
be in the system that needs the more advanced features), but that is a 
judgement call for sure
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to