HippoBaro commented on issue #12364:
URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7074977007

   Thanks to both of you!
   
   > Thus if you really store your data in a single row group per file, you'll 
have to buffer most of the file before you get the pages of the last column to 
begin merging
   
   @alamb There's a misunderstanding here. We do not consume this data in 
`RecordBatch`es like DataFusion would. We don’t need to reach the last column 
before starting the merge. Each input has its columns in lexicographic order. 
We merge column `bar` from the input(s) that contain it, emit the output, then 
move on to `foo`, and so on. **Within each column, we can process the data 
incrementally at data-page granularity**. We’re never assembling complete rows 
across all columns.
   
   So keeping each column chunk contiguous actually fits our access pattern 
best. We don’t need to buffer the whole row group, or split it into smaller row 
groups to make progress.
   
   That’s also why we want the descriptors alongside their column chunks: if we 
don’t need to keep future columns’ data in memory, we’d rather not keep all 
their descriptors either. Each cursor only needs the descriptor for the current 
column and can discard it when it moves on.
   
   >  One potential alternative [...] would be to write your column headers 
needed for the forward pass at the start of the file in a custom header.
   
   Putting all the descriptors at the start would indeed satisfy the single-GET 
requirement. But we’d need to retain the descriptors until their corresponding 
columns arrive. With very wide schemas (I'm dealing with 100k+ schemas on the 
regular!) and hundreds of inputs, that becomes a substantial working set. 
Wide-schema scaling is something we care deeply about, see #9722. 
   
   > This reminds me a bit of the discussion around the now deprecated 
`ColumnChunk::file_offset` field 
(https://github.com/apache/parquet-format/pull/440). Its idea was probably to 
store the `ColumnMetaData` out of line, and right before the data for that 
column.
   
   Thanks @jhorstmann! We considered reusing `ColumnMetaData` early on because 
it looks like a good fit on paper, but ran into three concerns:
   
    - The historical out-of-line metadata mechanism you mention is deprecated.
    - Thrift gives us an encoding, but we’d still need an application-defined 
convention for placing and discovering these objects between column chunks. 
Binary Protocol Extensions already provide framing and context within an 
existing structure. 
    - Most importantly, `ColumnMetaData` requires [the final compressed and 
uncompressed chunk 
sizes](https://github.com/apache/parquet-format/blob/f78561ef92fbb727e03724a7fac5dec4bc5d9849/src/main/thrift/parquet.thrift#L939-L944).
 For our forward-only writer, putting it before the chunk means buffering the 
entire output chunk instead of flushing pages as we produce them. This breaks 
our streaming hard requirement.
   
   We could omit those fields or fill them with placeholders, but then we 
wouldn’t be writing valid `ColumnMetaData`. At that point, much of the benefit 
of reusing a standard structure disappears.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to