HippoBaro commented on issue #12364: URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7074977007
Thanks to both of you! > Thus if you really store your data in a single row group per file, you'll have to buffer most of the file before you get the pages of the last column to begin merging @alamb There's a misunderstanding here. We do not consume this data in `RecordBatch`es like DataFusion would. We don’t need to reach the last column before starting the merge. Each input has its columns in lexicographic order. We merge column `bar` from the input(s) that contain it, emit the output, then move on to `foo`, and so on. **Within each column, we can process the data incrementally at data-page granularity**. We’re never assembling complete rows across all columns. So keeping each column chunk contiguous actually fits our access pattern best. We don’t need to buffer the whole row group, or split it into smaller row groups to make progress. That’s also why we want the descriptors alongside their column chunks: if we don’t need to keep future columns’ data in memory, we’d rather not keep all their descriptors either. Each cursor only needs the descriptor for the current column and can discard it when it moves on. > One potential alternative [...] would be to write your column headers needed for the forward pass at the start of the file in a custom header. Putting all the descriptors at the start would indeed satisfy the single-GET requirement. But we’d need to retain the descriptors until their corresponding columns arrive. With very wide schemas (I'm dealing with 100k+ schemas on the regular!) and hundreds of inputs, that becomes a substantial working set. Wide-schema scaling is something we care deeply about, see #9722. > This reminds me a bit of the discussion around the now deprecated `ColumnChunk::file_offset` field (https://github.com/apache/parquet-format/pull/440). Its idea was probably to store the `ColumnMetaData` out of line, and right before the data for that column. Thanks @jhorstmann! We considered reusing `ColumnMetaData` early on because it looks like a good fit on paper, but ran into three concerns: - The historical out-of-line metadata mechanism you mention is deprecated. - Thrift gives us an encoding, but we’d still need an application-defined convention for placing and discovering these objects between column chunks. Binary Protocol Extensions already provide framing and context within an existing structure. - Most importantly, `ColumnMetaData` requires [the final compressed and uncompressed chunk sizes](https://github.com/apache/parquet-format/blob/f78561ef92fbb727e03724a7fac5dec4bc5d9849/src/main/thrift/parquet.thrift#L939-L944). For our forward-only writer, putting it before the chunk means buffering the entire output chunk instead of flushing pages as we produce them. This breaks our streaming hard requirement. We could omit those fields or fill them with placeholders, but then we wouldn’t be writing valid `ColumnMetaData`. At that point, much of the benefit of reusing a standard structure disappears. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
