alamb commented on issue #12364:
URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7061657003
Another thing that came to my mind when thinking about this usecase is that
currently the parquet format effectively requires that the pages for each
column chunk are contiguous.
Thus if you really store your data in a single row group per file, you'll
have to buffer most of the file before you get the pages of the last column to
begin merging
For example, here is how a parquet file would look with 2 columns, 1 row
group, 3 data pages per column chunk:
```
┌────────────────────────────────┐
│ PAR1 Magic Number │
├────────────────────────────────┤
│ Data Page 1 (header + values) │ │ │
├────────────────────────────────┤ │ Column Chunk 1 │
│ Data Page 2 (header + values) │ │ (column "a") │
├────────────────────────────────┤ │ │
│ Data Page 3 (header + values) │ │ │
├────────────────────────────────┤ │ Row Group 1
│ Data Page 1 (header + values) │ │ │
├────────────────────────────────┤ │ Column Chunk 2 │
│ Data Page 2 (header + values) │ │ (column "b") │
├────────────────────────────────┤ │ │
│ Data Page 3 (header + values) │ │ │
├────────────────────────────────┤
│ Standard parquet footer │
└────────────────────────────────┘
```
If the goal is to streaming merge multiple files together with limited
memory, I think you might have to split the files into multiple row groups (and
basically plan to buffer a row group at a time). For example with three row row
groups:
```
┌────────────────────────────────┐
│ PAR1 Magic Number │
├────────────────────────────────┤
│ Data Page 1 (header + values) │ Column Chunk 1 (column "a") │
├────────────────────────────────┤ │ Row Group
1
│ Data Page 1 (header + values) │ Column Chunk 2 (column "b") │
├────────────────────────────────┤
│ Data Page 1 (header + values) │ Column Chunk 1 (column "a") │
├────────────────────────────────┤ │ Row Group
2
│ Data Page 1 (header + values) │ Column Chunk 2 (column "b") │
├────────────────────────────────┤
│ Data Page 1 (header + values) │ Column Chunk 1 (column "a") │
├────────────────────────────────┤ │ Row Group
3
│ Data Page 1 (header + values) │ Column Chunk 2 (column "b") │
├────────────────────────────────┤
│ Standard parquet footer │
└────────────────────────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]