alamb commented on issue #12364:
URL: https://github.com/apache/arrow-rs/issues/12364#issuecomment-7061657003

   Another thing that came to my mind when thinking about this usecase is that 
currently the parquet format effectively requires that the pages for each 
column chunk are contiguous. 
   
   Thus if you really store your data in a single row group per file, you'll 
have to buffer most of the file before you get the pages of the last column to 
begin merging
   
   For example, here is how a parquet file would look with 2 columns, 1 row 
group, 3 data pages per column chunk:
   
   ```
   ┌────────────────────────────────┐                                
   │       PAR1 Magic Number        │                                
   ├────────────────────────────────┤                                
   │ Data Page 1 (header + values)  │ │                │             
   ├────────────────────────────────┤ │ Column Chunk 1 │             
   │ Data Page 2 (header + values)  │ │ (column "a")   │             
   ├────────────────────────────────┤ │                │             
   │ Data Page 3 (header + values)  │ │                │             
   ├────────────────────────────────┤                  │ Row Group 1 
   │ Data Page 1 (header + values)  │ │                │             
   ├────────────────────────────────┤ │ Column Chunk 2 │             
   │ Data Page 2 (header + values)  │ │ (column "b")   │             
   ├────────────────────────────────┤ │                │             
   │ Data Page 3 (header + values)  │ │                │             
   ├────────────────────────────────┤                                
   │    Standard parquet footer     │                                
   └────────────────────────────────┘                                
   ```
   
   If the goal is to streaming merge multiple files together with limited 
memory, I think you might have to split the files into multiple row groups (and 
basically plan to buffer a row group at a time). For example with three row row 
groups:
   
   ```                                                                          
    
   ┌────────────────────────────────┐                                           
 
   │       PAR1 Magic Number        │                                           
 
   ├────────────────────────────────┤                                           
 
   │ Data Page 1 (header + values)  │  Column Chunk 1 (column "a")  │           
 
   ├────────────────────────────────┤                               │ Row Group 
1
   │ Data Page 1 (header + values)  │  Column Chunk 2 (column "b")  │           
 
   ├────────────────────────────────┤                                           
 
   │ Data Page 1 (header + values)  │  Column Chunk 1 (column "a")  │           
 
   ├────────────────────────────────┤                               │ Row Group 
2
   │ Data Page 1 (header + values)  │  Column Chunk 2 (column "b")  │           
 
   ├────────────────────────────────┤                                           
 
   │ Data Page 1 (header + values)  │  Column Chunk 1 (column "a")  │           
 
   ├────────────────────────────────┤                               │ Row Group 
3
   │ Data Page 1 (header + values)  │  Column Chunk 2 (column "b")  │           
 
   ├────────────────────────────────┤                                           
 
   │    Standard parquet footer     │                                           
 
   └────────────────────────────────┘                                           
 
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to