JingsongLi commented on PR #9837: URL: https://github.com/apache/paimon/pull/9837#issuecomment-5690982007
**[P2] Coalesce small input chunks before writing Parquet row groups** Location: [`map_shared_shredding_writer.py:167–173`](https://github.com/apache/paimon/blob/95ef49cf7ff3b6e7116835f419ad8e9ac7daa86c/paimon-python/pypaimon/write/map_shared_shredding_writer.py#L167-L173). `data.to_batches(max_chunksize=batch_rows)` preserves existing chunk boundaries; it does not combine small chunks. On the append-only path, each `write_row()` creates a one-row batch, and `WriteBuffer.materialize()` uses `pa.concat_tables`, which preserves those chunks. Consequently, this `writer.write_table(physical)` call creates a separate row group for every input row. I reproduced this with Python 3.11 / PyArrow 18.1, an append-only Parquet table with `id INT, m MAP<STRING, BIGINT>`, shared shredding enabled for `m`, and the default 256 physical columns. For 1,000 rows containing one entry `('a', i)` each: | Input method | Row groups | File size | Footer size | | --- | ---: | ---: | ---: | | One `write_arrow()` call with a contiguous table | 1 | 98,967 bytes | 76,623 bytes | | 1,000 `write_row()` calls | 1,000 | 32,456,977 bytes | 16,716,965 bytes | This substantially increases storage, footer memory, and read overhead for a supported write API. Adaptive column counts only take effect after a file completes, so they do not protect the first file. **How Java handles this:** [`ShreddingFormatWriter.addElement()`](https://github.com/apache/paimon/blob/95ef49cf7ff3b6e7116835f419ad8e9ac7daa86c/paimon-common/src/main/java/org/apache/paimon/format/shredding/ShreddingFormatWriter.java#L55-L69) converts one logical row and delegates to `ParquetBulkWriter` → `ParquetWriter.write()`. [`ColumnCompressionRecordWriter`](https://github.com/apache/paimon/blob/95ef49cf7ff3b6e7116835f419ad8e9ac7daa86c/paimon-format/src/main/java/org/apache/parquet/hadoop/ColumnCompressionRecordWriter.java#L155-L195) keeps column/page buffers across these calls and flushes a row group when the row-count limit or buffered-size threshold is reached; closing the file flushes the remainder. The default size target is 128 MiB. Thus, row-group boundaries are independent of individual input calls. Java also commits the final shredding dictionary metadata before closing the underlying writer, allowing a single pass over the rows. Please coalesce small logical chunks within bounded row/byte windows before conversion, and keep row-group boundaries independent of input chunk boundaries. Increasing `row_group_size` alone will not combine separate `write_table()` calls. A regression test comparing one batch with repeated `write_row()` calls should verify that row-group count does not grow with the number of input calls. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
