anniegracehu opened a new issue, #50866: URL: https://github.com/apache/arrow/issues/50866
### Describe the enhancement requested `parquet::arrow::FileWriter` can encode a row group's columns in parallel (`ArrowWriterProperties::set_use_threads`, GH-33655), but only via `NewBufferedRowGroup` + `WriteRecordBatch`. pyarrow only binds `WriteTable`, which encodes columns serially, and `ParquetWriter` has no `use_threads` argument, so every Parquet write from Python is single-threaded (`write_table`, `ParquetWriter`, pandas `to_parquet`). Measured against 23.0.1 on a 37-column flat table (18 string, 8 double, 6 bool, 5 int64), zstd, 1M rows: 2.5 s with `write_table`; 1.1 s with `use_threads` at 65,536-row groups; 0.4 s at 262,144-row groups. Rows read back identical, same row groups and statistics. Proposed: `ParquetWriter(..., use_threads=False)` sets `ArrowWriterProperties::set_use_threads`, and `write_table` then writes one buffered row group per `row_group_size` rows through `WriteRecordBatch`, so the file layout is unchanged and nothing changes unless the flag is passed. Python-side only, ~80 lines plus tests; patch ready. The alternative would be a `use_threads`-style option on C++ `WriteTable` so the dataset writer gets it too. Happy to send either; which shape do you prefer? ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
