anniegracehu opened a new pull request, #50867: URL: https://github.com/apache/arrow/pull/50867
### Rationale for this change Every Parquet write from Python is single-threaded. The C++ `FileWriter` can encode a row group's columns in parallel (`ArrowWriterProperties::use_threads`), but only on the buffered row group path (`NewBufferedRowGroup` + `WriteRecordBatch`), and pyarrow only binds `WriteTable`, which encodes columns one after another. This exposes the C++ writer's parallel column encoding through the Python bindings. ### What changes are included in this PR? - `ParquetWriter(..., use_threads=False)` and `pq.write_table(..., use_threads=False)`; the flag is passed to `ArrowWriterProperties::Builder::set_use_threads`. - With the flag, `write_table` opens one buffered row group per `row_group_size` rows and writes that range's batches into it with `WriteRecordBatch`. Row groups, statistics and the stored schema are the same as `WriteTable` produces; `row_group_size` defaults and the 64Mi cap are applied the same way. Empty tables and builds without threading take the existing path. - Docstring says the two costs: a row group's column chunks are all held until the group is complete, and it must not be used from code running on the CPU thread pool (same warning as the C++ API). - Default is off, so existing writes are unchanged. `ds.write_dataset` is not covered; it calls C++ `WriteTable` directly. Numbers, 23.0.1, 37-column flat table (18 string, 8 double, 6 bool, 5 int64), zstd, 1M rows, 24-thread pool: `write_table` 2.5 s; `use_threads=True` 1.1 s at 65,536-row groups (about 3 cores busy; small groups cap it), 0.4 s at 262,144-row groups (about 7 cores). ### Are these changes tested? Yes: serial vs threaded write of a chunked table with string, list, struct and dictionary columns and nulls, at four row group sizes, comparing row group lengths, per-column statistics, stored schema and the read-back table; the 1Mi default and 64Mi cap; an empty table; the schema-mismatch error. ### Are there any user-facing changes? New optional `use_threads` argument on `ParquetWriter` and `write_table`. No behavior change without it. This change was written with AI assistance and reviewed by the submitter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
