tustvold commented on code in PR #7655:
URL: https://github.com/apache/arrow-datafusion/pull/7655#discussion_r1371688699
##########
datafusion/common/src/config.rs:
##########
@@ -377,12 +377,24 @@ config_namespace! {
pub bloom_filter_ndv: Option<u64>, default = None
/// Controls whether DataFusion will attempt to speed up writing
- /// large parquet files by first writing multiple smaller files
- /// and then stitching them together into a single large file.
- /// This will result in faster write speeds, but higher memory usage.
- /// Also currently unsupported are bloom filters and column indexes
- /// when single_file_parallelism is enabled.
- pub allow_single_file_parallelism: bool, default = false
+ /// parquet files by serializing them in parallel. Each column
+ /// in each row group in each output file are serialized in parallel
+ /// leveraging a maximum possible core count of
n_files*n_row_groups*n_columns.
+ pub allow_single_file_parallelism: bool, default = true
+
+ /// If allow_single_file_parallelism=true, this setting allows
+ /// applying backpressure to prevent working on too many row groups in
+ /// parallel in case of limited memory or slow I/O speed causing
+ /// OOM errors. Lowering this number limits memory growth at the cost
+ /// of potentially slower write speeds.
+ pub maximum_parallel_row_group_writers: usize, default = 16
+
+ /// If allow_single_file_parallelism=true, this setting allows
+ /// applying backpressure to prevent too many RecordBatches building
+ /// up in memory in case the parallel writers cannot consume them fast
+ /// enough. Lowering this number limits memory growth at the cost
+ /// of potentially lower write speeds.
+ pub maximum_buffered_record_batches_per_stream: usize, default = 200
Review Comment:
Is the memory usage of these buffers tracked with a MemoryReservation at all?
For context https://github.com/apache/arrow-rs/issues/3871 was the issue
from back when ArrowWriter did something similar, and this caused operational
problems for us as writers could easily consume GB of memory when writing large
files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
