GitHub user ryux1 added a comment to the discussion: Feedback on high memory usage when merging N parquet files
The error points at the explicit output sort, not the Parquet writer itself: ```rust .with_sort_by(vec![signal_name_sort_expr, timestamp_sort_expr]) ``` That introduces the `ExternalSorterMerge` named in the error. Also, `file_sort_order` is a promise that each input file has that ordering; it does not make the concatenation of 40 files globally ordered when their key ranges overlap. So DataFusion may still need a merge/sort for the requested output ordering. The first change I would try is replacing `GreedyMemoryPool` with `FairSpillPool`: ```rust use datafusion::execution::memory_pool::FairSpillPool; let memory_pool = Arc::new(FairSpillPool::new(limit_bytes)); ``` In 50.3, the greedy pool is documented as appropriate for no spill or a single spillable operator, while the fair pool is intended when multiple spillable operators compete for the limit: [50.3 source](https://github.com/apache/datafusion/blob/50.3.0/datafusion/execution/src/memory_pool/pool.rs#L59-L63). A few other changes should reduce the peak: - Keep `target_partitions` low, as you found. - Move `parquet.write_batch_size` back down. Raising it from 8192 to 65536 increases writer buffering; it is a throughput tradeoff in the wrong direction for a memory-constrained run. - Reduce `max_row_group_size` from 1M while diagnosing. DataFusion's own 50.3 config docs note that larger row groups require more memory to write: [source](https://github.com/apache/datafusion/blob/50.3.0/datafusion/common/src/config.rs#L605-L611). - Because you partition output by year/month/day, bound writer concurrency too. In 50.3 the relevant settings are `datafusion.execution.minimum_parallel_output_files` (default 4) and `datafusion.execution.max_buffered_batches_per_output_file` (default 2): [source](https://github.com/apache/datafusion/blob/50.3.0/datafusion/common/src/config.rs#L434-L450). Finally, 6-8 MB Zstd-compressed input size is not a useful estimate of Arrow working memory: sorting operates on decoded arrays, plus sort buffers and active Parquet writers. I would test one change at a time in this order: fair pool, smaller write batch, smaller row groups, then lower output-writer concurrency. If removing `with_sort_by` makes memory collapse, that confirms the global output sort is the dominant consumer; only remove it if downstream readers do not require that physical ordering. GitHub link: https://github.com/apache/datafusion/discussions/18833#discussioncomment-18317644 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
