peterxcli opened a new issue, #6010: URL: https://github.com/apache/datafusion-comet/issues/6010
### What is the problem the feature request solves? Since #5916 a map task spills every partition into one temp file. `DiskManager::create_tmp_file` picks a random local dir per file, and Comet hands it Spark's local dirs, so the per-partition files it replaced spread a task's spill across the local dirs while the single file pins it to one. With several local dirs on separate disks the per-disk peak becomes one task's whole spill, and a single file can grow to the task's total spilled bytes. Rotation does not release space earlier: every file holds ranges from most partitions, so nothing can be unlinked before the merge finishes. What it can change is the spread across local dirs and the size of any one file. ### Describe the potential solution Compare the task-wide file against two alternatives that keep each partition's write order: - One file per spill round, as DataFusion's own operators spill. The file count then equals the round count, which a pool under pressure can push into the thousands. - A new file once the current one passes a byte threshold, rotating only between writes so no range straddles files. Ranges keep a task-wide offset space and stay 16 bytes; a small file table maps an offset to (file, local offset). The file count is bounded by spilled bytes / threshold whatever the round count. Either way the merge should keep its open readers bounded (opened lazily, capped by the file count or an LRU) and read each partition's ranges in offset order, which preserves write order. Measure on SSD/NVMe with many spill rounds and concurrent tasks, with two or more local dirs: spill write time and merge time, peak memory, open file counts, and per-disk peak space. `write_time` currently covers both the spill writes and the merge copies; splitting it into spill and merge time is a prerequisite for that report. ### Additional context Suggested in the review of #5916: https://github.com/apache/datafusion-comet/pull/5916#issuecomment-5718670089. #6009 covers the range metadata the merge keeps in memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
