andygrove opened a new issue, #2810: URL: https://github.com/apache/datafusion-ballista/issues/2810
## Summary This is a research item, not a decision to change the design. The goal is to find out whether a different spill file layout for sort-based shuffle would be simpler without costing performance, and to measure it before committing either way. This idea came up in the review of #2802 (thanks @NoahKusaba). ## Current design Each input partition's `SpillManager` keeps one spill file per output partition. Every spill event appends one complete Arrow IPC stream to each spilled partition's file. At finalize, each partition's spill file is copied verbatim into the consolidated data file. So one task can leave up to M x K spill files on disk (M input partitions, K output partitions), and each spill event opens up to K files. ## Alternative to evaluate Spark's sort shuffle uses a different layout: - [`ShuffleExternalSorter`](https://github.com/apache/spark/blob/2af0404f3eeea8e1d09fa428c7363c388655c184/core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java#L168-L268) writes **one file per spill event**, with partitions in order, and records only each partition's length ([`SpillInfo.partitionLengths`](https://github.com/apache/spark/blob/2af0404f3eeea8e1d09fa428c7363c388655c184/core/src/main/java/org/apache/spark/shuffle/sort/SpillInfo.java)). - [`mergeSpillsWithTransferTo`](https://github.com/apache/spark/blob/2af0404f3eeea8e1d09fa428c7363c388655c184/core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java#L457-L505) builds each output partition by copying its slice out of every spill file in turn. For Ballista that could look like: - **Spill:** write each partition as its own IPC stream into a single file per spill event, and keep a `Vec<u64>` of lengths. - **Finalize:** for each output partition, copy its slice from each spill file (e.g. `copy_file_range`). Readers already handle concatenated streams, so the output format and readers would not change. - **Code:** spilling and the in-memory encode already share one helper (`write_ipc_stream`, from #2802), and `SpillManager` could shrink to a list of `(file, lengths)`. ## Open questions - **Performance.** The bytes copied are the same. The trade is fewer file opens per spill against more, smaller copies at finalize. Spark's [own notes](https://github.com/apache/spark/blob/2af0404f3eeea8e1d09fa428c7363c388655c184/core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java#L369-L372) say those small copies hurt when the data per partition per spill is small, which is exactly the spill-heavy case. A first guess is roughly break-even, but this needs measuring. - **File descriptors and inodes.** How much does going from up to M x K files to one file per spill event matter in practice, for example with high K or many concurrent tasks per executor? - **Memory.** Is there any change to memory overhead (lengths table, copy buffers)? A memory regression would rule this out. ## Suggested approach 1. Prototype the layout behind the existing `SpillManager` interface. 2. Benchmark spill-heavy queries with a small `memory_limit_per_task_bytes` and a range of output partition counts, and compare wall time, finalize time, peak memory and number of files opened against the current design. 3. Decide based on the numbers. Keeping the current design is a fine outcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
