andygrove opened a new pull request, #2106:
URL: https://github.com/apache/datafusion-ballista/pull/2106
# Which issue does this PR close?
<!-- No dedicated tracking issue; this removes the legacy hash-partitioning
shuffle writer. -->
N/A
# Rationale for this change
Ballista had two shuffle writer implementations selected by a session config
(`ballista.shuffle.sort_based.enabled`, default `true`):
- **Sort-based** (`SortShuffleWriterExec`) — writes one consolidated data
file plus an
index file per input partition (`2 x N` files), coalesces small batches,
and bounds
memory via spilling.
- **Hash-based** (`ShuffleWriterExec`'s hash path) — keeps one open Arrow
`StreamWriter`
per output partition and writes one file per `(input_partition,
output_partition)`
pair (`N x M` files). This per-output-partition state does not scale to
high output
partition counts.
Sort-based has been the default and supersedes the hash path for the case
that matters
(hash-repartition stages). Keeping the non-scaling hash writer and a config
toggle to
select it added complexity and a foot-gun without a real use case. This PR
removes it.
# What changes are included in this PR?
- **Planner**: hash-repartition stages (`Partitioning::Hash`) now always use
`SortShuffleWriterExec`; the `ballista.shuffle.sort_based.enabled` gate is
gone.
- **Writer**: removed `ShuffleWriterExec`'s hash write path (the
per-output-partition
`WriteTracker` array + `BatchPartitioner`) and the unused `repart_time`
metric.
`ShuffleWriterExec` is retained for single-partition (`None`) output — the
final
output stage, `CoalescePartitionsExec`, `SortPreservingMergeExec`, and
broadcast-build
stages — and `try_new` now rejects `Some(Partitioning::Hash(..))`.
- **Reader / Flight**: sort-shuffle reads are gated purely on the on-disk
index file
(`is_sort_shuffle_output`), which was already the authoritative signal;
the redundant
config-derived boolean and the `ballista_sort_shuffle_enabled` trait
method are removed.
- **Config**: removed the `ballista.shuffle.sort_based.enabled` key, its
`ConfigEntry`,
and its accessor. Sort-shuffle tuning keys (`batch_size`,
`memory_limit_per_task_bytes`, writer channel capacity) are unchanged.
- **Benchmark**: `shuffle_bench` is now sort-only (dropped `--writer hash`).
- **Tests**: removed the hash-vs-sort comparison tests and the sort-shuffle
enable-toggle
tests; adapted writer/reader unit tests that constructed hash-partitioned
writers to the
single-partition path.
- **Docs**: removed the "Hash-based shuffle (opt-in)" documentation and the
`sort_based.enabled` config rows.
TPC-H plan-stability golden files are unchanged: they already show
`SortShuffleWriterExec` for hash-repartition stages and `ShuffleWriterExec:
partitioning:
None` for terminal stages, so query plan shape does not move under the
default
configuration.
# Are there any user-facing changes?
Yes — breaking. The `ballista.shuffle.sort_based.enabled` config key and the
public
`SessionConfigExt::ballista_sort_shuffle_enabled` trait method are removed,
and the
hash-based shuffle writer is no longer available. `BallistaConfig` does not
reject unknown
keys, so a config that still sets the removed key will be accepted and
silently ignored
(sort-based shuffle is used unconditionally) rather than erroring. Please
apply the
`api change` label.
Note on wire compatibility: scheduler and executor deploy together and
serialized plans
are ephemeral, but for completeness — a legacy physical plan carrying a hash
`output_partitioning` inside a `ShuffleWriterExecNode` would now fail to
decode (the
`try_new` guard rejects it), since hash-repartition stages are always
emitted as
`SortShuffleWriterExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
