andygrove opened a new issue, #2599: URL: https://github.com/apache/datafusion-ballista/issues/2599
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The three shuffle writer execs have identical `ExecutionPlan::execute` bodies and trailing trait boilerplate, apart from the struct name in error strings: - [RangeShuffleWriterExec (range_shuffle/writer.rs#L427-L544)](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/range_shuffle/writer.rs#L427-L544) - [ShuffleWriterExec (shuffle_writer.rs#L753-L852)](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/shuffle_writer.rs#L753-L852) - [SortShuffleWriterExec (sort_shuffle/writer.rs#L1007-L1114)](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/sort_shuffle/writer.rs#L1007-L1114) Each locks the state mutex, lazily creates K oneshot channels, spawns a coordinator and wraps `summaries_to_batch` in a `RecordBatchStreamAdapter`, followed by the same `metrics` / `statistics_from_inputs` / `child_stats_requests` / `ShuffleWriter` impls. The only real difference is what the spawned coordinator runs. A fix to the handoff protocol (for example the double-call error or poisoned-mutex handling) currently has to be applied three times. **Describe the solution you'd like** Extract a shared helper in `shuffle_writer.rs`, e.g. `execute_handoff(state, name, partition, ..., spawn_coordinator)`, taking the struct name for error messages and a closure that spawns the writer-specific coordinator. Each `execute` becomes a few lines. Optionally unify the two `run_coordinator` functions (`shuffle_writer.rs` L879 and `sort_shuffle/writer.rs` L1136) behind one generic that takes a future, and share the stats boilerplate via default methods or a small macro. **Describe alternatives you've considered** None. **Additional context** Line references are pinned to commit ea1126c0649b. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
