Sruhvx-jpg commented on issue #19774: URL: https://github.com/apache/datafusion/issues/19774#issuecomment-6831708609
Hi @ethan-tyler, @alamb, I would like to work on this issue if it is currently available. @ethan-tyler, are you actively working on this, or would it be okay if I take it over? Regarding the design, rather than introducing a separate `PartitionedDataSink` trait, what do you think about a parallel-first approach with a soft deprecation path? Specifically: 1. Add `write_partition()` directly to `DataSink`, with a default implementation delegating to `write_all()` so existing sinks continue to compile. 2. Mark `write_all()` as soon-to-be deprecated in doc comments and release notes (avoiding an immediate `#[deprecated]` attribute that breaks downstream CI on `-D warnings`). 3. Update `DataSinkExec` to execute partitions in parallel by default, inserting `CoalescePartitionsExec` only when a sink falls back to `write_all()`. If this direction sounds reasonable, could I be assigned this issue? Happy to put together a draft PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
