andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6440956527
I'm splitting this into smaller PRs for review, and keeping this one open as a draft for reference (full design notes, end-to-end tests, SF1000 results). Each split PR is cut from this branch's final state. 1. #2669: let a standalone cluster run push-staged (no pipelined concepts) 2. #2670: pipelined exchange operators, registry, protobuf and settings (nothing plans them yet; the scheduler rejects pipelined jobs) 3. Executor wiring (registry ownership, serving `FetchPipelinedPartition`, shared memory pool per job) 4. Scheduler: pipelined execution graph and all-at-once placement 5. Client streaming results, which turns the mode on, plus the end-to-end tests, TPC-H CI leg and user guide 6. Flight SQL support 7. Size-aware join selection for pipelined jobs I'll link the rest here as they're opened. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
