milenkovicm commented on PR #1372: URL: https://github.com/apache/datafusion-ballista/pull/1372#issuecomment-3921982197
> Woohoo! Nice work @milenkovicm. At a high level this all makes sense, acknowledging the TODOs. > > Some notes of my own for mile-high level understanding: > > * We have an adaptive planner that uses `AdaptiveDatafusionExec` and `ExchangeExec` as placeholder exec nodes > * As stages get scheduled, placeholder nodes are replaced with concrete shuffle read/write to get passed to executors, controlled by the `BallistaAdapter` > * In between stages, we replan X times allowing for modification/cancellation of the next stage > * Current capabilities include cancelling stages that produce empty results, (build can cancel probe on joins too) it can detect which states should be canceled but it does not cancel them at the moment (to be investigated if they can be cancelled) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
