andygrove commented on PR #2765: URL: https://github.com/apache/datafusion-ballista/pull/2765#issuecomment-6661611992
Thanks, this was a good call. I moved reuse to stage creation in 956a59cf2. `identify_runnable_stages` keys each exchange it's about to launch against a cache of launched stages, like `stageCache`. A nested exchange is encoded by its stage id, so two stages match only when they read the same stages, and each stage is keyed once on the plan that actually runs. That got rid of the side-channel flag, `MAX_PASSES`, the fingerprint pass and the pending-stage split. A late repeat of a finished stage is resolved there and triggers a replan before anything launches, as you described. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
