danielhumanmod commented on PR #1309: URL: https://github.com/apache/datafusion-ballista/pull/1309#issuecomment-3454633521
> Sorry for late reply @danielhumanmod I think your breakdown makes sense, I just wonder what would be the easiest way to implement this. > > Scheduler emits job completion events, so we could react on job completion, but this approach would be a bit messy with observers, as evens will be emitted for all jobs. > > An alternative would be to add ability to register job completion callback which triggers change of job B once the job A finishes. Trigger can change UnresolvedShuffle to actual Exec (we could use same logic like we use for simple EXPLAIN). > > I believe job finish callback would be useful for other cases, I'm not sure at this point how complicated would it be to implement, may have a look this weekend. Sorry for late response @milenkovicm ! The job completion callback sounds like a more reasonable approach for a dynamic scenario like this, and it also seems more generic for other future use cases. I’d be happy to help investigate or implement the callback if I could get a bit more context how you envision the design. Also, do you think it makes sense to finish EXPLAIN first, and then tackle EXPLAIN ANALYZE in a follow-up PR once the approach is finalized? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
