milenkovicm commented on issue #1344: URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3641482816
thank you @danielhumanmod for tackling this issue, I think it would bring few new possibilities if implemented . I'm not sure if we should utilise shuffles as base building block. Let's say we want to implement distributed cache functionality or some kind of job preparation, they may be reading from source which is not a shuffle file. Would it be possible to keep job submission pending until another job finishes? If we can attach a callback on a "leading" job which would change dependant job status from "pending" to "ready to run" or to "cancel". I'm just throwing ideas, will have to take a deeper look at this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
