milenkovicm commented on issue #2267: URL: https://github.com/apache/datafusion-ballista/issues/2267#issuecomment-5480442751
As its would be very hard to provide one size fit all solution i would suggest slight protocol change which would simplify implementations. I would suggest small changes to file names for running jobs : - running jobs to be named: `<event_log_dir>/<job_id>.eventlog.running` (or some prefix like that) - completed jobs, to be renamed from `<event_log_dir>/<job_id>.eventlog.running` to `<event_log_dir>/<job_id>.eventlog` this would give us ability to listen for file changes in `event_log_dir` and know when job has finished and event log completed. user could either listen for a directory changes or scheduler job notification to implement synchronisation any way it suits them wdyt @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
