danielhumanmod commented on PR #1309: URL: https://github.com/apache/datafusion-ballista/pull/1309#issuecomment-3367946791
> Sorry, for late reply @danielhumanmod I'm not quite sure, i guess all the metrics are collected at the scheduler side, so scheduler should have it all once job finishes. > > It's a question how can we get it out of scheduler? Once the job is scheduled its question where/how do we wait for job competition. Do we lunch another job, which would poll rest api or we change scheduler grpc to wait there, i'm not sure. > > #1292 may be related if we get it from rest api Oh, that’s a really good point. I was also thinking about how to handle this outside of the scheduler, since adding special-case logic directly into the scheduler feels like a bad pattern to me. Let me look into how we could leverage the REST API for this. But I do have a feeling we may need a new Exec node if we go down this path. I’ll do some investigation, dig into it further, and get back to you soon. Thanks for the inspiration! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
