milenkovicm commented on PR #1547: URL: https://github.com/apache/datafusion-ballista/pull/1547#issuecomment-4222406131
> Or we could possibly make it this way, let me know if these are viable: > > * send the system-level memory specification during executor registration in the **register_executor** function. Right now we are sending only the **available_parallelism** specification. Maybe the memory would also be viable for displaying in the UI, something like: **used_by_user_task_mem** / **total_executor_available_memory** we can continue sending what we send now, it would make sense to get overall system utilisation (also we can make this configurable ) perhaps we could use https://lib.rs/crates/memory-stats to capture process memory as well (not sure if this is expensive to do) > * send hearbeats to the scheduler with the info of currently running tasks? Something like a HashMap with job_id (if I'm not mistaken and this is the most top-level id that represents **app** that users created, similar to Spark). i think scheduler already knows list of running tasks/jobs not sure what would this do -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
