seancxmao commented on issue #23379: [SPARK-26440][WEBUI] Show total CPU time across all tasks on stage pages URL: https://github.com/apache/spark/pull/23379#issuecomment-452617564 Let me share some context. When we tune Spark applications, we usually focus on 3 kinds of performance metrics. * Run Time (at stage level, this is total run time across all tasks) * CPU * Time (at stage level, this is total cpu time across all tasks) * Memory * Time (maybe hard to measure this at stage level, since executor memory are shared across concurrently running stages) We found at stage level both run time and cpu time are useful. We can get both run time (in mili-seconds) and cpu time (in nano-seconds) via REST API, below is an example I ran locally. <img width="584" alt="screen shot 2019-01-09 at 4 26 46 pm" src="https://user-images.githubusercontent.com/12194089/50887029-0450d180-142d-11e9-9947-ecf55257519b.png";> run time is already displayed on stage page, however cpu time is not. <img width="373" alt="screen shot 2019-01-09 at 4 28 33 pm" src="https://user-images.githubusercontent.com/12194089/50887076-2b0f0800-142d-11e9-9366-e52066f03a2a.png";> Since REST API is usually targeted for monitoring tools rather than Spark users, maybe it's helpful to display cpu time side by side with run time. IIUC, currently we can't get stage level cpu time from the executors directly.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/12194089/50887029-0450d180-142d-11e9-9947-ecf55257519b.png"](https://user-images.githubusercontent.com/12194089/50887029-0450d180-142d-11e9-9947-ecf55257519b.png"){kind=link}
![https://user-images.githubusercontent.com/12194089/50887076-2b0f0800-142d-11e9-9366-e52066f03a2a.png"](https://user-images.githubusercontent.com/12194089/50887076-2b0f0800-142d-11e9-9366-e52066f03a2a.png"){kind=link}