I find that the Spark metrics system is quite useful to gather resource utilization metrics of Spark applications, including CPU, memory and I/O. If you are interested an example how this works for us at: https://db-blog.web.cern.ch/blog/luca-canali/2019-02-performance-dashboard-apache-spark If instead you are rather looking at ways to instrument your Spark code with performance metrics, Spark task metrics and event listeners are quite useful for that. See also https://github.com/apache/spark/blob/master/docs/monitoring.md and https://github.com/LucaCanali/sparkMeasure
Regards, Luca From: manish ranjan <cse1.man...@gmail.com> Sent: Tuesday, March 26, 2019 15:24 To: Jack Kolokasis <koloka...@ics.forth.gr> Cc: user <user@spark.apache.org> Subject: Re: Spark Profiler I have found ganglia very helpful in understanding network I/o , CPU and memory usage for a given spark cluster. I have not used , but have heard good things about Dr Elephant ( which I think was contributed by LinkedIn but not 100%sure). On Tue, Mar 26, 2019, 5:59 AM Jack Kolokasis <koloka...@ics.forth.gr<mailto:koloka...@ics.forth.gr>> wrote: Hello all, I am looking for a spark profiler to trace my application to find the bottlenecks. I need to trace CPU usage, Memory Usage and I/O usage. I am looking forward for your reply. --Iacovos --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
