Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Will Benton
Hi all, I've been evaluating YourKit and would like to profile the heap and CPU usage of certain tests from the Spark test suite. In particular, I'm very interested in tracking heap usage by allocation site. Unfortunately, I get a lot of crashes running Spark tests with profiling (and thus

Re: Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Matei Zaharia
I haven't seen issues using the JVM's own tools (jstack, jmap, hprof and such), so maybe there's a problem in YourKit or in your release of the JVM. Otherwise I'd suggest increasing the heap size of the unit tests a bit (you can do this in the SBT build file). Maybe they are very close to full

Re: Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Will Benton
Thanks, Matei; I have also had some success with jmap and friends and will probably just stick with them! best, wb - Original Message - From: Matei Zaharia matei.zaha...@gmail.com To: dev@spark.apache.org Sent: Monday, July 14, 2014 1:02:04 PM Subject: Re: Profiling Spark tests

Re: Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Will Benton
- Original Message - From: Aaron Davidson ilike...@gmail.com To: dev@spark.apache.org Sent: Monday, July 14, 2014 5:21:10 PM Subject: Re: Profiling Spark tests with YourKit (or something else) Out of curiosity, what problems are you seeing with Utils.getCallSite? Aaron, if I enable

Re: Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Aaron Davidson
Message - From: Aaron Davidson ilike...@gmail.com To: dev@spark.apache.org Sent: Monday, July 14, 2014 5:21:10 PM Subject: Re: Profiling Spark tests with YourKit (or something else) Out of curiosity, what problems are you seeing with Utils.getCallSite? Aaron, if I enable call site

Re: Profiling Spark tests with YourKit (or something else)

2014-07-14 Thread Will Benton
...@redhat.com wrote: - Original Message - From: Aaron Davidson ilike...@gmail.com To: dev@spark.apache.org Sent: Monday, July 14, 2014 5:21:10 PM Subject: Re: Profiling Spark tests with YourKit (or something else) Out of curiosity, what problems are you seeing