Thanks, Matei; I have also had some success with jmap and friends and will probably just stick with them!
best, wb ----- Original Message ----- > From: "Matei Zaharia" <matei.zaha...@gmail.com> > To: dev@spark.apache.org > Sent: Monday, July 14, 2014 1:02:04 PM > Subject: Re: Profiling Spark tests with YourKit (or something else) > > I haven't seen issues using the JVM's own tools (jstack, jmap, hprof and > such), so maybe there's a problem in YourKit or in your release of the JVM. > Otherwise I'd suggest increasing the heap size of the unit tests a bit (you > can do this in the SBT build file). Maybe they are very close to full and > profiling pushes them over the edge. > > Matei > > On Jul 14, 2014, at 9:51 AM, Will Benton <wi...@redhat.com> wrote: > > > Hi all, > > > > I've been evaluating YourKit and would like to profile the heap and CPU > > usage of certain tests from the Spark test suite. In particular, I'm very > > interested in tracking heap usage by allocation site. Unfortunately, I > > get a lot of crashes running Spark tests with profiling (and thus > > allocation-site tracking) enabled in YourKit; just using the sampler works > > fine, but it appears that enabling the profiler breaks Utils.getCallSite. > > > > Is there a way to make this combination work? If not, what are people > > using to understand the memory and CPU behavior of Spark and Spark apps? > > > > > > thanks, > > wb > >
