If you are profiling in standalone mode, I recommend you to try with Java Mission Control. You just need to start app with these params:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=$YOUR_PORT -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false And them connect with profiling agent, and perform recording. I have strong opinion, that you have epoll wait time spent in one concrete thread, but not in all spawned threads. On Tue, Apr 5, 2016 at 1:34 PM, Dmitry Olshansky <[email protected]> wrote: > Hi list, > > I'm curious as to what are the best practices of profiling spark apps? So > far I tried following this guide with hprof and/or yourkit but the profile > looks strange: > > https://cwiki.apache.org/confluence/display/SPARK/Profiling+Spark+Applications+Using+YourKit > > 55% of time spent in EPollWait. However I'm using standalone mode with > local master without starting separate daemon (could it be that I should?) > > --- > Dmitry Olshansky > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
