Umesh, Running task is a thread within the executor process. We need to take stack trace for the executor process. The executor will be running in any NodeManager machine as a container.
YARN RM UI running jobs will have the host details where executor is running. Login to that NodeManager machine and jps -l will list all java processes, jstack -l <pid> will give the stack trace. Thanks, Prabhu Joseph On Mon, Jan 11, 2016 at 7:56 PM, Umesh Kacha <umesh.ka...@gmail.com> wrote: > Hi Prabhu thanks for the response. How do I find pid of a slow running > task. Task is running in yarn cluster node. When I try to see pid of a > running task using my user I see some 7-8 digit number instead of user > running process any idea why spark creates this number instead of > displaying user > On Jan 3, 2016 6:06 AM, "Prabhu Joseph" <prabhujose.ga...@gmail.com> > wrote: > >> The attached image just has thread states, and WAITING threads need not >> be the issue. We need to take thread stack traces and identify at which >> area of code, threads are spending lot of time. >> >> Use jstack -l <pid> or kill -3 <pid>, where pid is the process id of the >> executor process. Take jstack stack trace for every 2 seconds and total 1 >> minute. This will help to identify the code where threads are spending lot >> of time and then try to tune. >> >> Thanks, >> Prabhu Joseph >> >> >> >> On Sat, Jan 2, 2016 at 1:28 PM, Umesh Kacha <umesh.ka...@gmail.com> >> wrote: >> >>> Hi thanks I did that and I have attached thread dump images. That was >>> the intention of my question asking for help to identify which waiting >>> thread is culprit. >>> >>> Regards, >>> Umesh >>> >>> On Sat, Jan 2, 2016 at 8:38 AM, Prabhu Joseph < >>> prabhujose.ga...@gmail.com> wrote: >>> >>>> Take thread dump of Executor process several times in a short time >>>> period and check what each threads are doing at different times which will >>>> help to identify the expensive sections in user code. >>>> >>>> Thanks, >>>> Prabhu Joseph >>>> >>>> On Sat, Jan 2, 2016 at 3:28 AM, unk1102 <umesh.ka...@gmail.com> wrote: >>>> >>>>> Sorry please see attached waiting thread log >>>>> >>>>> < >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n25851/Screen_Shot_2016-01-02_at_2.jpg >>>>> > >>>>> < >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n25851/Screen_Shot_2016-01-02_at_2.jpg >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-find-cause-waiting-threads-etc-of-hanging-job-for-7-hours-tp25850p25851.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>> >>
