Here is a UI of my thread dump. http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTYvMTEvMS8tLWpzdGFja19kdW1wX3dpbmRvd19pbnRlcnZhbF8xbWluX2JhdGNoX2ludGVydmFsXzFzLnR4dC0tNi0xNy00Ng==
On Mon, Oct 31, 2016 at 10:32 PM, kant kodali <kanth...@gmail.com> wrote: > Hi Vadim, > > Thank you so much this was a very useful command. This conversation is > going on here > > https://www.mail-archive.com/user@spark.apache.org/msg58656.html > > or you can just google " > > why spark driver program is creating so many threads? How can I limit this > number? > <https://www.mail-archive.com/search?l=user@spark.apache.org&q=subject:%22Re%5C%3A+why+spark+driver+program+is+creating+so+many+threads%5C%3F+How+can+I+limit+this+number%5C%3F%22&o=newest> > " > > please take a look if you are interested. > > Thanks a lot! > > On Mon, Oct 31, 2016 at 8:14 AM, Vadim Semenov < > vadim.seme...@datadoghq.com> wrote: > >> Have you tried to get number of threads in a running process using `cat >> /proc/<pid>/status` ? >> >> On Sun, Oct 30, 2016 at 11:04 PM, kant kodali <kanth...@gmail.com> wrote: >> >>> yes I did run ps -ef | grep "app_name" and it is root. >>> >>> >>> >>> On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <chin...@indetail.co.jp> >>> wrote: >>> >>>> sorry, the UID >>>> >>>> On 10/31/16 11:59 AM, Chan Chor Pang wrote: >>>> >>>> actually if the max user processes is not the problem, i have no idea >>>> >>>> but i still suspecting the user, >>>> as the user who run spark-submit is not necessary the pid for the JVM >>>> process >>>> >>>> can u make sure when you "ps -ef | grep {your app id} " the PID is root? >>>> On 10/31/16 11:21 AM, kant kodali wrote: >>>> >>>> The java process is run by the root and it has the same config >>>> >>>> sudo -i >>>> >>>> ulimit -a >>>> >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 0 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 120242 >>>> max locked memory (kbytes, -l) 64 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 1024 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 8192 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) 120242 >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> >>>> >>>> On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <chin...@indetail.co.jp >>>> > wrote: >>>> >>>>> I have the same Exception before and the problem fix after i change >>>>> the nproc conf. >>>>> >>>>> > max user processes (-u) 120242 >>>>> ↑this config does looks good. >>>>> are u sure the user who run ulimit -a is the same user who run the >>>>> Java process? >>>>> depend on how u submit the job and your setting, spark job may execute >>>>> by other user. >>>>> >>>>> >>>>> On 10/31/16 10:38 AM, kant kodali wrote: >>>>> >>>>> when I did this >>>>> >>>>> cat /proc/sys/kernel/pid_max >>>>> >>>>> I got 32768 >>>>> >>>>> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com> >>>>> wrote: >>>>> >>>>>> I believe for ubuntu it is unlimited but I am not 100% sure (I just >>>>>> read somewhere online). I ran ulimit -a and this is what I get >>>>>> >>>>>> core file size (blocks, -c) 0 >>>>>> data seg size (kbytes, -d) unlimited >>>>>> scheduling priority (-e) 0 >>>>>> file size (blocks, -f) unlimited >>>>>> pending signals (-i) 120242 >>>>>> max locked memory (kbytes, -l) 64 >>>>>> max memory size (kbytes, -m) unlimited >>>>>> open files (-n) 1024 >>>>>> pipe size (512 bytes, -p) 8 >>>>>> POSIX message queues (bytes, -q) 819200 >>>>>> real-time priority (-r) 0 >>>>>> stack size (kbytes, -s) 8192 >>>>>> cpu time (seconds, -t) unlimited >>>>>> max user processes (-u) 120242 >>>>>> virtual memory (kbytes, -v) unlimited >>>>>> file locks (-x) unlimited >>>>>> >>>>>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang < >>>>>> chin...@indetail.co.jp> wrote: >>>>>> >>>>>>> not sure for ubuntu, but i think you can just create the file by >>>>>>> yourself >>>>>>> the syntax will be the same as /etc/security/limits.conf >>>>>>> >>>>>>> nproc.conf not only limit java process but all process by the same >>>>>>> user >>>>>>> so even the jvm process does nothing, if the corresponding user is >>>>>>> busy in other way >>>>>>> the jvm process will still not able to create new thread. >>>>>>> >>>>>>> btw the default limit for centos is 1024 >>>>>>> >>>>>>> >>>>>>> On 10/31/16 9:51 AM, kant kodali wrote: >>>>>>> >>>>>>> >>>>>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang < >>>>>>> chin...@indetail.co.jp> wrote: >>>>>>> >>>>>>>> /etc/security/limits.d/90-nproc.conf >>>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am using Ubuntu 16.04 LTS. I have this directory >>>>>>> /etc/security/limits.d/ but I don't have any files underneath it. This >>>>>>> error happens after running for 4 to 5 hours. I wonder if this is a GC >>>>>>> issue? And I am thinking if I should use CMS. I have also posted this >>>>>>> on SO >>>>>>> since I havent got much response for this question >>>>>>> http://stackoverflow.com/questions/40315589/dag-sch >>>>>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat >>>>>>> e-new-native >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> kant >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ---*------------------------------------------------*---*---*---*--- >>>>>>> 株式会社INDETAIL >>>>>>> ニアショア総合サービス事業本部 >>>>>>> ゲームサービス事業部 >>>>>>> 陳 楚鵬 >>>>>>> E-mail :chin...@indetail.co.jp >>>>>>> URL : http://www.indetail.co.jp >>>>>>> >>>>>>> 【札幌本社／LABO／LABO2】 >>>>>>> 〒060-0042 >>>>>>> 札幌市中央区大通西9丁目3番地33 >>>>>>> キタコーセンタービルディング >>>>>>> （札幌本社／LABO2：2階、LABO：9階） >>>>>>> TEL：011-206-9235 FAX：011-206-9236 >>>>>>> >>>>>>> 【東京支店】 >>>>>>> 〒108-0014 >>>>>>> 東京都港区芝5丁目29番20号 クロスオフィス三田 >>>>>>> TEL：03-6809-6502 FAX：03-6809-6504 >>>>>>> >>>>>>> 【名古屋サテライト】 >>>>>>> 〒460-0002 >>>>>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD >>>>>>> TEL：052-971-0086 >>>>>>> >>>>>>> >>> >> >
