Re: Happy Diwali to those forum members who celebrate this great festival

Thanks a lot and same to you all as well ;) On 31-Oct-2016 1:35 AM, "Mich Talebzadeh" wrote: > I can hear and see plenty of firework in this foggy London tonight :) > > Dr Mich Talebzadeh > > > > LinkedIn * >

Re: java.lang.OutOfMemoryError: unable to create new native thread

yes I did run ps -ef | grep "app_name" and it is root. On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang wrote: > sorry, the UID > > On 10/31/16 11:59 AM, Chan Chor Pang wrote: > > actually if the max user processes is not the problem, i have no idea > > but i still

Re: java.lang.OutOfMemoryError: unable to create new native thread

sorry, the UID On 10/31/16 11:59 AM, Chan Chor Pang wrote: actually if the max user processes is not the problem, i have no idea but i still suspecting the user, as the user who run spark-submit is not necessary the pid for the JVM process can u make sure when you "ps -ef | grep {your app

Re: java.lang.OutOfMemoryError: unable to create new native thread

actually if the max user processes is not the problem, i have no idea but i still suspecting the user, as the user who run spark-submit is not necessary the pid for the JVM process can u make sure when you "ps -ef | grep {your app id} " the PID is root? On 10/31/16 11:21 AM, kant kodali

Re: java.lang.OutOfMemoryError: unable to create new native thread

The java process is run by the root and it has the same config sudo -i ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i)

Re: java.lang.OutOfMemoryError: unable to create new native thread

I have the same Exception before and the problem fix after i change the nproc conf. > max user processes (-u) 120242 ↑this config does looks good. are u sure the user who run ulimit -a is the same user who run the Java process? depend on how u submit the job and your setting,

Re: java.lang.OutOfMemoryError: unable to create new native thread

when I did this cat /proc/sys/kernel/pid_max I got 32768 On Sun, Oct 30, 2016 at 6:36 PM, kant kodali wrote: > I believe for ubuntu it is unlimited but I am not 100% sure (I just read > somewhere online). I ran ulimit -a and this is what I get > > core file size

Re: java.lang.OutOfMemoryError: unable to create new native thread

I believe for ubuntu it is unlimited but I am not 100% sure (I just read somewhere online). I ran ulimit -a and this is what I get core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f)

Re: java.lang.OutOfMemoryError: unable to create new native thread

not sure for ubuntu, but i think you can just create the file by yourself the syntax will be the same as /etc/security/limits.conf nproc.conf not only limit java process but all process by the same user so even the jvm process does nothing, if the corresponding user is busy in other way the

Re: java.lang.OutOfMemoryError: unable to create new native thread

On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang wrote: > /etc/security/limits.d/90-nproc.conf > Hi, I am using Ubuntu 16.04 LTS. I have this directory /etc/security/limits.d/ but I don't have any files underneath it. This error happens after running for 4 to 5 hours. I

Re: java.lang.OutOfMemoryError: unable to create new native thread

you may want to check the process limit of the user who responsible for starting the JVM. /etc/security/limits.d/90-nproc.conf On 10/29/16 4:47 AM, kant kodali wrote: "dag-scheduler-event-loop" java.lang.OutOfMemoryError: unable to create new native thread at

Re: Happy Diwali to those forum members who celebrate this great festival

I can hear and see plenty of firework in this foggy London tonight :) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Performance bug in UDAF?

Hi All, I have a UDAF that seems to perform poorly when its input is skewed. I have been debugging the UDAF implementation but I don't see any code that is causing the performance to degrade. More details on the data and the experiments I have run. DataSet: Assume 3 columns, column1 being the

Re: Happy Diwali to those forum members who celebrate this great festival

+1 -- Lefty On Sun, Oct 30, 2016 at 1:01 PM, Ashok Kumar wrote: > You are very kind Sir > > > On Sunday, 30 October 2016, 16:42, Devopam Mittra > wrote: > > > +1 > Thanks and regards > Devopam > > On 30 Oct 2016 9:37 pm, "Mich Talebzadeh"

Re: Happy Diwali to those forum members who celebrate this great festival

You are very kind Sir On Sunday, 30 October 2016, 16:42, Devopam Mittra wrote: +1 Thanks and regards Devopam On 30 Oct 2016 9:37 pm, "Mich Talebzadeh" wrote: Enjoy the festive season. Regards, Dr Mich Talebzadeh LinkedIn  

Re: Happy Diwali to those forum members who celebrate this great festival

Thank you Dr Mich :) Regards Sivakumaran S > On 30-Oct-2016, at 4:07 PM, Mich Talebzadeh wrote: > > Enjoy the festive season. > > Regards, > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >

Happy Diwali to those forum members who celebrate this great festival

Enjoy the festive season. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Spark 2.0 with Hadoop 3.0?

The version problems are related to using hadoop-aws-2.7.3 alongside aws-sdk-1.7.4 in hadoop-2.7.3 where DynamoDb functionality is limited (it may not even operate with deployed versions of the service). I've stripped out usage of DynamoDB from the driver program in the meantime (using it in a