Re: Client session timed out, have not heard from server in
Evan could you also share more logs on the error. Probably paste here or in pastebin. Also check zookeeper logs in case you find anything. - Thanks, via mobile, excuse brevity. On Dec 22, 2015 6:01 PM, "Dirceu Semighini Filho" < dirceu.semigh...@gmail.com> wrote: > Hi Yash, > I've experienced this behavior here when the process freeze in a worker. > This mainly happen, in my case, when the worker memory was full and the > java GC wasn't able to free memory for the process. > Try to search for outofmemory error in your worker logs. > > Regards, > Dirceu > > 2015-12-22 10:26 GMT-02:00 yaoxiaohua : > >> Thanks for your reply. >> >> I find spark-env.sh : >> >> SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300 >> -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true >> -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs >> -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash >> -Dspark.yarn.max.executor.failures=99999 -Dspark.worker.timeout=300" >> >> >> >> I just find log like this: >> >> >> INFO ClientCnxn: Client session timed out, have not heard from server in >> 40015ms for sessionid 0x351c416297a145a, closing socket connection and >> attempting reconnect >> >> Before spark2 master process shut down. >> >> I don’t see any zookeeper timeout setting . >> >> >> >> Best >> >> >> >> *From:* Yash Sharma [mailto:yash...@gmail.com] >> *Sent:* 2015年12月22日 19:55 >> *To:* yaoxiaohua >> *Cc:* user@spark.apache.org >> *Subject:* Re: Client session timed out, have not heard from server in >> >> >> >> Hi Evan, >> SPARK-9629 referred to connection issues with zookeeper. Could you check >> if its working fine in your setup. >> >> Also please share other error logs you might be getting. >> >> - Thanks, via mobile, excuse brevity. >> >> On Dec 22, 2015 5:00 PM, "yaoxiaohua" wrote: >> >> Hi, >> >> I encounter a similar question, spark1.4 >> >> Master2 run some days , then give a timeout exception, then shutdown. >> >> I found a bug : >> >> https://issues.apache.org/jira/browse/SPARK-9629 >> >> >> >> >> INFO ClientCnxn: Client session timed out, have not heard from server in >> 40015ms for sessionid 0x351c416297a145a, closing socket connection and >> attempting reconnect >> >> >> >> >> >> could you tell me what do you do for this? >> >> >> >> Best Regards, >> >> Evan >> > >
Re: Client session timed out, have not heard from server in
Hi Yash, I've experienced this behavior here when the process freeze in a worker. This mainly happen, in my case, when the worker memory was full and the java GC wasn't able to free memory for the process. Try to search for outofmemory error in your worker logs. Regards, Dirceu 2015-12-22 10:26 GMT-02:00 yaoxiaohua : > Thanks for your reply. > > I find spark-env.sh : > > SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300 > -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true > -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs > -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash > -Dspark.yarn.max.executor.failures=9 -Dspark.worker.timeout=300" > > > > I just find log like this: > > > INFO ClientCnxn: Client session timed out, have not heard from server in > 40015ms for sessionid 0x351c416297a145a, closing socket connection and > attempting reconnect > > Before spark2 master process shut down. > > I don’t see any zookeeper timeout setting . > > > > Best > > > > *From:* Yash Sharma [mailto:yash...@gmail.com] > *Sent:* 2015年12月22日 19:55 > *To:* yaoxiaohua > *Cc:* user@spark.apache.org > *Subject:* Re: Client session timed out, have not heard from server in > > > > Hi Evan, > SPARK-9629 referred to connection issues with zookeeper. Could you check > if its working fine in your setup. > > Also please share other error logs you might be getting. > > - Thanks, via mobile, excuse brevity. > > On Dec 22, 2015 5:00 PM, "yaoxiaohua" wrote: > > Hi, > > I encounter a similar question, spark1.4 > > Master2 run some days , then give a timeout exception, then shutdown. > > I found a bug : > > https://issues.apache.org/jira/browse/SPARK-9629 > > > > > INFO ClientCnxn: Client session timed out, have not heard from server in > 40015ms for sessionid 0x351c416297a145a, closing socket connection and > attempting reconnect > > > > > > could you tell me what do you do for this? > > > > Best Regards, > > Evan >
RE: Client session timed out, have not heard from server in
Thanks for your reply. I find spark-env.sh : SPARK_JAVA_OPTS="$SPARK_JAVA_OPTS -Dspark.akka.askTimeout=300 -Dspark.ui.retainedStages=1000 -Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs://sparkcluster/user/spark_history_logs -Dspark.shuffle.spill=false -Dspark.shuffle.manager=hash -Dspark.yarn.max.executor.failures=9 -Dspark.worker.timeout=300" I just find log like this: INFO ClientCnxn: Client session timed out, have not heard from server in 40015ms for sessionid 0x351c416297a145a, closing socket connection and attempting reconnect Before spark2 master process shut down. I don’t see any zookeeper timeout setting . Best From: Yash Sharma [mailto:yash...@gmail.com] Sent: 2015年12月22日 19:55 To: yaoxiaohua Cc: user@spark.apache.org Subject: Re: Client session timed out, have not heard from server in Hi Evan, SPARK-9629 referred to connection issues with zookeeper. Could you check if its working fine in your setup. Also please share other error logs you might be getting. - Thanks, via mobile, excuse brevity. On Dec 22, 2015 5:00 PM, "yaoxiaohua" wrote: Hi, I encounter a similar question, spark1.4 Master2 run some days , then give a timeout exception, then shutdown. I found a bug : https://issues.apache.org/jira/browse/SPARK-9629 INFO ClientCnxn: Client session timed out, have not heard from server in 40015ms for sessionid 0x351c416297a145a, closing socket connection and attempting reconnect could you tell me what do you do for this? Best Regards, Evan
Re: Client session timed out, have not heard from server in
Hi Evan, SPARK-9629 referred to connection issues with zookeeper. Could you check if its working fine in your setup. Also please share other error logs you might be getting. - Thanks, via mobile, excuse brevity. On Dec 22, 2015 5:00 PM, "yaoxiaohua" wrote: > Hi, > > I encounter a similar question, spark1.4 > > Master2 run some days , then give a timeout exception, then shutdown. > > I found a bug : > > https://issues.apache.org/jira/browse/SPARK-9629 > > > > > INFO ClientCnxn: Client session timed out, have not heard from server in > 40015ms for sessionid 0x351c416297a145a, closing socket connection and > attempting reconnect > > > > > > could you tell me what do you do for this? > > > > Best Regards, > > Evan >
Client session timed out, have not heard from server in
Hi, I encounter a similar question, spark1.4 Master2 run some days , then give a timeout exception, then shutdown. I found a bug : https://issues.apache.org/jira/browse/SPARK-9629 INFO ClientCnxn: Client session timed out, have not heard from server in 40015ms for sessionid 0x351c416297a145a, closing socket connection and attempting reconnect could you tell me what do you do for this? Best Regards, Evan
