This does not seem to help: export SPARK_JAVA_OPTS="-Dspark.local.dir=/app/spark/tmp -Dspark.worker.timeout=600 -Dspark.akka.timeout=200 -Dspark.storage.blockManagerSlaveTimeoutMs=300000"
Getting the message leads to GC failure followed by master declaring the worker as dead ! This is related to GC...Persisting the factors to disk at each iteration will resolve this issue with runtime loss of course... I also have another issue...I run with executor memory as 24g but I see 18.4 GB in executor ui...is that expected ? On Sat, Apr 5, 2014 at 8:16 AM, Debasish Das <[email protected]>wrote: > From the documentation this is what I understood: > > 1. spark.worker.timeout: Number of seconds after which the standalone > deploy master considers a worker lost if it receives no heartbeats. > default: 60 > > I increased it to be 600 > > It was pointed before that if there is GC overload and the worker takes > time to respond, master thinks worker JVM died. > > I have seen this issue as well several times. > > 2. spark.akka.timeout: Communication timeout between Spark nodes, in > seconds. > default: 100 > > I increased it to 200 as it was pointed before but I don't understand when > the communication timeout is triggered. Some explanation on this setting > will be very helpful. > > 3. spark.storage.blockManagerSlaveTimeoutMs: I could not find > documentation but as Patrick said the 45000 number coming from this. > > How is this related to spark.worker.timeout? > > I bumped it up to 300s but JVM can go to GC only if there is pressure on > JVM right....May be I need to do a yourkit run to understand the memory > usage in more detail. Any suggestions on how to setup yourkit for memory > analysis ? > > I set it using the following options in spark_env.sh: > > export SPARK_JAVA_OPTS="-Dspark.local.dir=/app/spark/tmp > -Dspark.storage.blockManagerSlaveTimeoutMs=300000 > -Dspark.worker.timeout=600 -Dspark.akka.timeout=200" > > > > This is the correct way to specify > spark.storage.blockManagerSlaveTimeoutMs ? > > > On Sat, Apr 5, 2014 at 4:00 AM, azurecoder <[email protected]>wrote: > >> Interested in a resolution to this. I'm building a large triangular >> matrix so >> doing similar to ALS - lots of work on the worker nodes and keep timing >> out. >> >> Tried a few updates to akka frame sizes, timeouts and blockmanager but >> unable to complete. Will try the blockmanagerslaves property now and let >> you >> know the effect. That property doesn't appear to be documented on the site >> though. >> >> Cheers! >> >> Richard >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Heartbeat-exceeds-tp3798p3809.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >
