Thanks Ak,
This problem has been solved, I use nmon to monitor the system I/O
and CPU pressure and found there is a very sharp peak.And after that peak many
process stops running, so I correct my code and this issue gone.
previous code looks like this:
Runtime.getRuntime.exec("command_a").waitFor()
Runtime.getRuntime.exec("command_b") correct code:
Runtime.getRuntime.exec("command_a+b.sh") Still appreciated to your
suggestion.
--------------------------------
Thanks&Best regards!
San.Luo
----- 原始邮件 -----
发件人:Akhil Das <[email protected]>
收件人:罗辉 <[email protected]>
抄送人:user <[email protected]>
主题:Re: got "java.lang.reflect.UndeclaredThrowableException" when running
multiply APPs in spark
日期:2015年06月30日 15点11分
This:
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [30
seconds]
Could happen for many reasons, one of them could be because of insufficient
memory. Are you running all 20 apps on the same node? How are you submitting
the apps? (with spark-submit?). I see you have driver memory specified as 8g,
does it mean all 20apps are running on the same machine and each will have 8g
of driver memory on a 12g machine?
Can you look in all worker nodes stderr and see the exact reason?
ThanksBest Regards
On Mon, Jun 29, 2015 at 3:08 PM, <[email protected]> wrote:
Hi there
I am running 30 APPs in my spark cluster, and some of the APPs got
exception like below:[root@slave3 0]# cat stderr
15/06/29 17:20:08 INFO executor.CoarseGrainedExecutorBackend: Registered signal
handlers for [TERM, HUP, INT]
15/06/29 17:20:09 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/06/29 17:20:09 INFO spark.SecurityManager: Changing view acls to: root
15/06/29 17:20:09 INFO spark.SecurityManager: Changing modify acls to: root
15/06/29 17:20:09 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/06/29 17:20:09 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/06/29 17:20:09 INFO Remoting: Starting remoting
15/06/29 17:20:10 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://driverPropsFetcher@slave3:51026]
15/06/29 17:20:10 INFO util.Utils: Successfully started service
'driverPropsFetcher' on port 51026.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:128)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:224)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [30
seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:144)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 4 more
when i am running 20 APPs,it is OK. So I doubt this problem looks like executor
get disassicated with the driver due to high I/O pressure or network
latency.however I have no idea which parameter is spark could fix this. Any
idea will be appreciated.
Here is some infomation about my cluster:1master and 6workers.every node has
8cores and 12GB memory.
And settings in my spark-default.conf and spark-env.sh is like this:
spark-default.conf
spark.master spark://master:7077
spark.eventLog.enabled true
spark.eventLog.dir /var/log/spark
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 8g
spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one
two three"
spark.kryoserializer.buffer.max.mb 128
spark.storage.memoryFraction 0.2
spark.shuffle.memoryFraction 0.4
spark.sql.shuffle.partitions 32
spark.scheduler.mode FAIR
spark.worker.cleanup.appDataTtl 259200
spark.port.maxRetries 10000
spark.scheduler.maxRegisteredResourcesWaitingTime 40
spark-env.sh:export SPARK_WORKER_INSTANCES=1
export SPARK_EXECUTOR_INSTANCES=8
export SPARK_EXECUTOR_CORES=1
export SPARK_EXECUTOR_MEMORY=1g
--------------------------------
Thanks&Best regards!
San.Luo