I am trying to run spark-shell on yarn (hadoop 2.2.0-cdh5.0.0-beta-1, spark 0.8.1).
I set up the shell to run via: export SPARK_JAR=/usr/lib/spark/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0-cdh5.0.0-beta-1.jar export SPARK_YARN_APP_JAR=/usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar export SPARK_WORKER_CORES=8 export SPARK_WORKER_MEMORY=2g export SPARK_WORKER_INSTANCES=5 export MASTER=yarn-client ./spark-shell The shell launches but the spark cluster immediately dies with this error: Application application_1390861968364_0092 failed 2 times due to AM Container for appattempt_1390861968364_0092_000002 exited with exitCode: 143 due to: Container [pid=18322,containerID=container_1390861968364_0092_02_000001] is running beyond virtual memory limits. Current usage: 277.2 MB of 8 GB physical memory used; 23.0 GB of 16.8 GB virtual memory used. Killing container. Dump of the process-tree for container_1390861968364_0092_02_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 18327 18322 18322 18322 (java) 722 28 24690708480 70669 java -server -Xmx512m -Djava.io.tmpdir=/disks/08/yarn/local/usercache/ekimbrel/appcache/application_1390861968364_0092/container_1390861968364_0092_02_000001/tmp -Xss1G org.apache.spark.deploy.yarn.WorkerLauncher --class notused --jar /usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar --args arcus2.silverdale.dev:45264 --worker-memory 184320 --worker-cores 8 --num-workers 5 |- 18322 24758 18322 18322 (bash) 0 0 9461760 285 /bin/bash -c java -server -Xmx512m -Djava.io.tmpdir=/disks/08/yarn/local/usercache/ekimbrel/appcache/application_1390861968364_0092/container_1390861968364_0092_02_000001/tmp -Xss1G org.apache.spark.deploy.yarn.WorkerLauncher --class notused --jar /usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar --args 'arcus2.silverdale.dev:45264' --worker-memory 184320 --worker-cores 8 --num-workers 5 1> /disks/17/yarn/logs/application_1390861968364_0092/container_1390861968364_0092_02_000001/stdout 2> /disks/17/yarn/logs/application_1390861968364_0092/container_1390861968364_0092_02_000001/stderr Container killed on request. Exit code is 143 .Failing this attempt.. Failing the application. I notice two things right away. The first is that my SPARK_WORKER_MEMORY setting isn't getting picked up and it seems to instead be using the value form spark_env.sh The second is of course this virtual memory error. Yarn by default allocates 2x physical to virtual memory, but i don't know why 8GB of memory is being allocated and i don't know why 23GB of virtual memory is being claimed. This is only launching the cluster and the spark-shell, no other action is taken. Thanks for any help / advice, Eric -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-with-yarn-runs-beyond-virtual-memory-limits-tp1050.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
