Henry Saputra created SPARK-2586: ------------------------------------ Summary: Lack of information to figure out connection to Tachyon master is inactive/ down Key: SPARK-2586 URL: https://issues.apache.org/jira/browse/SPARK-2586 Project: Spark Issue Type: Bug Components: Spark Core Reporter: Henry Saputra
When you running Spark with Tachyon, when the connection to Tachyon master is down (due to problem in network or the Master node is down) there is no clear log or error message to indicate it. Here is sample stack running SparkTachyonPi example with Tachyon connecting: 14/07/15 16:43:10 INFO Utils: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/07/15 16:43:10 WARN Utils: Your hostname, henry-pivotal.local resolves to a loopback address: 127.0.0.1; using 10.64.5.148 instead (on interface en5) 14/07/15 16:43:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/07/15 16:43:11 INFO SecurityManager: Changing view acls to: hsaputra 14/07/15 16:43:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hsaputra) 14/07/15 16:43:11 INFO Slf4jLogger: Slf4jLogger started 14/07/15 16:43:11 INFO Remoting: Starting remoting 14/07/15 16:43:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@office-5-148.pa.gopivotal.com:53203] 14/07/15 16:43:11 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@office-5-148.pa.gopivotal.com:53203] 14/07/15 16:43:11 INFO SparkEnv: Registering MapOutputTracker 14/07/15 16:43:11 INFO SparkEnv: Registering BlockManagerMaster 14/07/15 16:43:11 INFO DiskBlockManager: Created local directory at /var/folders/nv/nsr_3ysj0wgfq93fqp0rdt3w0000gp/T/spark-local-20140715164311-e63c 14/07/15 16:43:11 INFO ConnectionManager: Bound socket to port 53204 with id = ConnectionManagerId(office-5-148.pa.gopivotal.com,53204) 14/07/15 16:43:11 INFO MemoryStore: MemoryStore started with capacity 2.1 GB 14/07/15 16:43:11 INFO BlockManagerMaster: Trying to register BlockManager 14/07/15 16:43:11 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:53204 with 2.1 GB RAM 14/07/15 16:43:11 INFO BlockManagerMaster: Registered BlockManager 14/07/15 16:43:11 INFO HttpServer: Starting HTTP Server 14/07/15 16:43:11 INFO HttpBroadcast: Broadcast server started at http://10.64.5.148:53205 14/07/15 16:43:11 INFO HttpFileServer: HTTP File server directory is /var/folders/nv/nsr_3ysj0wgfq93fqp0rdt3w0000gp/T/spark-b2fb12ae-4608-4833-87b6-b335da00738e 14/07/15 16:43:11 INFO HttpServer: Starting HTTP Server 14/07/15 16:43:12 INFO SparkUI: Started SparkUI at http://office-5-148.pa.gopivotal.com:4040 2014-07-15 16:43:12.210 java[39068:1903] Unable to load realm info from SCDynamicStore 14/07/15 16:43:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/07/15 16:43:12 INFO SparkContext: Added JAR examples/target/scala-2.10/spark-examples-1.1.0-SNAPSHOT-hadoop2.4.0.jar at http://10.64.5.148:53206/jars/spark-examples-1.1.0-SNAPSHOT-hadoop2.4.0.jar with timestamp 1405467792813 14/07/15 16:43:12 INFO AppClient$ClientActor: Connecting to master spark://henry-pivotal.local:7077... 14/07/15 16:43:12 INFO SparkContext: Starting job: reduce at SparkTachyonPi.scala:43 14/07/15 16:43:12 INFO DAGScheduler: Got job 0 (reduce at SparkTachyonPi.scala:43) with 2 output partitions (allowLocal=false) 14/07/15 16:43:12 INFO DAGScheduler: Final stage: Stage 0(reduce at SparkTachyonPi.scala:43) 14/07/15 16:43:12 INFO DAGScheduler: Parents of final stage: List() 14/07/15 16:43:12 INFO DAGScheduler: Missing parents: List() 14/07/15 16:43:12 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at map at SparkTachyonPi.scala:39), which has no missing parents 14/07/15 16:43:13 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[1] at map at SparkTachyonPi.scala:39) 14/07/15 16:43:13 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 14/07/15 16:43:13 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140715164313-0000 14/07/15 16:43:13 INFO AppClient$ClientActor: Executor added: app-20140715164313-0000/0 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:43:13 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164313-0000/0 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:43:13 INFO AppClient$ClientActor: Executor updated: app-20140715164313-0000/0 is now RUNNING 14/07/15 16:43:15 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@office-5-148.pa.gopivotal.com:53213/user/Executor#-423405256] with ID 0 14/07/15 16:43:15 INFO TaskSetManager: Re-computing pending task lists. 14/07/15 16:43:15 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor 0: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:43:15 INFO TaskSetManager: Serialized task 0.0:0 as 1428 bytes in 3 ms 14/07/15 16:43:15 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor 0: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:43:15 INFO TaskSetManager: Serialized task 0.0:1 as 1428 bytes in 1 ms 14/07/15 16:43:15 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:53218 with 294.9 MB RAM 14/07/15 16:43:16 INFO BlockManagerInfo: Added rdd_0_1 on tachyon on office-5-148.pa.gopivotal.com:53218 (size: 977.2 KB) 14/07/15 16:43:16 INFO BlockManagerInfo: Added rdd_0_0 on tachyon on office-5-148.pa.gopivotal.com:53218 (size: 977.2 KB) 14/07/15 16:43:16 INFO TaskSetManager: Finished TID 0 in 1307 ms on office-5-148.pa.gopivotal.com (progress: 1/2) 14/07/15 16:43:16 INFO TaskSetManager: Finished TID 1 in 1300 ms on office-5-148.pa.gopivotal.com (progress: 2/2) 14/07/15 16:43:16 INFO DAGScheduler: Completed ResultTask(0, 0) 14/07/15 16:43:16 INFO DAGScheduler: Completed ResultTask(0, 1) 14/07/15 16:43:16 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/07/15 16:43:16 INFO DAGScheduler: Stage 0 (reduce at SparkTachyonPi.scala:43) finished in 3.336 s 14/07/15 16:43:16 INFO SparkContext: Job finished: reduce at SparkTachyonPi.scala:43, took 3.413498 s Pi is roughly 3.14254 14/07/15 16:43:16 INFO SparkUI: Stopped Spark web UI at http://office-5-148.pa.gopivotal.com:4040 14/07/15 16:43:16 INFO DAGScheduler: Stopping DAGScheduler 14/07/15 16:43:16 INFO SparkDeploySchedulerBackend: Shutting down all executors 14/07/15 16:43:16 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 14/07/15 16:43:17 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 14/07/15 16:43:17 INFO ConnectionManager: Selector thread was interrupted! 14/07/15 16:43:17 INFO ConnectionManager: ConnectionManager stopped 14/07/15 16:43:17 INFO MemoryStore: MemoryStore cleared 14/07/15 16:43:17 INFO BlockManager: BlockManager stopped 14/07/15 16:43:17 INFO BlockManagerMasterActor: Stopping BlockManagerMaster 14/07/15 16:43:17 INFO BlockManagerMaster: BlockManagerMaster stopped 14/07/15 16:43:17 INFO SparkContext: Successfully stopped SparkContext 14/07/15 16:43:17 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 14/07/15 16:43:17 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. Process finished with exit code 0 And here is the stack when Tachyon cannot be reached: 14/07/15 16:49:17 INFO Utils: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/07/15 16:49:17 WARN Utils: Your hostname, henry-pivotal.local resolves to a loopback address: 127.0.0.1; using 10.64.5.148 instead (on interface en5) 14/07/15 16:49:17 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/07/15 16:49:17 INFO SecurityManager: Changing view acls to: hsaputra 14/07/15 16:49:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hsaputra) 14/07/15 16:49:17 INFO Slf4jLogger: Slf4jLogger started 14/07/15 16:49:17 INFO Remoting: Starting remoting 14/07/15 16:49:17 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@office-5-148.pa.gopivotal.com:54541] 14/07/15 16:49:17 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@office-5-148.pa.gopivotal.com:54541] 14/07/15 16:49:17 INFO SparkEnv: Registering MapOutputTracker 14/07/15 16:49:17 INFO SparkEnv: Registering BlockManagerMaster 14/07/15 16:49:17 INFO DiskBlockManager: Created local directory at /var/folders/nv/nsr_3ysj0wgfq93fqp0rdt3w0000gp/T/spark-local-20140715164917-bf9e 14/07/15 16:49:17 INFO ConnectionManager: Bound socket to port 54542 with id = ConnectionManagerId(office-5-148.pa.gopivotal.com,54542) 14/07/15 16:49:17 INFO MemoryStore: MemoryStore started with capacity 2.1 GB 14/07/15 16:49:17 INFO BlockManagerMaster: Trying to register BlockManager 14/07/15 16:49:17 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:54542 with 2.1 GB RAM 14/07/15 16:49:17 INFO BlockManagerMaster: Registered BlockManager 14/07/15 16:49:17 INFO HttpServer: Starting HTTP Server 14/07/15 16:49:17 INFO HttpBroadcast: Broadcast server started at http://10.64.5.148:54543 14/07/15 16:49:17 INFO HttpFileServer: HTTP File server directory is /var/folders/nv/nsr_3ysj0wgfq93fqp0rdt3w0000gp/T/spark-400178c7-8c6e-4e44-9610-926bd1f84877 14/07/15 16:49:17 INFO HttpServer: Starting HTTP Server 14/07/15 16:49:18 INFO SparkUI: Started SparkUI at http://office-5-148.pa.gopivotal.com:4040 2014-07-15 16:49:18.144 java[39346:1903] Unable to load realm info from SCDynamicStore 14/07/15 16:49:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/07/15 16:49:18 INFO SparkContext: Added JAR examples/target/scala-2.10/spark-examples-1.1.0-SNAPSHOT-hadoop2.4.0.jar at http://10.64.5.148:54544/jars/spark-examples-1.1.0-SNAPSHOT-hadoop2.4.0.jar with timestamp 1405468158551 14/07/15 16:49:18 INFO AppClient$ClientActor: Connecting to master spark://henry-pivotal.local:7077... 14/07/15 16:49:18 INFO SparkContext: Starting job: reduce at SparkTachyonPi.scala:43 14/07/15 16:49:18 INFO DAGScheduler: Got job 0 (reduce at SparkTachyonPi.scala:43) with 2 output partitions (allowLocal=false) 14/07/15 16:49:18 INFO DAGScheduler: Final stage: Stage 0(reduce at SparkTachyonPi.scala:43) 14/07/15 16:49:18 INFO DAGScheduler: Parents of final stage: List() 14/07/15 16:49:18 INFO DAGScheduler: Missing parents: List() 14/07/15 16:49:18 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at map at SparkTachyonPi.scala:39), which has no missing parents 14/07/15 16:49:18 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[1] at map at SparkTachyonPi.scala:39) 14/07/15 16:49:18 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 14/07/15 16:49:18 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140715164918-0001 14/07/15 16:49:18 INFO AppClient$ClientActor: Executor added: app-20140715164918-0001/0 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:49:18 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164918-0001/0 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:49:18 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/0 is now RUNNING 14/07/15 16:49:20 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@office-5-148.pa.gopivotal.com:54548/user/Executor#-221675010] with ID 0 14/07/15 16:49:20 INFO TaskSetManager: Re-computing pending task lists. 14/07/15 16:49:20 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor 0: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:20 INFO TaskSetManager: Serialized task 0.0:0 as 1429 bytes in 1 ms 14/07/15 16:49:20 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor 0: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:20 INFO TaskSetManager: Serialized task 0.0:1 as 1429 bytes in 0 ms 14/07/15 16:49:20 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:54553 with 294.9 MB RAM 14/07/15 16:49:26 INFO SparkDeploySchedulerBackend: Executor 0 disconnected, so removing it 14/07/15 16:49:26 ERROR TaskSchedulerImpl: Lost executor 0 on office-5-148.pa.gopivotal.com: remote Akka client disassociated 14/07/15 16:49:26 INFO TaskSetManager: Re-queueing tasks for 0 from TaskSet 0.0 14/07/15 16:49:26 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/0 is now EXITED (Command exited with code 55) 14/07/15 16:49:26 WARN TaskSetManager: Lost TID 1 (task 0.0:1) 14/07/15 16:49:26 INFO SparkDeploySchedulerBackend: Executor app-20140715164918-0001/0 removed: Command exited with code 55 14/07/15 16:49:26 WARN TaskSetManager: Lost TID 0 (task 0.0:0) 14/07/15 16:49:26 INFO AppClient$ClientActor: Executor added: app-20140715164918-0001/1 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:49:26 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164918-0001/1 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:49:26 INFO DAGScheduler: Executor lost: 0 (epoch 0) 14/07/15 16:49:26 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/1 is now RUNNING 14/07/15 16:49:26 INFO BlockManagerMasterActor: Trying to remove executor 0 from BlockManagerMaster. 14/07/15 16:49:26 INFO BlockManagerMaster: Removed 0 successfully in removeExecutor 14/07/15 16:49:28 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@office-5-148.pa.gopivotal.com:54573/user/Executor#1564333236] with ID 1 14/07/15 16:49:28 INFO TaskSetManager: Re-computing pending task lists. 14/07/15 16:49:28 INFO TaskSetManager: Starting task 0.0:0 as TID 2 on executor 1: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:28 INFO TaskSetManager: Serialized task 0.0:0 as 1429 bytes in 0 ms 14/07/15 16:49:28 INFO TaskSetManager: Starting task 0.0:1 as TID 3 on executor 1: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:28 INFO TaskSetManager: Serialized task 0.0:1 as 1429 bytes in 0 ms 14/07/15 16:49:28 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:54578 with 294.9 MB RAM 14/07/15 16:49:34 INFO SparkDeploySchedulerBackend: Executor 1 disconnected, so removing it 14/07/15 16:49:34 ERROR TaskSchedulerImpl: Lost executor 1 on office-5-148.pa.gopivotal.com: remote Akka client disassociated 14/07/15 16:49:34 INFO TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0 14/07/15 16:49:34 WARN TaskSetManager: Lost TID 2 (task 0.0:0) 14/07/15 16:49:34 WARN TaskSetManager: Lost TID 3 (task 0.0:1) 14/07/15 16:49:34 INFO DAGScheduler: Executor lost: 1 (epoch 1) 14/07/15 16:49:34 INFO BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster. 14/07/15 16:49:34 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor 14/07/15 16:49:34 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/1 is now EXITED (Command exited with code 55) 14/07/15 16:49:34 INFO SparkDeploySchedulerBackend: Executor app-20140715164918-0001/1 removed: Command exited with code 55 14/07/15 16:49:34 INFO AppClient$ClientActor: Executor added: app-20140715164918-0001/2 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:49:34 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164918-0001/2 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:49:34 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/2 is now RUNNING 14/07/15 16:49:37 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@office-5-148.pa.gopivotal.com:54599/user/Executor#-557403228] with ID 2 14/07/15 16:49:37 INFO TaskSetManager: Re-computing pending task lists. 14/07/15 16:49:37 INFO TaskSetManager: Starting task 0.0:1 as TID 4 on executor 2: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:37 INFO TaskSetManager: Serialized task 0.0:1 as 1429 bytes in 1 ms 14/07/15 16:49:37 INFO TaskSetManager: Starting task 0.0:0 as TID 5 on executor 2: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:37 INFO TaskSetManager: Serialized task 0.0:0 as 1429 bytes in 0 ms 14/07/15 16:49:37 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:54604 with 294.9 MB RAM 14/07/15 16:49:43 INFO SparkDeploySchedulerBackend: Executor 2 disconnected, so removing it 14/07/15 16:49:43 ERROR TaskSchedulerImpl: Lost executor 2 on office-5-148.pa.gopivotal.com: remote Akka client disassociated 14/07/15 16:49:43 INFO TaskSetManager: Re-queueing tasks for 2 from TaskSet 0.0 14/07/15 16:49:43 WARN TaskSetManager: Lost TID 5 (task 0.0:0) 14/07/15 16:49:43 WARN TaskSetManager: Lost TID 4 (task 0.0:1) 14/07/15 16:49:43 INFO DAGScheduler: Executor lost: 2 (epoch 2) 14/07/15 16:49:43 INFO BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster. 14/07/15 16:49:43 INFO BlockManagerMaster: Removed 2 successfully in removeExecutor 14/07/15 16:49:43 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/2 is now EXITED (Command exited with code 55) 14/07/15 16:49:43 INFO SparkDeploySchedulerBackend: Executor app-20140715164918-0001/2 removed: Command exited with code 55 14/07/15 16:49:43 INFO AppClient$ClientActor: Executor added: app-20140715164918-0001/3 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:49:43 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164918-0001/3 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:49:43 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/3 is now RUNNING 14/07/15 16:49:45 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@office-5-148.pa.gopivotal.com:54627/user/Executor#-1697612197] with ID 3 14/07/15 16:49:45 INFO TaskSetManager: Re-computing pending task lists. 14/07/15 16:49:45 INFO TaskSetManager: Starting task 0.0:1 as TID 6 on executor 3: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:45 INFO TaskSetManager: Serialized task 0.0:1 as 1429 bytes in 0 ms 14/07/15 16:49:45 INFO TaskSetManager: Starting task 0.0:0 as TID 7 on executor 3: office-5-148.pa.gopivotal.com (PROCESS_LOCAL) 14/07/15 16:49:45 INFO TaskSetManager: Serialized task 0.0:0 as 1429 bytes in 0 ms 14/07/15 16:49:45 INFO BlockManagerMasterActor: Registering block manager office-5-148.pa.gopivotal.com:54634 with 294.9 MB RAM 14/07/15 16:49:51 INFO SparkDeploySchedulerBackend: Executor 3 disconnected, so removing it 14/07/15 16:49:51 ERROR TaskSchedulerImpl: Lost executor 3 on office-5-148.pa.gopivotal.com: remote Akka client disassociated 14/07/15 16:49:51 INFO TaskSetManager: Re-queueing tasks for 3 from TaskSet 0.0 14/07/15 16:49:51 WARN TaskSetManager: Lost TID 7 (task 0.0:0) 14/07/15 16:49:51 ERROR TaskSetManager: Task 0.0:0 failed 4 times; aborting job 14/07/15 16:49:51 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/07/15 16:49:51 INFO TaskSchedulerImpl: Cancelling stage 0 14/07/15 16:49:51 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/3 is now EXITED (Command exited with code 55) 14/07/15 16:49:51 INFO DAGScheduler: Failed to run reduce at SparkTachyonPi.scala:43 Exception in thread "main" 14/07/15 16:49:51 INFO SparkDeploySchedulerBackend: Executor app-20140715164918-0001/3 removed: Command exited with code 55 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:0 failed 4 times, most recent failure: TID 7 on host office-5-148.pa.gopivotal.com failed for unknown reason Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1046) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1030) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1028) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:632) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:632) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:632) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1231) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 14/07/15 16:49:51 INFO AppClient$ClientActor: Executor added: app-20140715164918-0001/4 on worker-20140715164009-office-5-148.pa.gopivotal.com-52519 (office-5-148.pa.gopivotal.com:52519) with 8 cores 14/07/15 16:49:51 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140715164918-0001/4 on hostPort office-5-148.pa.gopivotal.com:52519 with 8 cores, 512.0 MB RAM 14/07/15 16:49:51 INFO AppClient$ClientActor: Executor updated: app-20140715164918-0001/4 is now RUNNING 14/07/15 16:49:51 INFO DAGScheduler: Executor lost: 3 (epoch 3) 14/07/15 16:49:51 INFO BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster. 14/07/15 16:49:51 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor Process finished with exit code 1 -- This message was sent by Atlassian JIRA (v6.2#6252)