Hi, I have a really annoying issue that i cannot replicate consistently, still it happens every +- 100 submissions. (it's a job that's running every 3 minutes). Already reported an issue for this: https://issues.apache.org/jira/browse/SPARK-8592
Here are the Thread dump of the Driver and the Executor: https://docs.google.com/document/d/1x7ZwUzlvRqeJQ12FoGhpLV1zqDAmVsaF2HYhzkPNBKQ Any direction is should look into? Spark 1.4.0 Java 1.8.0_45 (Oracle Corporation) Scala 2.11.6 I already tried to resolve the NPE by not logging the ActorRef. This makes the NPE go away :) But the root cause lies deeper I expect, since then the driver then still hangs with the "*WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources*" messages. But there are enough resources available in the cluster, it has plenty of CPU and Memory left. Logs from Driver: 15/06/26 11:58:19 INFO Remoting: Starting remoting 15/06/26 11:58:19 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:51415] 15/06/26 11:58:19 INFO Utils: Successfully started service 'sparkDriver' on port 51415. 15/06/26 11:58:20 INFO SparkEnv: Registering MapOutputTracker 15/06/26 11:58:20 INFO SparkEnv: Registering BlockManagerMaster 15/06/26 11:58:20 INFO DiskBlockManager: Created local directory at /tmp/spark-ff1f5a88-4e1d-4fe0-9c54-890e4174f02a/blockmgr-92b1e974-53bb-45a3-b918-916759e14630 15/06/26 11:58:20 INFO MemoryStore: MemoryStore started with capacity 265.1 MB 15/06/26 11:58:20 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ff1f5a88-4e1d-4fe0-9c54-890e4174f02a/httpd-f5894293-33aa-4eaa-9740-4a36c054b6c8 15/06/26 11:58:20 INFO HttpServer: Starting HTTP Server 15/06/26 11:58:20 INFO Utils: Successfully started service 'HTTP file server' on port 33176. 15/06/26 11:58:20 INFO SparkEnv: Registering OutputCommitCoordinator 15/06/26 11:58:20 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/06/26 11:58:20 INFO SparkUI: Started SparkUI at http://172.17.0.123:4040 15/06/26 11:58:20 INFO SparkContext: Added JAR file:/opt/jar/spark/spark-job-1.0-SNAPSHOT.jar at http://172.17.0.123:33176/jars/spark-job-1.0-SNAPSHOT.jar with timestamp 1435319900257 15/06/26 11:58:20 INFO AppClient$ClientActor: Connecting to master akka.tcp://[email protected]:7077/user/Master... 15/06/26 11:58:20 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150626115820-0917 15/06/26 11:58:20 INFO AppClient$ClientActor: Executor added: app-20150626115820-0917/0 on worker-20150625133752-10.0.7.171-47050 ( 10.0.7.171:47050) with 1 cores 15/06/26 11:58:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150626115820-0917/0 on hostPort 10.0.7.171:47050 with 1 cores, 2.0 GB RAM 15/06/26 11:58:20 INFO TaskSchedulerImpl: Starting speculative execution thread 15/06/26 11:58:20 INFO AppClient$ClientActor: Executor updated: app-20150626115820-0917/0 is now LOADING 15/06/26 11:58:20 INFO AppClient$ClientActor: Executor updated: app-20150626115820-0917/0 is now RUNNING 15/06/26 11:58:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52000. 15/06/26 11:58:20 INFO NettyBlockTransferService: Server created on 52000 15/06/26 11:58:20 INFO BlockManagerMaster: Trying to register BlockManager 15/06/26 11:58:20 INFO BlockManagerMasterEndpoint: Registering block manager 172.17.0.123:52000 with 265.1 MB RAM, BlockManagerId(driver, 172.17.0.123, 52000) 15/06/26 11:58:20 INFO BlockManagerMaster: Registered BlockManager 15/06/26 11:58:20 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/06/26 11:58:24 INFO Exchange: Using SparkSqlSerializer2. 15/06/26 11:58:24 INFO Exchange: Using SparkSqlSerializer2. 15/06/26 11:58:24 INFO SparkContext: Starting job: map at SparkProductEventAggregator.scala:144 15/06/26 11:58:24 INFO Version: Elasticsearch Hadoop v2.1.0.rc1 [5cc3f53084] 15/06/26 11:58:24 INFO ScalaEsRowRDD: Reading from [675d42c8-9823-4d3c-8e86-5aa611d38770/events] 15/06/26 11:58:24 INFO ScalaEsRowRDD: Discovered mapping {675d42c8-9823-4d3c-8e86-5aa611d38770=[REMOVED]} for [675d42c8-9823-4d3c-8e86-5aa611d38770/events] 15/06/26 11:58:24 INFO DAGScheduler: Registering RDD 5 (map at SparkProductEventAggregator.scala:144) 15/06/26 11:58:24 INFO DAGScheduler: Got job 0 (map at SparkProductEventAggregator.scala:144) with 200 output partitions (allowLocal=false) 15/06/26 11:58:24 INFO DAGScheduler: Final stage: ResultStage 1(map at SparkProductEventAggregator.scala:144) 15/06/26 11:58:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 15/06/26 11:58:24 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) 15/06/26 11:58:24 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at map at SparkProductEventAggregator.scala:144), which has no missing parents 15/06/26 11:58:24 INFO MemoryStore: ensureFreeSpace(12384) called with curMem=0, maxMem=278019440 15/06/26 11:58:24 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.1 KB, free 265.1 MB) 15/06/26 11:58:24 INFO MemoryStore: ensureFreeSpace(5542) called with curMem=12384, maxMem=278019440 15/06/26 11:58:24 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.4 KB, free 265.1 MB) 15/06/26 11:58:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.17.0.123:52000 (size: 5.4 KB, free: 265.1 MB) 15/06/26 11:58:24 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:874 15/06/26 11:58:24 INFO DAGScheduler: Submitting 5 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at map at SparkProductEventAggregator.scala:144) 15/06/26 11:58:24 INFO TaskSchedulerImpl: Adding task set 0.0 with 5 tasks 15/06/26 11:58:39 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 15/06/26 11:58:54 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 15/06/26 11:59:09 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources Sjoerd
