What version of Spark are you using? TD
On Thu, Feb 19, 2015 at 2:45 AM, Emre Sevinc <emre.sev...@gmail.com> wrote: > Hello, > > We have a Spark Streaming application that watches an input directory, and > as files are copied there the application reads them and sends the contents > to a RESTful web service, receives a response and write some contents to an > output directory. > > When testing the application by copying a few thousand files at once to > its input directory, we have realized that after having processed about > 3800 files, it creates messages as the following in the log file: > > 15/02/19 10:22:06 INFO storage.MemoryStore: Block broadcast_17935 of size > 9960 dropped from memory (free 447798720) > 15/02/19 10:22:55 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > > and then the Spark Streaming application dies. > > What might be the potential causes to check for such errors? > > Below you can see last few lines before it dies: > > > 15/02/19 10:22:03 INFO broadcast.TorrentBroadcast: Started reading > broadcast variable 12894 > 15/02/19 10:22:04 INFO storage.MemoryStore: ensureFreeSpace(20978) called > with curMem=107884847, maxMem=556038881 > 15/02/19 10:22:04 INFO storage.MemoryStore: Block broadcast_12894_piece0 > stored as bytes in memory (estimated size 20.5 KB, free 427.4 MB) > 15/02/19 10:22:04 INFO storage.BlockManagerMaster: Updated info of block > broadcast_12894_piece0 > 15/02/19 10:22:04 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 12894 took 460 ms > 15/02/19 10:22:04 INFO storage.MemoryStore: ensureFreeSpace(347363) called > with curMem=107905825, maxMem=556038881 > 15/02/19 10:22:04 INFO storage.MemoryStore: Block broadcast_12894 stored > as values in memory (estimated size 339.2 KB, free 427.0 MB) > 15/02/19 10:22:04 INFO storage.MemoryStore: ensureFreeSpace(1079) called > with curMem=108253188, maxMem=556038881 > 15/02/19 10:22:04 INFO storage.MemoryStore: Block rdd_30466_35 stored as > bytes in memory (estimated size 1079.0 B, free 427.0 MB) > 15/02/19 10:22:04 INFO storage.BlockManagerMaster: Updated info of block > rdd_30466_35 > 15/02/19 10:22:05 INFO storage.MemoryStore: ensureFreeSpace(5) called with > curMem=108254267, maxMem=556038881 > 15/02/19 10:22:05 INFO storage.MemoryStore: Block rdd_30467_35 stored as > bytes in memory (estimated size 5.0 B, free 427.0 MB) > 15/02/19 10:22:05 INFO storage.BlockManagerMaster: Updated info of block > rdd_30467_35 > 15/02/19 10:22:05 INFO executor.Executor: Finished task 35.0 in stage > 351.0 (TID 12229). 2353 bytes result sent to driver > 15/02/19 10:22:06 INFO storage.BlockManager: Removing broadcast 17935 > 15/02/19 10:22:06 INFO storage.BlockManager: Removing block > broadcast_17935_piece0 > 15/02/19 10:22:06 INFO storage.MemoryStore: Block broadcast_17935_piece0 > of size 4151 dropped from memory (free 447788760) > 15/02/19 10:22:06 INFO storage.BlockManagerMaster: Updated info of block > broadcast_17935_piece0 > 15/02/19 10:22:06 INFO storage.BlockManager: Removing block broadcast_17935 > 15/02/19 10:22:06 INFO storage.MemoryStore: Block broadcast_17935 of size > 9960 dropped from memory (free 447798720) > 15/02/19 10:22:55 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:23:28 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:24:01 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:24:04 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@f445b44,BlockManagerId(2, node09.demo.hadoop, > 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:24:44 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:25:17 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:25:50 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:25:53 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@242d050,BlockManagerId(2, node09.demo.hadoop, > 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:26:33 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:27:06 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:27:39 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:27:42 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@4bf59b10,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:28:22 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:28:55 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:29:28 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:29:31 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@70ee8fc3,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:30:11 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:30:44 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:31:17 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:31:20 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@2c189a00,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:32:00 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:32:33 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:33:06 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:33:09 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@c040e36,BlockManagerId(2, node09.demo.hadoop, > 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:33:49 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:34:22 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:34:55 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:34:58 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@172a9e06,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:35:38 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:36:11 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:36:44 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:36:47 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@6ad1a9fc,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:37:27 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:38:00 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:38:33 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:38:36 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@21d8fa77,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:39:16 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:39:49 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:40:22 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:40:25 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@210049c8,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:41:05 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:41:38 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:42:11 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:42:14 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@28f7c8cf,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:42:54 WARN util.AkkaUtils: Error sending message in 1 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:43:27 WARN util.AkkaUtils: Error sending message in 2 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:44:00 WARN util.AkkaUtils: Error sending message in 3 attempts > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > 15/02/19 10:44:03 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(2,[Lscala.Tuple2;@64678d48,BlockManagerId(2, > node09.demo.hadoop, 50044))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:398) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187) > ... 1 more > 15/02/19 10:44:13 ERROR executor.CoarseGrainedExecutorBackend: Driver > Disassociated [akka.tcp://sparkexecu...@node09.demo.hadoop:57447] -> > [akka.tcp://sparkdri...@node05.demo.hadoop:59622] disassociated! Shutting > down. > > > > -- > Emre Sevinç > http://www.bigindustries.be/ > >
