Hi, I'm using spark on a 8 (+headnode) node cluster in standalone mode. The headnode runs the master instance and the compute nodes are supposed to run the workers. No worker runs on the headnode.
However, as soon as I connect more than one worker I get a weird behaviour: some applications run, some don't. My test cases have been a small parallelized list which is mapped and then reduced, the other one is the creation of an rMAT graph with graphx and printing out the number of vertices and edges. The parallelized list kernel runs perfectly fine while the graph kernel freezes after creation of the rMAT graph and before printing the number of vertices/edges. The problem could be related to this one: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-running-a-job-on-more-than-one-workers-tp12361.html The difference is that my applications don't get removed by the master but just freeze and I get a lot of EndpointAssociationExceptions in my worker logs. I doublechecked interconnectivity and all seems can reach eachother. Does any of you have an idea what might be causing the problem? Here is the data I collected: example code: ================================================================== import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf import org.apache.spark._ import org.apache.spark.graphx._ import org.apache.spark.graphx.util._ import org.apache.spark.rdd.RDD import java.io._ class NullOutStream extends OutputStream { def write(b: Byte): Unit = {} def write(i: Int): Unit = {} } object GraphMinimal { def main(args: Array[String]): Unit = { try { val conf = new SparkConf().setAppName("spark-graphx-minimal") val sc = new SparkContext(conf) val s = if (args.length >= 2) args(0).toInt else 10 val ef = if (args.length >= 2) args(1).toInt else 16 val n = math.pow(2, s).toInt val m = n * ef print(s"Generating graph with ${n} vertices and ${m} edges...") val g1 = scala.Console.withOut(new NullOutStream()) { GraphGenerators.rmatGraph(sc, n, m) } println("done") println(s"Vertices: ${g1.numVertices}, edges: ${g1.numEdges}") } catch { case e: Exception => e.printStackTrace() } } } object ListMinimal { def main(args: Array[String]): Unit = { try { val conf = new SparkConf().setAppName("spark-list-minimal") val sc = new SparkContext(conf) val max = if (args.length >= 1) args(0).toInt else 10000 val numElems = sc.parallelize(0 to max) .map(x => 1).reduce(_ + _) println(numElems) } catch { case e: Exception => e.printStackTrace() } } } Test Application output with one worker: ================================================================== list: ---------------------------------------------------------------------------------- Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 07:51:00 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:00 INFO AbstractConnector: Started SocketConnector@0.0.0.0:47461 14/08/21 07:51:00 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:00 INFO AbstractConnector: Started SocketConnector@0.0.0.0:58979 14/08/21 07:51:00 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:00 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/21 07:51:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 10001 (10001 shows that the application finished) graph: ---------------------------------------------------------------------------------- Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 07:51:08 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:08 INFO AbstractConnector: Started SocketConnector@0.0.0.0:49574 14/08/21 07:51:08 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:08 INFO AbstractConnector: Started SocketConnector@0.0.0.0:55816 14/08/21 07:51:09 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:51:09 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/21 07:51:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Generating graph with 1024 vertices and 16384 edges...done Vertices: 512, edges: 16384 (Vertices: 512, edges: 16384 indicates finished application) Test Application output with two workers: ================================================================== list: ---------------------------------------------------------------------------------- Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 07:52:18 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:18 INFO AbstractConnector: Started SocketConnector@0.0.0.0:55800 14/08/21 07:52:18 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:18 INFO AbstractConnector: Started SocketConnector@0.0.0.0:35448 14/08/21 07:52:19 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:19 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/21 07:52:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 10001 (10001 shows that the application finished with two workers, too) graph: ---------------------------------------------------------------------------------- Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 07:52:27 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:27 INFO AbstractConnector: Started SocketConnector@0.0.0.0:41677 14/08/21 07:52:27 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:27 INFO AbstractConnector: Started SocketConnector@0.0.0.0:39059 14/08/21 07:52:27 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:52:27 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/21 07:52:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Generating graph with 1024 vertices and 16384 edges...done (No number of vertices/edges shown, application was killed by manual timeout after 60 seconds) Master Log: ================================================================== Spark assembly has been built with Hive, including Datanucleus jars on classpath Spark Command: java -cp ::/opt/spark-1.0.2/conf:/opt/spark-1.0.2/lib/spark-assembly-1.0.2-hadoop2.2.0.jar:/opt/spark-1.0.2/lib/datanucleus-core-3.2.2.jar:/opt/spark-1.0.2/lib/datanucleus-rdbms-3.2.1.jar:/opt/spark-1.0.2/lib/datanucleus-api-jdo-3.2.1.jar -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip 192.168.23.1 --port 7077 --webui-port 8080 ======================================== 14/08/21 07:49:57 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 07:49:57 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080 14/08/21 07:51:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:58887]: Error [Association failed with [akka.tcp://spark@192.168.23.1:58887]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:58887] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:58887 ] 14/08/21 07:51:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:58887]: Error [Association failed with [akka.tcp://spark@192.168.23.1:58887]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:58887] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:58887 ] 14/08/21 07:51:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:58887]: Error [Association failed with [akka.tcp://spark@192.168.23.1:58887]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:58887] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:58887 ] 14/08/21 07:51:06 WARN Master: Got status update for unknown executor app-20140821075101-0000/0 14/08/21 07:51:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:45365]: Error [Association failed with [akka.tcp://spark@192.168.23.1:45365]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:45365] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:45365 ] 14/08/21 07:51:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:45365]: Error [Association failed with [akka.tcp://spark@192.168.23.1:45365]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:45365] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:45365 ] 14/08/21 07:51:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:45365]: Error [Association failed with [akka.tcp://spark@192.168.23.1:45365]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:45365] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:45365 ] 14/08/21 07:51:15 WARN Master: Got status update for unknown executor app-20140821075110-0001/0 14/08/21 07:52:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:34793]: Error [Association failed with [akka.tcp://spark@192.168.23.1:34793]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:34793] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:34793 ] 14/08/21 07:52:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:34793]: Error [Association failed with [akka.tcp://spark@192.168.23.1:34793]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:34793] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:34793 ] 14/08/21 07:52:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:34793]: Error [Association failed with [akka.tcp://spark@192.168.23.1:34793]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:34793] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:34793 ] 14/08/21 07:52:24 WARN Master: Got status update for unknown executor app-20140821075220-0002/1 14/08/21 07:52:24 WARN Master: Got status update for unknown executor app-20140821075220-0002/0 14/08/21 07:53:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:43727]: Error [Association failed with [akka.tcp://spark@192.168.23.1:43727]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:43727] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:43727 ] 14/08/21 07:53:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:43727]: Error [Association failed with [akka.tcp://spark@192.168.23.1:43727]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:43727] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:43727 ] 14/08/21 07:53:24 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.23.1:7077] -> [akka.tcp://spark@192.168.23.1:43727]: Error [Association failed with [akka.tcp://spark@192.168.23.1:43727]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.23.1:43727] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.23.1:43727 ] 14/08/21 07:53:25 WARN Master: Got status update for unknown executor app-20140821075228-0003/1 14/08/21 07:53:25 WARN Master: Got status update for unknown executor app-20140821075228-0003/0 Worker 1 Log: ================================================================== Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:01:53 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 12:01:53 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081 Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:02:30 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:48301 ] 14/08/21 12:02:30 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:48301 ] 14/08/21 12:02:30 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:48301] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:48301 ] Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:02:39 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:59128 ] 14/08/21 12:02:39 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:59128 ] 14/08/21 12:02:39 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:59128] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:59128 ] Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:03:48 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:50671 ] 14/08/21 12:03:48 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:50671 ] 14/08/21 12:03:48 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:50671] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:50671 ] Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:04:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:51517 ] 14/08/21 12:04:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:51517 ] 14/08/21 12:04:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane001.ziti.uni-heidelberg.de:43823] -> [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]: Error [Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane001.ziti.uni-heidelberg.de:51517] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane001.ziti.uni-heidelberg.de/127.0.1.1:51517 ] Worker 2 Log: ================================================================== Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:03:38 INFO Server: jetty-8.y.z-SNAPSHOT 14/08/21 12:03:38 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081 Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:04:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:44308 ] 14/08/21 12:04:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:44308 ] 14/08/21 12:04:15 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:44308] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:44308 ] Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/08/21 12:05:16 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:52528 ] 14/08/21 12:05:16 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:52528 ] 14/08/21 12:05:16 ERROR EndpointWriter: AssociationError [akka.tcp://sparkwor...@octane002.ziti.uni-heidelberg.de:39978] -> [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]: Error [Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkexecu...@octane002.ziti.uni-heidelberg.de:52528] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: octane002.ziti.uni-heidelberg.de/127.0.1.1:52528 ] Thanks in advance for your help and best regards, Alexander -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/More-than-one-worker-freezes-some-applications-tp12570.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
