I’ve got working cluster for more, than couple of weeks with 20 workers. 
Everything was perfect.
Today I added 4 more workers and all of them couldn’t fetch jar files from 

The following means to me that master is available to worker, it is registered 
there and it started everything. 
But when Executor got assigned task, it couldn’t download jar file from master.

16/12/01 06:25:36 INFO CoarseGrainedExecutorBackend: Connecting to driver: 
16/12/01 06:25:36 INFO WorkerWatcher: Connecting to worker 
spark://wor...@xx.xx.xx.xx:46799 <spark://wor...@xx.xx.xx.xx:46799>
16/12/01 06:25:36 INFO CoarseGrainedExecutorBackend: Successfully registered 
with driver
16/12/01 06:25:36 INFO Executor: Starting executor ID 304 on host 
machine059.company.com <http://machine059.company.com/>
16/12/01 06:25:36 INFO Utils: Successfully started service 
'org.apache.spark.network.netty.NettyBlockTransferService' on port 48
16/12/01 06:25:36 INFO NettyBlockTransferService: Server created on 48844
16/12/01 06:25:36 INFO BlockManagerMaster: Trying to register BlockManager
16/12/01 06:25:36 INFO BlockManagerMaster: Registered BlockManager
16/12/01 06:25:44 INFO CoarseGrainedExecutorBackend: Got assigned task 9239511
16/12/01 06:25:44 INFO Executor: Running task 46.0 in stage 159585.0 (TID 
16/12/01 06:25:44 INFO Executor: Fetching 
with timesta
mp 1479534760567
16/12/01 06:25:44 ERROR Executor: Exception in task 46.0 in stage 159585.0 (TID 

When I issue the same download manually I’m getting simple file not found.
$ curl 
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /jars/sparkws-spark-1.0.0-20161118.151127-286.jar. Reason:
<pre>    Not Found</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>

Spark Environment tells jars are there

And they are indeed there
[user@machine004]$ ll /opt/sparkws/lib/ | grep sparkws
  156455 Nov 18 15:12 sparkws-core-1.0.0-20161118.151127-286.jar
18944178 Nov 18 15:14 sparkws-spark-1.0.0-20161118.151127-286.jar

Anyone might know what might be an issue here?

Reply via email to