Hi all,
I am doing some experiments on spark standalone cluster setup
and I am facing the following issue:
I have a 4 node cluster setup. As per
http://spark.apache.org/docs/latest/spark-standalone.html#starting-a-cluster-manually
I tried to start the cluster with the scripts but, the slaves did not start
and gave permission denied error....My conf/slaves had a list of IP addresses
of the slaves... But Then I was able to start the worker nodes by going to the
respective slave machine and running the start-slave.sh script with the master
IP as parameter.... The webui showed the all the 3 worker nodes running as :
[cid:image001.png@01D10A93.B59D7300]
The application when submitted to the cluster will show in all the work nodes
in the webui.... But in the command line where is spark-submit command was run,
it gives this error periodically and continuously....
ERROR NettyTransport: failed to bind to /172.28.161.138:7077, shutting down
Netty transport
15/10/19 11:59:38 WARN Utils: Service 'sparkDriver' could not bind on port
7077. Attempting port 7078.
15/10/19 11:59:38 WARN AppClient$ClientActor: Could not connect to
akka.tcp://sparkMaster@localhost:7077: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077....
My conf/spark-env.sh has:
SPARK_MASTER_IP=172.28.161.138
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_WEBUI_PORT=8081
SPARK_WORKER_INSTANCES=1
And I have also put this in all the slave nodes too....
The applications are running fine in -master local mode but in -master
spark://masterip:7077, it is not working....
Any type of help would be appreciated..... Thanks in advance
