When you say remote cluster you need to make sure a few things like:
- No firewall/network is blocking any connection (Simply ping from
localmachine to remote ip and vice versa)
- Make sure all ports (unless you specify them manually) are open.
You can also refer this discussion,
I’ve setup a Spark cluster in the last few weeks and everything is working, but
I cannot run spark-shell interactively against the cluster from a remote host
* Deploy .jar to cluster from remote (laptop) spark-submit and have it run
– Check
* Run .jar on spark-shell locally – Check
*
