I have a Spark master deployed (0.8.0 for CDH4). We successfully run shark on top of that setup.
For a few days now I've been trying to run the SparkPI example from my desktop against the master, but my workers die with the following trace: 14/01/10 17:29:40 INFO executor.StandaloneExecutorBackend: Connecting to driver: akka://[email protected]:49081/user/StandaloneScheduler 14/01/10 17:30:41 ERROR executor.StandaloneExecutorBackend: Driver terminated or disconnected! Shutting down. The IP shown in the stacktrace is the IP of my desktop machine (the one I'm running the SparkPI program from). When I look at the stacks for the Shark client, it looks like the connection between spark master and shark-machine is successful...Why is it failing to my desktop client -- am I supposed to explicitly open ports and pass spark.driver.port explicitly? I was told no manual port-opening was done to get Shark going. Thanks for any insight.
