Howdy Scott, Please see the discussions about securing the Spark network [1] [2].
In a nut shell, Spark opens up a couple of well known ports. And,then the workers and the shell open up dynamic ports for each job. These dynamic ports make securing the Spark network difficult. Jacob [1] http://apache-spark-user-list.1001560.n3.nabble.com/Securing-Spark-s-Network-td4832.html [2] http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-driver-interacting-with-Workers-in-YARN-mode-firewall-blocking-communication-td5237.html Jacob D. Eisinger IBM Emerging Technologies jeis...@us.ibm.com - (512) 286-6075 From: Scott Clasen <scott.cla...@gmail.com> To: u...@spark.incubator.apache.org Date: 05/05/2014 11:39 AM Subject: Comprehensive Port Configuration reference? Is there somewhere documented how one would go about configuring every open port a spark application needs? This seems like one of the main things that make running spark hard in places like EC2 where you arent using the canned spark scripts. Starting an app looks like you'll see ports open for BlockManager OutoutTracker FileServer WebUI Local port to get callbacks from mesos master.. What else? How do I configure all of these? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Comprehensive-Port-Configuration-reference-tp5384.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
