I raised this question on the Spark ML but it may be more a Mesos question.
I would like to be able to configure the port used to communicate between the Mesos master and Spark tasks running across mesos slaves. It appears that spark's usage of the mesos library must use the default port = 0. At least that what I think happens in UPID. https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/pid.cpp Here is an example... In this case if the port 56311 is not opened up via iptables and security groups, the detecting new master step will hang indefinitely, and failures will be logged on the master. group.cpp:310] Group process ((2)@1.2.3.4:56311) connected to ZooKeeper group.cpp:752] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) group.cpp:367] Trying to create path '/mesos' in ZooKeeper detector.cpp:134] Detected a new leader: (id='2') group.cpp:629] Trying to get '/mesos/info_0000000002' in ZooKeeper detector.cpp:351] A new leading master ([email protected]:5050) is detected sched.cpp:230] Detecting new master Is there a way to tell the mesos native lib to use a specific port rather than a random port from the spark side of things? thanks SC
