Ok got it figured out. LIBPROCESS_PORT is being unset along the way somewhere by marathon/mesos, the solution was to start the spark driver process via a script that re exports LIBPROCESS_PORT by reading it from a var named something other than LIBPROCESS_PORT (in my case SPARK_LIBPROCESS_PORT)
On Mon, May 19, 2014 at 12:08 PM, Scott Clasen <[email protected]> wrote: > Oh for sure, setting quite a few that I am seeing in the env. > > > On Mon, May 19, 2014 at 12:05 PM, Vinod Kone <[email protected]> wrote: > >> Can you set some other variable via the same mechanism and see if you can >> see it in your app? >> >> >> On Mon, May 19, 2014 at 10:54 AM, Scott Clasen <[email protected]> wrote: >> >>> hmm, I am setting it via the Environment passed in the REST call to >>> create the app, and if I look at the marathon UI I do see the var set. >>> >>> >>> On Mon, May 19, 2014 at 10:19 AM, Vinod Kone <[email protected]>wrote: >>> >>>> Probably. How are you setting the LIBPROCESS_PORT in Marathon? It has >>>> to be set via CommandInfo.Environment() of the task/executor for this to >>>> take effect. >>>> >>>> >>>> On Fri, May 16, 2014 at 9:41 AM, Scott Clasen <[email protected]> wrote: >>>> >>>>> Aha, thanks! I am still having an issue. I am executing the process >>>>> via marathon, with LIBPROCESS_PORT set in the env. However when I log the >>>>> value of LIBPROCESS_PORT in my program, I get 0. >>>>> >>>>> Found a thread in this ML called >>>>> Re: Review Request: "Unset" LIBPROCESS_PORT before applying >>>>> executor/command environment variables (in the event they modify it). Is >>>>> this what is happening to me? >>>>> >>>>> >>>>> >>>>> On Thu, May 15, 2014 at 7:16 PM, Benjamin Mahler < >>>>> [email protected]> wrote: >>>>> >>>>>> You can set LIBPROCESS_PORT in the environment. >>>>>> >>>>>> >>>>>> On Wed, May 14, 2014 at 1:58 PM, Scott Clasen <[email protected]>wrote: >>>>>> >>>>>>> I raised this question on the Spark ML but it may be more a Mesos >>>>>>> question. >>>>>>> >>>>>>> I would like to be able to configure the port used to communicate >>>>>>> between the Mesos master and Spark tasks running across mesos slaves. It >>>>>>> appears that spark's usage of the mesos library must use the default >>>>>>> port = >>>>>>> 0. >>>>>>> >>>>>>> At least that what I think happens in UPID. >>>>>>> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/pid.cpp >>>>>>> >>>>>>> Here is an example... >>>>>>> >>>>>>> In this case if the port 56311 is not opened up via iptables and >>>>>>> security groups, the detecting new master step will hang indefinitely, >>>>>>> and >>>>>>> failures will be logged on the master. >>>>>>> >>>>>>> group.cpp:310] Group process ((2)@1.2.3.4:56311) connected to >>>>>>> ZooKeeper >>>>>>> group.cpp:752] Syncing group operations: queue size (joins, cancels, >>>>>>> datas) = (0, 0, 0) >>>>>>> group.cpp:367] Trying to create path '/mesos' in ZooKeeper >>>>>>> detector.cpp:134] Detected a new leader: (id='2') >>>>>>> group.cpp:629] Trying to get '/mesos/info_0000000002' in ZooKeeper >>>>>>> detector.cpp:351] A new leading master (UPID= >>>>>>> [email protected]:5050) is detected >>>>>>> sched.cpp:230] Detecting new master >>>>>>> >>>>>>> >>>>>>> Is there a way to tell the mesos native lib to use a specific port >>>>>>> rather than a random port from the spark side of things? >>>>>>> >>>>>>> >>>>>>> thanks >>>>>>> SC >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
