Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers]

A.  That would be satisfactory yes.

The framework is already "smart" (read as "overly complicated") since we
handle reassembly of fragmented offers, amongst other intricate coding
details.  So it does seem possible to add the kind of logic you suggest!
Thanks for the idea James!

My team currently uses the mesos containerizer, so the log access between
containers issue is not a problem like it might be with the docker
containerizer.

- Erik

On Sat, Jul 1, 2017 at 4:20 PM James Peach <jor...@gmail.com> wrote:

>
> > On Jul 1, 2017, at 11:14 AM, Erik Weathers <eweath...@groupon.com>
> wrote:
> >
> > Thanks for the info Kevin.  Seems there's no JIRAs nor design docs
> floating about yet for "admin tasks" or "daemon sets".
> >
> > Just FYI, this is the ticket in Storm for the problem I've been
> mentioning:
> >
> > https://issues.apache.org/jira/browse/STORM-1342
> >
> > I'll update it with the info you've provided below, so for now we'll
> rely on manually deploying logviewers.
>
> ISTM that this should be possible with a smart framework. If the framework
> keeps track of which agents it gets offers for, it could ensure that it
> launches a Storm logviewer task on the agent before launching any other
> Storm tasks. I expect that it might be a little tricky to get the
> containerization right so that the Storm tasks can rendezvous with the
> logviewer, but in principle it could be made to work?
>
> >
> > Thanks!
> >
> > - Erik
> >
> > On Sat, Jul 1, 2017 at 10:09 AM Kevin Klues <klue...@gmail.com> wrote:
> > What you are describing is a feature we call 'admin tasks' or 'daemon
> sets'.
> >
> > Unfortunately, there is no direct support for these yet, but we do have
> plans in the (relatively) near future to start working on it.
> >
> > One of our use cases is exactly what you describe with the logging
> service. On DC/OS we currently run our logging service as a systemd unit
> outside of mesos since we can't guarantee it gets launched everywhere (the
> same is true for a bunch of other services as well, namely metrics).
> >
> > We don't have an exact timeline for when we will build this support yet,
> but we will certainly announce it once we start actively working on it.
> >
> >
> > Erik Weathers <eweath...@groupon.com> schrieb am Sa. 1. Juli 2017 um
> 09:45:
> > That works for our particular use case, and is effectively what *we* do,
> but renders storm a "strange bird" amongst mesos frameworks.  Is there no
> trickery that could be played with mesos roles and/or reservations?
> >
> > - Erik
> >
> > On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <d...@hellooperator.net>
> wrote:
> > If it _needs_ to be there always then I'd roll it out with whatever
> > automation you use to deploy the mesos workers ; depending on
> > the scale you're running at launching it as a task is likely to be less
> > reliable due to outages etc.
> >
> > ( I understand the 'maybe all hosts' constraint but if it's 'up to one
> per
> > host', it sounds like a CM issue to me. )
> >
> > On 30 June 2017 at 23:58, Erik Weathers <eweath...@groupon.com> wrote:
> > > hi Mesos folks!
> > >
> > > My team is largely responsible for maintaining the Storm-on-Mesos
> framework.
> > > It suffers from a problem related to log retrieval:  Storm has a
> process
> > > called the "logviewer" that is assumed to exist on every host, and the
> Storm
> > > UI provides links to contact this process to download logs (and other
> > > debugging artifacts).   Our team manually runs this process on each
> Mesos
> > > host, but it would be nice to launch it automatically onto any Mesos
> host
> > > where Storm work gets allocated. [0]
> > >
> > > I have read that Mesos has added support for Kubernetes-esque "pods"
> as of
> > > version 1.1.0, but that feature seems somewhat insufficient for
> implementing
> > > our desired behavior from my naive understanding.  Specifically, Storm
> only
> > > has support for connecting to 1 logviewer per host, so unless pods can
> have
> > > separate containers inside each pod [1], and also dynamically change
> the set
> > > of executors and tasks inside of the pod [2], then I don't see how
> we'd be
> > > able to use them.
> > >
> > > Is there any existing feature in Mesos that might help us accomplish
> our
> > > goal?  Or any upcoming features?
> > >
> > > Thanks!!
> > >
> >

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers]

Thanks for the info Kevin.  Seems there's no JIRAs nor design docs floating
about yet for "admin tasks" or "daemon sets".

Just FYI, this is the ticket in Storm for the problem I've been mentioning:

https://issues.apache.org/jira/browse/STORM-1342

I'll update it with the info you've provided below, so for now we'll rely
on manually deploying logviewers.

Thanks!

- Erik

On Sat, Jul 1, 2017 at 10:09 AM Kevin Klues <klue...@gmail.com> wrote:

> What you are describing is a feature we call 'admin tasks' or 'daemon
> sets'.
>
> Unfortunately, there is no direct support for these yet, but we do have
> plans in the (relatively) near future to start working on it.
>
> One of our use cases is exactly what you describe with the logging
> service. On DC/OS we currently run our logging service as a systemd unit
> outside of mesos since we can't guarantee it gets launched everywhere (the
> same is true for a bunch of other services as well, namely metrics).
>
> We don't have an exact timeline for when we will build this support yet,
> but we will certainly announce it once we start actively working on it.
>
>
> Erik Weathers <eweath...@groupon.com> schrieb am Sa. 1. Juli 2017 um
> 09:45:
>
>> That works for our particular use case, and is effectively what *we* do,
>> but renders storm a "strange bird" amongst mesos frameworks.  Is there no
>> trickery that could be played with mesos roles and/or reservations?
>>
>> - Erik
>>
>> On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <d...@hellooperator.net>
>> wrote:
>>
>>> If it _needs_ to be there always then I'd roll it out with whatever
>>> automation you use to deploy the mesos workers ; depending on
>>> the scale you're running at launching it as a task is likely to be less
>>> reliable due to outages etc.
>>>
>>> ( I understand the 'maybe all hosts' constraint but if it's 'up to one
>>> per
>>> host', it sounds like a CM issue to me. )
>>>
>>> On 30 June 2017 at 23:58, Erik Weathers <eweath...@groupon.com> wrote:
>>> > hi Mesos folks!
>>> >
>>> > My team is largely responsible for maintaining the Storm-on-Mesos
>>> framework.
>>> > It suffers from a problem related to log retrieval:  Storm has a
>>> process
>>> > called the "logviewer" that is assumed to exist on every host, and the
>>> Storm
>>> > UI provides links to contact this process to download logs (and other
>>> > debugging artifacts).   Our team manually runs this process on each
>>> Mesos
>>> > host, but it would be nice to launch it automatically onto any Mesos
>>> host
>>> > where Storm work gets allocated. [0]
>>> >
>>> > I have read that Mesos has added support for Kubernetes-esque "pods"
>>> as of
>>> > version 1.1.0, but that feature seems somewhat insufficient for
>>> implementing
>>> > our desired behavior from my naive understanding.  Specifically, Storm
>>> only
>>> > has support for connecting to 1 logviewer per host, so unless pods can
>>> have
>>> > separate containers inside each pod [1], and also dynamically change
>>> the set
>>> > of executors and tasks inside of the pod [2], then I don't see how
>>> we'd be
>>> > able to use them.
>>> >
>>> > Is there any existing feature in Mesos that might help us accomplish
>>> our
>>> > goal?  Or any upcoming features?
>>> >
>>> > Thanks!!
>>> >
>>> > - Erik
>>> >
>>> > [0] Thus the "all" in quotes in the subject of this email, because it
>>> > *might* be all hosts, but it definitely would be all hosts where Storm
>>> gets
>>> > work assigned.
>>> >
>>> > [1] The Storm-on-Mesos framework leverages separate containers for each
>>> > topology's Supervisor and Worker processes, to provide isolation
>>> between
>>> > topologies.
>>> >
>>> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
>>> Workers
>>> > (a Mesos Task) onto hosts is ever changing in a given instance of a
>>> > Storm-on-Mesos framework.  i.e., as topologies get launched and die,
>>> or have
>>> > their worker processes die, the processes are dynamically distributed
>>> to the
>>> > various Mesos Worker hosts.  So existing containers often have more
>>> tasks
>>> > assigned into them (thus growing their footprint) or removed from them
>>> (thus
>>> > shrinking the footprint).
>>>
>> --
> ~Kevin
>

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers]

That works for our particular use case, and is effectively what *we* do,
but renders storm a "strange bird" amongst mesos frameworks.  Is there no
trickery that could be played with mesos roles and/or reservations?

- Erik

On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <d...@hellooperator.net> wrote:

> If it _needs_ to be there always then I'd roll it out with whatever
> automation you use to deploy the mesos workers ; depending on
> the scale you're running at launching it as a task is likely to be less
> reliable due to outages etc.
>
> ( I understand the 'maybe all hosts' constraint but if it's 'up to one per
> host', it sounds like a CM issue to me. )
>
> On 30 June 2017 at 23:58, Erik Weathers <eweath...@groupon.com> wrote:
> > hi Mesos folks!
> >
> > My team is largely responsible for maintaining the Storm-on-Mesos
> framework.
> > It suffers from a problem related to log retrieval:  Storm has a process
> > called the "logviewer" that is assumed to exist on every host, and the
> Storm
> > UI provides links to contact this process to download logs (and other
> > debugging artifacts).   Our team manually runs this process on each Mesos
> > host, but it would be nice to launch it automatically onto any Mesos host
> > where Storm work gets allocated. [0]
> >
> > I have read that Mesos has added support for Kubernetes-esque "pods" as
> of
> > version 1.1.0, but that feature seems somewhat insufficient for
> implementing
> > our desired behavior from my naive understanding.  Specifically, Storm
> only
> > has support for connecting to 1 logviewer per host, so unless pods can
> have
> > separate containers inside each pod [1], and also dynamically change the
> set
> > of executors and tasks inside of the pod [2], then I don't see how we'd
> be
> > able to use them.
> >
> > Is there any existing feature in Mesos that might help us accomplish our
> > goal?  Or any upcoming features?
> >
> > Thanks!!
> >
> > - Erik
> >
> > [0] Thus the "all" in quotes in the subject of this email, because it
> > *might* be all hosts, but it definitely would be all hosts where Storm
> gets
> > work assigned.
> >
> > [1] The Storm-on-Mesos framework leverages separate containers for each
> > topology's Supervisor and Worker processes, to provide isolation between
> > topologies.
> >
> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
> Workers
> > (a Mesos Task) onto hosts is ever changing in a given instance of a
> > Storm-on-Mesos framework.  i.e., as topologies get launched and die, or
> have
> > their worker processes die, the processes are dynamically distributed to
> the
> > various Mesos Worker hosts.  So existing containers often have more tasks
> > assigned into them (thus growing their footprint) or removed from them
> (thus
> > shrinking the footprint).
>

ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers]

hi Mesos folks!

My team is largely responsible for maintaining the Storm-on-Mesos
framework.  It suffers from a problem related to log retrieval:  Storm has
a process called the "logviewer" that is assumed to exist on every host,
and the Storm UI provides links to contact this process to download logs
(and other debugging artifacts).   Our team manually runs this process on
each Mesos host, but it would be nice to launch it automatically onto any
Mesos host where Storm work gets allocated. [0]

I have read that Mesos has added support for Kubernetes-esque "pods" as of
version 1.1.0, but that feature *seems* somewhat insufficient for
implementing our desired behavior from my naive understanding.
Specifically, Storm only has support for connecting to 1 logviewer per
host, so unless pods can have separate containers inside each pod [1], and
also dynamically change the set of executors and tasks inside of the pod
[2], then I don't see how we'd be able to use them.

Is there any existing feature in Mesos that might help us accomplish our
goal?  Or any upcoming features?

Thanks!!

- Erik

[0] Thus the "all" in quotes in the subject of this email, because it
*might* be all hosts, but it definitely would be all hosts where Storm gets
work assigned.

[1] The Storm-on-Mesos framework leverages separate containers for each
topology's Supervisor and Worker processes, to provide isolation between
topologies.

[2] The assignment of Storm Supervisors (a Mesos Executor) + Storm Workers
(a Mesos Task) onto hosts is ever changing in a given instance of a
Storm-on-Mesos framework.  i.e., as topologies get launched and die, or
have their worker processes die, the processes are dynamically distributed
to the various Mesos Worker hosts.  So existing containers often have more
tasks assigned into them (thus growing their footprint) or removed from
them (thus shrinking the footprint).

Re: Failure reason documentation

[Erik Weathers]

Thanks haosdent!

On Sun, Dec 4, 2016 at 10:45 AM haosdent <haosd...@gmail.com> wrote:

> Ohoh, sorry for misunderstanding the question. As far as I know, there is
> no documentation for that. We should add some comments to the reason enums.
> Create a ticket here https://issues.apache.org/jira/browse/MESOS-6686 to
> track it.
>
> On Mon, Dec 5, 2016 at 2:27 AM, Erik Weathers <eweath...@groupon.com>
> wrote:
>
> I think he's looking for documentation about what precisely each reason
> *means*. A la how there are comments beside the TaskState list in
> mesos.proto.
>
> - Erik
>
> On Sun, Dec 4, 2016 at 10:07 AM haosdent <haosd...@gmail.com> wrote:
>
> Hi @Wil You could find them here
> https://github.com/apache/mesos/blob/1.1.0/include/mesos/mesos.proto#L1577-L1609
>
> On Sat, Dec 3, 2016 at 6:09 AM, Wil Yegelwel <wyegel...@gmail.com> wrote:
>
> No I'm referring to the values of the enum Reason.
>
> On Fri, Dec 2, 2016, 4:52 PM Tomek Janiszewski <jani...@gmail.com> wrote:
>
> Hi
>
> Are you referring to task state? If yes then take a look at comments in
> proto
> https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1552
>
> http://mesos.apache.org/api/latest/java/org/apache/mesos/Protos.TaskState.html
>
> Best
>
> Tomek
>
> pt., 2.12.2016, 21:31 użytkownik Wil Yegelwel <wyegel...@gmail.com>
> napisał:
>
> Hey mesos users!
>
> I can't seem to find any documentation about the various reasons mesos
> includes when a job fails. Is there a place that describes what the reasons
> mean?
>
> Thanks,
> Wil
>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: Failure reason documentation

[Erik Weathers]

I think he's looking for documentation about what precisely each reason
*means*. A la how there are comments beside the TaskState list in
mesos.proto.

- Erik

On Sun, Dec 4, 2016 at 10:07 AM haosdent  wrote:

Hi @Wil You could find them here
https://github.com/apache/mesos/blob/1.1.0/include/mesos/mesos.proto#L1577-L1609

On Sat, Dec 3, 2016 at 6:09 AM, Wil Yegelwel  wrote:

No I'm referring to the values of the enum Reason.

On Fri, Dec 2, 2016, 4:52 PM Tomek Janiszewski  wrote:

Hi

Are you referring to task state? If yes then take a look at comments in
proto
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1552
http://mesos.apache.org/api/latest/java/org/apache/mesos/Protos.TaskState.html

Best

Tomek

pt., 2.12.2016, 21:31 użytkownik Wil Yegelwel  napisał:

Hey mesos users!

I can't seem to find any documentation about the various reasons mesos
includes when a job fails. Is there a place that describes what the reasons
mean?

Thanks,
Wil




-- 
Best Regards,
Haosdent Huang

Re: Using mesos' cfs limits on a docker container?

[Erik Weathers]

What was the problem and how did you overcome it?  (i.e. This would be a
sad resolution to this thread for someone faced with this same problem in
the future.)

On Sunday, August 14, 2016, Mark Hammons 
wrote:

> I finally got this working after fiddling with it all night. It works
> great so far!
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 04:50, Joseph Wu  > wrote:
>
> If you're not against running Docker containers without the Docker daemon,
> try using the Unified containerizer.
> See the latter half of this document: http://mesos.apache.org/
> documentation/latest/mesos-containerizer/
> 
>
> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
> mark.hamm...@inaf.cnrs-gif.fr
> > wrote:
>
>> Hi All,
>>
>>
>>
>> I was having a lot of success having mesos force sandboxed programs to
>> work within cpu and memory constraints, but when I added docker into the
>> mix, the cpu limitations go out the window (not sure about the memory
>> limitations. Is there any way to mix these two methods of isolation? I'd
>> like my executor/algorithm to run inside a docker container, but have that
>> container's memory and cpu usage controlled by systemd/mesos.
>>
>>
>>
>> Thanks,
>>
>> Mark
>> --
>>
>> Mark Hammons - +33 06 03 69 56 56
>>
>> Research Engineer @ BioEmergences
>> 
>>
>> Lab Phone: 01 69 82 34 19
>>
>
>

Re: Programmatically retrieve stdout/stderr from a node

[Erik Weathers]

Just for completeness and to provide an alternative, you can also probably
leverage the dcos command line tool (https://github.com/dcos/dcos-cli) to
get all the info you would need in a JSON format.

e.g.,
1. set up ~/.dcos/config.toml for your cluster
2. DCOS_CONFIG=~/.dcos/config.toml dcos task --json --completed

You could process that output with jq (https://stedolan.github.io/jq/) and
do all of this in a short script.  (Not that I have much luck using jq, I'm
not very skilled at working with such arcane syntax, a la XPath.)

- Erik

On Wed, Aug 10, 2016 at 9:27 AM, June Taylor  wrote:

> David,
>
> Thanks for the suggestions, this has the missing piece we needed!
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Aug 10, 2016 at 10:54 AM, David Greenberg 
> wrote:
>
>> The Cook framework has examples of how to do this. See here (
>> https://github.com/twosigma/Cook/blob/master/scheduler/src/
>> cook/mesos/api.clj#L322-L324) for constructing the stem of the URL, here
>> (https://github.com/twosigma/Cook/blob/7a49fbb98b281e3b23779
>> cd88d1d2b73428a0447/scheduler/src/cook/mesos/api.clj#L281-L297) for
>> finding the filesystem path, and here (https://github.com/twosigma/C
>> ook/blob/master/scheduler/docs/scheduler-rest-api.asc#using_output_url)
>> for getting data with the stem.
>>
>> Essentially, you need to scrape the Mesos master to get all the path info
>> you need. LMK if you have questions!
>>
>> On Wed, Aug 10, 2016 at 9:34 AM June Taylor  wrote:
>>
>>> Tomek,
>>>
>>> I'm not sure I understand your suggestion. We know how to ask for a file
>>> from an HTTP endpoint, but it is the construction of the correct URL which
>>> is not currently clear.
>>>
>>> We are not sure how to determine the Run ID of the executor.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
>>> wrote:
>>>
 Hi

 If you need simplest method then python SimpleHTTPServer could help.
 Just launch it in background before command you want to run, assign it port
 and query sandbox with : that can be obtained from
 state endpoint.

 -
 Tomek

 śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:

> We are trying to retrieve the stdout and stderr files from an executor
> programmatically.
>
> It appears that these are available via HTTP request, however,
> constructing the correct URL is posing to be a challenge.
>
> Our scenario is:
>
> 1. Use mesos-execute to submit a job. A framework ID is available at
> this point.
> 2. Using the framework ID, one can inquire with mesos-state to
> determine which slave ID is executing the task.
> 3. Using the slave ID, one can inquire with mesos-state to find the
> hostname for that slave ID
> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
> there is an Executor ID which we cannot programmatically determine, to
> complete this URL.
>
> Please advise the simplest option for retrieving the sandbox files
> given the scenario starts with mesos-execute commands.
>
> Thanks!
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>

>>>
>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Erik Weathers]

Heh, fair enough.  I know for the master/agent daemons you can put the
config values into files, which makes this pretty clean (though
that probably depends on the OS/init-launcher-daemon you are using).  But
yeah, when using the CLI you have to be pretty careful.

- Erik

On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com> wrote:

> On a side note requiring people to put JSON on the command line is a
> sadistic thing to do.
>
> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <eweath...@groupon.com>
> wrote:
>
>> +1 I would wrap every string in quotes...  otherwise your shell doesn't
>> know what you mean.  i.e.,
>>
>> How is the shell supposed to know that you want this all to be 1 string
>> value for the --command parameter?
>>
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read
>>
>> Similarly how is the shell to know that you *don't* want to end this
>> command and start a new one when you have a bare semicolon?
>>
>> --resources=cpus:1;mem:128
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <haosd...@gmail.com> wrote:
>>
>>> I use bash as well. How about
>>> ```
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command="/home/john/anaconda3/bin/python
>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>> ```
>>>
>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <j...@umn.edu> wrote:
>>>
>>>> hasodent,
>>>>
>>>> We've tried your suggestion but it seems like that's not working out:
>>>>
>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>>> "/home/john/temp_output/test.csv"}'
>>>>
>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>>>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>>>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>
>>>>
>>>> We're using the Bash shell. Is anyone else passing file-paths within
>>>> their environment variable argument?
>>>>
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <haosd...@gmail.com> wrote:
>>>>
>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>
>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <ch...@galacticfog.com>
>>>>> wrote:
>>>>>
>>>>>> It's your shell:
>>>>>>
>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>
>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <j...@umn.edu> wrote:
>>>>>>
>>>>>>> hasodent,
>>>>>>>
>>>>>>> I admit I don't see much of a difference in your version. Are you
>>>>>>> saying that Mesos doesn't escape the values, and that our program may be
>>>>>>> doing that on the executor end?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> June Taylor
>>>>>>> System Administrator, Minnesota Population Center
>>>>>>> University of Minnesota
>>>>>>>
>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <haosd...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, @Taylor I use
>>>>>>>>
>>>>>>>> ```
>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>>>>>>>> --command=env
>>>>>>>> --name="test"
>>>>>>>> ```
>>>>>>>>
>>>&g

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Erik Weathers]

+1 I would wrap every string in quotes...  otherwise your shell doesn't
know what you mean.  i.e.,

How is the shell supposed to know that you want this all to be 1 string
value for the --command parameter?

--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read

Similarly how is the shell to know that you *don't* want to end this
command and start a new one when you have a bare semicolon?

--resources=cpus:1;mem:128

- Erik

On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:

> I use bash as well. How about
> ```
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
> ```
>
> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>
>> hasodent,
>>
>> We've tried your suggestion but it seems like that's not working out:
>>
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>>
>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>>
>> We're using the Bash shell. Is anyone else passing file-paths within
>> their environment variable argument?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>>
>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>
>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
>>> wrote:
>>>
 It's your shell:

 $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
 {MY_FILEPATH: /home/john/temp_output/test.csv}

 On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:

> hasodent,
>
> I admit I don't see much of a difference in your version. Are you
> saying that Mesos doesn't escape the values, and that our program may be
> doing that on the executor end?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 2:02 PM, haosdent  wrote:
>
>> Hi, @Taylor I use
>>
>> ```
>> ./src/mesos-execute --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>> --command=env
>> --name="test"
>> ```
>>
>> to test in my env. The output looks correct in my side
>>
>> ```
>> ...
>> MY_FILEPATH=/home/john/temp_output/test.csv
>>
>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>> ...
>> ```
>>
>> Not sure if your problem caused by your program
>>
>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:
>>
>>> Greetings,
>>>
>>> We are using mesos-execute and passing an argument that contains a
>>> Unix filepath, as follows:
>>>
>>> *mesos-execute --master=cluster.example.com:5050
>>>  --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}*
>>>
>>> The slashes in the file path are then apparently escaped by
>>> mesos-execute and therefore fail to be useful once picked up by the
>>> executor:
>>>
>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>
>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>> shutdown*
>>>
>>> Are other people experiencing this? What is the method to avoid it?
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: New external dependency

[Erik Weathers]

@Kevin:

FYI, it's best practice to use a commit SHA in GitHub links so that future
readers are seeing the content you intended.

i.e., instead of:

   -
   
https://github.com/NVIDIA/nvidia-docker/blob/master/tools/src/nvidia/volumes.go#L109

It's best to do:

   -
   
https://github.com/NVIDIA/nvidia-docker/blob/101b436c89c3a74e9a3025a104587b6612d903d8/tools/src/nvidia/volumes.go#L109


And (awesomely!) GitHub makes it trivial to do this!  [1]

   - when you're looking at a file (such as the original link you pasted),
   just type "y" and GitHub will redirect to the latest commit in master:

- Erik

[1] https://help.github.com/articles/getting-permanent-links-to-files/

On Mon, Jun 20, 2016 at 6:59 PM, Kevin Klues  wrote:

> For now we've decided to actually remove the hard dependence on libelf
> for the 1.0 release and spend a bit more time thinking about the right
> way to pull it in.
>
> Jean, to answer your question though -- someone would still need to
> consolidate these libraries, even if it wasn't left to Mesos to do so.
> These libraries are spread across the file system, and need to be
> pulled into a single place for easy injection. The full list of
> binaries / libraries are here:
>
>
> https://github.com/NVIDIA/nvidia-docker/blob/master/tools/src/nvidia/volumes.go#L109
>
> We could put this burden on the operator and trust he gets it right,
> or we could have Mesos programmatically do it itself. We considered
> just leveraging the nvidia-docker-plugin itself (instead of
> duplicating its functionality into mesos), but ultimately decided it
> was better not to introduce an external dependency on it (since it is
> a separate running excutable, rather than a simple library, like
> libelf).
>
> On Mon, Jun 20, 2016 at 5:12 PM, Jean Christophe “JC” Martin
>  wrote:
> > As an operator not using GPUs, I feel that the burden seems misplaced,
> and disproportionate.
> > I assume that the operator of a GPU cluster knows the location of the
> libraries based on their OS, and could potentially provide this information
> at the time of creating the containers. I am not sure to see why this
> something that mesos is required to do (consolidating the libraries in the
> volume, versus being a configuration/external information).
> >
> > Thanks,
> >
> > JC
> >
> >> On Jun 20, 2016, at 2:30 PM, Kevin Klues  wrote:
> >>
> >> Sorry, the ticket just links to the nvidia-docker project without much
> >> further explanation. The information at the link below should make it
> >> a bit more clear:
> >>
> >> https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver.
> >>
> >> The crux of the issue is that we need to be able consolidate all of
> >> the Nvidia binaries/libraries into a single volume that we inject into
> >> a docker container.  We use libelf is used to get the canonical names
> >> of all the Nvidia libraries (i.e. SONAME in their dynamic sections) as
> >> well as lookup what external dependences they have (i.e. NEEDED in
> >> their dynamic sections) in order to build this volume.
> >>
> >> NOTE: None of this volume support is actually in Mesos yet -- we just
> >> added the libelf dependence in anticipation of it.
> >>
> >>
> >>
> >>
> >> On Mon, Jun 20, 2016 at 12:59 PM, Yan Xu  wrote:
> >>> It's not immediately clear form the ticket why the change from optional
> >>> dependency to required dependency though? Could you summarize?
> >>>
> >>>
> >>> On Sun, Jun 19, 2016 at 12:33 PM, Kevin Klues 
> wrote:
> 
>  Thanks Zhitao,
> 
>  I just pushed out a review for upgrades.md and added you as a
> reviewer.
> 
>  The new dependence was added in the JIRA that haosdent linked, but the
>  actual reason for adding the dependence is more related to:
>  https://issues.apache.org/jira/browse/MESOS-5401
> 
>  On Sun, Jun 19, 2016 at 9:34 AM, haosdent  wrote:
> > The related issue is Change build to always enable Nvidia GPU support
> > for
> > Linux
> > Last time my local build break before Kevin send out the email, and
> then
> > find this change.
> >
> > On Mon, Jun 20, 2016 at 12:11 AM, Zhitao Li 
> > wrote:
> >>
> >> Hi Kevin,
> >>
> >> Thanks for letting us know. It seems like this is not called out in
> >> upgrades.md, so can you please document this additional dependency
> >> there?
> >>
> >> Also, can you include the link to the JIRA or patch requiring this
> >> dependency so we can have some contexts?
> >>
> >> Thanks!
> >>
> >> On Sat, Jun 18, 2016 at 10:25 AM, Kevin Klues 
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> Just an FYI that the newest libmesos now has an external dependence
> >>> on
> >>> libelf on Linux. This dependence can be installed via the following
> >>> packages:
> >>>
> 

Re: Mesos 0.28.2 does not start

[Erik Weathers]

I'm not 100% sure what is implied by "public" and "floating" IPs here.  Are
the IPs of 10.20.250.x the *actual* IPs on the hosts?  i.e., if you run
"ifconfig -a" or "ip a" do those IPs appear?   If not then you cannot bind
to them on the host.

- Erik

On Sat, Jun 11, 2016 at 10:32 AM, Stefano Bianchi 
wrote:

> the problem is presented also on 0.27.2 i tried to perform a yum downgrade
> mesos-0.27.2 and i still have this problem. i don't understand why. i
> followed the common procedure as for my previous work.
>
> 2016-06-11 19:30 GMT+02:00 Stefano Bianchi :
>
>> The floating IPs are correclty set both on /etc/mesos-master/ip and
>> /etc/mesos-master/hostname
>>
>> 2016-06-11 19:27 GMT+02:00 Stefano Bianchi :
>>
>>> i checked now, and i have correctly set 10.250.0.124, 10.250.0.125, 
>>> 10.250.0.126
>>> according to the masters floating IPs
>>>
>>> 2016-06-11 19:18 GMT+02:00 haosdent :
>>>
 Oh, according to the log. It looks like your set /etc/mesos-master/ip
 to 10.250.0.12?

 On Sun, Jun 12, 2016 at 1:15 AM, Stefano Bianchi 
 wrote:

> i dont' have 10.250.0.12  floating IP, i
> have 10.250.0.124  10.250.0.125
>  and 10.250.0.126 
>
> 2016-06-11 19:13 GMT+02:00 Stefano Bianchi :
>
>> sorry i have made a mistakee in my previous message,
>> On each master: nano /etc/zookeeper/conf/zoo.cfg
>> the last three lines are:
>> server.1=192.168.100.3:2888:3888
>> server.2=192.168.100.4:2888:3888
>> server.3=192.168.10.3:2888:3888
>>
>> these are the correct IPs i have set in this file.
>>
>> 2016-06-11 15:36 GMT+02:00 haosdent :
>>
>>> If 10.250.0.12:5050 used by any other processes?
>>>
>>> On Sat, Jun 11, 2016 at 9:18 PM, Stefano Bianchi <
>>> jazzist...@gmail.com> wrote:
>>>
 I have set:

 1)
 On master1:
 sudo -u zookeeper zookeeper-server-initialize --myid=1

 On Master2:
 sudo -u zookeeper zookeeper-server-initialize --myid=2

 On Master3:
 sudo -u zookeeper zookeeper-server-initialize --myid=3

 i checked /var/lib/zookeeper/myid and i get correctly 1, 2 and 3


 2) On each master: nano /etc/zookeeper/conf/zoo.cfg
 the last three lines are:
 server.1=192.168.100.54:2888:3888
 server.2=192.168.100.55:2888:3888
 server.3=192.168.10.16:2888:3888

 where 2888 and 3888 are both open on openstack security group.

 3)nano /etc/mesos/zk

 zk://192.168.100.3:2181,192.168.100.4:2181,192.168.10.3:2181/mesos

 4) Set the quorum to 2with
 nano /etc/mesos-master/quorum  and i have written 2

 5)then in /etc/mesos-master/ i created hostname and ip files,
 within which i have written the public IPs of the masters such that i 
 can
 use sandbox on web ui.
 the same in /etc/marathon/conf/ just creating hostname file.

 let me say that the openstack cluster on which i'm now working on
 isnot on the public internet but under a private network of the 
 university
 lab.





 2016-06-10 22:21 GMT+02:00 Joseph Wu :

> Looks like the master can't bind to the IP you have specified (or,
> if you did not specify, the master resolved it from the computer's
> hostname).
>
> Either specify the --ip master flag or investigate the problem
> with your network.
>
> On Fri, Jun 10, 2016 at 11:03 AM, Stefano Bianchi <
> jazzist...@gmail.com> wrote:
>
>> this is what i get with your command:
>>
>> giu 10 18:02:23 master.novalocal systemd[1]: *Unit
>> mesos-master.service entered failed state.*
>>
>> giu 10 18:02:23 master.novalocal systemd[1]: *mesos-master.service
>> failed.*
>>
>> [root@master ~]# journalctl -u mesos-master
>>
>> -- Logs begin at gio 2016-06-09 23:24:02 UTC, end at ven
>> 2016-06-10 18:02:23 UTC. --
>>
>> giu 09 23:24:35 master.novalocal systemd[1]: Started Mesos Master.
>>
>> giu 09 23:24:35 master.novalocal systemd[1]: Starting Mesos
>> Master...
>>
>> giu 09 23:24:51 master.novalocal mesos-master[808]: *WARNING:
>> Logging before InitGoogleLogging() is written to STDERR*
>>
>> giu 09 23:24:51 master.novalocal mesos-master[808]: *F0609
>> 23:24:51.739418   718 process.cpp:892] Failed to initialize: Failed 

Re: Port Resource Offers

[Erik Weathers]

hi Pradeep,

Yes, that would *definitely* be a problem.  e.g., the Storm Framework could
easily assign Storm Workers to use those unavailable ports, and then they
would fail to come up since they wouldn't be able to bind to their assigned
port.  I've answered a similar question before:

https://unix.stackexchange.com/questions/211647/how-safe-is-it-to-change-the-linux-ephemeral-port-range/237543#237543

- Erik

On Tue, Mar 29, 2016 at 3:07 AM, Pradeep Chhetri <
pradeep.chhetr...@gmail.com> wrote:

> Hi Klaus,
>
> Thank you for the quick reply.
>
> One quick question:
>
> I have some of the ports like 8400,8500,8600 which are already in use by
> consul agent running on each mesos slave. But they are also being announced
> by each mesos slave. Will this cause any problem to tasks which maybe
> assigned those ports in future by mesos ?
>
> Thanks
>
> On Tue, Mar 29, 2016 at 11:01 AM, Klaus Ma  wrote:
>
>> Yes, all port resources must be ranges for now, e.g. 31000-35000.
>>
>> There’s already JIRA (MESOS-4627: Improve Ranges parsing to handle single
>> values) on that, patches are pending on review :).
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>>
>> --
>> Date: Tue, 29 Mar 2016 10:51:44 +0100
>> Subject: Port Resource Offers
>> From: pradeep.chhetr...@gmail.com
>> To: user@mesos.apache.org
>>
>>
>> Hello,
>>
>> I am running mesos slaves with the modified port announcement.
>>
>> $ cat /etc/mesos-slave/resources
>> ports(*):[6379, 9200, 9300, 27017, 31000-35000]
>>
>> I can that this is being picked up when starting the mesos slaves in ps
>> output:
>>
>> --resources=ports(*):[6379, 9200, 9300, 27017, 31000-35000]
>>
>> However, when i hit the /state.json endpoint of mesos-master, I am seeing
>> this:
>>
>>
>> ​
>> I can see the tasks are being assigned ports in the range of 9300-27017.
>> There are some of these ports which are already used by other applications
>> running on each mesos slaves but are being announced. I am not sure if this
>> will cause some issue. I am assuming that it will always check if the port
>> is already binded by some other process before assigning port to a task.
>>
>> By going through the code and test cases, it looks like it always expect
>> port resource in ranges.
>>
>>
>> https://github.com/apache/mesos/blob/master/src/v1/resources.cpp#L1255-L1263
>>
>> So I guess, I should always define ports in ranges rather than individual
>> port.
>>
>> It will be helpful if someone can confirm if it is the expected behaviour
>> and my configuration is wrong.
>>
>> --
>> Regards,
>> Pradeep Chhetri
>>
>
>
>
> --
> Regards,
> Pradeep Chhetri
>

Re: Reusing Task IDs

[Erik Weathers]

Thanks for the responses.  Filed a ticket for this:

   - https://issues.apache.org/jira/browse/MESOS-4737

- Erik

On Mon, Feb 22, 2016 at 1:23 PM, Sargun Dhillon <sar...@sargun.me> wrote:

> As someone who has been there and back again (Reusing task-IDs, and
> realizing it's a terrible idea), I'd put some advise in the docs +
> mesos.proto to compose task IDs from GUIDs, and add that it's
> dangerous to reuse them.
>
> I would advocate for a mechanism to prevent the usage of non-unique
> IDs for executors, tasks, and frameworks, but I feel that's a more
> complex, and larger problem.
>
> On Mon, Feb 22, 2016 at 1:19 PM, Vinod Kone <vinodk...@apache.org> wrote:
> > I would vote for updating comments in mesos.proto to warn users to not
> > re-use task id for now.
> >
> > On Sun, Feb 21, 2016 at 9:05 PM, Klaus Ma <klaus1982...@gmail.com>
> wrote:
> >>
> >> Yes, it's dangerous to reuse TaskID; there's a JIRA (MESOS-3070) that
> >> Master'll crash when Master failover with duplicated TaskID.
> >>
> >> Here's the case of MESOS-3070:
> >> T1: launch task (t1) on agent (agent_1)
> >> T2: master failover
> >> T3: launch another task (t1) on agent (agent_2) before agent_1
> >> re-registering back
> >> T4: agent_1 re-registered back; master'll crash because of `CHECK` when
> >> add task (t1) back to master
> >>
> >> Is there any special case that framework has to re-use the TaskID; if no
> >> special case, I think we should ask framework to avoid reuse TaskID.
> >>
> >> 
> >> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> >> Platform OpenSource Technology, STG, IBM GCG
> >> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >>
> >> On Mon, Feb 22, 2016 at 12:24 PM, Erik Weathers <eweath...@groupon.com>
> >> wrote:
> >>>
> >>> tldr; Reusing TaskIDs clashes with the mesos-agent recovery feature.
> >>>
> >>> Adam Bordelon wrote:
> >>> > Reusing taskIds may work if you're guaranteed to never be running two
> >>> > instances of the same taskId simultaneously
> >>>
> >>> I've encountered another scenario where reusing TaskIDs is dangerous,
> >>> even if you meet the guarantee of never running 2 task instances with
> the
> >>> same TaskID simultaneously.
> >>>
> >>> Scenario leading to a problem:
> >>>
> >>> Say you have a task with ID "T1", which terminates for some reason, so
> >>> its terminal status update gets recorded into the agent's current
> "run" in
> >>> the task's updates file:
> >>>
> >>>
> >>>
> MESOS_WORK_DIR/meta/slaves/latest/frameworks/FRAMEWORK_ID/executors/EXECUTOR_ID/runs/latest/tasks/T1/task.updates
> >>>
> >>> Then say a new task is launched with the same ID of T1, and it gets
> >>> scheduled under the same Executor on the same agent host. In that
> case, the
> >>> task will be reusing the same work_dir path, and thus have the already
> >>> recorded "terminal status update" in its task.updates file.  So the
> updates
> >>> file has a stream of updates that might look like this:
> >>>
> >>> TASK_RUNNING
> >>> TASK_FINISHED
> >>> TASK_RUNNING
> >>>
> >>> Say you subsequently restart the mesos-slave/agent, expecting all tasks
> >>> to survive the restart via the recovery process.  Unfortunately, T1 is
> >>> terminated because the task recovery logic [1] looks at the current
> run's
> >>> tasks' task.updates files, searching for tasks with "terminal status
> >>> updates", and then terminating any such tasks.  So, even though T1 was
> >>> actually running just fine, it gets terminated because at some point
> in its
> >>> previous incarnation it had a "terminal status update" recorded.
> >>>
> >>> Leads to inconsistent state
> >>>
> >>> Compounding the problem, this termination is done without informing the
> >>> Executor, and thus the process underlying the task continues to run,
> even
> >>> though mesos thinks it's gone.  Which is really bad since it leaves
> the host
> >>> with a different state than mesos thinks exists. e.g., if the task had
> a
> >>> port resource, then mesos incorrectly thinks the port is now free, so a
> >>> framework might try to launch a task/exe

Re: Reusing Task IDs

[Erik Weathers]

tldr; *Reusing TaskIDs clashes with the mesos-agent recovery feature.*

Adam Bordelon wrote:
> Reusing taskIds may work if you're guaranteed to never be running two
instances of the same taskId simultaneously

I've encountered another scenario where reusing TaskIDs is dangerous, even
if you meet the guarantee of never running 2 task instances with the same
TaskID simultaneously.

*Scenario leading to a problem:*

Say you have a task with ID "T1", which terminates for some reason, so its
terminal status update gets recorded into the agent's current "run" in the
task's updates file:

MESOS_WORK_DIR/meta/slaves/latest/frameworks/FRAMEWORK_ID/executors/EXECUTOR_ID/runs/latest/tasks/T1/task.updates

Then say a new task is launched with the same ID of T1, and it gets
scheduled under the same Executor on the same agent host. In that case, the
task will be reusing the same work_dir path, and thus have the already
recorded "terminal status update" in its task.updates file.  So the updates
file has a stream of updates that might look like this:

   - TASK_RUNNING
   - TASK_FINISHED
   - TASK_RUNNING

Say you subsequently restart the mesos-slave/agent, expecting all tasks to
survive the restart via the recovery process.  Unfortunately, T1 is
terminated because the task recovery logic

[1]
looks at the current run's tasks' task.updates files, searching for tasks
with "terminal status updates", and then terminating any such tasks.  So,
even though T1 was actually running just fine, it gets terminated because
at some point in its previous incarnation it had a "terminal status update"
recorded.

*Leads to inconsistent state*

Compounding the problem, this termination is done without informing the
Executor, and thus the process underlying the task continues to run, even
though mesos thinks it's gone.  Which is really bad since it leaves the
host with a different state than mesos thinks exists. e.g., if the task had
a port resource, then mesos incorrectly thinks the port is now free, so a
framework might try to launch a task/executor that uses the port, but it
will fail because the process cannot bind to the port.

*Change recovery code or just update comments in mesos.proto?*

Perhaps this behavior could be considered a "bug" and the recovery logic
that processes tasks status updates could be modified to ignore "terminal
status updates" if there is a subsequent TASK_RUNNING update in the
task.updates file.  If that sounds like a desirable change, I'm happy to
file a JIRA issue for that and work on the fix myself.

If we think the recovery logic is fine as it is, then we should update these
comments

[2]
in mesos.proto since they are incorrect given the behavior I just
encountered:

A framework generated ID to distinguish a task. The ID must remain
> unique while the task is active. However, a framework can reuse an
> ID _only_ if a previous task with the same ID has reached a
> terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).


*Conclusion*

It is dangerous indeed to reuse a TaskID for separate task runs, even if
they are guaranteed to not be running concurrently.

- Erik


P.S., I encountered this problem while trying to use mesos-agent recovery
with the storm-mesos framework  [3].
Notably, this framework sets the TaskID to
"-" for the storm worker tasks, so when a
storm worker dies and is reborn on that host, the TaskID gets reused.  But
then the task doesn't survive an agent restart (even though the worker
*process* does survive, putting us in an inconsistent state!).

P.P.S., being able to enable verbose logging in mesos-slave/agent with the
GLOG_v=3 environment variable is *super* convenient!  Would have taken me
*way* longer to figure this out if the verbose logging didn't exist.

P.P.P.S, To debug this, I wrote a tool
 [4] to decode
length-prefixed
protobuf

[5]
files, such as task.updates.

Here's an example of invoking the tool (notably, it has the same syntax as
"protoc --decode", but handles the length-prefix headers):

cat task.updates | \
  protoc-decode-lenprefix \
--decode mesos.internal.StatusUpdateRecord \
-I MESOS_CODE/src -I MESOS_CODE/include \
MESOS_CODE/src/messages/messages.proto


[1]
https://github.com/apache/mesos/blob/0.27.0/src/slave/slave.cpp#L5701-L5708
[2]
https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66
[3] https://github.com/mesos/storm
[4] https://github.com/erikdw/protoc-decode-lenprefix
[5]
http://eli.thegreenplace.net/2011/08/02/length-prefix-framing-for-protocol-buffers

On Sat, Jul 11, 2015 at 11:45 AM, CCAAT  wrote:

> I'd be most curious to see a working example of this idea, 

Re: memory limit exceeded ==> KILL instead of TERM (first)

[Erik Weathers]

hey Harry,

As Vinod said, the mesos-slave/agent will issue a status update about the
OOM condition.  This will be received by the scheduler of the framework.
In the storm-mesos framework we just log the messages (see below), but you
might consider somehow exposing these messages directly to the app owners:

Received status update:
{"task_id":"TASK_ID","slave_id":"20150806-001422-1801655306-5050-22041-S65","state":"TASK_FAILED","message":"Memory
limit exceeded: Requested: 2200MB Maximum Used: 2200MB\n\nMEMORY
STATISTICS: \ncache 20480\nrss 1811943424\nmapped_file 0\npgpgin
8777434\npgpgout 8805691\nswap 96878592\ninactive_anon
644186112\nactive_anon 1357594624\ninactive_file 20480\nactive_file
0\nunevictable 0\nhierarchical_memory_limit
2306867200\nhierarchical_memsw_limit 9223372036854775807\ntotal_cache
20480\ntotal_rss 1811943424\ntotal_mapped_file 0\ntotal_pgpgin
8777434\ntotal_pgpgout 8805691\ntotal_swap 96878592\ntotal_inactive_anon
644186112\ntotal_active_anon 1355497472\ntotal_inactive_file
20480\ntotal_active_file 0\ntotal_unevictable 0"}

- Erik

On Fri, Feb 12, 2016 at 10:24 AM, Harry Metske 
wrote:

> David,
>
> that's exactly the scenario I am afraid of, developers specifying way too
> large memory requirements, just to make sure their tasks don't get killed.
> Any suggestions on how to report this *why* to the developers, as far as I
> know the only place where you find the reason is in the logfile of the
> slave, the UI only tells the task failed, not the reason.
>
> (we could put some logfile monitoring in place picking up these messages
> of course, but if there are better ways, we are always interested)
>
> kind regards,
> Harry
>
>
> On 12 February 2016 at 15:08, David J. Palaitis <
> david.j.palai...@gmail.com> wrote:
>
>> In larger deployments, with many applications, you may not always be able
>> to ask good memory practices from app developers. We've found that
>> reporting *why* a job was killed, with details of container utilization, is
>> an effective way of helping app developers get better at mem mgmt.
>>
>> The alternative, just having jobs die, incentives bad behaviors. For
>> example, a hurried job owner may just double memory of the executor,
>> trading slack for stability.
>>
>> On Fri, Feb 12, 2016 at 6:36 AM Harry Metske 
>> wrote:
>>
>>> We don't want to use Docker (yet) in this environment, so 
>>> DockerContainerizer
>>> is not an option.
>>> After thinking a bit longer, I tend to agree with Kamil and let the
>>> problem be handled differently.
>>>
>>> Thanks for the amazing fast responses!
>>>
>>> kind regards,
>>> Harry
>>>
>>>
>>> On 12 February 2016 at 12:28, Kamil Chmielewski 
>>> wrote:
>>>
 On Fri, Feb 12, 2016 at 6:12 PM, Harry Metske 
>>> wrote:
>>>

 Is there a specific reason why the slave does not first send a TERM
 signal, and if that does not help after a certain timeout, send a KILL
 signal?
 That would give us a chance to cleanup consul registrations (and
 other cleanup).


 First of all it's wrong that you want to handle memory limit in your
 app. Things like this are outside of its scope. Your app can be lost
 because many different system or hardware failures that you just can't
 caught. You need to let it crash and design your architecture with this in
 mind.
 Secondly Mesos SIGKILL is consistent with linux OOM killer and it do
 the right thing
 https://github.com/torvalds/linux/blob/4e5448a31d73d0e944b7adb9049438a09bc332cb/mm/oom_kill.c#L586

 Best regards,
 Kamil

>>>
>>>
>

Re: Zookeeper cluster changes

[Erik Weathers]

Keep in mind that mesos is designed to "fail fast".  So when there are
problems (such as losing connectivity to the resolved ZooKeeper IP) the
daemon(s) (master & slave) die.

Due to this design, we are all supposed to run the mesos daemons under
"supervision", which means auto-restart after they crash.  This can be done
with monit/god/runit/etc.

So, to perform maintenance on ZooKeeper, I would firstly ensure the
mesos-master processes are running under "supervision" so that they restart
quickly after a ZK connectivity failure occurs.  Then proceed with standard
ZooKeeper maintenance (exhibitor-based or manual), pausing between downing
of ZK servers to ensure you have "enough" mesos-master processes running.
 (I *would* say a "pausing until you have a quorum of mesos-masters up",
but if you only have 2 of 3 up and then take down the ZK that the leader is
connected to, that would be temporarily bad.  So I'd make sure they're all
up.)

- Erik

On Mon, Nov 9, 2015 at 11:07 PM, Marco Massenzio 
wrote:

> The way I would do it in a production cluster would be *not* to use
> directly IP addresses for the ZK ensemble, but instead rely on some form of
> internal DNS and use internally-resolvable hostnames (eg, {zk1, zk2, ...}.
> prod.example.com etc) and have the provisioning tooling (Chef, Puppet,
> Ansible, what have you) handle the setting of the hostname when
> restarting/replacing a failing/crashed ZK server.
>
> This way your list of zk's to Mesos never changes, even though the FQN's
> will map to different IPs / VMs.
>
> Obviously, this may not be always desirable / feasible (eg, if your prod
> environment does not support DNS resolution).
>
> You are correct in that Mesos does not currently support dynamically
> changing the ZK's addresses, but I don't know whether that's a limitation
> of Mesos code or of the ZK C++ client driver.
> I'll look into it and let you know what I find (if anything).
>
> --
> *Marco Massenzio*
> Distributed Systems Engineer
> http://codetrips.com
>
> On Mon, Nov 9, 2015 at 6:01 AM, Donald Laidlaw  wrote:
>
>> How do mesos masters and slaves react to zookeeper cluster changes? When
>> the masters and slaves start they are given a set of addresses to connect
>> to zookeeper. But over time, one of those zookeepers fails, and is replaced
>> by a new server at a new address. How should this be handled in the mesos
>> servers?
>>
>> I am guessing that mesos does not automatically detect and react to that
>> change. But obviously we should do something to keep the mesos servers
>> happy as well. What should be do?
>>
>> The obvious thing is to stop the mesos servers, one at a time, and
>> restart them with the new configuration. But it would be really nice to be
>> able to do this dynamically without restarting the server. After all,
>> coordinating a rolling restart is a fairly hard job.
>>
>> Any suggestions or pointers?
>>
>> Best regards,
>> Don Laidlaw
>>
>>
>>
>

Re: Mesos and Zookeeper TCP keepalive

[Erik Weathers]

It would really help if you (Jeremy) explained the *actual* problem you are
facing.  I'm *guessing* that it's a firewall timing out the sessions
because there isn't activity on them for whatever the timeout of the
firewall is?   It seems likely to be unreasonably short, given that mesos
has constant activity between master and
slave/agent/whatever-it-is-being-called-nowadays-but-not-really-yet-maybe-someday-for-reals.

- Erik

On Mon, Nov 9, 2015 at 10:00 PM, Jojy Varghese  wrote:

> Hi Jeremy
>  Its great that you are making progress but I doubt if this is what you
> intend to achieve since network failures are a valid state in distributed
> systems. If you think there is a special case you are trying to solve, I
> suggest proposing a design document for review.
>   For ZK client code, I would suggest asking the zookeeper mailing list.
>
> thanks
> -Jojy
>
> On Nov 9, 2015, at 7:56 PM, Jeremy Olexa  wrote:
>
> Alright, great, I'm making some progress,
>
> I did a simple copy/paste modification and recompiled mesos. The keepalive
> timer is set from slave to master so this is an improvement for me. I
> didn't test the other direction yet -
> https://gist.github.com/jolexa/ee9e152aa7045c558e02 - I'd like to file an
> enhancement request for this since it seems like an improvement for other
> people as well, after some real world testing
>
> I'm having some harder time figuring out the zk client code. I started by
> modifying build/3rdparty/zookeeper-3.4.5/src/c/zookeeper.c but either a) my
> change wasn't correct or b) I'm modifying a wrong file, since I
> just assumed using the c client. Is this the correct place?
>
> Thanks much,
> Jeremy
>
>
> --
> *From:* Jojy Varghese 
> *Sent:* Monday, November 9, 2015 2:09 PM
> *To:* user@mesos.apache.org
> *Subject:* Re: Mesos and Zookeeper TCP keepalive
>
> Hi Jeremy
>  The “network” code is at
> "3rdparty/libprocess/include/process/network.hpp” ,
> "3rdparty/libprocess/src/poll_socket.hpp/cpp”.
>
> thanks
> jojy
>
>
> On Nov 9, 2015, at 6:54 AM, Jeremy Olexa  wrote:
>
> Hi all,
>
> Jojy, That is correct, but more specifically a keepalive timer from slave
> to master and slave to zookeeper. Can you send a link to the portion of the
> code that builds the socket/connection? Is there any reason to not set the
> SO_KEEPALIVE option in your opinion?
>
> hasodent, I'm not looking for keepalive between zk quorum members, like
> the ZOOKEEPER JIRA is referencing.
>
> Thanks,
> Jeremy
>
>
> --
> *From:* Jojy Varghese 
> *Sent:* Sunday, November 8, 2015 8:37 PM
> *To:* user@mesos.apache.org
> *Subject:* Re: Mesos and Zookeeper TCP keepalive
>
> Hi Jeremy
>   Are you trying to establish a keepalive timer between mesos master and
> mesos slave? If so, I don’t believe its possible today as SO_KEEPALIVE
> option is  not set on an accepting socket.
>
> -Jojy
>
> On Nov 8, 2015, at 8:43 AM, haosdent  wrote:
>
> I think keepalive option should be set in Zookeeper, not in Mesos. See
> this related issue in Zookeeper.
> https://issues.apache.org/jira/browse/ZOOKEEPER-2246?focusedCommentId=14724085=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14724085
>
> On Sun, Nov 8, 2015 at 4:47 AM, Jeremy Olexa 
> wrote:
>
>> Hello all,
>>
>> We have been fighting some network/session disconnection issues between
>> datacenters and I'm curious if there is anyway to enable tcp keepalive on
>> the zookeeper/mesos sockets? If there was a way, then the sysctl tcp
>> kernel settings would be used. I believe keepalive has to be enabled by the
>> software which is opening the connection. (That is my understanding anyway)
>>
>> Here is what I see via netstat --timers -tn:
>> tcp0  0 172.18.1.1:55842  10.10.1.1:2181
>>  ESTABLISHED off (0.00/0/0)
>> tcp0  0 172.18.1.1:49702  10.10.1.1:5050
>>  ESTABLISHED off (0.00/0/0)
>>
>>
>> Where 172 is the mesos-slave network and 10 is the mesos-master network.
>> The "off" keyword means that keepalive's are not being sent.
>>
>> I've trolled through JIRA, git, etc and cannot easily determine if this
>> is expected behavior or should be an enhancement request. Any ideas?
>>
>> Thanks much!
>> -Jeremy
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>

preventing registry failures from happening in mesos-master?

[Erik Weathers]

I know we're supposed to run the mesos daemons under supervision (i.e.,
bring them back up automatically if they fail).   But I'm interested in not
having the mesos-master fail at all, especially a failure in the registry /
replicated_log, which I am already a little scared of.

Situation:

   - Mesos version: 0.20.1
   - 30 mesos-slave hosts (on bare metal)
  - originally had 30, now have 39
   - 3 mesos-master hosts (on VMs)
   - 5 zookeepers (on bare metal)

Problems during slave addition:

(1) Brought up 1 brand new slave, this caused the acting master to die with
this error:

*Failed to admit slave ... Failed to update 'registry': Failed to perform
store within 5secs*


(2) 11 minutes later, brought up 8 more brand new slaves, this caused the
new acting master to die with this error:

*Failed to admit slave ... Failed to update 'registry': version mismatch*


I'm now even more afraid of the registry now. :(Is it likely that
there's some fundamental improperness in my configuration and/or setup that
would lead to the registry being so fragile?   I was guessing that running
the mesos-master on VMs might be bad and lead to the inital error about the
store not completing within 5 seconds.  But the latter problem is just
baffling to me.  Everything *seems* ok right now.  Maybe.  Hopefully.

Thanks!

- Erik

Re: docker based executor

[Erik Weathers]

hey Tyson,

I've also worked a bit on improving  simplifying the mesos-storm framework
-- spent the recent Mesosphere hackathon working with tnachen of Mesosphere
on this.  Nothing deliverable quite yet.

We didn't look at dockerization at all, the hacking we did was around these
goals:
* Avoiding the greedy hoarding of Offers done by the mesos-storm framework
(ditching RotatingMap, and only hoarding Offers when there are topologies
that need storm worker slots).
* Allowing the Mesos UI to distinguish the topologies, by having the Mesos
tasks be dedicated to a topology.
* Adding usable logging in MesosNimbus. (Some of this work should be usable
by other Mesos frameworks, since I'm pretty-printing the Mesos protobuf
objects in 1-line JSON instead of bazillion line protobuf toString()
pseudo-JSON output.  Would be nice to create a library out of it.)

Would you like to participate in an offline thread mesos-storm refactoring?

Thanks!

- Erik

On Fri, Apr 17, 2015 at 12:23 PM, Tyson Norris tnor...@adobe.com wrote:

 Hi -
 I am looking at revving the mesos-storm framework to be dockerized (and
 simpler).
 I’m using mesos 0.22.0-1.0.ubuntu1404
 mesos master + mesos slave are deployed in docker containers, in case it
 matters.

 I have the storm (nimbus) framework launching fine as a docker container,
 but launching tasks for a topology is having problems related to using a
 docker-based executor.

 For example.

 TaskInfo task = TaskInfo.newBuilder()
 .setName(worker  + slot.getNodeId() + : + slot.getPort())
 .setTaskId(taskId)
 .setSlaveId(offer.getSlaveId())
 .setExecutor(ExecutorInfo.newBuilder()

 .setExecutorId(ExecutorID.newBuilder().setValue(details.getId()))
 .setData(ByteString.copyFromUtf8(executorDataStr))
 .setContainer(ContainerInfo.newBuilder()
 .setType(ContainerInfo.Type.DOCKER)

 .setDocker(ContainerInfo.DockerInfo.newBuilder()
 .setImage(mesos-storm”)))

 .setCommand(CommandInfo.newBuilder().setShell(true).setValue(storm
 supervisor storm.mesos.MesosSupervisor))
 //rest is unchanged from existing mesos-storm framework
 code

 The executor launches and exits quickly - see the log msg:  Executor for
 container '88ce3658-7d9c-4b5f-b69a-cb5e48125dfd' has exited

 It seems like mesos loses track of the executor? I understand there is a 1
 min timeout on registering the executor, but the exit happens well before 1
 minute.

 I tried a few alternate commands to experiment, and I can see in the
 stdout for the task that
 echo testing123  echo testing456”
 prints to stdout correctly, both testing123 and testing456

 however:
 echo testing123a  sleep 10  echo testing456a”
 prints only testing123a, presumably because the container is lost and
 destroyed before the sleep time is up.

 So it’s like the container for the executor is only allowed to run for .5
 seconds, then it is detected as exited, and the task is lost.

 Thanks for any advice.

 Tyson



 slave logs look like:
 mesosslave_1  | I0417 19:07:27.46123011 slave.cpp:1121] Got assigned
 task mesos-slave1.service.consul-31000 for framework
 20150417-190611-2801799596-5050-1-
 mesosslave_1  | I0417 19:07:27.46147911 slave.cpp:1231] Launching task
 mesos-slave1.service.consul-31000 for framework
 20150417-190611-2801799596-5050-1-
 mesosslave_1  | I0417 19:07:27.46325011 slave.cpp:4160] Launching
 executor insights-1-1429297638 of framework
 20150417-190611-2801799596-5050-1- in work directory
 '/tmp/mesos/slaves/20150417-190611-2801799596-5050-1-S0/frameworks/20150417-190611-2801799596-5050-1-/executors/insights-1-1429297638/runs/6539127f-9dbb-425b-86a8-845b748f0cd3'
 mesosslave_1  | I0417 19:07:27.46344411 slave.cpp:1378] Queuing task
 'mesos-slave1.service.consul-31000' for executor insights-1-1429297638 of
 framework '20150417-190611-2801799596-5050-1-
 mesosslave_1  | I0417 19:07:27.467200 7 docker.cpp:755] Starting
 container '6539127f-9dbb-425b-86a8-845b748f0cd3' for executor
 'insights-1-1429297638' and framework
 '20150417-190611-2801799596-5050-1-'
 mesosslave_1  | I0417 19:07:27.985935 7 docker.cpp:1333] Executor for
 container '6539127f-9dbb-425b-86a8-845b748f0cd3' has exited
 mesosslave_1  | I0417 19:07:27.986359 7 docker.cpp:1159] Destroying
 container '6539127f-9dbb-425b-86a8-845b748f0cd3'
 mesosslave_1  | I0417 19:07:27.986021 9 slave.cpp:3135] Monitoring
 executor 'insights-1-1429297638' of framework
 '20150417-190611-2801799596-5050-1-' in container
 '6539127f-9dbb-425b-86a8-845b748f0cd3'
 mesosslave_1  | I0417 19:07:27.986464 7 docker.cpp:1248] Running
 docker stop on container '6539127f-9dbb-425b-86a8-845b748f0cd3'
 mesosslave_1  | I0417 19:07:28.28676110 slave.cpp:3186] Executor
 'insights-1-1429297638' of framework 20150417-190611-2801799596-5050-1-
 has terminated with unknown status
 mesosslave_1  | I0417