Re: Programmatically retrieve stdout/stderr from a node

[Benjamin Mahler]

Also I believe the CLI work that Haris / Kevin have been doing would make
this easy to do via the Mesos CLI (it's not integrated into the project
yet).

On Wed, Aug 10, 2016 at 9:57 AM, Erik Weathers 
wrote:

> Just for completeness and to provide an alternative, you can also probably
> leverage the dcos command line tool (https://github.com/dcos/dcos-cli) to
> get all the info you would need in a JSON format.
>
> e.g.,
> 1. set up ~/.dcos/config.toml for your cluster
> 2. DCOS_CONFIG=~/.dcos/config.toml dcos task --json --completed
>
> You could process that output with jq (https://stedolan.github.io/jq/)
> and do all of this in a short script.  (Not that I have much luck using jq,
> I'm not very skilled at working with such arcane syntax, a la XPath.)
>
> - Erik
>
> On Wed, Aug 10, 2016 at 9:27 AM, June Taylor  wrote:
>
>> David,
>>
>> Thanks for the suggestions, this has the missing piece we needed!
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Aug 10, 2016 at 10:54 AM, David Greenberg > > wrote:
>>
>>> The Cook framework has examples of how to do this. See here (
>>> https://github.com/twosigma/Cook/blob/master/scheduler/src/
>>> cook/mesos/api.clj#L322-L324) for constructing the stem of the URL,
>>> here (https://github.com/twosigma/Cook/blob/7a49fbb98b281e3b23779
>>> cd88d1d2b73428a0447/scheduler/src/cook/mesos/api.clj#L281-L297) for
>>> finding the filesystem path, and here (https://github.com/twosigma/C
>>> ook/blob/master/scheduler/docs/scheduler-rest-api.asc#using_output_url)
>>> for getting data with the stem.
>>>
>>> Essentially, you need to scrape the Mesos master to get all the path
>>> info you need. LMK if you have questions!
>>>
>>> On Wed, Aug 10, 2016 at 9:34 AM June Taylor  wrote:
>>>
 Tomek,

 I'm not sure I understand your suggestion. We know how to ask for a
 file from an HTTP endpoint, but it is the construction of the correct URL
 which is not currently clear.

 We are not sure how to determine the Run ID of the executor.


 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

 On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
 wrote:

> Hi
>
> If you need simplest method then python SimpleHTTPServer could help.
> Just launch it in background before command you want to run, assign it 
> port
> and query sandbox with : that can be obtained from
> state endpoint.
>
> -
> Tomek
>
> śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:
>
>> We are trying to retrieve the stdout and stderr files from an
>> executor programmatically.
>>
>> It appears that these are available via HTTP request, however,
>> constructing the correct URL is posing to be a challenge.
>>
>> Our scenario is:
>>
>> 1. Use mesos-execute to submit a job. A framework ID is available at
>> this point.
>> 2. Using the framework ID, one can inquire with mesos-state to
>> determine which slave ID is executing the task.
>> 3. Using the slave ID, one can inquire with mesos-state to find the
>> hostname for that slave ID
>> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
>> there is an Executor ID which we cannot programmatically determine, to
>> complete this URL.
>>
>> Please advise the simplest option for retrieving the sandbox files
>> given the scenario starts with mesos-execute commands.
>>
>> Thanks!
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>

>>
>

Re: Programmatically retrieve stdout/stderr from a node

[Erik Weathers]

Just for completeness and to provide an alternative, you can also probably
leverage the dcos command line tool (https://github.com/dcos/dcos-cli) to
get all the info you would need in a JSON format.

e.g.,
1. set up ~/.dcos/config.toml for your cluster
2. DCOS_CONFIG=~/.dcos/config.toml dcos task --json --completed

You could process that output with jq (https://stedolan.github.io/jq/) and
do all of this in a short script.  (Not that I have much luck using jq, I'm
not very skilled at working with such arcane syntax, a la XPath.)

- Erik

On Wed, Aug 10, 2016 at 9:27 AM, June Taylor  wrote:

> David,
>
> Thanks for the suggestions, this has the missing piece we needed!
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Aug 10, 2016 at 10:54 AM, David Greenberg 
> wrote:
>
>> The Cook framework has examples of how to do this. See here (
>> https://github.com/twosigma/Cook/blob/master/scheduler/src/
>> cook/mesos/api.clj#L322-L324) for constructing the stem of the URL, here
>> (https://github.com/twosigma/Cook/blob/7a49fbb98b281e3b23779
>> cd88d1d2b73428a0447/scheduler/src/cook/mesos/api.clj#L281-L297) for
>> finding the filesystem path, and here (https://github.com/twosigma/C
>> ook/blob/master/scheduler/docs/scheduler-rest-api.asc#using_output_url)
>> for getting data with the stem.
>>
>> Essentially, you need to scrape the Mesos master to get all the path info
>> you need. LMK if you have questions!
>>
>> On Wed, Aug 10, 2016 at 9:34 AM June Taylor  wrote:
>>
>>> Tomek,
>>>
>>> I'm not sure I understand your suggestion. We know how to ask for a file
>>> from an HTTP endpoint, but it is the construction of the correct URL which
>>> is not currently clear.
>>>
>>> We are not sure how to determine the Run ID of the executor.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
>>> wrote:
>>>
 Hi

 If you need simplest method then python SimpleHTTPServer could help.
 Just launch it in background before command you want to run, assign it port
 and query sandbox with : that can be obtained from
 state endpoint.

 -
 Tomek

 śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:

> We are trying to retrieve the stdout and stderr files from an executor
> programmatically.
>
> It appears that these are available via HTTP request, however,
> constructing the correct URL is posing to be a challenge.
>
> Our scenario is:
>
> 1. Use mesos-execute to submit a job. A framework ID is available at
> this point.
> 2. Using the framework ID, one can inquire with mesos-state to
> determine which slave ID is executing the task.
> 3. Using the slave ID, one can inquire with mesos-state to find the
> hostname for that slave ID
> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
> there is an Executor ID which we cannot programmatically determine, to
> complete this URL.
>
> Please advise the simplest option for retrieving the sandbox files
> given the scenario starts with mesos-execute commands.
>
> Thanks!
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>

>>>
>

Re: Programmatically retrieve stdout/stderr from a node

[June Taylor]

David,

Thanks for the suggestions, this has the missing piece we needed!


Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota

On Wed, Aug 10, 2016 at 10:54 AM, David Greenberg 
wrote:

> The Cook framework has examples of how to do this. See here (
> https://github.com/twosigma/Cook/blob/master/scheduler/
> src/cook/mesos/api.clj#L322-L324) for constructing the stem of the URL,
> here (https://github.com/twosigma/Cook/blob/7a49fbb98b281e3b23779cd88d1d2b
> 73428a0447/scheduler/src/cook/mesos/api.clj#L281-L297) for finding the
> filesystem path, and here (https://github.com/twosigma/
> Cook/blob/master/scheduler/docs/scheduler-rest-api.asc#using_output_url)
> for getting data with the stem.
>
> Essentially, you need to scrape the Mesos master to get all the path info
> you need. LMK if you have questions!
>
> On Wed, Aug 10, 2016 at 9:34 AM June Taylor  wrote:
>
>> Tomek,
>>
>> I'm not sure I understand your suggestion. We know how to ask for a file
>> from an HTTP endpoint, but it is the construction of the correct URL which
>> is not currently clear.
>>
>> We are not sure how to determine the Run ID of the executor.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
>> wrote:
>>
>>> Hi
>>>
>>> If you need simplest method then python SimpleHTTPServer could help.
>>> Just launch it in background before command you want to run, assign it port
>>> and query sandbox with : that can be obtained from
>>> state endpoint.
>>>
>>> -
>>> Tomek
>>>
>>> śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:
>>>
 We are trying to retrieve the stdout and stderr files from an executor
 programmatically.

 It appears that these are available via HTTP request, however,
 constructing the correct URL is posing to be a challenge.

 Our scenario is:

 1. Use mesos-execute to submit a job. A framework ID is available at
 this point.
 2. Using the framework ID, one can inquire with mesos-state to
 determine which slave ID is executing the task.
 3. Using the slave ID, one can inquire with mesos-state to find the
 hostname for that slave ID
 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
 there is an Executor ID which we cannot programmatically determine, to
 complete this URL.

 Please advise the simplest option for retrieving the sandbox files
 given the scenario starts with mesos-execute commands.

 Thanks!
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

>>>
>>

Re: Programmatically retrieve stdout/stderr from a node

[David Greenberg]

The Cook framework has examples of how to do this. See here (
https://github.com/twosigma/Cook/blob/master/scheduler/src/cook/mesos/api.clj#L322-L324)
for constructing the stem of the URL, here (
https://github.com/twosigma/Cook/blob/7a49fbb98b281e3b23779cd88d1d2b73428a0447/scheduler/src/cook/mesos/api.clj#L281-L297)
for finding the filesystem path, and here (
https://github.com/twosigma/Cook/blob/master/scheduler/docs/scheduler-rest-api.asc#using_output_url)
for getting data with the stem.

Essentially, you need to scrape the Mesos master to get all the path info
you need. LMK if you have questions!

On Wed, Aug 10, 2016 at 9:34 AM June Taylor  wrote:

> Tomek,
>
> I'm not sure I understand your suggestion. We know how to ask for a file
> from an HTTP endpoint, but it is the construction of the correct URL which
> is not currently clear.
>
> We are not sure how to determine the Run ID of the executor.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
> wrote:
>
>> Hi
>>
>> If you need simplest method then python SimpleHTTPServer could help. Just
>> launch it in background before command you want to run, assign it port and
>> query sandbox with : that can be obtained from state
>> endpoint.
>>
>> -
>> Tomek
>>
>> śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:
>>
>>> We are trying to retrieve the stdout and stderr files from an executor
>>> programmatically.
>>>
>>> It appears that these are available via HTTP request, however,
>>> constructing the correct URL is posing to be a challenge.
>>>
>>> Our scenario is:
>>>
>>> 1. Use mesos-execute to submit a job. A framework ID is available at
>>> this point.
>>> 2. Using the framework ID, one can inquire with mesos-state to determine
>>> which slave ID is executing the task.
>>> 3. Using the slave ID, one can inquire with mesos-state to find the
>>> hostname for that slave ID
>>> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
>>> there is an Executor ID which we cannot programmatically determine, to
>>> complete this URL.
>>>
>>> Please advise the simplest option for retrieving the sandbox files given
>>> the scenario starts with mesos-execute commands.
>>>
>>> Thanks!
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>

Re: Programmatically retrieve stdout/stderr from a node

[Tomek Janiszewski]

Oh, now I got it. To get path you need to query agent for it's state
with /slave(1)/state. In response you will get JSON. Path to sandbox is
stored in .frameworks[].executors[].directory.

śr., 10.08.2016 o 17:34 użytkownik June Taylor  napisał:

> Tomek,
>
> I'm not sure I understand your suggestion. We know how to ask for a file
> from an HTTP endpoint, but it is the construction of the correct URL which
> is not currently clear.
>
> We are not sure how to determine the Run ID of the executor.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Aug 10, 2016 at 10:08 AM, Tomek Janiszewski 
> wrote:
>
>> Hi
>>
>> If you need simplest method then python SimpleHTTPServer could help. Just
>> launch it in background before command you want to run, assign it port and
>> query sandbox with : that can be obtained from state
>> endpoint.
>>
>> -
>> Tomek
>>
>> śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:
>>
>>> We are trying to retrieve the stdout and stderr files from an executor
>>> programmatically.
>>>
>>> It appears that these are available via HTTP request, however,
>>> constructing the correct URL is posing to be a challenge.
>>>
>>> Our scenario is:
>>>
>>> 1. Use mesos-execute to submit a job. A framework ID is available at
>>> this point.
>>> 2. Using the framework ID, one can inquire with mesos-state to determine
>>> which slave ID is executing the task.
>>> 3. Using the slave ID, one can inquire with mesos-state to find the
>>> hostname for that slave ID
>>> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
>>> there is an Executor ID which we cannot programmatically determine, to
>>> complete this URL.
>>>
>>> Please advise the simplest option for retrieving the sandbox files given
>>> the scenario starts with mesos-execute commands.
>>>
>>> Thanks!
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>

Re: Programmatically retrieve stdout/stderr from a node

[Tomek Janiszewski]

Hi

If you need simplest method then python SimpleHTTPServer could help. Just
launch it in background before command you want to run, assign it port and
query sandbox with : that can be obtained from state
endpoint.

-
Tomek

śr., 10.08.2016 o 16:53 użytkownik June Taylor  napisał:

> We are trying to retrieve the stdout and stderr files from an executor
> programmatically.
>
> It appears that these are available via HTTP request, however,
> constructing the correct URL is posing to be a challenge.
>
> Our scenario is:
>
> 1. Use mesos-execute to submit a job. A framework ID is available at this
> point.
> 2. Using the framework ID, one can inquire with mesos-state to determine
> which slave ID is executing the task.
> 3. Using the slave ID, one can inquire with mesos-state to find the
> hostname for that slave ID
> 4. HTTP can be used to ask the /browse/ endpoint for a file, however,
> there is an Executor ID which we cannot programmatically determine, to
> complete this URL.
>
> Please advise the simplest option for retrieving the sandbox files given
> the scenario starts with mesos-execute commands.
>
> Thanks!
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>

Programmatically retrieve stdout/stderr from a node

[June Taylor]

We are trying to retrieve the stdout and stderr files from an executor
programmatically.

It appears that these are available via HTTP request, however, constructing
the correct URL is posing to be a challenge.

Our scenario is:

1. Use mesos-execute to submit a job. A framework ID is available at this
point.
2. Using the framework ID, one can inquire with mesos-state to determine
which slave ID is executing the task.
3. Using the slave ID, one can inquire with mesos-state to find the
hostname for that slave ID
4. HTTP can be used to ask the /browse/ endpoint for a file, however, there
is an Executor ID which we cannot programmatically determine, to complete
this URL.

Please advise the simplest option for retrieving the sandbox files given
the scenario starts with mesos-execute commands.

Thanks!
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota