[jira] [Commented] (MESOS-6804) Running 'tty' inside a debug container that has a tty reports "Not a tty"
[ https://issues.apache.org/jira/browse/MESOS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710510#comment-16710510 ] Kevin Klues commented on MESOS-6804: Here are some relevant tickets to help resolve this as other systems have run into this issue in the past: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1669578 https://github.com/opencontainers/runc/issues/814 https://github.com/lxc/lxd/issues/1724 https://github.com/moby/moby/issues/8755 https://github.com/opencontainers/runc/blob/v0.1.1/libcontainer/standard_init_linux.go#L58-L83 https://github.com/opencontainers/runc/blob/5d93fed3d27f1e2bab58bad13b180a7a81d0b378/libcontainer/standard_init_linux.go https://github.com/moby/moby/pull/33007 > Running 'tty' inside a debug container that has a tty reports "Not a tty" > - > > Key: MESOS-6804 > URL: https://issues.apache.org/jira/browse/MESOS-6804 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Priority: Major > Labels: debugging, mesosphere > > We need to inject `/dev/console` into the container and map it to the slave > end of the TTY we are attached to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9399) Update 'mesos task list' to only list running tasks
[
https://issues.apache.org/jira/browse/MESOS-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698954#comment-16698954
]
Kevin Klues commented on MESOS-9399:
{noformat}
commit 18e51b86fac848330dea640d7b7b7bf2d6584fe5
Author: Armand Grillet
Date: Mon Nov 26 08:47:25 2018 -0500
Replaced CLI test helper function 'running_tasks' by 'wait_for_task'.
Replaces 'running_tasks(master)', a function that was not generic nor
explicit, by 'wait_for_task(master, name, state, delay)'. This helper
function waits a 'delay' for a task with a given 'name' to be in a
certain 'state'.
All uses of 'running_tasks' have been replaced by the new function.
Review: https://reviews.apache.org/r/69426/
{noformat}
> Update 'mesos task list' to only list running tasks
> ---
>
> Key: MESOS-9399
> URL: https://issues.apache.org/jira/browse/MESOS-9399
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Doing a {{mesos task list}} currently returns all tasks that have ever been
> run (not just running tasks). The default behavior should be to return only
> the running tasks and offer an option to return all of them. To tell them
> apart, there should be a state field in the table returned by this command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9399) Update 'mesos task list' to only list running tasks
[
https://issues.apache.org/jira/browse/MESOS-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698924#comment-16698924
]
Kevin Klues commented on MESOS-9399:
{noformat}
commit 48cdd101c7a9730029471b8f881df46e136bfae4
Author: Armand Grillet
Date: Mon Nov 26 08:22:58 2018 -0500
Fixed name of task created when running mesos-cli-tests.
Review: https://reviews.apache.org/r/69425/
{noformat}
> Update 'mesos task list' to only list running tasks
> ---
>
> Key: MESOS-9399
> URL: https://issues.apache.org/jira/browse/MESOS-9399
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Doing a {{mesos task list}} currently returns all tasks that have ever been
> run (not just running tasks). The default behavior should be to return only
> the running tasks and offer an option to return all of them. To tell them
> apart, there should be a state field in the table returned by this command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9399) Update 'mesos task list' to only list running tasks
[
https://issues.apache.org/jira/browse/MESOS-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698909#comment-16698909
]
Kevin Klues commented on MESOS-9399:
{noformat}
commit 2b03f942b5cf9375a75f08b36091c3b3e7f096ff
Author: Armand Grillet
Date: Mon Nov 26 08:14:42 2018 -0500
Updated 'mesos task list' to only display running tasks.
Review: https://reviews.apache.org/r/69394/
{noformat}
> Update 'mesos task list' to only list running tasks
> ---
>
> Key: MESOS-9399
> URL: https://issues.apache.org/jira/browse/MESOS-9399
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Doing a {{mesos task list}} currently returns all tasks that have ever been
> run (not just running tasks). The default behavior should be to return only
> the running tasks and offer an option to return all of them. To tell them
> apart, there should be a state field in the table returned by this command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9399) Update 'mesos task list' to only list running tasks
[
https://issues.apache.org/jira/browse/MESOS-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696086#comment-16696086
]
Kevin Klues commented on MESOS-9399:
{noformat}
commit fcb8e2abeef52042a7f40312c461cab716c0bf3c
Author: Armand Grillet
Date: Thu Nov 22 11:17:43 2018 -0500
Updated 'mesos task list' to display a 'State' field.
Review: https://reviews.apache.org/r/69393/
{noformat}
> Update 'mesos task list' to only list running tasks
> ---
>
> Key: MESOS-9399
> URL: https://issues.apache.org/jira/browse/MESOS-9399
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Doing a {{mesos task list}} currently returns all tasks that have ever been
> run (not just running tasks). The default behavior should be to return only
> the running tasks and offer an option to return all of them. To tell them
> apart, there should be a state field in the table returned by this command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9333) Document usage and build of new Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696032#comment-16696032
]
Kevin Klues commented on MESOS-9333:
{noformat}
commit c132007309d2291537f6dc0ac6424092f47532db
Author: Armand Grillet
Date: Thu Nov 22 10:33:28 2018 -0500
Added docs describing how to use the new CLI.
The documentation describes the main commands of the new CLI, how to
activate it, how to build Mesos including this component, and how to
write a configuration file for it.
Review: https://reviews.apache.org/r/69390/
{noformat}
> Document usage and build of new Mesos CLI
> -
>
> Key: MESOS-9333
> URL: https://issues.apache.org/jira/browse/MESOS-9333
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Stating how to compile and use the Mesos CLI + its limitations (only Mesos
> containerizer, exec DEBUG follows task-user).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9333) Document usage and build of new Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691948#comment-16691948
]
Kevin Klues commented on MESOS-9333:
{noformat}
commit e7bd571e5c8bb3a3b91f589b9a03d07320070e3e
Author: Armand Grillet
Date: Mon Nov 19 10:49:28 2018 -0500
Added configuration docs describing how to use Python 3.
For Autotools, this means how to use 'PYTHON_3' and 'PYTHON_3_VERSION'.
For CMake, this means how to use '-DPYTHON_3'.
Review: https://reviews.apache.org/r/69380/
{noformat}
> Document usage and build of new Mesos CLI
> -
>
> Key: MESOS-9333
> URL: https://issues.apache.org/jira/browse/MESOS-9333
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Stating how to compile and use the Mesos CLI + its limitations (only Mesos
> containerizer, exec DEBUG follows task-user).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9333) Document usage and build of new Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691949#comment-16691949
]
Kevin Klues commented on MESOS-9333:
{noformat}
commit 2a899351641c8defb7d677bad25fe2b479d1f0b6
Author: Armand Grillet
Date: Mon Nov 19 10:50:47 2018 -0500
Updated configuration docs describing how to build the new CLI.
Review: https://reviews.apache.org/r/69381/
{noformat}
> Document usage and build of new Mesos CLI
> -
>
> Key: MESOS-9333
> URL: https://issues.apache.org/jira/browse/MESOS-9333
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Stating how to compile and use the Mesos CLI + its limitations (only Mesos
> containerizer, exec DEBUG follows task-user).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9396) Use the built CLI binary when running new CLI integration tests in CI
[
https://issues.apache.org/jira/browse/MESOS-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691943#comment-16691943
]
Kevin Klues commented on MESOS-9396:
{noformat}
commit 12bf198d8a6f0c651d5560defb9529fda4b8d212
Author: Armand Grillet
Date: Mon Nov 19 11:30:43 2018 -0500
Updated PyInstaller requirement for new CLI to support Python 3.7.
Review: https://reviews.apache.org/r/69255/
{noformat}
> Use the built CLI binary when running new CLI integration tests in CI
> -
>
> Key: MESOS-9396
> URL: https://issues.apache.org/jira/browse/MESOS-9396
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Labels: autotools, cmake
> Fix For: 1.8.0
>
>
> We currently use the CLI in the virtual environment which is just a Python
> file being interpreted, we should instead use the binary built by PyInstaller
> as it is what is gonna be used in production.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9396) Use the built CLI binary when running new CLI integration tests in CI
[
https://issues.apache.org/jira/browse/MESOS-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691942#comment-16691942
]
Kevin Klues commented on MESOS-9396:
{noformat}
commit 992b506e19829536b9407f04f4647af6642d9bec
Author: Armand Grillet agril...@mesosphere.io
Date: Mon Nov 19 10:41:13 2018 -0500
Updated new CLI test step to use binary created by PyInstaller.
The integration tests for the new CLI running while building Mesos now
directly use the binary created during the build. That way we make sure
that the binary created using PyInstaller is usable, which is the
artifact that we want to distribute to users in the future.
Previously, we were only activating the virtual environment to run the
tests thus the binary created by PyInstaller was never properly tested.
To use the binary created by PyInstaller, we simply update the PATH
before running 'mesos-cli-tests'.
Review: https://reviews.apache.org/r/69374/
{noformat}
> Use the built CLI binary when running new CLI integration tests in CI
> -
>
> Key: MESOS-9396
> URL: https://issues.apache.org/jira/browse/MESOS-9396
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Labels: autotools, cmake
>
> We currently use the CLI in the virtual environment which is just a Python
> file being interpreted, we should instead use the binary built by PyInstaller
> as it is what is gonna be used in production.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9363) Improve task exec to return correct exit status
[
https://issues.apache.org/jira/browse/MESOS-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673841#comment-16673841
]
Kevin Klues commented on MESOS-9363:
{noformat}
commit 25beea12f9f12143e6df7b0ad2d272d4116c217c
Author: Kevin Klues
Date: Fri Nov 2 19:55:44 2018 -0400
Updated new CLI task attach/exec exit strategy.
This code was pulled directly from:
https://github.com/dcos/dcos-core-cli/blob/
9d10e9d6fb2b16e46b58d67b7e9d79b2505f3451/
python/lib/dcos/dcos/mesos.py
Review: https://reviews.apache.org/r/69208/
{noformat}
> Improve task exec to return correct exit status
> ---
>
> Key: MESOS-9363
> URL: https://issues.apache.org/jira/browse/MESOS-9363
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Whatever the exit, {{mesos task exec}} always returns 0. We need to fix that
> to return the correct status code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9363) Improve task exec to return correct exit status
[
https://issues.apache.org/jira/browse/MESOS-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673823#comment-16673823
]
Kevin Klues commented on MESOS-9363:
{noformat}
commit 992e3c4efd2f607b60f5e4b5bea7999692a01c0a
Author: Armand Grillet
Date: Fri Nov 2 19:42:59 2018 -0400
Updated new CLI to propagate commands exit status properly.
In a later commit, this will be used by the two subcommands 'task
attach' and 'task exec' to return their proper exit statuses.
Review: https://reviews.apache.org/r/69206/
{noformat}
> Improve task exec to return correct exit status
> ---
>
> Key: MESOS-9363
> URL: https://issues.apache.org/jira/browse/MESOS-9363
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> Whatever the exit, {{mesos task exec}} always returns 0. We need to fix that
> to return the correct status code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9343) Add test(s) for `mesos task attach` on task launched with a TTY
[
https://issues.apache.org/jira/browse/MESOS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673820#comment-16673820
]
Kevin Klues commented on MESOS-9343:
{noformat}
commit 9441e48338a8c58adc1e88c9fc1804d10f201262
Author: Armand Grillet
Date: Fri Nov 2 19:39:06 2018 -0400
Simplified newline handling in 'test_exec()' test for new CLI.
The test was previously using '.strip()' to compare the command stdout
and the result we expect. This check was incorrect because it could
happen that the output ended in a bunch of extra whitespace that we
would then strip off unknowningly. By replacing the task command to use
'printf' instead of 'echo' (which artifically inserts an extra newline
in the output), we are able to simplify this assertion and make sure the
output is exactly the same as what we expect.
Review: https://reviews.apache.org/r/69237/
{noformat}
> Add test(s) for `mesos task attach` on task launched with a TTY
>
>
> Key: MESOS-9343
> URL: https://issues.apache.org/jira/browse/MESOS-9343
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9343) Add test(s) for `mesos task attach` on task launched with a TTY
[
https://issues.apache.org/jira/browse/MESOS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673818#comment-16673818
]
Kevin Klues commented on MESOS-9343:
{noformat}
commit 6b7b7e6f68ef891febcce2a38077a847288c1c10
Author: Armand Grillet
Date: Fri Nov 2 19:17:56 2018 -0400
Refactored 'running_tasks()' call for new CLI tests.
Review: https://reviews.apache.org/r/69207/
{noformat}
> Add test(s) for `mesos task attach` on task launched with a TTY
>
>
> Key: MESOS-9343
> URL: https://issues.apache.org/jira/browse/MESOS-9343
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9343) Add test(s) for `mesos task attach` on task launched with a TTY
[
https://issues.apache.org/jira/browse/MESOS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673806#comment-16673806
]
Kevin Klues commented on MESOS-9343:
{noformat}
commit 6d0cbda19ad8a5453c960e37424a38c4be1924a9
Author: Armand Grillet
Date: Fri Nov 2 19:08:24 2018 -0400
Added 'popen_tty()' to test util functions for the new CLI.
This code was pulled directly from:
https://github.com/dcos/dcos-core-cli/blob/
7fd55421939a7782c237e2b8719c0fe2f543acd7/
python/lib/dcoscli/dcoscli/test/common.py
This function will be used by tests requiring a TTY. This will be the
case for tests concerning the 'task attach' subcommand.
Review: https://reviews.apache.org/r/69116/
{noformat}
> Add test(s) for `mesos task attach` on task launched with a TTY
>
>
> Key: MESOS-9343
> URL: https://issues.apache.org/jira/browse/MESOS-9343
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9341) Add non-interactive test(s) for `mesos task exec`
[
https://issues.apache.org/jira/browse/MESOS-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672298#comment-16672298
]
Kevin Klues commented on MESOS-9341:
{noformat}
commit 1f574f72486e3adf789ffcf2c8624a3775071979
Author: Armand Grillet
Date: Wed Oct 31 19:07:17 2018 -0400
Added test for interactive 'task exec'.
Review: https://reviews.apache.org/r/69115/
{noformat}
> Add non-interactive test(s) for `mesos task exec`
> -
>
> Key: MESOS-9341
> URL: https://issues.apache.org/jira/browse/MESOS-9341
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9350) CLI build step is broken with CMake due to missing file.
[
https://issues.apache.org/jira/browse/MESOS-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670877#comment-16670877
]
Kevin Klues commented on MESOS-9350:
{noformat}
commit d8062f231b9f27889b7cae7a42eef49e4eed79ec
Author: Armand Grillet
Date: Wed Oct 31 18:43:44 2018 -0400
Updated 'CLI_FILES' in 'cli_new/CmakeLists.txt'.
Review: https://reviews.apache.org/r/69138/
{noformat}
> CLI build step is broken with CMake due to missing file.
>
>
> Key: MESOS-9350
> URL: https://issues.apache.org/jira/browse/MESOS-9350
> Project: Mesos
> Issue Type: Bug
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.8.0
>
>
> The file {{mesos.py}} was not added to {{CLI_FILES}} and this is now an issue
> when building the CLI using CMake.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8978) Command executor calling setsid breaks the tty support.
[
https://issues.apache.org/jira/browse/MESOS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665773#comment-16665773
]
Kevin Klues commented on MESOS-8978:
{noformat}
commit 4fa080a7eefd47697b9fab934d19c73f092c78f8 (HEAD -> master, origin/master)
Author: Kevin Klues
Date: Fri Oct 26 16:49:32 2018 -0700
Fixed bug in 'execute.cpp' with tty-based tasks and no 'containerInfo'.
Previously, we could only launch tasks using the '--tty' flag if they
had a backing docker image (or some other combination of other flags set
that would cause the task to have a 'containerInfo' created for it).
This commit makes sure that if '--tty' is passed, that a containerInfo
gets created and its TTYInfo field gets populated.
Review: https://reviews.apache.org/r/69103/
{noformat}
> Command executor calling setsid breaks the tty support.
> ---
>
> Key: MESOS-8978
> URL: https://issues.apache.org/jira/browse/MESOS-8978
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.4.1, 1.5.1, 1.6.0, 1.7.0
>Reporter: Jie Yu
>Assignee: Kevin Klues
>Priority: Major
> Fix For: 1.5.2, 1.6.2, 1.7.1, 1.8.0
>
>
> I was playing with
> [msh|https://github.com/mesos/mesos-go/blob/master/api/v1/cmd/msh/msh.go]
> (one example from [mesos-go|https://github.com/mesos/mesos-go]), which allows
> you to launch an interactive shell in the Mesos cluster. It works by
> launching a container with tty enabled, and then [attach to the container
> input|https://github.com/apache/mesos/blob/master/include/mesos/v1/agent/agent.proto#L191-L201]
> using the agent operator API.
> However, I got the following error when doing the following:
> {noformat}
> Jies-MacBook-Pro:mesos-go jie$ ./msh -master 127.0.0.1:5050 -tty -interactive
> -- /bin/sh -i
> ...
> 2018/06/05 11:51:35 original window size is 156 x 45
> sh: cannot set terminal process group (-1): Inappropriate ioctl for device
> sh: no job control in this shell
> {noformat}
> If I use `-pod`, the problem goes away. This only happens if command executor
> is used.
> A few research suggested that this issue is related to `setsid` (see this
> [thread|https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216]).
> Looks like we did an extra
> `[setsid|https://github.com/apache/mesos/blob/1.6.x/src/launcher/executor.cpp#L512]`
> in the command executor.
> The setsid() system call to create a new process group detaches the spawned
> process from a controlling tty. Therefore programs like bash complain, that
> they can't use job control. Re-attaching the controlling tty won't work,
> because the tty is still in use as a controlling tty for the command executor
> process.
> There are two possible solutions:
> 1) Get rid of setsid in command executor
> 2) Detach and re-attach the controlling TTY as suggested in
> https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9341) Add non-interactive test(s) for `mesos task exec`
[
https://issues.apache.org/jira/browse/MESOS-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659030#comment-16659030
]
Kevin Klues commented on MESOS-9341:
{noformat}
commit 06e5a3abe0f6241a06ca89ac6071b58556a9f07b
Author: Armand Grillet
Date: Mon Oct 22 10:18:29 2018 -0400
Added non-interactive test for 'task exec'.
Review: https://reviews.apache.org/r/69049/
{noformat}
> Add non-interactive test(s) for `mesos task exec`
> -
>
> Key: MESOS-9341
> URL: https://issues.apache.org/jira/browse/MESOS-9341
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9341) Add non-interactive test(s) for `mesos task exec`
[
https://issues.apache.org/jira/browse/MESOS-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659027#comment-16659027
]
Kevin Klues commented on MESOS-9341:
{noformat}
commit 4624823deb37df5469e1d1e548945985b59a8a73
Author: Armand Grillet
Date: Mon Oct 22 10:13:33 2018 -0400
Added new CLI constants 'TEST_DIRECTORY' and 'TEST_DATA_DIRECTORY'.
Review: https://reviews.apache.org/r/69119/
{noformat}
> Add non-interactive test(s) for `mesos task exec`
> -
>
> Key: MESOS-9341
> URL: https://issues.apache.org/jira/browse/MESOS-9341
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9341) Add non-interactive test(s) for `mesos task exec`
[
https://issues.apache.org/jira/browse/MESOS-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658986#comment-16658986
]
Kevin Klues commented on MESOS-9341:
{noformat}
commit aed4a743daab3c9e5aacb3e14c50d604ba6fd051
Author: Armand Grillet
Date: Mon Oct 22 09:33:02 2018 -0400
Added tenacity to 'pip-requirements' for new CLI.
This requirement will be used in upcoming new CLI integration tests.
Review: https://reviews.apache.org/r/69048/
{noformat}
> Add non-interactive test(s) for `mesos task exec`
> -
>
> Key: MESOS-9341
> URL: https://issues.apache.org/jira/browse/MESOS-9341
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9341) Add non-interactive test(s) for `mesos task exec`
[
https://issues.apache.org/jira/browse/MESOS-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658980#comment-16658980
]
Kevin Klues commented on MESOS-9341:
{noformat}
commit 963de3b1811ef569449102192d40ca2cbed73b3c
Author: Armand Grillet
Date: Mon Oct 22 09:28:28 2018 -0400
Added 'exec_command' to test util functions for the new CLI.
This code was mostly pulled directly from:
https://github.com/dcos/dcos-core-cli/blob/
7fd55421939a7782c237e2b8719c0fe2f543acd7/
python/lib/dcoscli/dcoscli/test/common.py
This function will be used by tests that do not return a specific output
but an error code, stdout, and stderr. This will be the case for tests
concerning the 'task exec' and 'task attach' subcommands.
Review: https://reviews.apache.org/r/69114/
{noformat}
> Add non-interactive test(s) for `mesos task exec`
> -
>
> Key: MESOS-9341
> URL: https://issues.apache.org/jira/browse/MESOS-9341
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> As a source, we could use the tests in
> https://github.com/dcos/dcos-core-cli/blob/b930d2004dceb47090004ab658f35cb608bc70e4/python/lib/dcoscli/tests/integrations/test_task.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657824#comment-16657824
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit 2d563794bc82a2cc799cdf2171fc47742fb573d1
Author: Armand Grillet armand.gril...@outlook.com
Date: Fri Oct 19 12:47:59 2018 +0200
Added 'task attach' subcommand to new CLI.
Review: https://reviews.apache.org/r/69105/
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655515#comment-16655515
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit 7f36ebc1775398a43b2aa3a81bb647fb296b8313
Author: Armand Grillet
Date: Thu Oct 11 15:12:32 2018 +0200
Added 'task exec' subcommand to new CLI.
This subcommand launches a process inside a Mesos task container.
Review: https://reviews.apache.org/r/69003
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655499#comment-16655499
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit cfa60e41e53ee1f5a647a7ecd26d0fedd24bc392
Author: Armand Grillet armand.gril...@outlook.com
Date: Wed Oct 10 06:35:13 2018 +0200
Added TaskIO to new CLI.
This code was pulled directly from:
https://github.com/dcos/dcos-core-cli/blob/
7fd55421939a7782c237e2b8719c0fe2f543acd7/
python/lib/dcos/dcos/mesos.py
It will be used by the new CLI for commands such as 'task exec'.
Changes have been made to work with our HTTP Python library and
make the code pass when linted by Pylint.
Review: https://reviews.apache.org/r/68978
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653431#comment-16653431
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit 5ad9bcc99783eca6c61c7d0851b24a391d417134
Author: Armand Grillet
Date: Mon Oct 15 13:00:54 2018 +0200
Added `retry` argument to `request()` method in Python library.
This allows us to do HTTP requests without using `tenacity.retry()`. We
need to be able to disable these retries to stream data for a long
period of time, such as when using the upcoming `mesos task exec`.
Review: https://reviews.apache.org/r/69023
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653394#comment-16653394
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit e9dc09516f2becad5e746c2410997ba9658eca70
Author: Armand Grillet
Date: Wed Oct 17 07:07:59 2018 -0400
Added `get_container_id` to util functions for the new CLI.
In a future commit, this function will be used by the `TaskIO` object to
know which container to use in order to run `task attach` and `task
exec.`.
Review: https://reviews.apache.org/r/69007/
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6551) Add attach/exec commands to the Mesos CLI
[
https://issues.apache.org/jira/browse/MESOS-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651616#comment-16651616
]
Kevin Klues commented on MESOS-6551:
{noformat}
commit e5532167c91ebabaa3add336a787f8192af59ec3
Author: Armand Grillet
Date: Tue Oct 16 08:26:51 2018 -0400
Added Record-IO encoder and decoder to Python library.
This code was pulled directly from:
https://github.com/dcos/dcos-core-cli/blob/
7fd55421939a7782c237e2b8719c0fe2f543acd7/
python/lib/dcos/dcos/recordio.py
https://github.com/dcos/dcos-core-cli/blob/
7fd55421939a7782c237e2b8719c0fe2f543acd7/
python/lib/dcos/tests/test_recordio.py
It will be used by the new CLI for commands such as `task exec`.
Review: https://reviews.apache.org/r/68977/
{noformat}
> Add attach/exec commands to the Mesos CLI
> -
>
> Key: MESOS-6551
> URL: https://issues.apache.org/jira/browse/MESOS-6551
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Labels: debugging, mesosphere
>
> After all of this support has landed, we need to update the Mesos CLI to
> implement {{attach}} and {{exec}} functionality as outlined in the Design Doc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8795) Catch up new CLI features to be the same as the old one.
[
https://issues.apache.org/jira/browse/MESOS-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651574#comment-16651574
]
Kevin Klues commented on MESOS-8795:
{noformat}
commit c6d7d5593a9792411a876a97303989fc3db7441b
Author: Armand Grillet agril...@mesosphere.io
Date: Tue Oct 16 07:35:02 2018 -0400
Updated Python library to properly import submodules.
The primary change was to add imports in '__init__.py' so that when
doing an 'import mesos' from some code using this module, we are
able to access submodules (e.g. 'mesos.http') as part of this.
In the process of making this change, we ran into the error pointed out
in variant 6 at the following link, for how to properly set the version
of a python module both in the code itself, and in its package
definition via setup.py.
https://packaging.python.org/guides/single-sourcing-package-version/
We decided to change to method 1 from this link to address this problem.
Review: https://reviews.apache.org/r/69047/
{noformat}
> Catch up new CLI features to be the same as the old one.
>
>
> Key: MESOS-8795
> URL: https://issues.apache.org/jira/browse/MESOS-8795
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> https://github.com/apache/mesos/tree/master/src/cli
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8795) Catch up new CLI features to be the same as the old one.
[
https://issues.apache.org/jira/browse/MESOS-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650101#comment-16650101
]
Kevin Klues commented on MESOS-8795:
{noformat}
commit c2ceb8d83447ad00a62e77d978d63d69b4da6567
Author: Armand Grillet
Date: Mon Oct 15 07:37:36 2018 -0400
Removed variable printing in error strings in new CLI.
By convention, the caller should print variables passed in as arguments
to other functions. In these cases the callee was printing them,
resulting in the variables being printed twice, redundantly.
Review: https://reviews.apache.org/r/68965/
{noformat}
{noformat}
commit 3b8be5a17ad9ce9e981ddc2c0789c37af1700de0
Author: Armand Grillet
Date: Mon Oct 15 07:36:27 2018 -0400
Added try/catch statements when using Mesos util functions in new CLI.
Review: https://reviews.apache.org/r/68965/
{noformat}
> Catch up new CLI features to be the same as the old one.
>
>
> Key: MESOS-8795
> URL: https://issues.apache.org/jira/browse/MESOS-8795
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> https://github.com/apache/mesos/tree/master/src/cli
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8025) Update the master field in the new CLI config to accept a URL instead of an
[
https://issues.apache.org/jira/browse/MESOS-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643379#comment-16643379
]
Kevin Klues commented on MESOS-8025:
{noformat}
commit d39b1e94fa1c7db09a246f1ac8b74929c577e071
Author: Armand Grillet
Date: Tue Oct 9 08:53:45 2018 -0400
Replaced 'verify_address_format' in new CLI with 'sanitize_address'.
In the process we cleaned up alot of the logic to make sure that the
addresses that are passed in are of the format we expect and that what
comes out is a sanitized address in canonical form.
As part of this change, we updated a number of call sites to use this
new function.
Review: https://reviews.apache.org/r/66683/
{noformat}
> Update the master field in the new CLI config to accept a URL instead of an
>
> -
>
> Key: MESOS-8025
> URL: https://issues.apache.org/jira/browse/MESOS-8025
> Project: Mesos
> Issue Type: Improvement
> Components: cli
> Environment: This will be useful in cases where the master is behind
> a proxy or when the master is sitting directly on port 80.
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8795) Catch up new CLI features to be the same as the old one.
[
https://issues.apache.org/jira/browse/MESOS-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641847#comment-16641847
]
Kevin Klues commented on MESOS-8795:
{noformat}
commit aa98a4918eab5c46383d352aee16e0b8ed4e2b13
Author: Armand Grillet
Date: Mon Oct 8 09:33:07 2018 -0400
Moved `get_agent_address` from `util.py` to `mesos.py` in new CLI.
Review: https://reviews.apache.org/r/68950/
{noformat}
> Catch up new CLI features to be the same as the old one.
>
>
> Key: MESOS-8795
> URL: https://issues.apache.org/jira/browse/MESOS-8795
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> https://github.com/apache/mesos/tree/master/src/cli
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8795) Catch up new CLI features to be the same as the old one.
[
https://issues.apache.org/jira/browse/MESOS-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641833#comment-16641833
]
Kevin Klues commented on MESOS-8795:
{noformat}
commit fee4803635776b8401a6deea3fd9b3d4133d30bc
Author: Armand Grillet
Date: Mon Oct 8 09:29:16 2018 -0400
Removed unused `lib/cli/tasks.py` for new CLI.
This file was introduced accidentally as part of
051a138d08ba3b9e28fd6ec4e4f707cbd4bb1563
Review: https://reviews.apache.org/r/68949/
{noformat}
> Catch up new CLI features to be the same as the old one.
>
>
> Key: MESOS-8795
> URL: https://issues.apache.org/jira/browse/MESOS-8795
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> https://github.com/apache/mesos/tree/master/src/cli
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8795) Catch up new CLI features to be the same as the old one.
[
https://issues.apache.org/jira/browse/MESOS-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635584#comment-16635584
]
Kevin Klues commented on MESOS-8795:
{noformat}
commit 051a138d08ba3b9e28fd6ec4e4f707cbd4bb1563
Author: Armand Grillet
Date: Tue Oct 2 10:09:54 2018 -0400
Refactored new CLI.
This adds a new file called `mesos.py` to the CLI library that abstracts
calling various Mesos APIs. The functions added here in this commit are
then used in the 'task' and 'agent' plugins to simplify them.
Review: https://reviews.apache.org/r/68734/
{noformat}
> Catch up new CLI features to be the same as the old one.
>
>
> Key: MESOS-8795
> URL: https://issues.apache.org/jira/browse/MESOS-8795
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> https://github.com/apache/mesos/tree/master/src/cli
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8978) Command executor calling setsid breaks the tty support.
[
https://issues.apache.org/jira/browse/MESOS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635220#comment-16635220
]
Kevin Klues commented on MESOS-8978:
{noformat}
commit 716bb9f65974e4479d9ad794c145aa8293cea203
Author: Kevin Klues klue...@gmail.com
Date: Tue Oct 2 11:40:52 2018 +0200
Added the ability to launch tasks with a TTY attached to mesos-execute.
Review: https://reviews.apache.org/r/68724/
{noformat}
{noformat}
commit 12b6ab6bdabfc37f73439ea55ef0a7f0ee680983
Author: Kevin Klues klue...@gmail.com
Date: Tue Oct 2 11:41:12 2018 +0200
Removed call to 'setsid()' in command executor if TTY attached.
Previously, we would unconditionally call 'setsid()' in the command
executor whenever we launched a process. This causes the process it
launches to start a new session and NOT inherit the controlling TTY
from it's parent. This obviously causes problems, if the goal of
attaching a TTY to a task is so that we can actually give it control
of that TTY while it is executing.
Interestingly, even if process does not control its TTY, it can still
read and write from the file descriptors associated with it (it just
can't receive any signals associated with it, such as SIGWINCH,
SIGHUP, etc.). This is why things "appeared" to mostly work until this
point because stdin/stdout/stderr were all being redirected properly
to it.
Where we saw problems was with trying to 'attach' to an already
running process launched via the command executor using the
ATTACH_NESTED_CONTAINER_INPUT/OUTPUT calls. If you ran something like
'bash', you would not be able to do job control, and you could not
resize the screen properly when running things like 'vim'.
This commit fixes this issue by checking to see if a TTY has been
attached to a task launched by the command executor, and skipping the
'setsid()' call when forking it. We considered a number of alternative
approaches, but finally settled on this one since it was the least
invasive. I.e. only tasks launched with a TTY (which is a failry new
concept in Mesos) will be affected by this change; the semantics of
all traditional tasks launched via the command executor will go
unchanged.
Note: this problem does not exists for the default executor because
the agent does the job of launching all containers, and there is no
executor sitting between the containerizer and the actual process of a
task intervening to call an extra 'setsid()'. This is why containers
launched via LAUNCH_NESTED_CONTAINER_SESSION have always worked as
expected.
Review: https://reviews.apache.org/r/68810/
{noformat}
> Command executor calling setsid breaks the tty support.
> ---
>
> Key: MESOS-8978
> URL: https://issues.apache.org/jira/browse/MESOS-8978
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.4.1, 1.5.1, 1.6.0
>Reporter: Jie Yu
>Assignee: Kevin Klues
>Priority: Major
> Fix For: 1.5.2, 1.6.2, 1.7.1, 1.8.0
>
>
> I was playing with
> [msh|https://github.com/mesos/mesos-go/blob/master/api/v1/cmd/msh/msh.go]
> (one example from [mesos-go|https://github.com/mesos/mesos-go]), which allows
> you to launch an interactive shell in the Mesos cluster. It works by
> launching a container with tty enabled, and then [attach to the container
> input|https://github.com/apache/mesos/blob/master/include/mesos/v1/agent/agent.proto#L191-L201]
> using the agent operator API.
> However, I got the following error when doing the following:
> {noformat}
> Jies-MacBook-Pro:mesos-go jie$ ./msh -master 127.0.0.1:5050 -tty -interactive
> -- /bin/sh -i
> ...
> 2018/06/05 11:51:35 original window size is 156 x 45
> sh: cannot set terminal process group (-1): Inappropriate ioctl for device
> sh: no job control in this shell
> {noformat}
> If I use `-pod`, the problem goes away. This only happens if command executor
> is used.
> A few research suggested that this issue is related to `setsid` (see this
> [thread|https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216]).
> Looks like we did an extra
> `[setsid|https://github.com/apache/mesos/blob/1.6.x/src/launcher/executor.cpp#L512]`
> in the command executor.
> The setsid() system call to create a new process group detaches the spawned
> process from a controlling tty. Therefore programs like bash complain, that
> they can't use job control. Re-attaching the controlling tty won't work,
> because the tty is still in use as a controlling tty for the command executor
> process.
> There are two possible solutions:
> 1) Get rid of setsid in command executor
> 2) Detach and re-attach the controlling TTY as suggested in
> https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (MESOS-8978) Command executor calling setsid breaks the tty support.
[
https://issues.apache.org/jira/browse/MESOS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues reassigned MESOS-8978:
--
Assignee: Kevin Klues
> Command executor calling setsid breaks the tty support.
> ---
>
> Key: MESOS-8978
> URL: https://issues.apache.org/jira/browse/MESOS-8978
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.4.1, 1.5.1, 1.6.0
>Reporter: Jie Yu
>Assignee: Kevin Klues
>Priority: Major
>
> I was playing with
> [msh|https://github.com/mesos/mesos-go/blob/master/api/v1/cmd/msh/msh.go]
> (one example from [mesos-go|https://github.com/mesos/mesos-go]), which allows
> you to launch an interactive shell in the Mesos cluster. It works by
> launching a container with tty enabled, and then [attach to the container
> input|https://github.com/apache/mesos/blob/master/include/mesos/v1/agent/agent.proto#L191-L201]
> using the agent operator API.
> However, I got the following error when doing the following:
> {noformat}
> Jies-MacBook-Pro:mesos-go jie$ ./msh -master 127.0.0.1:5050 -tty -interactive
> -- /bin/sh -i
> ...
> 2018/06/05 11:51:35 original window size is 156 x 45
> sh: cannot set terminal process group (-1): Inappropriate ioctl for device
> sh: no job control in this shell
> {noformat}
> If I use `-pod`, the problem goes away. This only happens if command executor
> is used.
> A few research suggested that this issue is related to `setsid` (see this
> [thread|https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216]).
> Looks like we did an extra
> `[setsid|https://github.com/apache/mesos/blob/1.6.x/src/launcher/executor.cpp#L512]`
> in the command executor.
> The setsid() system call to create a new process group detaches the spawned
> process from a controlling tty. Therefore programs like bash complain, that
> they can't use job control. Re-attaching the controlling tty won't work,
> because the tty is still in use as a controlling tty for the command executor
> process.
> There are two possible solutions:
> 1) Get rid of setsid in command executor
> 2) Detach and re-attach the controlling TTY as suggested in
> https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8770) Use Python3 for Mesos support scripts
[
https://issues.apache.org/jira/browse/MESOS-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596502#comment-16596502
]
Kevin Klues commented on MESOS-8770:
{noformat}
commit b55c5deb88b4a4e5713c9361e492e941972fe8db
Author: Kevin Klues
Date: Wed Aug 29 13:39:29 2018 +0200
Updated the python2 'PyLinter' to only lint python2 based code.
Specifically, this includes no longer linting all code under the
`src/python` directory.
Review: https://reviews.apache.org/r/68560
{noformat}
> Use Python3 for Mesos support scripts
> -
>
> Key: MESOS-8770
> URL: https://issues.apache.org/jira/browse/MESOS-8770
> Project: Mesos
> Issue Type: Task
>Reporter: Benjamin Bannier
>Assignee: Armand Grillet
>Priority: Major
>
> Our Python scripts under {{support/}} currently implicitly assume that
> developers have a python2 environment as their primary Python installation.
> We should consider updating these scripts so that they can be used with a
> python3 installation as well. There exist [some
> resources|http://python-future.org/overview.html#automatic-conversion-to-py2-3-compatible-code]
> on the web documenting best practices and tools for automatic rewrites which
> should get us a long way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545304#comment-16545304
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit da8cac1691cdb04c179e021980fbfd7773347095
Author: Armand Grillet
Date: Mon Jul 16 16:39:35 2018 +0200
Updated CLI to Python 3.
The build tools are also up to date thus the CLI can still be built
using Autotools and CMake. No features have been added to the CLI.
Along the way, we have also fixed some linting errors introduced when
upgrading pylint a few commits ago. They are included in this commit
instead of separated out because the commit chain would not apply
cleanly unless they were committed together.
Review: https://reviews.apache.org/r/67488/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8903) Update the Python CLI to use Python 3
[
https://issues.apache.org/jira/browse/MESOS-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545306#comment-16545306
]
Kevin Klues commented on MESOS-8903:
{noformat}
commit da8cac1691cdb04c179e021980fbfd7773347095
Author: Armand Grillet
Date: Mon Jul 16 16:39:35 2018 +0200
Updated CLI to Python 3.
The build tools are also up to date thus the CLI can still be built
using Autotools and CMake. No features have been added to the CLI.
Along the way, we have also fixed some linting errors introduced when
upgrading pylint a few commits ago. They are included in this commit
instead of separated out because the commit chain would not apply
cleanly unless they were committed together.
Review: https://reviews.apache.org/r/67488/
{noformat}
> Update the Python CLI to use Python 3
> -
>
> Key: MESOS-8903
> URL: https://issues.apache.org/jira/browse/MESOS-8903
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9074) Pylint is too noisy when using mesos-style.py
[
https://issues.apache.org/jira/browse/MESOS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543710#comment-16543710
]
Kevin Klues commented on MESOS-9074:
We are still waiting for tox 3.1.2 to come out to remove the line for e.g.:
{noformat}
Using config file /Users/Armand/Code/apache-mesos/support/pylint.config
{noformat}
> Pylint is too noisy when using mesos-style.py
> -
>
> Key: MESOS-9074
> URL: https://issues.apache.org/jira/browse/MESOS-9074
> Project: Mesos
> Issue Type: Bug
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Minor
>
> {code}
> apache-mesos (MESOS-9073) $ git commit -m "Test."
> No C++ files to lint
> No JavaScript files to lint
> Checking 1 Python file
> Using config file /Users/Armand/Code/apache-mesos/support/pylint.config
> ---
> Your code has been rated at 10.00/10 (previous run: 9.22/10, +0.78)
> Total errors found: 0
> [MESOS-9073 a3509d402] Test.
> 1 file changed, 1 insertion(+), 1 deletion(-)
> {code}
> The score needs to be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9074) Pylint is too noisy when using mesos-style.py
[
https://issues.apache.org/jira/browse/MESOS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543709#comment-16543709
]
Kevin Klues commented on MESOS-9074:
{noformat}
commit c2bdf07eb8771e7e00173606bedf2445a3a709b9
Author: Armand Grillet
Date: Fri Jul 13 22:37:50 2018 +0200
Updated pylint usage in mesos-style.py to be less verbose.
We started using Pylint 1.9 a few days ago to lint `.py` files, and this
version of pylint always prints a score after linting is complete. To
make the output less verbose, we now use the option `--score=n` when
using pylint in mesos-style.py.
Review: https://reviews.apache.org/r/67907/
{noformat}
> Pylint is too noisy when using mesos-style.py
> -
>
> Key: MESOS-9074
> URL: https://issues.apache.org/jira/browse/MESOS-9074
> Project: Mesos
> Issue Type: Bug
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Minor
>
> {code}
> apache-mesos (MESOS-9073) $ git commit -m "Test."
> No C++ files to lint
> No JavaScript files to lint
> Checking 1 Python file
> Using config file /Users/Armand/Code/apache-mesos/support/pylint.config
> ---
> Your code has been rated at 10.00/10 (previous run: 9.22/10, +0.78)
> Total errors found: 0
> [MESOS-9073 a3509d402] Test.
> 1 file changed, 1 insertion(+), 1 deletion(-)
> {code}
> The score needs to be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9073) Tox doesn't run in the support virtualenv when using Python 3 mesos-style.py
[
https://issues.apache.org/jira/browse/MESOS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543706#comment-16543706
]
Kevin Klues commented on MESOS-9073:
{noformat}
commit ec87ab575e698c447477fc6bb957f8270aea8c3f
Author: Armand Grillet
Date: Fri Jul 13 15:32:17 2018 +0200
Updated tox usage in mesos-style.py to run directly from virtualenv.
We were already doing this in the ptyhon2 mesos-style.py script. This
commit adds the same support to the python3 version of the script.
Review: https://reviews.apache.org/r/67906/
{noformat}
> Tox doesn't run in the support virtualenv when using Python 3 mesos-style.py
>
>
> Key: MESOS-9073
> URL: https://issues.apache.org/jira/browse/MESOS-9073
> Project: Mesos
> Issue Type: Bug
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9075) Virtualenv management in support directory is buggy.
[
https://issues.apache.org/jira/browse/MESOS-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543703#comment-16543703
]
Kevin Klues commented on MESOS-9075:
{noformat}
commit dd07b14df02d1a944004e63a03d97161c413e706
Author: Armand Grillet
Date: Fri Jul 13 22:10:46 2018 +0200
Fixed virtualenv management in support directory.
The switch from Python 2 to Python 3 creates problems with managing the
virtual environment in the support directory if a developer regularly
toggles between Python 2 and 3. For example, we want the virtual
environment to always use the same version of Python the user uses to
invoke the python linter from the command line through mesos-style.py.
This commit fixes the issue by adding a new check in the function of
`mesos-style.py` called `should_build_virtualenv` to see if the Python
interpreter version currently used is the one in the virtual
environement. If not, the virtual environment will get recreated.
Review: https://reviews.apache.org/r/67910/
{noformat}
> Virtualenv management in support directory is buggy.
>
>
> Key: MESOS-9075
> URL: https://issues.apache.org/jira/browse/MESOS-9075
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.7.0
>
>
> When switching back and forth from Python 2 to 3, the virtualenv does not get
> correctly reinstalled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9075) Virtualenv management in support directory is buggy.
[
https://issues.apache.org/jira/browse/MESOS-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543702#comment-16543702
]
Kevin Klues commented on MESOS-9075:
{noformat}
commit 6936fb553b463b99bf4bc07de9067dd9ca16457a
Author: Armand Grillet
Date: Fri Jul 13 22:21:00 2018 +0200
Fixed linting errors that appear with the recently updated pylint.
As part of this, we needed to remove some unnecessary delegate to
'super' calls that are legal in python2 but not in python3. That said,
they are also unnecessary in python2 so we remove them in both cases.
Specifically, we previously used to override 'main()' in our Python and
JS linters with:
def main(self, modified_files):
super(PyLinter, self).main(modified_files)
def main(self, modified_files):
super(JsLinter, self).main(modified_files)
However, in python3 this gives us linting errors because such a
delegation is unecessary. Just using the 'main()' function from the base
class is sufficient.
Review: http://reviews.apache.org/r/67910/
{noformat}
> Virtualenv management in support directory is buggy.
>
>
> Key: MESOS-9075
> URL: https://issues.apache.org/jira/browse/MESOS-9075
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.7.0
>
>
> When switching back and forth from Python 2 to 3, the virtualenv does not get
> correctly reinstalled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9075) Virtualenv management in support directory is buggy.
[
https://issues.apache.org/jira/browse/MESOS-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543699#comment-16543699
]
Kevin Klues commented on MESOS-9075:
{noformat}
commit 4a02bbca61be7451df99f89fc2b99615cc5a7064
Author: Armand Grillet
Date: Fri Jul 13 22:02:41 2018 +0200
Fixed `build-virtualenv` to correctly clear virtual environment.
Running `build-virtualenv` with Python 3 followed by running it again
with Python 2 leaves the directory with both versions of Python in the
virtualenv. This allows Python 3 to still be used from within the
virtualenv, which is incorrect.
This is caused by a long outstanding issue in `virtualenv` that we now
handle by unconditionally removing the 'support/.virtualenv' directory
before building it again.
Review: https://reviews.apache.org/r/67911/
{noformat}
> Virtualenv management in support directory is buggy.
>
>
> Key: MESOS-9075
> URL: https://issues.apache.org/jira/browse/MESOS-9075
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> When switching back and forth from Python 2 to 3, the virtualenv does not get
> correctly reinstalled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9058) ModuleNotFoundError: No module named 'nodeenv' when setting up support virtualenv.
[
https://issues.apache.org/jira/browse/MESOS-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537151#comment-16537151
]
Kevin Klues commented on MESOS-9058:
{noformat}
commit d7eb8a9a96b7abadd9dd778604f16383567369f9
Author: Armand Grillet
Date: Mon Jul 9 18:03:30 2018 +0200
Updated build-virtualenv to use Python 3 features.
With Python 3, virtualenv is built-in. We thus remove the error
messages if the script is run using Python 3 without virtualenv
installed on the machine. We also use `python` instead of
`virtualenv` to build the virtual environment.
This change also fixes an issue on my computer where running
`build-virtualenv` with Python 3 displayed an error message:
`ModuleNotFoundError: No module named 'nodeenv'`.
Review: https://reviews.apache.org/r/67853/
{noformat}
> ModuleNotFoundError: No module named 'nodeenv' when setting up support
> virtualenv.
> --
>
> Key: MESOS-9058
> URL: https://issues.apache.org/jira/browse/MESOS-9058
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Minor
>
> {code}
> apache-mesos (MESOS-8955) $ git rebase -i HEAD
> No C++ files to lint
> No JavaScript files to lint
> Checking 1 Python file
> Total errors found: 0
> The "pip-requirements.txt" file has changed.
> Rebuilding virtualenv...
> Traceback (most recent call last):
> File "/Users/Armand/Code/apache-mesos/support/.virtualenv/bin/nodeenv",
> line 7, in
> from nodeenv import main
> ModuleNotFoundError: No module named 'nodeenv'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534956#comment-16534956
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit 77921200a69564f966917bfc8e07a3d1e3ada196
Author: Armand Grillet
Date: Fri Jul 6 15:52:28 2018 +0200
Refactored base of Python CLI tests.
This patch mostly adds additional hardening to make sure we error out
with proper error messages in places where we weren't before.
However, as part of this, we also fixed a bug related to referencing
counting Master, Agent, and Task objects. Previously, this reference
count would increased in '__init__()' and decreased in '__del__()'.
However, this caused problems when limiting the number of agents and
masters to exactly 1, because we could never guarantee when the python
garbage collector would kick in to delete objects from previous test
runs. This manifested itself as flaky test failures. To fix this, we
now reference count only on explicit counts to 'launch()' and 'kill()'
instead of in '__init__()' and '__del__()'.
Review: https://reviews.apache.org/r/67842/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532830#comment-16532830
]
Kevin Klues edited comment on MESOS-8955 at 7/4/18 2:32 PM:
{noformat}
commit bf69856f241590b652485e012580dbbdb5e11674
Author: Armand Grillet
Date: Wed Jul 4 16:06:36 2018 +0200
Updated configure.ac and Makefile.am to use `$PYTHON` not `python`.
This ensures that we use Python 2 even if the default Python
installed in the path under `python` is Python 3.
Previously, even when explicitly setting PYTHON_VERSION or PYTHON to a
version of Python 2.6+, configuration would succeed, but compilation
would fail. This commit fixes that.
Review: https://reviews.apache.org/r/67487/
{noformat}
was (Author: klueska):
{noformat}
commit bf69856f241590b652485e012580dbbdb5e11674 (HEAD -> master,
upstream/master)
Author: Armand Grillet
Date: Wed Jul 4 16:06:36 2018 +0200
Updated configure.ac and Makefile.am to use `$PYTHON` not `python`.
This ensures that we use Python 2 even if the default Python
installed in the path under `python` is Python 3.
Previously, even when explicitly setting PYTHON_VERSION or PYTHON to a
version of Python 2.6+, configuration would succeed, but compilation
would fail. This commit fixes that.
Review: https://reviews.apache.org/r/67487/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532830#comment-16532830
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit bf69856f241590b652485e012580dbbdb5e11674 (HEAD -> master,
upstream/master)
Author: Armand Grillet
Date: Wed Jul 4 16:06:36 2018 +0200
Updated configure.ac and Makefile.am to use `$PYTHON` not `python`.
This ensures that we use Python 2 even if the default Python
installed in the path under `python` is Python 3.
Previously, even when explicitly setting PYTHON_VERSION or PYTHON to a
version of Python 2.6+, configuration would succeed, but compilation
would fail. This commit fixes that.
Review: https://reviews.apache.org/r/67487/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532782#comment-16532782
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit f9bb153f0abbc0e33787002bfd0d4abacb5f8645
Author: Armand Grillet
Date: Wed Jul 4 15:29:37 2018 +0200
Refactored logic for `PYTHON` and `PYTHON_VERSION` in `configure.ac`.
This will facilitate the introduction of `PYTHON_3` and
`PYTHON_3_VERSION` to build the CLI in a future commit.
Review: https://reviews.apache.org/r/67413/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513820#comment-16513820
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit 57091affe3380a01d516b03be2700db85c328ede
Author: Armand Grillet agril...@mesosphere.io
Date: Fri Jun 15 15:26:15 2018 +0200
Improved coverage with configure and `PYTHON` or `PYTHON_VERSION` set.
We set `PYTHON` and `PYTHON_VERSION` when configuring the build.
We now cover all possible cases (both variables set, only one, none).
This ensures that both variables are set after being checked.
Review: https://reviews.apache.org/r/67412/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8955) Manage Python2 and 3 in build steps
[
https://issues.apache.org/jira/browse/MESOS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513805#comment-16513805
]
Kevin Klues commented on MESOS-8955:
{noformat}
commit 3bf6eb7c88b7ea365c27fae2623322876ed2af45
Author: Armand Grillet
Date: Fri Jun 15 15:17:09 2018 +0200
Broadened check for Autotools Python environment variables.
The checks now also apply if we run configure with disabled Python
bindings but enabled new CLI. Another check regarding the Python
version has been kept separated as we will use different Python
versions for both components in a later commit.
Review: https://reviews.apache.org/r/67411/
{noformat}
> Manage Python2 and 3 in build steps
> ---
>
> Key: MESOS-8955
> URL: https://issues.apache.org/jira/browse/MESOS-8955
> Project: Mesos
> Issue Type: Improvement
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8978) Command executor calling setsid breaks the tty support.
[
https://issues.apache.org/jira/browse/MESOS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502500#comment-16502500
]
Kevin Klues commented on MESOS-8978:
We discussed on slack and confirmed there is a similar issue with the DC/OS CLI.
{{msh}} is more like a launch-then-attach, than an {{exec}}.
Running the equivalent in the {{dcos}} CLI, we see:
{noformat}
Launch marathon app:
{
"id": "/bash-forever",
"cmd": "/bin/sh -i",
"cpus": 0.1,
"disk": 0,
"instances": 1,
"mem": 128,
"tty": true
}
Run attach and print stdout:
$ dcos task attach bash-forever
sh-4.3# cat stdout
Executing pre-exec command
'{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/opt\/mesosphere\/active\/mesos\/libexec\/mesos\/mesos-containerizer"}'
Executing pre-exec command '{"shell":true,"value":"mount -n -t proc proc \/proc
-o nosuid,noexec,nodev"}'
Executing pre-exec command
'{"arguments":["mount","-n","-t","ramfs","ramfs","\/var\/lib\/mesos\/slave\/slaves\/7afdd628-3cfb-44f8-b6dc-dbe3ba18901e-S4\/frameworks\/7afdd628-3cfb-44f8-b6dc-dbe3ba18901e-0001\/executors\/bash-forever.76230e3e-6902-11e8-b489-1a42a410974d\/runs\/056fda1e-e8d1-4f96-ae3d-d5fdc902b908\/.secret-c5c4bf09-d6a5-419c-8c8f-103c601d7480"],"shell":false,"value":"mount"}'
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0605 20:53:13.284710 5 openssl.cpp:476] Failed SSL connections will be
downgraded to a non-SSL socket
I0605 20:53:13.284783 5 openssl.cpp:498] CA directory path unspecified!
NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
I0605 20:53:13.284796 5 openssl.cpp:503] Will not verify peer certificate!
NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate verification
I0605 20:53:13.284811 5 openssl.cpp:509] Will only verify peer certificate
if presented!
NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate verification
W0605 20:53:13.285053 5 process.cpp:1197] Failed SSL connections will be
downgraded to a non-SSL socket
I0605 20:53:13.289585 5 exec.cpp:162] Version: 1.4.2
I0605 20:53:13.300499 9 exec.cpp:236] Executor registered on agent
7afdd628-3cfb-44f8-b6dc-dbe3ba18901e-S4
I0605 20:53:13.30170913 executor.cpp:171] Received SUBSCRIBED event
I0605 20:53:13.30195613 executor.cpp:175] Subscribed executor on 10.0.0.71
I0605 20:53:13.30203813 executor.cpp:171] Received LAUNCH event
I0605 20:53:13.30222713 executor.cpp:633] Starting task
bash-forever.76230e3e-6902-11e8-b489-1a42a410974d
I0605 20:53:13.30286613 executor.cpp:477] Running
'/opt/mesosphere/active/mesos/libexec/mesos/mesos-containerizer launch
'
I0605 20:53:13.30372213 executor.cpp:646] Forked command at 15
sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
{noformat}
> Command executor calling setsid breaks the tty support.
> ---
>
> Key: MESOS-8978
> URL: https://issues.apache.org/jira/browse/MESOS-8978
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.4.1, 1.5.1, 1.6.0
>Reporter: Jie Yu
>Priority: Major
>
> I was playing with
> [msh|https://github.com/mesos/mesos-go/blob/master/api/v1/cmd/msh/msh.go]
> (one example from [mesos-go|https://github.com/mesos/mesos-go]), which allows
> you to launch an interactive shell in the Mesos cluster. It works by
> launching a container with tty enabled, and then [attach to the container
> input|https://github.com/apache/mesos/blob/master/include/mesos/v1/agent/agent.proto#L191-L201]
> using the agent operator API.
> However, I got the following error when doing the following:
> {noformat}
> Jies-MacBook-Pro:mesos-go jie$ ./msh -master 127.0.0.1:5050 -tty -interactive
> -- /bin/sh -i
> ...
> 2018/06/05 11:51:35 original window size is 156 x 45
> sh: cannot set terminal process group (-1): Inappropriate ioctl for device
> sh: no job control in this shell
> {noformat}
> If I use `-pod`, the problem goes away. This only happens if command executor
> is used.
> A few research suggested that this issue is related to `setsid` (see this
> [thread|https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216]).
> Looks like we did an extra
> `[setsid|https://github.com/apache/mesos/blob/1.6.x/src/launcher/executor.cpp#L512]`
> in the command executor.
> The setsid() system call to create a new process group detaches the spawned
> process from a controlling tty. Therefore programs like bash complain, that
> they can't use job control. Re-attaching the controlling tty won't work,
> because the tty is still in use as a controlling tty for the command executor
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8978) Command executor calling setsid breaks the tty support.
[
https://issues.apache.org/jira/browse/MESOS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502455#comment-16502455
]
Kevin Klues commented on MESOS-8978:
Why don't I have this problem with the DC/OS CLI and launching {{sh -i`}}?
My command line looks similar:
{noformat}
dcos task exec --tty --interactive /bin/sh -i
{noformat}
and it works just fine
Concrete example I just ran:
{noformat}
$ dcos task exec -it sleep /bin/sh -i
sh-4.3# ls
containers stderr stderr.logrotate.conf stdout stdout.logrotate.conf
{noformat}
However, without the {{-it}} flags I see the same error as you:
{noformat}
$ dcos task exec sleep /bin/sh -i
sh: cannot set terminal process group (48): Inappropriate ioctl for device
sh: no job control in this shell
sh-4.3#
{noformat}
Are you sure there aren't issues with the way `msh` interprets the `-tty` and
`-interactive` flags?
> Command executor calling setsid breaks the tty support.
> ---
>
> Key: MESOS-8978
> URL: https://issues.apache.org/jira/browse/MESOS-8978
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.4.1, 1.5.1, 1.6.0
>Reporter: Jie Yu
>Priority: Major
>
> I was playing with
> [msh|https://github.com/mesos/mesos-go/blob/master/api/v1/cmd/msh/msh.go]
> (one example from [mesos-go|https://github.com/mesos/mesos-go]), which allows
> you to launch an interactive shell in the Mesos cluster. It works by
> launching a container with tty enabled, and then [attach to the container
> input|https://github.com/apache/mesos/blob/master/include/mesos/v1/agent/agent.proto#L191-L201]
> using the agent operator API.
> However, I got the following error when doing the following:
> {noformat}
> Jies-MacBook-Pro:mesos-go jie$ ./msh -master 127.0.0.1:5050 -tty -interactive
> -- /bin/sh -i
> ...
> 2018/06/05 11:51:35 original window size is 156 x 45
> sh: cannot set terminal process group (-1): Inappropriate ioctl for device
> sh: no job control in this shell
> {noformat}
> If I use `-pod`, the problem goes away. This only happens if command executor
> is used.
> A few research suggested that this issue is related to `setsid` (see this
> [thread|https://github.com/Yelp/dumb-init/issues/51#issuecomment-227792216]).
> Looks like we did an extra
> `[setsid|https://github.com/apache/mesos/blob/1.6.x/src/launcher/executor.cpp#L512]`
> in the command executor.
> The setsid() system call to create a new process group detaches the spawned
> process from a controlling tty. Therefore programs like bash complain, that
> they can't use job control. Re-attaching the controlling tty won't work,
> because the tty is still in use as a controlling tty for the command executor
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-6918) Prometheus exporter endpoints for metrics
[
https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484034#comment-16484034
]
Kevin Klues commented on MESOS-6918:
Which endpoints is this targeted for specifically?
Useful metrics are kind of spread out all over the place at the moment.
{noformat}
masters:
/metrics/snapshot
/state -- per agent resource usage stats are in here
agents:
/metrics/snapshot
/monitor/statistics
{noformat}
> Prometheus exporter endpoints for metrics
> -
>
> Key: MESOS-6918
> URL: https://issues.apache.org/jira/browse/MESOS-6918
> Project: Mesos
> Issue Type: Bug
> Components: statistics
>Reporter: James Peach
>Assignee: James Peach
>Priority: Major
>
> There are a couple of [Prometheus|https://prometheus.io] metrics exporters
> for Mesos, of varying quality. Since the Mesos stats system actually knows
> about statistics data types and semantics, and Mesos has reasonable HTTP
> support we could add Prometheus metrics endpoints to directly expose
> statistics in [Prometheus wire
> format|https://prometheus.io/docs/instrumenting/exposition_formats/],
> removing the need for operators to run separate exporter processes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8240) Add an option to build the new CLI and run unit tests.
[
https://issues.apache.org/jira/browse/MESOS-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435560#comment-16435560
]
Kevin Klues commented on MESOS-8240:
{noformat}
commit bffa9f3c5b8dc5ed27fc4f079c0376c2a64ea042
Author: Armand Grillet
Date: Thu Apr 12 06:10:28 2018 -0700
Added options to build the Python CLI and run unit tests.
An update of the discarded review /r/52543.
Works with Autotools and CMake.
Review: https://reviews.apache.org/r/64211/
{noformat}
> Add an option to build the new CLI and run unit tests.
> --
>
> Key: MESOS-8240
> URL: https://issues.apache.org/jira/browse/MESOS-8240
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
> Fix For: 1.6.0
>
>
> An update of the discarded [https://reviews.apache.org/r/52543/]
> Also needs to be available for CMake.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8240) Add an option to build the new CLI and run unit tests.
[
https://issues.apache.org/jira/browse/MESOS-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435505#comment-16435505
]
Kevin Klues commented on MESOS-8240:
{noformat}
commit 2f7f03f249a724d61265dabfadb5bb83eb2008e8
Author: Armand Grillet
Date: Thu Apr 12 14:37:10 2018 +0200
Improved documentation regarding the new CLI setup.
Explains how to create the necessary virtual environment from
anywhere and how to set up autocompletion in such case.
Also removes an unnecessary activation of the virtual environment
in `mesos` and `mesos-cli-tests`.
Review: https://reviews.apache.org/r/65585/
{noformat}
{noformat}
commit b6ea804322c85eae066c6b52239063b1169f54a3 (HEAD)
Author: Armand Grillet
Date: Thu Apr 12 14:43:54 2018 +0200
Removed 'source activate' from new CLI wrapper scripts.
Having these command in these scripts was only necessary if you wanted
to run them from outside the virtual environment they were meant to run
in. Since we will soon be allowing people to create this virtual
environment anywhere on their filesystem, it doesn't make sense to
assume they have done it in the 'src/python/cli_new' directory anymore.
Since we can't assume that, we will just have to mandate that the
virtualenv is actually set up before invoking these scripts.
{noformat}
> Add an option to build the new CLI and run unit tests.
> --
>
> Key: MESOS-8240
> URL: https://issues.apache.org/jira/browse/MESOS-8240
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> An update of the discarded [https://reviews.apache.org/r/52543/]
> Also needs to be available for CMake.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-7693) DEBUG container does not inherit env variable properly for command tasks.
[
https://issues.apache.org/jira/browse/MESOS-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432898#comment-16432898
]
Kevin Klues commented on MESOS-7693:
The underlying mesos implementation of debug containers doesn't allow an exec'd
container to inherit environment variables from those defined in the parent
container's base docker image.
For example, when running the following directly on docker, we see the
{{LD_LIBRARY_PATH}} set from the base {{nvidia/cuda}} image inside the exec'd
process:
{noformat}
[klueska@core-dev ~]$ docker run -d nvidia/cuda sleep
1a78ee1eed45947952bc0a80354d3419423e6d5a52ee43909a54f54e3b751d3e
[klueska@core-dev ~]$ docker ps
CONTAINER IDIMAGE COMMAND CREATED
STATUS PORTS NAMES
1a78ee1eed45nvidia/cuda "sleep "9 seconds ago
Up 4 secondsvibrant_volhard
[klueska@core-dev ~]$ docker exec -it 1a78ee1eed45 bash -c "echo
\$LD_LIBRARY_PATH"
/usr/local/nvidia/lib:/usr/local/nvidia/lib64
{noformat}
But when running the equivalent on {{dcos}} via marathon:
{noformat}
$ cat cuda.json
{
"id": "nvidia-test",
"mem": 2048,
"disk": 0,
"instances": 1,
"cmd": "sleep 999",
"container": {
"type": "MESOS",
"docker": {
"image": "nvidia/cuda"
}
}
}
$ dcos marathon app add cuda.json
Created deployment 3d02ceb1-519c-4f08-a85c-bb0b8134c630
$ dcos task
NAME HOST USER STATE ID
MESOS ID REGION ZONE
nvidia-test 10.10.0.183 rootR
nvidia-test.3f941a9f-0a8c-11e8-b838-f638b54f6560
e896367a-8dbe-40d6-9867-1af79b77fb04-S0 us-west-2 us-west-2c
$ dcos task exec -it nvidia-test bash -c "echo \$LD_LIBRARY_PATH"
{noformat}
We need to update the logic for debug containers to inherit this properly:
This will likely involve an update to
https://github.com/apache/mesos/blob/56100adc14a8acbcf2ee0aae967f758f592fcd31/src/slave/containerizer/mesos/containerizer.cpp#L1613
> DEBUG container does not inherit env variable properly for command tasks.
> -
>
> Key: MESOS-7693
> URL: https://issues.apache.org/jira/browse/MESOS-7693
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Jie Yu
>Assignee: Alexander Rukletsov
>Priority: Major
>
> I can repo the issue:
> {code}
> sudo /home/vagrant/workspace/dist/mesos-1.4.0/bin/mesos-execute
> --master=172.28.128.3:5050 --name=java8 --docker_image=java:8
> --command="sleep 1000"
> I0618 17:42:21.410598 3356 scheduler.cpp:184] Version: 1.4.0
> I0618 17:42:21.413465 3356 scheduler.cpp:470] New master detected at
> master@172.28.128.3:5050
> Subscribed with ID cacf5c08-cbbc-401a-a84d-2cfc4edc6519-0006
> Submitted task 'java8' to agent 'cacf5c08-cbbc-401a-a84d-2cfc4edc6519-S0'
> Received status update TASK_RUNNING for task 'java8'
> source: SOURCE_EXECUTOR
> Jies-MacBook-Pro:script jie$ ./dcos task
> NAME HOST USER STATE ID
> java8 172.28.128.3 rootRjava8
> Jies-MacBook-Pro:script jie$ ./dcos task exec -t -i java8 bash
> root@vagrant-ubuntu-trusty-64:/mnt/mesos/sandbox# env
> LIBPROCESS_IP=172.28.128.3
> MESOS_AGENT_ENDPOINT=172.28.128.3:5051
> MESOS_DIRECTORY=/tmp/mesos/slave/slaves/cacf5c08-cbbc-401a-a84d-2cfc4edc6519-S0/frameworks/cacf5c08-cbbc-401a-a84d-2cfc4edc6519-0006/executors/java8/runs/1b06c661-20f3-460a-8cfd-475dc3e60aa3
> MESOS_EXECUTOR_ID=java8
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> PWD=/mnt/mesos/sandbox
> MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD=5secs
> MESOS_NATIVE_JAVA_LIBRARY=/home/vagrant/workspace/dist/mesos-1.4.0/lib/libmesos-1.4.0.so
> MESOS_NATIVE_LIBRARY=/home/vagrant/workspace/dist/mesos-1.4.0/lib/libmesos-1.4.0.so
> MESOS_HTTP_COMMAND_EXECUTOR=0
> MESOS_SLAVE_PID=slave(1)@172.28.128.3:5051
> MESOS_FRAMEWORK_ID=cacf5c08-cbbc-401a-a84d-2cfc4edc6519-0006
> MESOS_CHECKPOINT=0
> SHLVL=1
> LIBPROCESS_PORT=0
> MESOS_SLAVE_ID=cacf5c08-cbbc-401a-a84d-2cfc4edc6519-S0
> MESOS_SANDBOX=/mnt/mesos/sandbox
> _=/usr/bin/env
> {code}
> As you can see, environment variables like JAVA_HOME defined in the docker
> image are not in the debug container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8758) Document LABEL and ENV VAR dependency for GPU support.
[
https://issues.apache.org/jira/browse/MESOS-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425228#comment-16425228
]
Kevin Klues commented on MESOS-8758:
This is the nvidia-docker documentation about this.
https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection-%28version-1.0%29
In the past when questions have come up I've sent the following to the mailing
list:
{noformat}
Just deriving from "ubuntu:16.04" wont work out of the box.
In order for the libs to be mounted, and available in the path, the docker
image needs to have:
1) A label set for "com.nvidia.volumes.needed = nvidia_driver"
2) The env set for
"LD_LIBRARY_PATH=/usr/local/nvidia/lib;/usr/local/nvidia/lib64"
3) The env set for "PATH=/usr/local/nvidia/bin;$PATH"
The nvidia images do this automatically as described here:
https://github.com/NVIDIA/nvidia-docker/wiki/Internals
{noformat}
In general, we've always just referred people to the nvidia-docker
documentation since that is the source of truth for what is required. We should
probably duplicate this in our docs though. Especially since (for more than a
year now) nvidia-docker has a new ay of doing things and we've never had the
opportunity to catch up with them. That is, they don't actually need this label
or env vars set anymore. Luckily they stay backwards compatible with their old
way of doing things though, so these labels and envvars are still set in all of
their images. It's very risky to keep assuming they will do this forever
though. My guess is that they will stop doing it within the next year.
> Document LABEL and ENV VAR dependency for GPU support.
> --
>
> Key: MESOS-8758
> URL: https://issues.apache.org/jira/browse/MESOS-8758
> Project: Mesos
> Issue Type: Documentation
> Components: documentation
>Reporter: Gilbert Song
>Priority: Major
> Labels: document
>
> If using GPU on UCR with non-Nvidia based images, the following LABEL and ENV
> VAR is needed in the image:
> {noformat}
> LABEL com.nvidia.volumes.needed="nvidia_driver"
> LABEL com.nvidia.cuda.version="${CUDA_VERSION}"
> {noformat}
> {noformat}
> ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
> ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-8240) Add an option to build the new CLI and run unit tests.
[
https://issues.apache.org/jira/browse/MESOS-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367405#comment-16367405
]
Kevin Klues commented on MESOS-8240:
{noformat}
commit 550d3723210490a47c15b2cd7ba7a2a0322bffcf
Author: Armand Grillet
Date: Fri Feb 16 15:45:58 2018 +0100
Updated Mesos CLI test base to use shell to start masters and agents.
We use the shell scripts in `build/src` to start masters and agents
for the CLI tests. Instead of running them as executables, we now use
the shell launch them using the new Executable attribute 'shell'.
Review: https://reviews.apache.org/r/65655/
{noformat}
> Add an option to build the new CLI and run unit tests.
> --
>
> Key: MESOS-8240
> URL: https://issues.apache.org/jira/browse/MESOS-8240
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>Priority: Major
>
> An update of the discarded [https://reviews.apache.org/r/52543/]
> Also needs to be available for CMake.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-7310) Implement a separate python client library for the new cli
[
https://issues.apache.org/jira/browse/MESOS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355574#comment-16355574
]
Kevin Klues commented on MESOS-7310:
{noformat}
commit 6906310e568355724933503c85a7248ce8cf641d
Author: Eric Chung
Date: Wed Feb 7 15:49:32 2018 +0100
Added extra python package requirements for the new python-based CLI.
These extra python packages will be used by subsequent commits.
Review: https://reviews.apache.org/r/61172/
{noformat}
{noformat}
commit 9571adc9ac87873bf7688eefcb315d4141f4b3e1
Author: Eric Chung
Date: Wed Feb 7 15:52:56 2018 +0100
Whitelisted the 'ujson' package for linting in the new python-based CLI.
The 'usjon' package contains C-bindings which break the python linter
when imported into our CLI python modules. This commit whitelists this
package for the linter so that we can import it without linting errors.
Review: https://reviews.apache.org/r/61172/
{noformat}
{noformat}
commit dea1b31ec6ac466831095d82e280911365d2f174
Author: Eric Chung
Date: Wed Feb 7 16:01:08 2018 +0100
Updated the python-based CLIs tox.ini file.
This file has been updated to pull in its python dependencies directly
from the relevant requirements.txt and requirements-test.txt files
instead of duplicating the list of packages from these files.
Review: https://reviews.apache.org/r/61172/
{noformat}
{noformat}
commit 994213739b1afc473bbd9d15ded7c3fd26eaa924 (HEAD -> master,
upstream/master, origin/master)
Author: Eric Chung
Date: Wed Feb 7 16:02:17 2018 +0100
Added 'mesos.http' and 'mesos.exceptions' for the python-based CLI.
Part of MESOS-7310, this patch adds the mesos.http and mesos.exceptions
modules, which provides a Resource class and its descendants for
abstracting away common operations over http connections with JSON
serialization.
Review: https://reviews.apache.org/r/61172/
{noformat}
> Implement a separate python client library for the new cli
> --
>
> Key: MESOS-7310
> URL: https://issues.apache.org/jira/browse/MESOS-7310
> Project: Mesos
> Issue Type: Task
> Components: cli
>Affects Versions: 1.3.0
>Reporter: Eric Chung
>Assignee: Eric Chung
>Priority: Major
>
> cli_new in its current form is very difficult to package due to the following
> reasons:
> 1. src/cli_new/lib/mesos imports plugins using relative imports, which fails
> if it is built into a pip package
> 2. there is no setup.py script which defines what should be installed
> 3. plugins/tests are unnecessarily included in the package, which are things
> consumers of the package shouldn’t be able to import
> having such a package will allow external consumers to be able to add
> application-specific wrappers on it, e.g. integration with ACL libraries of
> their choice.
> The plan as discussed will create a `mesos` package under `src/python/lib`,
> potentially including a `setup.py` for building the package into a PyPI
> package.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-7924) Add a javascript linter to the webui.
[
https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247536#comment-16247536
]
Kevin Klues commented on MESOS-7924:
{noformat}
commit f03e75c98b67d2e610bbd5eae366f7b206306399
Author: Armand Grillet
Date: Fri Nov 10 14:50:15 2017 +0100
Updated JavaScript linter rules.
The rules we added do the following:
1. They force us to use 'this_' for any alias of 'this'
that we create.
2. They allow us to have arguments to functions that go unused so long
as the variable names begin with an '_'.
Review: https://reviews.apache.org/r/63724/
{noformat}
{noformat}
commit af8404e7bf49201d7f3a0a563f43d767539558f0
Author: Armand Grillet
Date: Fri Nov 10 14:52:43 2017 +0100
Linted JavaScript files.
Review: https://reviews.apache.org/r/63703/
{noformat}
{noformat}
commit 5516adba30324bcb1a8e9aa9ab95e489663c70a4
Author: Armand Grillet
Date: Fri Nov 10 15:03:10 2017 +0100
Improved support/mesos-style.py structure.
We have updated the following three things:
1. The Python linter now also use the function
`run_command_in_virtualenv` to run.
2. The methods of LinterBase have been reordered alphabetically.
3. Some unecessary excluded files have been removed for the JS linter.
Review: https://reviews.apache.org/r/63585/
{noformat}
> Add a javascript linter to the webui.
> -
>
> Key: MESOS-7924
> URL: https://issues.apache.org/jira/browse/MESOS-7924
> Project: Mesos
> Issue Type: Improvement
> Components: webui
>Reporter: Benjamin Mahler
>Assignee: Armand Grillet
> Labels: tech-debt
> Fix For: 1.5.0
>
>
> As far as I can tell, javascript linters (e.g. ESLint) help catch some
> functional errors as well, for example, we've made some "strict" mistakes a
> few times that ESLint can catch: MESOS-6624, MESOS-7912.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7924) Add a javascript linter to the webui.
[
https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245903#comment-16245903
]
Kevin Klues commented on MESOS-7924:
{noformat}
commit 844590611067d04de86a2de923b21ef377554728 (HEAD -> master,
upstream/master)
Author: Armand Grillet
Date: Thu Nov 9 16:53:40 2017 +0100
Added JavaScript linter.
The linter runs when changes on a JavaScript file are being committed.
We use ESLint with a configuration based on our current JS code base.
The linter and its dependencies (i.e. Node.js) are installed in a
virtual environment using Virtualenv and then Nodeenv.
Review: https://reviews.apache.org/r/62214/
{noformat}
> Add a javascript linter to the webui.
> -
>
> Key: MESOS-7924
> URL: https://issues.apache.org/jira/browse/MESOS-7924
> Project: Mesos
> Issue Type: Improvement
> Components: webui
>Reporter: Benjamin Mahler
>Assignee: Armand Grillet
> Labels: tech-debt
> Fix For: 1.5.0
>
>
> As far as I can tell, javascript linters (e.g. ESLint) help catch some
> functional errors as well, for example, we've made some "strict" mistakes a
> few times that ESLint can catch: MESOS-6624, MESOS-7912.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (MESOS-7924) Add a javascript linter to the webui.
[ https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Klues updated MESOS-7924: --- Shepherd: Kevin Klues (was: Benjamin Mahler) > Add a javascript linter to the webui. > - > > Key: MESOS-7924 > URL: https://issues.apache.org/jira/browse/MESOS-7924 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Benjamin Mahler >Assignee: Armand Grillet > Labels: tech-debt > Fix For: 1.5.0 > > > As far as I can tell, javascript linters (e.g. ESLint) help catch some > functional errors as well, for example, we've made some "strict" mistakes a > few times that ESLint can catch: MESOS-6624, MESOS-7912. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (MESOS-7924) Add a javascript linter to the webui.
[
https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245903#comment-16245903
]
Kevin Klues edited comment on MESOS-7924 at 11/9/17 4:01 PM:
-
{noformat}
commit 844590611067d04de86a2de923b21ef377554728
Author: Armand Grillet
Date: Thu Nov 9 16:53:40 2017 +0100
Added JavaScript linter.
The linter runs when changes on a JavaScript file are being committed.
We use ESLint with a configuration based on our current JS code base.
The linter and its dependencies (i.e. Node.js) are installed in a
virtual environment using Virtualenv and then Nodeenv.
Review: https://reviews.apache.org/r/62214/
{noformat}
was (Author: klueska):
{noformat}
commit 844590611067d04de86a2de923b21ef377554728 (HEAD -> master,
upstream/master)
Author: Armand Grillet
Date: Thu Nov 9 16:53:40 2017 +0100
Added JavaScript linter.
The linter runs when changes on a JavaScript file are being committed.
We use ESLint with a configuration based on our current JS code base.
The linter and its dependencies (i.e. Node.js) are installed in a
virtual environment using Virtualenv and then Nodeenv.
Review: https://reviews.apache.org/r/62214/
{noformat}
> Add a javascript linter to the webui.
> -
>
> Key: MESOS-7924
> URL: https://issues.apache.org/jira/browse/MESOS-7924
> Project: Mesos
> Issue Type: Improvement
> Components: webui
>Reporter: Benjamin Mahler
>Assignee: Armand Grillet
> Labels: tech-debt
> Fix For: 1.5.0
>
>
> As far as I can tell, javascript linters (e.g. ESLint) help catch some
> functional errors as well, for example, we've made some "strict" mistakes a
> few times that ESLint can catch: MESOS-6624, MESOS-7912.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7924) Add a javascript linter to the webui.
[
https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245805#comment-16245805
]
Kevin Klues commented on MESOS-7924:
{noformat}
commit 0f674cb7fcc827ef241dc76fa40139e86717
Author: Armand Grillet
Date: Thu Nov 9 16:17:12 2017 +0100
Removed pylint from the CLI requirements.
Due to the new virtual environment located in /support, we do
not need to have pylint in the CLI virtual environment anymore.
Review: https://reviews.apache.org/r/63582/
{noformat}
> Add a javascript linter to the webui.
> -
>
> Key: MESOS-7924
> URL: https://issues.apache.org/jira/browse/MESOS-7924
> Project: Mesos
> Issue Type: Improvement
> Components: webui
>Reporter: Benjamin Mahler
>Assignee: Armand Grillet
> Labels: tech-debt
> Fix For: 1.5.0
>
>
> As far as I can tell, javascript linters (e.g. ESLint) help catch some
> functional errors as well, for example, we've made some "strict" mistakes a
> few times that ESLint can catch: MESOS-6624, MESOS-7912.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7924) Add a javascript linter to the webui.
[
https://issues.apache.org/jira/browse/MESOS-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245799#comment-16245799
]
Kevin Klues commented on MESOS-7924:
{noformat}
commit 102a36304ca1a78384b8841c94b122a4a93e5a4e
Author: Armand Grillet
Date: Thu Nov 9 16:10:22 2017 +0100
Created virtual environment for linters in /support.
This change affects the Python linter but not the C++ linter as we
use a customized version of cpplint that we cannot get using pip.
Review: https://reviews.apache.org/r/63581/
{noformat}
> Add a javascript linter to the webui.
> -
>
> Key: MESOS-7924
> URL: https://issues.apache.org/jira/browse/MESOS-7924
> Project: Mesos
> Issue Type: Improvement
> Components: webui
>Reporter: Benjamin Mahler
>Assignee: Armand Grillet
> Labels: tech-debt
> Fix For: 1.5.0
>
>
> As far as I can tell, javascript linters (e.g. ESLint) help catch some
> functional errors as well, for example, we've made some "strict" mistakes a
> few times that ESLint can catch: MESOS-6624, MESOS-7912.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-6390) Ensure Python support scripts are linted
[
https://issues.apache.org/jira/browse/MESOS-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192922#comment-16192922
]
Kevin Klues commented on MESOS-6390:
{noformat}
commit 38af943ad6a2f8e7a47148ab6637692978545500
Author: Armand Grillet
Date: Thu Oct 5 16:03:51 2017 +0200
Added support/ to the list of the linted directories.
By adding the support directory to 'mesos-style.py', we make sure
that all our support scripts follow the same coding style that the
rest of our Python codebase uses.
We also added invalid-name to 'pylint.config' as all the Python files
in support/ use dashes instead of underscores and we also added
'file-ignored' as we do not lint 'support/post-reviews.py' yet.
Review: https://reviews.apache.org/r/62788/
{noformat}
> Ensure Python support scripts are linted
>
>
> Key: MESOS-6390
> URL: https://issues.apache.org/jira/browse/MESOS-6390
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Bannier
>Assignee: Armand Grillet
> Labels: newbie, python
> Fix For: 1.5.0
>
>
> Currently {{support/mesos-style.py}} does not lint files under {{support/}}.
> This is mostly due to the fact that these scripts are too inconsistent
> style-wise that they wouldn't even pass the linter now.
> We should clean up all Python scripts under {{support/}} so they pass the
> Python linter, and activate that directory in the linter for future
> additions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (MESOS-8025) Update the master field in the new CLI config to accept a URL instead of an
Kevin Klues created MESOS-8025: -- Summary: Update the master field in the new CLI config to accept a URL instead of an Key: MESOS-8025 URL: https://issues.apache.org/jira/browse/MESOS-8025 Project: Mesos Issue Type: Improvement Components: cli Environment: This will be useful in cases where the master is behind a proxy or when the master is sitting directly on port 80. Reporter: Kevin Klues Assignee: Armand Grillet -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-8024) Add Mesos CLI command to list agents
[
https://issues.apache.org/jira/browse/MESOS-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182529#comment-16182529
]
Kevin Klues commented on MESOS-8024:
{noformat}
commit 1270703291a132d8d959d71bf99e4dfe4cf4292e
Author: Armand Grillet
Date: Wed Sep 27 15:02:04 2017 +0200
Added 'mesos agent list' command to CLI.
This command displays the agents in a cluster by
reaching the slaves endpoint of a master.
Review: https://reviews.apache.org/r/62065/
{noformat}
> Add Mesos CLI command to list agents
>
>
> Key: MESOS-8024
> URL: https://issues.apache.org/jira/browse/MESOS-8024
> Project: Mesos
> Issue Type: Task
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>
> We should have a command listing the agents in a Mesos cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7840) Add Mesos CLI command to list active tasks
[
https://issues.apache.org/jira/browse/MESOS-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180899#comment-16180899
]
Kevin Klues commented on MESOS-7840:
{noformat}
commit 3d4c1cec5ee655654e24c75509bda594c7980f05
Author: Armand Grillet
Date: Tue Sep 26 16:57:24 2017 +0200
Added 'mesos task list' command to CLI.
This command displays the active tasks in a cluster
by hitting the tasks endpoint of a master.
Review: https://reviews.apache.org/r/61213/
{noformat}
> Add Mesos CLI command to list active tasks
> --
>
> Key: MESOS-7840
> URL: https://issues.apache.org/jira/browse/MESOS-7840
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>
> We need to add a command to list all the tasks running in a Mesos cluster by
> checking the endpoint {{/tasks}} and reporting the results.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7840) Add Mesos CLI command to list active tasks
[
https://issues.apache.org/jira/browse/MESOS-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180867#comment-16180867
]
Kevin Klues commented on MESOS-7840:
{noformat}
commit ebc277c324b6a88aa86c1f1fcb3a47876d37c8a2
Author: Armand Grillet
Date: Tue Sep 26 16:38:36 2017 +0200
Added default configuration file for CLI tests.
This will be used by future tests, it overwrites
the configuration file defined by the user.
Review: https://reviews.apache.org/r/61211/
{noformat}
> Add Mesos CLI command to list active tasks
> --
>
> Key: MESOS-7840
> URL: https://issues.apache.org/jira/browse/MESOS-7840
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>
> We need to add a command to list all the tasks running in a Mesos cluster by
> checking the endpoint {{/tasks}} and reporting the results.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7284) Allow Mesos CLI to take masters IP
[
https://issues.apache.org/jira/browse/MESOS-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180862#comment-16180862
]
Kevin Klues commented on MESOS-7284:
{noformat}
commit b35e8169d200d41118793993f5007e2c573ece06
Author: Armand Grillet
Date: Tue Sep 26 16:35:42 2017 +0200
Added 'master' key as an acceptable key in the CLI's config.toml.
This key is a field that has to be composed of
an `address` or `zookeeper` field, but not both.
Review: https://reviews.apache.org/r/60088/
{noformat}
> Allow Mesos CLI to take masters IP
> --
>
> Key: MESOS-7284
> URL: https://issues.apache.org/jira/browse/MESOS-7284
> Project: Mesos
> Issue Type: Task
>Reporter: Avinash Sridharan
>Assignee: Armand Grillet
>
> Allow the Mesos CLI to take master IPs. This will allow the CLI to send HTTP
> requests to one master in a cluster with one or multiple ones.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7840) Add Mesos CLI command to list active tasks
[
https://issues.apache.org/jira/browse/MESOS-7840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180857#comment-16180857
]
Kevin Klues commented on MESOS-7840:
{noformat}
commit 016a565f3c8f0b2028fa6e70816531bc3b13bcc5 (HEAD -> master,
upstream/master)
Author: Armand Grillet
Date: Tue Sep 26 16:32:28 2017 +0200
Added CLI utility function to verify addresses.
This will be used by future plugins.
Review: https://reviews.apache.org/r/61212/
{noformat}
> Add Mesos CLI command to list active tasks
> --
>
> Key: MESOS-7840
> URL: https://issues.apache.org/jira/browse/MESOS-7840
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Armand Grillet
>Assignee: Armand Grillet
>
> We need to add a command to list all the tasks running in a Mesos cluster by
> checking the endpoint {{/tasks}} and reporting the results.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (MESOS-8012) Support Znode paths for masters in the new CLI
Kevin Klues created MESOS-8012:
--
Summary: Support Znode paths for masters in the new CLI
Key: MESOS-8012
URL: https://issues.apache.org/jira/browse/MESOS-8012
Project: Mesos
Issue Type: Improvement
Components: cli
Reporter: Kevin Klues
Assignee: Armand Grillet
Right now the new Mesos CLI only works in single master mode with a single
master IP and port. We should add support for finding the mesos leader in HA
mode by hitting a set of zk instances similar to how {{mesos-resolve}} works.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7730) CUDA not working anymore on 1.3.0
[
https://issues.apache.org/jira/browse/MESOS-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109070#comment-16109070
]
Kevin Klues commented on MESOS-7730:
https://reviews.apache.org/r/59140/
> CUDA not working anymore on 1.3.0
> -
>
> Key: MESOS-7730
> URL: https://issues.apache.org/jira/browse/MESOS-7730
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Affects Versions: 1.3.0
>Reporter: Adam Cecile
>Assignee: Kevin Klues
> Fix For: 1.3.1
>
>
> Hello,
> My docker container using CUDA do not detect it anymore.
> Here the tensorflow output with 1.2.1:
> {noformat}
> I0628 12:39:45.505900 16309 exec.cpp:162] Version: 1.2.1
> I0628 12:39:45.508358 16301 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcuda.so.1 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcurand.so.8.0 locally
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE3 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.1 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use FMA instructions, but these are available on your
> machine and could speed up CPU computations.
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with
> properties:
> name: GeForce GTX 1080
> major: 6 minor: 1 memoryClockRate (GHz) 1.7335
> pciBusID :82:00.0
> Total memory: 7.92GiB
> Free memory: 7.81GiB
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow
> device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id:
> :82:00.0)
> {noformat}
> And with 1.3.0
> {noformat}
> I0628 12:40:30.833947 16854 exec.cpp:162] Version: 1.3.0
> I0628 12:40:30.836612 16845 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library
> libcuda.so.1. LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname:
> zelda.service.earthlab.lu
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported
> version is: Not found: was unable to find libcuda.so DSO loaded into this
> program
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version
> file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 375.66 Mon
> May 1 15:29:16 PDT 2017
> GCC version: gcc version 4.9.2 (Debian 4.9.2-10)
> """
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported
> version is: 375.66.0
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1065] LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1066] failed to find
> libcuda.so on this system: Failed precondition: could not dlopen DSO:
> libcuda.so.1; dlerror: libcuda.so.1: cannot open shared object file: No such
> file or directory
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcurand.so.8.0 locally
> W
[jira] [Commented] (MESOS-7823) Reorganize the new Mesos CLI to live under src/python
[
https://issues.apache.org/jira/browse/MESOS-7823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097828#comment-16097828
]
Kevin Klues commented on MESOS-7823:
{noformat}
commit 09fa758b89d3149ae8910d84377a2656064903a6
Author: Kevin Klues
Date: Sun Jul 23 16:42:55 2017 -0700
Moved new Mesos CLI to 'src/python/cli_new'.
This change allows us to have all Python files under
only two directories : 'src/python/' and 'support/'. This
will improve the structure of our Python code in the future
by sharing configuration files and virtual environments.
Review: https://reviews.apache.org/r/60899
{noformat}
> Reorganize the new Mesos CLI to live under src/python
> -
>
> Key: MESOS-7823
> URL: https://issues.apache.org/jira/browse/MESOS-7823
> Project: Mesos
> Issue Type: Improvement
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (MESOS-7823) Reorganize the new Mesos CLI to live under src/python
Kevin Klues created MESOS-7823: -- Summary: Reorganize the new Mesos CLI to live under src/python Key: MESOS-7823 URL: https://issues.apache.org/jira/browse/MESOS-7823 Project: Mesos Issue Type: Improvement Components: cli Reporter: Kevin Klues Assignee: Armand Grillet -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MESOS-7310) Implement a separate python client library for the new cli
[
https://issues.apache.org/jira/browse/MESOS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097824#comment-16097824
]
Kevin Klues commented on MESOS-7310:
{noformat}
commit 456a0ac34deb9321c20f57b7ed74fb61253999c3
Author: Kevin Klues klue...@gmail.com
Date: Sun Jul 23 16:21:36 2017 -0700
Added package and test infrastructure for 'src/python/lib/mesos'.
Part of MESOS-7310, this patch adds the test infrastructure necessary
for reliably running unit tests for the mesos package located under
'src/python/lib/mesos'.
'setup.py' is added under src/python/lib to define the Python package.
'tox.ini' is added under the same dir to enable automated unit tests via
the command tox, which runs tests via pytest.
Review: https://reviews.apache.org/r/60719/
{noformat}
{noformat}
commit 2d19111e4852aed25161e4549ff704f9d4c2f37b
Author: Kevin Klues klue...@gmail.com
Date: Sun Jul 23 16:27:58 2017 -0700
Added extra directives to 'src/python/.gitignore'.
The new ignored files are in support of adding a new 'lib/mesos'
directory that contains shared mesos library code. The '.virtualenv'
folder is the standard python virtual environment we use to build,
run, and test the python code under 'lib/mesos'. The '.tox' and
'.coverage' files are files generated as part of a unit test runner
called 'tox'. The 'build' and 'mesos.egg-info' folders are standard
folders created when running 'python setup.py build' on the new
'lib/mesos' python package.
Review: https://reviews.apache.org/r/60719/
{noformat}
> Implement a separate python client library for the new cli
> --
>
> Key: MESOS-7310
> URL: https://issues.apache.org/jira/browse/MESOS-7310
> Project: Mesos
> Issue Type: Task
> Components: cli
>Affects Versions: 1.3.0
>Reporter: Eric Chung
>Assignee: Eric Chung
>
> cli_new in its current form is very difficult to package due to the following
> reasons:
> 1. src/cli_new/lib/mesos imports plugins using relative imports, which fails
> if it is built into a pip package
> 2. there is no setup.py script which defines what should be installed
> 3. plugins/tests are unnecessarily included in the package, which are things
> consumers of the package shouldn’t be able to import
> having such a package will allow external consumers to be able to add
> application-specific wrappers on it, e.g. integration with ACL libraries of
> their choice.
> The plan as discussed will create a `mesos` package under `src/python/lib`,
> potentially including a `setup.py` for building the package into a PyPI
> package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-7730) CUDA not working anymore on 1.3.0
[
https://issues.apache.org/jira/browse/MESOS-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096554#comment-16096554
]
Kevin Klues commented on MESOS-7730:
Hmm. I'm not sure what would have changed to cause this error. I'll dig into it
soon.
> CUDA not working anymore on 1.3.0
> -
>
> Key: MESOS-7730
> URL: https://issues.apache.org/jira/browse/MESOS-7730
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Affects Versions: 1.3.0
>Reporter: Adam Cecile
>Assignee: Kevin Klues
> Fix For: 1.2.1
>
>
> Hello,
> My docker container using CUDA do not detect it anymore.
> Here the tensorflow output with 1.2.1:
> {noformat}
> I0628 12:39:45.505900 16309 exec.cpp:162] Version: 1.2.1
> I0628 12:39:45.508358 16301 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcuda.so.1 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcurand.so.8.0 locally
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE3 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.1 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use FMA instructions, but these are available on your
> machine and could speed up CPU computations.
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with
> properties:
> name: GeForce GTX 1080
> major: 6 minor: 1 memoryClockRate (GHz) 1.7335
> pciBusID :82:00.0
> Total memory: 7.92GiB
> Free memory: 7.81GiB
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow
> device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id:
> :82:00.0)
> {noformat}
> And with 1.3.0
> {noformat}
> I0628 12:40:30.833947 16854 exec.cpp:162] Version: 1.3.0
> I0628 12:40:30.836612 16845 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library
> libcuda.so.1. LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname:
> zelda.service.earthlab.lu
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported
> version is: Not found: was unable to find libcuda.so DSO loaded into this
> program
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version
> file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 375.66 Mon
> May 1 15:29:16 PDT 2017
> GCC version: gcc version 4.9.2 (Debian 4.9.2-10)
> """
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported
> version is: 375.66.0
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1065] LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1066] failed to find
> libcuda.so on this system: Failed precondition: could not dlopen DSO:
> libcuda.so.1; dlerror: libcuda.so.1: cannot open shared object file: No such
> file or directory
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened
[jira] [Assigned] (MESOS-7730) CUDA not working anymore on 1.3.0
[
https://issues.apache.org/jira/browse/MESOS-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues reassigned MESOS-7730:
--
Assignee: Kevin Klues
> CUDA not working anymore on 1.3.0
> -
>
> Key: MESOS-7730
> URL: https://issues.apache.org/jira/browse/MESOS-7730
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Affects Versions: 1.3.0
>Reporter: Adam Cecile
>Assignee: Kevin Klues
> Fix For: 1.2.1
>
>
> Hello,
> My docker container using CUDA do not detect it anymore.
> Here the tensorflow output with 1.2.1:
> {noformat}
> I0628 12:39:45.505900 16309 exec.cpp:162] Version: 1.2.1
> I0628 12:39:45.508358 16301 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcuda.so.1 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcurand.so.8.0 locally
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE3 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.1 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use SSE4.2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use AVX2 instructions, but these are available on your
> machine and could speed up CPU computations.
> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library
> wasn't compiled to use FMA instructions, but these are available on your
> machine and could speed up CPU computations.
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with
> properties:
> name: GeForce GTX 1080
> major: 6 minor: 1 memoryClockRate (GHz) 1.7335
> pciBusID :82:00.0
> Total memory: 7.92GiB
> Free memory: 7.81GiB
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow
> device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id:
> :82:00.0)
> {noformat}
> And with 1.3.0
> {noformat}
> I0628 12:40:30.833947 16854 exec.cpp:162] Version: 1.3.0
> I0628 12:40:30.836612 16845 exec.cpp:237] Executor registered on agent
> 84c99d0b-8551-4f30-a9bc-6c1edbf7c18c-S1
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcublas.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcudnn.so.5 locally
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcufft.so.8.0 locally
> I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library
> libcuda.so.1. LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname:
> zelda.service.earthlab.lu
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported
> version is: Not found: was unable to find libcuda.so DSO loaded into this
> program
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version
> file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 375.66 Mon
> May 1 15:29:16 PDT 2017
> GCC version: gcc version 4.9.2 (Debian 4.9.2-10)
> """
> I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported
> version is: 375.66.0
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1065] LD_LIBRARY_PATH:
> I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1066] failed to find
> libcuda.so on this system: Failed precondition: could not dlopen DSO:
> libcuda.so.1; dlerror: libcuda.so.1: cannot open shared object file: No such
> file or directory
> I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA
> library libcurand.so.8.0 locally
> W tensorflow/core/platform/cpu_feature_guard.cc:45]
[jira] [Commented] (MESOS-7310) Implement a separate python client library for the new cli
[
https://issues.apache.org/jira/browse/MESOS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077041#comment-16077041
]
Kevin Klues commented on MESOS-7310:
{noformat}
commit 1b75c37cff41f1d6955e1b202f73193b57ea636f
Author: Eric Chung
Date: Thu Jul 6 10:54:37 2017 -0700
Setup new directory for importable python libs in src/python.
Part of MESOS-7310, this patch adds a new directory
(src/python/lib/mesos), which will be importable via 'import mesos'
from the cli.
Review: https://reviews.apache.org/r/58394/
{noformat}
> Implement a separate python client library for the new cli
> --
>
> Key: MESOS-7310
> URL: https://issues.apache.org/jira/browse/MESOS-7310
> Project: Mesos
> Issue Type: Task
> Components: cli
>Affects Versions: 1.3.0
>Reporter: Eric Chung
>Assignee: Eric Chung
>
> cli_new in its current form is very difficult to package due to the following
> reasons:
> 1. src/cli_new/lib/mesos imports plugins using relative imports, which fails
> if it is built into a pip package
> 2. there is no setup.py script which defines what should be installed
> 3. plugins/tests are unnecessarily included in the package, which are things
> consumers of the package shouldn’t be able to import
> having such a package will allow external consumers to be able to add
> application-specific wrappers on it, e.g. integration with ACL libraries of
> their choice.
> The plan as discussed will create a `mesos` package under `src/python/lib`,
> potentially including a `setup.py` for building the package into a PyPI
> package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (MESOS-7585) Added 'mesos config show' command to the new Mesos CLI.
Kevin Klues created MESOS-7585: -- Summary: Added 'mesos config show' command to the new Mesos CLI. Key: MESOS-7585 URL: https://issues.apache.org/jira/browse/MESOS-7585 Project: Mesos Issue Type: Improvement Components: cli Reporter: Kevin Klues Assignee: Armand Grillet This command should display the contents of the user-defined config.toml file. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7584) ASF Jenkins build errors out on missing 'python-six' dependency
[
https://issues.apache.org/jira/browse/MESOS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7584:
---
Shepherd: Till Toenshoff (was: Kevin Klues)
> ASF Jenkins build errors out on missing 'python-six' dependency
> ---
>
> Key: MESOS-7584
> URL: https://issues.apache.org/jira/browse/MESOS-7584
> Project: Mesos
> Issue Type: Bug
> Components: build
>Reporter: Kevin Klues
>Assignee: Kevin Klues
>Priority: Blocker
>
> {noformat}
> [ RUN ] ExamplesTest.PythonFramework
> Using temporary directory '/tmp/ExamplesTest_PythonFramework_T0LN9L'
> Traceback (most recent call last):
> File "/mesos/mesos-1.4.0/_build/../src/examples/python/test_framework.py",
> line 24, in
> from mesos.interface import mesos_pb2
> File "build/bdist.linux-x86_64/egg/mesos/interface/mesos_pb2.py", line 7,
> in
> File
> "/mesos/mesos-1.4.0/_build/3rdparty/protobuf-3.3.0/python/google/protobuf/descriptor.py",
> line 37, in
> import six
> ImportError: No module named six
> ../../src/tests/script.cpp:83: Failure
> Failed
> python_framework_test.sh exited with status 1
> [ FAILED ] ExamplesTest.PythonFramework (149 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-7584) ASF Jenkins build errors out on missing 'pyton-six' dependency
[
https://issues.apache.org/jira/browse/MESOS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028619#comment-16028619
]
Kevin Klues commented on MESOS-7584:
{noformat}
commit 8ea46990999b61d7d79a23b582b5eb7d586bd164
Author: Kevin Klues
Date: Mon May 29 12:04:23 2017 -0700
Added 'python-six' to the list of build dependencies.
Review: https://reviews.apache.org/r/59632
{noformat}
> ASF Jenkins build errors out on missing 'pyton-six' dependency
> --
>
> Key: MESOS-7584
> URL: https://issues.apache.org/jira/browse/MESOS-7584
> Project: Mesos
> Issue Type: Bug
> Components: build
>Reporter: Kevin Klues
>Assignee: Kevin Klues
>Priority: Blocker
>
> {noformat}
> [ RUN ] ExamplesTest.PythonFramework
> Using temporary directory '/tmp/ExamplesTest_PythonFramework_T0LN9L'
> Traceback (most recent call last):
> File "/mesos/mesos-1.4.0/_build/../src/examples/python/test_framework.py",
> line 24, in
> from mesos.interface import mesos_pb2
> File "build/bdist.linux-x86_64/egg/mesos/interface/mesos_pb2.py", line 7,
> in
> File
> "/mesos/mesos-1.4.0/_build/3rdparty/protobuf-3.3.0/python/google/protobuf/descriptor.py",
> line 37, in
> import six
> ImportError: No module named six
> ../../src/tests/script.cpp:83: Failure
> Failed
> python_framework_test.sh exited with status 1
> [ FAILED ] ExamplesTest.PythonFramework (149 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7584) ASF Jenkins build errors out on missing 'python-six' dependency
[
https://issues.apache.org/jira/browse/MESOS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7584:
---
Summary: ASF Jenkins build errors out on missing 'python-six' dependency
(was: ASF Jenkins build errors out on missing 'pyton-six' dependency)
> ASF Jenkins build errors out on missing 'python-six' dependency
> ---
>
> Key: MESOS-7584
> URL: https://issues.apache.org/jira/browse/MESOS-7584
> Project: Mesos
> Issue Type: Bug
> Components: build
>Reporter: Kevin Klues
>Assignee: Kevin Klues
>Priority: Blocker
>
> {noformat}
> [ RUN ] ExamplesTest.PythonFramework
> Using temporary directory '/tmp/ExamplesTest_PythonFramework_T0LN9L'
> Traceback (most recent call last):
> File "/mesos/mesos-1.4.0/_build/../src/examples/python/test_framework.py",
> line 24, in
> from mesos.interface import mesos_pb2
> File "build/bdist.linux-x86_64/egg/mesos/interface/mesos_pb2.py", line 7,
> in
> File
> "/mesos/mesos-1.4.0/_build/3rdparty/protobuf-3.3.0/python/google/protobuf/descriptor.py",
> line 37, in
> import six
> ImportError: No module named six
> ../../src/tests/script.cpp:83: Failure
> Failed
> python_framework_test.sh exited with status 1
> [ FAILED ] ExamplesTest.PythonFramework (149 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7584) ASF Jenkins build errors out on missing 'pyton-six' dependency
Kevin Klues created MESOS-7584:
--
Summary: ASF Jenkins build errors out on missing 'pyton-six'
dependency
Key: MESOS-7584
URL: https://issues.apache.org/jira/browse/MESOS-7584
Project: Mesos
Issue Type: Bug
Components: build
Reporter: Kevin Klues
Assignee: Kevin Klues
Priority: Blocker
{noformat}
[ RUN ] ExamplesTest.PythonFramework
Using temporary directory '/tmp/ExamplesTest_PythonFramework_T0LN9L'
Traceback (most recent call last):
File "/mesos/mesos-1.4.0/_build/../src/examples/python/test_framework.py",
line 24, in
from mesos.interface import mesos_pb2
File "build/bdist.linux-x86_64/egg/mesos/interface/mesos_pb2.py", line 7, in
File
"/mesos/mesos-1.4.0/_build/3rdparty/protobuf-3.3.0/python/google/protobuf/descriptor.py",
line 37, in
import six
ImportError: No module named six
../../src/tests/script.cpp:83: Failure
Failed
python_framework_test.sh exited with status 1
[ FAILED ] ExamplesTest.PythonFramework (149 ms)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-7582) Add Config class to manage the Mesos CLI config file.
[
https://issues.apache.org/jira/browse/MESOS-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028292#comment-16028292
]
Kevin Klues commented on MESOS-7582:
{noformat}
commit f6c2ee0df8a3170d6cd211211c5ab2e50802efde
Author: Kevin Klues
Date: Mon May 29 05:28:27 2017 -0700
Moved 'current_word' calculation for autocomplete in the new Mesos CLI.
{noformat}
{noformat}
commit 8740ce8d3adc723162350f35b5508d811214ac7e
Author: Armand Grillet
Date: Mon May 29 05:12:33 2017 -0700
Added Config class to manage the config file in the new Mesos CLI.
This new class simplifies the management of the configuration file
given by the user; it loads the TOML file on initialization and
has one method for each element that the user can set.
This new class and its associated content is also passed to the
plugins at initialization so that they can read the user configuration
and use it.
Review: https://reviews.apache.org/r/59177/
{noformat}
> Add Config class to manage the Mesos CLI config file.
> -
>
> Key: MESOS-7582
> URL: https://issues.apache.org/jira/browse/MESOS-7582
> Project: Mesos
> Issue Type: Task
> Components: cli
>Reporter: Kevin Klues
>Assignee: Armand Grillet
>
> This new class will simplify the management of the configuration file given
> by the user; it will load the TOML file on initialization and have one method
> for each element that the user can set.
> This new class and its associated content should also be passed to the
> plugins at initialization so that they can read the user configuration and
> use it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7582) Add Config class to manage the Mesos CLI config file.
Kevin Klues created MESOS-7582: -- Summary: Add Config class to manage the Mesos CLI config file. Key: MESOS-7582 URL: https://issues.apache.org/jira/browse/MESOS-7582 Project: Mesos Issue Type: Task Components: cli Reporter: Kevin Klues Assignee: Armand Grillet This new class will simplify the management of the configuration file given by the user; it will load the TOML file on initialization and have one method for each element that the user can set. This new class and its associated content should also be passed to the plugins at initialization so that they can read the user configuration and use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7579) Deprecate GPU_RESOURCES capability and master flag `--filter-gpu-resources={true|false}`
Kevin Klues created MESOS-7579:
--
Summary: Deprecate GPU_RESOURCES capability and master flag
`--filter-gpu-resources={true|false}`
Key: MESOS-7579
URL: https://issues.apache.org/jira/browse/MESOS-7579
Project: Mesos
Issue Type: Task
Components: allocation, gpu
Reporter: Kevin Klues
Once we reach Mesos 2.0, we should completely remove the GPU_RESOURCES
capability and the corresponding {{--filter-gpu-resources}} that controls
whether the allocator honors this capability or not.
It will have been deprecated once support for {{dynamic reservations}},
{{hierarchical roles}}, and {{support for reservations to multiple roles}} has
landed. The JIRA tracking these features as blockers to this ticket are linked
below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7577) Remove GPU_RESOURCES capability and master flag `--filter-gpu-resources={true|false}`
[
https://issues.apache.org/jira/browse/MESOS-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7577:
---
Target Version/s: 2.1.0
Description:
Once we reach Mesos 2.0, we should completely remove the GPU_RESOURCES
capability and the corresponding {{--filter-gpu-resources}} that controls
whether the allocator honors this capability or not.
It will have been deprecated once support for {{dynamic reservations}},
{{hierarchical roles}}, and {{support for reservations to multiple roles}} has
landed. The JIRA tracking these features as blockers to this ticket are linked
below.
was:
This flag was added as a temporary way to to enable / disable honoring the
GPU_RESOURCES framework capability. We should remove it once we have better
support for achieving the same functionality that the GPU_RESOURCES capability
gives you.
This support relies on dynamic reservations, hierarchical roles, and support
for reservations to multiple roles (an unyet implemented feature). The JIRA
tracking these features as blockers to this ticket are linked below.
> Remove GPU_RESOURCES capability and master flag
> `--filter-gpu-resources={true|false}`
> -
>
> Key: MESOS-7577
> URL: https://issues.apache.org/jira/browse/MESOS-7577
> Project: Mesos
> Issue Type: Task
> Components: allocation, gpu
>Reporter: Kevin Klues
>
> Once we reach Mesos 2.0, we should completely remove the GPU_RESOURCES
> capability and the corresponding {{--filter-gpu-resources}} that controls
> whether the allocator honors this capability or not.
> It will have been deprecated once support for {{dynamic reservations}},
> {{hierarchical roles}}, and {{support for reservations to multiple roles}}
> has landed. The JIRA tracking these features as blockers to this ticket are
> linked below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7577) Remove GPU_RESOURCES capability and remove master flag `--filter-gpu-resources={true|false}`
[
https://issues.apache.org/jira/browse/MESOS-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7577:
---
Summary: Remove GPU_RESOURCES capability and remove master flag
`--filter-gpu-resources={true|false}` (was: Remove master flag
`--filter-gpu-resources={true|false}`)
> Remove GPU_RESOURCES capability and remove master flag
> `--filter-gpu-resources={true|false}`
>
>
> Key: MESOS-7577
> URL: https://issues.apache.org/jira/browse/MESOS-7577
> Project: Mesos
> Issue Type: Task
> Components: allocation, gpu
>Reporter: Kevin Klues
>
> This flag was added as a temporary way to to enable / disable honoring the
> GPU_RESOURCES framework capability. We should remove it once we have better
> support for achieving the same functionality that the GPU_RESOURCES
> capability gives you.
> This support relies on dynamic reservations, hierarchical roles, and support
> for reservations to multiple roles (an unyet implemented feature). The JIRA
> tracking these features as blockers to this ticket are linked below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7577) Remove GPU_RESOURCES capability and master flag `--filter-gpu-resources={true|false}`
[
https://issues.apache.org/jira/browse/MESOS-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7577:
---
Summary: Remove GPU_RESOURCES capability and master flag
`--filter-gpu-resources={true|false}` (was: Remove GPU_RESOURCES capability
and remove master flag `--filter-gpu-resources={true|false}`)
> Remove GPU_RESOURCES capability and master flag
> `--filter-gpu-resources={true|false}`
> -
>
> Key: MESOS-7577
> URL: https://issues.apache.org/jira/browse/MESOS-7577
> Project: Mesos
> Issue Type: Task
> Components: allocation, gpu
>Reporter: Kevin Klues
>
> This flag was added as a temporary way to to enable / disable honoring the
> GPU_RESOURCES framework capability. We should remove it once we have better
> support for achieving the same functionality that the GPU_RESOURCES
> capability gives you.
> This support relies on dynamic reservations, hierarchical roles, and support
> for reservations to multiple roles (an unyet implemented feature). The JIRA
> tracking these features as blockers to this ticket are linked below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7577) Remove master flag `--filter-gpu-resources={true|false}`
Kevin Klues created MESOS-7577:
--
Summary: Remove master flag `--filter-gpu-resources={true|false}`
Key: MESOS-7577
URL: https://issues.apache.org/jira/browse/MESOS-7577
Project: Mesos
Issue Type: Task
Components: allocation, gpu
Reporter: Kevin Klues
This flag was added as a temporary way to to enable / disable honoring the
GPU_RESOURCES framework capability. We should remove it once we have better
support for achieving the same functionality that the GPU_RESOURCES capability
gives you.
This support relies on dynamic reservations, hierarchical roles, and support
for reservations to multiple roles (an unyet implemented feature). The JIRA
tracking these features as blockers to this ticket are linked below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7576) Add master flag `--filter-gpu-resources={true|false}`
[
https://issues.apache.org/jira/browse/MESOS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7576:
---
Description:
Per the email thread below, we are adding a new flag on the master called
{{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
framework capability.
https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
When set to {{true}}, this flag will cause the mesos master to continue to
function as it does today. That is, it will filter offers containing GPU
resources and only send them to frameworks that opt into the {{GPU_RESOURCES}}
framework capability. When set to {{false}}, this flag will cause the master to
*not* filter offers containing GPU resources, and indiscriminately send them to
all frameworks whether they set the {{GPU_RESOURCES}} capability or not.
This is a temporary flag that will eventually be removed. We will remove it
once we have better support for achieving the same functionality that the
{{GPU_RESOURCES}} capability gives you.
As described in the email, this support relies on {{dynamic reservations}},
{{hierarchical roles}}, and support for {{reservations to multiple roles}} (an
unyet implemented feature). The JIRA tracking these features are linked below.
was:
Per the email thread below, we are adding a new flag on the master called
{{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
framework capability.
https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
When set to {{true}}, this flag will cause the mesos master to continue to
function as it does today. That is, it will filter offers containing GPU
resources and only send them to frameworks that opt into the {{GPU_RESOURCES}}
framework capability. When set to {{false}}, this flag will cause the master to
*not* filter offers containing GPU resources, and indiscriminately send them to
all frameworks whether they set the {{GPU_RESOURCES}} capability or not.
This is a temporary flag that will eventually be removed. We will remove it
once we have better support for achieving the same functionality that the
{{GPU_RESOURCES}} capability gives you.
As described in the email, this support relies {{dynamic reservations}},
{{hierarchical roles}}, and support for {{reservations to multiple roles}} (an
unyet implemented feature). The JIRA tracking these features are linked below.
> Add master flag `--filter-gpu-resources={true|false}`
> -
>
> Key: MESOS-7576
> URL: https://issues.apache.org/jira/browse/MESOS-7576
> Project: Mesos
> Issue Type: Task
> Components: gpu
>Affects Versions: 1.2.0
>Reporter: Kevin Klues
>Assignee: Kevin Klues
>
> Per the email thread below, we are adding a new flag on the master called
> {{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
> framework capability.
> https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
> When set to {{true}}, this flag will cause the mesos master to continue to
> function as it does today. That is, it will filter offers containing GPU
> resources and only send them to frameworks that opt into the
> {{GPU_RESOURCES}} framework capability. When set to {{false}}, this flag will
> cause the master to *not* filter offers containing GPU resources, and
> indiscriminately send them to all frameworks whether they set the
> {{GPU_RESOURCES}} capability or not.
> This is a temporary flag that will eventually be removed. We will remove it
> once we have better support for achieving the same functionality that the
> {{GPU_RESOURCES}} capability gives you.
> As described in the email, this support relies on {{dynamic reservations}},
> {{hierarchical roles}}, and support for {{reservations to multiple roles}}
> (an unyet implemented feature). The JIRA tracking these features are linked
> below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7576) Add master flag `--filter-gpu-resources={true|false}`
[
https://issues.apache.org/jira/browse/MESOS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Klues updated MESOS-7576:
---
Description:
Per the email thread below, we are adding a new flag on the master called
{{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
framework capability.
https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
When set to {{true}}, this flag will cause the mesos master to continue to
function as it does today. That is, it will filter offers containing GPU
resources and only send them to frameworks that opt into the {{GPU_RESOURCES}}
framework capability. When set to {{false}}, this flag will cause the master to
*not* filter offers containing GPU resources, and indiscriminately send them to
all frameworks whether they set the {{GPU_RESOURCES}} capability or not.
This is a temporary flag that will eventually be removed. We will remove it
once we have better support for achieving the same functionality that the
{{GPU_RESOURCES}} capability gives you.
As described in the email, this support relies {{dynamic reservations}},
{{hierarchical roles}}, and support for {{reservations to multiple roles}} (an
unyet implemented feature). The JIRA tracking these features are linked below.
was:
Per the email thread below, we are adding a new flag on the master called
{{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
framework capability.
https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
When set to {{true}}, this flag will cause the mesos master to continue to
function as it does today. That is, it will filter offers containing GPU
resources and only send them to frameworks that opt into the {{GPU_RESOURCES}}
framework capability. When set to {{false}}, this flag will cause the master to
*not* filter offers containing GPU resources, and indiscriminately send them to
all frameworks whether they set the {{GPU_RESOURCES}} capability or not.
This is a temporary flag that will eventually be removed. We will remove it
once we have better support for achieving the same functionality that the
{{GPU_RESOURCES}} capability gives you.
As described in the email, this support relies {{reservations}}, {{hierarchical
roles}}, and support for {{reservations to multiple roles}} (an unyet
implemented feature). The JIRA tracking these features are linked below.
> Add master flag `--filter-gpu-resources={true|false}`
> -
>
> Key: MESOS-7576
> URL: https://issues.apache.org/jira/browse/MESOS-7576
> Project: Mesos
> Issue Type: Task
> Components: gpu
>Affects Versions: 1.2.0
>Reporter: Kevin Klues
>Assignee: Kevin Klues
>
> Per the email thread below, we are adding a new flag on the master called
> {{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
> framework capability.
> https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
> When set to {{true}}, this flag will cause the mesos master to continue to
> function as it does today. That is, it will filter offers containing GPU
> resources and only send them to frameworks that opt into the
> {{GPU_RESOURCES}} framework capability. When set to {{false}}, this flag will
> cause the master to *not* filter offers containing GPU resources, and
> indiscriminately send them to all frameworks whether they set the
> {{GPU_RESOURCES}} capability or not.
> This is a temporary flag that will eventually be removed. We will remove it
> once we have better support for achieving the same functionality that the
> {{GPU_RESOURCES}} capability gives you.
> As described in the email, this support relies {{dynamic reservations}},
> {{hierarchical roles}}, and support for {{reservations to multiple roles}}
> (an unyet implemented feature). The JIRA tracking these features are linked
> below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7576) Add master flag `--filter-gpu-resources={true|false}`
Kevin Klues created MESOS-7576:
--
Summary: Add master flag `--filter-gpu-resources={true|false}`
Key: MESOS-7576
URL: https://issues.apache.org/jira/browse/MESOS-7576
Project: Mesos
Issue Type: Task
Components: gpu
Affects Versions: 1.2.0
Reporter: Kevin Klues
Assignee: Kevin Klues
Per the email thread below, we are adding a new flag on the master called
{{--filter-gpu-resources}} to enable / disable honoring the {{GPU_RESOURCES}}
framework capability.
https://www.mail-archive.com/dev@mesos.apache.org/msg37571.html
When set to {{true}}, this flag will cause the mesos master to continue to
function as it does today. That is, it will filter offers containing GPU
resources and only send them to frameworks that opt into the {{GPU_RESOURCES}}
framework capability. When set to {{false}}, this flag will cause the master to
*not* filter offers containing GPU resources, and indiscriminately send them to
all frameworks whether they set the {{GPU_RESOURCES}} capability or not.
This is a temporary flag that will eventually be removed. We will remove it
once we have better support for achieving the same functionality that the
{{GPU_RESOURCES}} capability gives you.
As described in the email, this support relies {{reservations}}, {{hierarchical
roles}}, and support for {{reservations to multiple roles}} (an unyet
implemented feature). The JIRA tracking these features are linked below.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (MESOS-7375) provide additional insight for framework developers re: GPU_RESOURCES capability
[
https://issues.apache.org/jira/browse/MESOS-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983645#comment-15983645
]
Kevin Klues edited comment on MESOS-7375 at 4/25/17 9:24 PM:
-
The flag you are thinking of is
{{\-\-allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{\-\-allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
"""
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( `--allocator_fairness_excluded_resource_names=gpus`), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set {{\-\-allocator_fairness_excluded_resource_names=gpus}}
in DC/OS (but maybe we should?). Is it the case that most DC/OS users only
install GPUs on a small number of nodes in their cluster? If so, we should
consider it a scarce resource and set this flag by default. If not, then GPUs
aren't actually a scarce resource and we shouldn't be setting this flag -- DRF
will perform as expected without it.
"""
was (Author: klueska):
The flag you are thinking of is
{{\-\-allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{\-\-allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
"""
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( `--allocator_fairness_excluded_resource_names=gpus`), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set `--allocator_fairness_excluded_resource_names=gpus` in
DC/OS (but maybe we should?). Is it the case that most DC/OS users only install
GPUs on a small number of nodes in their cluster? If so, we should consider it
a scarce resource and set this flag by default. If not, then GPUs aren't
actually a scarce resource and we shouldn't be setting this flag-- DRF will
perform as expected without it.
"""
> provide additional insight for framework developers re: GPU_RESOURCES
> capability
>
>
> Key: MESOS-7375
> URL: https://issues.apache.org/jira/browse/MESOS-7375
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: James DeFelice
> Labels: mesosphere
>
> On clusters where all nodes are equal and every node has a GPU, frameworks
> that **don't** opt-in to the `GPU_RESOURCES` capability won't get any offers.
> This is surprising for
[jira] [Comment Edited] (MESOS-7375) provide additional insight for framework developers re: GPU_RESOURCES capability
[
https://issues.apache.org/jira/browse/MESOS-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983645#comment-15983645
]
Kevin Klues edited comment on MESOS-7375 at 4/25/17 9:24 PM:
-
The flag you are thinking of is
{{\-\-allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{\-\-allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
"""
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( {{\-\-allocator_fairness_excluded_resource_names=gpus}}), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set {{\-\-allocator_fairness_excluded_resource_names=gpus}}
in DC/OS (but maybe we should?). Is it the case that most DC/OS users only
install GPUs on a small number of nodes in their cluster? If so, we should
consider it a scarce resource and set this flag by default. If not, then GPUs
aren't actually a scarce resource and we shouldn't be setting this flag -- DRF
will perform as expected without it.
"""
was (Author: klueska):
The flag you are thinking of is
{{\-\-allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{\-\-allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
"""
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( `--allocator_fairness_excluded_resource_names=gpus`), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set {{\-\-allocator_fairness_excluded_resource_names=gpus}}
in DC/OS (but maybe we should?). Is it the case that most DC/OS users only
install GPUs on a small number of nodes in their cluster? If so, we should
consider it a scarce resource and set this flag by default. If not, then GPUs
aren't actually a scarce resource and we shouldn't be setting this flag -- DRF
will perform as expected without it.
"""
> provide additional insight for framework developers re: GPU_RESOURCES
> capability
>
>
> Key: MESOS-7375
> URL: https://issues.apache.org/jira/browse/MESOS-7375
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: James DeFelice
> Labels: mesosphere
>
> On clusters where all nodes are equal and every node has a GPU, frameworks
> that **don't** opt-in to the `GPU_RESOURCES` capability won't get any offers.
> This is
[jira] [Comment Edited] (MESOS-7375) provide additional insight for framework developers re: GPU_RESOURCES capability
[
https://issues.apache.org/jira/browse/MESOS-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983645#comment-15983645
]
Kevin Klues edited comment on MESOS-7375 at 4/25/17 9:23 PM:
-
The flag you are thinking of is
{{\-\-allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{\-\-allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
{noformat}
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( `--allocator_fairness_excluded_resource_names=gpus`), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set `--allocator_fairness_excluded_resource_names=gpus` in
DC/OS (but maybe we should?). Is it the case that most DC/OS users only install
GPUs on a small number of nodes in their cluster? If so, we should consider it
a scarce resource and set this flag by default. If not, then GPUs aren't
actually a scarce resource and we shouldn't be setting this flag-- DRF will
perform as expected without it.
{noformat}
was (Author: klueska):
The flag you are thinking of is
{{--allocator_fairness_excluded_resource_names}} (i.e. you can set it as
{{--allocator_fairness_excluded_resource_names=gpus}}).
Regarding motivation for the GPU_RESOURCES capability-- here is an excerpt from
an email I sent out recently:
{noformat}
Ideally, marathon (and any other frameworks -- SDK include) should do some sort
of preferential scheduling when they opt-in to use GPUs. That is, they should
*prefer* to run GPU jobs on GPU machines and non-GPU jobs on non-GPU machines
(falling back to running them on GPU machines only if that is all that is
available).
Additionally, we need a way for an operator to indicate whether GPUs are a
scarce resource in their cluster or not. We have a flag in mesos that allows us
to set this ( `--allocator_fairness_excluded_resource_names=gpus`), but we
don't yet have a way of setting this through DC/OS. If we don't set this flag,
we run the risk of Mesos's DRF algorithm choosing to very rarely send out
offers from GPU machines once the first GPU job has been launched on them.
As a concrete example, imagine you have a machine with only 1 GPU and you
launch a task that consumes it -- from DRF's perspective that node now has 100%
usage of one of its resources. Even if you have 2 GPUs, and one gets consumed,
DRF still thinks you have consumed 50% of one of its resources. Out of
fairness, DRF will choose not to send offers from you until some other resource
on *all* other nodes approaches 50% as well (which may take a while if you are
allocating CPUs, memory, and disk in small increments).
Right now we don't set `--allocator_fairness_excluded_resource_names=gpus` in
DC/OS (but maybe we should?). Is it the case that most DC/OS users only install
GPUs on a small number of nodes in their cluster? If so, we should consider it
a scarce resource and set this flag by default. If not, then GPUs aren't
actually a scarce resource and we shouldn't be setting this flag-- DRF will
perform as expected without it.
{noformat}
> provide additional insight for framework developers re: GPU_RESOURCES
> capability
>
>
> Key: MESOS-7375
> URL: https://issues.apache.org/jira/browse/MESOS-7375
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: James DeFelice
> Labels: mesosphere
>
> On clusters where all nodes are equal and every node has a GPU, frameworks
> that **don't** opt-in to the `GPU_RESOURCES` capability won't get any offers.
> This is
