[jira] [Updated] (MESOS-5753) Command executor should use `mesos-containerizer launch` to launch user task.
[ https://issues.apache.org/jira/browse/MESOS-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-5753: -- Sprint: Mesosphere Sprint 38 Story Points: 8 > Command executor should use `mesos-containerizer launch` to launch user task. > - > > Key: MESOS-5753 > URL: https://issues.apache.org/jira/browse/MESOS-5753 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Jie Yu >Assignee: Jie Yu > > Currently, command executor and `mesos-containerizer launch` share a lot of > the logic. Command executor should in fact, just use `mesos-containerizer > launch` to launch the user task. > Potentially, `mesos-containerizer launch` can be also used by custom executor > to launch user tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5753) Command executor should use `mesos-containerizer launch` to launch user task.
[ https://issues.apache.org/jira/browse/MESOS-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-5753: -- Component/s: containerization > Command executor should use `mesos-containerizer launch` to launch user task. > - > > Key: MESOS-5753 > URL: https://issues.apache.org/jira/browse/MESOS-5753 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Jie Yu >Assignee: Jie Yu > > Currently, command executor and `mesos-containerizer launch` share a lot of > the logic. Command executor should in fact, just use `mesos-containerizer > launch` to launch the user task. > Potentially, `mesos-containerizer launch` can be also used by custom executor > to launch user tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5753) Command executor should use `mesos-containerizer launch` to launch user task.
[ https://issues.apache.org/jira/browse/MESOS-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360421#comment-15360421 ] Jie Yu commented on MESOS-5753: --- https://reviews.apache.org/r/49415 https://reviews.apache.org/r/49424 https://reviews.apache.org/r/49425 https://reviews.apache.org/r/49472 https://reviews.apache.org/r/49473 https://reviews.apache.org/r/49479 https://reviews.apache.org/r/49524 https://reviews.apache.org/r/49523 https://reviews.apache.org/r/49540 https://reviews.apache.org/r/49541 https://reviews.apache.org/r/49542 https://reviews.apache.org/r/49548 https://reviews.apache.org/r/49549 https://reviews.apache.org/r/49568 https://reviews.apache.org/r/49569 > Command executor should use `mesos-containerizer launch` to launch user task. > - > > Key: MESOS-5753 > URL: https://issues.apache.org/jira/browse/MESOS-5753 > Project: Mesos > Issue Type: Improvement >Reporter: Jie Yu >Assignee: Jie Yu > > Currently, command executor and `mesos-containerizer launch` share a lot of > the logic. Command executor should in fact, just use `mesos-containerizer > launch` to launch the user task. > Potentially, `mesos-containerizer launch` can be also used by custom executor > to launch user tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5770) Mesos state api reporting Host IP instead of container IP with health check
Lax created MESOS-5770: -- Summary: Mesos state api reporting Host IP instead of container IP with health check Key: MESOS-5770 URL: https://issues.apache.org/jira/browse/MESOS-5770 Project: Mesos Issue Type: Bug Reporter: Lax Priority: Critical Fix For: 0.28.0 Am using Mesos IP per container using docker containerizer (via Calico). Mesos state API (/master/state.json) seems to report container IP as long as I have no health check on my task. As soon as I add health check to the task, mesos start reporting Host IP instead of Container IP. Had initially opened this bug on Marathon (https://github.com/mesosphere/marathon/issues/3907). But then told the issue is with Mesos reporting the wrong IP. Here are versions of Mesos and Marathon I was using. Mesos: 0.28.0.2 Marathon: 0.15.3-1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5401) Add ability to inject a Volume of Nvidia GPU-related libraries into a docker container.
[
https://issues.apache.org/jira/browse/MESOS-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360358#comment-15360358
]
Benjamin Mahler commented on MESOS-5401:
{noformat}
commit a7185a88c67eb462a0fe8da805193ae027fadca3
Author: Kevin Klues
Date: Sat Jul 2 15:41:05 2016 -0700
Added a new 'NvidiaVolume' component.
This component is responsible for building up a consolidated volume of
binaries / libraries from the user-space portion of the Nvidia GPU
drivers so that it can be injected into a container as a single
volume. Its core functionality mimics that of the
'nvidia-docker-plugin': https://github.com/NVIDIA/nvidia-docker/
We currently only implement the 'create()' function which is
responsible for building up the volume itself. In a subsequent commit
we will add support for reading a Docker image manifest and deciding
whether we should inject the volume into the docker container.
Review: https://reviews.apache.org/r/49565/
{noformat}
{noformat}
commit 14458c81ecdc0f592a6f07db551a237f7173eca7
Author: Kevin Klues
Date: Sat Jul 2 16:39:35 2016 -0700
Integrated the 'NvidiaVolume' component into 'NvidiaComponents'.
This makes it so that the 'NvidiaVolume' component can be shared
across containerizers in the same way that the 'NvidiaGpuAllocator' is
currently being shared.
Review: https://reviews.apache.org/r/49566/
{noformat}
> Add ability to inject a Volume of Nvidia GPU-related libraries into a docker
> container.
> ---
>
> Key: MESOS-5401
> URL: https://issues.apache.org/jira/browse/MESOS-5401
> Project: Mesos
> Issue Type: Improvement
>Reporter: Kevin Klues
>Assignee: Kevin Klues
> Labels: gpu, mesosphere
> Fix For: 1.0.0
>
>
> In order to support Nvidia GPUs with docker containers in Mesos, we need to
> be able to consolidate all Nvidia libraries into a common volume and inject
> that volume into the container.
> More info on why this is necessary here:
> https://github.com/NVIDIA/nvidia-docker/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5401) Add ability to inject a Volume of Nvidia GPU-related libraries into a docker container.
[ https://issues.apache.org/jira/browse/MESOS-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360310#comment-15360310 ] Kevin Klues edited comment on MESOS-5401 at 7/2/16 7:42 PM: This is a partial implementation of this support. It includes the ability to create the volume, but not the support to decide if it should be injected into a container or not. The logic for this injection decision will come in a subsequent patch. https://reviews.apache.org/r/49565/ https://reviews.apache.org/r/49566/ was (Author: klueska): This is a partial implementation of this support. It includes the ability to create the volume, but not the support to decide if it should be injected into a container or not. The logic for this injection decision will come in a subsequent patch. https://reviews.apache.org/r/49565/ > Add ability to inject a Volume of Nvidia GPU-related libraries into a docker > container. > --- > > Key: MESOS-5401 > URL: https://issues.apache.org/jira/browse/MESOS-5401 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > In order to support Nvidia GPUs with docker containers in Mesos, we need to > be able to consolidate all Nvidia libraries into a common volume and inject > that volume into the container. > More info on why this is necessary here: > https://github.com/NVIDIA/nvidia-docker/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5401) Add ability to inject a Volume of Nvidia GPU-related libraries into a docker container.
[ https://issues.apache.org/jira/browse/MESOS-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360310#comment-15360310 ] Kevin Klues edited comment on MESOS-5401 at 7/2/16 7:43 PM: These patches implement a partial implementation of this volume injection support. They include the ability to create a volume, but not the support to decide if it should be injected into a container or not. The logic for this injection decision will come in a subsequent patch. https://reviews.apache.org/r/49565/ https://reviews.apache.org/r/49566/ was (Author: klueska): This is a partial implementation of this support. It includes the ability to create the volume, but not the support to decide if it should be injected into a container or not. The logic for this injection decision will come in a subsequent patch. https://reviews.apache.org/r/49565/ https://reviews.apache.org/r/49566/ > Add ability to inject a Volume of Nvidia GPU-related libraries into a docker > container. > --- > > Key: MESOS-5401 > URL: https://issues.apache.org/jira/browse/MESOS-5401 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > In order to support Nvidia GPUs with docker containers in Mesos, we need to > be able to consolidate all Nvidia libraries into a common volume and inject > that volume into the container. > More info on why this is necessary here: > https://github.com/NVIDIA/nvidia-docker/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5769) Add get_abi_version() to ELF abstraction in stout
Kevin Klues created MESOS-5769:
--
Summary: Add get_abi_version() to ELF abstraction in stout
Key: MESOS-5769
URL: https://issues.apache.org/jira/browse/MESOS-5769
Project: Mesos
Issue Type: Improvement
Reporter: Kevin Klues
Assignee: Kevin Klues
Fix For: 1.0.0
This function allows us to inspect the {{.note.ABI-tag}} section of an ELF
binary to determine the ABI version of the executable / library. This is
needed for checking soe of the logic in building up an NvidiaVolume for
injection into a container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5768) Reimplement the stout ELF abstraction in terms of ELFIO
[ https://issues.apache.org/jira/browse/MESOS-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Klues updated MESOS-5768: --- Description: With the introduction of the new bundled ELFIO library, we need to reimplement our stout ELF abstraction in terms of it. As part of this, we need to update the tests that use it (i.e. ldcache_test.cpp) was:With the introduction of the new bundled ELFIO library, we need to reimplement our stout ELF abstraction in terms of it. > Reimplement the stout ELF abstraction in terms of ELFIO > --- > > Key: MESOS-5768 > URL: https://issues.apache.org/jira/browse/MESOS-5768 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > With the introduction of the new bundled ELFIO library, we need to > reimplement our stout ELF abstraction in terms of it. > As part of this, we need to update the tests that use it (i.e. > ldcache_test.cpp) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5768) Reimplement the stout ELF abstraction in terms of ELFIO
[ https://issues.apache.org/jira/browse/MESOS-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360303#comment-15360303 ] Kevin Klues commented on MESOS-5768: https://reviews.apache.org/r/49563/ > Reimplement the stout ELF abstraction in terms of ELFIO > --- > > Key: MESOS-5768 > URL: https://issues.apache.org/jira/browse/MESOS-5768 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > With the introduction of the new bundled ELFIO library, we need to > reimplement our stout ELF abstraction in terms of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5768) Reimplement the stout ELF abstraction in terms of ELFIO
Kevin Klues created MESOS-5768: -- Summary: Reimplement the stout ELF abstraction in terms of ELFIO Key: MESOS-5768 URL: https://issues.apache.org/jira/browse/MESOS-5768 Project: Mesos Issue Type: Improvement Reporter: Kevin Klues Assignee: Kevin Klues Fix For: 1.0.0 With the introduction of the new bundled ELFIO library, we need to reimplement our stout ELF abstraction in terms of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5767) Add ELFIO as bundled Dependency to Mesos
[ https://issues.apache.org/jira/browse/MESOS-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360300#comment-15360300 ] Kevin Klues commented on MESOS-5767: https://reviews.apache.org/r/49561/ > Add ELFIO as bundled Dependency to Mesos > > > Key: MESOS-5767 > URL: https://issues.apache.org/jira/browse/MESOS-5767 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > ELFIO is a header-only replacement for parsing ELF binaries. Previously we > were using libelf, which introduced both a new build-time dependency as well > as a runtime dependence even though we only really needed this library when > operating on machines that have GPUs. > By using this header-only library and bundling it with Mesos, we can remove > this external dependence altogether. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5767) Add ELFIO as bundled Dependency to Mesos
[ https://issues.apache.org/jira/browse/MESOS-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360298#comment-15360298 ] Kevin Klues commented on MESOS-5767: https://reviews.apache.org/r/49560/ > Add ELFIO as bundled Dependency to Mesos > > > Key: MESOS-5767 > URL: https://issues.apache.org/jira/browse/MESOS-5767 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Kevin Klues > Labels: gpu, mesosphere > Fix For: 1.0.0 > > > ELFIO is a header-only replacement for parsing ELF binaries. Previously we > were using libelf, which introduced both a new build-time dependency as well > as a runtime dependence even though we only really needed this library when > operating on machines that have GPUs. > By using this header-only library and bundling it with Mesos, we can remove > this external dependence altogether. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5767) Add ELFIO as bundled Dependency to Mesos
Kevin Klues created MESOS-5767: -- Summary: Add ELFIO as bundled Dependency to Mesos Key: MESOS-5767 URL: https://issues.apache.org/jira/browse/MESOS-5767 Project: Mesos Issue Type: Improvement Reporter: Kevin Klues Assignee: Kevin Klues Fix For: 1.0.0 ELFIO is a header-only replacement for parsing ELF binaries. Previously we were using libelf, which introduced both a new build-time dependency as well as a runtime dependence even though we only really needed this library when operating on machines that have GPUs. By using this header-only library and bundling it with Mesos, we can remove this external dependence altogether. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5766) Missing License Information for Bundled NVML headers
Kevin Klues created MESOS-5766: -- Summary: Missing License Information for Bundled NVML headers Key: MESOS-5766 URL: https://issues.apache.org/jira/browse/MESOS-5766 Project: Mesos Issue Type: Bug Reporter: Kevin Klues Assignee: Kevin Klues Fix For: 1.0.0 See Summary -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5757) Authorize orphaned tasks
[ https://issues.apache.org/jira/browse/MESOS-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360292#comment-15360292 ] Joerg Schad commented on MESOS-5757: The problem with orphaned tasks is that we don't necessarily access to the FrameworkInfo and hence we cannot authorize them as we would do for running/staged task (i.e., first check Task/TaskInfo and as fallback check FrameworInfo). I see three options for dealing with this problem: 1. Coarse granular authz for orphaned tasks Introduce a new acls/authz action for allowing a user to view orphaned tasks (i.e., all or none). 2. Add FrameworkInfo to Task (and then authorize orphaned Tasks the same way we would running/staged ones) The drawback here would be the increased size (and hence memory usage) for storing Tasks. FrameworkInfo can have large fields. Note that FrameworkInfo will be stored for each Task and hence potentially duplicated. 3. Add a new datastructure to the master storing a mapping of frameworkID to frameworkInfo (the frameworkInfo is part of Task and hence we can authorize orphaned Tasks the same way we would running/staged ones) This would lead to less storage overhead compared to 2, but might be a little more complex to implement as we have to fill/cleanup such datastructure. > Authorize orphaned tasks > > > Key: MESOS-5757 > URL: https://issues.apache.org/jira/browse/MESOS-5757 > Project: Mesos > Issue Type: Bug > Components: security >Affects Versions: 1.0.0 >Reporter: Vinod Kone > Labels: mesosphere, security > Fix For: 1.0.0 > > > Currently, orphaned tasks are not filtered (i.e., using authorization) when a > request is made to /state endpoint. This is inconsistent (and unexpected) > with how we filter un-orphaned tasks. > This is tricky because master and hence the authorizer do not have > FrameworkInfos for these orphaned tasks, until after the corresponding > frameworks re-register. > One option is for the agent to include FrameworkInfos of all its tasks and > executors in its re-registration message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5708) Add authz to /files/debug
[
https://issues.apache.org/jira/browse/MESOS-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360109#comment-15360109
]
Alexander Rojas commented on MESOS-5708:
The only issue there is if we want to construct the files object with an
authorizer, or if {{/files/debug}} should use a callback function as the
sandboxes are protected. I like the first one better because its consistent
with how the authorizers are build, but the second separates the files for even
having to know about an authorizer.
> Add authz to /files/debug
> -
>
> Key: MESOS-5708
> URL: https://issues.apache.org/jira/browse/MESOS-5708
> Project: Mesos
> Issue Type: Task
> Components: security
>Reporter: Adam B
>Assignee: Abhishek Dasgupta
>Priority: Minor
> Labels: mesosphere, security
> Fix For: 1.0.0
>
>
> The /files/debug endpoint exposes the attached master/agent log paths and
> every attached sandbox path, which includes the frameworkId and executorId.
> Even if sandboxes are protected, we still don't want to expose this
> information to unauthorized users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5514) Implement LIST_FILES Call in v1 agent API.
[ https://issues.apache.org/jira/browse/MESOS-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357119#comment-15357119 ] Abhishek Dasgupta edited comment on MESOS-5514 at 7/2/16 7:53 AM: -- Discarding my old reviews due to complete reorganization and design changes: New Reviews are : https://reviews.apache.org/r/49443/ https://reviews.apache.org/r/49444/ https://reviews.apache.org/r/49445/ https://reviews.apache.org/r/49529/ https://reviews.apache.org/r/49446/ https://reviews.apache.org/r/49447/ https://reviews.apache.org/r/49550/ https://reviews.apache.org/r/49551/ https://reviews.apache.org/r/49448/ was (Author: a10gupta): Discarding my old reviews due to complete reorganization and design changes: New Reviews are : https://reviews.apache.org/r/49443/ https://reviews.apache.org/r/49444/ https://reviews.apache.org/r/49445/ https://reviews.apache.org/r/49446/ https://reviews.apache.org/r/49447/ https://reviews.apache.org/r/49448/ > Implement LIST_FILES Call in v1 agent API. > -- > > Key: MESOS-5514 > URL: https://issues.apache.org/jira/browse/MESOS-5514 > Project: Mesos > Issue Type: Task >Reporter: Vinod Kone >Assignee: Abhishek Dasgupta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5487) Implement LIST_FILES Call in v1 master API.
[ https://issues.apache.org/jira/browse/MESOS-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357118#comment-15357118 ] Abhishek Dasgupta edited comment on MESOS-5487 at 7/2/16 7:52 AM: -- Discarding old reviews due to complete reorganization and design changes: New Reviews are : https://reviews.apache.org/r/49443/ https://reviews.apache.org/r/49444/ https://reviews.apache.org/r/49445/ https://reviews.apache.org/r/49529/ https://reviews.apache.org/r/49446/ https://reviews.apache.org/r/49447/ https://reviews.apache.org/r/49550/ https://reviews.apache.org/r/49551/ https://reviews.apache.org/r/49448/ was (Author: a10gupta): Discarding old reviews due to complete reorganization and design changes: New Reviews are : https://reviews.apache.org/r/49443/ https://reviews.apache.org/r/49444/ https://reviews.apache.org/r/49445/ https://reviews.apache.org/r/49446/ https://reviews.apache.org/r/49447/ https://reviews.apache.org/r/49448/ > Implement LIST_FILES Call in v1 master API. > --- > > Key: MESOS-5487 > URL: https://issues.apache.org/jira/browse/MESOS-5487 > Project: Mesos > Issue Type: Task >Reporter: Vinod Kone >Assignee: Abhishek Dasgupta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
