[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204341#comment-16204341
]
Sidharta Seethana commented on YARN-7286:
-
[~ebadger], thanks for the updated patch. I'll wait for jenkins to come back
before looking at the patch in more detail. One thing though : I see
negative/failure tests in 'testDockerCapabilities' ; do you mind adding a test
to ensure that using just "none" (or "NONE") leads to an empty capability list
on container launch?
[~jlowe], If you want to review the patch instead, that is fine too.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch, YARN-7286.004.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201121#comment-16201121
]
Sidharta Seethana commented on YARN-7286:
-
Agreed, we shouldn't be changing that behavior at this point. I am ok with the
"NONE" approach. With respect to mixing "NONE" and other capabilities, how
about moving capability configuration handling to the initialization function
in {{DockerLinuxContainerRuntime}} and throw an exception there in case they
are mixed? It should probably have been in that function in the first place -
all other config handling is in that function.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200904#comment-16200904
]
Sidharta Seethana edited comment on YARN-7286 at 10/11/17 8:30 PM:
---
The {{DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES}} list was based on the
capabilities docker enabled by default at the time. This default list of
capabilities is meant to stay consistent even if docker changes this list over
time.
{quote}
Well, yes, but that's at the discretion of the admin. If they want to give the
user 0 capabilities, then they should be able to. The question is what the best
way to do that is. If I were to look at yarn-site.xml and see
for the capabilities, I would implicitly think there are no capabilities given,
since this is an empty list. However, this would actually give the default list
of capabilities.
{quote}
This is a bit surprising - is this the behavior expected from
{{Configuration.setStrings(key, "")}} as well ? The behavior I see is this :
{code}
final YarnConfiguration conf = new YarnConfiguration();
//set test.key1 and not test.key2
conf.setStrings("test.key1", "");
Assert.assertTrue(conf.getStrings("test.key1", "val1",
"val2") == null);
Assert.assertEquals(2, conf.getStrings("test.key2", "val1", "val2")
.length);
{code}
was (Author: sidharta-s):
The {{DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES}} list was based on the
capabilities docker enabled by default at the time. This default list of
capabilities is meant to stay consistent even if docker changes this list over
time.
{quote}
Well, yes, but that's at the discretion of the admin. If they want to give the
user 0 capabilities, then they should be able to. The question is what the best
way to do that is. If I were to look at yarn-site.xml and see
for the capabilities, I would implicitly think there are no capabilities given,
since this is an empty list. However, this would actually give the default list
of capabilities.
{quote}
This is bit that is surprising - is this the behavior expected from
{{Configuration.setStrings(key, "")}} as well ? The behavior I see is this :
{code}
final YarnConfiguration conf = new YarnConfiguration();
//set test.key1 and not test.key2
conf.setStrings("test.key1", "");
Assert.assertTrue(conf.getStrings("test.key1", "val1",
"val2") == null);
Assert.assertEquals(2, conf.getStrings("test.key2", "val1", "val2")
.length);
{code}
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200904#comment-16200904
]
Sidharta Seethana commented on YARN-7286:
-
The {{DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES}} list was based on the
capabilities docker enabled by default at the time. This default list of
capabilities is meant to stay consistent even if docker changes this list over
time.
{quote}
Well, yes, but that's at the discretion of the admin. If they want to give the
user 0 capabilities, then they should be able to. The question is what the best
way to do that is. If I were to look at yarn-site.xml and see
for the capabilities, I would implicitly think there are no capabilities given,
since this is an empty list. However, this would actually give the default list
of capabilities.
{quote}
This is bit that is surprising - is this the behavior expected from
{{Configuration.setStrings(key, "")}} as well ? The behavior I see is this :
{code}
final YarnConfiguration conf = new YarnConfiguration();
//set test.key1 and not test.key2
conf.setStrings("test.key1", "");
Assert.assertTrue(conf.getStrings("test.key1", "val1",
"val2") == null);
Assert.assertEquals(2, conf.getStrings("test.key2", "val1", "val2")
.length);
{code}
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190674#comment-16190674
]
Sidharta Seethana commented on YARN-7226:
-
[~jlowe] I committed the branch-2.8 patch. Unfortunately, YARN-6550 was
committed to branch-2 a couple of hours after you uploaded the patch and the
branch-2 patch no longer applies. Apologies for the confusion - do you mind
rebasing the branch-2 patch?
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch,
> YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch,
> YARN-7226.006.patch, YARN-7226-branch-2.006.patch,
> YARN-7226-branch-2.8.006.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16189191#comment-16189191
]
Sidharta Seethana commented on YARN-7226:
-
[~jlowe]
+1 to the latest patch - committed to trunk and branch-3.0 . Could you please
add a branch-2 version ? Thanks.
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch,
> YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch,
> YARN-7226.006.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188820#comment-16188820
]
Sidharta Seethana commented on YARN-7226:
-
[~jlowe], recent changes to trunk (YARN-6550) seem to have modified
{{ContainerLaunch.java}} and the latest patch from this JIRA no longer applies
cleanly - could you please take a look?
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch,
> YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186627#comment-16186627
]
Sidharta Seethana commented on YARN-7226:
-
I'll take a look at the latest patch today or Monday at the latest.
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch,
> YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181815#comment-16181815
]
Sidharta Seethana commented on YARN-7226:
-
[~jlowe], a couple of comments :
* Is there a reason for adding {{useWhitelistEnv}} to {{LinuxContainerRuntime}}
and not {{ContainerRuntime}} ?
* Now that the whitelist behavior is different for the docker runtime, a test
for this would be a good thing to add.
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch,
> YARN-7226.003.patch, YARN-7226.004.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179698#comment-16179698
]
Sidharta Seethana commented on YARN-7226:
-
{quote}
But how does this work today for Docker containers? The launch script is using
the var:-value syntax, which means if the image specified the value then it
will not take the value the user desires. In other words, it looks like for
Docker containers the semantics of the whitelist is the list of variables that
cannot be overridden by the container.
I hope we can all agree that if the container explicitly sets an environment
variable then that variable should be set to the value the user specified. I
think the only issue then is what to do about variables that are getting
implicitly set via the NM whitelist that end up overriding the Docker image
variables unintentionally.
{quote}
[~jlowe], you are right on both counts - short of moving more of the common
functionality into the executors/container runtimes, there wasn't a better way
of handling this at the time. The goal was to ensure that non-docker
containers' behavior is not affected (however, based on this JIRA it looks like
there was a scenario that wasn't considered).
The approach of moving the implementation into the executors/runtimes makes
sense to me as well - though keeping the semantics in sync (to the extent
possible) is something we'll need to be careful about in the future. Maybe, at
some point in the future, the docker implementation could use a different
approach altogether (e.g specifying these env vars as a part of the docker run
command).
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177109#comment-16177109
]
Sidharta Seethana commented on YARN-7226:
-
[~ebadger], [~jlowe]
{quote}
The client gets to decide what image to use, so they should know the layout of
the image anyway. And if they don't, then they can fall back to the environment
of the docker image.
{quote}
{quote}
I do see the problem where a container did not ask for any setting of some
variable before entering the Docker container, but it gets a setting anyway
because of NM whitelist processing (e.g.: HADOOP_COMMON_HOME or
HADOOP_CONF_DIR). That whitelist-inherited setting can end up being problematic
for the Docker image, e.g.: Hadoop is installed in a different place in the
image compared to where the NM is picking it up. However I would counter-argue
that since the user picked the Docker image, the user knows what values to set
for these variables and can override them explicitly in the container startup.
Basically I see it as a "would be nice to have" feature rather than "Docker
containers do not work without it" feature. Am I missing the use case where
it's really necessary?
{quote}
I believe this is not simply a 'nice to have' feature - the use case that does
not work is exactly the one described above - a different HADOOP_COMMON_HOME or
JAVA_HOME that is specified in the docker image is not picked up correctly.
'Software isolation' is the canonical use case for integrating docker
containers with YARN - different java versions, hadoop versions etc., If the
user doesn't override the env vars, it is expected that what is specified in
the image takes precedence.
If I understand correctly, 'delayed variable expansion' is broken (but other
forms of env var override still work as expected for non-docker containers?) -
is this expected behavior described somewhere ?
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175635#comment-16175635
]
Sidharta Seethana commented on YARN-7226:
-
{quote}
As I understand it, a whitelisted variable is simply an environment variable
that will be propagated from the nodemanager's environment to the container's
environment unless the container has specified it's own value for that variable.
{quote}
[~jlowe],
Using docker containers introduces a new way in which environment variables for
a container can be specified - this is through the docker image itself, which
is a fairly common scenario. For example, an image used for running MR tasks
could specify its own JAVA_HOME that needs to be used instead of the JAVA_HOME
that is specified in the nodemanager's environment. Since
{{launch_container.sh}} runs inside the docker container, using the specified
docker image, using {{putEnvIfAbsent}} for whitelisted env vars doesn't do the
right thing in this case.
> Whitelisted variables do not support delayed variable expansion
> ---
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6374) Improve test coverage and add utility classes for common Docker operations
[ https://issues.apache.org/jira/browse/YARN-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996188#comment-15996188 ] Sidharta Seethana commented on YARN-6374: - [~shaneku...@gmail.com], could you please fix the (new) compiler warnings? Thanks. > Improve test coverage and add utility classes for common Docker operations > -- > > Key: YARN-6374 > URL: https://issues.apache.org/jira/browse/YARN-6374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6374.001.patch, YARN-6374.002.patch, > YARN-6374.003.patch, YARN-6374-branch-2.001.patch > > > Currently, it is tedious to execute Docker related operations due to the > plumbing needed to define the DockerCommand, writing the command file, > configuring privileged operation, and finally executing the command and > validating the result. Obtaining the current status of a Docker container can > also be improved. Finally, the test coverage is lacking for Docker Commands. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6374) Improve test coverage and add utility classes for common Docker operations
[ https://issues.apache.org/jira/browse/YARN-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-6374: Fix Version/s: 3.0.0-alpha3 > Improve test coverage and add utility classes for common Docker operations > -- > > Key: YARN-6374 > URL: https://issues.apache.org/jira/browse/YARN-6374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6374.001.patch, YARN-6374.002.patch, > YARN-6374.003.patch > > > Currently, it is tedious to execute Docker related operations due to the > plumbing needed to define the DockerCommand, writing the command file, > configuring privileged operation, and finally executing the command and > validating the result. Obtaining the current status of a Docker container can > also be improved. Finally, the test coverage is lacking for Docker Commands. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6374) Improve test coverage and add utility classes for common Docker operations
[ https://issues.apache.org/jira/browse/YARN-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995823#comment-15995823 ] Sidharta Seethana commented on YARN-6374: - [~shaneku...@gmail.com], thanks for the updated patch. It looks good to me. I have checked this patch into trunk. However, I am running into compilation errors if I cherry-pick the patch into branch-2 - could you please take a look? > Improve test coverage and add utility classes for common Docker operations > -- > > Key: YARN-6374 > URL: https://issues.apache.org/jira/browse/YARN-6374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-6374.001.patch, YARN-6374.002.patch, > YARN-6374.003.patch > > > Currently, it is tedious to execute Docker related operations due to the > plumbing needed to define the DockerCommand, writing the command file, > configuring privileged operation, and finally executing the command and > validating the result. Obtaining the current status of a Docker container can > also be improved. Finally, the test coverage is lacking for Docker Commands. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6374) Improve test coverage and add utility classes for common Docker operations
[
https://issues.apache.org/jira/browse/YARN-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985754#comment-15985754
]
Sidharta Seethana commented on YARN-6374:
-
[~shaneku...@gmail.com], thanks for the updated patch. Here is some feedback :
{code}
dockerOp.disableFailureLogging();
{code}
Why is failure logging disabled for all operations that use this method? So far
this has only been the case for signaling containers.
{code}
LOG.debug("Running docker rm. ContainerId: " + containerId);
{code}
Could you enclose such debug log lines with checks to see if debug logging is
enabled?
{code}
public static void executeDockerRm(String containerId,
{code}
I see the value in {{executeDockerCommand}} - but the other methods (e.g
{{executeDockerRm}}) might not actually be beneficial. Instead of
{{DockerCommandExecutor.executeDockerRm(containerId, …)}}, it would be
{{DockerCommandExecutor.executeDockerCommand(new DockerRmCommand(containerId),
... )}} . I am not sure there is much point to these additional wrappers. In
fact, this would mean additional work every time a new subclass of
{{DockerCommand}} is added.
{code}
public DockerContainerStatus getContainerStatus(String containerId,
Configuration conf,
PrivilegedOperationExecutor privilegedOperationExecutor) {
{code}
Is there a reason this method isn’t static like the utility methods in the
{{DockerCommandExecutor}} class? What do you think about simplifying this by
combining both of the these into one class (i.e is there a reason
{{getContainerStatus}} can’t be bundled with {{DockerCommandExecutor}} ?)
{code}
public DockerLinuxContainerRuntime(PrivilegedOperationExecutor
privilegedOperationExecutor, CGroupsHandler cGroupsHandler,
DockerContainerStatusHandler dockerContainerStatusHandler) {
{code}
So far, everything docker related has been encapsulated within the runtime
class ( and this includes the {{DockerClient}} which {{DockerCommandExecutor}}
uses). It seems counter-intuitive to inject something as specific as a
{{DockerContainerStatusHandler}} as a dependency for
{{DockerLinuxContainerRuntime}} itself.
{code}
public class MockDockerContainerStatusHandler
extends DockerContainerStatusHandler {
.
.
.
protected String executeStatusCommand(String containerId,
Configuration conf,
PrivilegedOperationExecutor privilegedOperationExecutor)
throws ContainerExecutionException {
{code}
What would be the reason to execute the inspect if the objective is return a
mocked/injected status?
{code}
@Test
public void testExecuteDockerRm() throws Exception {
DockerCommandExecutor
{code}
Same question as earlier - is it necessary to add all these wrappers?
{code}
@Test
public void testGetContainerStatusForCreated()
{code}
There seem to be a lot of these test methods to test one if-else tree in
{{getContainerStatus}}. Wouldn’t it easier to list all the distinct enum values
and test this in a loop as opposed to writing 8 functions?
{code}
@Test
public void testRemoveContainerOnExit() throws Exception {
{code}
Lot of new tests here to test argument concatenation in the
{{DockerRunCommand}} class - do you think these can be consolidated into fewer
methods? Some of this is already tested in the {{TestDockerContainerRuntime}}
class.
> Improve test coverage and add utility classes for common Docker operations
> --
>
> Key: YARN-6374
> URL: https://issues.apache.org/jira/browse/YARN-6374
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6374.001.patch, YARN-6374.002.patch
>
>
> Currently, it is tedious to execute Docker related operations due to the
> plumbing needed to define the DockerCommand, writing the command file,
> configuring privileged operation, and finally executing the command and
> validating the result. Obtaining the current status of a Docker container can
> also be improved. Finally, the test coverage is lacking for Docker Commands.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954431#comment-15954431 ] Sidharta Seethana commented on YARN-4757: - As per the comment above, It looks like the YARN-4757 branch was merged into the yarn-native-services branch (YARN-5079) . Linking the two issues so that we don't lose track of the discussion here while we make progress on YARN-5079. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Labels: oct16-hard > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, > YARN-4757.001.patch, YARN-4757.002.patch, YARN-4757- Simplified discovery of > services via DNS mechanisms.pdf, YARN-4757-YARN-4757.001.patch, > YARN-4757-YARN-4757.002.patch, YARN-4757-YARN-4757.003.patch, > YARN-4757-YARN-4757.004.patch, YARN-4757-YARN-4757.005.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5669: Fix Version/s: 3.0.0-alpha3 2.9.0 > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904134#comment-15904134 ] Sidharta Seethana commented on YARN-5669: - Committed to trunk and branch-2. Thanks, [~luhuichun] ! > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904125#comment-15904125 ] Sidharta Seethana commented on YARN-5669: - The javac warning and the checkstyle issue don't seem to be relevant to this specific patch. > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903519#comment-15903519 ] Sidharta Seethana commented on YARN-5669: - I have launched a commit build again - I'll take a look at this patch once the build is done. > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900256#comment-15900256
]
Sidharta Seethana commented on YARN-4266:
-
Based on the discussions in this JIRA and on YARN-5360, it looks like all we
have are less than ideal choices. Like I mentioned on YARN-5360, using the uid
has readability issues and it still wouldn’t guarantee that an image would work
correctly. In my opinion, we shouldn’t be adding *more* requirements on images
- the whole objective of this jira was to remove a requirement where possible
({{--user}}). launch_container.sh already uses bash, ln, cp, chmod, ls, find.
To this list we are considering adding usermod, su, getent and so on. In
addition to this we are considering making (expensive) changes to a container
prior to launching the application process - usermod only changes the files in
a user’s home directory and even then we still have no way of predicting how
long this operation would take - making application (process) launch time
unpredictable. IMO, This is not the direction we should be going in.
In the interest of making some progress, perhaps we could add support for
optionally using {{--user=:}}(turned off by default). A subset of
images that wouldn’t otherwise work, would be usable because of this change -
for example : images that don’t have the user being specified (say foo) but
would otherwise work with an arbitrary user (i.e the values supplied in
{{--user=:}} don’t matter).
I might have said this on other JIRAs and I’ll repeat here : docker containers
and applications using them are just one category of workloads that are going
to be run on a production YARN cluster. While we would like to use as much of
the power and flexibility that docker provides, we have to do this with due
consideration given to existing YARN/hadoop paradigms - security model
(users/groups/permissions), localization, log aggregation and so on.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5258: Fix Version/s: 2.9.0 > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5258.001.patch, YARN-5258.002.patch, > YARN-5258.003.patch, YARN-5258.004.patch, YARN-5258.005.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5258: Fix Version/s: 3.0.0-alpha3 > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5258.001.patch, YARN-5258.002.patch, > YARN-5258.003.patch, YARN-5258.004.patch, YARN-5258.005.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15857115#comment-15857115 ] Sidharta Seethana commented on YARN-5258: - Thanks, [~templedf]. This version looks good to me. Committed to trunk and branch-2. > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Attachments: YARN-5258.001.patch, YARN-5258.002.patch, > YARN-5258.003.patch, YARN-5258.004.patch, YARN-5258.005.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806246#comment-15806246
]
Sidharta Seethana edited comment on YARN-5258 at 1/7/17 12:32 AM:
--
[~templedf], the doc looks good. There is one thing, though :
{quote}
the entry point will be ignored when LCE launches the image because the
LCE always specifies the command to execute as YARN's container launch script.
{quote}
Actually, images with ‘entrypoint’s won’t work correctly - the entry point is
not ignored, the launch command (in this case launch_container.sh) will be sent
as an argument to the entrypoint which is completely unexpected. The only
scenario where this could work is if
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE is in use - where the entry
point is used with the default command. We’ll need to find a fix/work around
for this for other scenarios.
was (Author: sidharta-s):
[~templedf], the doc looks good. The is one thing, though :
{quote}
the entry point will be ignored when LCE launches the image because the
LCE always specifies the command to execute as YARN's container launch script.
{quote}
Actually, images with ‘entrypoint’s won’t work correctly - the entry point is
not ignored, the launch command (in this case launch_container.sh) will be sent
as an argument to the entrypoint which is completely unexpected. The only
scenario where this could work is if
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE is in use - where the entry
point is used with the default command. We’ll need to find a fix/work around
for this for other scenarios.
> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch,
> YARN-5258.003.patch, YARN-5258.004.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all
> of its various options aside from reading all of the JIRAs. We need to
> document the configuration, use, and troubleshooting, along with helpful
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806246#comment-15806246
]
Sidharta Seethana commented on YARN-5258:
-
[~templedf], the doc looks good. The is one thing, though :
{quote}
the entry point will be ignored when LCE launches the image because the
LCE always specifies the command to execute as YARN's container launch script.
{quote}
Actually, images with ‘entrypoint’s won’t work correctly - the entry point is
not ignored, the launch command (in this case launch_container.sh) will be sent
as an argument to the entrypoint which is completely unexpected. The only
scenario where this could work is if
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE is in use - where the entry
point is used with the default command. We’ll need to find a fix/work around
for this for other scenarios.
> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch,
> YARN-5258.003.patch, YARN-5258.004.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all
> of its various options aside from reading all of the JIRAs. We need to
> document the configuration, use, and troubleshooting, along with helpful
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803791#comment-15803791 ] Sidharta Seethana commented on YARN-5258: - I'll take a look. Thanks for the updated patch. > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Attachments: YARN-5258.001.patch, YARN-5258.002.patch, > YARN-5258.003.patch, YARN-5258.004.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669088#comment-15669088 ] Sidharta Seethana commented on YARN-5669: - Thanks for the clarification. YARN-3854 mentions that YARN-5669 and YARN-5670 are the subtasks for implementing docker image localization. I (mistakenly) thought that YARN-3854 was the parent JIRA. > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5669) Add support for Docker pull
[ https://issues.apache.org/jira/browse/YARN-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668932#comment-15668932 ] Sidharta Seethana commented on YARN-5669: - Hi [~luhuichun], Isn't the scope of this patch to add localization support via docker pull? If this is not the case, this jira and YARN-5670 do not fully cover the changes needed to support docker image localization. thanks. > Add support for Docker pull > --- > > Key: YARN-5669 > URL: https://issues.apache.org/jira/browse/YARN-5669 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: luhuichun > Attachments: YARN-5669.001.patch > > > We need to add docker pull to support Docker image localization. Refer to > YARN-3854 for the details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665610#comment-15665610
]
Sidharta Seethana commented on YARN-4266:
-
IMO, the usermod approach described in the design document is error prone and
likely to break in various ways.
* Like I mentioned in a earlier comment, the usermod operation only makes
changes to the home directory and not elsewhere in the image. Modifying the
rest of the image is not scalable and could significantly slow down the launch
of every container where we go down this path.
* Running this usermod operation also requires that launch_container.sh be
launched as a privileged user. Also, note that running {{docker run --pid=host
… bash ../launch_container.sh $newUID $containerUsername}} does not guarantee
that launch_container.sh as described here will work correctly - if the image
has a ‘USER’ directive, launch_container.sh will be run as that user and that
user might not have privileges to run a usermod operation.
* In addition, I don’t believe we should be using —pid=host. This exposes
other containers’s processes into this container - which will break isolation
and possibly behavior for certain applications (applications that assume a
single instance is running on a ’node’, for example).
* Lastly, adding more commands that run inside the container (usermod in this
case) adds even more requirements for the docker image being launched : we
already require that bash, find, ls etc be present in the image.
I can’t think of a way where we can find a generic solution for disabling —user
that will work for all (or even a large number of) scenarios while still being
in line with YARN’s security model/log aggregation etc., . (I’d be happy to be
proven wrong here). I think we need to acknowledge this and find a set of
canonical use cases that we want to support - and see how we can enable each
one of them. Couple of examples off the top of my head : 1) Spark/MRv2 - IMO,
it maybe easier to support these because they are already first class YARN
applications. 2) Apache/httpd based images - more digging to do there.
Thoughts ? I apologize for my spotty availability on this JIRA - I’ll try to
provide more prompt responses/feedback in the future.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager
[ https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665449#comment-15665449 ] Sidharta Seethana commented on YARN-5280: - YARN-3853 added the ContainerRuntime interface. The objective of adding the ‘prepareContainer()’ and ‘reapContainer()’ methods was to provide finer grained control of the container lifecycle (and at some point add corresponding instrumentation to track where time is spent in the container lifecycle). Using docker containers as an example, ‘prepareContainer()’ could be hooked into docker ‘create’ (and maybe even image localization). reapContainer() could be used for (optional) post complete container deletion. Once container runtimes were introduced, the interaction with resource handlers got a bit … trickier. Right now, the same cgroups based resource handlers can be used across the ‘default’ and ‘docker’ container runtimes (mainly due to docker’s cgroup-parent support). In this case, ‘postExecute()’ is used to clean up the container cgroups created by YARN and ‘reapContainer()’ could be used to clean up/remove the container itself. I hope that helps. > Allow YARN containers to run with Java Security Manager > --- > > Key: YARN-5280 > URL: https://issues.apache.org/jira/browse/YARN-5280 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, yarn >Affects Versions: 2.6.4 >Reporter: Greg Phillips >Assignee: Greg Phillips >Priority: Minor > Labels: oct16-medium > Attachments: YARN-5280.001.patch, YARN-5280.002.patch, > YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.patch, > YARNContainerSandbox.pdf > > > YARN applications have the ability to perform privileged actions which have > the potential to add instability into the cluster. The Java Security Manager > can be used to prevent users from running privileged actions while still > allowing their core data processing use cases. > Introduce a YARN flag which will allow a Hadoop administrator to enable the > Java Security Manager for user code, while still providing complete > permissions to core Hadoop libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634608#comment-15634608
]
Sidharta Seethana edited comment on YARN-5258 at 11/3/16 11:10 PM:
---
Feedback :
{code}
Docker combines an easy-to-use interface to Linux containers with
easy-to-construct image files for those containers. In short, Docker launches
very light weight virtual machines.
{code}
IMO, this is not an accurate characterization and we should drop it. This blog
post explains this is more detail :
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a
good idea to link to http://docs.docker.com/ .
{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN
containers into Docker containers.
{code}
I believe a brief description of ‘container runtimes’ would be warranted in
this section - LCE currently supports two - the default/‘process’ based runtime
and the docker runtime. It is possible to choose between these on a per
container basis. Alternatively, additional information could be added in a
follow up patch(es).
{code}
The Docker suuport in the LCE is still evolving.
{code}
minor typo. suuport -> support
{code}
To track progress, follow JIRA-3611,
{code}
It might be better to say YARN-3611 - with a link.
{code}
sudo docker pull images/hadoop-docker:latest
{code}
IMO, this should be a working example. That said, I am not aware of any popular
vendor-neutral images that would be a good candidate. This has been one of the
barriers to creating good documentation for this functionality. Should we
consider hosting ‘official’ apache hadoop images on docker hub ? Thoughts ?
{code}
The following properties should be set in yarn-site.xml:
{code}
Some of the properties described here don’t have values that are inline with
yarn-default.xml . Specifically,
{{yarn.nodemanager.runtime.linux.docker.allowed-container-networks}} and
{{yarn.nodemanager.runtime.linux.docker.privileged-containers.acl}} . There is
also a setting that isn’t mentioned here :
{{yarn.nodemanager.runtime.linux.docker.default-container-network}} . I think a
separate section on networking is warranted - I’ll submit a follow up patch
with additional documentation.
{code}
feature.docker.enabled
{code}
In the context of this functionality, this is not ‘optional’ - this must be set
of 1.
{code}
In order to work with YARN, there are two requirements for Docker images.
{code}
There are additional limitations - again, these could be added in subsequent
updates to the documentation. An important limitation that comes to mind is
that because YARN always overrides the command the container is launched with,
images with an {{ENTRYPOINT}} directive will not work. Application frameworks
may impose their additional requirements. For example, using slider with Docker
and YARN (currently) requires that all images have python installed in them (in
order to run the slider agent).
{code}
First, the Docker container will be explicitly launched with the application
owner as the container user. If the application owner is not a valid user (by
UID) in the Docker image, the application will fail.
{code}
By UID? This is not clear - it might be useful to provide an example here. One
example I can think of here - the UID of ‘nobody’ is different in CentOS vs
Ubuntu - so running an Ubuntu container on CentOS as user ‘nobody’ is likely to
cause failures.
{code}
In order to run an application in a Docker container, set the following
environment variables in the application's environment:
{code}
It might be worth pointing out that while this is not ideal, it does allow for
some existing applications that can inject environment variables to run in
docker containers without modifications e.g spark and map reduce.
{code}
Example: Spark
{code}
As mentioned earlier : I think this should be an actual working example and we
should consider exploring what it would take to make that possible.
I think this is a great start, thanks again [~templedf] for taking this on.
was (Author: sidharta-s):
Feedback :
{code}
Docker combines an easy-to-use interface to Linux containers with
easy-to-construct image files for those containers. In short, Docker launches
very light weight virtual machines.
{code}
IMO, this is not an accurate characterization and we should drop it. This blog
post explains this is more detail :
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a
good idea to link to http://docs.docker.com/ .
{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN
containers into Docker containers.
{code}
I believe a brief description of ‘container runtimes’ would be warranted in
this section - LCE currently supports two - the default/‘process’ based runtime
and the docker runti
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634608#comment-15634608
]
Sidharta Seethana commented on YARN-5258:
-
Feedback :
{code}
Docker combines an easy-to-use interface to Linux containers with
easy-to-construct image files for those containers. In short, Docker launches
very light weight virtual machines.
{code}
IMO, this is not an accurate characterization and we should drop it. This blog
post explains this is more detail :
https://blog.docker.com/2016/03/containers-are-not-vms/ . It might also be a
good idea to link to http://docs.docker.com/ .
{code}
The Linux Container Executor (LCE) allows the YARN NodeManager to launch YARN
containers into Docker containers.
{code}
I believe a brief description of ‘container runtimes’ would be warranted in
this section - LCE currently supports two - the default/‘process’ based runtime
and the docker runtime. It is possible to choose between these on a per
container basis. Alternatively, additional information could be added in a
follow up patch(es).
{code}
The Docker suuport in the LCE is still evolving.
{code}
minor typo. suuport -> support
{code}
To track progress, follow JIRA-3611,
{code}
It might be better to say YARN-3611 - with a link.
{code}
sudo docker pull images/hadoop-docker:latest
{code}
IMO, this should be a working example. That said, I am not aware of any popular
vendor-neutral images that would be a good candidate. This has been one of the
barriers to creating good documentation for this functionality. Should we
consider hosting ‘official’ apache hadoop images on docker hub ? Thoughts ?
{code}
The following properties should be set in yarn-site.xml:
{code}
Some of the properties described here don’t have values that are inline with
yarn-default.xml . Specifically,
yarn.nodemanager.runtime.linux.docker.allowed-container-networks and
yarn.nodemanager.runtime.linux.docker.privileged-containers.acl . There is also
a setting that isn’t mentioned here :
yarn.nodemanager.runtime.linux.docker.default-container-network . I think a
separate section on networking is warranted - I’ll submit a follow up patch
with additional documentation.
{code}
feature.docker.enabled
{code}
In the context of this functionality, this is not ‘optional’ - this must be set
of 1.
{code}
In order to work with YARN, there are two requirements for Docker images.
{code}
There are additional limitations - again, these could be added in subsequent
updates to the documentation. An important limitation that comes to mind is
that because YARN always overrides the command the container is launched with,
images with an {{ENTRYPOINT}} directive will not work. Application frameworks
may impose their additional requirements. For example, using slider with Docker
and YARN (currently) requires that all images have python installed in them (in
order to run the slider agent).
{code}
First, the Docker container will be explicitly launched with the application
owner as the container user. If the application owner is not a valid user (by
UID) in the Docker image, the application will fail.
{code}
By UID? This is not clear - it might be useful to provide an example here. One
example I can think of here - the UID of ‘nobody’ is different in CentOS vs
Ubuntu - so running an Ubuntu container on CentOS as user ‘nobody’ is likely to
cause failures.
{code}
In order to run an application in a Docker container, set the following
environment variables in the application's environment:
{code}
It might be worth pointing out that while this is not ideal, it does allow for
some existing applications that can inject environment variables to run in
docker containers without modifications e.g spark and map reduce.
{code}
Example: Spark
{code}
As mentioned earlier : I think this should be an actual working example and we
should consider exploring what it would take to make that possible.
I think this is a great start, thanks again [~templedf] for taking this on.
> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all
> of its various options aside from reading all of the JIRAs. We need to
> document the configuration, use, and troubleshooting, along with helpful
> examples.
--
This message was sent by Atlassian JIRA
(v6.
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634430#comment-15634430 ] Sidharta Seethana commented on YARN-5258: - [~templedf], I was able to apply the patch to branch-2. I'll do a review and get back to you soon. > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Attachments: YARN-5258.001.patch, YARN-5258.002.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634373#comment-15634373 ] Sidharta Seethana commented on YARN-5258: - [~templedf] thank you for this patch. I had some trouble applying this patch to trunk - was this generated against branch-2? > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Attachments: YARN-5258.001.patch, YARN-5258.002.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5822) Log ContainerRuntime initialization error in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5822: Attachment: YARN-5822.001.patch Uploading a quick patch to log the container runtime initialization failure. > Log ContainerRuntime initialization error in LinuxContainerExecutor > > > Key: YARN-5822 > URL: https://issues.apache.org/jira/browse/YARN-5822 > Project: Hadoop YARN > Issue Type: Task > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Trivial > Attachments: YARN-5822.001.patch > > > LinuxContainerExecutor does not log information corresponding to > ContainerRuntime initialization failure. This makes it hard to identify the > root cause for Nodemanager start failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5822) Log ContainerRuntime initialization error in LinuxContainerExecutor
Sidharta Seethana created YARN-5822: --- Summary: Log ContainerRuntime initialization error in LinuxContainerExecutor Key: YARN-5822 URL: https://issues.apache.org/jira/browse/YARN-5822 Project: Hadoop YARN Issue Type: Task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Priority: Trivial LinuxContainerExecutor does not log information corresponding to ContainerRuntime initialization failure. This makes it hard to identify the root cause for Nodemanager start failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576903#comment-15576903
]
Sidharta Seethana commented on YARN-4266:
-
Usermod seems to be of limited use. From usermod's man page :
{code}
-u, --uid UID
The new numerical value of the user's ID.
This value must be unique, unless the -o option is used. The value
must be non-negative.
The user's mailbox, and any files which the user owns and which are
located in the user's home directory will have the file user ID changed
automatically.
The ownership of files outside of the user's home directory must be
fixed manually.
No checks will be performed with regard to the UID_MIN, UID_MAX,
SYS_UID_MIN, or SYS_UID_MAX from /etc/login.defs.
{code}
If nothing outside the user's home directory is updated, this is likely to
break many images that use custom/non-root users ?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576471#comment-15576471 ] Sidharta Seethana commented on YARN-5704: - Thanks, [~vvasudev]. > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5717) Add tests for container-executor's is_feature_enabled function
[ https://issues.apache.org/jira/browse/YARN-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576469#comment-15576469 ] Sidharta Seethana commented on YARN-5717: - Thanks, [~chris.douglas]. > Add tests for container-executor's is_feature_enabled function > -- > > Key: YARN-5717 > URL: https://issues.apache.org/jira/browse/YARN-5717 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5717.001.patch, YARN-5717.002.patch > > > YARN-5704 added functionality to disable certain features in > container-executor. Most of the changes cannot be tested via > container-executor - however, is_feature_enabled can be tested if it were > made public. (It is currently static). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5673) [Umbrella] Re-write container-executor to improve security, extensibility, and portability
[ https://issues.apache.org/jira/browse/YARN-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572625#comment-15572625 ] Sidharta Seethana commented on YARN-5673: - [~templedf], I think this is restricted to the container-executor binary, specifically, not ContainerExecutor. You are right, though - about additional functionality being pushed into lower layers and the use ENV variables. This former was to ensure backward compatibility at the ContainerExecutor interface layer. The latter was to ensure a) no changes to protocols b) allow for existing apps to use the new functionality without changes (e.g MR, Spark). We should discuss these in a new/different JIRA, though. > [Umbrella] Re-write container-executor to improve security, extensibility, > and portability > -- > > Key: YARN-5673 > URL: https://issues.apache.org/jira/browse/YARN-5673 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: container-executor Re-write Design Document.pdf > > > As YARN adds support for new features that require administrator > privileges(such as support for network throttling and docker), we’ve had to > add new capabilities to the container-executor. This has led to a recognition > that the current container-executor security features as well as the code > could be improved. The current code is fragile and it’s hard to add new > features without causing regressions. Some of the improvements that need to > be made are - > *Security* > Currently the container-executor has limited security features. It relies > primarily on the permissions set on the binary but does little additional > security beyond that. There are few outstanding issues today - > - No audit log > - No way to disable features - network throttling and docker support are > built in and there’s no way to turn them off at a container-executor level > - Code can be improved - a lot of the code switches users back and forth in > an arbitrary manner > - No input validation - the paths, and files provided at invocation are not > validated or required to be in some specific location > - No signing functionality - there is no way to enforce that the binary was > invoked by the NM and not by any other process > *Code Issues* > The code layout and implementation themselves can be improved. Some issues > there are - > - No support for log levels - everything is logged and this can’t be turned > on or off > - Extremely long set of invocation parameters(specifically during container > launch) which makes turning features on or off complicated > - Poor test coverage - it’s easy to introduce regressions today due to the > lack of a proper test setup > - Duplicate functionality - there is some amount of code duplication > - Hard to make improvements or add new features due to the issues raised above > *Portability* > - The container-executor mixes platform dependent APIs with platform > independent APIs making it hard to run it on multiple platforms. Allowing it > to run on multiple platforms also improves the overall code structure . > One option is to improve the existing container-executor, however it might be > easier to start from scratch. That allows existing functionality to be > supported until we are ready to switch to the new code. > This umbrella JIRA is to capture all the work required for the new code. I'm > going to work on a design doc for the changes - any suggestions or > improvements are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5388) MAPREDUCE-6719 requires changes to DockerContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571140#comment-15571140
]
Sidharta Seethana commented on YARN-5388:
-
[~templedf] branch-2 patch : should we update {{DockerContainerExecutor.md.vm}}
here as well?
> MAPREDUCE-6719 requires changes to DockerContainerExecutor
> --
>
> Key: YARN-5388
> URL: https://issues.apache.org/jira/browse/YARN-5388
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: YARN-5388.001.patch, YARN-5388.002.patch,
> YARN-5388.branch-2.001.patch, YARN-5388.branch-2.002.patch
>
>
> Because the {{DockerContainerExecuter}} overrides the {{writeLaunchEnv()}}
> method, it must also have the wildcard processing logic from
> YARN-4958/YARN-5373 added to it. Without it, the use of -libjars will fail
> unless wildcarding is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5388) MAPREDUCE-6719 requires changes to DockerContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571092#comment-15571092
]
Sidharta Seethana commented on YARN-5388:
-
[~templedf] Apologies for the late response. Took at look at the trunk patch -
I think {{DockerContainerExecutor.md.vm}} needs to be removed as well. I also
see a reference to {{DockerContainerExecutor}} in
{{hadoop-project/src/site/site.xml}} .
> MAPREDUCE-6719 requires changes to DockerContainerExecutor
> --
>
> Key: YARN-5388
> URL: https://issues.apache.org/jira/browse/YARN-5388
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: YARN-5388.001.patch, YARN-5388.002.patch,
> YARN-5388.branch-2.001.patch, YARN-5388.branch-2.002.patch
>
>
> Because the {{DockerContainerExecuter}} overrides the {{writeLaunchEnv()}}
> method, it must also have the wildcard processing logic from
> YARN-4958/YARN-5373 added to it. Without it, the use of -libjars will fail
> unless wildcarding is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5717) Add tests for container-executor's is_feature_enabled function
[ https://issues.apache.org/jira/browse/YARN-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567674#comment-15567674 ] Sidharta Seethana commented on YARN-5717: - [~aw], [~chris.douglas] : could you please review this patch? thanks, -Sidharta > Add tests for container-executor's is_feature_enabled function > -- > > Key: YARN-5717 > URL: https://issues.apache.org/jira/browse/YARN-5717 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5717.001.patch, YARN-5717.002.patch > > > YARN-5704 added functionality to disable certain features in > container-executor. Most of the changes cannot be tested via > container-executor - however, is_feature_enabled can be tested if it were > made public. (It is currently static). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5717) Add tests for container-executor's is_feature_enabled function
[ https://issues.apache.org/jira/browse/YARN-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5717: Attachment: YARN-5717.002.patch Uploaded a new patch to fix compiler warnings. > Add tests for container-executor's is_feature_enabled function > -- > > Key: YARN-5717 > URL: https://issues.apache.org/jira/browse/YARN-5717 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5717.001.patch, YARN-5717.002.patch > > > YARN-5704 added functionality to disable certain features in > container-executor. Most of the changes cannot be tested via > container-executor - however, is_feature_enabled can be tested if it were > made public. (It is currently static). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5717) Add tests for container-executor's is_feature_enabled function
[ https://issues.apache.org/jira/browse/YARN-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5717: Attachment: YARN-5717.001.patch Attaching a patch with tests for is_feature_enabled and changes to display_usage to use fprintf directly. > Add tests for container-executor's is_feature_enabled function > -- > > Key: YARN-5717 > URL: https://issues.apache.org/jira/browse/YARN-5717 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5717.001.patch > > > YARN-5704 added functionality to disable certain features in > container-executor. Most of the changes cannot be tested via > container-executor - however, is_feature_enabled can be tested if it were > made public. (It is currently static). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560712#comment-15560712 ] Sidharta Seethana commented on YARN-5704: - [~aw], again : my reason for testing this manually was that most of the changes were to main.c - that was the only way to test if enabling/disabling these features worked in the first place. The fact that the function was static ( and the fact that the patch was already committed to trunk/branch-2) were the reasons for filing a separate JIRA for the tests to keep things clean - I was not using that as a reason to not add tests. container-executor in general needs a complete overhaul - for testability if not anything else. There is no disagreement that bad practices should not be further propagated - but we disagree on 'where variables should be declared in C code' being one among them. > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[
https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560552#comment-15560552
]
Sidharta Seethana commented on YARN-5704:
-
/cc [~chris.douglas], [~vinodkv]
[~aw], going back and forth discussing new concerns every time is not
productive. Do you have more concerns with this patch or is the list above
complete? Regarding your comments above :
# I explained already that most of the changes are in main.c which cannot be
tested via test-container-executor - I'll go ahead add tests for the function
you referred to.
# I don't believe it is a "useless abstraction" - those functions provide
useful, less error-prone shorthands.
# “In the middle” ? This variable is declared at the beginning of a new scope.
Even if it weren’t, declaring variables ‘near’ where they are needed is common
practice and is a widely used GNU extension. If this is a problem, we could use
static checking as part of our build or add the {{-ansi}} and {{-pedantic}}
compiler flags (and in the process break code that has existed for 5+ years).
> Provide config knobs to control enabling/disabling new/work in progress
> features in container-executor
> --
>
> Key: YARN-5704
> URL: https://issues.apache.org/jira/browse/YARN-5704
> Project: Hadoop YARN
> Issue Type: Task
> Components: yarn
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch
>
>
> Provide a mechanism to enable/disable Docker and TC (Traffic Control)
> functionality at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559506#comment-15559506 ] Sidharta Seethana commented on YARN-5704: - [~aw], the function you are referring to is static currently - and is not visible outside of that file and hence cannot be tested via test-container-executor either. I have filed YARN-5717 to make this function public and add tests for it. I'll follow up soon with a patch - I am hoping you could help review/commit it. [~vvasudev], if the branch-2.8 version of patch looks ok to you, could you please go ahead and commit it? > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5717) Add tests for container-executor's is_feature_enabled function
Sidharta Seethana created YARN-5717: --- Summary: Add tests for container-executor's is_feature_enabled function Key: YARN-5717 URL: https://issues.apache.org/jira/browse/YARN-5717 Project: Hadoop YARN Issue Type: Task Components: yarn Affects Versions: 3.0.0-alpha1, 2.8.0, 3.0.0-alpha2 Reporter: Sidharta Seethana Assignee: Sidharta Seethana YARN-5704 added functionality to disable certain features in container-executor. Most of the changes cannot be tested via container-executor - however, is_feature_enabled can be tested if it were made public. (It is currently static). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554516#comment-15554516 ] Sidharta Seethana commented on YARN-5704: - [~vvasudev], could you please take a look? Also, It looks like neither of these features are in branch-2.7 so a patch isn't necessary for that branch. > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5704: Affects Version/s: (was: 2.7.3) > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5704: Attachment: YARN-5704-branch-2.8.001.patch Uploaded a patch for branch-2.8. > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554438#comment-15554438 ] Sidharta Seethana commented on YARN-5704: - [~vvasudev], this patch needs YARN-4749 to be backported into branch-2.8 and branch-2.7 (that patch was committed as YARN-4245) . > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551296#comment-15551296 ] Sidharta Seethana commented on YARN-5704: - [~aw] As you can see from the patch, most of the changes in the patch are in main.c . Changes to this file cannot be tested via test-container-executor, I believe ? > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5704: Attachment: YARN-5704.001.patch Uploaded a patch that provides config knobs to enable/disable docker/tc functionality. The usage message is also updated accordingly. I tested various enable/disable combinations manually on Centos 7.2. > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544451#comment-15544451 ] Sidharta Seethana commented on YARN-5704: - Submitted to Jenkins. [~vvasudev], could you please take a look? > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
Sidharta Seethana created YARN-5704: --- Summary: Provide config knobs to control enabling/disabling new/work in progress features in container-executor Key: YARN-5704 URL: https://issues.apache.org/jira/browse/YARN-5704 Project: Hadoop YARN Issue Type: Task Components: yarn Affects Versions: 3.0.0-alpha1, 2.7.3, 2.8.0, 3.0.0-alpha2 Reporter: Sidharta Seethana Assignee: Sidharta Seethana Provide a mechanism to enable/disable Docker and TC (Traffic Control) functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504533#comment-15504533 ] Sidharta Seethana commented on YARN-4591: - Thanks, [~leftnoteasy] ! > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Sidharta Seethana >Priority: Trivial > Attachments: YARN-4591.001.patch, YARN-4591.002.patch > > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4591: Attachment: YARN-4591.002.patch Uploaded a new patch that fixes the reported checkstyle issues. > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Sidharta Seethana >Priority: Trivial > Attachments: YARN-4591.001.patch, YARN-4591.002.patch > > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491805#comment-15491805 ] Sidharta Seethana commented on YARN-4591: - Submitted to Jenkins. [~wangda], could you please take a look ? Thanks! > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Sidharta Seethana >Priority: Trivial > Attachments: YARN-4591.001.patch > > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4591: Attachment: YARN-4591.001.patch Uploaded a patch that provides a robot.txt for all YARN web UIs that disallows crawling. Added a unit test and tested this manually with RM/NM web UIs. > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Sidharta Seethana >Priority: Trivial > Attachments: YARN-4591.001.patch > > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488191#comment-15488191 ] Sidharta Seethana commented on YARN-4591: - I'll take a crack at this. Thanks. > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Priority: Trivial > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4591) YARN Web UIs should provide a robots.txt
[ https://issues.apache.org/jira/browse/YARN-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana reassigned YARN-4591: --- Assignee: Sidharta Seethana > YARN Web UIs should provide a robots.txt > > > Key: YARN-4591 > URL: https://issues.apache.org/jira/browse/YARN-4591 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Sidharta Seethana >Priority: Trivial > > To prevent well-behaved crawlers from indexing public YARN UIs. > Similar to HDFS-330 / HDFS-9651. > I took a quick look at the Webapp stuff in YARN and it looks complicated so I > can't provide a quick patch. If anyone can point me in the right direction I > might take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5596) TestDockerContainerRuntime fails on OS X
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1545#comment-1545
]
Sidharta Seethana commented on YARN-5596:
-
Thanks, [~templedf]!
[~vvasudev], could you please take a look? Thanks
> TestDockerContainerRuntime fails on OS X
>
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-5596.001.patch, YARN-5596.002.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5596) TestDockerContainerRuntime fails on OS X
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharta Seethana updated YARN-5596:
Attachment: YARN-5596.002.patch
Uploaded a new patch with proposed change.
> TestDockerContainerRuntime fails on OS X
>
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-5596.001.patch, YARN-5596.002.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5596) TestDockerContainerRuntime fails on OS X
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449971#comment-15449971
]
Sidharta Seethana commented on YARN-5596:
-
Thanks, [~templedf]. I'll upload a new patch with the visibility changed.
> TestDockerContainerRuntime fails on OS X
>
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-5596.001.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5596) TestDockerContainerRuntime fails on OS X
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharta Seethana updated YARN-5596:
Attachment: YARN-5596.001.patch
Uploading a patch that fixes the tests.
> TestDockerContainerRuntime fails on OS X
>
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-5596.001.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5596) TestDockerContainerRuntime fails on OS X
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharta Seethana updated YARN-5596:
Summary: TestDockerContainerRuntime fails on OS X (was:
TestDockerContainerRuntime fails on the mac)
> TestDockerContainerRuntime fails on OS X
>
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5596) TestDockerContainerRuntime fails on the mac
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sidharta Seethana updated YARN-5596:
Description:
/sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because of
this.
{code}
Failed tests:
TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testDockerContainerLaunch:297
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
{code}
was:
/sys/fs/cgroup doesn't exist on the Mac. And the tests seem to fail because of
this.
{code}
Failed tests:
TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testDockerContainerLaunch:297
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
{code}
> TestDockerContainerRuntime fails on the mac
> ---
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5596) TestDockerContainerRuntime fails on the mac
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449822#comment-15449822
]
Sidharta Seethana commented on YARN-5596:
-
I'll take a look at this. Patch coming up.
> TestDockerContainerRuntime fails on the mac
> ---
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
>
> /sys/fs/cgroup doesn't exist on the Mac. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5596) TestDockerContainerRuntime fails on the mac
Sidharta Seethana created YARN-5596:
---
Summary: TestDockerContainerRuntime fails on the mac
Key: YARN-5596
URL: https://issues.apache.org/jira/browse/YARN-5596
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, yarn
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Priority: Minor
/sys/fs/cgroup doesn't exist on the Mac. And the tests seem to fail because of
this.
{code}
Failed tests:
TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
TestDockerContainerRuntime.testDockerContainerLaunch:297
expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
/]test_container_local...> but was:<...ET_BIND_SERVICE -v
/[]test_container_local...>
Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5042) Mount /sys/fs/cgroup into Docker containers as read only mount
[ https://issues.apache.org/jira/browse/YARN-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433544#comment-15433544 ] Sidharta Seethana commented on YARN-5042: - +1 for the latest patch from me as well. Though the existing default behavior is to create the mounts on the host system if they don't exist, I think it makes sense to not create container work dir, log dirs etc if they don't exist. That can be taken up in a separate JIRA. > Mount /sys/fs/cgroup into Docker containers as read only mount > -- > > Key: YARN-5042 > URL: https://issues.apache.org/jira/browse/YARN-5042 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: luhuichun > Attachments: YARN-5042.001.patch, YARN-5042.002.patch, > YARN-5042.003.patch, YARN-5042.004.patch, YARN-5042.005.patch > > > Containers running systemd need access to /sys/fs/cgroup. We should mount it > into the container as a read only mount. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422403#comment-15422403 ] Sidharta Seethana commented on YARN-4266: - [~tangzhankun], please feel free to take this over. Do you have a proposal for dealing with logs, directories etc? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5394) Remove bind-mount /etc/passwd to Docker Container
[
https://issues.apache.org/jira/browse/YARN-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408082#comment-15408082
]
Sidharta Seethana commented on YARN-5394:
-
+1 from me too.
> Remove bind-mount /etc/passwd to Docker Container
> -
>
> Key: YARN-5394
> URL: https://issues.apache.org/jira/browse/YARN-5394
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5394.002.patch, YARN-5394.003.patch
>
>
> Current LCE (DockerLinuxContainerRuntime) is mounting /etc/passwd to the
> container. And it seems uses wrong file name "/etc/password" for container.
> {panel}
> .addMountLocation("/etc/passwd", "/etc/password:ro");
> {panel}
> The biggest issue of bind-mount /etc/passwd is that it overrides the users
> defined in Docker image which is not expected. Remove it won't affect
> existing use cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5042) Mount /sys/fs/cgroup into Docker containers as read only mount
[ https://issues.apache.org/jira/browse/YARN-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403039#comment-15403039 ] Sidharta Seethana commented on YARN-5042: - [~luhuichun], the latest jenkins run still shows broken unit tests - could you please update the tests? > Mount /sys/fs/cgroup into Docker containers as read only mount > -- > > Key: YARN-5042 > URL: https://issues.apache.org/jira/browse/YARN-5042 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: luhuichun > Attachments: YARN-5042.001.patch, YARN-5042.002.patch > > > Containers running systemd need access to /sys/fs/cgroup. We should mount it > into the container as a read only mount. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5394) Remove bind-mount /etc/passwd to Docker Container
[
https://issues.apache.org/jira/browse/YARN-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399675#comment-15399675
]
Sidharta Seethana commented on YARN-5394:
-
[~tangzhankun], could you please submit the patch to jenkins if its ready for
review?
> Remove bind-mount /etc/passwd to Docker Container
> -
>
> Key: YARN-5394
> URL: https://issues.apache.org/jira/browse/YARN-5394
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5394-branch-2.8.001.patch,
> YARN-5394-branch-2.8.002.patch
>
>
> Current LCE (DockerLinuxContainerRuntime) is mounting /etc/passwd to the
> container. And it seems uses wrong file name "/etc/password" for container.
> {panel}
> .addMountLocation("/etc/passwd", "/etc/password:ro");
> {panel}
> The biggest issue of bind-mount /etc/passwd is that it overrides the users
> defined in Docker image which is not expected. Remove it won't affect
> existing use cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Decouple host user and Docker container user
[ https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398593#comment-15398593 ] Sidharta Seethana commented on YARN-5360: - To clarify a few things here - yes, customized images are required in some cases (especially in secure mode) to make apps work for certain users. This is a limitation we have to work with for the moment given the hadoop security model - it may not be reasonable or practical to drop it altogether except under controlled situations. Also, log aggregation does not work in secure mode if you drop "--user" (it works in non-secure mode, I think but I'll have to check the code/test again). Artifact deletion will not work if the artifacts are created as a different user in the docker container (artifact cleanup is done as the 'run as' user). In your first table above, the yarn/nobody case likely did not work because the sequenceiq image is based on centos (nobody uid=99) and the system you were testing on was not centos (ubuntu? nobody uid=65534). We have tested spark with other images on centos (I have test images on docker hub if you'd like to try). I am pretty sure [~templedf] has successfully run spark using the current implementation as well. ([~templedf] : please confirm) . If this the discussion here is now only about dropping the "--user" in certain cases, this is captured in YARN-4266. > Decouple host user and Docker container user > > > Key: YARN-5360 > URL: https://issues.apache.org/jira/browse/YARN-5360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > > There is *a dependency between job submitting user and the user in the Docker > image* in LCE currently. For instance, in order to run the Docker container > as yarn user, we can choose set the > "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn > and leave > "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" > default (true). Then LCE will choose yarn ( UID maybe 1001) as the user > running jobs. > LCE will mount the generated launch_container.sh (owned by the running job > user) and /etc/passwd (*current the code is mounting to container's > /etc/password, I think it's a mistake*) into the Docker container and > utilizes "docker run --user=" option to get it done internally. > Mounting /etc/passwd to the container is a not good choice due to override > original users defined in Docker image. As far as I know, since Docker v1.8 > (or maybe earlier), the Docker run command "--user=" option accepts UID and > *when passing UID, the user does not have to exist in the container*. So we > could use UID instead of user name to construct the Docker run command to > eliminate the dependency that create the same user in the Docker image. This > enables LCE the ability to launch any Docker container safely regardless what > users in it. > But this is not enough to decouple host user and Docker container user. The > final solution we are searching for are focused on allowing users to run > their Docker images flexibly without involving dependencies of YARN and make > sure the container won't bring in security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Decouple host user and Docker container user
[ https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398111#comment-15398111 ] Sidharta Seethana commented on YARN-5360: - [~tangzhankun], dropping the '--user' works in this because you are running the container processes as root - once the processes are run root, the restrictive permissions on launch_container.sh don't matter. An image that is setup to use a different arbitrary user will not work. To reiterate, I don't believe it makes sense to have different default behavior for secure/non-secure modes - we should find a model that works across both. > Decouple host user and Docker container user > > > Key: YARN-5360 > URL: https://issues.apache.org/jira/browse/YARN-5360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > > There is *a dependency between job submitting user and the user in the Docker > image* in LCE currently. For instance, in order to run the Docker container > as yarn user, we can choose set the > "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn > and leave > "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" > default (true). Then LCE will choose yarn ( UID maybe 1001) as the user > running jobs. > LCE will mount the generated launch_container.sh (owned by the running job > user) and /etc/passwd (*current the code is mounting to container's > /etc/password, I think it's a mistake*) into the Docker container and > utilizes "docker run --user=" option to get it done internally. > Mounting /etc/passwd to the container is a not good choice due to override > original users defined in Docker image. As far as I know, since Docker v1.8 > (or maybe earlier), the Docker run command "--user=" option accepts UID and > *when passing UID, the user does not have to exist in the container*. So we > could use UID instead of user name to construct the Docker run command to > eliminate the dependency that create the same user in the Docker image. This > enables LCE the ability to launch any Docker container safely regardless what > users in it. > But this is not enough to decouple host user and Docker container user. The > final solution we are searching for are focused on allowing users to run > their Docker images flexibly without involving dependencies of YARN and make > sure the container won't bring in security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3531) Make "good" local directories available to ContainerExecutors at initialization time
[ https://issues.apache.org/jira/browse/YARN-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana reassigned YARN-3531: --- Assignee: Sidharta Seethana (was: Yufei Gu) > Make "good" local directories available to ContainerExecutors at > initialization time > > > Key: YARN-3531 > URL: https://issues.apache.org/jira/browse/YARN-3531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > > Currently, in the NodeManager's serviceInit() function, the configured > executor is initialized before the node health checker/directory handler > service are initialized. There are use cases where executor initialization > requires access to 'good' local directories ( e.g for creation of temporary > files , see YARN-3366 ). We need to figure out a way to make this possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Decouple host user and Docker container user
[
https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392734#comment-15392734
]
Sidharta Seethana commented on YARN-5360:
-
{quote}
For under what conditions to drop the "--user", I prefer that in non-secure
mode, we drop it by default and leave the chance of setting it to the
application thru interface (default yarn container launch is not affected,
still run the container based on configuration). But in secure mode,
application is not allowed to run Docker container as arbitrary user.
{quote}
I am afraid I disagree with this. Having such completely different default
behavior in secure vs non-secure mode can lead to a lot of confusion and break
existing supported use cases. Dropping --user by default, for example would
break spark/map-reduce in non-secure mode.
> Decouple host user and Docker container user
>
>
> Key: YARN-5360
> URL: https://issues.apache.org/jira/browse/YARN-5360
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>
> There is *a dependency between job submitting user and the user in the Docker
> image* in LCE currently. For instance, in order to run the Docker container
> as yarn user, we can choose set the
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn
> and leave
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users"
> default (true). Then LCE will choose yarn ( UID maybe 1001) as the user
> running jobs.
> LCE will mount the generated launch_container.sh (owned by the running job
> user) and /etc/passwd (*current the code is mounting to container's
> /etc/password, I think it's a mistake*) into the Docker container and
> utilizes "docker run --user=" option to get it done internally.
> Mounting /etc/passwd to the container is a not good choice due to override
> original users defined in Docker image. As far as I know, since Docker v1.8
> (or maybe earlier), the Docker run command "--user=" option accepts UID and
> *when passing UID, the user does not have to exist in the container*. So we
> could use UID instead of user name to construct the Docker run command to
> eliminate the dependency that create the same user in the Docker image. This
> enables LCE the ability to launch any Docker container safely regardless what
> users in it.
> But this is not enough to decouple host user and Docker container user. The
> final solution we are searching for are focused on allowing users to run
> their Docker images flexibly without involving dependencies of YARN and make
> sure the container won't bring in security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5394) Correct the wrong file name when mounting /etc/passwd to Docker Container
[
https://issues.apache.org/jira/browse/YARN-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392727#comment-15392727
]
Sidharta Seethana commented on YARN-5394:
-
Unfortunately, "correcting the typo" would change the current behavior and this
could be disruptive. Removing the (bad) mount altogether and fixing it
correctly in a different JIRA - that seems fine to me. Thoughts ?
> Correct the wrong file name when mounting /etc/passwd to Docker Container
> -
>
> Key: YARN-5394
> URL: https://issues.apache.org/jira/browse/YARN-5394
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5394-branch-2.8.001.patch
>
>
> Current LCE (DockerLinuxContainerRuntime) is mounting /etc/passwd to the
> container. But it seems to use wrong file name "/etc/password" for container.
> {panel}
> .addMountLocation("/etc/passwd", "/etc/password:ro");
> {panel}
> This causes LCE failed to launch the Docker container if the Docker images
> don't create the same user name and UID in it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392721#comment-15392721 ] Sidharta Seethana commented on YARN-3854: - YARN-3289 was filed in the context of DockerContainerExecutor - though the discussion there is still quite relevant. My preference would be to keep all discussions related to features/fixes here under the YARN-3611 umbrella JIRA . > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf > > > We need the ability to localize docker images when those images aren't > already available locally. There are various approaches that could be used > here with different trade-offs/issues : image archives on HDFS + docker load > , docker pull during the localization phase or (automatic) docker pull > during the run/launch phase. > We also need the ability to clean-up old/stale, unused images. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3531) Make "good" local directories available to ContainerExecutors at initialization time
[ https://issues.apache.org/jira/browse/YARN-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390275#comment-15390275 ] Sidharta Seethana commented on YARN-3531: - [~yufeigu], are you still working on this JIRA? If not, would you mind if I took it over? Please let me know either way. Thanks! > Make "good" local directories available to ContainerExecutors at > initialization time > > > Key: YARN-3531 > URL: https://issues.apache.org/jira/browse/YARN-3531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Yufei Gu > > Currently, in the NodeManager's serviceInit() function, the configured > executor is initialized before the node health checker/directory handler > service are initialized. There are use cases where executor initialization > requires access to 'good' local directories ( e.g for creation of temporary > files , see YARN-3366 ). We need to figure out a way to make this possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-3854: Description: We need the ability to localize docker images when those images aren't already available locally. There are various approaches that could be used here with different trade-offs/issues : image archives on HDFS + docker load , docker pull during the localization phase or (automatic) docker pull during the run/launch phase. We also need the ability to clean-up old/stale, unused images. was:We need the ability to localize images from HDFS and load them for use when launching docker containers. > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf > > > We need the ability to localize docker images when those images aren't > already available locally. There are various approaches that could be used > here with different trade-offs/issues : image archives on HDFS + docker load > , docker pull during the localization phase or (automatic) docker pull > during the run/launch phase. > We also need the ability to clean-up old/stale, unused images. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390226#comment-15390226 ] Sidharta Seethana commented on YARN-3854: - I updated the description based on our discussion here. > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf > > > We need the ability to localize docker images when those images aren't > already available locally. There are various approaches that could be used > here with different trade-offs/issues : image archives on HDFS + docker load > , docker pull during the localization phase or (automatic) docker pull > during the run/launch phase. > We also need the ability to clean-up old/stale, unused images. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3854) Add localization support for docker images
[
https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386568#comment-15386568
]
Sidharta Seethana commented on YARN-3854:
-
I had spent some time looking into this as well. My biggest concern with using
HDFS + save/load is that it is very easy for users to clobber each others
images. This can lead to some hard to trace bugs and also has security
implications. Here is one example - UserA is using fedora:latest on a given
node. UserB comes along, localizes their own image file and triggers a docker
load. If this image is also tagged fedora:latest - the existing tag is
replaced. This is shown below (on a single node - the example is somewhat
contrived, but it demonstrates the issue I am describing )
{code}
sidharta-laptop (13:11:11) /tmp$ docker images | grep ^centos
centos centos6 ae6154bcd4459 weeks
ago 194.5 MB
sidharta-laptop (13:11:13) /tmp$ docker images | grep ^fedora
sidharta-laptop (13:11:35) /tmp$ docker tag ae6154bcd445 fedora:latest
sidharta-laptop (13:12:40) /tmp$ docker images | grep ^fedora
fedora latest ae6154bcd4459 weeks
ago 194.5 MB
sidharta-laptop (13:12:42) /tmp$ docker save fedora:latest > fedora.tar
sidharta-laptop (13:14:05) /tmp$ docker rmi fedora:latest
Untagged: fedora:latest
sidharta-laptop (13:14:19) /tmp$ docker pull fedora:latest
latest: Pulling from library/fedora
7c91a140e7a1: Pull complete
Digest: sha256:a97914edb6ba15deb5c5acf87bd6bd5b6b0408c96f48a5cbd450b5b04509bb7d
Status: Downloaded newer image for fedora:latest
sidharta-laptop (13:14:37) /tmp$ docker images | grep ^fedora
fedora latest f9873d5305884 weeks
ago 204.4 MB
sidharta-laptop (13:14:43) /tmp$ docker load -i fedora.tar
The image fedora:latest already exists, renaming the old one with ID
sha256:f9873d530588316311ac1d3d15e95487b947f5d8b560e72bdd6eb73a7831b2c4 to
empty string
Loaded image: fedora:latest
sidharta-laptop (13:15:01) /tmp$ docker images | grep ^fedora
fedora f9873d5305884 weeks
ago 204.4 MB
fedora latest ae6154bcd4459 weeks
ago 194.5 MB
sidharta-laptop (13:15:15) /tmp$
{code}
It is very easy for a UserB to wreak havoc on UserA by reusing tags. You can
see above that the 'official' fedora:latest tag got replaced by one loaded from
an archive. Using a registry directly alleviates some of the concerns - but
there are other aspects to consider as mentioned by [~templedf] and
[~shaneku...@gmail.com].
> Add localization support for docker images
> --
>
> Key: YARN-3854
> URL: https://issues.apache.org/jira/browse/YARN-3854
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-3854-branch-2.8.001.patch,
> YARN-3854_Localization_support_for_Docker_image_v1.pdf,
> YARN-3854_Localization_support_for_Docker_image_v2.pdf
>
>
> We need the ability to localize images from HDFS and load them for use when
> launching docker containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5394) Correct the wrong file name when mounting /etc/passwd to Docker Container
[
https://issues.apache.org/jira/browse/YARN-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384999#comment-15384999
]
Sidharta Seethana commented on YARN-5394:
-
/cc [~zyluo], [~vvasudev]
As discussed in YARN-5360, I don't think its a good idea to mount /etc/passwd
into a container without any way to disable it. At a minimum, we should add a
(cluster-wide?) mechanism to control this (and it should be disabled by
default, IMO).
> Correct the wrong file name when mounting /etc/passwd to Docker Container
> -
>
> Key: YARN-5394
> URL: https://issues.apache.org/jira/browse/YARN-5394
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5394-branch-2.8.001.patch
>
>
> Current LCE (DockerLinuxContainerRuntime) is mounting /etc/passwd to the
> container. But it seems to use wrong file name "/etc/password" for container.
> {panel}
> .addMountLocation("/etc/passwd", "/etc/password:ro");
> {panel}
> This causes LCE failed to launch the Docker container if the Docker images
> don't create the same user name and UID in it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Decouple host user and Docker container user
[ https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384786#comment-15384786 ] Sidharta Seethana commented on YARN-5360: - [~templedf], running a container as root does in fact have security implications (there are other things to consider in conjunction with this - capabilites, selinux and so on). There are (at least) a couple of reasons why --user is enforced currently : 1) YARN security model requires the launched process run as the designated user 2) Log aggregation/local permissions etc - some of these things would stop working if the generated logs have ownership that is different from what YARN expects. These are also the reasons that need to be considered for YARN-4266 > Decouple host user and Docker container user > > > Key: YARN-5360 > URL: https://issues.apache.org/jira/browse/YARN-5360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > > There is *a dependency between job submitting user and the user in the Docker > image* in LCE currently. For instance, in order to run the Docker container > as yarn user, we can choose set the > "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn > and leave > "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" > default (true). Then LCE will choose yarn ( UID maybe 1001) as the user > running jobs. > LCE will mount the generated launch_container.sh (owned by the running job > user) and /etc/passwd (*current the code is mounting to container's > /etc/password, I think it's a mistake*) into the Docker container and > utilizes "docker run --user=" option to get it done internally. > Mounting /etc/passwd to the container is a not good choice due to override > original users defined in Docker image. As far as I know, since Docker v1.8 > (or maybe earlier), the Docker run command "--user=" option accepts UID and > *when passing UID, the user does not have to exist in the container*. So we > could use UID instead of user name to construct the Docker run command to > eliminate the dependency that create the same user in the Docker image. This > enables LCE the ability to launch any Docker container safely regardless what > users in it. > But this is not enough to decouple host user and Docker container user. The > final solution we are searching for are focused on allowing users to run > their Docker images flexibly without involving dependencies of YARN and make > sure the container won't bring in security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Decouple host user and Docker container user
[
https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384777#comment-15384777
]
Sidharta Seethana commented on YARN-5360:
-
[~zyluo],
{quote}
I think this is inconsistent with Docker's motto to "build, ship and run".
There is no point of using Docker if the user has to use every image as a base
to add the correct user.
{quote}
While that may be Docker's motto - the objective of YARN-3611, in my opinion
has never been to use docker for docker's sake - we needed to adapt it to the
YARN/hadoop world - hadoop security, log aggregation, localization - all of
these need to work.
> Decouple host user and Docker container user
>
>
> Key: YARN-5360
> URL: https://issues.apache.org/jira/browse/YARN-5360
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>
> There is *a dependency between job submitting user and the user in the Docker
> image* in LCE currently. For instance, in order to run the Docker container
> as yarn user, we can choose set the
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn
> and leave
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users"
> default (true). Then LCE will choose yarn ( UID maybe 1001) as the user
> running jobs.
> LCE will mount the generated launch_container.sh (owned by the running job
> user) and /etc/passwd (*current the code is mounting to container's
> /etc/password, I think it's a mistake*) into the Docker container and
> utilizes "docker run --user=" option to get it done internally.
> Mounting /etc/passwd to the container is a not good choice due to override
> original users defined in Docker image. As far as I know, since Docker v1.8
> (or maybe earlier), the Docker run command "--user=" option accepts UID and
> *when passing UID, the user does not have to exist in the container*. So we
> could use UID instead of user name to construct the Docker run command to
> eliminate the dependency that create the same user in the Docker image. This
> enables LCE the ability to launch any Docker container safely regardless what
> users in it.
> But this is not enough to decouple host user and Docker container user. The
> final solution we are searching for are focused on allowing users to run
> their Docker images flexibly without involving dependencies of YARN and make
> sure the container won't bring in security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378155#comment-15378155 ] Sidharta Seethana commented on YARN-5298: - hi [~vvasudev], the resolution field is set to "Cannot Reproduce" - is this intentional? > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: Sidharta Seethana > Fix For: 2.9.0 > > Attachments: YARN-5298.001.patch, YARN-5298.002.patch > > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375910#comment-15375910 ] Sidharta Seethana commented on YARN-5298: - Thanks, [~vvasudev] and [~templedf] ! [~vvasudev] , about the container specific directories : The docker container runtime itself makes no assumptions about the location of the container specific directories/non container-specific directories. It does not know of or assume a parent/sub-dir structure and explicitly mounts all required directories. I hope that answers your question. > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: Sidharta Seethana > Attachments: YARN-5298.001.patch, YARN-5298.002.patch > > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5360) Use UID instead of user name to build the Docker run command
[ https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374621#comment-15374621 ] Sidharta Seethana commented on YARN-5360: - To be honest, I am not in favor of going down the path of using ‘uid’s instead of the actual username. Unix usernames are more readable and using the uid instead of the unix username only ensures that the container can be launched using that uid but does not guarantee that container/image will work correctly using that uid. For example, launching a container as uid 99 (nobody in centos) does not guarantee that an ubuntu container will work correctly when launched with using uid 99. In general it is good practice to ensure that image being launched is setup correctly for the user(s) that are used to launch the image. We also need to consider if ‘run as user’ can be turned off for some users (see YARN-4266). About /etc/passwd being mounted incorrectly, I am not quite sure how that crept in - [~ashahab] and I were working on initial patches for docker support and this was moved over from DockerContainerExecutor ([~ashahab], know the background behind this?). In any case, I don’t think we should be always mounting /etc/passwd from the host (or at least we should make it a cluster-wide option). Always mounting /etc/passwd would mean that any additional users that are already in the image will be unavailable (examples include images with users like apache, jenkins and so on that may not be available on the cluster). We also need to consider the implications of user namespaces/user remapping that is supported by docker int recent versions (still a work in progress I think). Given all of these, I think switching to uids now would only make things more complicated and behavior difficult to understand. > Use UID instead of user name to build the Docker run command > > > Key: YARN-5360 > URL: https://issues.apache.org/jira/browse/YARN-5360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > > There is *a dependency between job submitting user and the user in the Docker > image* in LCE currently. For instance, in order to run the Docker container > as yarn user, we can choose set the > "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn > and leave > "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" > default (true). Then LCE will choose yarn ( UID maybe 1001) as the user > running jobs. > LCE will mount the generated launch_container.sh (owned by the running job > user) and /etc/passwd (*current the code is mounting to container's > /etc/password, I think it's a mistake*) into the Docker container and > utilizes "docker run --user=" option to get it done internally. > But I don't think mounting /etc/passwd to the container is a good choice. As > far as I know, since Docker v1.8 (or maybe earlier), the Docker run command > "--user=" option accepts UID and *when passing UID, the user does not have to > exist in the container*. So we should use UID instead of user name to > construct the Docker run command to eliminate the dependency that create the > same user in the Docker image. This enables LCE the ability to launch any > Docker container safely regardless what users in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5298: Attachment: YARN-5298.002.patch Uploaded a new patch with some checkstyle fixes. Some of the checkstyle issues seem to correspond to pre-existing code. The test failure is unrelated. [~vvasudev], could you please review the patch? > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: Sidharta Seethana > Attachments: YARN-5298.001.patch, YARN-5298.002.patch > > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-5298: Attachment: YARN-5298.001.patch Uploaded a first version of the patch. > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: Sidharta Seethana > Attachments: YARN-5298.001.patch > > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364957#comment-15364957 ] Sidharta Seethana commented on YARN-3854: - [~tangzhankun], thanks for uploading the doc and the patch. I'll take a look at them both and get back to you. > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf > > > We need the ability to localize images from HDFS and load them for use when > launching docker containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351517#comment-15351517 ] Sidharta Seethana commented on YARN-5298: - Thanks for filing this ticket, [~vvasudev] ! I'll take a look at this. > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5298) Mount usercache and NM filecache directories into Docker container
[ https://issues.apache.org/jira/browse/YARN-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana reassigned YARN-5298: --- Assignee: Sidharta Seethana > Mount usercache and NM filecache directories into Docker container > -- > > Key: YARN-5298 > URL: https://issues.apache.org/jira/browse/YARN-5298 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Varun Vasudev >Assignee: Sidharta Seethana > > Currently, we don't mount the usercache and the NM filecache directories into > the Docker container. This can lead to issues with containers that rely on > public and application scope resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345220#comment-15345220 ] Sidharta Seethana commented on YARN-5168: - YARN today does not dynamically allocate/manage ports for containers, nor does it provide a way to return arbitrary container attributes back from the executor layer. For this to work end to end (for example to avoid port collisions), YARN also needs to add support for scheduling based on ports. This makes adding port support specifically for bridge mode quite a large task. Also, bridge mode itself has certain limitations - since the IPs assigned are (usually) not externally routable, certain kinds of applications will run into issues ( e.g applications/tasks that need both forward/reverse lookups to work). IMO, using a container SDN (that allows containers to see each other and the outside world to see them) along with a discovery mechanism (e.g YARN-4757) is a better path forward. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
