[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696510#comment-16696510 ] Charo Zhang edited comment on YARN-8986 at 11/23/18 8:39 AM: - [~eyang] All unit test cases passed the in 009 patch.:D Is there anything else I can do for this issue. was (Author: charo zhang): [~eyang] All unit test cases passed the in 008 patch.:D Is there anything else I can do for this issue. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch, > YARN-8986.009.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696510#comment-16696510 ] Charo Zhang edited comment on YARN-8986 at 11/23/18 8:40 AM: - [~eyang] All unit test cases passed in the 009 patch.:D Is there anything else I can do for this issue. was (Author: charo zhang): [~eyang] All unit test cases passed the in 009 patch.:D Is there anything else I can do for this issue. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch, > YARN-8986.009.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696395#comment-16696395
]
Eric Yang edited comment on YARN-8986 at 11/23/18 5:37 AM:
---
[~Charo Zhang] The test can be run locally using in
hadoop-yarn-server-nodemanager:
{code}
mvn test -Dtest=cetest -Pnative
{code}
The result will be stored in target/surefire-reports/cetest.*.
Test results output link also show the reason of the
[failures|https://builds.apache.org/job/PreCommit-YARN-Build/22680/testReport/].
The test cases passed fine without this patch. Net=bridge is specified, and
causing docker network inspect to fail, and returned error result.
was (Author: eyang):
[~Charo Zhang] The test can be run locally using in
hadoop-yarn-server-nodemanager:
{code}
mvn test -Dtest=cetest -Pnative
{code}
Test results output link also show the reason of the
[failures|https://builds.apache.org/job/PreCommit-YARN-Build/22680/testReport/].
The test cases passed fine without this patch. Net=bridge is specified, and
causing docker network inspect to fail, and returned error result.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695628#comment-16695628
]
Xun Liu edited comment on YARN-8986 at 11/22/18 7:49 AM:
-
hi, [~Charo Zhang], [~eyang]
{code:java}
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
was (Author: liuxun323):
hi, [~Charo Zhang], [~eyang]
{code:java}
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695040#comment-16695040
]
Eric Yang edited comment on YARN-8986 at 11/21/18 6:04 PM:
---
[~Charo Zhang] Jenkins runs with docker 17.05.0-ce. Hadoop compilation runs
inside docker container, which does not have docker binary available to unit
test. This is the reason that it fails. If you change
add_ports_mapping_to_command, then it will work correctly:
{code}
+ char *network_name = get_configuration_value("net",
DOCKER_COMMAND_FILE_SECTION, command_config);
+ if (network_name == NULL) {
+network_name = make_string("%s", "bridge");
+ }
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+ FILE* docker_network = popen(docker_network_command, "r");
+ ret = fscanf(docker_network, "%s", network_type);
+ if (pclose (docker_network) != 0 || ret <= 0) {
+fprintf (ERRORFILE, "Could not inspect docker network to get type %s.\n",
docker_network_command);
+goto cleanup;
+ }
{code}
to
{code}
+ char *network_name = get_configuration_value("net",
DOCKER_COMMAND_FILE_SECTION, command_config);
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }
{code}
If network name is not specified, it will be allowed to use -p and -P flags.
This side step the requirement to have docker binary available to unit test and
validate the default case is working. It would be possible the refactor the
first 3 lines into another function and test the docker_network_command for
null and also artificial name for generating docker_network_command for maximum
coverage, but it is okay if you don't do the refactoring because some existing
unit tests should not fail when net parameter is not passed. It would be okay
to skip the actual docker command execution in unit test, since we already unit
test that part manually in our discussions.
was (Author: eyang):
[~Charo Zhang] Jenkins runs with docker 17.05.0-ce. Hadoop compilation runs
inside docker container, which does not have docker binary available to unit
test. This is the reason that it fails. If you change
add_ports_mapping_to_command, then it will work correctly:
{code}
+ if (network_name == NULL) {
+network_name = make_string("%s", "bridge");
+ }
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+ FILE* docker_network = popen(docker_network_command, "r");
+ ret = fscanf(docker_network, "%s", network_type);
+ if (pclose (docker_network) != 0 || ret <= 0) {
+fprintf (ERRORFILE, "Could not inspect docker network to get type %s.\n",
docker_network_command);
+goto cleanup;
+ }
{code}
to
{code}
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }
{code}
If network name is not specified, it will be allowed to use -p and -P flags.
This side step the requirement to have docker binary available to unit test and
validate the default case is working. It would be possible the refactor the
first 3 lines into another function and test the docker_network_command for
null and also artificial name for generating
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694740#comment-16694740
]
Charo Zhang edited comment on YARN-8986 at 11/21/18 2:36 PM:
-
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
the first test case exit code is -1, which means docker network inspect
command execute failed.
was (Author: charo zhang):
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
-1 exit cod mean docker network inspect command execute failed.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694740#comment-16694740
]
Charo Zhang edited comment on YARN-8986 at 11/21/18 2:35 PM:
-
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
-1 exit cod mean docker network inspect command execute failed.
was (Author: charo zhang):
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694313#comment-16694313 ] Charo Zhang edited comment on YARN-8986 at 11/21/18 7:53 AM: - [~eyang] I know what your mean, it will use bridge network as default network type if ''--net" option don't exist for docker run. But in my opinion: the default network is controlled by yarn.nodemanager.runtime.linux.docker.default-container-network for Yarn Docker, we should't consider Docker default network type. That's mean when network name is not specified( it 's same to "net" is null in command_config), it's better to use yarn.nodemanager.runtime.linux.docker.default-container-network value or add "--net=none". was (Author: charo zhang): [~eyang] I know what your mean, it will use bridge network as default network type if ''--net" option don't exist for docker run. But in my opinion: the default network is controlled by yarn.nodemanager.runtime.linux.docker.default-container-network for Yarn Docker, we should't consider Docker default network type. That's mean when network name is not specified( it 's same to "net" is null in command_config), it's better to use yarn.nodemanager.runtime.linux.docker.default-container-network value or add "--net=none". At same time , i set network_name=bridge when it's not specified as you said in 007 patch. Hope it's the last one. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694313#comment-16694313 ] Charo Zhang edited comment on YARN-8986 at 11/21/18 7:42 AM: - [~eyang] I know what your mean, it will use bridge network as default network type if ''--net" option don't exist for docker run. But in my opinion: the default network is controlled by yarn.nodemanager.runtime.linux.docker.default-container-network for Yarn Docker, we should't consider Docker default network type. That's mean when network name is not specified( it 's same to "net" is null in command_config), it's better to use yarn.nodemanager.runtime.linux.docker.default-container-network value or add "--net=none". At same time , i set network_name=bridge when it's not specified as you said in 007 patch. Hope it's the last one. was (Author: charo zhang): [~eyang] I know what your mean, it will use bridge network as default network type if ''--net" option don't exist for docker run. But in my opinion: the default network is controlled by yarn.nodemanager.runtime.linux.docker.default-container-network for Yarn Docker, we should't consider Docker default network type. That's mean when network name is not specified( it 's same to "net" is null in command_config), it's better to use yarn.nodemanager.runtime.linux.docker.default-container-network value or add "--net=none". > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693797#comment-16693797 ] Charo Zhang edited comment on YARN-8986 at 11/20/18 9:29 PM: - [~eyang] I think network_name is allowed to not exist, and i find set_network return 0 when do not get "net" key. so i process it with the same way in add_ports_mapping_to_command, if network_name is null, it return 0 to do nothing, there is no need to set default network is bridge. At same time, i add two more test cases in test_docker_util.cc if network_name not exist. The latest 005 patch is uploaded. was (Author: charo zhang): [~eyang] I think network_name is allowed to not exist, and i find set_network return 0 when do not get "net" key. so i process it with the same way in add_ports_mapping_to_command, if network_name is null, it return 0 to do nothing, there is not need to set default network is bridge. At same time, i add two more test cases in test_docker_util.cc if network_name not exist. The latest 005 patch is uploaded. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690921#comment-16690921 ] Charo Zhang edited comment on YARN-8986 at 11/18/18 2:58 PM: - [~eyang] According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and validation of ports-mapping in java code, there is no need to check again. was (Author: charo zhang): [~eyang] According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and validation of ports-mapping in java code, there was no need to check again. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690921#comment-16690921 ] Charo Zhang edited comment on YARN-8986 at 11/18/18 2:57 PM: - [~eyang] According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and validation of ports-mapping in java code, there was no need to check again. was (Author: charo zhang): [~eyang] According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and Validation of ports-mapping in java code, there's no need to check again. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690921#comment-16690921 ] Charo Zhang edited comment on YARN-8986 at 11/18/18 2:56 PM: - [~eyang] According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and Validation of ports-mapping in java code, there's no need to check again. was (Author: charo zhang): [~eyang]According to you suggestions, i verify the network_name against the allowed list, and update a validation of ports-mapping format in container-executor. Please review again, I'm sorry to bother you so much.:D I thought there is a verification network_name in "set_network" and Validation of ports-mapping in java code, there's no need to check again. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441
]
Charo Zhang edited comment on YARN-8986 at 11/17/18 9:59 AM:
-
[~eyang] Does yarn have version requirement for Docker. I am using Docker
1.9.1, I didn't find "--format" option for "docker network ls ".
I update my Docker to latest version, and modify the patch from
{code:c}
char *docker_network_command = make_string("%s network ls | grep %s | awk
'{print $3}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
to
{code:c}
char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
The latest patch has uploaded, do you think that is ok?
was (Author: charo zhang):
[~eyang] Does yarn have version requirement for Docker. I am using Docker
1.9.1, I didn't find "--format" option for "docker network ls ".
I update my Docker to latest version, and modify the patch from
{code:c}
char *docker_network_command = make_string("%s network ls | grep %s | awk
'{print $3}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
to
{code:c}
char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
The latest patch has uploaded, do you that is ok?
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441
]
Charo Zhang edited comment on YARN-8986 at 11/17/18 9:53 AM:
-
[~eyang] Does yarn have version requirement for Docker. I am using Docker
1.9.1, I didn't find "--format" option for "docker network ls ".
I update my Docker to latest version, and modify the patch from
{code:c}
char *docker_network_command = make_string("%s network ls | grep %s | awk
'{print $3}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
to
{code:c}
char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
FILE* docker_network = popen(docker_network_command, "r");
{code}
The latest patch has uploaded, do you that is ok?
was (Author: charo zhang):
[~eyang] Does yarn have version requirement for Docker. I am using Docker
1.9.1, I didn't find "--format" option for "docker network ls ".
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441 ] Charo Zhang edited comment on YARN-8986 at 11/17/18 8:32 AM: - [~eyang] Does yarn have version requirement for Docker. I am using Docker 1.9.1, I didn't find "--format" option for "docker network ls ". was (Author: charo zhang): [~eyang] Does yarn have version requirement for Docker. I am using Docker 1.9.1, I can not find "--format" option for "docker network ls ". > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441 ] Charo Zhang edited comment on YARN-8986 at 11/17/18 8:31 AM: - [~eyang] Does yarn have version requirement for Docker. I am using Docker 1.9.1, I can not find "--format" option for "docker network ls ". was (Author: charo zhang): [~eyang] Does yarn have version requirement for Docker. I am using Docker 1.9.1, do not find "--format" option for "docker network ls ". > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441 ] Charo Zhang edited comment on YARN-8986 at 11/17/18 8:28 AM: - [~eyang] Does yarn have version requirement for Docker. I am using Docker 1.9.1, do not find "--format" option for "docker network ls ". was (Author: charo zhang): [~eyang] Dose yarn have version requirement for Docker. I am using Docker 1.9.1, do not find "--format" option for "docker network ls ". > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690441#comment-16690441 ] Charo Zhang edited comment on YARN-8986 at 11/17/18 8:27 AM: - [~eyang] Dose yarn have version requirement for Docker. I am using Docker 1.9.1, do not find "--format" option for "docker network ls ". was (Author: charo zhang): [~eyang]Dose yarn have version requirement for Docker. I am using Docker 1.9.1, do not find "--format" option for "docker network ls ". > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689356#comment-16689356 ] Charo Zhang edited comment on YARN-8986 at 11/16/18 3:41 PM: - [~eyang] Thanks for you suggestions. About 2/3/4 point, i have modified according to you opinion. but about the first suggestion: 1, network name is only allowed by yarn.nodemanager.runtime.linux.docker.allowed-container-networks configuration, so i think "null >/dev/null && rm -rf *" can not pass to the commmand. 2, Another reason, i didn't find a better way to get network type except command with pipe.:( was (Author: charo zhang): [~eyang] Thanks for you suggestions. About 2/3/4 point, i have Modified according to you opinion. but about the first suggestion: 1, network name is only allowed by yarn.nodemanager.runtime.linux.docker.allowed-container-networks configuration, so i think "null >/dev/null && rm -rf *" can not pass to the commmand. 2, Another reason, i didn't find a better way to get network type except command with pipe.:( > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689356#comment-16689356 ] Charo Zhang edited comment on YARN-8986 at 11/16/18 3:41 PM: - [~eyang] Thanks for you suggestions. About 2/3/4 point, i have Modified according to you opinion. but about the first suggestion: 1, network name is only allowed by yarn.nodemanager.runtime.linux.docker.allowed-container-networks configuration, so i think "null >/dev/null && rm -rf *" can not pass to the commmand. 2, Another reason, i didn't find a better way to get network type except command with pipe.:( was (Author: charo zhang): [~eyang] Thanks for you suggestions. About 2/3/4 point, i will change and test later. but about the first suggestion: 1, network name is only allowed by yarn.nodemanager.runtime.linux.docker.allowed-container-networks configuration, so i think "null >/dev/null && rm -rf *" can not pass to the commmand. 2, Another reason, i didn't find a better way to get network type except command with pipe.:( > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8986.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686938#comment-16686938 ] Eric Yang edited comment on YARN-8986 at 11/14/18 6:23 PM: --- [~Charo Zhang] Thank you for the patch. This patch assumes YARN user has ability to run "docker" command line. This is not true in secure clusters. [Docker access|https://docs.docker.com/install/linux/linux-postinstall/] should be given to trusted system admin with sudo access only. YARN user can only acquire privileges to run docker command via C version of container-executor binary. This ensures that we are not giving too much power to YARN user. We should route "docker network ls" check through C version of container-executor to perform docker operations. The decision making process of adding "-P" probably belongs to get_docker_run_command. YARN_CONTAINER_RUNTIME_DOCKER_PORTS_MAPPING looks ok. Do you plan to support specific binding of host IP? i.e. 127.0.0.1:8080:80 to restrict the container port 80 to map to host 127.0.0.1:8080. was (Author: eyang): [~Charo Zhang] Thank you for the patch. This patch assumes YARN user has ability to run "docker" command line. This is not true in secure clusters. [Docker access|https://docs.docker.com/install/linux/linux-postinstall/] should be given to trusted system admin with sudo access only. YARN user can only acquire privileges to run docker command via C version of container-executor binary. This ensures that we are not giving too much power to YARN user. We should route "docker network ls" check through C version of container-executor to perform docker operations. The decision making process of adding "-P" probably belongs to get_docker_run_command. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Fix For: 3.1.2 > > Attachments: 20181108155450.png, YARN-8986.001.patch, > YARN-8986.002.patch, YARN-8986.003.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
