[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436207#comment-16436207
]
Jim Brennan commented on YARN-8071:
---
MAPREDUCE-7069 resolved this problem for the following properties:
{quote}mapreduce.map.env.VARNAME=value
mapreduce.reduce.env.VARNAME=value
yarn.app.mapreduce.am.env.VARNAME=value
yarn.app.mapreduce.am.admin.user.env.VARNAME=value
{quote}
The remaining YARN environment variable property is:
{{yarn.nodemanager.admin-env}}
I am planning to use this Jira to add support for the
{{yarn.nodemanager.admin-env.VARNAME=value}} syntax to allow variables with
commas to be specified for this property.
[~jlowe], [~shaneku...@gmail.com], please let me know if you agree this is
needed, and also if I'm missing any other properties.
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417505#comment-16417505
]
Jim Brennan commented on YARN-8071:
---
@jlowe, I've filed [MAPREDUCE-7069] for addressing the mapreduce properties. I
will use this one to address the yarn properties.
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415791#comment-16415791
]
Jason Lowe commented on YARN-8071:
--
It sounds like separate JIRAs, at least for MapReduce and YARN. One JIRA would
implement the change for MapReduce and all of its env properties (e.g.:
mapreduce.map.env, mapreduce.admin.user.env, etc.) while a separate JIRA would
implement the changes for YARN properties (e.g.: yarn.nodemanager.admin-env).
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415782#comment-16415782
]
Jim Brennan commented on YARN-8071:
---
[~jlowe], yes I think this will affect mapreduce, yarn, and common code. I
haven't done the analysis yet to figure out everything this will affect.
Should this be refiled in hadoop common, or or should we add additional
components to this Jira?
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412177#comment-16412177
]
Jason Lowe commented on YARN-8071:
--
Is this a YARN JIRA? I would think the changes would be in MapReduce to fix
the handling of mapreduce.map.env (maybe also Common if we have a utility
method on Configuration to collect properties with a specified prefix).
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Provide Spark-like API for setting Environment Variables to enable vars with commas
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412063#comment-16412063
]
Jim Brennan commented on YARN-8071:
---
Alternate solutions to this problem (with patches) are provided in the
following Jiras:
[YARN-8029] - addresses the specific issue for docker yarn mount paths by
changing the delimiter for those environment variables from comma to semicolon.
[YARN-6830] - Modifies the regex used by Apps.setEnvFromInputString() to allow
commas in a quoted string. This requires that users of those variables remove
the surrounding strings before using them.
See discussion in [YARN-6830] about the alternatives.
> Provide Spark-like API for setting Environment Variables to enable vars with
> commas
> ---
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
