[
https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408656#comment-16408656
]
Jim Brennan commented on YARN-6830:
-----------------------------------
There was some discussion of this at the latest docker sync-up meeting. There
was a slight preference in that meeting for this over [YARN-8029], or the
solution that [~aw] suggested earlier:
{quote}
Rather than fight with a regex why not redefine the API instead?
When running the MR job, these environment variables are supplied as a comma
delimited string.
-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"
-Dmapreduce.map.env.MODE=bar
-Dmapreduce.map.env.IMAGE_NAME=foo
-Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
...
e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
This greatly simplifies the input validation needed and makes it clear what is
actually being defined.
{quote}
This would be a more significant change, but since it would be orthogonal to
the existing behavior, it might be less risky than changing the regex.
So there are now three options on the table:
1 - [YARN-8029] - just use a different delimiter for the docker mount vars
2 - [YARN-6830] - this one, which changes the regex to allow commas inside a
quoted string (note that docker code needs to strip the quotes)
3 - Change the API to allow specifying environment variables as separate
command line options ([~aw]'s proposal)
> Support quoted strings for environment variables
> ------------------------------------------------
>
> Key: YARN-6830
> URL: https://issues.apache.org/jira/browse/YARN-6830
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Shane Kumpf
> Assignee: Jim Brennan
> Priority: Major
> Attachments: YARN-6830.001.patch, YARN-6830.002.patch,
> YARN-6830.003.patch, YARN-6830.004.patch
>
>
> There are cases where it is necessary to allow for quoted string literals
> within environment variables values when passed via the yarn command line
> interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma
> delimited string.
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as
> {{MOUNTS=/tmp/foo}}. Any attempts to quote the embedded comma separated value
> results in quote characters becoming part of the value, and parsing still
> breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double
> or single quote). This was mentioned on YARN-4595 and will impact YARN-5534
> as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
