[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409856#comment-16409856
]
Jason Lowe commented on YARN-7654:
----------------------------------
bq. Can you share how spark does tokenizer for environment variables that we
talked about in the meeting? The current approach is a bit adhoc in this jira,
and I like to address it, if possible.
That is a separate, mostly unrelated thing having to do with
Configuration.getStrings behavior when the values within the commas have commas
themselves. See the discussion in YARN-6830, and
http://spark.apache.org/docs/latest/running-on-yarn.html#configuration
discusses how Spark handles environment variables being passed by the user (see
spark.yarn.appMasterEnv on that page).
The clean way to handle this would be to either avoid using
ShellCommandExecutor in PrivilegedOperationExecutor and use ProcessBuilder
directly or fix Shell#runCommand to not smash all of its arguments together
into one big string. It's a shame that we actually do go through all the
motions to build up an array of separate arguments only to smash them all
together with space separators before running the child process.
ProcessBuilder supports an array of arguments which is what we need to avoid
marshalling multiple values into strings and the pitfalls that arise from
trying to decode it. That's clearly a separate JIRA which could be a
prerequisite if we want to leverage it here.
bq. Let me know if there is a subset that would be interesting to have. I will
modify code accordingly.
Yeah, we can start out with no variables being passed through from the
nodemanager and see if that's going to be problematic for the entry point use
cases. In essence we're bundling "my container has an entry point" with "I
don't need to inherit any environment variables." We may need a way to
distinguish those two later, but we can try combining them for now. I would
add a comment to the relevant code explaining why we're not putting in those
variables to help document the decision to combine these two concepts.
> Support ENTRY_POINT for docker container
> ----------------------------------------
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Affects Versions: 3.1.0
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
