[
https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563058#comment-15563058
]
Remi Catherinot edited comment on YARN-5714 at 10/10/16 7:32 PM:
-----------------------------------------------------------------
HOMELOC depends on both HADOOP_MAPRED_HOME and HADOOP_COMMON_HOME. This is more
or less the test done line 1560 in TestContainerLaunch when i'm testing that
$\{A:-$B\} extract A and B as its dependencies. Other tests take care of
testing $var, $\{var\} and $\{#var\} syntaxes, so I limited the test of a var
in a var to just the $B and not the $\{B\} one.
I don't parse using simplier indexOf( "$" ) because it would miss escapes by
backslashes and single quote protections.
It is also a best effort dependency parsing, which means it it fails to see the
dependencies, we end up with a "random" ordered map which is what we have
without the patch.
The parser in fact does not see that B is used inside A. It just sees A and B
which is enough for dependency checking. Parsers to evaluate $\{A:-$B\} would
require to see that B is inside A are harder to code and would certainly need a
grammar and generated code, which would be overkill for such a need.
was (Author: rcatherinot):
HOMELOC depends on both HADOOP_MAPRED_HOME and HADOOP_COMMON_HOME. This is more
or less the test done line 1560 in TestContainerLaunch when i'm testing that
${A:-$B} extract A and B as dependencies. Other tests take care of testing $var
${var} and ${#var} syntaxes, so i limited the test of a var in a var to just $B
and not ${B}.
The parser in fact does not see that B is used inside A, it just sees A and B
which is enought for dependency checking. The parser is designed to evalutate
which would require to see that B is inside A.
> ContainerExecutor does not order environment map
> ------------------------------------------------
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
> Reporter: Remi Catherinot
> Assignee: Remi Catherinot
> Priority: Trivial
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch,
> YARN-5714.003.patch, YARN-5714.004.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped
> based on the order internally used by the map implementation (hash based). It
> does not take into consideration that some env varibales may refer each
> other, and so that some env variables must be declared before those
> referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on
> HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a
> wrong value and so native libraries weren't loaded. jobs were running but not
> at their best efficiency. This is just a use case falling into that bug, but
> i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate
> to 5 days for packaging the patch in the right fashion for JIRA + try my best
> to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will
> only implement the unix version on a 1st release. I'm not used to windows env
> variables syntax so it will take me more time/research for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
