[jira] [Commented] (YARN-9591) JAR archive expansion doesn't faithfully restore the jar file content
[
https://issues.apache.org/jira/browse/YARN-9591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913447#comment-16913447
]
Terence Yim commented on YARN-9591:
---
Ping again. Is anyone available reviewing the fix?
> JAR archive expansion doesn't faithfully restore the jar file content
> -
>
> Key: YARN-9591
> URL: https://issues.apache.org/jira/browse/YARN-9591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Terence Yim
>Priority: Major
>
> Due to change in YARN-2185, jar archives are unjar using {{JarInputStream}},
> which will skip writing out the {{META-INF/MANIFEST.MF}} file.
> Before the YARN-2185 change, it was unarchive using {{JarFile}} instead, and
> the {{META-INF/MANIFEST.MF}} file was written out correctly.
> This change of behavior is causing some applications to fail.
> Suggested to change to use {{ZipInputStream}} instead to faithfully restoring
> all entries inside a jar file.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9591) JAR archive expansion doesn't faithfully restore the jar file content
[
https://issues.apache.org/jira/browse/YARN-9591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898553#comment-16898553
]
Terence Yim commented on YARN-9591:
---
Hi,
The fixes is provided in GitHub PR for 2 months. Is there any update?
Thanks,
Terence
> JAR archive expansion doesn't faithfully restore the jar file content
> -
>
> Key: YARN-9591
> URL: https://issues.apache.org/jira/browse/YARN-9591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Terence Yim
>Priority: Major
>
> Due to change in YARN-2185, jar archives are unjar using {{JarInputStream}},
> which will skip writing out the {{META-INF/MANIFEST.MF}} file.
> Before the YARN-2185 change, it was unarchive using {{JarFile}} instead, and
> the {{META-INF/MANIFEST.MF}} file was written out correctly.
> This change of behavior is causing some applications to fail.
> Suggested to change to use {{ZipInputStream}} instead to faithfully restoring
> all entries inside a jar file.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9591) JAR archive expansion doesn't faithfully restore the jar file content
Terence Yim created YARN-9591:
-
Summary: JAR archive expansion doesn't faithfully restore the jar
file content
Key: YARN-9591
URL: https://issues.apache.org/jira/browse/YARN-9591
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.1.0
Reporter: Terence Yim
Due to change in YARN-2185, jar archives are unjar using {{JarInputStream}},
which will skip writing out the {{META-INF/MANIFEST.MF}} file.
Before the YARN-2185 change, it was unarchive using {{JarFile}} instead, and
the {{META-INF/MANIFEST.MF}} file was written out correctly.
This change of behavior is causing some applications to fail.
Suggested to change to use {{ZipInputStream}} instead to faithfully restoring
all entries inside a jar file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4727) Unable to override the $HADOOP_CONF_DIR env variable for container
[
https://issues.apache.org/jira/browse/YARN-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terence Yim updated YARN-4727:
--
Description:
Given the default config of "yarn.nodemanager.env-whitelist", application
should be able to set the env variable $HADOOP_CONF_DIR to value other than the
one in the NodeManager system environment. However, I believe due to a bug in
the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
class, it is not possible so.
>From the {{sanitizeEnv()}} method in the ContainerLaunch class
>(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
{noformat}
putEnvIfNotNull(environment,
Environment.HADOOP_CONF_DIR.name(),
System.getenv(Environment.HADOOP_CONF_DIR.name())
);
if (!Shell.WINDOWS) {
environment.put("JVM_PID", "$$");
}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
{noformat}
So there two issues here.
1. the environment is already set with the system environment of the NM in the
{{putEnvIfNotNull}} call, hence the {{putEnvIfAbsent}} call will never set it
to some new value
2. Inside the {{putEnvIfAbsent}} call, it uses the system environment of the
NM, which it should be using the one from the {{launchContext}} instead.
was:
Given the default config of "yarn.nodemanager.env-whitelist", application
should be able to set the env variable $HADOOP_CONF_DIR to value other than the
one in the NodeManager system environment. However, I believe due to a bug in
the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
class, it is not possible so.
>From the {{sanitizeEnv()}} method in the ContainerLaunch class
>(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L991)
{noformat}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
{noformat}
The {{putEnvIfAbsent}} call simply use the {{whitelistEnvVariable}} to call
{{System.getenv(whitelistEnvVariable)}} and set it to the environment map. It
should be using the one from the {{launchContext}} instead.
> Unable to override the $HADOOP_CONF_DIR env variable for container
> --
>
> Key: YARN-4727
> URL: https://issues.apache.org/jira/browse/YARN-4727
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.4.1, 2.5.2, 2.7.2, 2.6.4
>Reporter: Terence Yim
>
> Given the default config of "yarn.nodemanager.env-whitelist", application
> should be able to set the env variable $HADOOP_CONF_DIR to value other than
> the one in the NodeManager system environment. However, I believe due to a
> bug in the
> {{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
> class, it is not possible so.
> From the {{sanitizeEnv()}} method in the ContainerLaunch class
> (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
> {noformat}
> putEnvIfNotNull(environment,
> Environment.HADOOP_CONF_DIR.name(),
> System.getenv(Environment.HADOOP_CONF_DIR.name())
> );
> if (!Shell.WINDOWS) {
> environment.put("JVM_PID", "$$");
> }
> String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
> YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
>
> for(String whitelistEnvVariable : whitelist) {
> putEnvIfAbsent(environment, whitelistEnvVariable.trim());
> }
> ...
> private static void putEnvIfAbsent(
> Map environment, String variable) {
> if (environment.get(variable) == null) {
>
[jira] [Created] (YARN-4727) Unable to override the $HADOOP_CONF_DIR env variable for container
Terence Yim created YARN-4727:
-
Summary: Unable to override the $HADOOP_CONF_DIR env variable for
container
Key: YARN-4727
URL: https://issues.apache.org/jira/browse/YARN-4727
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.6.4, 2.7.2, 2.5.2, 2.4.1
Reporter: Terence Yim
Given the default config of "yarn.nodemanager.env-whitelist", application
should be able to set the env variable $HADOOP_CONF_DIR to value other than the
one in the NodeManager system environment. However, I believe due to a bug in
the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
class, it is not possible so.
>From the {{sanitizeEnv()}} method in the ContainerLaunch class
>(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L991)
{noformat}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
{noformat}
The {{putEnvIfAbsent}} call simply use the {{whitelistEnvVariable}} to call
{{System.getenv(whitelistEnvVariable)}} and set it to the environment map. It
should be using the one from the {{launchContext}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
