[
https://issues.apache.org/jira/browse/HADOOP-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607592#comment-16607592
]
Jason Lowe commented on HADOOP-15722:
-------------------------------------
I haven't been able to reproduce the issue yet, but looking closer at the logs
I think it's related to variable expansion. Another aspect of restricted
parsing is they are unable to access system properties or environment variables
from the config since those could potentially contain secrets. Looks like in
the following log snippets for the good and bad runs, the user.name system
property is not getting expanded in the bad run because the conf resource is
untrusted:
Log excerpt from the session with hadoop 2.7.3:
{noformat}
18/09/06 08:12:04 INFO SessionState: Created HDFS directory:
/tmp/hive-admin/user_b/799640f8-3d34-4cb7-90fe-5368c22881d5
{noformat}
Log excerpt from the session with hadoop 2.7.7:
{noformat}
18/09/06 07:23:09 INFO SessionState: Created HDFS directory:
/tmp/hive-${user.name}/user_b
{noformat}
[~yumwang] would you mind running with the following patch to Hadoop 2.7.7's
Configuration to see if this fixes the issue or at least gets significantly
farther? That would help validate my theory as to what's going on here. The
patch keeps XML directives restricted for untrusted sources but re-enables
system property access.
{noformat}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 5ce3e65..4df8491 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -905,7 +905,7 @@ public synchronized void reloadConfiguration() {
private synchronized void addResourceObject(Resource resource) {
resources.add(resource); // add to resources
- restrictSystemProps |= resource.isParserRestricted();
+ restrictSystemProps = false;
reloadConfiguration();
}
{noformat}
If it indeed is the issue then we may need to reconsider the restriction on
system properties. Choices include:
- Removing the property expansion restriction completely so all system and env
properties are available, and it would be up to admins to sanitize these when
starting proxy servers
- Allowing system properties but restricting environment variables, if we feel
env variables are more common for passing secrets
- Using a whitelist for system properties
> regression: Hadoop 2.7.7 release breaks spark submit
> ----------------------------------------------------
>
> Key: HADOOP-15722
> URL: https://issues.apache.org/jira/browse/HADOOP-15722
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, conf, security
> Affects Versions: 2.7.7
> Reporter: Steve Loughran
> Priority: Major
>
> SPARK-25330 highlights that upgrading spark to hadoop 2.7.7 is causing a
> regression in client setup, with things only working when
> {{Configuration.getRestrictParserDefault(Object resource)}} = false.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
