[
https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605906#comment-16605906
]
Yuming Wang commented on SPARK-25330:
-------------------------------------
[~brahmareddy] Sorry. I didn't have script because we need a kerberos env to
reproduce it. I try to change {{getRestrictParserDefault}} to
{code:java}
private static boolean getRestrictParserDefault(Object resource) {
if (resource instanceof String || !UserGroupInformation.isInitialized()) {
System.err.println("resource instanceof String.");
return false;
} else {
UserGroupInformation user;
try {
System.err.println("resource: " + resource);
System.err.println("UserGroupInformation.isInitialized(): " +
UserGroupInformation.isInitialized());
user = UserGroupInformation.getCurrentUser();
System.err.println("UserGroupInformation.getCurrentUser(): " +
UserGroupInformation.getCurrentUser().toString());
System.err.println("user.getRealUser(): " +
UserGroupInformation.getCurrentUser().getRealUser());
System.err.println("user.getRealUser().isFromKeytab(): " +
UserGroupInformation.getCurrentUser().isFromKeytab());
System.err.println("user.getRealUser().hasKerberosCredentials(): " +
UserGroupInformation.getCurrentUser().hasKerberosCredentials());
System.err.println("user.getLoginUser().isFromKeytab(): " +
UserGroupInformation.getLoginUser().isFromKeytab());
System.err.println("user.getLoginUser().getRealUser(): " +
UserGroupInformation.getLoginUser().getRealUser());
System.err.println("user.getLoginUser().hasKerberosCredentials(): " +
UserGroupInformation.getLoginUser().hasKerberosCredentials());
System.err.println("user.getLoginUser().isFromKeytab(): " +
UserGroupInformation.getLoginUser().isFromKeytab());
} catch (IOException e) {
throw new RuntimeException("Unable to determine current user", e);
}
return user.getRealUser() != null;
}
}
{code}
The output is:
{noformat}
...
resource: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@62e6b5c8
UserGroupInformation.isInitialized(): true
UserGroupInformation.getCurrentUser(): user_b (auth:PROXY) via
[email protected] (auth:KERBEROS)
user.getRealUser(): [email protected] (auth:KERBEROS)
user.getRealUser().isFromKeytab(): false
user.getRealUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
user.getLoginUser().getRealUser(): [email protected] (auth:KERBEROS)
user.getLoginUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
resource: file:/apache/hive/conf/hive-site.xml
UserGroupInformation.isInitialized(): true
UserGroupInformation.getCurrentUser(): user_b (auth:PROXY) via
[email protected] (auth:KERBEROS)
user.getRealUser(): [email protected] (auth:KERBEROS)
user.getRealUser().isFromKeytab(): false
user.getRealUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
user.getLoginUser().getRealUser(): [email protected] (auth:KERBEROS)
user.getLoginUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
18/09/06 07:49:29 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no
longer has any effect. Use hive.hmshandler.retry.* instead
18/09/06 07:49:29 WARN HiveConf: HiveConf of name hive.metastore.local does not
exist
18/09/06 07:49:29 WARN HiveConf: HiveConf of name
hive.metastore.ds.retry.attempts does not exist
18/09/06 07:49:29 WARN HiveConf: HiveConf of name
hive.metastore.ds.retry.interval does not exist
18/09/06 07:49:29 WARN HiveConf: HiveConf of name
hive.server2.enable.impersonation does not exist
18/09/06 07:49:29 INFO HiveMetaStore: 0: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
18/09/06 07:49:30 INFO ObjectStore: ObjectStore, initialize called
18/09/06 07:49:30 INFO Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/09/06 07:49:30 INFO Persistence: Property datanucleus.cache.level2 unknown -
will be ignored
resource: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@ae7950d
UserGroupInformation.isInitialized(): true
UserGroupInformation.getCurrentUser(): user_b (auth:PROXY) via
[email protected] (auth:KERBEROS)
user.getRealUser(): [email protected] (auth:KERBEROS)
user.getRealUser().isFromKeytab(): false
user.getRealUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
user.getLoginUser().getRealUser(): [email protected] (auth:KERBEROS)
user.getLoginUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
resource: file:/apache/hive/conf/hive-site.xml
UserGroupInformation.isInitialized(): true
UserGroupInformation.getCurrentUser(): user_b (auth:PROXY) via
[email protected] (auth:KERBEROS)
user.getRealUser(): [email protected] (auth:KERBEROS)
user.getRealUser().isFromKeytab(): false
user.getRealUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
user.getLoginUser().getRealUser(): [email protected] (auth:KERBEROS)
user.getLoginUser().hasKerberosCredentials(): false
user.getLoginUser().isFromKeytab(): false
......
18/09/06 07:23:08 INFO deprecation: user.name is deprecated. Instead, use
mapreduce.job.user.name
18/09/06 07:23:09 INFO SessionState: Created HDFS directory:
/tmp/hive-${user.name}/user_b
18/09/06 07:23:09 INFO SessionState: Created local directory:
/tmp/136f93db-8924-4366-a795-49960acde4d0_resources
Exception in thread "main" java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=user_b, access=EXECUTE,
inode="/tmp/hive-$%7Buser.name%7D/user_b/136f93db-8924-4366-a795-49960acde4d0":user_a:hadoop:drwx------
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
{noformat}
The correct output should be:
{noformat}
......
18/09/06 08:12:04 INFO SessionState: Created local directory:
/tmp/799640f8-3d34-4cb7-90fe-5368c22881d5_resources
18/09/06 08:12:04 INFO SessionState: Created HDFS directory:
/tmp/hive-admin/user_b/799640f8-3d34-4cb7-90fe-5368c22881d5
18/09/06 08:12:04 INFO SessionState: Created local directory:
/tmp/admin/799640f8-3d34-4cb7-90fe-5368c22881d5
18/09/06 08:12:04 INFO SessionState: Created HDFS directory:
/tmp/hive-admin/user_b/799640f8-3d34-4cb7-90fe-5368c22881d5/_tmp_space.db
......
{noformat}
> Permission issue after upgrade hadoop version to 2.7.7
> ------------------------------------------------------
>
> Key: SPARK-25330
> URL: https://issues.apache.org/jira/browse/SPARK-25330
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 2.3.2, 2.4.0
> Reporter: Yuming Wang
> Priority: Major
>
> How to reproduce:
> {code:java}
> # build spark
> ./dev/make-distribution.sh --name SPARK-25330 --tgz -Phadoop-2.7 -Phive
> -Phive-thriftserver -Pyarn
> tar -zxf spark-2.4.0-SNAPSHOT-bin-SPARK-25330.tgz && cd
> spark-2.4.0-SNAPSHOT-bin-SPARK-25330
> export HADOOP_PROXY_USER=user_a
> bin/spark-sql
> export HADOOP_PROXY_USER=user_b
> bin/spark-sql{code}
>
> {noformat}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=user_b, access=EXECUTE,
> inode="/tmp/hive-$%7Buser.name%7D/user_b/668748f2-f6c5-4325-a797-fd0a7ee7f4d4":user_b:hadoop:drwx------
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
