[
https://issues.apache.org/jira/browse/SPARK-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14977151#comment-14977151
]
Yu Gao commented on SPARK-10181:
--------------------------------
No problem. Please try out the changes in the second pull request which does
explicit kerberos login during hive client initialization to prepare the
credentials, instead of making UserGroupInformation class shared which would
introduce other class collision issues.
> HiveContext is not used with keytab principal but with user principal/unix
> username
> -----------------------------------------------------------------------------------
>
> Key: SPARK-10181
> URL: https://issues.apache.org/jira/browse/SPARK-10181
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Environment: kerberos
> Reporter: Bolke de Bruin
> Labels: hive, hivecontext, kerberos
>
> `bin/spark-submit --num-executors 1 --executor-cores 5 --executor-memory 5G
> --driver-java-options -XX:MaxPermSize=4G --driver-class-path
> lib/datanucleus-api-jdo-3.2.6.jar:lib/datanucleus-core-3.2.10.jar:lib/datanucleus-rdbms-3.2.9.jar:conf/hive-site.xml
> --files conf/hive-site.xml --master yarn --principal sparkjob --keytab
> /etc/security/keytabs/sparkjob.keytab --conf
> spark.yarn.executor.memoryOverhead=18000 --conf
> "spark.executor.extraJavaOptions=-XX:MaxPermSize=4G" --conf
> spark.eventLog.enabled=false ~/test.py`
> With:
> #!/usr/bin/python
> from pyspark import SparkContext
> from pyspark.sql import HiveContext
> sc = SparkContext()
> sqlContext = HiveContext(sc)
> query = """ SELECT * FROM fm.sk_cluster """
> rdd = sqlContext.sql(query)
> rdd.registerTempTable("test")
> sqlContext.sql("CREATE TABLE wcs.test LOCATION '/tmp/test_gl' AS SELECT *
> FROM test")
> Ends up with:
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denie
> d: user=ua80tl, access=READ_EXECUTE,
> inode="/tmp/test_gl/.hive-staging_hive_2015-08-24_10-43-09_157_78057390024057878
> 34-1/-ext-10000":sparkjob:hdfs:drwxr-x---
> (Our umask denies read access to other by default)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
