[
https://issues.apache.org/jira/browse/SPARK-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595205#comment-14595205
]
Bolke de Bruin commented on SPARK-5111:
---------------------------------------
This patch does not work on a hadoop 2.6 + kerberos + yarn cluster. The
SparkContext cannot be setup if the patch is nor earlier in the process (in
SparkContext.init). However, just moving the patch makes the connection fail in
HiveContext. Keeping it (ie. duplicating) in two places does not work as the
class became "stale" if I remember correctly.
Latest spark build (1.5.0-SNAPSHOT) from git does not go beyond 0.13 yet it
seems even if something like
spark.sql.hive.metastore.version 0.14
spark.sql.hive.metastore.jars
/usr/hdp/2.2.0.0-2041/hadoop/conf:/usr/hdp/2.2.0.0-2041/hadoop/lib/*:/usr/hdp/2.2.0.0-2041/hadoop/.//*:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java-5.1.34-bin.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.2.0.0-2041/tez/*:/usr/hdp/2.2.0.0-2041/tez/lib/*:/etc/tez/conf:/usr/hdp/2.2.0.0-2041/hive/lib/*
is configured (an out of memory error is then thrown, could not get rid of it).
> HiveContext and Thriftserver cannot work in secure cluster beyond hadoop2.5
> ---------------------------------------------------------------------------
>
> Key: SPARK-5111
> URL: https://issues.apache.org/jira/browse/SPARK-5111
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Zhan Zhang
>
> Due to "java.lang.NoSuchFieldError: SASL_PROPS" error. Need to backport some
> hive-0.14 fix into spark, since there is no effort to upgrade hive to 0.14
> support in spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
