[
https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974239#comment-14974239
]
Steve Loughran edited comment on SPARK-11265 at 10/26/15 2:08 PM:
------------------------------------------------------------------
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method; it has, in order, {{get(HiveConf}}, {{get(HiveConf,
boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class<?> clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
was (Author: [email protected]):
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order,
{{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class<?> clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
> YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
> ---------------------------------------------------------------------
>
> Key: SPARK-11265
> URL: https://issues.apache.org/jira/browse/SPARK-11265
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.5.1
> Environment: Kerberized Hadoop cluster
> Reporter: Steve Loughran
>
> As reported on the dev list, trying to run a YARN client which wants to talk
> to Hive in a Kerberized hadoop cluster fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
