On 22 Oct 2015, at 08:25, Chester Chen
<[email protected]<mailto:[email protected]>> wrote:
Doug
We are not trying to compiling against different version of hive. The
1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving from
spark 1.3.1 to 1.5.1. Simply trying to supply the needed dependency. The rest
of application (besides spark) simply uses hive 0.13.1.
Yes we are using yarn client directly, there are many functions we need and
modified are not provided in yarn client. The spark launcher in the current
form does not satisfy our requirements (at least last time I see it) there is a
discussion thread about several month ago.
From spark 1.x to 1.3.1, we fork the yarn client to achieve these goals (
yarn listener call backs, killApplications, yarn capacities call back etc). In
current integration for 1.5.1, to avoid forking the spark, we simply subclass
the yarn client overwrites a few methods. But we lost resource capacity call
back and estimation by doing this.
This is bit off the original topic.
I still think there is a bug related to the spark yarn client in case of
Kerberos + spark hive-exec dependency.
Chester
I think I understand what's being implied here.
1. In a secure cluster, a spark app needs a hive delegation token to talk
to hive
2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses reflection
to get the delegation token
3. The reflection doesn't work, a CFNE exception is logged
4. The app should still launch, but it'll be without a hive token , so
attempting to work with Hive will fail.
I haven't seen this, because while I do test runs against a kerberos cluster, I
wasn't talking to hive from the deployed app.
It sounds like this workaround works because the hive RPC protocol is
compatible enough with 0.13 that a 0.13 client can ask hive for the token,
though then your remote CP is stuck on 0.13
Looking at the hive class, the metastore has now made the hive constructor
private and gone to a factory method (public static Hive get(HiveConf c) throws
HiveException) to get an instance. The reflection code would need to be updated.
I'll file a bug with my name next to it
