A similar issue occurs when interacting with Hive secured by Sentry. https://issues.apache.org/jira/browse/SPARK-9042
By changing how Hive Context instance is created, this issue might also be resolved. On Thu, Oct 22, 2015 at 11:33 AM Steve Loughran <ste...@hortonworks.com> wrote: > On 22 Oct 2015, at 08:25, Chester Chen <ches...@alpinenow.com> wrote: > > Doug > > We are not trying to compiling against different version of hive. The > 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving > from spark 1.3.1 to 1.5.1. Simply trying to supply the needed > dependency. The rest of application (besides spark) simply uses hive 0.13.1. > > Yes we are using yarn client directly, there are many functions we need > and modified are not provided in yarn client. The spark launcher in the > current form does not satisfy our requirements (at least last time I see > it) there is a discussion thread about several month ago. > > From spark 1.x to 1.3.1, we fork the yarn client to achieve these > goals ( yarn listener call backs, killApplications, yarn capacities call > back etc). In current integration for 1.5.1, to avoid forking the spark, we > simply subclass the yarn client overwrites a few methods. But we lost > resource capacity call back and estimation by doing this. > > This is bit off the original topic. > > I still think there is a bug related to the spark yarn client in case > of Kerberos + spark hive-exec dependency. > > Chester > > > I think I understand what's being implied here. > > > 1. In a secure cluster, a spark app needs a hive delegation token to > talk to hive > 2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses > reflection to get the delegation token > 3. The reflection doesn't work, a CFNE exception is logged > 4. The app should still launch, but it'll be without a hive token , so > attempting to work with Hive will fail. > > I haven't seen this, because while I do test runs against a kerberos > cluster, I wasn't talking to hive from the deployed app. > > > It sounds like this workaround works because the hive RPC protocol is > compatible enough with 0.13 that a 0.13 client can ask hive for the token, > though then your remote CP is stuck on 0.13 > > Looking at the hive class, the metastore has now made the hive constructor > private and gone to a factory method (public static Hive get(HiveConf c) > throws HiveException) to get an instance. The reflection code would need to > be updated. > > I'll file a bug with my name next to it > > > >
