Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r43242989
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala
---
@@ -245,4 +247,55 @@ class YarnSparkHadoopUtilSuite extends SparkFunSuite
with Matchers with Logging
System.clearProperty("SPARK_YARN_MODE")
}
}
+
+ test("Obtain tokens For HiveMetastore") {
+ val hadoopConf = new Configuration()
+ hadoopConf.set("hive.metastore.kerberos.principal", "bob")
+ // thrift picks up on port 0 and bails out, without trying to talk to
endpoint
+ hadoopConf.set("hive.metastore.uris", "http://localhost:0";)
+ val util = new YarnSparkHadoopUtil
+ val e = intercept[InvocationTargetException] {
+ util.obtainTokenForHiveMetastoreInner(hadoopConf, "alice")
+ }
+ assertNestedHiveException(e)
+ // expect exception trapping code to unwind this hive-side exception
+ assertNestedHiveException(intercept[InvocationTargetException] {
+ util.obtainTokenForHiveMetastore(hadoopConf)
+ })
+ }
+
+ def assertNestedHiveException(e: InvocationTargetException): Throwable =
{
+ val inner = e.getCause
+ if (inner == null) {
+ fail("No inner cause", e)
+ }
+ if (!inner.isInstanceOf[HiveException]) {
+ fail(s"Not a hive exception", inner)
+ }
+ inner
+ }
+
+ test("handleTokenIntrospectionFailure") {
+ val util = new YarnSparkHadoopUtil
+ // downgraded exceptions
+ util.handleTokenIntrospectionFailure("hive", new
ClassNotFoundException("cnfe"))
--- End diff --
As soon as this patch is in I'll turn to
[SPARK-11317](https://issues.apache.org/jira/browse/SPARK-11317), which is
essentially "apply the same catching, filtering and reporting strategy for
HBase tokens as for Hive ones". It's not as critical as this one (token
retrieval is working), but as nothing gets logged except
"InvocationTargetException" with no stack trace, trying to recognise the issue
is a Kerberos auth problem, let alone trying to fix it, is a weekend's effort,
rather than 20 minutes worth.
Because the policy goes in both places, having it separate and re-usable
makes it a zero-cut-and-paste reuse, with that single test for failures without
having to mock up failures across two separate clauses. And future maintenance
costs are kept down if someone ever decides to change the policy again.
Would you be happier if I cleaned up the HBase code as part of this same
patch? Because I can and it will make the benefits of the factored out
behaviour clearer. It's just messy to fix two things in one patch, especially
if someone ever needs to play cherry pick or reverting games.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
