Updated Branches:
refs/heads/master 39af914b2 -> 9290e5bcd
spark-assembly.jar fails to authenticate with YARN ResourceManager
sbt-assembly is setup to pick the first
META-INF/services/org.apache.hadoop.security.SecurityInfo file instead of
merging them. This causes Kerberos authentication to fail, this manifests
itself in the "info:null" debug log statement:
DEBUG SaslRpcClient: Get token info proto:interface
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:null
DEBUG SaslRpcClient: Get kerberos info proto:interface
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:null
ERROR UserGroupInformation: PriviledgedActionException as:foo@BAR
(auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
DEBUG UserGroupInformation: PrivilegedAction as:foo@BAR (auth:KERBEROS)
from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583)
WARN Client: Exception encountered while connecting to the server :
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
ERROR UserGroupInformation: PriviledgedActionException as:foo@BAR
(auth:KERBEROS) cause:java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
This previously would just contain a single class:
$ unzip -c
assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar
META-INF/services/org.apache.hadoop.security.SecurityInfo
Archive:
assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar
inflating: META-INF/services/org.apache.hadoop.security.SecurityInfo
org.apache.hadoop.security.AnnotatedSecurityInfo
And now has the full list of classes:
$ unzip -c
assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar
META-INF/services/org.apache.hadoop.security.SecurityInfoArchive:
assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop2.2.0.jar
inflating: META-INF/services/org.apache.hadoop.security.SecurityInfo
org.apache.hadoop.security.AnnotatedSecurityInfo
org.apache.hadoop.mapreduce.v2.app.MRClientSecurityInfo
org.apache.hadoop.mapreduce.v2.security.client.ClientHSSecurityInfo
org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo
org.apache.hadoop.yarn.security.ContainerManagerSecurityInfo
org.apache.hadoop.yarn.security.SchedulerSecurityInfo
org.apache.hadoop.yarn.security.admin.AdminSecurityInfo
org.apache.hadoop.yarn.server.RMNMSecurityInfoClass
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/23146a67
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/23146a67
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/23146a67
Branch: refs/heads/master
Commit: 23146a67052889797d6761388cbc19ae6bfe6e21
Parents: dfd1ebc
Author: Nathan Howell <[email protected]>
Authored: Tue Nov 12 13:17:48 2013 -0800
Committer: Nathan Howell <[email protected]>
Committed: Tue Nov 12 13:27:50 2013 -0800
----------------------------------------------------------------------
project/SparkBuild.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/23146a67/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 115570d..2db167f 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -327,7 +327,7 @@ object SparkBuild extends Build {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") =>
MergeStrategy.discard
case "log4j.properties" => MergeStrategy.discard
- case "META-INF/services/org.apache.hadoop.fs.FileSystem" =>
MergeStrategy.concat
+ case m if m.toLowerCase.startsWith("meta-inf/services/") =>
MergeStrategy.filterDistinctLines
case "reference.conf" => MergeStrategy.concat
case _ => MergeStrategy.first
}
