[
https://issues.apache.org/jira/browse/SPARK-23527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431514#comment-16431514
]
Ron Gonzalez commented on SPARK-23527:
--------------------------------------
My admin team has resolved it, and now I get this same problem when I wrap
org.apache.spark.deploy.SparkSubmit and invoke it directly using Java.
Is there some kind of jar files or configuration that I need to add into
classpath?
> Error with spark-submit and kerberos with TLS-enabled Hadoop cluster
> --------------------------------------------------------------------
>
> Key: SPARK-23527
> URL: https://issues.apache.org/jira/browse/SPARK-23527
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.2.1
> Environment: core-site.xml
> <property>
> <name>hadoop.security.key.provider.path</name>
> <value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
> hdfs-site.xml
> <property>
> <name>dfs.encryption.key.provider.uri</name>
> <value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
> Reporter: Ron Gonzalez
> Priority: Critical
>
> For current configuration of our enterprise cluster, I submit using
> spark-submit:
> ./spark-submit --master yarn --deploy-mode cluster --class
> org.apache.spark.examples.SparkPi --conf
> spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar
> ../examples/jars/spark-examples_2.11-2.2.1.jar 10
> I am getting the following problem:
>
> 18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 3351181 for svchdc236d on ha-hdfs:nameservice1
> Exception in thread "main" java.lang.IllegalArgumentException:
> java.net.UnknownHostException: host1.domain.com;host2.domain.com
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781)
> at
> org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
> at
> org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52)
>
> If I get rid of the other host for the properties so instead of
> kms://ht...@host1.domain.com;host2.domain.com:16000/kms, I convert it to:
> kms://ht...@host1.domain.com:16000/kms
> it fails with a different error:
> java.io.IOException: javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> If I do the same thing using spark 1.6, it works so it seems like a
> regression...
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
