[ https://issues.apache.org/jira/browse/SPARK-23527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-23527. ------------------------------------ Resolution: Not A Problem Doesn't seem like a bug. Either a Spark or KMS config issue. Please try the mailing lists first. > Error with spark-submit and kerberos with TLS-enabled Hadoop cluster > -------------------------------------------------------------------- > > Key: SPARK-23527 > URL: https://issues.apache.org/jira/browse/SPARK-23527 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 2.2.1 > Environment: core-site.xml > <property> > <name>hadoop.security.key.provider.path</name> > <value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value> > </property> > hdfs-site.xml > <property> > <name>dfs.encryption.key.provider.uri</name> > <value>kms://ht...@host1.domain.com;host2.domain.com:16000/kms</value> > </property> > Reporter: Ron Gonzalez > Priority: Critical > > For current configuration of our enterprise cluster, I submit using > spark-submit: > ./spark-submit --master yarn --deploy-mode cluster --class > org.apache.spark.examples.SparkPi --conf > spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar > ../examples/jars/spark-examples_2.11-2.2.1.jar 10 > I am getting the following problem: > > 18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token > 3351181 for svchdc236d on ha-hdfs:nameservice1 > Exception in thread "main" java.lang.IllegalArgumentException: > java.net.UnknownHostException: host1.domain.com;host2.domain.com > at > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781) > at > org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) > at > org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046) > at > org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52) > > If I get rid of the other host for the properties so instead of > kms://ht...@host1.domain.com;host2.domain.com:16000/kms, I convert it to: > kms://ht...@host1.domain.com:16000/kms > it fails with a different error: > java.io.IOException: javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: unable to find > valid certification path to requested target > If I do the same thing using spark 1.6, it works so it seems like a > regression... > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org