eigakow opened a new issue #2100:
URL: https://github.com/apache/hudi/issues/2100
**Describe the problem you faced**
While uplifting deltastreamer from 0.5.3 to 0.6.0, I can no longer use
--principal and --keytab start parameters as it results in error
_java.util.ServiceConfigurationError:
org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier not found_
**To Reproduce**
Steps to reproduce the behavior:
1. Start the job with the following
`spark-submit --master yarn --deploy-mode cluster --conf
spark.app.name=hudi_impl2 --conf spark.yarn.maxAppAttempts=4
--conf spark.yarn.am.attemptFailuresValidityInterval=1h
--conf spark.yarn.executor.failuresValidityInterval=1h
--conf spark.task.maxFailures=8
--principal iga@XXX
--keytab /home/director/XXX.keytab
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--jars
hdfs:///app/ingestion/hudi/v0.3/hive-service-2.1.1-cdh6.1.0.jar,hdfs:///app/ingestion/hudi/v0.3/hive-jdbc-2.1.1-cdh6.1.0.jar
hdfs:///app/ingestion/hudi/v0.6.0/hudi-utilities-bundle_2.11-0.6.0.jar
--props hdfs:///app/ingestion/impl2/hudi-default.properties
--target-base-path adl://XXX.azuredatalakestore.net/impl2
--table-type COPY_ON_WRITE
--source-class org.apache.hudi.utilities.sources.AvroKafkaSource
--target-table impl2
--schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider
--enable-hive-sync
--source-ordering-field commit_messageid`
The job will fail, with ServiceConfigurationError errror. The same command,
just with 0.5.3 (or earlier) hudi-utilities jar, will work fine.
**Expected behavior**
A successful run
**Environment Description**
Hudi version : 0.6.0
Spark version : 2.4.0-cdh6.1.0
Hive version : 2.1.1-cdh6.1.0
Hadoop version : 3.0.0-cdh6.1.0
Storage (HDFS/S3/GCS..) : ADLS
Running on Docker? (yes/no) : no
**Additional context**
The 0.6.0 release was compiled with a change to
hudi-utilities-bundle/pom.xml adding the following _<relocation>
<pattern>org.eclipse.jetty.</pattern>
<shadedPattern>org.apache.hudi.org.apache.jetty.</shadedPattern>
</relocation>_
**Stacktrace**
```
20/09/22 07:34:49 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:50 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:51 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:52 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:53 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:54 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: ACCEPTED)
20/09/22 07:34:55 INFO yarn.Client: Application report for
application_1600754659032_0016 (state: FAILED)
20/09/22 07:34:55 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1600754659032_0016 failed 2
times in previous 3600000 milliseconds due to AM Container for
appattempt_1600754659032_0016_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-09-22 07:34:54.678]Exception from
container-launch.
Container id: container_1600754659032_0016_02_000001
Exit code: 1
Shell output: main : command provided 1
main : run as user is iga
main : requested yarn user is iga
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file
/data3/yarn/nm/nmPrivate/application_1600754659032_0016/container_1600754659032_0016_02_000001/container_1600754659032_0016_02_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...
[2020-09-22 07:34:54.681]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ovider org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier
not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:150)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at
scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:115)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:111)
at scala.Option.flatMap(Option.scala:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.getTokenRenewalInterval(HadoopFSDelegationTokenProvider.scala:111)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:53)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:132)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:130)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:130)
at
org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer$$anon$3.run(AMCredentialRenewer.scala:160)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer$$anon$3.run(AMCredentialRenewer.scala:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer.obtainTokensAndScheduleRenewal(AMCredentialRenewer.scala:157)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer.start(AMCredentialRenewer.scala:98)
at
org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:113)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:806)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
[2020-09-22 07:34:54.682]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ovider org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier
not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:150)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at
scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:115)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:111)
at scala.Option.flatMap(Option.scala:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.getTokenRenewalInterval(HadoopFSDelegationTokenProvider.scala:111)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:53)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:132)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:130)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:130)
at
org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer$$anon$3.run(AMCredentialRenewer.scala:160)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer$$anon$3.run(AMCredentialRenewer.scala:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer.obtainTokensAndScheduleRenewal(AMCredentialRenewer.scala:157)
at
org.apache.spark.deploy.yarn.security.AMCredentialRenewer.start(AMCredentialRenewer.scala:98)
at
org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:113)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:806)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
