eigakow commented on issue #2100:
URL: https://github.com/apache/hudi/issues/2100#issuecomment-699839103
Below the stack trace of the failure. I have removed hive-sync option to
simplify troubleshooting.
```
20/09/28 07:27:54 WARN deltastreamer.SchedulerConfGenerator: Job Scheduling
Configs will not be in effect as spark.scheduler.mode is not set to FAIR at
instantiation time. Continuing without scheduling configs
20/09/28 07:27:54 INFO spark.SparkContext: Running Spark version
2.4.0-cdh6.1.0
20/09/28 07:27:54 INFO spark.SparkContext: Submitted application:
delta-streamer_impl2
20/09/28 07:27:54 INFO spark.SecurityManager: Changing view acls to:
director,iga
20/09/28 07:27:54 INFO spark.SecurityManager: Changing modify acls to:
director,iga
20/09/28 07:27:54 INFO spark.SecurityManager: Changing view acls groups to:
20/09/28 07:27:54 INFO spark.SecurityManager: Changing modify acls groups to:
20/09/28 07:27:54 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(director, iga); groups with view permissions: Set(); users with modify
permissions: Set(director, iga); groups with modify permissions: Set()
20/09/28 07:27:54 INFO util.Utils: Successfully started service
'sparkDriver' on port 38195.
20/09/28 07:27:54 INFO spark.SparkEnv: Registering MapOutputTracker
20/09/28 07:27:54 INFO spark.SparkEnv: Registering BlockManagerMaster
20/09/28 07:27:54 INFO storage.BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/09/28 07:27:54 INFO storage.BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
20/09/28 07:27:54 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-2d0abd3b-1755-43bc-9313-36a9cb19e802
20/09/28 07:27:54 INFO memory.MemoryStore: MemoryStore started with capacity
366.3 MB
20/09/28 07:27:54 INFO spark.SparkEnv: Registering OutputCommitCoordinator
20/09/28 07:27:54 INFO util.log: Logging initialized @3357ms
20/09/28 07:27:54 INFO server.Server: jetty-9.3.z-SNAPSHOT
20/09/28 07:27:54 INFO server.Server: Started @3430ms
20/09/28 07:27:54 INFO server.AbstractConnector: Started
ServerConnector@108d55c4{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
20/09/28 07:27:54 INFO util.Utils: Successfully started service 'SparkUI' on
port 4040.
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@2eaef76d{/jobs,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@c6da8bb{/jobs/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@3bae64d0{/jobs/job,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1fba386c{/jobs/job/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7e736350{/stages,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@36b310aa{/stages/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@76c387f9{/stages/stage,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@736048ed{/stages/stage/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1976f537{/stages/pool,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@45f421c{/stages/pool/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1816e24a{/storage,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6940f685{/storage/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@47b179d7{/storage/rdd,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@39008c9f{/storage/rdd/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@162b3d47{/environment,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@783115d9{/environment/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@3402b4c9{/executors,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@95bb2a2{/executors/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@46d567cb{/executors/threadDump,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@363a3d15{/executors/threadDump/json,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@173f73e7{/static,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@387bf2d9{/,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@74aa9c72{/api,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@c1fca2a{/jobs/job/kill,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5533dc72{/stages/stage/kill,null,AVAILABLE,@Spark}
20/09/28 07:27:54 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://biedva-worker-1.bigdatapoc.local:4040
20/09/28 07:27:54 INFO Configuration.deprecation:
mapred.output.compression.codec is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.codec
20/09/28 07:27:54 INFO Configuration.deprecation:
mapred.output.compression.type is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.type
20/09/28 07:27:54 INFO Configuration.deprecation: mapred.output.compress is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/09/28 07:27:54 INFO spark.SparkContext: Added JAR
hdfs:///app/ingestion/hudi/v0.6.0/hudi-utilities-bundle_2.11-0.6.0.jar at
hdfs:///app/ingestion/hudi/v0.6.0/hudi-utilities-bundle_2.11-0.6.0.jar with
timestamp 1601278074902
20/09/28 07:27:54 INFO util.Utils: Using initial executors = 0, max of
spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors
and spark.executor.instances
20/09/28 07:27:55 INFO yarn.Client: Kerberos credentials: principal =
iga@XXX, keytab = /home/director/XXX.keytab
20/09/28 07:27:55 INFO client.RMProxy: Connecting to ResourceManager at
biedva-master-1.bigdatapoc.local/172.18.157.17:8032
20/09/28 07:27:55 INFO yarn.Client: Requesting a new application from
cluster with 6 NodeManagers
20/09/28 07:27:55 INFO conf.Configuration: resource-types.xml not found
20/09/28 07:27:55 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
20/09/28 07:27:55 INFO yarn.Client: Verifying our application has not
requested more than the maximum memory capability of the cluster (28672 MB per
container)
20/09/28 07:27:55 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
20/09/28 07:27:55 INFO yarn.Client: Setting up container launch context for
our AM
20/09/28 07:27:55 INFO yarn.Client: Setting up the launch environment for
our AM container
20/09/28 07:27:55 INFO yarn.Client: Preparing resources for our AM container
20/09/28 07:27:55 INFO yarn.Client: To enable the AM to login from keytab,
credentials are being copied over to the AM via the YARN Secure Distributed
Cache.
20/09/28 07:27:55 INFO yarn.Client: Uploading resource
file:/home/director/XXX.keytab ->
hdfs://nameservice1/user/iga/.sparkStaging/application_1601272990574_0015/XXX.keytab
20/09/28 07:27:55 INFO yarn.Client: Source and destination file systems are
the same. Not copying
hdfs://nameservice1/app/ingestion/hudi/v0.3/hive-service-2.1.1-cdh6.1.0.jar
20/09/28 07:27:55 INFO yarn.Client: Source and destination file systems are
the same. Not copying
hdfs://nameservice1/app/ingestion/hudi/v0.3/hive-jdbc-2.1.1-cdh6.1.0.jar
20/09/28 07:27:55 INFO yarn.Client: Uploading resource
file:/tmp/spark-431edee5-628f-4887-900b-ee4a354c574d/__spark_conf__2055697676300668872.zip
->
hdfs://nameservice1/user/iga/.sparkStaging/application_1601272990574_0015/__spark_conf__.zip
20/09/28 07:27:55 INFO spark.SecurityManager: Changing view acls to:
director,iga
20/09/28 07:27:55 INFO spark.SecurityManager: Changing modify acls to:
director,iga
20/09/28 07:27:55 INFO spark.SecurityManager: Changing view acls groups to:
20/09/28 07:27:55 INFO spark.SecurityManager: Changing modify acls groups to:
20/09/28 07:27:55 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(director, iga); groups with view permissions: Set(); users with modify
permissions: Set(director, iga); groups with modify permissions: Set()
20/09/28 07:27:55 INFO conf.HiveConf: Found configuration file
file:/etc/hive/conf.cloudera.hive/hive-site.xml
20/09/28 07:27:55 INFO security.HadoopFSDelegationTokenProvider: getting
token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1511656334_1,
ugi=iga@XXX (auth:KERBEROS)]]
20/09/28 07:27:55 INFO hdfs.DFSClient: Created token for iga:
HDFS_DELEGATION_TOKEN owner=iga@XXX, renewer=yarn, realUser=,
issueDate=1601278075790, maxDate=1601882875790, sequenceNumber=26153,
masterKeyId=880 on ha-hdfs:nameservice1
20/09/28 07:27:55 INFO security.HadoopFSDelegationTokenProvider: getting
token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1511656334_1,
ugi=iga@XXX (auth:KERBEROS)]]
20/09/28 07:27:55 INFO hdfs.DFSClient: Created token for iga:
HDFS_DELEGATION_TOKEN owner=iga@XXX, renewer=iga, realUser=,
issueDate=1601278075804, maxDate=1601882875804, sequenceNumber=26154,
masterKeyId=880 on ha-hdfs:nameservice1
20/09/28 07:27:55 INFO yarn.Client: Deleted staging directory
hdfs://nameservice1/user/iga/.sparkStaging/application_1601272990574_0015
20/09/28 07:27:55 ERROR spark.SparkContext: Error initializing SparkContext.
java.util.ServiceConfigurationError:
org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:150)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at
scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:115)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:111)
at scala.Option.flatMap(Option.scala:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.getTokenRenewalInterval(HadoopFSDelegationTokenProvider.scala:111)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:53)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:132)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:130)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:130)
at
org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
at
org.apache.spark.deploy.yarn.Client.setupSecurityToken(Client.scala:322)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1028)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:180)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:179)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:507)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at
org.apache.hudi.utilities.UtilHelpers.buildSparkContext(UtilHelpers.java:214)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:457)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/09/28 07:27:55 INFO server.AbstractConnector: Stopped
Spark@108d55c4{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
20/09/28 07:27:55 INFO ui.SparkUI: Stopped Spark web UI at
http://biedva-worker-1.bigdatapoc.local:4040
20/09/28 07:27:55 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint:
Attempted to request executors before the AM has registered!
20/09/28 07:27:55 INFO cluster.YarnClientSchedulerBackend: Stopped
20/09/28 07:27:55 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
20/09/28 07:27:55 INFO memory.MemoryStore: MemoryStore cleared
20/09/28 07:27:55 INFO storage.BlockManager: BlockManager stopped
20/09/28 07:27:55 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/09/28 07:27:55 WARN metrics.MetricsSystem: Stopping a MetricsSystem that
is not running
20/09/28 07:27:55 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
20/09/28 07:27:55 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:150)
at
org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1$$anonfun$4.apply(HadoopFSDelegationTokenProvider.scala:116)
at
scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at
scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:115)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider$$anonfun$getTokenRenewalInterval$1.apply(HadoopFSDelegationTokenProvider.scala:111)
at scala.Option.flatMap(Option.scala:171)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.getTokenRenewalInterval(HadoopFSDelegationTokenProvider.scala:111)
at
org.apache.spark.deploy.security.HadoopFSDelegationTokenProvider.obtainDelegationTokens(HadoopFSDelegationTokenProvider.scala:53)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:132)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:130)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at
scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.obtainDelegationTokens(HadoopDelegationTokenManager.scala:130)
at
org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.obtainDelegationTokens(YARNHadoopDelegationTokenManager.scala:59)
at
org.apache.spark.deploy.yarn.Client.setupSecurityToken(Client.scala:322)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1028)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:180)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:179)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:507)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at
org.apache.hudi.utilities.UtilHelpers.buildSparkContext(UtilHelpers.java:214)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:457)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/09/28 07:27:55 INFO util.ShutdownHookManager: Shutdown hook called
20/09/28 07:27:55 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-431edee5-628f-4887-900b-ee4a354c574d
20/09/28 07:27:55 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-e18d8494-7656-405c-87e2-0c3d456a08e6
```
Exactly the same start command, but using hudi 0.5.2 is working
successfully, therefore I do not think it is related to the environment, but to
the changes that were implemented between 0.5.2 and 0.6.0 versions of hudi.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
