duntonr opened a new issue, #11589: URL: https://github.com/apache/hudi/issues/11589
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? YES **Describe the problem you faced** A clear and concise description of the problem. **To Reproduce** Steps to reproduce the behavior: 1. Launch Hudi Delta Streamer (command used below) 2. Wait a while 3. Boom **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.15 * Spark version : 3.5.1 * Hive version : n/a * Hadoop version : 3.3.4 * Storage (HDFS/S3/GCS..) : S3 (Wasabi) * Running on Docker? (yes/no) : yes **Additional context** Hudi Launch Command: ``` spark-submit \ --jars /opt/jars/aws-java-sdk-bundle-1.12.735.jar,/opt/jars/hadoop-aws-3.3.4.jar,/opt/jars/hadoop-common-3.3.4.jar,/opt/jars/woodstox-core-6.5.1.jar,/opt/jars/stax2-api-4.2.1.jar,/opt/jars/commons-configuration2-2.9.0.jar,/opt/jars/re2j-1.7.jar,/opt/jars/hbase-client-2.6.0-hadoop3.jar,/opt/jars/hudi-spark3.5-bundle_2.12-0.15.0.jar,/opt/jars/hudi-utilities-bundle_2.12-0.15.0.jar \ --conf spark.driver.extraClassPath=/opt/jars/aws-java-sdk-bundle-1.12.735.jar:/opt/jars/hadoop-aws-3.3.4.jar:/opt/jars/hadoop-common-3.3.4.jar:/opt/jars/woodstox-core-6.5.1.jar:/opt/jars/stax2-api-4.2.1.jar:/opt/jars/commons-configuration2-2.9.0.jar:/opt/jars/re2j-1.7.jar:/opt/jars/hbase-client-2.6.0-hadoop3.jar:/opt/jars/hudi-spark3.5-bundle_2.12-0.15.0.jar:/opt/jars/hudi-utilities-bundle_2.12-0.15.0.jar \ --conf spark.executor.extraClassPath=/opt/jars/aws-java-sdk-bundle-1.12.735.jar:/opt/jars/hadoop-aws-3.3.4.jar:/opt/jars/hadoop-common-3.3.4.jar:/opt/jars/woodstox-core-6.5.1.jar:/opt/jars/stax2-api-4.2.1.jar:/opt/jars/commons-configuration2-2.9.0.jar:/opt/jars/re2j-1.7.jar:/opt/jars/hbase-client-2.6.0-hadoop3.jar:/opt/jars/hudi-spark3.5-bundle_2.12-0.15.0.jar:/opt/jars/hudi-utilities-bundle_2.12-0.15.0.jar \ --deploy-mode cluster \ --supervise \ --num-executors 10 \ --executor-memory 2700m \ --conf spark.executor.cores=4 \ --driver-memory 2700m \ --name xxxxxxx \ --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \ --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ --conf 'spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem' \ --conf 'spark.hadoop.fs.s3a.path.style.access=true' \ --conf 'spark.hadoop.fs.s3a.access.key=xxxxxxx' \ --conf 'spark.hadoop.fs.s3a.secret.key=xxxxxxxxxxxxx' \ --conf 'spark.hadoop.fs.s3a.endpoint=s3.us-east-1.wasabisys.com' \ --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+UseCompressedOops -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof -Djdk.attach.allowAttachSelf=true --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" \ --conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof -Djdk.attach.allowAttachSelf=true --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" \ --conf spark.driver.maxResultSize=2g \ --conf spark.rdd.compress=true \ --conf spark.decommission.enabled=true \ --conf spark.kryoserializer.buffer.max=512m \ --conf spark.task.cpus=1 \ --conf spark.sql.shuffle.partitions=100 \ --conf spark.sql.hive.convertMetastoreParquet=false \ --conf spark.task.maxFailures=4 \ --conf spark.sql.catalogImplementation=hive \ --conf spark.scheduler.mode=FAIR \ --conf spark.executor.heartbeatInterval=120s \ --conf spark.network.timeout=600s \ --conf spark.hadoop.fs.s3a.connection.maximum=80 \ --master spark://spark-master.xxxxxxxxxxxxx:7077 \ --class org.apache.hudi.utilities.streamer.HoodieStreamer /opt/jars/hudi-utilities-bundle_2.12-0.15.0.jar \ --enable-sync \ --continuous \ --table-type 'MERGE_ON_READ' \ --target-base-path 's3a://xxxxxxxxxxcrypto_ohlcv_2' \ --target-table 'crypto_ohlcv_2' \ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \ --op 'UPSERT' \ --hoodie-conf bootstrap.servers=kafka.xxxxxxxxxxxxx:9092 \ --hoodie-conf schema.registry.url=https://xxxxxxxx.us-east-2.aws.confluent.cloud \ --hoodie-conf schema.registry.basic.auth.user.info=xxxxxxx:xxxxxxxxxxx \ --hoodie-conf basic.auth.credentials.source=USER_INFO \ --hoodie-conf 'hoodie.streamer.schemaprovider.registry.url=https://xxxxxxxxxx:[email protected]/xxxxxxxxx/versions/latest' \ --hoodie-conf hoodie.streamer.source.kafka.topic=ohlcv_to_hudi_coinapi_spot_loader \ --schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \ --hoodie-conf auto.offset.reset=earliest \ --hoodie-conf group.id=xxxxx \ --hoodie-conf hoodie.table.name=xxxxxxx \ --hoodie-conf hoodie.datasource.write.table.type=MERGE_ON_READ \ --hoodie-conf hoodie.datasource.write.recordkey.field=key_str_hash \ --hoodie-conf hoodie.datasource.write.partitionpath.field=time_period_start_year,time_period_start_month,time_period_start_day,period,exchange_id,base_symbol,quote_symbol \ --hoodie-conf hoodie.datasource.write.hive_style_partitioning=true \ --hoodie-conf hoodie.metadata.log.compaction.enable=true \ --hoodie-conf hoodie.metadata.enable=true \ --hoodie-conf hoodie.metadata.index.bloom.filter.enable=true \ --hoodie-conf hoodie.metadata.index.column.stats.enable=true \ --hoodie-conf hoodie.metadata.record.index.enable=true \ --hoodie-conf hoodie.metadata.optimized.log.blocks.scan.enable=true \ --hoodie-conf hoodie.enable.data.skipping=true \ --hoodie-conf hoodie.datasource.hive_sync.database=warehouse \ --hoodie-conf hoodie.datasource.write.precombine.field=ts \ --hoodie-conf hoodie.datasource.hive_sync.support_timestamp=true \ --hoodie-conf hoodie.datasource.hive_sync.table=xxxxxx \ --hoodie-conf hoodie.datasource.hive_sync.partition_fields=time_period_start_year,time_period_start_month,time_period_start_day,period,exchange_id,base_symbol,quote_symbol \ --hoodie-conf hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.MultiPartKeysValueExtractor \ --hoodie-conf hoodie.datasource.hive_sync.use_jdbc=false \ --hoodie-conf hoodie.datasource.hive_sync.mode=hms \ --hoodie-conf hoodie.datasource.meta.sync.base.path=s3a://xxxxxxxxxxxxxxxx \ --hoodie-conf hoodie.meta.sync.metadata_file_listing=true \ --hoodie-conf hoodie.streamer.source.dfs.root=s3a://xxxxxxxxx \ --props 's3a://therackio-warehouse/warehouse/data/hudi/dfs-source.properties' \ --hoodie-conf hoodie.clustering.async.enabled=true \ --hoodie-conf hoodie.index.type=RECORD_INDEX \ --hoodie-conf hoodie.record.index.update.partition.path=true \ --hoodie-conf hoodie.cleaner.policy.failed.writes=LAZY \ --hoodie-conf hoodie.clean.automatic=true \ --hoodie-conf hoodie.streamer.kafka.source.maxEvents=100000 \ --hoodie-conf hoodie.client.heartbeat.interval_in_ms=130000 \ --hoodie-conf hoodie.streamer.source.kafka.enable.commit.offset=true \ --hoodie-conf hoodie.bloom.index.update.partition.path=true \ --source-ordering-field ts ``` **Stacktrace** ```24/07/07 19:51:05 ERROR HoodieStreamer: Shutting down delta-sync due to exception org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:326) at org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288) at org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284) at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154) at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:988) at org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:843) at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:493) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:793) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.hudi.exception.HoodieRollbackException: Generating rollback requests failed for 20240620171530629005 at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:199) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:111) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:134) at org.apache.hudi.table.HoodieSparkMergeOnReadTable.scheduleRollback(HoodieSparkMergeOnReadTable.java:198) at org.apache.hudi.table.HoodieTable.rollbackInflightLogCompaction(HoodieTable.java:683) at org.apache.hudi.client.BaseHoodieTableServiceClient.logCompact(BaseHoodieTableServiceClient.java:219) at org.apache.hudi.client.BaseHoodieTableServiceClient.lambda$runAnyPendingLogCompactions$6(BaseHoodieTableServiceClient.java:258) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.apache.hudi.client.BaseHoodieTableServiceClient.runAnyPendingLogCompactions(BaseHoodieTableServiceClient.java:256) at org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingLogCompactions(BaseHoodieWriteClient.java:611) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.runPendingTableServicesOperations(HoodieBackedTableMetadataWriter.java:1289) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1251) at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323) ... 12 more Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:410) at org.apache.spark.rdd.RDD.collect(RDD.scala:1048) at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362) at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361) at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) at org.apache.hudi.client.common.HoodieSparkEngineContext.flatMap(HoodieSparkEngineContext.java:150) at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:111) ... 25 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) ... 3 more 24/07/07 19:51:05 INFO HoodieStreamer: Delta Sync shutdown. Error ?true 24/07/07 19:51:05 WARN HoodieStreamer: Gracefully shutting down compactor 24/07/07 19:51:05 WARN TaskSetManager: Lost task 3.1 in stage 2.0 (TID 11) (spark-worker-10.service.rack01.consul.internal.therack.io executor 5): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace:) 24/07/07 19:51:06 WARN TaskSetManager: Lost task 2.0 in stage 2.0 (TID 8) (spark-worker-4.service.rack01.consul.internal.therack.io executor 6): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace:) 24/07/07 19:51:06 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID 7) (spark-worker-8.service.rack01.consul.internal.therack.io executor 1): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace:) 24/07/07 19:51:06 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool hoodiedeltasync 24/07/07 19:51:10 INFO AsyncCompactService: Compactor shutting down properly!! 24/07/07 19:51:10 WARN HoodieStreamer: Gracefully shutting down clustering service 24/07/07 19:51:10 INFO AsyncClusteringService: Clustering executor shutting down properly 24/07/07 19:51:10 INFO HoodieStreamer: Ingestion completed. Has error: true 24/07/07 19:51:10 INFO HoodieHeartbeatClient: Stopping heartbeat for instant 20240707195005576 24/07/07 19:51:10 INFO HoodieHeartbeatClient: Stopped heartbeat for instant 20240707195005576 24/07/07 19:51:10 INFO TransactionManager: Transaction manager closed 24/07/07 19:51:10 ERROR HoodieAsyncService: Service shutdown with error java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103) at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:65) at org.apache.hudi.common.util.Option.ifPresent(Option.java:101) at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:214) at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:606) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:63) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:832) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:326) at org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288) at org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284) at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154) at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:988) at org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:843) at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:493) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:793) ... 4 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Generating rollback requests failed for 20240620171530629005 at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:199) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:111) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:134) at org.apache.hudi.table.HoodieSparkMergeOnReadTable.scheduleRollback(HoodieSparkMergeOnReadTable.java:198) at org.apache.hudi.table.HoodieTable.rollbackInflightLogCompaction(HoodieTable.java:683) at org.apache.hudi.client.BaseHoodieTableServiceClient.logCompact(BaseHoodieTableServiceClient.java:219) at org.apache.hudi.client.BaseHoodieTableServiceClient.lambda$runAnyPendingLogCompactions$6(BaseHoodieTableServiceClient.java:258) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.apache.hudi.client.BaseHoodieTableServiceClient.runAnyPendingLogCompactions(BaseHoodieTableServiceClient.java:256) at org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingLogCompactions(BaseHoodieWriteClient.java:611) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.runPendingTableServicesOperations(HoodieBackedTableMetadataWriter.java:1289) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1251) at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323) ... 12 more Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:410) at org.apache.spark.rdd.RDD.collect(RDD.scala:1048) at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362) at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361) at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) at org.apache.hudi.client.common.HoodieSparkEngineContext.flatMap(HoodieSparkEngineContext.java:150) at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:111) ... 25 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) ... 3 more 24/07/07 19:51:10 INFO TransactionManager: Transaction manager closed 24/07/07 19:51:10 INFO StreamSync: Shutting down embedded timeline server 24/07/07 19:51:10 INFO EmbeddedTimelineService: Closing Timeline server 24/07/07 19:51:10 INFO TimelineService: Closing Timeline Service 24/07/07 19:51:10 INFO Javalin: Stopping Javalin ... 24/07/07 19:51:10 INFO SparkContext: SparkContext is stopping with exitCode 0. 24/07/07 19:51:10 ERROR Javalin: Javalin failed to stop gracefully java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:276) at org.apache.hudi.org.apache.jetty.server.AbstractConnector.doStop(AbstractConnector.java:373) at org.apache.hudi.org.apache.jetty.server.AbstractNetworkConnector.doStop(AbstractNetworkConnector.java:88) at org.apache.hudi.org.apache.jetty.server.ServerConnector.doStop(ServerConnector.java:246) at org.apache.hudi.org.apache.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at org.apache.hudi.org.apache.jetty.server.Server.doStop(Server.java:459) at org.apache.hudi.org.apache.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94) at io.javalin.Javalin.stop(Javalin.java:209) at org.apache.hudi.timeline.service.TimelineService.close(TimelineService.java:411) at org.apache.hudi.client.embedded.EmbeddedTimelineService.stopForBasePath(EmbeddedTimelineService.java:249) at org.apache.hudi.utilities.streamer.StreamSync.close(StreamSync.java:1272) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.close(HoodieStreamer.java:962) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.onIngestionCompletes(HoodieStreamer.java:950) at org.apache.hudi.async.HoodieAsyncService.lambda$shutdownCallback$0(HoodieAsyncService.java:171) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 24/07/07 19:51:10 INFO SparkUI: Stopped Spark web UI at http://spark-worker-6.service.rack01.consul.internal.therack.io:8090 24/07/07 19:51:10 INFO StandaloneSchedulerBackend: Shutting down all executors 24/07/07 19:51:10 INFO StandaloneSchedulerBackend$StandaloneDriverEndpoint: Asking each executor to shut down 24/07/07 19:51:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 24/07/07 19:51:11 INFO MemoryStore: MemoryStore cleared 24/07/07 19:51:11 INFO BlockManager: BlockManager stopped 24/07/07 19:51:11 INFO BlockManagerMaster: BlockManagerMaster stopped 24/07/07 19:51:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 24/07/07 19:51:11 INFO SparkContext: Successfully stopped SparkContext 24/07/07 19:51:11 ERROR TransportRequestHandler: Error sending result StreamResponse[streamId=/jars/hudi-spark3.5-bundle_2.12-0.15.0.jar,byteCount=108580053,body=FileSegmentManagedBuffer[file=/opt/jars/hudi-spark3.5-bundle_2.12-0.15.0.jar,offset=0,length=108580053]] to /10.100.100.97:45156; closing connection io.netty.channel.StacklessClosedChannelException at io.netty.channel.AbstractChannel.close(ChannelPromise)(Unknown Source) Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:63) at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) Caused by: org.apache.hudi.utilities.ingestion.HoodieIngestionException: Ingestion service was shut down with exception. at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:67) at org.apache.hudi.common.util.Option.ifPresent(Option.java:101) at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:214) at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:606) ... 6 more Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103) at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:65) ... 9 more Caused by: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:832) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.hudi.exception.HoodieException: Failed to instantiate Metadata table at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:326) at org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288) at org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284) at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154) at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:988) at org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:843) at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:493) at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:793) ... 4 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Generating rollback requests failed for 20240620171530629005 at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:199) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.requestRollback(BaseRollbackPlanActionExecutor.java:111) at org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor.execute(BaseRollbackPlanActionExecutor.java:134) at org.apache.hudi.table.HoodieSparkMergeOnReadTable.scheduleRollback(HoodieSparkMergeOnReadTable.java:198) at org.apache.hudi.table.HoodieTable.rollbackInflightLogCompaction(HoodieTable.java:683) at org.apache.hudi.client.BaseHoodieTableServiceClient.logCompact(BaseHoodieTableServiceClient.java:219) at org.apache.hudi.client.BaseHoodieTableServiceClient.lambda$runAnyPendingLogCompactions$6(BaseHoodieTableServiceClient.java:258) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at org.apache.hudi.client.BaseHoodieTableServiceClient.runAnyPendingLogCompactions(BaseHoodieTableServiceClient.java:256) at org.apache.hudi.client.BaseHoodieWriteClient.runAnyPendingLogCompactions(BaseHoodieWriteClient.java:611) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.runPendingTableServicesOperations(HoodieBackedTableMetadataWriter.java:1289) at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1251) at org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323) ... 12 more Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (spark-worker-1.service.rack01.consul.internal.therack.io executor 9): org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2856) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2792) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2791) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2791) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1247) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1247) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3060) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2994) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2983) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:989) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:410) at org.apache.spark.rdd.RDD.collect(RDD.scala:1048) at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362) at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361) at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) at org.apache.hudi.client.common.HoodieSparkEngineContext.flatMap(HoodieSparkEngineContext.java:150) at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.getRollbackRequests(ListingBasedRollbackStrategy.java:111) ... 25 more Caused by: org.apache.hudi.exception.HoodieRollbackException: Unknown listing type, during rollback of [==>20240620171530629005__logcompaction__INFLIGHT] at org.apache.hudi.table.action.rollback.ListingBasedRollbackStrategy.lambda$getRollbackRequests$742513f$1(ListingBasedRollbackStrategy.java:189) at org.apache.hudi.client.common.HoodieSparkEngineContext.lambda$flatMap$7d470b86$1(HoodieSparkEngineContext.java:150) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMap$1(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166) at org.apache.spark.scheduler.Task.run(Task.scala:141) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) ... 3 more 24/07/07 19:51:11 INFO ShutdownHookManager: Shutdown hook called 24/07/07 19:51:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-23554983-0acf-42d9-a96b-2864374faac0 24/07/07 19:51:11 INFO ShutdownHookManager: Deleting directory /alloc/tmp/spark-d843b8b1-a916-4516-868e-0cc02f0062d5 24/07/07 19:51:11 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system... 24/07/07 19:51:11 INFO MetricsSystemImpl: s3a-file-system metrics system stopped. 24/07/07 19:51:11 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.``` NOTE: Initially appeared similar to https://github.com/apache/hudi/issues/10982 but after "forward porting" the fix from https://github.com/apache/hudi/pull/10194/files to the 0.15 release and recompiling, the error changed a little to be more consistant (see thread from 10982) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
