gbcoder2020 commented on issue #13002:
URL: https://github.com/apache/hudi/issues/13002#issuecomment-2743922673
@nsivabalan
```
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 ERROR YarnScheduler: Lost executor 1492 on <masked ***>:
Executor heartbeat timed out after 149584 ms
25/03/11 15:03:59 ERROR YarnScheduler: Lost executor 1492 on <masked ***>
Executor heartbeat timed out after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 INFO DAGScheduler: ResultStage 260 (collect at
HoodieJavaRDD.java:177) failed in 7303.427 s due to Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
25/03/11 15:03:59 INFO DAGScheduler: ResultStage 260 (collect at
HoodieJavaRDD.java:177) failed in 7303.427 s due to Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
2025-03-11T15:04:00.241Z
Driver stacktrace:
Driver stacktrace:
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1473.2 in stage 260.0 (TID
1152417) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1473.2 in stage 260.0 (TID
1152417) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 873.2 in stage 260.0 (TID
1152416) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 873.2 in stage 260.0 (TID
1152416) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 2073.2 in stage 260.0 (TID
1152419) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 2073.2 in stage 260.0 (TID
1152419) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 2373.2 in stage 260.0 (TID
1152413) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 2373.2 in stage 260.0 (TID
1152413) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1773.2 in stage 260.0 (TID
1152412) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1773.2 in stage 260.0 (TID
1152412) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 273.2 in stage 260.0 (TID
1152415) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 273.2 in stage 260.0 (TID
1152415) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 573.2 in stage 260.0 (TID
1152418) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 573.2 in stage 260.0 (TID
1152418) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1173.2 in stage 260.0 (TID
1152414) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
25/03/11 15:03:59 WARN TaskSetManager: Lost task 1173.2 in stage 260.0 (TID
1152414) (<masked ***> executor 1492): ExecutorLostFailure (executor 1492
exited caused by one of the running tasks) Reason: Executor heartbeat timed out
after 149584 ms
2025-03-11T15:04:00.241Z
25/03/11 15:03:59 INFO ExecutorMonitor: Executor 1492 is removed. Remove
reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0,
driver killed: 603, unexpectedly exited: 877).
25/03/11 15:03:59 INFO ExecutorMonitor: Executor 1492 is removed. Remove
reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0,
driver killed: 603, unexpectedly exited: 877).
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO DAGScheduler: Executor lost: 1493 (epoch 102)
25/03/11 15:04:00 INFO DAGScheduler: Executor lost: 1493 (epoch 102)
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Trying to remove executor
1493 from BlockManagerMaster.
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Trying to remove executor
1493 from BlockManagerMaster.
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(1493, <masked ***>, 44649, None)
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(1493, <masked ***>, 44649, None)
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMaster: Removed 1493 successfully in
removeExecutor
25/03/11 15:04:00 INFO BlockManagerMaster: Removed 1493 successfully in
removeExecutor
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO DAGScheduler: Job 117 failed: collect at
HoodieJavaRDD.java:177, took 7303.439556 s
25/03/11 15:04:00 INFO DAGScheduler: Job 117 failed: collect at
HoodieJavaRDD.java:177, took 7303.439556 s
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO DAGScheduler: Executor lost: 1492 (epoch 102)
25/03/11 15:04:00 INFO DAGScheduler: Executor lost: 1492 (epoch 102)
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Trying to remove executor
1492 from BlockManagerMaster.
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Trying to remove executor
1492 from BlockManagerMaster.
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(1492, <masked ***>, 33069, None)
25/03/11 15:04:00 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(1492, <masked ***>, 33069, None)
2025-03-11T15:04:00.241Z
25/03/11 15:04:00 INFO BlockManagerMaster: Removed 1492 successfully in
removeExecutor
25/03/11 15:04:00 INFO BlockManagerMaster: Removed 1492 successfully in
removeExecutor
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 ERROR HoodieBackedTableMetadataWriter: Exception in
running table services on metadata table
25/03/11 15:04:00 ERROR HoodieBackedTableMetadataWriter: Exception in
running table services on metadata table
2025-03-11T15:04:00.242Z
org.apache.hudi.exception.HoodieCompactionException: Could not compact
s3://<s3 path>/.hoodie/metadata
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:129)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.compact(HoodieSparkMergeOnReadTable.java:155)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.compact(BaseHoodieTableServiceClient.java:297)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1126)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1046)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1316)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1267)
at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323)
at
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288)
at
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154)
at
org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:225)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.liftedTree1$1(HoodieSparkSqlWriter.scala:492)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:490)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
at
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:104)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:123)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:160)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$8(SQLExecution.scala:160)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:271)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:159)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:101)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:107)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:530)
at
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:97)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:84)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:82)
at
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
at
org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
.
.
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
.
.
.
.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
org.apache.hudi.exception.HoodieCompactionException: Could not compact
s3://<s3 path>/.hoodie/metadata at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:129)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.compact(HoodieSparkMergeOnReadTable.java:155)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.compact(BaseHoodieTableServiceClient.java:297)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1126)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1046)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1316)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1267)
at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323)
at org.apache.hudi.client.SparkRDDWriteClient.initMet
adataTable(SparkRDDWriteClient.java:288) at
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154)
at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:225)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.liftedTree1$1(HoodieSparkSqlWriter.scala:492)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:490)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168) at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectR
esult$lzycompute(commands.scala:75) at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:104)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:123)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:160)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$8(SQ
LExecution.scala:160) at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:271)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:159)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:101)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:107)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans
$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:530)
at
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:97)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:84)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:82)
at
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142)
at org.apache
.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856) at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
.
.
.
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2025-03-11T15:04:00.242Z
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
2025-03-11T15:04:00.242Z
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2974)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2910)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2909)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2909)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1263)
at scala.Option.foreach(Option.scala:407)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3173)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3112)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3101)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2271)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2292)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2311)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2336)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1019)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:405)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1018)
at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
at
org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
at
org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:177)
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:113)
... 74 more
Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2974)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2910)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2909)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2909) at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1263)
at scala.Option.foreach(Option.scala:407) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1263)
at org.apache.spark.scheduler.DAGSchedulerEventProcess
Loop.doOnReceive(DAGScheduler.scala:3173) at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3112)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3101)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2271) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2292) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2311) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2336) at
org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1019) at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:405) at
org.apache.spark.rdd.RDD.collect(RDD.scala:1018) at
org.apache.spark.api.java.Java
RDDLike.collect(JavaRDDLike.scala:362) at
org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361) at
org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) at
org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:177) at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:113)
... 74 more
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152414 for unknown executor 1492.
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152414 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO TransactionManager: Transaction ending with
transaction owner Option{val=[==>20250311130120164__commit__INFLIGHT]}
25/03/11 15:04:00 INFO TransactionManager: Transaction ending with
transaction owner Option{val=[==>20250311130120164__commit__INFLIGHT]}
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO InProcessLockProvider: Base Path s3://<s3 path to
hoodie table>, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@1212c905[Write locks = 1,
Read locks = 0], Thread main, In-process lock state RELEASING
25/03/11 15:04:00 INFO InProcessLockProvider: Base Path s3://<s3 path to
hoodie table>, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@1212c905[Write locks = 1,
Read locks = 0], Thread main, In-process lock state RELEASING
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO InProcessLockProvider: Base Path s3://<s3 path to
hoodie table>, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@1212c905[Write locks = 0,
Read locks = 0], Thread main, In-process lock state RELEASED
25/03/11 15:04:00 INFO InProcessLockProvider: Base Path s3://<s3 path to
hoodie table>, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@1212c905[Write locks = 0,
Read locks = 0], Thread main, In-process lock state RELEASED
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO InProcessLockProvider: Base Path s3://<s3 path to
hoodie table>, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@1212c905[Write locks = 0,
Read locks = 0], Thread main, In-process lock state ALREADY_RELEASED
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO LockManager: Released connection created for
acquiring lock
25/03/11 15:04:00 INFO LockManager: Released connection created for
acquiring lock
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO TransactionManager: Transaction ended with
transaction owner Option{val=[==>20250311130120164__commit__INFLIGHT]}
25/03/11 15:04:00 INFO TransactionManager: Transaction ended with
transaction owner Option{val=[==>20250311130120164__commit__INFLIGHT]}
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO HoodieSparkSqlWriterInternal:
Config.inlineCompactionEnabled ? false
25/03/11 15:04:00 INFO HoodieSparkSqlWriterInternal:
Config.inlineCompactionEnabled ? false
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO HoodieSparkSqlWriterInternal:
Config.asyncClusteringEnabled ? false
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN HoodieSparkSqlWriterInternal: Closing write client
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO HoodieHeartbeatClient: Stopping heartbeat for instant
20250311130120164
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO HoodieHeartbeatClient: Stopped heartbeat for instant
20250311130120164
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO TransactionManager: Transaction manager closed
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO TransactionManager: Transaction manager closed
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152419 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152417 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152412 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152416 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152415 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152413 for unknown executor 1492.
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 WARN YarnSchedulerBackend$YarnDriverEndpoint: Attempted to
kill task 1152418 for unknown executor 1492.
2025-03-11T15:04:00.242Z
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed
to instantiate Metadata table
at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:326)
at
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288)
at
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154)
at
org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:225)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.liftedTree1$1(HoodieSparkSqlWriter.scala:492)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:490)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
at
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:104)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:123)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:160)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$8(SQLExecution.scala:160)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:271)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:159)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:101)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:107)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:530)
at
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:97)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:84)
at
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:82)
at
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
at
org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
.
.
.
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
.
.
.
.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed
to instantiate Metadata table at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:326)
at
org.apache.hudi.client.SparkRDDWriteClient.initMetadataTable(SparkRDDWriteClient.java:288)
at
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1244)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1284)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154)
at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:225)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.liftedTree1$1(HoodieSparkSqlWriter.scala:492)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:490)
at
org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:1
25) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:104)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:123)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:160)
a
t
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$8(SQLExecution.scala:160)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:271)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:159)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:101)
at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:554)
at org.apa
che.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:107) at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:554)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:530)
at
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:97)
at org.apache.spark.sql.execution.Query
Execution.commandExecuted$lzycompute(QueryExecution.scala:84) at
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:82)
at
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
.
.
.
.
.
.
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192) at
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215) at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2025-03-11T15:04:00.242Z
Caused by: org.apache.hudi.exception.HoodieCompactionException: Could not
compact s3://<s3 path>/.hoodie/metadata
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:129)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.compact(HoodieSparkMergeOnReadTable.java:155)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.compact(BaseHoodieTableServiceClient.java:297)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1126)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1046)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1316)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1267)
at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323)
... 67 more
Caused by: org.apache.hudi.exception.HoodieCompactionException: Could not
compact s3://<s3 path>/.hoodie/metadata at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:129)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.compact(HoodieSparkMergeOnReadTable.java:155)
at
org.apache.hudi.client.BaseHoodieTableServiceClient.compact(BaseHoodieTableServiceClient.java:297)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1126)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1046)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.compactIfNecessary(HoodieBackedTableMetadataWriter.java:1316)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.performTableServices(HoodieBackedTableMetadataWriter.java:1267)
at
org.apache.hudi.client.SparkRDDWriteClient.initializeMetadataTable(SparkRDDWriteClient.java:323)
... 67 more
2025-03-11T15:04:00.242Z
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 2075 in stage 260.0 failed 4 times, most recent failure: Lost
task 2075.3 in stage 260.0 (TID 1152449) (<masked ***> executor 1493):
ExecutorLostFailure (executor 1493 exited caused by one of the running tasks)
Reason: Executor heartbeat timed out after 131337 ms
2025-03-11T15:04:00.242Z
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2974)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2910)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2909)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2909)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1263)
at scala.Option.foreach(Option.scala:407)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3173)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3112)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3101)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2271)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2292)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2311)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2336)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1019)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:405)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1018)
at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
at
org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
at
org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:177)
at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:113)
... 74 more
Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2974)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2910)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2909)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2909) at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1263)
at scala.Option.foreach(Option.scala:407) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1263)
at org.apache.spark.scheduler.DAGSchedulerEventProcess
Loop.doOnReceive(DAGScheduler.scala:3173) at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3112)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3101)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2271) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2292) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2311) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2336) at
org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1019) at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:405) at
org.apache.spark.rdd.RDD.collect(RDD.scala:1018) at
org.apache.spark.api.java.Java
RDDLike.collect(JavaRDDLike.scala:362) at
org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361) at
org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45) at
org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:177) at
org.apache.hudi.table.action.compact.RunCompactionActionExecutor.execute(RunCompactionActionExecutor.java:113)
... 74 more
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO SparkContext: Invoking stop() from shutdown hook
2025-03-11T15:04:00.242Z
25/03/11 15:04:00 INFO SparkContext: SparkContext is stopping with exitCode
0.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
