abhiNB-star commented on issue #13410:
URL: https://github.com/apache/hudi/issues/13410#issuecomment-3043975383
@danny0405
**Hi tried with this Fcaing this issue :**
25/07/07 07:46:05 INFO TaskSetManager: Finished task 399.0 in stage 61.0
(TID 5522) in 4 ms on 10.51.10.19 (executor 2) (401/401)
25/07/07 07:46:05 INFO TaskSchedulerImpl: Removed TaskSet 61.0, whose tasks
have all completed, from pool
25/07/07 07:46:05 INFO DAGScheduler: ResultStage 61 (countByKey at
HoodieJavaPairRDD.java:109) finished in 0.472 s
25/07/07 07:46:05 INFO DAGScheduler: Job 31 is finished. Cancelling
potential speculative or zombie tasks for this job
25/07/07 07:46:05 INFO TaskSchedulerImpl: Killing all running tasks in stage
61: Stage finished
25/07/07 07:46:05 INFO DAGScheduler: Job 31 finished: countByKey at
HoodieJavaPairRDD.java:109, took 130.559404 s
25/07/07 07:46:05 INFO UpsertPartitioner: Total Buckets: 4, bucketInfoMap
size: 4, partitionPathToInsertBucketInfos size: 0, updateLocationToBucket size:
4
25/07/07 07:46:05 INFO HoodieHeartbeatClient: Stopping heartbeat for instant
20250707073821517
25/07/07 07:46:05 INFO HoodieHeartbeatClient: Stopped heartbeat for instant
20250707073821517
25/07/07 07:46:05 INFO TransactionManager: Transaction manager closed
25/07/07 07:46:05 INFO TransactionManager: Transaction manager closed
25/07/07 07:46:05 INFO TransactionManager: State change ending for action
instant Option{val=[==>20250707073821517__commit__INFLIGHT]}
25/07/07 07:46:05 INFO InProcessLockProvider: Base Path
gs://starship-hood-prod/nbdata/data/v4/test/party_test/, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@2d9d42a4[Write locks = 1,
Read locks = 0], Thread pool-37-thread-1, In-process lock state RELEASING
25/07/07 07:46:05 INFO InProcessLockProvider: Base Path
gs://starship-hood-prod/nbdata/data/v4/test/party_test/, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@2d9d42a4[Write locks = 0,
Read locks = 0], Thread pool-37-thread-1, In-process lock state RELEASED
25/07/07 07:46:05 INFO InProcessLockProvider: Base Path
gs://starship-hood-prod/nbdata/data/v4/test/party_test/, Lock Instance
java.util.concurrent.locks.ReentrantReadWriteLock@2d9d42a4[Write locks = 0,
Read locks = 0], Thread pool-37-thread-1, In-process lock state ALREADY_RELEASED
25/07/07 07:46:05 INFO LockManager: Released connection created for
acquiring lock
25/07/07 07:46:05 INFO TransactionManager: State change ended for action
instant Option{val=[==>20250707073821517__commit__INFLIGHT]}
25/07/07 07:46:05 INFO HoodieHeartbeatClient: Stopping heartbeat for instant
20250707073821517
25/07/07 07:46:05 INFO HoodieHeartbeatClient: Stopped heartbeat for instant
20250707073821517
25/07/07 07:46:05 INFO GoogleCloudStorageFileSystem: Successfully repaired
'gs://starship-hood-prod/nbdata/data/v4/test/party_test/.hoodie/.heartbeat/'
directory.
25/07/07 07:46:05 INFO HeartbeatUtils: Deleted the heartbeat for instant
20250707073821517
25/07/07 07:46:05 INFO HoodieHeartbeatClient: Deleted heartbeat file for
instant 20250707073821517
25/07/07 07:46:05 INFO UnionRDD: Removing RDD 140 from persistence list
25/07/07 07:46:05 INFO BlockManager: Removing RDD 140
25/07/07 07:46:05 INFO MapPartitionsRDD: Removing RDD 148 from persistence
list
25/07/07 07:46:05 INFO BlockManager: Removing RDD 148
25/07/07 07:46:05 INFO UnionRDD: Removing RDD 156 from persistence list
25/07/07 07:46:05 INFO BlockManager: Removing RDD 156
25/07/07 07:46:05 INFO MapPartitionsRDD: Removing RDD 150 from persistence
list
25/07/07 07:46:05 INFO MapPartitionsRDD: Removing RDD 155 from persistence
list
25/07/07 07:46:05 INFO BlockManager: Removing RDD 150
25/07/07 07:46:05 INFO UnionRDD: Removing RDD 174 from persistence list
25/07/07 07:46:05 INFO BlockManager: Removing RDD 155
25/07/07 07:46:05 INFO BlockManager: Removing RDD 174
25/07/07 07:46:05 ERROR HoodieStreamer: Shutting down delta-sync due to
exception
org.apache.hudi.exception.HoodieException: Error while completing streaming
commit to metadata with instant 20250707073821517
at
org.apache.hudi.client.StreamingMetadataWriteHandler.commitToMetadataTable(StreamingMetadataWriteHandler.java:81)
at
org.apache.hudi.client.SparkRDDWriteClient.writeToMetadataTable(SparkRDDWriteClient.java:163)
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:318)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:274)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:146)
at
org.apache.hudi.utilities.streamer.StreamSync.writeToSinkAndDoMetaSync(StreamSync.java:821)
at
org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:479)
at
org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:836)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to commit
20250707073821517 unable to save inflight metadata
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.saveWorkloadProfileMetadataToInflight(BaseCommitActionExecutor.java:169)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:189)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:162)
at
org.apache.hudi.table.action.deltacommit.SparkUpsertPreppedDeltaCommitActionExecutor.execute(SparkUpsertPreppedDeltaCommitActionExecutor.java:44)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:126)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsertPrepped(HoodieSparkMergeOnReadTable.java:88)
at
org.apache.hudi.client.SparkRDDMetadataWriteClient.upsertPreppedRecords(SparkRDDMetadataWriteClient.java:135)
at
org.apache.hudi.client.SparkRDDMetadataWriteClient.upsertPreppedRecords(SparkRDDMetadataWriteClient.java:54)
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.secondaryWriteToMetadataTablePartitions(SparkHoodieBackedTableMetadataWriter.java:176)
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.secondaryWriteToMetadataTablePartitions(SparkHoodieBackedTableMetadataWriter.java:75)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.prepareAndWriteToNonStreamingPartitions(HoodieBackedTableMetadataWriter.java:1218)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.completeStreamingCommit(HoodieBackedTableMetadataWriter.java:1205)
at
org.apache.hudi.client.StreamingMetadataWriteHandler.commitToMetadataTable(StreamingMetadataWriteHandler.java:79)
... 11 more
Caused by: org.apache.hudi.exception.HoodieIOException: Failed to create
file
gs://starship-hood-prod/nbdata/data/v4/test/party_test/.hoodie/metadata/.hoodie/timeline/20250707073821517.deltacommit.inflight
at
org.apache.hudi.storage.HoodieStorage.createImmutableFileInPath(HoodieStorage.java:353)
at
org.apache.hudi.storage.HoodieStorage.createImmutableFileInPath(HoodieStorage.java:312)
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionPendingState(ActiveTimelineV2.java:563)
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionRequestedToInflight(ActiveTimelineV2.java:626)
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.saveWorkloadProfileMetadataToInflight(BaseCommitActionExecutor.java:167)
... 23 more
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
'gs://starship-hood-prod/nbdata/data/v4/test/party_test/.hoodie/metadata/.hoodie/timeline/20250707073821517.deltacommit.inflight'
already exists
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:92)
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.<init>(GoogleHadoopOutputStream.java:76)
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.lambda$create$3(GoogleHadoopFileSystemBase.java:680)
at
com.google.cloud.hadoop.fs.gcs.GhfsStorageStatistics.trackDuration(GhfsStorageStatistics.java:104)
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:653)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1064)
at
org.apache.hudi.hadoop.fs.HoodieWrapperFileSystem.lambda$create$2(HoodieWrapperFileSystem.java:243)
at
org.apache.hudi.hadoop.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:118)
at
org.apache.hudi.hadoop.fs.HoodieWrapperFileSystem.create(HoodieWrapperFileSystem.java:242)
at
org.apache.hudi.storage.hadoop.HoodieHadoopStorage.create(HoodieHadoopStorage.java:129)
at
org.apache.hudi.storage.HoodieStorage.createImmutableFileInPath(HoodieStorage.java:348)
... 27 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
