Leoyzen commented on issue #7824:
URL: https://github.com/apache/hudi/issues/7824#issuecomment-1415707154
@danny0405 No, the path doesn't exists.
After upgrading latest branch(0.13.0-rc1), the exception has changed.
```LOG
2023-02-03 18:54:52,537 [flink-akka.actor.default-dispatcher-39] INFO
org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to
recover from a global failure.
org.apache.flink.util.FlinkException: Global failure triggered by
OperatorCoordinator for 'bucket_write: tbl_resp_sink' (operator
2b7fd632fd7edc109c5698c0b534dcae).
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:597)
~[flink-dist-1.15-vvr-6.0.4-SNAPSHOT.jar:1.15-vvr-6.0.4-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:196)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:142)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
~[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
~[?:1.8.0_102]
at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]
Caused by: org.apache.hudi.exception.HoodieException: Executor executes
action [initialize instant 20230203185127227] error
... 6 more
Caused by: org.apache.hudi.exception.HoodieException: Failed to update
metadata
at
org.apache.hudi.client.HoodieFlinkTableServiceClient.writeTableMetadata(HoodieFlinkTableServiceClient.java:181)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:279)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:282)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:233)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:111)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:199)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:548)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:524)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:495)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:413)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[?:?]
... 3 more
Caused by: java.lang.IllegalStateException: Metadata table is not fully
initialized yet.
at
org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67)
~[?:?]
at
org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:109)
~[?:?]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:823)
~[?:?]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:890)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkTableServiceClient.writeTableMetadata(HoodieFlinkTableServiceClient.java:179)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:279)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:282)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:233)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:111)
~[?:?]
at
org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:74)
~[?:?]
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:199)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:548)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:524)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:495)
~[?:?]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:413)
~[?:?]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[?:?]
... 3 more
```
It seems the procedure:
1. The hudi launched and found there isn't any metadata exists, try to
create one.
2. The creation fails because of there is delta commit inflighting.
3. When the data flushing to the disk, the error occurs(the metadata has not
been initilization).
```LOG
2023-02-03 18:54:08,449 [pool-24-thread-1] WARN
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter [] - Cannot
initialize metadata table as operation(s) are in progress on the dataset:
[[==>20230203184713442__deltacommit__INFLIGHT],
[==>20230203185127185__rollback__INFLIGHT],
[==>20230203185127227__deltacommit__INFLIGHT]]
..........
2023-02-03 18:54:08,238 [pool-24-thread-1] WARN
org.apache.hudi.metadata.HoodieBackedTableMetadata [] - Metadata
table was not found at path
oss://dengine-lake-zjk/cloudcode_pre/dwd_egc_adv_resp_intra/.hoodie/metadata
............
Caused by: java.lang.IllegalStateException: Metadata table is not fully
initialized yet.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
