Riddle4045 commented on issue #9435: URL: https://github.com/apache/hudi/issues/9435#issuecomment-1676756322
> Did you check the log of JM to ensure the hive sync works as expected? @danny0405 , just checked JM logs. I do see some errors, and they seem to be related to HMS DDL execution failure. ``` 023-08-14 06:23:44.846 [] pool-44124-thread-1 INFO flink org.apache.hudi.client.HoodieTimelineArchiver 179 No Instants to archive 2023-08-14 06:23:44.916 [] pool-44124-thread-1 INFO flink org.apache.hudi.sink.StreamWriteOperatorCoordinator 542 Commit instant [20230814062340927] success! 2023-08-14 06:23:44.921 [] pool-44124-thread-1 INFO flink apache.hudi.common.table.HoodieTableMetaClient 130 Loading HoodieTableMetaClient from abfs://[email protected]/hudi/t1hms3 2023-08-14 06:23:44.934 [] pool-44124-thread-1 INFO flink apache.hudi.common.table.HoodieTableConfig 268 Loading table properties from abfs://[email protected]/hudi/t1hms3/.hoodie/hoodie.properties 2023-08-14 06:23:44.960 [] pool-44124-thread-1 INFO flink apache.hudi.common.table.HoodieTableMetaClient 149 Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from abfs://[email protected]/hudi/t1hms3 2023-08-14 06:23:44.960 [] pool-44124-thread-1 INFO flink apache.hudi.common.table.HoodieTableMetaClient 152 Loading Active commit timeline for abfs://[email protected]/hudi/t1hms3 2023-08-14 06:23:44.979 [] pool-44124-thread-1 INFO flink hudi.common.table.timeline.HoodieActiveTimeline 171 Loaded instants upto : Option{val=[20230814062340927__commit__COMPLETED]} 2023-08-14 06:23:44.982 [] pool-44124-thread-1 ERROR flink org.apache.hudi.sink.StreamWriteOperatorCoordinator 140 Executor executes action [handle end input event for instant 20230814062340927] error java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?] at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?] at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?] at java.lang.Class.getMethod0(Unknown Source) ~[?:?] at java.lang.Class.getMethod(Unknown Source) ~[?:?] at org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[blob_p-f6d17d46787238a244a654163d9291f028bd4b50-f555f20a32e666dc6370967ae417e027:0.13.0] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.ClassNotFoundException: org.apache.calcite.plan.RelOptRule at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?] at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?] ... 20 more 2023-08-14 06:23:44.983 [] flink-akka.actor.default-dispatcher-20 INFO flink apache.flink.runtime.executiongraph.ExecutionGraph 1435 stream_write: flink_hudi_hms3 -> Sink: clean_commits (1/1) (95e11513c14db026e5f45d6c8222d2aa_8d3451599e14a899830b68533f64c818_0_0) switched from RUNNING to FINISHED. 2023-08-14 06:23:44.983 [] flink-akka.actor.default-dispatcher-15 INFO flink flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager 292 Clearing resource requirements of job feae5f184bade9247fe744f103bc101a 2023-08-14 06:23:44.984 [] flink-akka.actor.default-dispatcher-20 INFO flink apache.flink.runtime.executiongraph.ExecutionGraph 1152 Job insert-into_myhive.flink.flink_hudi_hms3 (feae5f184bade9247fe744f103bc101a) switched from state RUNNING to FINISHED. 2023-08-14 06:23:44.984 [] flink-akka.actor.default-dispatcher-20 INFO flink apache.flink.runtime.checkpoint.CheckpointCoordinator 411 Stopping checkpoint coordinator for job feae5f184bade9247fe744f103bc101a. 2023-08-14 06:23:44.985 [] flink-akka.actor.default-dispatcher-20 INFO flink apache.flink.runtime.jobmaster.JobMaster 297 Trying to recover from a global failure. org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'stream_write: flink_hudi_hms3 -> Sink: clean_commits' (operator 8d3451599e14a899830b68533f64c818). at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:617) ~[flink-dist-1.16.0-0.0.18.jar:1.16.0-0.0.18] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:190) ~[?:?] at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:142) ~[?:?] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at java.lang.Thread.run(Unknown Source) ~[?:?] Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [handle end input event for instant 20230814062340927] error ... 8 more Caused by: java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?] at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?] at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?] at java.lang.Class.getMethod0(Unknown Source) ~[?:?] at java.lang.Class.getMethod(Unknown Source) ~[?:?] at org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86) ~[?:?] at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87) ~[?:?] at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119) ~[?:?] at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113) ~[?:?] at org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281) ~[?:?] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[?:?] ... 5 more Caused by: java.lang.ClassNotFoundException: org.apache.calcite.plan.RelOptRule at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?] at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?] at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:?] at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[?:?] at java.lang.Class.getMethodsRecursive(Unknown Source) ~[?:?] at java.lang.Class.getMethod0(Unknown Source) ~[?:?] at java.lang.Class.getMethod(Unknown Source) ~[?:?] at org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:86) ~[?:?] at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:87) ~[?:?] at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119) ~[?:?] at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113) ~[?:?] at org.apache.hudi.sink.utils.HiveSyncContext.hiveSyncTool(HiveSyncContext.java:81) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:338) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:329) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:433) ~[?:?] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:281) ~[?:?] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[?:?] ... 5 more 2023-08-14 06:23:44.991 [] flink-akka.actor.default-dispatcher-20 INFO flink apache.flink.runtime.dispatcher.StandaloneDispatcher 1113 Job feae5f184bade9247fe744f103bc101a reached terminal state FINISHED. 2023-08-14 06:23:45.108 [] cluster-io-thread-1 INFO flink apache.flink.runtime.history.FsJobArchivist 91 Job feae5f184bade9247fe744f103bc101a has been archived at abfs://[email protected]/completed-jobs/feae5f184bade9247fe744f103bc101a. 2023-08-14 06:23:45.162 [] cluster-io-thread-3 INFO flink apache.flink.kubernetes.highavailability.KubernetesMultipleComponentLeaderElectionHaServices 214 Clean up the high availability data for job 8b89dcbd2662be08651d7a1fcacb1c4f. 2023-08-14 06:23:45.164 [] cluster-io-thread-3 WARN flink flink.runtime.dispatcher.cleanup.DefaultResourceCleaner 223 Cleanup of HighAvailabilityServices failed for job 8b89dcbd2662be08651d7a1fcacb1c4f due to a CompletionException: java.util.concurrent.ExecutionException: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://10.0.0.1/api/v1/namespaces/97e6cfd01ddd497db1cfdd13ed38d5c9/configmaps/flink-97e6cfd01ddd497db1cfdd13ed38d5c9-8b89dcbd2662be08651d7a1fcacb1c4f-config-map. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "flink-97e6cfd01ddd497db1cfdd13ed38d5c9-8b89dcbd2662be08651d7a1fcacb1c4f-config-map" is forbidden: User "system:serviceaccount:97e6cfd01ddd497db1cfdd13ed38d5c9:97e6cfd01ddd497db1cfdd13ed38d5c9" cannot delete resource "configmaps" in API group "" in the namespace "97e6cfd01ddd497db1cfdd13ed38d5c9". 2023-08-14 06:23:45.258 [] cluster-io-thread-4 INFO flink apache.flink.runtime.dispatcher.StandaloneDispatcher 1161 Job feae5f184bade9247fe744f103bc101a has been registered for cleanup in the JobResultStore after reaching a terminal state. 2023-08-14 06:23:45.258 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.flink.runtime.jobmaster.JobMaster 434 Stopping the JobMaster for job 'insert-into_myhive.flink.flink_hudi_hms3' (feae5f184bade9247fe744f103bc101a). 2023-08-14 06:23:45.259 [] flink-akka.actor.default-dispatcher-15 INFO flink org.apache.hudi.client.BaseHoodieClient 111 Stopping Timeline service !! 2023-08-14 06:23:45.260 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.hudi.client.embedded.EmbeddedTimelineService 151 Closing Timeline server 2023-08-14 06:23:45.260 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.hudi.timeline.service.TimelineService 399 Closing Timeline Service 2023-08-14 06:23:45.260 [] flink-akka.actor.default-dispatcher-15 INFO flink io.javalin.Javalin 22 Stopping Javalin ... 2023-08-14 06:23:45.264 [] flink-akka.actor.default-dispatcher-15 INFO flink io.javalin.Javalin 22 Javalin has stopped 2023-08-14 06:23:45.265 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.hudi.timeline.service.TimelineService 408 Closed Timeline Service 2023-08-14 06:23:45.265 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.hudi.client.embedded.EmbeddedTimelineService 155 Closed Timeline server 2023-08-14 06:23:45.266 [] flink-akka.actor.default-dispatcher-15 INFO flink org.apache.hudi.client.BaseHoodieClient 111 Stopping Timeline service !! 2023-08-14 06:23:45.266 [] flink-akka.actor.default-dispatcher-15 INFO flink apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore 182 Shutting down 2023 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
