aib628 commented on issue #8848:
URL: https://github.com/apache/hudi/issues/8848#issuecomment-1676988983
> Yeah, maybe it's my fault, we do not exclude calcite when packaging the
bundle with hive-exec, maybe for some Hive version since 3.x, the calcite
related classes are required, but the hive-exec itself does not include the
calcite, do you package by using the same verison hive-exec as your hive server?
@danny0405 Hi, new problem found in hadoop3.2.2, build command: mvn clean
install -DskipTests -Dhadoop.version=3.2.2 -Dhive.version=3.1.2 -Dflink1.13
-Drat.skip=true -Pflink-bundle-shade-hive3
```
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.utils.HiveSyncContext.create(HiveSyncContext.java:87)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.initHiveSync(StreamWriteOperatorCoordinator.java:323)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:200)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:198)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:589)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
... 20 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_282]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_282]
at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_282]
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.utils.HiveSyncContext.create(HiveSyncContext.java:87)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.initHiveSync(StreamWriteOperatorCoordinator.java:323)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:200)
~[blob_p-8fb0a9a43a0b21fb8067cce0a27f3f694247a52c-7896267fa9a5412ec3662db78cba584e:0.14.0-SNAPSHOT]
at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:198)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:589)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.12-1.13.1.jar:1.13.1]
... 20 more
```
same as calcite-core.jar, hive sync work normally after add
hadoop-client-api.jar into flink runtime lib with --jar option manually.
as you said, maybe for some hive version since 3.x, there had some changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
