hanson2021 opened a new issue #3838:
URL: https://github.com/apache/hudi/issues/3838
Hadoop: 3.0.0-cdh6.3.2
Hive：2.1.1-cdh6.3.2
Flink: 1.13.1
Scala: 2.11
(1) set hudi/pom.xml hive.version=2.1.1 to recompile like this：
mvn clean install -DskipTests -DskipITs -Dcheckstyle.skip=true
-Drat.skip=true -Dscala-2.11 -Dhive.version=2.1.1 -Pflink-bundle-shade-hive2
-Dflink.version=1.13.1
Error info as following:
```
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project hudi-hadoop-mr: Compilation failure
[ERROR]
/data1/download/apachehudi/hudi-0.10.0/tmp/hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:[32,37]
找不到符号
[ERROR] 符号: 类 StringInternUtils
[ERROR] 位置: 程序包 org.apache.hadoop.hive.common
[ERROR]
[ERROR] -> [Help 1]
```
(2) After use default pom.xml to recompile ,when run flink sql on yarn with
hudi、hive、flink, Error info as following:
```
2021-10-21 10:56:00,215 INFO hive.metastore
[] - Closed a connection to metastore, current connections: 0
2021-10-21 10:56:00,215 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [sync hive metadata for instant 20211021105600] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive
syncing unknown
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:120)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:305)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if
table exists unknown_ro
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:214)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name:
'get_table_req'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2303)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:212)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
2021-10-21 10:56:01,034 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 192 for job aa1c0e656a89bab1cd28f729d3356fba (56712 bytes in 1982
ms).
2021-10-21 10:56:01,037 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 193 (type=CHECKPOINT) @ 1634784961035 for job
aa1c0e656a89bab1cd28f729d3356fba.
2021-10-21 10:56:01,080 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Commit
instant [20211021105600] success!
2021-10-21 10:56:02,041 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Create
instant [20211021105602] for table [hudi_lcy_test2] with type [MERGE_ON_READ]
2021-10-21 10:56:02,041 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [commits the instant 20211021105600] success!
2021-10-21 10:56:02,041 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [taking checkpoint 193] success!
2021-10-21 10:56:02,053 INFO hive.metastore
[] - Trying to connect to metastore with URI
thrift://sh1-bi-thdwcdh-hm02:9083
2021-10-21 10:56:02,053 INFO hive.metastore
[] - Opened a connection to metastore, current connections: 1
2021-10-21 10:56:02,053 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [handle write metadata event for instant 20211021105602]
success!
2021-10-21 10:56:02,054 INFO hive.metastore
[] - Connected to metastore.
2021-10-21 10:56:02,146 INFO
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [handle write metadata event for instant 20211021105602]
success!
2021-10-21 10:56:02,209 INFO hive.metastore
[] - Closed a connection to metastore, current connections: 0
2021-10-21 10:56:02,210 ERROR
org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor
executes action [sync hive metadata for instant 20211021105602] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive
syncing unknown
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:120)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:305)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if
table exists unknown_ro
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:214)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name:
'get_table_req'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at
org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2303)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:212)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117)
~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
