LIKE-HUB opened a new issue, #18077:
URL: https://github.com/apache/hudi/issues/18077
### Bug Description
Title: Hive Sync Fails with TTransportException When Using Kerberos
Authentication
Environment:
Hudi: 1.1.1
Flink: 1.20.3
Hive: 3.1.2
Authentication: Kerberos enabled
Problem Description:
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to create
HiveMetaStoreClient
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:112)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.transport.TTransportException
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:279)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:92)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.transport.TTransportException
at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4606)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:92)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:4357)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:4345)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2861)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_301]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_301]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_301]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_301]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:92)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at
org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137)
~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Key Log Errors:
set_ugi() not successful, Likely cause: new client talking to old server.
MetaStoreClient lost connection with TTransportException
Failed to register all functions due to TTransportException
Final error: Failed to create HiveMetaStoreClient → Got runtime exception
when hive syncing
Hudi Table Configuration (relevant sync settings):
sql
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.kerberos.enable' = 'true',
'hive_sync.kerberos.principal' = '...',
'hive_sync.metastore.uris' = 'thrift://100.150.0.23:9083',
'hive_sync.db' = 'jxk',
'hive_sync.table' = 'c_bton_cdc100222223hudi2'
What Works:
Hudi data writes and commits succeed (Commit instant success!)
Compaction scheduling works normally
flink/lib
flink-cep-1.20.3.jar
flink-table-api-java-uber-1.20.3.jar
flink-scala_2.12-1.20.3.jar
flink-connector-files-1.20.3.jar
flink-table-runtime-1.20.3.jar
log4j-core-2.24.3.jar
log4j-slf4j-impl-2.24.3.jar
log4j-1.2-api-2.24.3.jar
flink-json-1.20.3.jar
flink-table-planner-loader-1.20.3.jar
log4j-api-2.24.3.jar
flink-csv-1.20.3.jar
mysql-connector-j-8.0.33.jar
flink-connector-jdbc-3.3.0-1.20.jar
flink-dist-1.20.3.jar
flink-cdc-pipeline-connector-doris-3.5.0.jar
flink-cdc-pipeline-connector-mysql-3.5.0.jar
flink_udf-1.0-SNAPSHOT.jar
flink-cdc-pipeline-connector-postgres-3.5.0.jar
flink-doris-connector-1.20-25.1.0.jar
ojdbc8-21.1.0.0.jar
flink-sql-connector-oracle-cdc-3.5.0.jar
htrace-core-3.1.0-incubating.jar
seatunnel-core-flink.jar
hudi-flink1.20-bundle-1.1.1.jar
guava-27.0.1-jre.jar
hadoop-mapreduce-client-core-3.2.2.jar
calcite-core-1.10.0.jar
What Fails:
Hive metadata synchronization step fails consistently
Hive Metastore connection/client initialization fails with transport
exceptions
Additional Context:
The Hive Metastore server is Kerberos-enabled (principal:
hive/[email protected]). The client appears to connect initially but fails during
subsequent operations like getAllFunctions.
Question:
Is there a known compatibility issue between Hudi 1.1.1 (Flink bundle) and
Hive 3.1.2 with Kerberos authentication, particularly regarding Thrift
transport? Are there specific configuration requirements or workarounds for
this setup?
Let me know if you'd like to include any additional details or adjust the
tone.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
