xiaozhongcheng created HUDI-4603:
------------------------------------
Summary: Improve HMS Catalog function in flink
Key: HUDI-4603
URL: https://issues.apache.org/jira/browse/HUDI-4603
Project: Apache Hudi
Issue Type: Improvement
Reporter: xiaozhongcheng
Fix For: 0.12.0
1. Make users can choose to sync ro and rt table or not when using hms catalog
in flink.
Currently, if users use hms catalog in flink, ro and rt table will be also sync
to the hms.
But If I just want to sync the metadata of hudi table, I don't want to sync ro
and rt table.
So, I think users should be able to choose to sync the ro and rt table or not.
2. Make users can sync ro and rt table correctly when the table is partitioned
table.
Currently, If users create the partitioned table in hms catalog, but the
partition field is not in the form of yyyy/mm/dd. If they don't make
hive_sync.partition_extractor_class to
org.apache.hudi.hive.HiveStylePartitionValueExtractor, ro and rt table will not
be synced correctly
The stacks is list below:
{code:java}
org.apache.hudi.exception.HoodieException: Got runtime exception when hive
syncing student22
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:326)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:426)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:278)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_241]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_241]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync
partitions for table student22_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
... 10 more
Caused by: java.lang.IllegalArgumentException: Partition path school=beida is
not in the form yyyy/mm/dd
at
org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor.extractPartitionValuesInPath(SlashEncodedDayPartitionValueExtractor.java:58)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at
org.apache.hudi.sync.common.HoodieSyncClient.getPartitionEvents(HoodieSyncClient.java:144)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:318)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
... 10 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
