[
https://issues.apache.org/jira/browse/HUDI-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaozhongcheng closed HUDI-4603.
--------------------------------
Fix Version/s: (was: 0.13.0)
Resolution: Fixed
> Improve HMS Catalog function in flink
> -------------------------------------
>
> Key: HUDI-4603
> URL: https://issues.apache.org/jira/browse/HUDI-4603
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: xiaozhongcheng
> Priority: Major
> Labels: pull-request-available
>
> 1. Make users can choose to sync ro and rt table or not when using hms
> catalog in flink.
> Currently, if users use hms catalog in flink, ro and rt table will be also
> sync to the hms.
> But If I just want to sync the metadata of hudi table, I don't want to sync
> ro and rt table.
> So, I think users should be able to choose to sync the ro and rt table or not.
>
> 2. Make users can sync ro and rt table correctly when the table is
> partitioned table.
> Currently, If users create the partitioned table in hms catalog, but the
> partition field is not in the form of yyyy/mm/dd. If they don't make
> hive_sync.partition_extractor_class to
> org.apache.hudi.hive.HiveStylePartitionValueExtractor, ro and rt table will
> not be synced correctly
> The stacks is list below:
>
> {code:java}
> org.apache.hudi.exception.HoodieException: Got runtime exception when hive
> syncing student22
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:326)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:426)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:278)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_241]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_241]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_241]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
> Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync
> partitions for table student22_ro
> at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Partition path school=beida is
> not in the form yyyy/mm/dd
> at
> org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor.extractPartitionValuesInPath(SlashEncodedDayPartitionValueExtractor.java:58)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at
> org.apache.hudi.sync.common.HoodieSyncClient.getPartitionEvents(HoodieSyncClient.java:144)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:318)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
> ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
> ... 10 more {code}
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
