[
https://issues.apache.org/jira/browse/HUDI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udit Mehrotra updated HUDI-5601:
--------------------------------
Description:
*SHOW PARTITIONS* in Hudi is integrated with Hudi metadata table instead of
with Hive catalog. Thus, adding partition using *ALTER TABLE ADD PARTITION* or
doing *MSCK REPAIR* does not result in partitions being reflected when users
run *SHOW PARTITIONS* later. Additionally, *ALTER TABLE ADD PARTITION* adds the
partition in the table catalog metadata using Hive style partitioning which
would not work well with Hudi by default. Thus, to support these we probably
need to override the implementation of these commands specifically for Hudi
like how we have done for other commands.
was:
Currently it seems that when a user tries to add a partition to their table
using `spark.sql("alter table hudi_table add partition(year='2021')")` and then
run a `show partition`, nothing is returned. When running the command
`spark.sql("msck repair table hudi_table").show(20)` nothing is returned as
well.
I believe in hudi we have not integrated this with the external catalog (hive
metastore), and thus we should look into this integration.
scala> spark.sql("SHOW PARTITIONS hudi_table ").show(20)
+---------+ |partition|
+--------{-}{-}+ +--------+
> Support for commands ALTER TABLE ADD PARTITION, and MSCK REPAIR
> ---------------------------------------------------------------
>
> Key: HUDI-5601
> URL: https://issues.apache.org/jira/browse/HUDI-5601
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Rahil Chertara
> Priority: Major
>
> *SHOW PARTITIONS* in Hudi is integrated with Hudi metadata table instead of
> with Hive catalog. Thus, adding partition using *ALTER TABLE ADD PARTITION*
> or doing *MSCK REPAIR* does not result in partitions being reflected when
> users run *SHOW PARTITIONS* later. Additionally, *ALTER TABLE ADD PARTITION*
> adds the partition in the table catalog metadata using Hive style
> partitioning which would not work well with Hudi by default. Thus, to support
> these we probably need to override the implementation of these commands
> specifically for Hudi like how we have done for other commands.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
