[
https://issues.apache.org/jira/browse/HIVE-29736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-29736 started by Raghav Aggarwal.
----------------------------------------------
> MSCK REPAIR TABLE throws "partition already exists" exception when Metastore
> URI contains an explicit port and HDFS uses HA Nameservice
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29736
> URL: https://issues.apache.org/jira/browse/HIVE-29736
> Project: Hive
> Issue Type: Bug
> Reporter: Raghav Aggarwal
> Assignee: Raghav Aggarwal
> Priority: Major
> Labels: pull-request-available
>
> It fixes an edge case in {{HiveMetaStoreChecker}} where{{ MSCK REPAIR
> TABLE}} crashes with a {{MetastoreException}} when comparing partitions on an
> HDFS HA cluster. When a partition's Metastore URI contains an explicit port
> (e.g., :8020 ) but the physical Hadoop FileSystem returns paths normalized to
> the HA nameservice.
> The PR updates the validation loop to ignore these leftover physical
> directories if they perfectly map to a partition that was already
> successfully matched against the Metastore database.
> For example:
> Desc formatted tbl -> hdfs://namenode/
> Desc formatted tbl partition.. -> hdfs://namenode:8020/;
> the _{{checkTable}}_ will detemine the diff of HDFS vs HMS lookup and it
> compares complete URI and picks the already existing / register partition as
> a candidate for msck operation. Causing the following stacktrace:
> Stacktrace:
> {code:java}
> org.apache.hadoop.hive.metastore.api.MetastoreException: The partition
> 'dummy_tbl_name:xxxx=xxxx/xxxxxxx=xxxxxx/dt=2026-06-07' already exists for
> table aip_auditlog at
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:468)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:401)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:287)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:162)
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1812) at
> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:411) at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
