[
https://issues.apache.org/jira/browse/HIVE-28757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098165#comment-18098165
]
Feng Liu edited comment on HIVE-28757 at 7/22/26 12:23 PM:
-----------------------------------------------------------
I reproduced this on our Hive 3.1.2-based branch. In the dynamic-partition
path, Hive.loadPartition calls add_partition but does not keep the Partition
returned by HMS. It continues with the request object, whose createTime is
still 0. A later StatsTask can send that object back through alter_partition,
overwriting the createTime stored in HMS.
The same pattern is present in current master in Hive.addPartitionToMetastore.
Master also has a batch partition-add path, which needs separate coverage.
Our 3.1.2 patch keeps the Partition returned by HMS. It also preserves the
existing createTime in alterPartition/alterPartitions when the incoming value
is not set, so another client path cannot erase it. We added regression tests
on both the client and metastore sides, and a clean ql compile passes.
I am adapting the change to apache/master now. I will post the master PR here,
and keep the 3.1.2 patch separately for backport/reference.
was (Author: feng liu):
We investigated this issue on a production-derived Hive 3.1.2 fork and
confirmed the root cause.
Root cause
The metastore initializes server-owned fields, including createTime, when a new
partition is added. However, the Hive client path in Hive.loadPartition ignores
the Partition returned by add_partition and keeps using the original
client-side object whose createTime is 0. If a following StatsTask alters that
partition, the stale client object is sent back to the metastore and overwrites
the persisted createTime with 0.
The same pattern is still present on the current master branch:
addPartitionToMetastore calls getMSC().add_partition(...) without retaining the
returned partition. The current master also has a batch dynamic-partition path,
so both single and batch creation flows need to be covered.
Proposed fix
1. Preserve the Partition returned by the metastore after a successful add
operation and continue with that server-initialized object.
2. Add a defensive metastore-side check in alterPartition and alterPartitions
so an unset or non-positive incoming createTime cannot overwrite an existing
valid value.
3. Cover both single-partition and batch dynamic-partition creation paths on
master.
Validation on Hive 3.1.2
We implemented the fix on Hive 3.1.2 and added client-side and metastore
regression tests. A clean ql compilation also passed.
We are preparing two clean patches: one adapted to the latest Apache Hive
master, and one based on the official Hive 3.1.2 source for reference/backport.
The upstream patch will be created from apache/master and will not include any
downstream-specific changes.
> The inserted dynamic partitions loss createTime
> -----------------------------------------------
>
> Key: HIVE-28757
> URL: https://issues.apache.org/jira/browse/HIVE-28757
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.1.3
> Reporter: FangBO
> Priority: Minor
> Attachments: image.png
>
>
> The inserted dynamic partitions loss createTime。
>
> {code:java}
> // code placeholder
> create table p (id bigint) partitioned by (ds string);
> create table src (id bigint, ds string);
> insert into src values(1, '00');
> insert overwrite table p partition(ds) select id,ds from src;{code}
> After the operation, the inserted partition does not have createTime.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
