sivabalan narayanan created HUDI-5464:
-----------------------------------------
Summary: Fix instantiation of a new partition in MDT re-using the
same instant time as a regular commit
Key: HUDI-5464
URL: https://issues.apache.org/jira/browse/HUDI-5464
Project: Apache Hudi
Issue Type: Bug
Components: metadata
Reporter: sivabalan narayanan
we re-use the same instant time as the commit being applied to MDT while
instantiating a new partition in MDT. this needs to be fixed.
for eg, lets say we have 10 commits w/ already FILES enabled.
for C11, we are enabling col-stats.
after data table business, when we enter metadata writer instantiation, we
deduct that col-stats has to be instantiated and then instantiate using DC11.
and then we go ahead and apply actual C11 from DT to MDT. here, we overwrite
the same DC11 w/ records pertaining to C11.
which is buggy. we definitely need to fix this.
We can add a suffix to C11 (say C11_003 or C11_001) as we do for compaction and
clean in MDT so that any additional operation in MDT has a diff commit time
format. For everything else, it should match w/ DT 1 on 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
