lokeshj1703 commented on code in PR #19717:
URL: https://github.com/apache/hudi/pull/19717#discussion_r3872380105
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieIndexer.java:
##########
@@ -184,6 +188,49 @@ public void testIndexerForRecordIndex() {
/**
* Test indexer for RLI and secondary index.
*/
+ /**
+ * An indexing action builds the partition it names. With two
secondary-index definitions registered and neither
+ * initialized (the shape of a table whose metadata table was rebuilt with
its definitions intact), the action for
+ * one of them builds exactly that one, on storage and not only in the table
config.
+ */
+ @Test
+ void
testIndexerBuildsTheRequestedSecondaryIndexWhenSeveralAreUninitialized() {
Review Comment:
This covers one secondary index among several uninitialized - could we also
add the expression-index analog, and a case where the indexer creates multiple
secondary indexes and multiple expression indexes on the same table? That is
the shape a rebuilt metadata table with several definitions intact produces,
and it is really what `resolvePartitionsToInit` is for. The indexer-level
`TestSecondaryIndexer` / `TestExpressionIndexer` build the context with the
5-arg `of` (empty request), so the requested-partition branch - including the
ambiguous-request failure and the already-initialized skip - is otherwise
unexercised.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -435,8 +453,16 @@ private void initializeMetadataPartition(
Indexer indexer,
String dataTableInstantTime,
Map<String, List<FileInfo>> partitionToAllFilesMap,
- Lazy<List<FileSliceAndPartition>> lazyMergedFileSlices) throws
IOException {
- String instantTimeForPartition =
generateUniqueInstantTime(dataTableInstantTime);
+ Lazy<List<FileSliceAndPartition>> lazyMergedFileSlices,
+ Option<String> requestedIndexPartition) throws IOException {
+ // A requested partition initializes under a fresh solo-family instant,
never the indexing
+ // action's own instant. The action's completion applies its data commit
to the metadata table
+ // too, and finding that instant already completed there is treated as a
partially applied
+ // earlier commit: it is rolled back and re-applied, destroying the
initialization records
+ // while leaving the file groups. The solo family is the established shape
for
+ // metadata-table-only bootstrap commits and survives that reconciliation.
+ String instantTimeForPartition = requestedIndexPartition.isPresent()
+ ? generateUniqueSoloInstantTime() :
generateUniqueInstantTime(dataTableInstantTime);
Review Comment:
The solo-family instant is chosen only for a requested SECONDARY_INDEX /
EXPRESSION_INDEX partition here - every other type built by an async indexing
action still goes through `generateUniqueInstantTime`, which returns the
action's own instant for an indexing commit. If the record-wiping rollback
comes from committing the init under the action's instant and then re-applying
that instant on completion, wouldn't RECORD_INDEX / COLUMN_STATS /
BLOOM_FILTERS / PARTITION_STATS initialized via the indexer hit the same path?
Could you clarify why those stay safe, or whether the solo instant should cover
them too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
