[jira] [Created] (HUDI-4181) Async indexer fails for simple index creation

[sivabalan narayanan (Jira)]

sivabalan narayanan created HUDI-4181:
-----------------------------------------

             Summary: Async indexer fails for simple index creation
                 Key: HUDI-4181
                 URL: https://issues.apache.org/jira/browse/HUDI-4181
             Project: Apache Hudi
          Issue Type: Bug
          Components: metadata
            Reporter: sivabalan narayanan


I tried to ran async indexing to build out col stats and ran into issues. Had 
to make following fix to make progress. but this needs proper fix. 

 

Local fix that worked for me.
{code:java}
diff --git 
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
 
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
index f5a96fb676..1e67020810 100644
--- 
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
+++ 
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
@@ -955,7 +955,6 @@ public abstract class HoodieBackedTableMetadataWriter 
implements HoodieTableMeta
     HoodieTableFileSystemView fsView = 
HoodieTableMetadataUtil.getFileSystemView(metadataMetaClient);
     for (Map.Entry<MetadataPartitionType, HoodieData<HoodieRecord>> entry : 
partitionRecordsMap.entrySet()) {
       final String partitionName = entry.getKey().getPartitionPath();
-      final int fileGroupCount = entry.getKey().getFileGroupCount();
       HoodieData<HoodieRecord> records = entry.getValue();
 
       List<FileSlice> fileSlices =
@@ -965,9 +964,10 @@ public abstract class HoodieBackedTableMetadataWriter 
implements HoodieTableMeta
         // so if there are no committed file slices, look for inflight slices
         fileSlices = 
HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient,
 Option.ofNullable(fsView), partitionName);
       }
-      ValidationUtils.checkArgument(fileSlices.size() == fileGroupCount,
+      final int fileGroupCount = fileSlices.size();
+      /*ValidationUtils.checkArgument(fileSlices.size() == fileGroupCount,
           String.format("Invalid number of file groups for partition:%s, 
found=%d, required=%d",
-              partitionName, fileSlices.size(), fileGroupCount));
+              partitionName, fileSlices.size(), fileGroupCount));*/
 
       List<FileSlice> finalFileSlices = fileSlices;
       HoodieData<HoodieRecord> rddSinglePartitionRecords = records.map(r -> { 
{code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)