vinothchandar commented on a change in pull request #4206:
URL: https://github.com/apache/hudi/pull/4206#discussion_r762091656
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -413,6 +413,58 @@ public void testVirtualKeysInBaseFiles(boolean
populateMetaFields) throws Except
});
}
+
+ /**
+ * Tests that virtual key configs are honored in base files after compaction
in metadata table.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testMetadataTableWithPendingCompaction() throws Exception {
+ HoodieTableType tableType = COPY_ON_WRITE;
+ init(tableType, false);
+ writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .enableFullScan(true)
+ .enableMetrics(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(3)
+ .build()).build();
+ initWriteConfigAndMetatableWriter(writeConfig, true);
+
+ doWriteOperation(testTable, "0000001", INSERT);
+ // create an inflight compaction in metadata table.
+ // not easy to create an inflight in metadata table directly, hence
letting compaction succeed and then deleting the completed instant.
+ // this new write is expected to trigger metadata table compaction
+ String commitInstant = "0000002";
+ doWriteOperation(testTable, commitInstant, INSERT);
+
+ HoodieTableMetadata tableMetadata = metadata(writeConfig, context);
+ String metadataCompactionInstant = commitInstant + "001";
+ assertTrue(tableMetadata.getLatestCompactionTime().isPresent());
+ assertEquals(tableMetadata.getLatestCompactionTime().get(),
metadataCompactionInstant);
+
+ validateMetadata(testTable);
+ // Fetch compaction Commit file and rename to some other file. completed
compaction meta file should have some serialized info that table interprets
+ // for future upserts. so, renaming the file here to some temp name and
later renaming it back to same name.
+ java.nio.file.Path parentPath = Paths.get(metadataTableBasePath,
HoodieTableMetaClient.METAFOLDER_NAME);
Review comment:
dont we have utils for some of these?
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
##########
@@ -352,7 +352,7 @@ public static int mapRecordKeyToFileGroupIndex(String
recordKey, int numFileGrou
}
HoodieTableFileSystemView fsView = new
HoodieTableFileSystemView(metaClient, timeline);
- return fsView.getLatestFileSlices(partition).sorted((s1, s2) ->
s1.getFileId().compareTo(s2.getFileId()))
+ return fsView.getLatestUnCompactedFileSlices(partition).sorted((s1, s2) ->
s1.getFileId().compareTo(s2.getFileId()))
Review comment:
With metadata table we begin without any base files. Can this handle the
fact that there is no previous file slice during compaction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
