nsivabalan commented on a change in pull request #4206:
URL: https://github.com/apache/hudi/pull/4206#discussion_r762186699
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -413,6 +413,58 @@ public void testVirtualKeysInBaseFiles(boolean
populateMetaFields) throws Except
});
}
+
+ /**
+ * Tests that virtual key configs are honored in base files after compaction
in metadata table.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testMetadataTableWithPendingCompaction() throws Exception {
+ HoodieTableType tableType = COPY_ON_WRITE;
+ init(tableType, false);
+ writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .enableFullScan(true)
+ .enableMetrics(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(3)
+ .build()).build();
+ initWriteConfigAndMetatableWriter(writeConfig, true);
+
+ doWriteOperation(testTable, "0000001", INSERT);
+ // create an inflight compaction in metadata table.
+ // not easy to create an inflight in metadata table directly, hence
letting compaction succeed and then deleting the completed instant.
+ // this new write is expected to trigger metadata table compaction
+ String commitInstant = "0000002";
+ doWriteOperation(testTable, commitInstant, INSERT);
+
+ HoodieTableMetadata tableMetadata = metadata(writeConfig, context);
+ String metadataCompactionInstant = commitInstant + "001";
+ assertTrue(tableMetadata.getLatestCompactionTime().isPresent());
+ assertEquals(tableMetadata.getLatestCompactionTime().get(),
metadataCompactionInstant);
+
+ validateMetadata(testTable);
+ // Fetch compaction Commit file and rename to some other file. completed
compaction meta file should have some serialized info that table interprets
+ // for future upserts. so, renaming the file here to some temp name and
later renaming it back to same name.
+ java.nio.file.Path parentPath = Paths.get(metadataTableBasePath,
HoodieTableMetaClient.METAFOLDER_NAME);
Review comment:
yes. we have utils to delete the meta file fully. but not to move it
around. also, I did not see this being widely used in other tests, and so kept
it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
