nsivabalan commented on code in PR #13642:
URL: https://github.com/apache/hudi/pull/13642#discussion_r2254721043
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java:
##########
@@ -254,6 +293,20 @@ private static Stream<Arguments>
upgradeDowngradeVersionPairs() {
);
}
+ /**
+ * Version pairs that should be tested for metadata upgrade/downgrade
failures.
+ * Excludes pairs that trigger rollback operations which delete metadata
tables.
+ */
+ private static Stream<Arguments> metadataTableCorruptionTestVersionPairs() {
+ return Stream.of(
+ Arguments.of(HoodieTableVersion.FOUR, HoodieTableVersion.FIVE), //
V4 -> V5
+ Arguments.of(HoodieTableVersion.FIVE, HoodieTableVersion.FOUR) //
V5 -> V4
+ // Note: Other pairs like V5->V6, V6->V8 are excluded because they may
trigger
+ // rollback operations in rollbackFailedWritesAndCompact() that disable
metadata
Review Comment:
why would it trigger deletion of MDT?
Our test set up should be such that.
testing 4 to 5:
create data table and mdt in v4. corrupt mdt. and try upgrade.
testing 6 to 8:
create data table and mdt in v8. corrupt mdt. try upgrade.
given this test structure, why would mdt be deleted.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java:
##########
@@ -153,6 +156,42 @@ public void testAutoUpgradeDisabled(HoodieTableVersion
originalVersion) throws E
LOG.info("Auto-upgrade disabled test passed for version {}",
originalVersion);
}
+ @ParameterizedTest
+ @MethodSource("metadataTableCorruptionTestVersionPairs")
+ public void testMetadataTableUpgradeDowngradeFailure(HoodieTableVersion
fromVersion, HoodieTableVersion toVersion) throws Exception {
+ boolean isUpgrade = fromVersion.lesserThan(toVersion);
+ String operation = isUpgrade ? "upgrade" : "downgrade";
+ LOG.info("Testing metadata table failure during {} from version {} to {}",
operation, fromVersion, toVersion);
+
+ HoodieTableMetaClient originalMetaClient = loadFixtureTable(fromVersion);
+ assertEquals(fromVersion,
originalMetaClient.getTableConfig().getTableVersion(),
+ "Fixture table should be at expected version");
+
+ HoodieWriteConfig cfg = createWriteConfig(originalMetaClient, true);
+
+ String metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(
+ originalMetaClient.getBasePath().toString());
+ StoragePath metadataHoodiePath = new StoragePath(metadataTablePath,
HoodieTableMetaClient.METAFOLDER_NAME);
+ StoragePath propsPath = new StoragePath(metadataHoodiePath,
HoodieTableConfig.HOODIE_PROPERTIES_FILE);
+ StoragePath backupPropsPath = new StoragePath(metadataHoodiePath,
HoodieTableConfig.HOODIE_PROPERTIES_FILE_BACKUP);
+
+ String corruptedContent =
"CORRUPTED_INVALID_CONTENT\n\nTHIS_IS_NOT_VALID_PROPERTIES_FORMAT";
+ try (OutputStream propsOut =
originalMetaClient.getStorage().create(propsPath, true);
+ OutputStream backupOut =
originalMetaClient.getStorage().create(backupPropsPath, true)) {
+ propsOut.write(corruptedContent.getBytes());
+ backupOut.write(corruptedContent.getBytes());
+ }
+
+ HoodieUpgradeDowngradeException exception = assertThrows(
+ HoodieUpgradeDowngradeException.class,
+ () -> new UpgradeDowngrade(originalMetaClient, cfg, context(),
SparkUpgradeDowngradeHelper.getInstance())
+ .run(toVersion, null)
+ );
+
+ LOG.info("Successfully caught expected exception during {} from {} to {}:
{}",
+ operation, fromVersion, toVersion, exception.getMessage());
Review Comment:
there is a way to assert the exception message as well.
you can find examples in other tests in this repo.
can we do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
