prashantwason opened a new pull request, #18181: URL: https://github.com/apache/hudi/pull/18181
### Describe the issue this Pull Request addresses Closes #18180 When the write config does not have correct enable flags for MDT (Metadata Table) partitions, they are automatically deleted. This does not work well for production datasets with multiple writers where accidental config changes can cause large MDT indexes to be deleted. ### Summary and Changelog Added a new configuration `hoodie.metadata.auto.delete.partitions` that controls whether metadata table partitions can be automatically deleted when the corresponding config flags are disabled in the write config. **Changes:** - Added `hoodie.metadata.auto.delete.partitions` config in `HoodieMetadataConfig.java` (default: `true`) - Added `isAutoDeleteMdtPartitionsEnabled()` getter in `HoodieWriteConfig.java` - Modified `maybeDeleteMetadataTable()` in `HoodieTable.java` to check the config before deleting - Modified `deleteMetadataIndexIfNecessary()` in `HoodieTable.java` to check the config before deleting - Updated `HoodieSparkTable.java` to check `isMetadataTableAvailable()` in addition to `isMetadataTableEnabled()` for metadata writer initialization - Replaced `storage.exists()` check with `metadataWriter.isInitialized()` for better reliability ### Impact - New config: `hoodie.metadata.auto.delete.partitions` (default: `true`) - No breaking changes - default behavior is unchanged (auto-delete enabled) - Production users can set to `false` to prevent accidental deletion of metadata indexes ### Risk Level low - This is an additive change with a feature flag. Default behavior remains unchanged. ### Documentation Update Config description is included in the code. The new config `hoodie.metadata.auto.delete.partitions`: - Default: `true` - When enabled (default), metadata table partitions that are disabled in the write config will be automatically deleted - When disabled, users must explicitly drop metadata partitions using hudi-cli or DROP INDEX command - Recommended to disable for production datasets with multiple writers ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
