danny0405 commented on code in PR #18295:
URL: https://github.com/apache/hudi/pull/18295#discussion_r2969022195
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -350,6 +388,26 @@ public static HoodieWriteConfig createMetadataWriteConfig(
}
HoodieWriteConfig metadataWriteConfig = builder.build();
+ if (deriveMetadataLockConfigsFromDataTableConfigs) {
+ // We need to update the MDT write config to have the same lock related
configs as the data table.
+ // All data table props with the lock prefix are always copied (to
override MDT defaults with
+ // user-configured values). Other data table props not present in MDT
config are also copied to
+ // support custom lock providers that may use non-standard config keys.
+ Properties lockProps = new Properties();
+ TypedProperties dataTableProps = writeConfig.getProps();
+ TypedProperties mdtProps = metadataWriteConfig.getProps();
+ for (String key : dataTableProps.stringPropertyNames()) {
Review Comment:
> is that we need to "inherit" the lock configs from data table. But we
can't directly add all the props from data table
+1 to this part, I see that the props we want to set up into the MDT config
is as following:
- key.startsWith(LockConfiguration.LOCK_PREFIX)
- HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key()
- HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key()
we can prepare this props in one shot from data table write config, and set
it in the mdt builder.
Two concens here:
1. it seems we want to include all the options that are contained in the
data table config but not in the metadata table config with check
`!mdtProps.containsKey(key)`, are there any risk that wong config are leaked
from DT into MDT and cause unexpected behaviors? looks dangerous though.
2. get the props from HoodieWriteConfig and rebuild with the props generally
works but there is risk for properties lost, we have some POJO configs within
the `HoodieWriteConfig` that instantiated within the constructor of the
`HoodieWriteConfig`, those config could infer options that are not included in
the `HoodieWriteConfig` props.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
