rangareddy commented on issue #19967: URL: https://github.com/apache/hudi/issues/19967#issuecomment-5694773837
Confirmed, and thanks for offering to pick it up. The diagnosis holds on master: - `GLUE_CATALOG_ID` is `GLUE_CLIENT_PROPERTY_PREFIX + "catalogId"` ([GlueCatalogSyncClientConfig.java#L108](https://github.com/apache/hudi/blob/master/hudi-aws/src/main/java/org/apache/hudi/config/GlueCatalogSyncClientConfig.java#L108)), the only camelCase key among the twelve in that class; every sibling is snake_case (`database_name`, `skip_table_archive`, ...). - Lookup is an exact `props.containsKey(...)` for the key **and** for each alternative ([`ConfigUtils.getRawValueWithAltKeys`](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/util/ConfigUtils.java#L406-L418)), so nothing normalises case. - It is read once, as `config.getStringOrDefault(GLUE_CATALOG_ID, identityResponse.account())` ([AWSGlueCatalogSyncClient.java#L183](https://github.com/apache/hudi/blob/master/hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java#L183)). That default is why it is silent rather than loud: a miss falls back to the STS caller identity account, which is a valid catalog id, so the sync succeeds against the wrong account. One correction on the plan, though, because as described it would not fix the case you reported. Renaming the key to `catalog_id` and putting `catalogId` in `withAlternatives` still leaves your repro broken: Spark hands Hudi the key already lower-cased, so the props hold `...glue.catalogid`, which exact-matches neither `catalog_id` nor `catalogId`. The alternative that actually has to be there is the all-lowercase form: ```java public static final ConfigProperty<String> GLUE_CATALOG_ID = ConfigProperty .key(GLUE_CLIENT_PROPERTY_PREFIX + "catalog_id") .noDefaultValue() .sinceVersion("1.1.0") .markAdvanced() .withAlternatives( GLUE_CLIENT_PROPERTY_PREFIX + "catalogId", // set exactly, e.g. a Streamer props file or Flink GLUE_CLIENT_PROPERTY_PREFIX + "catalogid") // what Spark .option()/.options() actually delivers .withDocumentation("..."); ``` Keeping both alternatives matters: the camelCase one covers callers that pass properties through verbatim and are working today, and the lowercase one is the actual reported break. Alternatives also emit a deprecation warning on hit, so users get nudged toward the new key instead of silently drifting. Worth a test that pins the lowercase path specifically, since that is the one with no coverage today: build a `HiveSyncConfig` whose props hold only `hoodie.datasource.meta.sync.glue.catalogid` and assert `getStringOrDefault(GLUE_CATALOG_ID, "fallback")` returns the configured id rather than the fallback. `TestAWSGlueSyncClient` in `hudi-aws` is the natural home. Please go ahead and send the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
