jarredhj0214 commented on code in PR #12121:
URL: https://github.com/apache/gravitino/pull/12121#discussion_r3629075920
##########
flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalog.java:
##########
@@ -129,43 +134,75 @@ public void open() throws CatalogException {
super.open();
return;
}
+ Map<String, String> paimonOptions = new HashMap<>(mutableOptions);
+ Configuration hadoopConf =
+
HadoopUtils.getHadoopConfiguration(Options.fromMap(withoutHadoopOptions(paimonOptions)));
+ moveHadoopOptionsToConf(paimonOptions, hadoopConf);
try {
CredentialPropertyUtils.applyPaimonCredentials(
- CredentialPropertyUtils.getCredentials(catalog()), mutableOptions);
Review Comment:
> `CredentialPropertyUtils.applyPaimonCredentials` may put credentials to
mutableOptions.
>
> We need to confirm one thing: should these properties be removed from
`paimonOptions`?
Good point. These two sets of properties are intentionally handled
differently.
The Hadoop filesystem options from catalog properties (`hadoop.*`, `fs.*`,
and `dfs.*`) are removed from `paimonOptions` and moved into Hadoop
`Configuration`.
The credentials injected by `CredentialPropertyUtils.applyPaimonCredentials`
are vended credentials, and they need to stay in `paimonOptions` because Paimon
native S3/OSS FileIO loaders declare these keys as required options. If we
remove them from `paimonOptions`, native Paimon S3/OSS FileIO may fail to
initialize.
I added a comment to make this distinction explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
