kbendick commented on issue #3044: URL: https://github.com/apache/iceberg/issues/3044#issuecomment-908635309
There's an open PR for upgrading to Flink 1.13. I left some comments there, but my larger concern is not being able to access additional catalog options. I've left some comments on that PR: https://github.com/apache/iceberg/pull/2629 > One possible solution is to add glue as one of the known catalog-type's and avoid calling clusterHadoopConf() when the user sets this value (I tested this change with a local build and it worked in a non-hadoop environment). At present, I think adding to the known `catalog-type`s might be the best path forward to resolve the issue more immediately. I asked on the upgrade to Flink 1.13 PR, but it's possible that generically allowing `*` as an additional catalog option might not be possible. In which case, we'll definitely need to add `glue` etc (or at least all of their possible options). > I believe that one potential root issue is that FileIO has leaked out from TableOperations into catalog implementations like GlueCatalog. Theoretically, provided a catalog extending BaseMetastoreCatalog can implement newTableOps then all the FileIO configuration should happen inside TableOperations. In the longer term, if these larger API changes to the interactions between `FileIO` and `TableOperations` and `Catalog` are undertaken / considered, we can remove the added `catalog-type`s. But this is possibly a larger API change that will probably need to be considered a bit more closely. In the near-term, I think it's most realistic to add the catalogs we support here in the repo to `catalog-type` to address the immediate issue. They can be removed later on if they're not needed / if the API changes. Plus, the additional configs (even for S3FileIO / AWS Authentication) might need to be added as predeclared to the `FlinkCatalogFactoryOptions` class as part of the 1.13 upgrade. If somebody can provide a workaround for KDA, then the issue wouldn't necessarily be as pressing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
