rdblue commented on pull request #1558:
URL: https://github.com/apache/iceberg/pull/1558#issuecomment-707908716
Okay, I understand why Flink won't load `hive-site.xml`: because Flink
doesn't have direct Hive integration and that is the application's
responsibility. Then, it also makes sense that the user might be creating a
catalog using SQL, and would have no ability to set up the Hive config directly.
What I don't understand is why Iceberg should do anything other than ensure
that `hive-site.xml` is added as a default resource from the classpath.
Most configuration for the `HiveCatalog` should come from catalog options,
not from `hive-site.xml`. That config file would be primarily used for global
defaults, so it makes more sense to me for it to be a default resource that is
shared across all catalogs. If it is shared, then there is no need to load a
path that is specific to each catalog. Would it be valuable to support multiple
`hive-site.xml` configs, one per catalog?
If Hive only tuning defaults are controlled by `hive-site.xml`, then why not
load it the normal way from the application classpath? That gives the user a
way to add folders (by adding to the classpath) or to bundle the config file
into their application Jar. When an application's Jars are being localized, the
config file could be handled just like other classpath artifacts. That's the
normal way of distributing config files.
In summary, unless we need catalog-specific Hive configs, I don't see why we
wouldn't use the "normal" ways to distribute and add a `hive-site.xml` file. It
doesn't seem like something we need Iceberg to handle, beyond ensuring that the
`hive-site.xml` from the classpath is added as a default resource. Is there a
use case that I'm misunderstanding?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
