JingsongLi commented on pull request #1558:
URL: https://github.com/apache/iceberg/pull/1558#issuecomment-707444552
> > From hiveConfDir, it should be URI which contains scheme and full path.
This means it can be loaded by Hadoop Filesystem.
>
> Can you help me understand Flink's behavior a bit more?
>
> * If `hiveConfDir` is a Flink option, then why doesn't Flink automatically
load the `hive-site.xml` file when it creates a `Configuration`? It doesn't
seem like this should be delegated to libraries that run inside Flink. I would
expect Flink to create the `Configuration` based on application options and
pass that to libraries.
> * If Flink allows the user to set a separate `hiveConfDir`, why is that
not just add that directory to the classpath so that `hive-site.xml` can be
loaded like normal? That's what happens in most Hadoop setup scripts: extra
config folders are added to the classpath so that the regular method of loading
config files works.
Why not automatically load the `hive-site.xml` file when it creates a
`Configuration`?
Flink just want to get Hadoop Configuration for connecting Yarn and reading
HDFS.
The hive connector is only an optional connector of Flink, it is not
built-in, so the read of built-in Hadoop config will not process hive
configuration (hive-site.xml). And the `hiveConfDir` is in Hive connector, it
should be called explicitly.
Why is that not just add that directory to the classpath so that
`hive-site.xml` can be loaded like normal?
Now, the only way is system env, just like:
```
String hadoopHome = System.getenv("HADOOP_HOME");
if (hadoopHome != null) {
loadConfs(hadoopHome + "/conf");
loadConfs(hadoopHome + "/etc/hadoop");
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
