[
https://issues.apache.org/jira/browse/FLINK-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233983#comment-16233983
]
ASF GitHub Bot commented on FLINK-7944:
---------------------------------------
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4920
You're right, I added a longer description in the commit message. This is
the message:
```
Flink comes with bundled Hadoop dependencies in
lib/flink-shaded-hadoop2-uber-*.jar. Flink can also use the Hadoop jars
that are available on the system (from the distro, EMR, GCE, ...) and
Flink can run completely without Hadoop dependencies. Switching between
these comes down to either adding or removing the bundled jar from lib/
and/or telling Flink to use the system Hadoop classpaths or not.
This change introduces a new possible configuration file in
"bin/hadoop-classpath". If present, this should contain a classpath
definition. When the file is present Flink will append this to the
internal classpaths.
The new configuration script allows easily switching the different modes:
- use-system-hadoop: Try and determine the system Hadoop classpaths and
store them in bin/hadoop-classpath. The file can be tweaked manually
afterwards. Move the bundled Hadoop jar from lib/ to opt/ if present so
that Flink doesn't use it.
- use-bundled-hadoop: Remove the bin/hadoop-classpath configuration
file. Move the bundled Hadoop jar from opt/ to lib/ if it is not
already in lib/
- no-hadoop: Remove the bin/hadoop-classpath configuration and move the
bundled Hadoop jar from lib/ to opt/.
Flink will still add this to the classpath in config.sh:
"${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${YARN_CONF_DIR}:${HBASE_CONF_DIR}"
i.e. env variables on the system are still being picked up, no matter
the configuration in bin/hadoop-classpath.
```
That last part is interesting. We've always done it but now I'm wondering
whether we should. (By default `HADOOP_CLASSPATH` is empty on AWS and GCE.
> Allow configuring Hadoop Classpath
> ----------------------------------
>
> Key: FLINK-7944
> URL: https://issues.apache.org/jira/browse/FLINK-7944
> Project: Flink
> Issue Type: Improvement
> Components: Startup Shell Scripts
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Priority: Major
> Fix For: 1.4.0
>
>
> Currently, we have some paths hardcoded in {{config.sh}} in addition to some
> magic that tries to derive a classpath from the {{hadoop}}/{{hbase}}
> commands. We should make the classpath configurable using a separate script
> and put the classpath in a {{classpath}} file that will be picked up by the
> scripts.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
