[
https://issues.apache.org/jira/browse/FLINK-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368677#comment-16368677
]
Ken Krugler commented on FLINK-7477:
------------------------------------
It works (at least when running with YARN via EMR). I believe that's because
the version of Hadoop on the EMR master matches what we're running against; on
my machine, I have to switch between multiple versions of Hadoop for various
(consulting) clients who are on different versions of Hadoop, and my {{hadoop}}
symlink wound up pointing to a different version of Hadoop than what Flink was
using.
Related note - the 1.4 release fixed some shading issues we were running into
with AWS client classes (mostly around {{HttpCore}} stuff), but to get
everything working properly I felt like I did some voodoo with class exclusions
in the {{maven-shade-plugin}} section of my {{pom.xml}}, which still feels
fragile.
> Use "hadoop classpath" to augment classpath when available
> ----------------------------------------------------------
>
> Key: FLINK-7477
> URL: https://issues.apache.org/jira/browse/FLINK-7477
> Project: Flink
> Issue Type: Bug
> Components: Startup Shell Scripts
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Priority: Major
> Fix For: 1.4.0
>
>
> Currently, some cloud environments don't properly put the Hadoop jars into
> {{HADOOP_CLASSPATH}} (or don't set {{HADOOP_CLASSPATH}}) at all. We should
> check in {{config.sh}} if the {{hadoop}} binary is on the path and augment
> our {{INTERNAL_HADOOP_CLASSPATHS}} with the result of {{hadoop classpath}} in
> our scripts.
> This will improve the out-of-box experience of users that otherwise have to
> manually set {{HADOOP_CLASSPATH}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
