[
https://issues.apache.org/jira/browse/FLINK-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154177#comment-15154177
]
Till Rohrmann commented on FLINK-3441:
--------------------------------------
Yes the "provided" approach would be the cleanest in my opinion. With SBT you
effectively do it as you've described it with IntelliJ. You have a special
module which sets automatically all provided dependencies to "compile". So in
order to run your job locally you have to choose the classpath of that module.
Concerning possible problems with incompatible classes, I think that shading
away all of Hadoop's dependencies would make it sound.
> Revisit quickstarts exclusion policy
> ------------------------------------
>
> Key: FLINK-3441
> URL: https://issues.apache.org/jira/browse/FLINK-3441
> Project: Flink
> Issue Type: Improvement
> Components: Quickstarts
> Affects Versions: 1.0.0
> Reporter: Till Rohrmann
> Priority: Minor
>
> Flink provides quickstart projects to quickly set up a Flink project. In
> order to decrease the size of the resulting jar, the predefined {{pom.xml}}
> file contains a list of excluded dependencies.
> The problem with this approach is two-fold: First of all, the list seems to
> be rather arbitrary. Why excluding {{org.apache.commons:commons-math}} but
> not {{org.apache.commons:commons-math3}}? It seems as if this list would have
> to be kept up to date with every changing dependency of Flink.
> Secondly and the more severe problem is the following: The exclusions assume
> that Flink always provides a compatible version for an excluded dependency.
> However, since the exclusions are not bound to a specific version, this won't
> necessarily be true. For example, {{snappy-java}} is excluded from the user
> job jar, because Flink with Hadoop {{2.3.0}} comes with
> {{snappy-java:1.0.5}}. However, the {{kafka-clients:0.9.0}} which comes with
> the {{flink-connector-kafka-0.9}} dependency has a transitive dependency on
> {{snappy-java:1.1.1.7}}. Consequently, this dependency will be excluded from
> the user job jar per default. If version {{1.1.1.7}} is not compatible to
> {{snappy-java:1.0.5}}, then the program will crash at runtime. This will be
> very hard to understand for the user because it is a very subtle bug since
> the exclusions also affect the transitive dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
