[
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007041#comment-16007041
]
ASF GitHub Bot commented on FLINK-6514:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/3876
[FLINK-6514] [build] Create a proper separate Hadoop uber jar for
'flink-dist' assembly
This fixes the issue that Flink cannot be started locally if built with
Maven 3.3+
There are two big fixes in this pull request, because they do not
build/pass tests individually. The wrong Mesos dependencies where the reason
that the broken Hadoop fat jar building actually passed the Yarn tests.
# Hadoop Uber Jar
- This builds a proper Hadoop Uber Jar with all of Hadoop's needed
dependencies. The prior build was missing many important dependencies in the
Hadoop Uber Jar.
- The Hadoop-jar is no longer excluded in `flink-dist` via setting the
dependency to `provided`, but by explicit exclusion. That way, Hadoop's
transitive dependencies are not excluded from other dependencies as well.
Before this patch, various decompression and Avro were broken in a Flink build,
due to accidental exclusion of their transitive dependencies.
# Dependency fixing
- This also fixes the dependencies of `flink-mesos`, which made all of
Hadoop's transitive dependencies its own dependencies, by promoting them during
shading. That way, Flink had various unnecessary dependencies in its
`flink-dist` jar.
- Incidentally, that brought Hadoop's accidentally excluded dependencies
back in, but into the `flink-dist` jar, not the `shaded-hadoop2` jar.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink fix_fat_jar
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3876.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3876
----
commit 86803d23dfdc56f2be64274a6a90a76c1e782f08
Author: Stephan Ewen <[email protected]>
Date: 2017-05-11T13:12:50Z
[FLINK-6546] [build] Fix dependencies of flink-mesos
- This makes all flink-related dependencies 'provided' to not have the
transitive dependencies promoted
- Drops the unnecessary dependency on the Hadoop artifact
- Adds directly referenced libraries, like jackson
- Deactivates default logging of tests
commit a14137c109e73738a3d1f89a3d99e2fd2a799219
Author: Stephan Ewen <[email protected]>
Date: 2017-05-11T15:32:03Z
[build] Reduce flink-avro's compile dependency from 'flink-java' to
'flink-core'
commit 32e8574498d7963e2ab58f1530b41a6853f23601
Author: Stephan Ewen <[email protected]>
Date: 2017-05-11T15:35:43Z
[FLINK-6514] [build] Remove 'flink-shaded-hadoop2' from 'flink-dist' via
exclusions
This is more tedious/manual than setting it to 'provided' once, but it
is also correct.
For example, in the case of Hadoop 2.3, having 'flink-shaded-hadoop2' as
'provided'
removes other needed dependencies as well, such as 'org.codehaus.jackson'
from avro.
commit efaed902a78c6a7f236e0dad4f72ed7ae8bad1c0
Author: Stephan Ewen <[email protected]>
Date: 2017-05-11T11:52:25Z
[FLINK-6514] [build] Merge bin and lib assembly
commit df8efd5cdadaeef9323472f20871871c94d14af5
Author: Stephan Ewen <[email protected]>
Date: 2017-05-11T15:00:03Z
[FLINK-6514] [build] Create a proper separate Hadoop uber jar for
'flink-dist' assembly
----
> Cannot start Flink Cluster in standalone mode
> ---------------------------------------------
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
> Issue Type: Bug
> Components: Build System, Cluster Management
> Reporter: Aljoscha Krettek
> Assignee: Stephan Ewen
> Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster
> results in a {{ClassNotFoundException}} of {{LogFactory}} in
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that
> we again have all the dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
