[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042832#comment-16042832
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user StephanEwen closed the pull request at:

https://github.com/apache/flink/pull/3877


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0, 1.4.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013915#comment-16013915
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3877
  
@StephanEwen can you manually close the PR?
Thx.


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0, 1.4.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010239#comment-16010239
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3877
  
Merging this change ...


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010233#comment-16010233
 ] 

Robert Metzger commented on FLINK-6514:
---

Merged for 1.4 in 
- http://git-wip-us.apache.org/repos/asf/flink/commit/aa135914
- http://git-wip-us.apache.org/repos/asf/flink/commit/3c77812e
- http://git-wip-us.apache.org/repos/asf/flink/commit/63bbb4f5
- http://git-wip-us.apache.org/repos/asf/flink/commit/e4cdf125
- http://git-wip-us.apache.org/repos/asf/flink/commit/ae423e1d

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010230#comment-16010230
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3876


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010225#comment-16010225
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3876
  
I'm merging this change now


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010195#comment-16010195
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3876
  
+1 to merge


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008114#comment-16008114
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3877#discussion_r116227807
  
--- Diff: flink-shaded-hadoop/flink-shaded-hadoop2-uber/pom.xml ---
@@ -0,0 +1,129 @@
+
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-shaded-hadoop
+   1.4-SNAPSHOT
--- End diff --

This needs to be 1.3-SNAPSHOT.


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007986#comment-16007986
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3876
  
This PR works in my VM. There's just some weird behavior when starting 
Flink on YARN.
I'll cross check with AWS EMR to see if its a YARN or a VM/Cloudera issue.

I wanted to check on EMR anyways for the release :)


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007906#comment-16007906
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3876
  
The change looks good for me.
I'll verify it with my Cloudera VM to make sure it works with provided 
hadoop versions as well.


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007882#comment-16007882
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/3876#discussion_r116190819
  
--- Diff: flink-dist/pom.xml ---
@@ -403,15 +403,6 @@ under the License.


 
-   
--- End diff --

I think this is an artifact from the time when the checker was defined in 
the root pom and we disabled it for all modules that don't need checking.


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007672#comment-16007672
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3877
  
The only diff from #3876 is the dropping of the jersey/jettison/servlet 
deps, right?


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007670#comment-16007670
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/3876#discussion_r116157739
  
--- Diff: flink-shaded-hadoop/flink-shaded-hadoop2-uber/pom.xml ---
@@ -0,0 +1,129 @@
+
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-shaded-hadoop
+   1.4-SNAPSHOT
+   ..
+   
+
+   flink-shaded-hadoop2-uber
+   flink-shaded-hadoop2-uber
+
+   jar
+
+   
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007669#comment-16007669
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/3877#discussion_r116157591
  
--- Diff: flink-shaded-hadoop/flink-shaded-hadoop2-uber/pom.xml ---
@@ -0,0 +1,129 @@
+
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-shaded-hadoop
+   1.4-SNAPSHOT
+   ..
+   
+
+   flink-shaded-hadoop2-uber
+   flink-shaded-hadoop2-uber
+
+   jar
+
+   
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007668#comment-16007668
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/3876#discussion_r116157373
  
--- Diff: flink-dist/pom.xml ---
@@ -403,15 +403,6 @@ under the License.


 
-   
--- End diff --

This is removed because we have it in all the modules itself?


> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007050#comment-16007050
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/3877

[backport] [FLINK-6514] [build] Create a proper separate Hadoop uber jar 
for 'flink-dist' assembly

Backport of #3876 to `release-1.3`

This fixes the issue that Flink cannot be started locally if built with 
Maven 3.3+

There are two big fixes in this pull request, because they do not 
build/pass tests individually. The wrong Mesos dependencies where the reason 
that the broken Hadoop fat jar building actually passed the Yarn tests.

# Hadoop Uber Jar

  - This builds a proper Hadoop Uber Jar with all of Hadoop's needed 
dependencies. The prior build was missing many important dependencies in the 
Hadoop Uber Jar.

  - The Hadoop-jar is no longer excluded in `flink-dist` via setting the 
dependency to `provided`, but by explicit exclusion. That way, Hadoop's 
transitive dependencies are not excluded from other dependencies as well. 
Before this patch, various decompression and Avro were broken in a Flink build, 
due to accidental exclusion of their transitive dependencies.

# Dependency fixing

  - This also fixes the dependencies of `flink-mesos`, which made all of 
Hadoop's transitive dependencies its own dependencies, by promoting them during 
shading. That way, Flink had various unnecessary dependencies in its 
`flink-dist` jar.

  - Incidentally, that brought Hadoop's accidentally excluded dependencies 
back in, but into the `flink-dist` jar, not the `shaded-hadoop2` jar.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink fix_fat_jar_13

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3877


commit ac15bc32a3d786b50e8864a903d31d0b3e0c3042
Author: Stephan Ewen 
Date:   2017-05-11T13:12:10Z

[hotfix] [build] Drop transitive jersey/jettison/servlet dependencies 
pulled via Hadoop

commit 5e94787ce4e14f5da88e71418200b4bbe517483b
Author: Stephan Ewen 
Date:   2017-05-11T13:12:50Z

[FLINK-6546] [build] Fix dependencies of flink-mesos

  - This makes all flink-related dependencies 'provided' to not have the
transitive dependencies promoted

  - Drops the unnecessary dependency on the Hadoop artifact

  - Adds directly referenced libraries, like jackson

  - Deactivates default logging of tests

commit b568ccfdf7366056d29ee43d14c606cfc4448bab
Author: Stephan Ewen 
Date:   2017-05-11T15:32:03Z

[build] Reduce flink-avro's compile dependency from 'flink-java' to 
'flink-core'

commit 84c150a5798f029bb9aced998ad6b81dd8dc8de5
Author: Stephan Ewen 
Date:   2017-05-11T15:35:43Z

[FLINK-6514] [build] Remove 'flink-shaded-hadoop2' from 'flink-dist' via 
exclusions

This is more tedious/manual than setting it to 'provided' once, but it
is also correct.

For example, in the case of Hadoop 2.3, having 'flink-shaded-hadoop2' as 
'provided'
removes other needed dependencies as well, such as 'org.codehaus.jackson' 
from avro.

commit 99658870865c15ad0996066cf94c721f30bc86ca
Author: Stephan Ewen 
Date:   2017-05-11T11:52:25Z

[FLINK-6514] [build] Merge bin and lib assembly

commit 93e37c666aba50988a48b9273d7b531434c5d5b1
Author: Stephan Ewen 
Date:   2017-05-11T15:00:03Z

[FLINK-6514] [build] Create a proper separate Hadoop uber jar for 
'flink-dist' assembly




> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007041#comment-16007041
 ] 

ASF GitHub Bot commented on FLINK-6514:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/3876

[FLINK-6514] [build] Create a proper separate Hadoop uber jar for 
'flink-dist' assembly

This fixes the issue that Flink cannot be started locally if built with 
Maven 3.3+

There are two big fixes in this pull request, because they do not 
build/pass tests individually. The wrong Mesos dependencies where the reason 
that the broken Hadoop fat jar building actually passed the Yarn tests.

# Hadoop Uber Jar

  - This builds a proper Hadoop Uber Jar with all of Hadoop's needed 
dependencies. The prior build was missing many important dependencies in the 
Hadoop Uber Jar.

  - The Hadoop-jar is no longer excluded in `flink-dist` via setting the 
dependency to `provided`, but by explicit exclusion. That way, Hadoop's 
transitive dependencies are not excluded from other dependencies as well. 
Before this patch, various decompression and Avro were broken in a Flink build, 
due to accidental exclusion of their transitive dependencies.

# Dependency fixing

  - This also fixes the dependencies of `flink-mesos`, which made all of 
Hadoop's transitive dependencies its own dependencies, by promoting them during 
shading. That way, Flink had various unnecessary dependencies in its 
`flink-dist` jar.

  - Incidentally, that brought Hadoop's accidentally excluded dependencies 
back in, but into the `flink-dist` jar, not the `shaded-hadoop2` jar.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink fix_fat_jar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3876.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3876


commit 86803d23dfdc56f2be64274a6a90a76c1e782f08
Author: Stephan Ewen 
Date:   2017-05-11T13:12:50Z

[FLINK-6546] [build] Fix dependencies of flink-mesos

  - This makes all flink-related dependencies 'provided' to not have the
transitive dependencies promoted

  - Drops the unnecessary dependency on the Hadoop artifact

  - Adds directly referenced libraries, like jackson

  - Deactivates default logging of tests

commit a14137c109e73738a3d1f89a3d99e2fd2a799219
Author: Stephan Ewen 
Date:   2017-05-11T15:32:03Z

[build] Reduce flink-avro's compile dependency from 'flink-java' to 
'flink-core'

commit 32e8574498d7963e2ab58f1530b41a6853f23601
Author: Stephan Ewen 
Date:   2017-05-11T15:35:43Z

[FLINK-6514] [build] Remove 'flink-shaded-hadoop2' from 'flink-dist' via 
exclusions

This is more tedious/manual than setting it to 'provided' once, but it
is also correct.

For example, in the case of Hadoop 2.3, having 'flink-shaded-hadoop2' as 
'provided'
removes other needed dependencies as well, such as 'org.codehaus.jackson' 
from avro.

commit efaed902a78c6a7f236e0dad4f72ed7ae8bad1c0
Author: Stephan Ewen 
Date:   2017-05-11T11:52:25Z

[FLINK-6514] [build] Merge bin and lib assembly

commit df8efd5cdadaeef9323472f20871871c94d14af5
Author: Stephan Ewen 
Date:   2017-05-11T15:00:03Z

[FLINK-6514] [build] Create a proper separate Hadoop uber jar for 
'flink-dist' assembly




> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Assignee: Stephan Ewen
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006903#comment-16006903
 ] 

Stephan Ewen commented on FLINK-6514:
-

There are more issues. The way this is set up right now also drops transitive 
dependencies from Flink that should not be dropped (compression, parts of Avro 
support, ...)

I have a more comprehensive fix for this...

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006106#comment-16006106
 ] 

Robert Metzger commented on FLINK-6514:
---

We could use the maven enforcer plugin to enforce maven version 3.2.5 in the 
release profile.
But I think almost everybody doing a custom Flink build will run into the issue.

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004497#comment-16004497
 ] 

Chesnay Schepler commented on FLINK-6514:
-

Why was this closed as not a problem? Having to downgrade maven is not a good 
solution, especially when we don't even know what the actual underlying problem 
is.

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004471#comment-16004471
 ] 

Aljoscha Krettek commented on FLINK-6514:
-

Yes, this seems to have been caused by my Maven version.

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004403#comment-16004403
 ] 

Aljoscha Krettek commented on FLINK-6514:
-

Checking this out now.

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002964#comment-16002964
 ] 

Greg Hogan commented on FLINK-6514:
---

[~aljoscha] are you running a newer version of Maven? I was seeing this issue 
yesterday until reverting to 3.2.5.

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6514) Cannot start Flink Cluster in standalone mode

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002949#comment-16002949
 ] 

Aljoscha Krettek commented on FLINK-6514:
-

Proposed solution (that has other possible complications): 
https://github.com/apache/flink/pull/3856

> Cannot start Flink Cluster in standalone mode
> -
>
> Key: FLINK-6514
> URL: https://issues.apache.org/jira/browse/FLINK-6514
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Cluster Management
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.0
>
>
> The changes introduced for FLINK-5998 change what goes into the 
> {{flink-dost}} fat jar. As it is, this means that trying to start a cluster 
> results in a {{ClassNotFoundException}} of {{LogFactory}} in 
> {{commons-logging}}.
> One solution is to now make the shaded Hadoop jar a proper fat-jar, so that 
> we again have all the dependencies.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)