[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977227#comment-15977227
]
Stephan Ewen commented on FLINK-3328:
-
That's all right.
We would not shade/bundle if we did not need to. The libraries we shade are
simply frequently conflicting with different version used by users, that's why
we need to make them disappear.
There is simply no good way in Java currently to allow different versions of a
library to co-exist. For everyone to always use the newest one, libraries would
have to be perfectly backwards compatible, which most are not.
Hence the ugly shading business...
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
>
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977155#comment-15977155
]
Luke Hutchison commented on FLINK-3328:
---
Right, but shading requires a complete copy of the deps to be shipped inside
the jar. I understand the desire to reduce namespace clashes and to not allow
users to depend upon Flink's deps (or to not expose users to Flink's deps). But
I think it's actually worse to ship a complete copy of all deps (shaded) with a
library, due to increased overall size, if a consumer of the library also uses
the dep.
(Also, in the general case, bundling is almost always a bad idea for security
reasons -- if an urgent patch is needed to a shaded jar, users have to wait for
the producer of the library to re-build with the fix, rather than just
depending upon an ABI-compatible fixed version themselves -- although this is
mitigated here by the fact that Flink is open source, so users can build their
own Flink libraries if they need to. Bundling in general is generally frowned
upon for this reason, e.g. in the situation outside the JVM world where a large
number of programs statically link in OpenSSH and/or zlib, rather than using
the systemwide version of it.)
Apologies if I'm still misunderstanding the issues here.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar,
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977106#comment-15977106
]
Stephan Ewen commented on FLINK-3328:
-
What you suggest (let Maven or Gradle to pull in deps) is exactly what we do
currently. Shading hides a dependency from downstream dependencies, so they
pull the dependency again, and again hide it in the shading process.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976649#comment-15976649
]
Luke Hutchison commented on FLINK-3328:
---
What's the advantage to Flink of including all these shaded deps, rather than
just relying on something like Maven or Gradle to pull in deps? There are
numerous reasons why it is not usually a good idea to incorporate copies of
external dependencies.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976621#comment-15976621
]
Stephan Ewen commented on FLINK-3328:
-
Yeah, I see your point. It would work more smooth if there is a separate
project with separate release for the big shaded artifact.
Yet one more point for the way we have it currently...
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976408#comment-15976408
]
Chesnay Schepler commented on FLINK-3328:
-
Would IDE's work properly out-of-the-box when working against the shaded
namespaces?
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING] -
>
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976371#comment-15976371
]
Stephan Ewen commented on FLINK-3328:
-
To do that, one would have to explicitly code against the shaded namespaces,
meaning explicitly use
{{org.apache.flink.shaded.com.google.common.collect.Lists}} rather than
{{com.google.common.collect.Lists}}.
That's not ideal either.
I think the best think is to actually avoid libraries that are used in many
modules need shading. We started to do that with Guava. With ASM it will not be
possible, I think.
An alternative that I could see is to have a single
{{flink-shaded-dependencies}} project in which we have the libraries shaded
once, and then we develop against the shaded namespaces directly.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975268#comment-15975268
]
Luke Hutchison commented on FLINK-3328:
---
[~StephanEwen] It's only not a problem if you know that in every case, the
version of every single shared library will be the same between all Flink jars.
And even if you know that will be the case, it's hardly ideal space-wise,
because you end up pulling in the same dependencies multiple times for every
Flink project, until a fat jar is built. In general, there is a good reason for
that Maven warning, because if there's any way that multiple different versions
of a dep can occur, you can trigger bugs that are maddeningly hard to trace.
If every Flink jar (like `flink-runtime`) depends on one of the core Flink jars
(e.g. `flink-core`), couldn't all the shared dependencies just be put in that
one jar?
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING]
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975227#comment-15975227
]
Stephan Ewen commented on FLINK-3328:
-
That is not a problem. Guava and asm must be in each artifact ({{flink-core}},
{{flink-runtime}}, etc) to make them self contained.
When assembling the final jar, they all have guava, so you have an overlap of
the exact same classes.
The alternative is to relocate it to a different namespace in each artifact,
but then we get guava 10 times into the {{flink-dist}}, and that is not very
desirable.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
>
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969820#comment-15969820
]
Luke Hutchison commented on FLINK-3328:
---
[~rmetzger] I'm still seeing this in Flink 1.2.0:
{noformat}
[WARNING] flink-core-1.2.0.jar, flink-java-1.2.0.jar,
flink-runtime_2.10-1.2.0.jar define 166 overlapping classes:
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$LocalVarRule
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.tree.InvokeDynamicInsnNode
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$ExceptionRule
[WARNING] - org.apache.flink.shaded.org.objectweb.asm.Attribute
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$LookupSwitchLabelRule
[WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.LdcInsnNode
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.xml.Processor$OutputSlicingHandler
[WARNING] -
org.apache.flink.shaded.org.objectweb.asm.tree.LocalVariableAnnotationNode
[WARNING] - org.apache.flink.shaded.org.objectweb.asm.commons.StaticInitMerger
[WARNING] - 156 more...
[WARNING] flink-runtime-web_2.10-1.2.0.jar, flink-runtime_2.10-1.2.0.jar,
flink-optimizer_2.10-1.2.0.jar define 1690 overlapping classes:
[WARNING] -
org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedSortedSetMultimap
[WARNING] -
org.apache.flink.shaded.com.google.common.util.concurrent.FutureCallback
[WARNING] -
org.apache.flink.shaded.com.google.common.util.concurrent.Monitor$Guard
[WARNING] -
org.apache.flink.shaded.com.google.common.util.concurrent.Striped$2
[WARNING] -
org.apache.flink.shaded.com.google.common.collect.ImmutableEnumSet$1
[WARNING] -
org.apache.flink.shaded.com.google.common.primitives.Bytes$ByteArrayAsList
[WARNING] - org.apache.flink.shaded.com.google.common.math.IntMath$1
[WARNING] -
org.apache.flink.shaded.com.google.common.hash.Hashing$ConcatenatedHashFunction
[WARNING] -
org.apache.flink.shaded.com.google.common.hash.Funnels$UnencodedCharsFunnel
[WARNING] -
org.apache.flink.shaded.com.google.common.collect.DenseImmutableTable$ImmutableArrayMap$1$1
[WARNING] - 1680 more...
{noformat}
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING]
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132134#comment-15132134
]
Robert Metzger commented on FLINK-3328:
---
I had an offline chat with Stephan. The file is only 7.6 MB.
The netty, sfl4j, .. dependencies are coming from the
"flink-shaded-curator-recipes.jar"
Further investigating
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132378#comment-15132378
]
ASF GitHub Bot commented on FLINK-3328:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1584#issuecomment-179866261
It works with CDH 5.4.2, which uses Hadoop 2.6.0.
I'll merge it.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132380#comment-15132380
]
ASF GitHub Bot commented on FLINK-3328:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1584
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING] -
>
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132372#comment-15132372
]
ASF GitHub Bot commented on FLINK-3328:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1584#issuecomment-179861292
Cool, thanks for fixing this.
Changes look good to me!
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] -
[jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132269#comment-15132269
]
ASF GitHub Bot commented on FLINK-3328:
---
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1584#issuecomment-179823460
Till told me that we do all this curator shading because of Hadoop 2.6.0.
I'll verify if everything is working with 2.6 as well.
> Incorrectly shaded dependencies in flink-runtime
>
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
>
