[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975268#comment-15975268
]
Luke Hutchison edited comment on FLINK-3328 at 4/19/17 6:43 PM:
----------------------------------------------------------------
[~StephanEwen] It's only not a problem if you know that in every case, the
version of every single shared library will be the same between all Flink jars.
And even if you know that will be the case, it's hardly ideal space-wise,
because you end up pulling in the same dependencies multiple times for every
Flink project, until a fat jar is built. In general, there is a good reason for
that Maven warning, because if there's any way that multiple different versions
of a dep can occur, you can trigger bugs that are maddeningly hard to trace.
If every Flink jar (like {{flink-runtime}}) depends on one of the core Flink
jars (e.g. {{flink-core}}), couldn't all the shared dependencies just be put in
that one jar?
was (Author: lukehutch):
[~StephanEwen] It's only not a problem if you know that in every case, the
version of every single shared library will be the same between all Flink jars.
And even if you know that will be the case, it's hardly ideal space-wise,
because you end up pulling in the same dependencies multiple times for every
Flink project, until a fat jar is built. In general, there is a good reason for
that Maven warning, because if there's any way that multiple different versions
of a dep can occur, you can trigger bugs that are maddeningly hard to trace.
If every Flink jar (like `flink-runtime`) depends on one of the core Flink jars
(e.g. `flink-core`), couldn't all the shared dependencies just be put in that
one jar?
> Incorrectly shaded dependencies in flink-runtime
> ------------------------------------------------
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
> Affects Versions: 1.0.0
> Reporter: Stephan Ewen
> Assignee: Robert Metzger
> Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1
> [WARNING] - 1680 more...
> [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar,
> flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes:
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Context
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldWriter
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypePath
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Handler
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypeReference
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Frame
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ByteVector
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor
> [WARNING] - 15 more...
> [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35
> overlapping classes:
> [WARNING] - jline.ArgumentCompletor$ArgumentList
> [WARNING] - jline.UnsupportedTerminal
> [WARNING] - jline.Terminal
> [WARNING] - jline.WindowsTerminal$ReplayPrefixOneCharInputStream
> [WARNING] - jline.History
> [WARNING] - jline.WindowsTerminal$1
> [WARNING] - jline.ConsoleReader
> [WARNING] - jline.ClassNameCompletor
> [WARNING] - jline.SimpleCompletor$SimpleCompletorFilter
> [WARNING] - jline.CandidateCycleCompletionHandler
> [WARNING] - 25 more...
> [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define
> 810 overlapping classes:
> [WARNING] -
> org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker
> [WARNING] - org.jboss.netty.handler.codec.base64.Base64Decoder
> [WARNING] - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1
> [WARNING] - org.jboss.netty.util.VirtualExecutorService
> [WARNING] - org.jboss.netty.util.DefaultObjectSizeEstimator
> [WARNING] -
> org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry
> [WARNING] - org.jboss.netty.channel.socket.oio.OioDatagramChannel
> [WARNING] - org.jboss.netty.logging.InternalLoggerFactory
> [WARNING] - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame
> [WARNING] - org.jboss.netty.channel.LifeCycleAwareChannelHandler
> [WARNING] - 800 more...
> [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar
> define 141 overlapping classes:
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule
> [WARNING] - 131 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar
> define 82 overlapping classes:
> [WARNING] - org.apache.commons.beanutils.Converter
> [WARNING] - org.apache.commons.beanutils.WrapDynaBean
> [WARNING] - org.apache.commons.beanutils.converters.IntegerConverter
> [WARNING] - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
> [WARNING] -
> org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter
> [WARNING] -
> org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter
> [WARNING] - org.apache.commons.beanutils.converters.ShortConverter
> [WARNING] - org.apache.commons.beanutils.converters.StringArrayConverter
> [WARNING] - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
> [WARNING] - org.apache.commons.beanutils.LazyDynaClass
> [WARNING] - 72 more...
> [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define
> 217 overlapping classes:
> [WARNING] - org.apache.commons.lang3.builder.DiffResult
> [WARNING] - org.apache.commons.lang3.CharRange
> [WARNING] -
> org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle
> [WARNING] - org.apache.commons.lang3.concurrent.ConcurrentException
> [WARNING] - org.apache.commons.lang3.builder.DiffBuilder$1
> [WARNING] - org.apache.commons.lang3.builder.DiffBuilder
> [WARNING] - org.apache.commons.lang3.builder.Diff
> [WARNING] - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField
> [WARNING] - org.apache.commons.lang3.ObjectUtils$Null
> [WARNING] - org.apache.commons.lang3.reflect.MemberUtils
> [WARNING] - 207 more...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
