I like the idea, thank you for bringing it up.
Given that the raised problems aren't really ASM specific would it make
sense to create one flink-shaded module that contains all frequently
shaded libraries? (or maybe even all shaded dependencies by core
modules) The proposal limits the scope of this to ASM and i was
wondering why.
I also remember that there was a discussion recently about why we shade
things at all, and the idea of working against the shaded namespaces was
brought up. Back then i was expressing doubts as to whether IDE's would
properly support this; what's the state on that?
On 10.05.2017 18:18, Stephan Ewen wrote:
Hi!
This is a discussion about altering the way we handle dependencies and
shading in Flink.
I ran into quite a view problems trying to adjust / fix some shading issues
during release validation.
The issue is tracked under: https://issues.apache.org/jira/browse/FLINK-6529
Bring this discussion thread up because it is a bigger issue
*Problem*
Currently, Flink shades dependencies like ASM and Guava into all jars of
projects that reference it and relocate the classes.
There are some drawbacks to that approach, let's discuss them at the
example of ASM:
- The ASM classes are for example in flink-core, flink-java, flink-scala,
flink-runtime, etc.
- Users that reference these dependencies have the classes multiple times
in the classpath. That is unclean (works, through, because the classes are
identical). The same happens when building the final dist. jar.
- Some of these dependencies require to include license files in the
shaded jar. It is hard to impossible to build a good automatic solution for
that, partly due to Maven's very poor cross-project path support
- Most importantly: Scala does not support shading really well. Scala
classes have references to classes in more places than just the class names
(apparently for Scala reflect support). Referencing a Scala project with
shaded ASM still requires to add a reference to unshaded ASM (at least as a
compile dependency).
*Proposal*
I propose that we build and deploy a asm-flink-shaded version of ASM and
directly program against the relocated namespaces. Since we never use
classes that we relocate in public interfaces, Flink users will never see
the relocated class names. Internally, it does not hurt to use them.
- Proper maven dependency management, no hidden (shaded) dependencies
- One copy of each class for shaded dependencies
- Proper Scala interoperability
- Natural License management (license is part of deployed
asm-flink-shaded jar)
Happy to hear thoughts!
Stephan
