Re: [DISCUSS] Handling dependencies for Storm 2.0.0 (STORM-2882)

[Bobby Evans]

Sorry for the late reply.

I am 100% in favor of shading again for very common dependencies.  The big
issue is around how do we do this cleanly?  I like the way flink does it.
That can help with some of the issues we see with IDEs, but we just need to
be very careful with transitive dependencies when we do this.  I am not
aware of any issues like this, but if we have something like storm-client
depends on A and B, and B also depends on A.  If we shade A by itself, the
original A would still be needed on the classpath for B to work properly,
or we would have to do some kind of shading for B too, which if we don't do
it carefully might result in two separate copies of a shaded A on the
classpath.

Thanks,

Bobby

On Wed, Feb 21, 2018 at 1:20 AM Jungtaek Lim  wrote:

> Hi devs,
>
> I'm initiating this one for part of Storm 2.0.0. Actually we had some
> discussion from earlier thread [1] but it was originated for IDE
> complaining and we completely remove shading/relocating in Storm 2.0.0 at
> that time, ends up with exposing all the transitive dependencies in
> 'storm-client' to user topology code. We've reduced lots of dependencies
> while breaking down storm-core, but troublesome dependencies (for example,
> Guava, Jackson, Netty, and so on) are still left.
>
> There're two questions we should address:
>
> 1. Do we want to (and need to) shade/relocate the dependencies?
> (We may also want to see it as regression issue, since end users should
> tackle with dependencies if we don't shade/relocate.)
>
> 2. If we want to relocate the dependencies, how?
>
> If we would want to relocate them, there's a good reference on Flink: Flink
> creates separate repository [2] which contains pom for relocation for each
> target artifact. Links for related discussion [3] and issue [4] are
> available, so if you are interested on Flink's approach, please refer the
> links. I guess it may require considerable efforts, so if someone finds
> simpler and easier way that should be great.
>
> Please share your voices regarding two questions, and idea/knowledge of
> how. Thanks in advance.
>
> Jungtaek Lim (HeartSaVioR)
>
> 1.
>
> https://mail-archives.apache.org/mod_mbox/incubator-storm-dev/201703.mbox/%3ccaf5108gjjqzsyywcp99bgpgec7tufe-tbt9fi0m78ah8rkm...@mail.gmail.com%3E
> 2. https://github.com/apache/flink-shaded
> 3.
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html
> 4. https://issues.apache.org/jira/browse/FLINK-6529
>

[DISCUSS] Handling dependencies for Storm 2.0.0 (STORM-2882)

[Jungtaek Lim]

Hi devs,

I'm initiating this one for part of Storm 2.0.0. Actually we had some
discussion from earlier thread [1] but it was originated for IDE
complaining and we completely remove shading/relocating in Storm 2.0.0 at
that time, ends up with exposing all the transitive dependencies in
'storm-client' to user topology code. We've reduced lots of dependencies
while breaking down storm-core, but troublesome dependencies (for example,
Guava, Jackson, Netty, and so on) are still left.

There're two questions we should address:

1. Do we want to (and need to) shade/relocate the dependencies?
(We may also want to see it as regression issue, since end users should
tackle with dependencies if we don't shade/relocate.)

2. If we want to relocate the dependencies, how?

If we would want to relocate them, there's a good reference on Flink: Flink
creates separate repository [2] which contains pom for relocation for each
target artifact. Links for related discussion [3] and issue [4] are
available, so if you are interested on Flink's approach, please refer the
links. I guess it may require considerable efforts, so if someone finds
simpler and easier way that should be great.

Please share your voices regarding two questions, and idea/knowledge of
how. Thanks in advance.

Jungtaek Lim (HeartSaVioR)

1.
https://mail-archives.apache.org/mod_mbox/incubator-storm-dev/201703.mbox/%3ccaf5108gjjqzsyywcp99bgpgec7tufe-tbt9fi0m78ah8rkm...@mail.gmail.com%3E
2. https://github.com/apache/flink-shaded
3.
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html
4. https://issues.apache.org/jira/browse/FLINK-6529