[
https://issues.apache.org/jira/browse/STORM-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970319#comment-13970319
]
Derek Dagit commented on STORM-169:
-----------------------------------
I think by "uber jar" revans2 meant a hypothetical jar containing both the
user's classes and storm's classes, not merely the storm-core jar. (I have
seen the same term for this idea used in Hadoop.)
What we have now are the user's jar and storm's jar. When the topology is
launched, the storm-core jar is on the classpath before the user's jar. So the
version of, e.g., hiccup that will be used when the worker runs will be
whatever version is in the storm jar—not the version in the user's jar—since
the storm-core jar takes precedence.
He is suggesting that, first, we separate the necessary dependencies—those that
the storm code needs to launch a worker—into a different, new jar. Perhaps we
could call it the "worker jar."
When workers would launch, they would include *only* this worker jar, with the
essential storm dependencies for launching a worker. When storm daemons
launch, they would need both the classes in the worker jar, plus the remainder
from what is now in storm-core. The remainder would include the old versions
of conflicting libraries that would otherwise cause dependency problems like
those we are seeing.
Second, we look at what we have in the new worker jar, and we shade those storm
dependencies so that even they will not conflict with user code that uses
different versions.
That is my understanding of the general idea, but please correct me if I
misunderstood.
> Storm depends on outdated versions of hiccup and ring
> -----------------------------------------------------
>
> Key: STORM-169
> URL: https://issues.apache.org/jira/browse/STORM-169
> Project: Apache Storm (Incubating)
> Issue Type: Wish
> Reporter: James Xu
> Priority: Minor
>
> https://github.com/nathanmarz/storm/issues/707
> Storm core depends on a very old versions of
> hiccup 0.3.6 => 1.0.4
> ring-* 0.3.11 => 1.2.0
> Especially the hiccup version conflicts with modern libraries like liberator
> (see also weavejester/hiccup#86
> IllegalStateException escape-html already refers to:
> #'hiccup.core/escape-html in namespace: hiccup.page
> ----------
> revans2: I agree that storm depends on old versions of libraries that I would
> love to see updated, but simply updating the versions can cause a lot of pain
> too. Having worked through this on Hadoop the only real solution is to get to
> true dependency isolation. There are just too many libraries that are not
> backwards and forwards compatible in very subtle ways, even for minor
> releases. If you upgrade one dependency to fix a specific incompatibility it
> will break someone else with a new incompatibility.
> I am not saying that we should not do this. People should be willing to pay
> off tech debt and upgrade, but until we can isolate a user's dependencies
> from storm's dependencies we need to consider major dependency upgrades as
> incompatible changes.
> ----------
> jocrau: Yes, updating versions can cause a lot of pain. But deferring every
> update to the time when the user's dependency is isolated might also cause a
> lot of pain and slow down the adoption of Storm in some domains (like RESTful
> services). I would love to see a more differentiated approach. Hiccup for
> example is only used in the Storm UI; a place where I expect little impact.
> ----------
> revans2: I totally agree. I just wanted to be sure that the storm version
> number was bumped when we did this, so that we are indicating to downstream
> users that something incompatible may have happened and they need to retest.
> Now that storm is a multi-module project splitting out the UI and logviewer
> to be their own jars with separate dependencies really seems like a good
> start for isolation. More then that gets a little bit harry but is still
> doable.
> If you think about it the worker only needs to pull in a few clojure APIs,
> disruptor, curator, and jzmq or netty depending on how it is configured.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
