[
https://issues.apache.org/jira/browse/STORM-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13971530#comment-13971530
]
Robert Joseph Evans commented on STORM-169:
-------------------------------------------
Yes there are pre-compiled versions of clojure code packaged directly into the
storm jar. This is how clojure does ahead of time compiling. There are also
shaded thrift7 classes in the jar. So I concede that storm-core could be
considered an uberjar. I don't consider it as such because clojure does not
automatically go off of the precompiled versions. If you include a jar in the
classpath that has the corresponding .clj files in it, clojure will dynamically
compile them and they will be used for execution. Also as Derek and I both
said it really depends on the order of the classpath when the topology is
launched. It does not matter if storm-core has lots of stuff in it, if
something else overrides them and is on the classpath first.
There are a few ways to isolate dependencies. We can do like what JBoss or
OSGI do and have a classloader for user code separate from the one the system
uses. This is painful to get right, and my personal opinion would be to leave
it up to the experts and integrate storm with an existing OSGI framework, but
this can be kind of heavy weight. If there are some OSGI experts or class
loader gurus out there that would like to volunteer to help out with this we
would gladly accept your contributions.
The route I would go with is to remove everything from the classpath that is
not needed by the worker and then shade what is left that can be shaded. This
is also not a trivial task, and as such even though it has been a known issue
for a very long time we have not taken any real decisive steps to fix this.
Even Hadoop has only recently taken steps trying to add in their own class
loaded for user code, it falls over in a number of different situations, and is
off by default.
> Storm depends on outdated versions of hiccup and ring
> -----------------------------------------------------
>
> Key: STORM-169
> URL: https://issues.apache.org/jira/browse/STORM-169
> Project: Apache Storm (Incubating)
> Issue Type: Wish
> Reporter: James Xu
> Priority: Minor
>
> https://github.com/nathanmarz/storm/issues/707
> Storm core depends on a very old versions of
> hiccup 0.3.6 => 1.0.4
> ring-* 0.3.11 => 1.2.0
> Especially the hiccup version conflicts with modern libraries like liberator
> (see also weavejester/hiccup#86
> IllegalStateException escape-html already refers to:
> #'hiccup.core/escape-html in namespace: hiccup.page
> ----------
> revans2: I agree that storm depends on old versions of libraries that I would
> love to see updated, but simply updating the versions can cause a lot of pain
> too. Having worked through this on Hadoop the only real solution is to get to
> true dependency isolation. There are just too many libraries that are not
> backwards and forwards compatible in very subtle ways, even for minor
> releases. If you upgrade one dependency to fix a specific incompatibility it
> will break someone else with a new incompatibility.
> I am not saying that we should not do this. People should be willing to pay
> off tech debt and upgrade, but until we can isolate a user's dependencies
> from storm's dependencies we need to consider major dependency upgrades as
> incompatible changes.
> ----------
> jocrau: Yes, updating versions can cause a lot of pain. But deferring every
> update to the time when the user's dependency is isolated might also cause a
> lot of pain and slow down the adoption of Storm in some domains (like RESTful
> services). I would love to see a more differentiated approach. Hiccup for
> example is only used in the Storm UI; a place where I expect little impact.
> ----------
> revans2: I totally agree. I just wanted to be sure that the storm version
> number was bumped when we did this, so that we are indicating to downstream
> users that something incompatible may have happened and they need to retest.
> Now that storm is a multi-module project splitting out the UI and logviewer
> to be their own jars with separate dependencies really seems like a good
> start for isolation. More then that gets a little bit harry but is still
> doable.
> If you think about it the worker only needs to pull in a few clojure APIs,
> disruptor, curator, and jzmq or netty depending on how it is configured.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
