I tend to agree. The move to maven makes package relocation very easy, but we should be careful not to abuse it if that’s a direction we decide to take.
IMHO, users having to modify the contents of $STORM_HOME/lib is really bad and should only be done as a last resort. If someone asks for help with an issue after they’ve modified the contents of $STORM_HOM/lib, all bets are off. I also agree that we need to keep a close watch on our dependencies, and that some might warrant re-evaluation. I’d love to others opinions. - Taylor On Feb 6, 2014, at 9:28 PM, Adam Lewis <[email protected]> wrote: > My $0.02 on this subject: > > Without going down the path of class loader or OSGi mania and becoming a full > container, I'd definitely be in favor of storm relocating its own > dependencies. In this way edge cases around things like reflection can be > handled once within storm rather than burdening every topology builder with > those details. Part of the problem seems to be that storm makes extensive > use (directly or transitively) of a lot of go-to utility libraries like > guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm sure > that leveraging these libraries allowed storm's development to proceed > rapidly, but from a maturity perspective, it is problematic to impose these > version choices on users. And while I might want Storm to, say, try to track > the latest Guava version, that same policy could be very problematic for > others. > > If storm can relocate even some of its own dependencies, I think that would > be a great help to me at least. Longer term, I wonder how much of some of > these libraries are really being used. For example, is clj-time (and by > extension joda-time) really needed? Or just a small fraction of the > functionality in that library? I can probably pitch in some of the effort > required to do this, if this is the direction people want to go in. > > > > > On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <[email protected]> wrote: > I’m glad the shader plugin worked for you. > > Updating dependencies can be tricky as it can easily introduce regressions. > > Ultimately we need to figure out the best solution to avoiding conflicts > between user code (i.e. dependencies in topology jar files) and Storm’s > libraries. > > The classloader approach has been attempted, but IMO Storm’s use of > serialization complicates things significantly. Package relocation seems to > be a relatively lightweight solution. > > If that’s a direction we pursue, then it introduces the question of whether > Storm should relocate its dependencies, or if that should be left up to the > user (topology developer). > > Elastic Search has gone down the path of relocating some of their > dependencies [1] (not necessarily an endorsement, just an observation). > > I’ve CC’d dev@ since this is all related to the infamous issue #115, which is > now STORM-129 [2]. > > - Taylor > > [1] https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474 > [2] https://issues.apache.org/jira/browse/STORM-129 > > > > > > On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <[email protected]> wrote: > >> Thank you all for replies! The shader-plugin solution seems to work for us. >> >> I wonder if we can create a JIRA ticket for storm to upgrade the http-client >> library as part of their next release. >> >> -Vinay >> >> >> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <[email protected]> wrote: >> We've done this with SLF4j and Guava as well without issues. >> >> Michael Rose (@Xorlev) >> Senior Platform Engineer, FullContact >> [email protected] >> >> >> >> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <[email protected]> wrote: >> We had this problem as well. We modified our chef cookbook to just replace >> the older version with the newer one and storm didn't complain or have any >> other issues as a result. >> >> >> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <[email protected]> wrote: >> Your best bet is probably to use the shade plugin to relocate the >> http-client package so it doesn’t conflict with the version storm uses. >> >> Storm does this with the libtrhift dependency in storm-core: >> >> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220 >> >> (You can ignore the clojure transformer in that config, unless you have >> non-AOT clojure code that uses the http-client library). >> >> More information on using the shade plugin to do package relocations can be >> found here: >> >> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html >> >> - Taylor >> >> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <[email protected]> wrote: >> >> > Hello, >> > >> > I am using storm version 0.9.0.1. >> > My application depends on apache http-client version 4.3.2 - but storm >> > depends on http-client version 4.1.1. >> > >> > What is the best way to override this dependency? >> > >> > Thanks >> > Vinay >> >> >> >> > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
