+1 for relocating storm's dependencies. I just ran into this a couple weeks ago and have had to side-shelf the project until I find a work-around. Will check-out the shade plugin. Thanks for the rec!
-Cody On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <[email protected]> wrote: > I tend to agree. > > The move to maven makes package relocation very easy, but we should be > careful not to abuse it if that's a direction we decide to take. > > IMHO, users having to modify the contents of $STORM_HOME/lib is really bad > and should only be done as a last resort. If someone asks for help with an > issue after they've modified the contents of $STORM_HOM/lib, all bets are > off. > > I also agree that we need to keep a close watch on our dependencies, and > that some might warrant re-evaluation. > > I'd love to others opinions. > > - Taylor > > On Feb 6, 2014, at 9:28 PM, Adam Lewis <[email protected]> wrote: > > My $0.02 on this subject: > > Without going down the path of class loader or OSGi mania and becoming a > full container, I'd definitely be in favor of storm relocating its own > dependencies. In this way edge cases around things like reflection can be > handled once within storm rather than burdening every topology builder with > those details. Part of the problem seems to be that storm makes extensive > use (directly or transitively) of a lot of go-to utility libraries like > guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm > sure that leveraging these libraries allowed storm's development to proceed > rapidly, but from a maturity perspective, it is problematic to impose these > version choices on users. And while I might want Storm to, say, try to > track the latest Guava version, that same policy could be very problematic > for others. > > If storm can relocate even some of its own dependencies, I think that > would be a great help to me at least. Longer term, I wonder how much of > some of these libraries are really being used. For example, is clj-time > (and by extension joda-time) really needed? Or just a small fraction of the > functionality in that library? I can probably pitch in some of the effort > required to do this, if this is the direction people want to go in. > > > > > On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <[email protected]> wrote: > >> I'm glad the shader plugin worked for you. >> >> Updating dependencies can be tricky as it can easily introduce >> regressions. >> >> Ultimately we need to figure out the best solution to avoiding conflicts >> between user code (i.e. dependencies in topology jar files) and Storm's >> libraries. >> >> The classloader approach has been attempted, but IMO Storm's use of >> serialization complicates things significantly. Package relocation seems to >> be a relatively lightweight solution. >> >> If that's a direction we pursue, then it introduces the question of >> whether Storm should relocate its dependencies, or if that should be left >> up to the user (topology developer). >> >> Elastic Search has gone down the path of relocating some of their >> dependencies [1] (not necessarily an endorsement, just an observation). >> >> I've CC'd dev@ since this is all related to the infamous issue #115, >> which is now STORM-129 [2]. >> >> - Taylor >> >> [1] >> https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474 >> [2] https://issues.apache.org/jira/browse/STORM-129 >> >> >> >> >> >> On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <[email protected]> >> wrote: >> >> Thank you all for replies! The shader-plugin solution seems to work for >> us. >> >> I wonder if we can create a JIRA ticket for storm to upgrade the >> http-client library as part of their next release. >> >> -Vinay >> >> >> On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <[email protected]>wrote: >> >>> We've done this with SLF4j and Guava as well without issues. >>> >>> Michael Rose (@Xorlev <https://twitter.com/xorlev>) >>> Senior Platform Engineer, FullContact <http://www.fullcontact.com/> >>> [email protected] >>> >>> >>> On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <[email protected]> wrote: >>> >>>> We had this problem as well. We modified our chef cookbook to just >>>> replace the older version with the newer one and storm didn't complain or >>>> have any other issues as a result. >>>> >>>> >>>> On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <[email protected]>wrote: >>>> >>>>> Your best bet is probably to use the shade plugin to relocate the >>>>> http-client package so it doesn't conflict with the version storm uses. >>>>> >>>>> Storm does this with the libtrhift dependency in storm-core: >>>>> >>>>> >>>>> https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220 >>>>> >>>>> (You can ignore the clojure transformer in that config, unless you >>>>> have non-AOT clojure code that uses the http-client library). >>>>> >>>>> More information on using the shade plugin to do package relocations >>>>> can be found here: >>>>> >>>>> >>>>> http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html >>>>> >>>>> - Taylor >>>>> >>>>> On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <[email protected]> >>>>> wrote: >>>>> >>>>> > Hello, >>>>> > >>>>> > I am using storm version 0.9.0.1. >>>>> > My application depends on apache http-client version 4.3.2 - but >>>>> storm depends on http-client version 4.1.1. >>>>> > >>>>> > What is the best way to override this dependency? >>>>> > >>>>> > Thanks >>>>> > Vinay >>>>> >>>>> >>>> >>> >> >> > > -- Cody A. Ray, LEED AP [email protected] 215.501.7891
