On Fri, Feb 7, 2014 at 1:21 PM, Bobby Evans <[email protected]> wrote:
> I am +1 for relocating storm's dependencies, but I also think that we want > to split some parts out too. There are a number of things on the worker > classpath that the worker just does not need. Jetty for example. I think > in these cases it seems simpler to have the worker process/client process > and it's dependencies separated out from other storm system daemons. And > then only look at relocating the dependencies the client/worker uses. > +1. This makes the most sense to me. > > --Bobby > > From: "Cody A. Ray" <[email protected]<mailto:[email protected]>> > Reply-To: "[email protected]<mailto: > [email protected]>" <[email protected]<mailto: > [email protected]>> > Date: Thursday, February 6, 2014 at 9:09 PM > To: "[email protected]<mailto: > [email protected]>" <[email protected]<mailto: > [email protected]>> > Cc: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > Subject: Re: http-client version conflict > > +1 for relocating storm's dependencies. I just ran into this a couple > weeks ago and have had to side-shelf the project until I find a > work-around. Will check-out the shade plugin. Thanks for the rec! > > -Cody > > > On Thu, Feb 6, 2014 at 9:07 PM, P. Taylor Goetz <[email protected]<mailto: > [email protected]>> wrote: > I tend to agree. > > The move to maven makes package relocation very easy, but we should be > careful not to abuse it if that's a direction we decide to take. > > IMHO, users having to modify the contents of $STORM_HOME/lib is really bad > and should only be done as a last resort. If someone asks for help with an > issue after they've modified the contents of $STORM_HOM/lib, all bets are > off. > > I also agree that we need to keep a close watch on our dependencies, and > that some might warrant re-evaluation. > > I'd love to others opinions. > > - Taylor > > On Feb 6, 2014, at 9:28 PM, Adam Lewis <[email protected]<mailto: > [email protected]>> wrote: > > My $0.02 on this subject: > > Without going down the path of class loader or OSGi mania and becoming a > full container, I'd definitely be in favor of storm relocating its own > dependencies. In this way edge cases around things like reflection can be > handled once within storm rather than burdening every topology builder with > those details. Part of the problem seems to be that storm makes extensive > use (directly or transitively) of a lot of go-to utility libraries like > guava, thrift, jodatime, json-simple, snakeyaml, commons-io, etc... I'm > sure that leveraging these libraries allowed storm's development to proceed > rapidly, but from a maturity perspective, it is problematic to impose these > version choices on users. And while I might want Storm to, say, try to > track the latest Guava version, that same policy could be very problematic > for others. > > If storm can relocate even some of its own dependencies, I think that > would be a great help to me at least. Longer term, I wonder how much of > some of these libraries are really being used. For example, is clj-time > (and by extension joda-time) really needed? Or just a small fraction of the > functionality in that library? I can probably pitch in some of the effort > required to do this, if this is the direction people want to go in. > > > > > On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz <[email protected]<mailto: > [email protected]>> wrote: > I'm glad the shader plugin worked for you. > > Updating dependencies can be tricky as it can easily introduce regressions. > > Ultimately we need to figure out the best solution to avoiding conflicts > between user code (i.e. dependencies in topology jar files) and Storm's > libraries. > > The classloader approach has been attempted, but IMO Storm's use of > serialization complicates things significantly. Package relocation seems to > be a relatively lightweight solution. > > If that's a direction we pursue, then it introduces the question of > whether Storm should relocate its dependencies, or if that should be left > up to the user (topology developer). > > Elastic Search has gone down the path of relocating some of their > dependencies [1] (not necessarily an endorsement, just an observation). > > I've CC'd dev@ since this is all related to the infamous issue #115, > which is now STORM-129 [2]. > > - Taylor > > [1] > https://github.com/elasticsearch/elasticsearch/blob/master/pom.xml#L474 > [2] https://issues.apache.org/jira/browse/STORM-129 > > > > > > On Feb 6, 2014, at 7:25 PM, Vinay Pothnis <[email protected]<mailto: > [email protected]>> wrote: > > Thank you all for replies! The shader-plugin solution seems to work for us. > > I wonder if we can create a JIRA ticket for storm to upgrade the > http-client library as part of their next release. > > -Vinay > > > On Thu, Feb 6, 2014 at 2:38 PM, Michael Rose <[email protected] > <mailto:[email protected]>> wrote: > We've done this with SLF4j and Guava as well without issues. > > > Michael Rose (@Xorlev<https://twitter.com/xorlev>) > Senior Platform Engineer, FullContact<http://www.fullcontact.com/> > [email protected]<mailto:[email protected]> > > > On Thu, Feb 6, 2014 at 3:03 PM, Mark Greene <[email protected]<mailto: > [email protected]>> wrote: > We had this problem as well. We modified our chef cookbook to just replace > the older version with the newer one and storm didn't complain or have any > other issues as a result. > > > On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz <[email protected] > <mailto:[email protected]>> wrote: > Your best bet is probably to use the shade plugin to relocate the > http-client package so it doesn't conflict with the version storm uses. > > Storm does this with the libtrhift dependency in storm-core: > > > https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220 > > (You can ignore the clojure transformer in that config, unless you have > non-AOT clojure code that uses the http-client library). > > More information on using the shade plugin to do package relocations can > be found here: > > > http://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html > > - Taylor > > On Feb 4, 2014, at 4:27 PM, Vinay Pothnis <[email protected]<mailto: > [email protected]>> wrote: > > > Hello, > > > > I am using storm version 0.9.0.1. > > My application depends on apache http-client version 4.3.2 - but storm > depends on http-client version 4.1.1. > > > > What is the best way to override this dependency? > > > > Thanks > > Vinay > > > > > > > > > > > -- > Cody A. Ray, LEED AP > [email protected]<mailto:[email protected]> > 215.501.7891 >
