I've submitted pull requests for most of these changes (I think I still need to submit one for http-client), where there wasn't already one.
The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I didn't investigate it. But it seems upgrading to 1.6.x will require some code change. Anyway I created a snapshot build with all these updates pulled in. All unit tests pass, and my basic fully distributed mode smoke tests pass, so I'm leaning toward a +1 for all. However, if anyone who is interested in seeing these upgrades merged has the time and resources to try it out, that would be a huge help to us. I'll post the binaries soon, when I have access to a real computer. -Taylor > On Mar 26, 2014, at 10:49 AM, Brian O'Neill <[email protected]> wrote: > > > I’m +1 on all of those (including the clojure version bump). > > We’ve been hit with the maven-clojure-plugin issue as well, with no real > way to diagnose. > > -brian > > --- > Brian O'Neill > Chief Technology Officer > > Health Market Science > The Science of Better Results > 2700 Horizon Drive · King of Prussia, PA · 19406 > M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> · > healthmarketscience.com > > This information transmitted in this email message is for the intended > recipient only and may contain confidential and/or privileged material. If > you received this email in error and are not the intended recipient, or > the person responsible to deliver it to the intended recipient, please > contact the sender at the email above and delete this email and any > attachments and destroy any copies thereof. Any review, retransmission, > dissemination, copying or other use of, or taking any action in reliance > upon, this information by persons or entities other than the intended > recipient is strictly prohibited. > > > > > > > >> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <[email protected]> wrote: >> >> There are a number of email threads and JIRA issues regarding upgrading >> various storm dependencies, so I’d like to enumerate them and discuss >> them in one thread. >> >> Here’s the list so far: >> >> 1. Kryo/Carbonite (STORM-263)[1] >> 2. Clojure (STORM-265) [2] >> 3. commons-io (STORM-258) [3] >> 4. curator (STORM-252) [4] >> 5. http-client [5] >> >> I am +1 for all of the above, with the exception of #2, which I am +0 >> only because I personally haven’t had a chance to do any testing with >> newer versions of clojure. I’d be interested to hear if anyone has done >> any testing with newer versions of clojure. >> >> I think we should at least consider a bump to clojure 1.5 since it >> includes Bobby Evan’s patch that fixes error output getting swallowed >> (this manifests itself as the maven-clojure-plugin failing without any >> useful information when there are certain AOT compilation issues — very >> annoying). >> >> - Taylor >> >> [1] https://issues.apache.org/jira/browse/STORM-263 >> [2] https://issues.apache.org/jira/browse/STORM-265 >> [3] https://issues.apache.org/jira/browse/STORM-258 >> [4] https://issues.apache.org/jira/browse/STORM-252 >> [5] >> http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22 >> [email protected]%3E >> >>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <[email protected]> wrote: >>> >>> Agreed. One of our guys got hung up on that just yesterday. >>> It isn’t hard to track down the assembly, but it might be worth making >>> it a bit easier. >>> >>> Also, I’d love to see STORM-263 included in the next release. It >>> seems like a quick win. >>> >>> -brian >>> >>> --- >>> Brian O'Neill >>> Chief Technology Officer >>> >>> Health Market Science >>> The Science of Better Results >>> 2700 Horizon Drive · King of Prussia, PA · 19406 >>> M: 215.588.6024 · @boneill42 · healthmarketscience.com >>> >>> This information transmitted in this email message is for the intended >>> recipient only and may contain confidential and/or privileged material. >>> If you received this email in error and are not the intended recipient, >>> or the person responsible to deliver it to the intended recipient, >>> please contact the sender at the email above and delete this email and >>> any attachments and destroy any copies thereof. Any review, >>> retransmission, dissemination, copying or other use of, or taking any >>> action in reliance upon, this information by persons or entities other >>> than the intended recipient is strictly prohibited. >>> >>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <[email protected]> wrote: >>>> >>>> hi guys >>>> >>>> How about adding a bin/build_release.sh (http://build_release.sh) >>>> script in this 0.9.2-incubating release? >>>> >>>> Since some guys asked the question about building storm release >>>> package more than once in the mail list. >>>> >>>> -- >>>> Best Regards! >>>> >>>> 肖康(Kang Xiao,<[email protected] (mailto:[email protected])>) >>>> Distributed Software Engineer >>>> >>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道： >>>>> Taylor, I am not very clear on "Lazy consensus +2". By this >>>>> definition code >>>>> can committed with no +1 from a committer, right? >>>>> >>>>> You may want to look at Hadoop bylaws - >>>>> https://hadoop.apache.org/bylaws.html >>>>> >>>>> The code commits in Hadoop are consensus approval with minimum +1 >>>>> from an >>>>> active committer and no veto. This has worked well in my experience. >>>>> It may >>>>> be a good idea to also adapt minimum 3 +1s from active committers for >>>>> merging feature branches. >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz <[email protected] >>>>> (mailto:[email protected])> wrote: >>>>> >>>>>> That brings up a good point that probably deserves a separate thread. >>>>>> >>>>>> We should establish by-laws soon. Specifically a commit/merge policy. >>>>>> >>>>>> For code changes, I've been operating under a "lazy consensus +2" >>>>>> model: 2 >>>>>> committer +1 votes and no vetoes (-1). If a committer submits the >>>>>> patch, >>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've been >>>>>> waiting >>>>>> for 3 binding votes and no vetoes. >>>>>> >>>>>> That's kind of a middle ground between the traditional code >>>>>> modification >>>>>> rule and lazy consensus [1]. >>>>>> >>>>>> When wearing my "release manager" hat, I've also interpreted "code >>>>>> change" >>>>>> to mean "anything that alters the behavior of the software we >>>>>> produce." In >>>>>> terms of the build/packaging I've been a little looser. For large >>>>>> changes >>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For some >>>>>> changes I've committed directly -- I don't think we need to have a >>>>>> 3-day >>>>>> vote on updating the CHANGELOG, for example. >>>>>> >>>>>> Anyway, it's something to think about. Sorry for hijacking the >>>>>> thread. >>>>>> >>>>>> +1 (again ;) ) >>>>>> >>>>>> Taylor >>>>>> >>>>>> [1] https://www.apache.org/foundation/voting.html >>>>>> >>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <[email protected] >>>>>>> (mailto:[email protected])> wrote: >>>>>>> >>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in there. >>>>>>> Just >>>>>>> one more vote needed by a committer. >>>>>>> >>>>>>> >>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <[email protected] >>>>>>>> (mailto:[email protected])> wrote: >>>>>>>> >>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it >>>>>>>> impossible >>>>>> to >>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle >>>>>>> >>>>>> >>>>>> grouping. >>>>>>>> >>>>>>>> I'm not sure why there haven't been more reports of problems with >>>>>>>> it. >>>>>> Two >>>>>>>> possibilities I can think of are that we are using exclusively >>>>>>>> shell >>>>>>>> components--perhaps there's a root-cause bug in those component >>>>>>>> classes--and >>>>>>>> that we are dealing with a high volume stream of large tuples. >>>>>>>> >>>>>>> >>>>>> >>>>>> (thousands / >>>>>>>> sec, KB in size) >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz >>>>>>>> <[email protected] (mailto:[email protected])> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Never mind... just found it. >>>>>>>>> >>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz <[email protected] >>>>>>>>>> (mailto:[email protected])> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Derek do you have an idea for a fix? >>>>>>>>>> >>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit <[email protected] >>>>>>>>>> (mailto:[email protected])> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>>>> As I said above, this fix is the most important in my opinion. >>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether >>>>>>>>>>>> it's as >>>>>>>>>>>> important as STORM-187 or not. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yeah, we found it recently, and I created it this morning after >>>>>>>> reading >>>>>>>>> Taylor's mail. >>>>>>>>>>> >>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries >>>>>>>>>>> (likelihood >>>>>>>>> depends on configuration), but we will hit STORM-259 when retries >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> exceeds >>>>>>>>> 30. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Derek >>>>>>>>>>> >>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote: >>>>>>>>>>>> On my side the most important change is, as you point out, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> STORM-187. >>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out because >>>>>>>>>>>> it's a >>>>>>>>>>>> stability problem. The secondary aspect is that this issue >>>>>>>>>>>> taints >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> the >>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> confidently >>>>>>>>>>>> bury ZeroMQ the better. :-) >>>>>>>>>>>> >>>>>>>>>>>> As I said above, this fix is the most important in my opinion. >>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether >>>>>>>>>>>> it's as >>>>>>>>>>>> important as STORM-187 or not. >>>>>>>>>>>> >>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252 >>>>>> (Upgrade >>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running ZK >>>>>>>>>>>> 3.4.5 >>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> official >>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently >>>>>>>>>>>> released >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> ZK >>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know >>>>>>>>>>>> how >>>>>>>>>>>> confident we are that the code in STORM-252 actually works, >>>>>>>>>>>> i.e. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> whether >>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would be >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> jumping >>>>>>>>>>>> the gun or a safe move. >>>>>>>>>>>> >>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the same >>>>>>>>>>>> boat: >>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use >>>>>>>>>>>> of ZK >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> 3.3.4 >>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's >>>>>>>>>>>> basically >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> the >>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the >>>>>>>>>>>> Kafka >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> team >>>>>>>>> is >>>>>>>>>>>> working on that. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Michael >>>>>>>>>>>> >>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote: >>>>>>>>>>>>> I'd like to get this discussion started, largely because the >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not >>>>>>>>> seen it >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> in >>>>>>>>> the wild, but I've heard of a few cases where it was enough to >>>>>>>>> hinder >>>>>>>>> upgrading. >>>>>>>>>>>>> >>>>>>>>>>>>> HEAD looks good to me at the moment, with the major difference >>>>>> being >>>>>>>>> the zookeeper update and the patch mentioned above. >>>>>>>>>>>>> >>>>>>>>>>>>> Any thoughts on other PRs or patches to include? >>>>>>>>>>>>> >>>>>>>>>>>>> -Taylor >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Patrick Lucas >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Twitter: @nathanmarz >>>>>>> http://nathanmarz.com >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> http://hortonworks.com/download/ >>>>> >>>>> -- >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to >>>>> which it is addressed and may contain information that is >>>>> confidential, >>>>> privileged and exempt from disclosure under applicable law. If the >>>>> reader >>>>> of this message is not the intended recipient, you are hereby >>>>> notified that >>>>> any printing, copying, dissemination, distribution, disclosure or >>>>> forwarding of this communication is strictly prohibited. If you have >>>>> received this communication in error, please contact the sender >>>>> immediately >>>>> and delete it from your system. Thank You. >>>>> >>>>> >>>>> >>>> >>>> >>> >> > >
