STORM-196 is an issue affecting anyone trying to use STORM_JAR_JVM_OPTS. The PR on github is concise and ready to go: https://github.com/apache/incubator-storm/pull/27
On Mon, Mar 31, 2014 at 1:46 PM, Milinda Pathirage <[email protected]>wrote: > Hi Taylor, > > Is there a fork or branch with Clojure 1.5 support? Or is this just a > change of Clojure version in master branch? > > Thanks > Milinda > > On Thu, Mar 27, 2014 at 1:05 AM, P. Taylor Goetz <[email protected]> > wrote: > > As promised, binaries can be found here: > > > > > http://people.apache.org/~ptgoetz/storm/storm-0.9.2-incubating/dep-frankenbuild/ > > > > This is not a release. It is merely a convenience build provided to > enable further testing among the community. > > > > -Taylor > > > > On Mar 26, 2014, at 9:54 PM, P. Taylor Goetz <[email protected]> wrote: > > > >> I've submitted pull requests for most of these changes (I think I still > need to submit one for http-client), where there wasn't already one. > >> > >> The only problem I saw was with upgrading to clojure 1.6.0. IIRC it had > to do with ns conflicts with "some.*". Falling back to 1.5.1 solved it. I > didn't investigate it. But it seems upgrading to 1.6.x will require some > code change. > >> > >> Anyway I created a snapshot build with all these updates pulled in. All > unit tests pass, and my basic fully distributed mode smoke tests pass, so > I'm leaning toward a +1 for all. > >> > >> However, if anyone who is interested in seeing these upgrades merged > has the time and resources to try it out, that would be a huge help to us. > >> > >> I'll post the binaries soon, when I have access to a real computer. > >> > >> -Taylor > >> > >>> On Mar 26, 2014, at 10:49 AM, Brian O'Neill <[email protected]> > wrote: > >>> > >>> > >>> I’m +1 on all of those (including the clojure version bump). > >>> > >>> We’ve been hit with the maven-clojure-plugin issue as well, with no > real > >>> way to diagnose. > >>> > >>> -brian > >>> > >>> --- > >>> Brian O'Neill > >>> Chief Technology Officer > >>> > >>> Health Market Science > >>> The Science of Better Results > >>> 2700 Horizon Drive · King of Prussia, PA · 19406 > >>> M: 215.588.6024 · @boneill42 <http://www.twitter.com/boneill42> · > >>> healthmarketscience.com > >>> > >>> This information transmitted in this email message is for the intended > >>> recipient only and may contain confidential and/or privileged > material. If > >>> you received this email in error and are not the intended recipient, or > >>> the person responsible to deliver it to the intended recipient, please > >>> contact the sender at the email above and delete this email and any > >>> attachments and destroy any copies thereof. Any review, retransmission, > >>> dissemination, copying or other use of, or taking any action in > reliance > >>> upon, this information by persons or entities other than the intended > >>> recipient is strictly prohibited. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>>> On 3/26/14, 10:09 AM, "P. Taylor Goetz" <[email protected]> wrote: > >>>> > >>>> There are a number of email threads and JIRA issues regarding > upgrading > >>>> various storm dependencies, so I’d like to enumerate them and discuss > >>>> them in one thread. > >>>> > >>>> Here’s the list so far: > >>>> > >>>> 1. Kryo/Carbonite (STORM-263)[1] > >>>> 2. Clojure (STORM-265) [2] > >>>> 3. commons-io (STORM-258) [3] > >>>> 4. curator (STORM-252) [4] > >>>> 5. http-client [5] > >>>> > >>>> I am +1 for all of the above, with the exception of #2, which I am +0 > >>>> only because I personally haven’t had a chance to do any testing with > >>>> newer versions of clojure. I’d be interested to hear if anyone has > done > >>>> any testing with newer versions of clojure. > >>>> > >>>> I think we should at least consider a bump to clojure 1.5 since it > >>>> includes Bobby Evan’s patch that fixes error output getting swallowed > >>>> (this manifests itself as the maven-clojure-plugin failing without any > >>>> useful information when there are certain AOT compilation issues — > very > >>>> annoying). > >>>> > >>>> - Taylor > >>>> > >>>> [1] https://issues.apache.org/jira/browse/STORM-263 > >>>> [2] https://issues.apache.org/jira/browse/STORM-265 > >>>> [3] https://issues.apache.org/jira/browse/STORM-258 > >>>> [4] https://issues.apache.org/jira/browse/STORM-252 > >>>> [5] > >>>> > http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CA732B22 > >>>> [email protected]%3E > >>>> > >>>>> On Mar 26, 2014, at 7:13 AM, Brian O'Neill <[email protected]> > wrote: > >>>>> > >>>>> Agreed. One of our guys got hung up on that just yesterday. > >>>>> It isn’t hard to track down the assembly, but it might be worth > making > >>>>> it a bit easier. > >>>>> > >>>>> Also, I’d love to see STORM-263 included in the next release. It > >>>>> seems like a quick win. > >>>>> > >>>>> -brian > >>>>> > >>>>> --- > >>>>> Brian O'Neill > >>>>> Chief Technology Officer > >>>>> > >>>>> Health Market Science > >>>>> The Science of Better Results > >>>>> 2700 Horizon Drive · King of Prussia, PA · 19406 > >>>>> M: 215.588.6024 · @boneill42 · healthmarketscience.com > >>>>> > >>>>> This information transmitted in this email message is for the > intended > >>>>> recipient only and may contain confidential and/or privileged > material. > >>>>> If you received this email in error and are not the intended > recipient, > >>>>> or the person responsible to deliver it to the intended recipient, > >>>>> please contact the sender at the email above and delete this email > and > >>>>> any attachments and destroy any copies thereof. Any review, > >>>>> retransmission, dissemination, copying or other use of, or taking any > >>>>> action in reliance upon, this information by persons or entities > other > >>>>> than the intended recipient is strictly prohibited. > >>>>> > >>>>>> On Mar 26, 2014, at 2:19 AM, Kang Xiao <[email protected]> > wrote: > >>>>>> > >>>>>> hi guys > >>>>>> > >>>>>> How about adding a bin/build_release.sh (http://build_release.sh) > >>>>>> script in this 0.9.2-incubating release? > >>>>>> > >>>>>> Since some guys asked the question about building storm release > >>>>>> package more than once in the mail list. > >>>>>> > >>>>>> -- > >>>>>> Best Regards! > >>>>>> > >>>>>> 肖康(Kang Xiao,<[email protected] (mailto:[email protected] > )>) > >>>>>> Distributed Software Engineer > >>>>>> > >>>>>>> 在 2014年3月25日 星期二，1:38，Suresh Srinivas 写道： > >>>>>>> Taylor, I am not very clear on "Lazy consensus +2". By this > >>>>>>> definition code > >>>>>>> can committed with no +1 from a committer, right? > >>>>>>> > >>>>>>> You may want to look at Hadoop bylaws - > >>>>>>> https://hadoop.apache.org/bylaws.html > >>>>>>> > >>>>>>> The code commits in Hadoop are consensus approval with minimum +1 > >>>>>>> from an > >>>>>>> active committer and no veto. This has worked well in my > experience. > >>>>>>> It may > >>>>>>> be a good idea to also adapt minimum 3 +1s from active committers > for > >>>>>>> merging feature branches. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Mar 21, 2014 at 6:22 PM, P. Taylor Goetz < > [email protected] > >>>>>>> (mailto:[email protected])> wrote: > >>>>>>> > >>>>>>>> That brings up a good point that probably deserves a separate > thread. > >>>>>>>> > >>>>>>>> We should establish by-laws soon. Specifically a commit/merge > policy. > >>>>>>>> > >>>>>>>> For code changes, I've been operating under a "lazy consensus +2" > >>>>>>>> model: 2 > >>>>>>>> committer +1 votes and no vetoes (-1). If a committer submits the > >>>>>>>> patch, > >>>>>>>> that's an implicit +1. Unless it's a somewhat urgent fix, I've > been > >>>>>>>> waiting > >>>>>>>> for 3 binding votes and no vetoes. > >>>>>>>> > >>>>>>>> That's kind of a middle ground between the traditional code > >>>>>>>> modification > >>>>>>>> rule and lazy consensus [1]. > >>>>>>>> > >>>>>>>> When wearing my "release manager" hat, I've also interpreted "code > >>>>>>>> change" > >>>>>>>> to mean "anything that alters the behavior of the software we > >>>>>>>> produce." In > >>>>>>>> terms of the build/packaging I've been a little looser. For large > >>>>>>>> changes > >>>>>>>> (e.g. The switch to maven), I've waited for 3 binding votes. For > some > >>>>>>>> changes I've committed directly -- I don't think we need to have a > >>>>>>>> 3-day > >>>>>>>> vote on updating the CHANGELOG, for example. > >>>>>>>> > >>>>>>>> Anyway, it's something to think about. Sorry for hijacking the > >>>>>>>> thread. > >>>>>>>> > >>>>>>>> +1 (again ;) ) > >>>>>>>> > >>>>>>>> Taylor > >>>>>>>> > >>>>>>>> [1] https://www.apache.org/foundation/voting.html > >>>>>>>> > >>>>>>>>> On Mar 21, 2014, at 7:48 PM, Nathan Marz <[email protected] > >>>>>>>>> (mailto:[email protected])> wrote: > >>>>>>>>> > >>>>>>>>> Let's get https://issues.apache.org/jira/browse/STORM-262 in > there. > >>>>>>>>> Just > >>>>>>>>> one more vote needed by a committer. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On Fri, Mar 21, 2014 at 3:21 PM, Patrick Lucas <[email protected] > >>>>>>>>>> (mailto:[email protected])> wrote: > >>>>>>>>>> > >>>>>>>>>> A fix STORM-120 would be greatly appreciated. It's making it > >>>>>>>>>> impossible > >>>>>>>> to > >>>>>>>>>> increase tasks/executors > 1 when there is a downstream shuffle > >>>>>>>>> > >>>>>>>> > >>>>>>>> grouping. > >>>>>>>>>> > >>>>>>>>>> I'm not sure why there haven't been more reports of problems > with > >>>>>>>>>> it. > >>>>>>>> Two > >>>>>>>>>> possibilities I can think of are that we are using exclusively > >>>>>>>>>> shell > >>>>>>>>>> components--perhaps there's a root-cause bug in those component > >>>>>>>>>> classes--and > >>>>>>>>>> that we are dealing with a high volume stream of large tuples. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> (thousands / > >>>>>>>>>> sec, KB in size) > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Mar 20, 2014 at 2:14 PM, P. Taylor Goetz > >>>>>>>>>> <[email protected] (mailto:[email protected])> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Never mind... just found it. > >>>>>>>>>>> > >>>>>>>>>>>> On Mar 20, 2014, at 5:09 PM, P. Taylor Goetz < > [email protected] > >>>>>>>>>>>> (mailto:[email protected])> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Derek do you have an idea for a fix? > >>>>>>>>>>>> > >>>>>>>>>>>> On Mar 20, 2014, at 3:43 PM, Derek Dagit < > [email protected] > >>>>>>>>>>>> (mailto:[email protected])> > >>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>>> As I said above, this fix is the most important in my > opinion. > >>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether > >>>>>>>>>>>>>> it's as > >>>>>>>>>>>>>> important as STORM-187 or not. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yeah, we found it recently, and I created it this morning > after > >>>>>>>>>> reading > >>>>>>>>>>> Taylor's mail. > >>>>>>>>>>>>> > >>>>>>>>>>>>> STORM-187 can be a problem with fewer than 30 retries > >>>>>>>>>>>>> (likelihood > >>>>>>>>>>> depends on configuration), but we will hit STORM-259 when > retries > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> exceeds > >>>>>>>>>>> 30. > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Derek > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On 3/20/14, 14:18, Michael G. Noll wrote: > >>>>>>>>>>>>>> On my side the most important change is, as you point out, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> STORM-187. > >>>>>>>>>>>>>> The primary reason is like Adam Lewis is pointing out > because > >>>>>>>>>>>>>> it's a > >>>>>>>>>>>>>> stability problem. The secondary aspect is that this issue > >>>>>>>>>>>>>> taints > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> the > >>>>>>>>>>>>>> new Netty backend, and at least IMHO the faster Storm could > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> confidently > >>>>>>>>>>>>>> bury ZeroMQ the better. :-) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As I said above, this fix is the most important in my > opinion. > >>>>>>>>>>>>>> STORM-259 (Random#nextInt) is new to me -- can't say whether > >>>>>>>>>>>>>> it's as > >>>>>>>>>>>>>> important as STORM-187 or not. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Switching to my non-essential wishlist I'd also +1 STORM-252 > >>>>>>>> (Upgrade > >>>>>>>>>>>>>> Curator and thus ZooKeeper to 3.4.5). We have been running > ZK > >>>>>>>>>>>>>> 3.4.5 > >>>>>>>>>>>>>> anyway for a couple of reasons, and it would be nice to have > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> official > >>>>>>>>>>>>>> Storm support for the latest ZK version (ok, the recently > >>>>>>>>>>>>>> released > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> ZK > >>>>>>>>>>>>>> 3.4.6 is actually the latest but hey). Although I don't know > >>>>>>>>>>>>>> how > >>>>>>>>>>>>>> confident we are that the code in STORM-252 actually works, > >>>>>>>>>>>>>> i.e. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> whether > >>>>>>>>>>>>>> integrating STORM-252 into 0.9.2 on such short notice would > be > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> jumping > >>>>>>>>>>>>>> the gun or a safe move. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Btw, in terms of Storm/Kafka integration Kafka is in the > same > >>>>>>>>>>>>>> boat: > >>>>>>>>>>>>>> it's built against ZK 3.3.x, and LinkedIn recommends the use > >>>>>>>>>>>>>> of ZK > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> 3.3.4 > >>>>>>>>>>>>>> in the docs. There's an open ticket KAFKA-854 [1] that's > >>>>>>>>>>>>>> basically > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> the > >>>>>>>>>>>>>> equivalent of STORM-252, but I'm not sure how actively the > >>>>>>>>>>>>>> Kafka > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> team > >>>>>>>>>>> is > >>>>>>>>>>>>>> working on that. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>> Michael > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-854 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 03/20/2014 02:33 AM, P. Taylor Goetz wrote: > >>>>>>>>>>>>>>> I'd like to get this discussion started, largely because > the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> "negative timeout" bug (STORM-187) really bothers me. I've not > >>>>>>>>>>> seen it > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> in > >>>>>>>>>>> the wild, but I've heard of a few cases where it was enough to > >>>>>>>>>>> hinder > >>>>>>>>>>> upgrading. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> HEAD looks good to me at the moment, with the major > difference > >>>>>>>> being > >>>>>>>>>>> the zookeeper update and the patch mentioned above. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Any thoughts on other PRs or patches to include? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -Taylor > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Patrick Lucas > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Twitter: @nathanmarz > >>>>>>>>> http://nathanmarz.com > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> http://hortonworks.com/download/ > >>>>>>> > >>>>>>> -- > >>>>>>> CONFIDENTIALITY NOTICE > >>>>>>> NOTICE: This message is intended for the use of the individual or > >>>>>>> entity to > >>>>>>> which it is addressed and may contain information that is > >>>>>>> confidential, > >>>>>>> privileged and exempt from disclosure under applicable law. If the > >>>>>>> reader > >>>>>>> of this message is not the intended recipient, you are hereby > >>>>>>> notified that > >>>>>>> any printing, copying, dissemination, distribution, disclosure or > >>>>>>> forwarding of this communication is strictly prohibited. If you > have > >>>>>>> received this communication in error, please contact the sender > >>>>>>> immediately > >>>>>>> and delete it from your system. Thank You. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > > > > > > -- > Milinda Pathirage > > PhD Student | Research Assistant > School of Informatics and Computing | Data to Insight Center > Indiana University > > twitter: milindalakmal > skype: milinda.pathirage > blog: http://milinda.pathirage.org > -- Patrick Lucas
