Do we have a clear date to release Storm 2.0 beta version? I saw some users expecting to use Java8.
BTW. Could somebody help to update the Storm site? The 2.0.0-SNAPSHOT documents should be updated. Thanks. - Xin 2017-06-29 23:27 GMT+08:00 Jungtaek Lim <kabh...@gmail.com>: > FYI I just gave it a try with separating 1.x branch and 1.1.x branch (sure > experimental in forked repo) > > https://github.com/heartsavior/storm/tree/1.1.x-branch-experimental > > I've updated CHANGELOG only once in that branch so you can see full of the > changelog which contains the issues ported back. > > https://github.com/HeartSaVioR/storm/commit/81e9d65793abc5defc0ab83c09b26a > 7dcba7e0eb > > Most of the issues are classified to the bug fix, but there're also some > issues filtered out. > (For example, wildcard classpath, refactor storm-autocreds, binary > storm-redis state, and so on) > > Please let me know what's your opinion on filtering out non-bugfix issues > from 1.1.1. > If there's no objection I'll do the change: > - rename the branch to 1.1.x-branch and push > - change the version of 1.x-branch to 1.2.0-SNAPSHOT > - reflect the version change to JIRA issues > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2017년 6월 29일 (목) 오전 6:41, Alexandre Vermeerbergen < > avermeerber...@gmail.com>님이 > 작성: > > > Hi Hugo, > > > > Thanks for your concerns about our troubles with the new > > storm-kafka-client. > > > > Our "bench" is based on our live production data of our cloud supervision > > system, collecting at least 1million metrics/min in our Kafka Brokers > > cluster (currently based on Kafka 0.10.1.0, with "compatibility flag > > active"). > > > > More details are available in the thread in the same dev list entitled > "Lag > > issues using Storm 1.1.1 latest build with StormKafkaClient 1.1.1 vs old > > StormKafka spouts". > > > > The latest post on this thread from Stig Døssing is giving me back some > > hope to see some progress in understanding our issues. > > > > My point about writing our own Spout come from our past experience: we've > > been using Kafka for a very long time in our supervision application. Way > > before we decided to use Storm, we had our own Java daemons consuming the > > same topics as today, doing some evaluation and writing them into an > > in-memory store for later consumption by our web services - see this as > > a"poor man's streaming system" ;-) In this legacy code, the part in > charge > > of consuming data from Kafka wasn't the most complex which we had: a > small > > pool of threads using the old Kafka consumer API... so maybe I'm wrong, > but > > for *this* purpose I do not feel like writing a Spout consuming a few > > topics to be a big effort. But of course, if we do that, then we'll miss > > the fancy integration in StormUI, flux, and the ability to subscribe to > > multiple topics based on a wildcard expression. > > > > So we're going to carefully dig into Stig's answer, and probably provide > > more details on our bench before jumping into our home-brewed Kafka > > spout... then I'll have to make some decision based on how much progress > we > > have vs time remaining before our next delivery gate. > > > > Hope it clarifies my position with regard to storm-kafka-client > > > > Best regards, > > Alexandre > > > > > > > > > > > > > > 2017-06-28 22:49 GMT+02:00 Hugo Da Cruz Louro <hlo...@hortonworks.com>: > > > > > Hi Alexandre, > > > > > > In my benchmarks the storm-kafka-client spout improves throughput by > 70% > > > and latency by 40% vs the storm-kafka implementation. I am surprised by > > > your findings substantiating the opposite. Can you share your benchmark > > > where you compare the performances of both implementations? > > > > > > As for you writing your own version of the spout. Why not contribute to > > > this one instead? Do you think the implementation is that poor? If so, > > why > > > do you think it is that poor? Do you expect your first version to be > much > > > better than a version that is already in production in several > customers, > > > and seems to be working fairly well? > > > > > > All the bugs found so far have been addressed. Since it’s a new feature > > > there may be a few bugs - it is expected. However, I don’t think that > it > > is > > > as bad as you make it sound as there are several people using it in > > > production for extended periods of time. > > > > > > Cheers > > > > > > > On Jun 28, 2017, at 12:04 PM, Alexandre Vermeerbergen < > > > avermeerber...@gmail.com> wrote: > > > > > > > > Hello, > > > > > > > > If that matters, our current experiences with StormKafkaClient > > > > isdisappointing (see my recent posts "Lag issues using Storm 1.1.1 > > latest > > > > build with StormKafkaClient 1.1.1 vs old StormKafka spouts" in this > > > mailing > > > > list). > > > > > > > > Our current experience is that the old StormKafka spout always beats > > the > > > > new one in term of performance & stability. > > > > > > > > Therefore, I am surprised when I see talks about deprecation of the > old > > > > StormKafka spout when the new one which just came "General Available" > > > with > > > > Storm 1.1.0, is not stable, and it's not better when we try it from > > > current > > > > 1.1.x builds to take into account recently closed JIRAs. > > > > > > > > We're even considering writing our own Kafka spout with Kafka 0.10.x > > API > > > to > > > > overcome the incompatibility of the old StormKafka spout with Kafka > > 0.10 > > > > libraries. > > > > > > > > Thus, for people which are comfortable with old Kafka spout, I'like > to > > > give > > > > a -1 (non binding) to the proposal of withdrawal of the old > StormKafka > > > > spout until the new one converges. > > > > > > > > Best regards, > > > > Alexandre Vermeerbergen > > > > > > > > > > > > 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <ptgo...@gmail.com>: > > > > > > > >> > > > >>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro < > > > hlo...@hortonworks.com> > > > >> wrote: > > > >>> > > > >>> I still need to go over the entire discussion thread in more > detail, > > > but > > > >> one thing I would like to bring up right way is the proposal to > > > DEPRECATE, > > > >> and eventually remove, the KafkaSpout with the old Kafka Consumer > > APIs. > > > The > > > >> storm-kafka-client KafkaSpout is getting stabilized, and I think we > > are > > > all > > > >> in agreement that the storm-kafka KafkaSpout has presented > continuous > > > >> maintainability problems with some fixes that got in not being > > backwards > > > >> compatible. > > > >> > > > >> I’m fine with deprecating the old KafkaSpout, but I feel the > decision > > to > > > >> actually remove it needs to take into account the user community. > The > > > main > > > >> sticking point here is compatibility with earlier versions of Kafka. > > > Like > > > >> with JDK versions, there are many valid reasons whey users may not > be > > > in a > > > >> position to upgrade to a newer version of Kafka. Outright removal > > could > > > >> leave some users in the lurch. > > > >> > > > >> Ideally, we could just poll the user community to get an idea of how > > > much > > > >> of the user base depends on the old KafkaSpout and use the results > to > > > guide > > > >> our decision. Unfortunately, at least in my past experience, polling > > the > > > >> user@ list doesn’t elicit much of a response and the results don’t > > > >> provide an accurate view of the user community. > > > >> > > > >> > > > >>> > > > >>> I am pretty confident how things are looking at this point for the > > > >> KafkaSpout. The Trident Kafka Spout is likely in between alpha and > > beta, > > > >> and that should be taken into account. I just recently submitted a > PR< > > > >> https://github.com/apache/storm/pull/2174> with some improvements > to > > > the > > > >> Trident Kafka Spout (including the refactoring done to support > manual > > > >> partition assignment), and there are some customers using it in > > > >> pre-production. However, it definitely would benefit from some more > > > testing. > > > >>> > > > >>> Thanks, > > > >>> Hugo > > > >> > > > >> -Taylor > > > >> > > > >> > > > >>> > > > >>> On Jun 28, 2017, at 7:48 AM, Bobby Evans > <ev...@yahoo-inc.com.INVALID > > < > > > >> mailto:ev...@yahoo-inc.com.INVALID>> wrote: > > > >>> > > > >>> +1. > > > >>> If the 1.1 and 1.2 lines start to become difficult to maintain we > can > > > >> look at putting them in maintenance mode too once we have a 2.x > > release. > > > >>> I am a little nervous about merging a new feature into 1.x branch > > > >> without first going to master, but I hope that it will not be too > much > > > work > > > >> to port it to master, and I trust the devs on that branch to do the > > > right > > > >> thing. > > > >>> On a related note we have not done much with feature branches > before > > so > > > >> I am not sure what we want to do about merging in the new metrics > API > > > >> branch to 1.x. I know for me I have not had time to keep up with > the > > > >> development work going on there. I would at least like to have a > pull > > > >> request put up for review before we merge it in. This would fit > with > > > our > > > >> current bylaws that do not mention feature branches. If all of the > > > changes > > > >> have already followed the review process then technically I think it > > is > > > OK > > > >> to just merge it in, but I still would like to take some time to > look > > at > > > >> the changes, and especially the new APIs. > > > >>> > > > >>> - Bobby > > > >>> > > > >>> > > > >>> On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim < > > > >> kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote: > > > >>> > > > >>> That's great news that metrics work is ready! > > > >>> > > > >>> I'm +1 to Taylor's proposal, but in order to respect semantic > > > >> versioning, I > > > >>> propose some modifications from Taylor's proposal: > > > >>> > > > >>> - create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port > > back > > > >> only > > > >>> bug fixes to the 1.1.x-branch > > > >>> - change the target version of 1.x-branch to 1.2.0-SNAPSHOT > > > >>> > > > >>> If we also agree above, I would like to volunteer the back-port > work. > > > >>> > > > >>> Thanks, > > > >>> Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io<mailto:storm@ > > > >> harsha.io>>님이 작성: > > > >>> > > > >>> +1 for above stated approach on releasing 1.2.0 with metrics > > > >>> -Harsha > > > >>> > > > >>> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote: > > > >>> The work on metrics is ready for a pull request to 1.x-branch from > > the > > > >>> feature branch. I’ve held off because we haven’t reached consensus > > on a > > > >>> path forward with the 1.x release lines . > > > >>> > > > >>> I’d like to propose the following for the 1.x line: > > > >>> > > > >>> 1. Create a branch for 1.2 so we have a branch to review the > metrics > > > >>> stuff. > > > >>> 2. Release 1.1.1 > > > >>> 3. Review/merge metrics work. Port metrics to master. > > > >>> 4. Release 1.2.0 > > > >>> 5. Put the entire 1.x line into maintenance mode. Drop support for > > > 1.0.x. > > > >>> (we would only support 1.2.x and 1.1.x which are very closely > > aligned). > > > >>> > > > >>> Dropping support for 1.0.x line would eliminate the need to > maintain > > > one > > > >>> of the fairly heavily diverged branches. The 1.2.x and 1.1.x would > be > > > >>> very closely aligned. I just up merged metrics_v2 against > 1.x-branch > > > >>> after a while, and there were no conflicts. > > > >>> > > > >>> That would give us a little more bandwidth to focus on 2.0 and > needed > > > bug > > > >>> fixes to the 1.x line like some of the issues raised with > > > >>> storm-kafka-client. We could even start releasing alpha/beta > versions > > > of > > > >>> 2.0 in parallel to the steps above. > > > >>> > > > >>> Any thoughts on that approach? > > > >>> > > > >>> -Taylor > > > >>> > > > >>> > > > >>> On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabh...@gmail.com > <mailto: > > > kabh > > > >> w...@gmail.com>> wrote: > > > >>> > > > >>> Yes I prefer option 1, but it might depend on the progress of > metrics > > > >>> V2. > > > >>> If it can be done within predictable near future I'm OK to pick > > option > > > >>> 2, > > > >>> but if not, we may be better to focus releasing 2.0.0 and make it > > > >>> really > > > >>> happen. > > > >>> > > > >>> Whichever we go, I feel it's time to track remaining work on Storm > > > >>> 2.0.0. I > > > >>> found some bugs on master branch so filed issues, and we've > remaining > > > >>> port > > > >>> work (UI and logviewer). We've some other improvements target for > > > >>> 2.0.0: > > > >>> worker redesign, beam integration, and so on, and we don't track > its > > > >>> progress at all. I don't think we should wait for features which > > > >>> progress > > > >>> is not transparent (in other words we don't know when it will be > > > >>> finished). > > > >>> > > > >>> - Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgo...@gmail.com > <mailto: > > > ptgo > > > >> e...@gmail.com>>님이 작성: > > > >>> > > > >>> Bobby/Jungtaek, > > > >>> > > > >>> Are you saying you want to forego the 1.2 “metrics_v2” release and > > > >>> include > > > >>> it only in 2.0? (I ask because that work is already based on > > > >>> 1.x-branch, > > > >>> and forward-porting it to master is relatively simple.) I’d kind of > > > >>> like > > > >>> that work go out soon. > > > >>> > > > >>> If we go with option 1, I would want to see a 2.0 release (even if > > > >>> it’s a > > > >>> “beta” or “preview) before putting the 1.x line into maintenance > > mode. > > > >>> > > > >>> -Taylor > > > >>> > > > >>> On Jun 23, 2017, at 9:51 AM, Bobby Evans > <ev...@yahoo-inc.com.INVALID > > < > > > >> mailto:ev...@yahoo-inc.com.INVALID> > > > >>> > > > >>> wrote: > > > >>> > > > >>> I see 2 ways to address this. > > > >>> 1) We put the 1.x line into maintenance mode like with 0.10. We > > > >>> don't > > > >>> backport anything except bug fixes.2) We backport a lot of the > > > >>> backwards > > > >>> compatible changes from 2.x to 1.x. > > > >>> My personal preference is 1. It makes it clear the direction we > > > >>> want to > > > >>> go in. The biggest issue is that we probably would want to do a > 2.x > > > >>> release sooner rather then later. Even if we don't get all of the > > > >>> features > > > >>> that people want, if we just get a release out we can add in new > > > >>> features > > > >>> if they are backwards compatible, or we can create a 3.x line that > > > >>> would > > > >>> have the breaking changes in it. > > > >>> > > > >>> - Bobby > > > >>> > > > >>> > > > >>> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim < > > > >>> kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote: > > > >>> > > > >>> I'd like to bump this again instead of initiating new discussion > > > >>> thread. > > > >>> > > > >>> I had having hard time to create and apply pull requests for both > > > >>> master > > > >>> and 1.x-branch and that's really painful and sometimes blocker for > > > >>> me to > > > >>> do > > > >>> merge step. > > > >>> Two branches are heavily diverged more than between 0.10 and 1.0.0, > > > >>> even > > > >>> IDE can't switch between the branch smoothly. We didn't even > address > > > >>> checkstyle issue yet, but after addressing, it could be > "completely" > > > >>> diverged. JDK version is another major issue, since the pull > requests > > > >>> targeted for master branch are not checked against JDK 7, and some > of > > > >>> them > > > >>> make some issues regarding JDK version while porting back. > > > >>> > > > >>> So personally I really would like to see the plan for 1.x version > > > >>> line > > > >>> changed - skipping any minor releases including 1.2.0 - and have > epic > > > >>> issue > > > >>> for 2.0.0 and just go ahead. That was our proposed plan indeed. > (even > > > >>> proposed plan was having 2.0.0 directly after 1.0.0) > > > >>> > > > >>> Would like to hear everyone's opinions. If we have consensus to not > > > >>> having > > > >>> any minor releases for 1.x version line, I will not port back > > > >>> non-bugfix > > > >>> pull requests to 1.x-branch, and guide contributors to create pull > > > >>> requests > > > >>> against master branch, not 1.x version line. > > > >>> > > > >>> Thanks, > > > >>> Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen < > > > >>> avermeerber...@gmail.com<mailto:avermeerber...@gmail.com>>님이 > > > >>> 작성: > > > >>> > > > >>> +1 for Roshan's suggestion : in our Storm 1.x based supervision > > > >>> system, > > > >>> we're very interested anything that can provide better throughput. > > > >>> > > > >>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ros...@hortonworks.com > > <mailto: > > > >> ros...@hortonworks.com>>: > > > >>> > > > >>> For 2.0 beta … it would be good to incorporate some of the Worker > > > >>> improvements (STORM-2284) IMO. Changes to messaging subsystem can > > > >>> be > > > >>> delivered sooner and my in-progress implementation suggests that it > > > >>> will > > > >>> yield substantial latency improvements. The 2.0 beta phase would > > > >>> really > > > >>> help kick the tires on the revised messaging system and the > > > >>> performance > > > >>> improvements will also be a good incentive for trying out the 2.0 > > > >>> line. > > > >>> > > > >>> I notice multiple other bottlenecks that are holding back > > > >>> throughput a > > > >>> lot, which can be addressed in a subsequent 2.x minor release. > > > >>> -roshan > > > >>> > > > >>> > > > >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabh...@gmail.com<mailto:kabh > > > >> w...@gmail.com>> wrote: > > > >>> > > > >>> I also would love to see metrics V2 code sooner or later too. > > > >>> If we > > > >>> can get > > > >>> it before releasing 2.0.0 that will be great, and then maybe we > > > >>> could > > > >>> just > > > >>> move toward to 2.0.0, not adding any improvements to 1.x version > > > >>> line. > > > >>> (And that's what I would want to.) > > > >>> > > > >>> If we would really want to have 1.2.0, I suggest that we make > > > >>> the > > > >>> 1.1.1 > > > >>> version correct right now rather than after releasing 1.1.1. We > > > >>> also > > > >>> merged > > > >>> non-bugfix things to 1.x-branch but that's not what users > > > >>> expect. I > > > >>> agree > > > >>> that work may be painful, but anyway need to do it. > > > >>> > > > >>> - Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 6월 3일 (토) 오전 3:49, Bobby Evans > > > >>> <ev...@yahoo-inc.com.invalid<mailto:ev...@yahoo-inc.com.invalid> > > > >>> 님이 > > > >>> 작성: > > > >>> > > > >>> I would love to see the metrics V2 code come out sooner rather > > > >>> than > > > >>> later. +1. > > > >>> My biggest blocker for a 1.x release is > > > >>> https://github.com/apache/storm/pull/2142 Even though the pull > > > >>> request > > > >>> says it is minor it showed that we messed up pushing back some > > > >>> changes for > > > >>> pacemaker to open source (the code does not run at all which for > > > >>> me > > > >>> is a > > > >>> blocker) and I really want to get that fully fixed/tested before > > > >>> another > > > >>> release. > > > >>> As for 2.x I think we are very close to being able to so a 2.x > > > >>> alpha > > > >>> release. I would like to see metrics V2 merged in simply because > > > >>> it > > > >>> is a > > > >>> big change for user facing APIs. But after that I would love to > > > >>> see > > > >>> us > > > >>> starting to push forward on getting that out. > > > >>> > > > >>> > > > >>> - Bobby > > > >>> > > > >>> > > > >>> On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz < > > > >>> ptgo...@gmail.com<mailto:ptgo...@gmail.com>> wrote: > > > >>> > > > >>> I’d like to bump this thread and start a discussion around our > > > >>> next > > > >>> release. Here are my thoughts. > > > >>> > > > >>> There are a number of important fixes in 1.x-branch so I’d like > > > >>> to > > > >>> consider releasing 1.1.1 soon. I’d appreciate input on any open > > > >>> issues that > > > >>> should be resolved for that release. > > > >>> > > > >>> I’d like us to consider releasing the metrics improvements in > > > >>> STORM-2153 > > > >>> [1] as version 1.2.0. That work is in the metrics_v2 feature > > > >>> branch > > > >>> right > > > >>> now and would need to be reviewed and merged. That work is > > > >>> against > > > >>> the > > > >>> 1.x-branch right now. I would recommend porting it to master > > > >>> *after* > > > >>> the > > > >>> review/merge since there will likely be changes as a result of > > > >>> the > > > >>> review. > > > >>> > > > >>> Maybe related to or not, but would we want to create a new > > > >>> branch > > > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2? > > > >>> > > > >>> > > > >>> > > > >>> If wee agree to the above, I would say yes. After the 1.1.1 > > > >>> release, > > > >>> we > > > >>> could create a 1.1.x-branch that would be the maintenance/release > > > >>> branch > > > >>> for that version line. 1.x-branch would then become the target > > > >>> for > > > >>> the > > > >>> 1.2.0 release. > > > >>> > > > >>> There are a few fixes in the 0.10.x branch that probably warrant > > > >>> a > > > >>> release. After that we may want to back away from that version > > > >>> line > > > >>> a bit > > > >>> so we can focus more on newer versions. > > > >>> > > > >>> In the past, we’ve shied away form doing “beta” releases, but I’m > > > >>> wondering if we might want to revisit that for the 2.0 release — > > > >>> the > > > >>> idea > > > >>> being that it would give early adopter users a chance to kick the > > > >>> tires on > > > >>> what’s coming in 2.0 and provide feedback, find bugs, etc. to > > > >>> help > > > >>> make the > > > >>> final release more solid. I’m on the fence here and could go > > > >>> either > > > >>> way. > > > >>> > > > >>> I’d appreciate any input others may have. > > > >>> > > > >>> > > > >>> Thanks, > > > >>> > > > >>> -Taylor > > > >>> > > > >>> > > > >>> [1] https://issues.apache.org/jira/browse/STORM-2153 > > > >>> > > > >>> > > > >>> > > > >>> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabh...@gmail.com > <mailto: > > > kabh > > > >> w...@gmail.com>> > > > >>> wrote: > > > >>> > > > >>> Maybe related to or not, but would we want to create a new > > > >>> branch > > > >>> "1.1.x-branch", and make "1.x-branch" target for 1.2? > > > >>> > > > >>> I'm not clear we don't release 1.2 for moving toward to 2.0.0, > > > >>> so > > > >>> hence > > > >>> the > > > >>> question. > > > >>> > > > >>> - Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro < > > > >>> hlo...@hortonworks.com<mailto:hlo...@hortonworks.com>>님이 > > > >>> 작성: > > > >>> > > > >>> +1 for finishing the porting to Java ahead of anything else - > > > >>> it > > > >>> will > > > >>> be a > > > >>> significant milestone. I have a JIRA assigned concerning to > > > >>> the > > > >>> porting. I > > > >>> will work on it for the 2.0 release. > > > >>> > > > >>> it’s a priority to guarantee no performance regressions. As > > > >>> part > > > >>> of this > > > >>> endeavor, explore an automated (or easy) way to run and assert > > > >>> major > > > >>> performance benchmarks. Ideally any contributor should be able > > > >>> to > > > >>> fairly > > > >>> easily test the impact of changes under certain performance > > > >>> test > > > >>> scenarios. > > > >>> > > > >>> Beam Runner work should take into account the impact of > > > >>> incorporating > > > >>> new > > > >>> JStorm features and Storm Worker Redesign< > > > >>> https://issues.apache.org/jira/browse/STORM-2284>. Not very > > > >>> efficient > > > >>> to > > > >>> start doing it, to find out that it will have to chance in > > > >>> face > > > >>> of > > > >>> Storm > > > >>> and worker redesign. That is, it should be done after it’s > > > >>> building > > > >>> blocks > > > >>> are stable. > > > >>> > > > >>> Thanks, > > > >>> Hugo > > > >>> > > > >>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan < > > > >>> ar...@apache.org<mailto:ar...@apache.org> > > > >>> <mailto: > > > >>> ar...@apache.org>> wrote: > > > >>> > > > >>> +1 to release with the porting completed. I think its mainly > > > >>> the > > > >>> UI > > > >>> server > > > >>> and log viewer that’s pending. > > > >>> > > > >>> We can start doing the regression and performance tests for > > > >>> whatever is > > > >>> already ported. > > > >>> > > > >>> If anyone is running the master branch in their pre-prod / > > > >>> prod > > > >>> environments, it will be good to know and give us more > > > >>> confidence. > > > >>> > > > >>> The other features can be added in follow up releases. > > > >>> > > > >>> Regards, > > > >>> Arun > > > >>> > > > >>> > > > >>> On 3/24/17, 11:47 AM, "Satish Duggana" < > > > >>> satish.dugg...@gmail.com > > > >>> <mailto: > > > >>> satish.dugg...@gmail.com>> wrote: > > > >>> > > > >>> +1 to have 2.0 with porting and performance(it should be at > > > >>> least > > > >>> as > > > >>> good > > > >>> as 1.x release) issues addressed > > > >>> > > > >>> We can target other tasks(mentioned by Taylor and Jungtaek) > > > >>> for > > > >>> 2.x-branch. > > > >>> > > > >>> > > > >>> Exactly-once support: > > > >>> While thinking through the exactlyonce support design, it is > > > >>> realized > > > >>> better to avoid acking tuples and implement exactly once by > > > >>> snapshotting > > > >>> barriers. It seems JStorm folks followed similar design, they > > > >>> claim it > > > >>> gives better performance. This feature is essential for beam > > > >>> runner and > > > >>> we > > > >>> can decide on respective approaches though. > > > >>> > > > >>> Beam Runner > > > >>> Lets hold on this for now and keep it in Storm till 2.x. We > > > >>> should avoid > > > >>> having a minimal beam runner in haste. It is better to address > > > >>> STORM-2284, > > > >>> exactly-once and other windowing enhancements to enable beam > > > >>> runner. > > > >>> > > > >>> JStorm > > > >>> Agree with Jungtaek on looking at the latest JStorm and > > > >>> align/scope with > > > >>> the features for 2.x. > > > >>> > > > >>> STORM-2284 > > > >>> We may want to look at JStorm worker before working on > > > >>> respective > > > >>> components in this epic to pull appropriate enhancements. > > > >>> > > > >>> YARN/MESOS > > > >>> Supporting Storm on YARN/Mesos for 2.x. > > > >>> > > > >>> Thanks, > > > >>> Satish. > > > >>> > > > >>> > > > >>> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim < > > > >>> kabh...@gmail.com > > > >>> <mailto: > > > >>> kabh...@gmail.com>> wrote: > > > >>> > > > >>> First of all, +1 to complete only port work and do sanity > > > >>> check > > > >>> (including > > > >>> performance regression), and release. > > > >>> > > > >>> If we can get STORM-2284 within deterministic time frame (say > > > >>> 2~3 > > > >>> months) > > > >>> that should be great, but if not I'd in favor of postponing > > > >>> that > > > >>> to > > > >>> later > > > >>> 2.x release. > > > >>> > > > >>> JStorm released their new versions after code donation. So > > > >>> there're more > > > >>> things we could get ideas from, or even adopt from. > > > >>> https://github.com/alibaba/jstorm/blob/master/history.md > > > >>> As you noticed from release note link, we also need to update > > > >>> phase 2 > > > >>> since > > > >>> they already changed what we're planning to do in phase 2. For > > > >>> example, > > > >>> they changed backpressure to end-to-end, and changed to use > > > >>> snapshot > > > >>> rather > > > >>> than acker. > > > >>> May be sure, JStorm pulled many features from today's Storm, > > > >>> like > > > >>> Flux, > > > >>> Windowing, more shuffle groupings, log search, log level > > > >>> change, > > > >>> and so > > > >>> on. > > > >>> > > > >>> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> > > > >>> is > > > >>> due to > > > >>> the > > > >>> limitation of Spout lifecycle (all the things are done in > > > >>> single > > > >>> thread), > > > >>> and STORM-1358 < > > > >>> https://issues.apache.org/jira/browse/STORM-1358 > > > >>> (JStorm's > > > >>> multi-thread Spout) can remedy this (despite that Spout > > > >>> implementation > > > >>> may > > > >>> need to guarantee thread-safety later). It's not a just > > > >>> improvement but > > > >>> close to design concern so would like to address sooner than > > > >>> other > > > >>> things > > > >>> in phase 2. > > > >>> > > > >>> For Storm SQL side, I've lost progress but major work would be > > > >>> adopting > > > >>> group by with windowing. It was not available from Calcite but > > > >>> will be > > > >>> available at next release (1.12.0). > > > >>> I've filed this to STORM-2405 > > > >>> <https://issues.apache.org/jira/browse/STORM-2405>, but > > > >>> windowing & > > > >>> micro > > > >>> batch is not intuitive, so I would like to change the > > > >>> underlying > > > >>> API to > > > >>> stream API in SQL. Also filed this to STORM-2406 > > > >>> <https://issues.apache.org/jira/browse/STORM-2406>. > > > >>> > > > >>> Just 2 cents btw, hopefully I would like to see metrics V2 > > > >>> sooner > > > >>> since > > > >>> we > > > >>> lost metrics even when doing normal operation like restarting > > > >>> worker, > > > >>> rebalancing, and so on. Eventually we need to fight with > > > >>> dynamic > > > >>> scaling, > > > >>> and then metrics will be broken often. > > > >>> > > > >>> Thanks, > > > >>> Jungtaek Lim (HeartSaVioR) > > > >>> > > > >>> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani < > > > >>> st...@harsha.io > > > >>> 님이 > > > >>> 작성: > > > >>> > > > >>> Storm 2.0 migration to java in itself is a big win and would > > > >>> attract > > > >>> wider > > > >>> community and adoption. So my vote would be to resolve the > > > >>> first > > > >>> 3 items > > > >>> to > > > >>> get a release out. > > > >>> All the other featured mentioned are great to have but > > > >>> shouldn't > > > >>> be > > > >>> blockers for 2.0 release. > > > >>> > > > >>> -Harsha > > > >>> > > > >>> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz < > > > >>> ptgo...@gmail.com> > > > >>> wrote: > > > >>> > > > >>> With the 1.1.0 release nearing completion, I’d like to turn > > > >>> our > > > >>> attention > > > >>> to 2.0 and develop a plan for what features, etc. to include. > > > >>> > > > >>> The following 3 are what I feel are the minimum for a 2.0 > > > >>> release. > > > >>> These > > > >>> could likely be resolved relatively quickly: > > > >>> > > > >>> * Performance — I’ve not benchmarked the master branch vs. > > > >>> 1.0.x > > > >>> or > > > >>> 1.1.x > > > >>> in a while, but I feel it will be important to make sure there > > > >>> are no > > > >>> performance regressions, and would hope that we actually have > > > >>> a > > > >>> performance > > > >>> improvement over previous versions. To that end (e.g. if there > > > >>> is > > > >>> in > > > >>> fact a > > > >>> performance regression), the proposals that Roshan Naik put > > > >>> together > > > >>> for > > > >>> revising the threading and execution model (STORM-2307) and > > > >>> replacing > > > >>> Disruptor with JCTools (STORM-2306) warrant review and > > > >>> consideration. > > > >>> See > > > >>> also STORM-2284 which is the parent JIRA. > > > >>> > > > >>> * Finish porting Storm UI to java (STORM-1311) > > > >>> > > > >>> * Finish porting log viewer to java (STORM-1280) > > > >>> > > > >>> The following are items that are nice to have in 2.0, but I > > > >>> don’t > > > >>> feel > > > >>> are > > > >>> absolutely necessary for an initial 2.0 release: > > > >>> > > > >>> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it > > > >>> because > > > >>> it’s > > > >>> relevant) — Initially there seemed to be a lot of interest in > > > >>> this, but > > > >>> that seems to have trailed off. I spoke with some Beam > > > >>> developers > > > >>> and > > > >>> there > > > >>> seems to be interest from that community as well. Do we want > > > >>> to > > > >>> move > > > >>> that > > > >>> effort to the Beam community, or keep it here? Moving it to > > > >>> the > > > >>> Beam > > > >>> community might lead to better collaboration between projects. > > > >>> > > > >>> * Bounded Spouts (needed for Beam Runner implementation) — > > > >>> Currently > > > >>> spouts are unbounded, there no end to the stream. Beam has the > > > >>> concept > > > >>> of > > > >>> bounded sources (roughly analogous to batch processing). To > > > >>> support > > > >>> that, > > > >>> we would need to implement a similar concept in Storm. One > > > >>> benefit of > > > >>> such > > > >>> a feature would be the ability to handle both bounded and > > > >>> unbounded > > > >>> workflows in Storm. > > > >>> > > > >>> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers > > > >>> behind > > > >>> this > > > >>> effort. What improvements do you envision for 2.0? > > > >>> > > > >>> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been > > > >>> targeting this > > > >>> for 1.2.0, but it’s designed to be easily portable to > > > >>> master/2.0. > > > >>> > > > >>> * JStorm Migration — Original outline can be found here [1]. > > > >>> Note > > > >>> a lot > > > >>> of > > > >>> the associated JIRAs below are assigned, but there hasn’t been > > > >>> any > > > >>> recent > > > >>> activity or pull requests, we should probably consider them > > > >>> unassigned > > > >>> and > > > >>> up for grabs.: > > > >>> > > > >>> * Worker Classloader Isolation (STORM-1338) — Lack of this has > > > >>> been the > > > >>> bane of a lot of Storm users almost since day one. We have > > > >>> largely > > > >>> addressed it by shading/relocating dependencies. It would be > > > >>> great to > > > >>> see > > > >>> this addressed once and for all. > > > >>> > > > >>> * JStorm back pressure implementation (STORM-1324) — The > > > >>> current > > > >>> back > > > >>> pressure implementation leaves a bit to be desired, and the > > > >>> JStorm > > > >>> approach > > > >>> looks promising, though it also depends on the JStorm concept > > > >>> of > > > >>> “topology > > > >>> master” (STORM-1323), which may have some implications > > > >>> regarding > > > >>> security. > > > >>> > > > >>> * Dynamic Topology Updates (STORM-1335) — This would provide a > > > >>> command > > > >>> to > > > >>> update topology jars and configuration without stopping the > > > >>> topology, > > > >>> and > > > >>> is well suited to leverage the blobstore. The restart command > > > >>> (that can > > > >>> also update the topology configuration) also looks compelling > > > >>> (STORM-1334). > > > >>> > > > >>> * Additional Scheduler Implementations (STORM-1320) > > > >>> > > > >>> * Additional Grouping Implementations (STORM-1328) > > > >>> > > > >>> > > > >>> As always I’m open to any opinions and suggestions. > > > >>> > > > >>> -Taylor > > > >>> > > > >>> [1] > > > >>> > > > >>> https://cwiki.apache.org/confluence/pages/viewpage. > > > >>> action?pageId=61328109 > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > >
