Yes I would prefer to see us get a 2.x alpha release out. Are the new metrics APIs going to be a breaking change? I assume not, because we wouldn't want to put them in 1.x otherwise. If that is the case then we don't need to wait for them before doing a 2.x release.
- Bobby On Friday, June 23, 2017, 3:19:28 PM CDT, P. Taylor Goetz <ptgo...@gmail.com> wrote: Bobby/Jungtaek, Are you saying you want to forego the 1.2 “metrics_v2” release and include it only in 2.0? (I ask because that work is already based on 1.x-branch, and forward-porting it to master is relatively simple.) I’d kind of like that work go out soon. If we go with option 1, I would want to see a 2.0 release (even if it’s a “beta” or “preview) before putting the 1.x line into maintenance mode. -Taylor > On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote: > > I see 2 ways to address this. > 1) We put the 1.x line into maintenance mode like with 0.10. We don't > backport anything except bug fixes.2) We backport a lot of the backwards > compatible changes from 2.x to 1.x. > My personal preference is 1. It makes it clear the direction we want to go > in. The biggest issue is that we probably would want to do a 2.x release > sooner rather then later. Even if we don't get all of the features that > people want, if we just get a release out we can add in new features if they > are backwards compatible, or we can create a 3.x line that would have the > breaking changes in it. > > - Bobby > > > On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <kabh...@gmail.com> > wrote: > > I'd like to bump this again instead of initiating new discussion thread. > > I had having hard time to create and apply pull requests for both master > and 1.x-branch and that's really painful and sometimes blocker for me to do > merge step. > Two branches are heavily diverged more than between 0.10 and 1.0.0, even > IDE can't switch between the branch smoothly. We didn't even address > checkstyle issue yet, but after addressing, it could be "completely" > diverged. JDK version is another major issue, since the pull requests > targeted for master branch are not checked against JDK 7, and some of them > make some issues regarding JDK version while porting back. > > So personally I really would like to see the plan for 1.x version line > changed - skipping any minor releases including 1.2.0 - and have epic issue > for 2.0.0 and just go ahead. That was our proposed plan indeed. (even > proposed plan was having 2.0.0 directly after 1.0.0) > > Would like to hear everyone's opinions. If we have consensus to not having > any minor releases for 1.x version line, I will not port back non-bugfix > pull requests to 1.x-branch, and guide contributors to create pull requests > against master branch, not 1.x version line. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <avermeerber...@gmail.com>님이 > 작성: > >> +1 for Roshan's suggestion : in our Storm 1.x based supervision system, >> we're very interested anything that can provide better throughput. >> >> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ros...@hortonworks.com>: >> >>> For 2.0 beta … it would be good to incorporate some of the Worker >>> improvements (STORM-2284) IMO. Changes to messaging subsystem can be >>> delivered sooner and my in-progress implementation suggests that it will >>> yield substantial latency improvements. The 2.0 beta phase would really >>> help kick the tires on the revised messaging system and the performance >>> improvements will also be a good incentive for trying out the 2.0 line. >>> >>> I notice multiple other bottlenecks that are holding back throughput a >>> lot, which can be addressed in a subsequent 2.x minor release. >>> -roshan >>> >>> >>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote: >>> >>> I also would love to see metrics V2 code sooner or later too. If we >>> can get >>> it before releasing 2.0.0 that will be great, and then maybe we could >>> just >>> move toward to 2.0.0, not adding any improvements to 1.x version >> line. >>> (And that's what I would want to.) >>> >>> If we would really want to have 1.2.0, I suggest that we make the >> 1.1.1 >>> version correct right now rather than after releasing 1.1.1. We also >>> merged >>> non-bugfix things to 1.x-branch but that's not what users expect. I >>> agree >>> that work may be painful, but anyway need to do it. >>> >>> - Jungtaek Lim (HeartSaVioR) >>> >>> 2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 >>> 작성: >>> >>> > I would love to see the metrics V2 code come out sooner rather than >>> > later. +1. >>> > My biggest blocker for a 1.x release is >>> > https://github.com/apache/storm/pull/2142 Even though the pull >>> request >>> > says it is minor it showed that we messed up pushing back some >>> changes for >>> > pacemaker to open source (the code does not run at all which for me >>> is a >>> > blocker) and I really want to get that fully fixed/tested before >>> another >>> > release. >>> > As for 2.x I think we are very close to being able to so a 2.x >> alpha >>> > release. I would like to see metrics V2 merged in simply because >> it >>> is a >>> > big change for user facing APIs. But after that I would love to >> see >>> us >>> > starting to push forward on getting that out. >>> > >>> > >>> > - Bobby >>> > >>> > >>> > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz < >>> > ptgo...@gmail.com> wrote: >>> > >>> > I’d like to bump this thread and start a discussion around our next >>> > release. Here are my thoughts. >>> > >>> > There are a number of important fixes in 1.x-branch so I’d like to >>> > consider releasing 1.1.1 soon. I’d appreciate input on any open >>> issues that >>> > should be resolved for that release. >>> > >>> > I’d like us to consider releasing the metrics improvements in >>> STORM-2153 >>> > [1] as version 1.2.0. That work is in the metrics_v2 feature branch >>> right >>> > now and would need to be reviewed and merged. That work is against >>> the >>> > 1.x-branch right now. I would recommend porting it to master >> *after* >>> the >>> > review/merge since there will likely be changes as a result of the >>> review. >>> > >>> > > Maybe related to or not, but would we want to create a new branch >>> > > "1.1.x-branch", and make "1.x-branch" target for 1.2? >>> > >>> > >>> > >>> > If wee agree to the above, I would say yes. After the 1.1.1 >> release, >>> we >>> > could create a 1.1.x-branch that would be the maintenance/release >>> branch >>> > for that version line. 1.x-branch would then become the target for >>> the >>> > 1.2.0 release. >>> > >>> > There are a few fixes in the 0.10.x branch that probably warrant a >>> > release. After that we may want to back away from that version line >>> a bit >>> > so we can focus more on newer versions. >>> > >>> > In the past, we’ve shied away form doing “beta” releases, but I’m >>> > wondering if we might want to revisit that for the 2.0 release — >> the >>> idea >>> > being that it would give early adopter users a chance to kick the >>> tires on >>> > what’s coming in 2.0 and provide feedback, find bugs, etc. to help >>> make the >>> > final release more solid. I’m on the fence here and could go either >>> way. >>> > >>> > I’d appreciate any input others may have. >>> > >>> > >>> > Thanks, >>> > >>> > -Taylor >>> > >>> > >>> > [1] https://issues.apache.org/jira/browse/STORM-2153 >>> > >>> > >>> > >>> > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabh...@gmail.com> >>> wrote: >>> > > >>> > > Maybe related to or not, but would we want to create a new branch >>> > > "1.1.x-branch", and make "1.x-branch" target for 1.2? >>> > > >>> > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so >>> hence >>> > the >>> > > question. >>> > > >>> > > - Jungtaek Lim (HeartSaVioR) >>> > > >>> > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro < >>> hlo...@hortonworks.com>님이 >>> > 작성: >>> > > >>> > >> +1 for finishing the porting to Java ahead of anything else - it >>> will >>> > be a >>> > >> significant milestone. I have a JIRA assigned concerning to the >>> > porting. I >>> > >> will work on it for the 2.0 release. >>> > >> >>> > >> it’s a priority to guarantee no performance regressions. As part >>> of this >>> > >> endeavor, explore an automated (or easy) way to run and assert >>> major >>> > >> performance benchmarks. Ideally any contributor should be able >> to >>> fairly >>> > >> easily test the impact of changes under certain performance test >>> > scenarios. >>> > >> >>> > >> Beam Runner work should take into account the impact of >>> incorporating >>> > new >>> > >> JStorm features and Storm Worker Redesign< >>> > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very >>> efficient >>> > to >>> > >> start doing it, to find out that it will have to chance in face >>> of >>> > Storm >>> > >> and worker redesign. That is, it should be done after it’s >>> building >>> > blocks >>> > >> are stable. >>> > >> >>> > >> Thanks, >>> > >> Hugo >>> > >> >>> > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <ar...@apache.org >>> <mailto: >>> > >> ar...@apache.org>> wrote: >>> > >> >>> > >> +1 to release with the porting completed. I think its mainly the >>> UI >>> > server >>> > >> and log viewer that’s pending. >>> > >> >>> > >> We can start doing the regression and performance tests for >>> whatever is >>> > >> already ported. >>> > >> >>> > >> If anyone is running the master branch in their pre-prod / prod >>> > >> environments, it will be good to know and give us more >> confidence. >>> > >> >>> > >> The other features can be added in follow up releases. >>> > >> >>> > >> Regards, >>> > >> Arun >>> > >> >>> > >> >>> > >> On 3/24/17, 11:47 AM, "Satish Duggana" < >> satish.dugg...@gmail.com >>> > <mailto: >>> > >> satish.dugg...@gmail.com>> wrote: >>> > >> >>> > >> +1 to have 2.0 with porting and performance(it should be at >> least >>> as >>> > good >>> > >> as 1.x release) issues addressed >>> > >> >>> > >> We can target other tasks(mentioned by Taylor and Jungtaek) for >>> > 2.x-branch. >>> > >> >>> > >> >>> > >> Exactly-once support: >>> > >> While thinking through the exactlyonce support design, it is >>> realized >>> > >> better to avoid acking tuples and implement exactly once by >>> snapshotting >>> > >> barriers. It seems JStorm folks followed similar design, they >>> claim it >>> > >> gives better performance. This feature is essential for beam >>> runner and >>> > we >>> > >> can decide on respective approaches though. >>> > >> >>> > >> Beam Runner >>> > >> Lets hold on this for now and keep it in Storm till 2.x. We >>> should avoid >>> > >> having a minimal beam runner in haste. It is better to address >>> > STORM-2284, >>> > >> exactly-once and other windowing enhancements to enable beam >>> runner. >>> > >> >>> > >> JStorm >>> > >> Agree with Jungtaek on looking at the latest JStorm and >>> align/scope with >>> > >> the features for 2.x. >>> > >> >>> > >> STORM-2284 >>> > >> We may want to look at JStorm worker before working on >> respective >>> > >> components in this epic to pull appropriate enhancements. >>> > >> >>> > >> YARN/MESOS >>> > >> Supporting Storm on YARN/Mesos for 2.x. >>> > >> >>> > >> Thanks, >>> > >> Satish. >>> > >> >>> > >> >>> > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim < >> kabh...@gmail.com >>> > <mailto: >>> > >> kabh...@gmail.com>> wrote: >>> > >> >>> > >> First of all, +1 to complete only port work and do sanity check >>> > (including >>> > >> performance regression), and release. >>> > >> >>> > >> If we can get STORM-2284 within deterministic time frame (say >> 2~3 >>> > months) >>> > >> that should be great, but if not I'd in favor of postponing that >>> to >>> > later >>> > >> 2.x release. >>> > >> >>> > >> JStorm released their new versions after code donation. So >>> there're more >>> > >> things we could get ideas from, or even adopt from. >>> > >> https://github.com/alibaba/jstorm/blob/master/history.md >>> > >> As you noticed from release note link, we also need to update >>> phase 2 >>> > since >>> > >> they already changed what we're planning to do in phase 2. For >>> example, >>> > >> they changed backpressure to end-to-end, and changed to use >>> snapshot >>> > rather >>> > >> than acker. >>> > >> May be sure, JStorm pulled many features from today's Storm, >> like >>> Flux, >>> > >> Windowing, more shuffle groupings, log search, log level change, >>> and so >>> > on. >>> > >> >>> > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> >> is >>> due to >>> > >> the >>> > >> limitation of Spout lifecycle (all the things are done in single >>> > thread), >>> > >> and STORM-1358 < >> https://issues.apache.org/jira/browse/STORM-1358 >>> > >(JStorm's >>> > >> multi-thread Spout) can remedy this (despite that Spout >>> implementation >>> > may >>> > >> need to guarantee thread-safety later). It's not a just >>> improvement but >>> > >> close to design concern so would like to address sooner than >> other >>> > things >>> > >> in phase 2. >>> > >> >>> > >> For Storm SQL side, I've lost progress but major work would be >>> adopting >>> > >> group by with windowing. It was not available from Calcite but >>> will be >>> > >> available at next release (1.12.0). >>> > >> I've filed this to STORM-2405 >>> > >> <https://issues.apache.org/jira/browse/STORM-2405>, but >>> windowing & >>> > micro >>> > >> batch is not intuitive, so I would like to change the underlying >>> API to >>> > >> stream API in SQL. Also filed this to STORM-2406 >>> > >> <https://issues.apache.org/jira/browse/STORM-2406>. >>> > >> >>> > >> Just 2 cents btw, hopefully I would like to see metrics V2 >> sooner >>> since >>> > we >>> > >> lost metrics even when doing normal operation like restarting >>> worker, >>> > >> rebalancing, and so on. Eventually we need to fight with dynamic >>> > scaling, >>> > >> and then metrics will be broken often. >>> > >> >>> > >> Thanks, >>> > >> Jungtaek Lim (HeartSaVioR) >>> > >> >>> > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io >>> 님이 >>> 작성: >>> > >> >>> > >> Storm 2.0 migration to java in itself is a big win and would >>> attract >>> > >> wider >>> > >> community and adoption. So my vote would be to resolve the first >>> 3 items >>> > >> to >>> > >> get a release out. >>> > >> All the other featured mentioned are great to have but shouldn't >>> be >>> > >> blockers for 2.0 release. >>> > >> >>> > >> -Harsha >>> > >> >>> > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz < >>> ptgo...@gmail.com> >>> > >> wrote: >>> > >> >>> > >> With the 1.1.0 release nearing completion, I’d like to turn our >>> > >> attention >>> > >> to 2.0 and develop a plan for what features, etc. to include. >>> > >> >>> > >> The following 3 are what I feel are the minimum for a 2.0 >> release. >>> > >> These >>> > >> could likely be resolved relatively quickly: >>> > >> >>> > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x >>> or >>> > >> 1.1.x >>> > >> in a while, but I feel it will be important to make sure there >>> are no >>> > >> performance regressions, and would hope that we actually have a >>> > >> performance >>> > >> improvement over previous versions. To that end (e.g. if there >> is >>> in >>> > >> fact a >>> > >> performance regression), the proposals that Roshan Naik put >>> together >>> > >> for >>> > >> revising the threading and execution model (STORM-2307) and >>> replacing >>> > >> Disruptor with JCTools (STORM-2306) warrant review and >>> consideration. >>> > >> See >>> > >> also STORM-2284 which is the parent JIRA. >>> > >> >>> > >> * Finish porting Storm UI to java (STORM-1311) >>> > >> >>> > >> * Finish porting log viewer to java (STORM-1280) >>> > >> >>> > >> The following are items that are nice to have in 2.0, but I >> don’t >>> feel >>> > >> are >>> > >> absolutely necessary for an initial 2.0 release: >>> > >> >>> > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because >>> it’s >>> > >> relevant) — Initially there seemed to be a lot of interest in >>> this, but >>> > >> that seems to have trailed off. I spoke with some Beam >> developers >>> and >>> > >> there >>> > >> seems to be interest from that community as well. Do we want to >>> move >>> > >> that >>> > >> effort to the Beam community, or keep it here? Moving it to the >>> Beam >>> > >> community might lead to better collaboration between projects. >>> > >> >>> > >> * Bounded Spouts (needed for Beam Runner implementation) — >>> Currently >>> > >> spouts are unbounded, there no end to the stream. Beam has the >>> concept >>> > >> of >>> > >> bounded sources (roughly analogous to batch processing). To >>> support >>> > >> that, >>> > >> we would need to implement a similar concept in Storm. One >>> benefit of >>> > >> such >>> > >> a feature would be the ability to handle both bounded and >>> unbounded >>> > >> workflows in Storm. >>> > >> >>> > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers >>> behind >>> > >> this >>> > >> effort. What improvements do you envision for 2.0? >>> > >> >>> > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been >>> targeting this >>> > >> for 1.2.0, but it’s designed to be easily portable to >> master/2.0. >>> > >> >>> > >> * JStorm Migration — Original outline can be found here [1]. >> Note >>> a lot >>> > >> of >>> > >> the associated JIRAs below are assigned, but there hasn’t been >> any >>> > >> recent >>> > >> activity or pull requests, we should probably consider them >>> unassigned >>> > >> and >>> > >> up for grabs.: >>> > >> >>> > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has >>> been the >>> > >> bane of a lot of Storm users almost since day one. We have >> largely >>> > >> addressed it by shading/relocating dependencies. It would be >>> great to >>> > >> see >>> > >> this addressed once and for all. >>> > >> >>> > >> * JStorm back pressure implementation (STORM-1324) — The current >>> back >>> > >> pressure implementation leaves a bit to be desired, and the >> JStorm >>> > >> approach >>> > >> looks promising, though it also depends on the JStorm concept of >>> > >> “topology >>> > >> master” (STORM-1323), which may have some implications regarding >>> > >> security. >>> > >> >>> > >> * Dynamic Topology Updates (STORM-1335) — This would provide a >>> command >>> > >> to >>> > >> update topology jars and configuration without stopping the >>> topology, >>> > >> and >>> > >> is well suited to leverage the blobstore. The restart command >>> (that can >>> > >> also update the topology configuration) also looks compelling >>> > >> (STORM-1334). >>> > >> >>> > >> * Additional Scheduler Implementations (STORM-1320) >>> > >> >>> > >> * Additional Grouping Implementations (STORM-1328) >>> > >> >>> > >> >>> > >> As always I’m open to any opinions and suggestions. >>> > >> >>> > >> -Taylor >>> > >> >>> > >> [1] >>> > >> >>> > >> https://cwiki.apache.org/confluence/pages/viewpage. >>> > >> action?pageId=61328109 >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >>> >>> >>>
