+1 for Roshan's suggestion : in our Storm 1.x based supervision system, we're very interested anything that can provide better throughput.
2017-06-03 18:12 GMT+02:00 Roshan Naik <ros...@hortonworks.com>: > For 2.0 beta … it would be good to incorporate some of the Worker > improvements (STORM-2284) IMO. Changes to messaging subsystem can be > delivered sooner and my in-progress implementation suggests that it will > yield substantial latency improvements. The 2.0 beta phase would really > help kick the tires on the revised messaging system and the performance > improvements will also be a good incentive for trying out the 2.0 line. > > I notice multiple other bottlenecks that are holding back throughput a > lot, which can be addressed in a subsequent 2.x minor release. > -roshan > > > On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote: > > I also would love to see metrics V2 code sooner or later too. If we > can get > it before releasing 2.0.0 that will be great, and then maybe we could > just > move toward to 2.0.0, not adding any improvements to 1.x version line. > (And that's what I would want to.) > > If we would really want to have 1.2.0, I suggest that we make the 1.1.1 > version correct right now rather than after releasing 1.1.1. We also > merged > non-bugfix things to 1.x-branch but that's not what users expect. I > agree > that work may be painful, but anyway need to do it. > > - Jungtaek Lim (HeartSaVioR) > > 2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 > 작성: > > > I would love to see the metrics V2 code come out sooner rather than > > later. +1. > > My biggest blocker for a 1.x release is > > https://github.com/apache/storm/pull/2142 Even though the pull > request > > says it is minor it showed that we messed up pushing back some > changes for > > pacemaker to open source (the code does not run at all which for me > is a > > blocker) and I really want to get that fully fixed/tested before > another > > release. > > As for 2.x I think we are very close to being able to so a 2.x alpha > > release. I would like to see metrics V2 merged in simply because it > is a > > big change for user facing APIs. But after that I would love to see > us > > starting to push forward on getting that out. > > > > > > - Bobby > > > > > > On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz < > > ptgo...@gmail.com> wrote: > > > > I’d like to bump this thread and start a discussion around our next > > release. Here are my thoughts. > > > > There are a number of important fixes in 1.x-branch so I’d like to > > consider releasing 1.1.1 soon. I’d appreciate input on any open > issues that > > should be resolved for that release. > > > > I’d like us to consider releasing the metrics improvements in > STORM-2153 > > [1] as version 1.2.0. That work is in the metrics_v2 feature branch > right > > now and would need to be reviewed and merged. That work is against > the > > 1.x-branch right now. I would recommend porting it to master *after* > the > > review/merge since there will likely be changes as a result of the > review. > > > > > Maybe related to or not, but would we want to create a new branch > > > "1.1.x-branch", and make "1.x-branch" target for 1.2? > > > > > > > > If wee agree to the above, I would say yes. After the 1.1.1 release, > we > > could create a 1.1.x-branch that would be the maintenance/release > branch > > for that version line. 1.x-branch would then become the target for > the > > 1.2.0 release. > > > > There are a few fixes in the 0.10.x branch that probably warrant a > > release. After that we may want to back away from that version line > a bit > > so we can focus more on newer versions. > > > > In the past, we’ve shied away form doing “beta” releases, but I’m > > wondering if we might want to revisit that for the 2.0 release — the > idea > > being that it would give early adopter users a chance to kick the > tires on > > what’s coming in 2.0 and provide feedback, find bugs, etc. to help > make the > > final release more solid. I’m on the fence here and could go either > way. > > > > I’d appreciate any input others may have. > > > > > > Thanks, > > > > -Taylor > > > > > > [1] https://issues.apache.org/jira/browse/STORM-2153 > > > > > > > > > On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabh...@gmail.com> > wrote: > > > > > > Maybe related to or not, but would we want to create a new branch > > > "1.1.x-branch", and make "1.x-branch" target for 1.2? > > > > > > I'm not clear we don't release 1.2 for moving toward to 2.0.0, so > hence > > the > > > question. > > > > > > - Jungtaek Lim (HeartSaVioR) > > > > > > 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro < > hlo...@hortonworks.com>님이 > > 작성: > > > > > >> +1 for finishing the porting to Java ahead of anything else - it > will > > be a > > >> significant milestone. I have a JIRA assigned concerning to the > > porting. I > > >> will work on it for the 2.0 release. > > >> > > >> it’s a priority to guarantee no performance regressions. As part > of this > > >> endeavor, explore an automated (or easy) way to run and assert > major > > >> performance benchmarks. Ideally any contributor should be able to > fairly > > >> easily test the impact of changes under certain performance test > > scenarios. > > >> > > >> Beam Runner work should take into account the impact of > incorporating > > new > > >> JStorm features and Storm Worker Redesign< > > >> https://issues.apache.org/jira/browse/STORM-2284>. Not very > efficient > > to > > >> start doing it, to find out that it will have to chance in face > of > > Storm > > >> and worker redesign. That is, it should be done after it’s > building > > blocks > > >> are stable. > > >> > > >> Thanks, > > >> Hugo > > >> > > >> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan <ar...@apache.org > <mailto: > > >> ar...@apache.org>> wrote: > > >> > > >> +1 to release with the porting completed. I think its mainly the > UI > > server > > >> and log viewer that’s pending. > > >> > > >> We can start doing the regression and performance tests for > whatever is > > >> already ported. > > >> > > >> If anyone is running the master branch in their pre-prod / prod > > >> environments, it will be good to know and give us more confidence. > > >> > > >> The other features can be added in follow up releases. > > >> > > >> Regards, > > >> Arun > > >> > > >> > > >> On 3/24/17, 11:47 AM, "Satish Duggana" <satish.dugg...@gmail.com > > <mailto: > > >> satish.dugg...@gmail.com>> wrote: > > >> > > >> +1 to have 2.0 with porting and performance(it should be at least > as > > good > > >> as 1.x release) issues addressed > > >> > > >> We can target other tasks(mentioned by Taylor and Jungtaek) for > > 2.x-branch. > > >> > > >> > > >> Exactly-once support: > > >> While thinking through the exactlyonce support design, it is > realized > > >> better to avoid acking tuples and implement exactly once by > snapshotting > > >> barriers. It seems JStorm folks followed similar design, they > claim it > > >> gives better performance. This feature is essential for beam > runner and > > we > > >> can decide on respective approaches though. > > >> > > >> Beam Runner > > >> Lets hold on this for now and keep it in Storm till 2.x. We > should avoid > > >> having a minimal beam runner in haste. It is better to address > > STORM-2284, > > >> exactly-once and other windowing enhancements to enable beam > runner. > > >> > > >> JStorm > > >> Agree with Jungtaek on looking at the latest JStorm and > align/scope with > > >> the features for 2.x. > > >> > > >> STORM-2284 > > >> We may want to look at JStorm worker before working on respective > > >> components in this epic to pull appropriate enhancements. > > >> > > >> YARN/MESOS > > >> Supporting Storm on YARN/Mesos for 2.x. > > >> > > >> Thanks, > > >> Satish. > > >> > > >> > > >> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim <kabh...@gmail.com > > <mailto: > > >> kabh...@gmail.com>> wrote: > > >> > > >> First of all, +1 to complete only port work and do sanity check > > (including > > >> performance regression), and release. > > >> > > >> If we can get STORM-2284 within deterministic time frame (say 2~3 > > months) > > >> that should be great, but if not I'd in favor of postponing that > to > > later > > >> 2.x release. > > >> > > >> JStorm released their new versions after code donation. So > there're more > > >> things we could get ideas from, or even adopt from. > > >> https://github.com/alibaba/jstorm/blob/master/history.md > > >> As you noticed from release note link, we also need to update > phase 2 > > since > > >> they already changed what we're planning to do in phase 2. For > example, > > >> they changed backpressure to end-to-end, and changed to use > snapshot > > rather > > >> than acker. > > >> May be sure, JStorm pulled many features from today's Storm, like > Flux, > > >> Windowing, more shuffle groupings, log search, log level change, > and so > > on. > > >> > > >> STORM-2426 <https://issues.apache.org/jira/browse/STORM-2426> is > due to > > >> the > > >> limitation of Spout lifecycle (all the things are done in single > > thread), > > >> and STORM-1358 <https://issues.apache.org/jira/browse/STORM-1358 > > >(JStorm's > > >> multi-thread Spout) can remedy this (despite that Spout > implementation > > may > > >> need to guarantee thread-safety later). It's not a just > improvement but > > >> close to design concern so would like to address sooner than other > > things > > >> in phase 2. > > >> > > >> For Storm SQL side, I've lost progress but major work would be > adopting > > >> group by with windowing. It was not available from Calcite but > will be > > >> available at next release (1.12.0). > > >> I've filed this to STORM-2405 > > >> <https://issues.apache.org/jira/browse/STORM-2405>, but > windowing & > > micro > > >> batch is not intuitive, so I would like to change the underlying > API to > > >> stream API in SQL. Also filed this to STORM-2406 > > >> <https://issues.apache.org/jira/browse/STORM-2406>. > > >> > > >> Just 2 cents btw, hopefully I would like to see metrics V2 sooner > since > > we > > >> lost metrics even when doing normal operation like restarting > worker, > > >> rebalancing, and so on. Eventually we need to fight with dynamic > > scaling, > > >> and then metrics will be broken often. > > >> > > >> Thanks, > > >> Jungtaek Lim (HeartSaVioR) > > >> > > >> 2017년 3월 24일 (금) 오전 5:05, Harsha Chintalapani <st...@harsha.io>님이 > 작성: > > >> > > >> Storm 2.0 migration to java in itself is a big win and would > attract > > >> wider > > >> community and adoption. So my vote would be to resolve the first > 3 items > > >> to > > >> get a release out. > > >> All the other featured mentioned are great to have but shouldn't > be > > >> blockers for 2.0 release. > > >> > > >> -Harsha > > >> > > >> On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz < > ptgo...@gmail.com> > > >> wrote: > > >> > > >> With the 1.1.0 release nearing completion, I’d like to turn our > > >> attention > > >> to 2.0 and develop a plan for what features, etc. to include. > > >> > > >> The following 3 are what I feel are the minimum for a 2.0 release. > > >> These > > >> could likely be resolved relatively quickly: > > >> > > >> * Performance — I’ve not benchmarked the master branch vs. 1.0.x > or > > >> 1.1.x > > >> in a while, but I feel it will be important to make sure there > are no > > >> performance regressions, and would hope that we actually have a > > >> performance > > >> improvement over previous versions. To that end (e.g. if there is > in > > >> fact a > > >> performance regression), the proposals that Roshan Naik put > together > > >> for > > >> revising the threading and execution model (STORM-2307) and > replacing > > >> Disruptor with JCTools (STORM-2306) warrant review and > consideration. > > >> See > > >> also STORM-2284 which is the parent JIRA. > > >> > > >> * Finish porting Storm UI to java (STORM-1311) > > >> > > >> * Finish porting log viewer to java (STORM-1280) > > >> > > >> The following are items that are nice to have in 2.0, but I don’t > feel > > >> are > > >> absolutely necessary for an initial 2.0 release: > > >> > > >> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because > it’s > > >> relevant) — Initially there seemed to be a lot of interest in > this, but > > >> that seems to have trailed off. I spoke with some Beam developers > and > > >> there > > >> seems to be interest from that community as well. Do we want to > move > > >> that > > >> effort to the Beam community, or keep it here? Moving it to the > Beam > > >> community might lead to better collaboration between projects. > > >> > > >> * Bounded Spouts (needed for Beam Runner implementation) — > Currently > > >> spouts are unbounded, there no end to the stream. Beam has the > concept > > >> of > > >> bounded sources (roughly analogous to batch processing). To > support > > >> that, > > >> we would need to implement a similar concept in Storm. One > benefit of > > >> such > > >> a feature would be the ability to handle both bounded and > unbounded > > >> workflows in Storm. > > >> > > >> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers > behind > > >> this > > >> effort. What improvements do you envision for 2.0? > > >> > > >> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been > targeting this > > >> for 1.2.0, but it’s designed to be easily portable to master/2.0. > > >> > > >> * JStorm Migration — Original outline can be found here [1]. Note > a lot > > >> of > > >> the associated JIRAs below are assigned, but there hasn’t been any > > >> recent > > >> activity or pull requests, we should probably consider them > unassigned > > >> and > > >> up for grabs.: > > >> > > >> * Worker Classloader Isolation (STORM-1338) — Lack of this has > been the > > >> bane of a lot of Storm users almost since day one. We have largely > > >> addressed it by shading/relocating dependencies. It would be > great to > > >> see > > >> this addressed once and for all. > > >> > > >> * JStorm back pressure implementation (STORM-1324) — The current > back > > >> pressure implementation leaves a bit to be desired, and the JStorm > > >> approach > > >> looks promising, though it also depends on the JStorm concept of > > >> “topology > > >> master” (STORM-1323), which may have some implications regarding > > >> security. > > >> > > >> * Dynamic Topology Updates (STORM-1335) — This would provide a > command > > >> to > > >> update topology jars and configuration without stopping the > topology, > > >> and > > >> is well suited to leverage the blobstore. The restart command > (that can > > >> also update the topology configuration) also looks compelling > > >> (STORM-1334). > > >> > > >> * Additional Scheduler Implementations (STORM-1320) > > >> > > >> * Additional Grouping Implementations (STORM-1328) > > >> > > >> > > >> As always I’m open to any opinions and suggestions. > > >> > > >> -Taylor > > >> > > >> [1] > > >> > > >> https://cwiki.apache.org/confluence/pages/viewpage. > > >> action?pageId=61328109 > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > >
