Longda, "2.1 Copy modules from JStorm, one module from one module > 2.2 The sequence is extern > modules/client/utils/nimbus/supervisor/drpc/worker & task/web ui/others"
Are you suggesting we just copy the Jstorm code in place of clojure? If so thats not going to work. There might be some code that can be easily replaceable with Jstorm's . But not everything will be that straightforward especially with feature disparity between Storm & JStorm. We should be moving code to java i.e rewriting parts of the code where needed and if something that can be picked up from JStorm we should do that . Thanks, Harsha On Wed, Nov 18, 2015, at 10:13 PM, Longda Feng wrote: > Sorry for changing the Subject. > > I am +1 for releasing Storm 2.0 with java core, which is merged with > JStorm. > > I think the change of this release will be the biggest one in history. It > will probably take a long time to develop. At the same time, Heron is > going to open source, and the latest release of Flink provides the > compatibility to Storm’s API. These might be the threat to Storm. So I > suggest we start the development of Storm 2.0 as quickly as possible. In > order to accelerate the development cycle, I proposed to take JStorm > 2.1.0 core and UI as the base version since this version is stable and > compatible with API of Storm 1.0. Please refer to the phases below for > the detailed merging plan. > > Note: We provide a demo of JStorm’s web UI. Please refer to > storm.taobao.org . I think JStorm will give a totally different view to > you. > > I would like to share the experience of initial development of JStorm > (Migrate from clojure core to java core). > Our team(4 developers) have spent almost one year to finish the > migration. We took 4 months to release the first JStorm version, and 6 > months to make JStorm stable. During this period, we tried to switch more > than online 100 applications with different scenarios from Storm to > JStorm, and many bugs were fixed. Then more and more applications were > switched to JStorm in Alibaba. > Currently, there are 7000+ nodes of JStorm clusters in Alibaba and 2000+ > applications are running on them. The JStorm Clusters here can handle 1.5 > PB/2 Trillion messages per day. The use cases are not only in BigData > field but also in many other online scenarios. > Besides it, we have experienced the November 11th Shopping Festival of > Alibaba for last three years. At that day, the computation in our cluster > increased several times than usual. All applications worked well during > the peak time. I can say the stability of JStorm is no doubt today. > Actually, besides Alibaba, the most powerful Chinese IT company are also > using JStorm. > > > Phase 1: > > Define the target of Storm 2.0. List the requirement of Storm 2.0 > 1. Open a new Umbrella Jira > (https://issues.apache.org/jira/browse/STORM-717) > 2. Create one 2.0 branch, > 2.1 Copy modules from JStorm, one module from one module > 2.2 The sequence is extern > modules/client/utils/nimbus/supervisor/drpc/worker & task/web ui/others > 3. Create jira for all differences between JStorm 2.1.0 and Storm 1.0 > 3.1 Discuss solution for each difference(jira) > 3.2 Once the solution is finalized, we can start the merging. (Some > issues could be start concurrently. It depends on the discussion.) > > The phase mainly try to define target and finalize the solution. > Hopefully this phase could be finished in 2 month(before 2016/1/31). . > > > Phase 2: > Release Storm 2.0 beta > 1. Based on phrase 1's discussion, finish all features of Storm 2.0 > 2. Integrate all modules, make the simplest storm example can run on the > system. > 3. Test with all example and modules in Storm code base. > 4. All daily test can be passed. > > Hopefully this phase could be finished in 2 month(before 2016/3/31) > > > Phase 3: > Persuade some user to have a try. > Alibaba will try to run some online applications on the beta version > > Hopefully this phase could be finished in 1 month(before 2016/4/31). > > > Any comments are welcome. > > > Thanks > Longda------------------------------------------------------------------From:P. > Taylor Goetz <[email protected]>Send Time:2015年11月19日(星期四) 06:23To:dev > <[email protected]>，Bobby Evans <[email protected]>Subject:Re: > [DISCUSS] Plan for Merging JStorm Code > All I have at this point is a placeholder wiki entry [1], and a lot of > local notes that likely would only make sense to me. > > Let me know your wiki username and I’ll give you permissions. The same > goes for anyone else who wants to help. > > -Taylor > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109 > > > On Nov 18, 2015, at 2:08 PM, Bobby Evans <[email protected]> > >wrote: > > > > Taylor and others I was hoping to get started filing JIRA and planning on > >how we are going to do the java migration + JStorm merger. Is anyone else > >starting to do this? If not would anyone object to me starting on it? - > >Bobby > > > > > > On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz > ><[email protected]> wrote: > > > > > > Thanks for putting this together Basti, that comparison helps a lot. > > > > And thanks Bobby for converting it into markdown. I was going to just > >attach the spreadsheet to JIRA, but markdown is a much better solution. > > > > -Taylor > > > >> On Nov 12, 2015, at 12:03 PM, Bobby Evans <[email protected]> > >>wrote: > >> > >> I translated the excel spreadsheet into a markdown file and put up a pull > >>request for it. > >> https://github.com/apache/storm/pull/877 > >> I did a few edits to it to make it work with Markdown, and to add in a few > >>of my own comments. I also put in a field for JIRAs to be able to track > >>the migration. > >> Overall I think your evaluation was very good. We have a fair amount of > >>work ahead of us to decide what version of various features we want to go > >>forward with. > >> - Bobby > >> > >> > >> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) > >><[email protected]> wrote: > >> > >> > >> Hi Bobby & Jungtaek, > >> > >> Thanks for your replay. > >> I totally agree that compatibility is the most important thing. Actually, > >>JStorm has been compatible with the user API of Storm. > >> As you mentioned below, we indeed still have some features different > >>between Storm and JStorm. I have tried to list them (minor update or > >>improvements are not included). > >> Please refer to attachment for details. If any missing, please help to > >>point out. (The current working features are probably missing here.) > >> Just have a look at these differences. For the missing features in JStorm, > >>I did not see any obstacle which will block the merge to JStorm. > >> For the features which has different solution between Storm and JStorm, we > >>can evaluate the solution one by one to decision which one is appropriate. > >> After the finalization of evaluation, I think JStorm team can take the > >>merging job and publish a stable release in 2 months. > >> But anyway, the detailed implementation for these features with different > >>solution is transparent to user. So, from user's point of view, there is > >>not any compatibility problem. > >> > >> Besides compatibility, by our experience, stability is also important and > >>is not an easy job. 4 people in JStorm team took almost one year to finish > >>the porting from "clojure core" > >> to "java core", and to make it stable. Of course, we have many devs in > >>community to make the porting job faster. But it still needs a long time to > >>run many online complex topologys to find bugs and fix them. So, that is > >>the reason why I proposed to do merging and build on a stable "java core". > >> > >> -----Original Message----- > >> From: Bobby Evans [mailto:[email protected]] > >> Sent: Wednesday, November 11, 2015 10:51 PM > >> To: [email protected] > >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code > >> > >> +1 for doing a 1.0 release based off of the clojure 0.11.x code. > >>Migrating the APIs to org.apache.storm is a big non-backwards compatible > >>move, and a major version bump to 2.x seems like a good move there. > >> +1 for the release plan > >> > >> I would like the move for user facing APIs to org.apache to be one of the > >>last things we do. Translating clojure code into java and moving it to > >>org.apache I am not too concerned about. > >> > >> Basti, > >> We have two code bases that have diverged significantly from one another > >>in terms of functionality. The storm code now or soon will have A > >>Heartbeat Server, Nimbus HA (Different Implementation), Resource Aware > >>Scheduling, a distributed cache like API, log searching, security, massive > >>performance improvements, shaded almost all of our dependencies, a REST API > >>for programtically accessing everything on the UI, and I am sure I am > >>missing a few other things. JStorm also has many changes including cgroup > >>isolation, restructured zookeeper layout, classpath isolation, and more too. > >> No matter what we do it will be a large effort to port changes from one > >>code base to another, and from clojure to java. I proposed this initially > >>because it can be broken up into incremental changes. It may take a little > >>longer, but we will always have a working codebase that is testable and > >>compatible with the current storm release, at least until we move the user > >>facing APIs to be under org.apache. This lets the community continue to > >>build and test the master branch and report problems that they find, which > >>is incredibly valuable. I personally don't think it will be much easier, > >>especially if we are intent on always maintaining compatibility with storm. > >>- Bobby > >> > >> > >> On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) > >><[email protected]> wrote: > >> > >> > >> Hi Taylor, > >> > >> > >> > >> Thanks for the merge plan. I have a question about “Phase 2.2”. > >> > >> Do you mean community plan to create a fresh new “java core” based on > >>current “clojure core” firstly, and then migrate the features from JStorm? > >> > >> If so, it confused me. It is really a huge job which might require a long > >>developing time to make it stable, while JStorm is already a stable version. > >> > >> The release planned to be release after Nov 11th has already run online > >>stably several month in Alibaba. > >> > >> Besides this, there are many valuable internal requirements in Alibaba, > >>the fast evolution of JStorm is forseeable in next few months. > >> > >> If the “java core” is totally fresh new, it might bring many problems for > >>the coming merge. > >> > >> So, from the point of this view, I think it is much better and easier to > >>migrate the features of “clojure core” basing on JStorm for the “java core”. > >> > >> Please correct me, if any misunderstanding. > >> > >> > >> > >> Regards > >> > >> Basti > >> > >> > >> > >> 发件人: P. Taylor Goetz [mailto:[email protected]] > >> 发送时间: 2015年11月11日 5:32 > >> 收件人: [email protected] > >> 主题: [DISCUSS] Plan for Merging JStorm Code > >> > >> > >> > >> Based on a number of discussions regarding merging the JStorm code, I’ve > >>tried to distill the ideas presented and inserted some of my own. The > >>result is below. > >> > >> > >> > >> I’ve divided the plan into three phases, though they are not necessarily > >>sequential — obviously some tasks can take place in parallel. > >> > >> > >> > >> None of this is set in stone, just presented for discussion. Any and all > >>comments are welcome. > >> > >> > >> > >> ------- > >> > >> > >> > >> Phase 1 - Plan for 0.11.x Release > >> > >> 1. Determine feature set for 0.11.x and publish to wiki [1]. > >> > >> 2. Announce feature-freeze for 0.11.x > >> > >> 3. Create 0.11.x branch from master (Phase 2.4 can begin.) > >> > >> 4. Release 0.11.0 (or whatever version # we want to use) > >> > >> 5. Bug fixes and subsequent releases from 0.11.x-branch > >> > >> > >> > >> > >> > >> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches) > >> > >> 1. Determine/document unique features in JStorm (e.g. classpath isolation, > >>cgroups, etc.) and create JIRA for migrating the feature. > >> > >> 2. Create JIRA for migrating each clojure component (or logical group of > >>components) to Java. Assumes tests will be ported as well. > >> > >> 3. Discuss/establish style guide for Java coding conventions. Consider > >>using Oracle’s or Google’s Java conventions as a base — they are both > >>pretty solid. > >> > >> 4. align package names (e.g backtype.storm --> org.apache.storm / > >>com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin) > >> > >> > >> > >> > >> > >> Phase 3 - Migrate Clojure --> Java > >> > >> 1. Port code/tests to Java, leveraging existing JStorm code wherever > >>possible (core functionality only, features distinct to JStorm migrated > >>separately). > >> > >> 2. Port JStorm-specific features. > >> > >> 3. Begin releasing preview/beta versions. > >> > >> 4. Code cleanup (across the board) and refactoring using established > >>coding conventions, and leveraging PMD/Checkstyle reports for reference. > >>(Note: good oportunity for new contributors.) > >> > >> 5. Release 0.12.0 (or whatever version # we want to use) and lift feature > >>freeze. > >> > >> > >> > >> > >> > >> Notes: > >> > >> We should consider bumping up to version 1.0 sometime soon and then > >>switching to semantic versioning [3] from then on. > >> > >> > >> > >> > >> > >> With the exception of package name alignment, the "jstorm-import" branch > >>will largely be read-only throughout the process. > >> > >> > >> > >> During migration, it's probably easiest to operate with two local clones > >>of the Apache Storm repo: one for working (i.e. checked out to working > >>branch) and one for reference/copying (i.e. checked out to "jstorm-import"). > >> > >> > >> > >> Feature-freeze probably only needs to be enforced against core > >>functionality. Components under "external" can likely be exempt, but we > >>should figure out a process for accepting and releasing new features during > >>the migration. > >> > >> > >> > >> Performance testing should be continuous throughout the process. Since we > >>don't really have ASF infrastructure for performance testing, we will need > >>a volunteer(s) to host and run the performance tests. Performance test > >>results can be posted to the wiki [2]. It would probably be a good idea to > >>establish a baseline with the 0.10.0 release. > >> > >> > >> > >> I’ve attached an analysis document Sean Zhong put together a while back to > >>the JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but > >>is still relevant and has a lot of good information. > >> > >> > >> > >> > >> > >> [1] > >>https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set > >> > >> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home > >> > >> [3] http://semver.org > >> > >> [4] https://issues.apache.org/jira/browse/STORM-717 > >> > >> > >> > >> > >> > >> -Taylor > >> > >> > >> > >> > >> > >> > > > > > >
