Hi Taylor, I'd like to help too, could you add me in? my id is: hustfxj -----邮件原件----- 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] 发送时间: 2015年11月24日 23:54 收件人: dev@storm.apache.org 主题: Re: [DISCUSS] Plan for Merging JStorm Code
Hi Cody, I wasn’t able to find your username. Are you sure you have an account on cwiki.apache.org? -Taylor > On Nov 22, 2015, at 8:46 AM, Cody Innowhere <e.neve...@gmail.com> wrote: > > Hi Taylor, > I'd like to help too, could you add me in? my id is: Cody > > On Sat, Nov 21, 2015 at 11:51 AM, 刘键(Basti Liu) > <basti...@alibaba-inc.com> > wrote: > >> Hi Taylor， >> >> Sorry for the late response. >> I'd like to help on this. Could you please help to give me the permission? >> Thanks. >> UserName: basti.lj >> >> Regards >> Basti >> >> -----Original Message----- >> From: P. Taylor Goetz [mailto:ptgo...@gmail.com] >> Sent: Thursday, November 19, 2015 6:24 AM >> To: dev@storm.apache.org; Bobby Evans >> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >> >> All I have at this point is a placeholder wiki entry [1], and a lot >> of local notes that likely would only make sense to me. >> >> Let me know your wiki username and I’ll give you permissions. The >> same goes for anyone else who wants to help. >> >> -Taylor >> >> [1] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6132 >> 8109 >> >>> On Nov 18, 2015, at 2:08 PM, Bobby Evans >>> <ev...@yahoo-inc.com.INVALID> >> wrote: >>> >>> Taylor and others I was hoping to get started filing JIRA and >>> planning on how we are going to do the java migration + JStorm >>> merger. Is anyone else starting to do this? If not would anyone >>> object to me starting on it? - Bobby >>> >>> >>> On Thursday, November 12, 2015 12:04 PM, P. Taylor Goetz < >> ptgo...@gmail.com> wrote: >>> >>> >>> Thanks for putting this together Basti, that comparison helps a lot. >>> >>> And thanks Bobby for converting it into markdown. I was going to >>> just >> attach the spreadsheet to JIRA, but markdown is a much better solution. >>> >>> -Taylor >>> >>>> On Nov 12, 2015, at 12:03 PM, Bobby Evans >>>> <ev...@yahoo-inc.com.INVALID> >> wrote: >>>> >>>> I translated the excel spreadsheet into a markdown file and put up >>>> a >> pull request for it. >>>> https://github.com/apache/storm/pull/877 >>>> I did a few edits to it to make it work with Markdown, and to add >>>> in a >> few of my own comments. I also put in a field for JIRAs to be able >> to track the migration. >>>> Overall I think your evaluation was very good. We have a fair >>>> amount >> of work ahead of us to decide what version of various features we >> want to go forward with. >>>> - Bobby >>>> >>>> >>>> On Thursday, November 12, 2015 9:37 AM, 刘键(Basti Liu) < >> basti...@alibaba-inc.com> wrote: >>>> >>>> >>>> Hi Bobby & Jungtaek, >>>> >>>> Thanks for your replay. >>>> I totally agree that compatibility is the most important thing. >> Actually, JStorm has been compatible with the user API of Storm. >>>> As you mentioned below, we indeed still have some features >>>> different >> between Storm and JStorm. I have tried to list them (minor update or >> improvements are not included). >>>> Please refer to attachment for details. If any missing, please help >>>> to point out. (The current working features are probably missing >>>> here.) >> Just have a look at these differences. For the missing features in >> JStorm, I did not see any obstacle which will block the merge to JStorm. >>>> For the features which has different solution between Storm and >>>> JStorm, >> we can evaluate the solution one by one to decision which one is >> appropriate. >>>> After the finalization of evaluation, I think JStorm team can take >>>> the >> merging job and publish a stable release in 2 months. >>>> But anyway, the detailed implementation for these features with >> different solution is transparent to user. So, from user's point of >> view, there is not any compatibility problem. >>>> >>>> Besides compatibility, by our experience, stability is also >>>> important >> and is not an easy job. 4 people in JStorm team took almost one year >> to finish the porting from "clojure core" >>>> to "java core", and to make it stable. Of course, we have many devs >>>> in >> community to make the porting job faster. But it still needs a long >> time to run many online complex topologys to find bugs and fix them. >> So, that is the reason why I proposed to do merging and build on a stable >> "java core". >>>> >>>> -----Original Message----- >>>> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] >>>> Sent: Wednesday, November 11, 2015 10:51 PM >>>> To: dev@storm.apache.org >>>> Subject: Re: [DISCUSS] Plan for Merging JStorm Code >>>> >>>> +1 for doing a 1.0 release based off of the clojure 0.11.x code. >> Migrating the APIs to org.apache.storm is a big non-backwards >> compatible move, and a major version bump to 2.x seems like a good move >> there. >>>> +1 for the release plan >>>> >>>> I would like the move for user facing APIs to org.apache to be one >>>> of >> the last things we do. Translating clojure code into java and moving >> it to org.apache I am not too concerned about. >>>> >>>> Basti, >>>> We have two code bases that have diverged significantly from one >> another in terms of functionality. The storm code now or soon will >> have A Heartbeat Server, Nimbus HA (Different Implementation), >> Resource Aware Scheduling, a distributed cache like API, log >> searching, security, massive performance improvements, shaded almost >> all of our dependencies, a REST API for programtically accessing >> everything on the UI, and I am sure I am missing a few other things. >> JStorm also has many changes including cgroup isolation, restructured >> zookeeper layout, classpath isolation, and more too. >>>> No matter what we do it will be a large effort to port changes from >>>> one code base to another, and from clojure to java. I proposed >>>> this initially because it can be broken up into incremental >>>> changes. It may take a little longer, but we will always have a >>>> working codebase that is testable and compatible with the current >>>> storm release, at least until we move the user facing APIs to be under >>>> org.apache. >>>> This lets the community continue to build and test the master >>>> branch and report problems that they find, which is incredibly >>>> valuable. I personally don't think it will be much easier, >>>> especially if we are intent on always maintaining compatibility >>>> with storm. - Bobby >>>> >>>> >>>> On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu) < >> basti...@alibaba-inc.com> wrote: >>>> >>>> >>>> Hi Taylor, >>>> >>>> >>>> >>>> Thanks for the merge plan. I have a question about “Phase 2.2”. >>>> >>>> Do you mean community plan to create a fresh new “java core” based >>>> on >> current “clojure core” firstly, and then migrate the features from JStorm? >>>> >>>> If so, it confused me. It is really a huge job which might require >>>> a >> long developing time to make it stable, while JStorm is already a >> stable version. >>>> >>>> The release planned to be release after Nov 11th has already run >>>> online >> stably several month in Alibaba. >>>> >>>> Besides this, there are many valuable internal requirements in >>>> Alibaba, >> the fast evolution of JStorm is forseeable in next few months. >>>> >>>> If the “java core” is totally fresh new, it might bring many >>>> problems >> for the coming merge. >>>> >>>> So, from the point of this view, I think it is much better and >>>> easier >> to migrate the features of “clojure core” basing on JStorm for the >> “java core”. >>>> >>>> Please correct me, if any misunderstanding. >>>> >>>> >>>> >>>> Regards >>>> >>>> Basti >>>> >>>> >>>> >>>> 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] >>>> 发送时间: 2015年11月11日 5:32 >>>> 收件人: dev@storm.apache.org >>>> 主题: [DISCUSS] Plan for Merging JStorm Code >>>> >>>> >>>> >>>> Based on a number of discussions regarding merging the JStorm code, >> I’ve tried to distill the ideas presented and inserted some of my >> own. The result is below. >>>> >>>> >>>> >>>> I’ve divided the plan into three phases, though they are not >> necessarily sequential — obviously some tasks can take place in parallel. >>>> >>>> >>>> >>>> None of this is set in stone, just presented for discussion. Any >>>> and >> all comments are welcome. >>>> >>>> >>>> >>>> ------- >>>> >>>> >>>> >>>> Phase 1 - Plan for 0.11.x Release >>>> >>>> 1. Determine feature set for 0.11.x and publish to wiki [1]. >>>> >>>> 2. Announce feature-freeze for 0.11.x >>>> >>>> 3. Create 0.11.x branch from master (Phase 2.4 can begin.) >>>> >>>> 4. Release 0.11.0 (or whatever version # we want to use) >>>> >>>> 5. Bug fixes and subsequent releases from 0.11.x-branch >>>> >>>> >>>> >>>> >>>> >>>> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches) >>>> >>>> 1. Determine/document unique features in JStorm (e.g. classpath >> isolation, cgroups, etc.) and create JIRA for migrating the feature. >>>> >>>> 2. Create JIRA for migrating each clojure component (or logical >>>> group >> of components) to Java. Assumes tests will be ported as well. >>>> >>>> 3. Discuss/establish style guide for Java coding conventions. >>>> Consider >> using Oracle’s or Google’s Java conventions as a base — they are both >> pretty solid. >>>> >>>> 4. align package names (e.g backtype.storm --> org.apache.storm / >>>> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin) >>>> >>>> >>>> >>>> >>>> >>>> Phase 3 - Migrate Clojure --> Java >>>> >>>> 1. Port code/tests to Java, leveraging existing JStorm code >>>> wherever >> possible (core functionality only, features distinct to JStorm >> migrated separately). >>>> >>>> 2. Port JStorm-specific features. >>>> >>>> 3. Begin releasing preview/beta versions. >>>> >>>> 4. Code cleanup (across the board) and refactoring using >>>> established coding conventions, and leveraging PMD/Checkstyle >>>> reports for reference. (Note: good oportunity for new >>>> contributors.) >>>> >>>> 5. Release 0.12.0 (or whatever version # we want to use) and lift >> feature freeze. >>>> >>>> >>>> >>>> >>>> >>>> Notes: >>>> >>>> We should consider bumping up to version 1.0 sometime soon and then >> switching to semantic versioning [3] from then on. >>>> >>>> >>>> >>>> >>>> >>>> With the exception of package name alignment, the "jstorm-import" >> branch will largely be read-only throughout the process. >>>> >>>> >>>> >>>> During migration, it's probably easiest to operate with two local >> clones of the Apache Storm repo: one for working (i.e. checked out to >> working branch) and one for reference/copying (i.e. checked out to >> "jstorm-import"). >>>> >>>> >>>> >>>> Feature-freeze probably only needs to be enforced against core >> functionality. Components under "external" can likely be exempt, but >> we should figure out a process for accepting and releasing new >> features during the migration. >>>> >>>> >>>> >>>> Performance testing should be continuous throughout the process. >>>> Since >> we don't really have ASF infrastructure for performance testing, we >> will need a volunteer(s) to host and run the performance tests. >> Performance test results can be posted to the wiki [2]. It would >> probably be a good idea to establish a baseline with the 0.10.0 release. >>>> >>>> >>>> >>>> I’ve attached an analysis document Sean Zhong put together a while >>>> back >> to the JStorm merge JIRA [4]. The analysis was against the 0.9.3 >> release but is still relevant and has a lot of good information. >>>> >>>> >>>> >>>> >>>> >>>> [1] >>>> https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Fe >>>> at >>>> ure+Set >>>> >>>> [2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home >>>> >>>> [3] http://semver.org >>>> >>>> [4] https://issues.apache.org/jira/browse/STORM-717 >>>> >>>> >>>> >>>> >>>> >>>> -Taylor >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >> >> >>
