After taking some time to review the JStorm code and history of both projects, here are my thoughts:
Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes. The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]: "Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.” While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java. Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5]. To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process. So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.” Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved. That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members. In short, I would rather see us work together than apart. -Taylor [1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a [2] https://github.com/alibaba/jstorm/commits/master [3] https://github.com/alibaba/jstorm/blob/master/history.md [4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote: > Sorry I am coming late to the conversation on this. Looking at the code for > jstorm I am willing to entertain the idea of merging the two projects. I > understand that by using java we can make the project more accessible to a > larger number of developers, and that does seem to fit with the apache model > of community over code. But I don't really want change just for change's > sake. Look at kill_topology in both > java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java > > and in > clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj > > The java version is 61 lines of code and is not actually compatible with the > clojure version because it just counts argument for the wait parameter > instead of using a -w. The clojure version is 29, but the 15 of which are > the apache header, so it is closer to 14 lines. Granted, to someone who does > not know you have to work a bit to read it, but not that much. > My concerns would be two fold. > First is the effort involved in maintaining two separate code bases, sub > project or otherwise, is a lot. I don't want to do it, and I assume you > don't want to either, that is why you are talking about merging. We would > need to decide which code to keep and which code to drop, and in all > likelihood some of it will still be clojure. I also would not want to > restrict people from using clojure for new work if they want to, just like we > don't restrict java usage for new work. > Second is true feature parity. Storm is a moving target, I would not want to > replace any code, even if it is faster, without verifying that it has feature > parity, or it is a conscious decision that the missing features are not > needed. Especially around the security branch, which is scheduled for the > 0.10.0 release, but is not a part of jstorm at all. > - Bobby > > From: 封仲淹(纪君祥) <zhongyan.f...@alibaba-inc.com> > To: dev@storm.apache.org; tdunn...@maprtech.com > Cc: jstorm_...@googlegroups.com > Sent: Wednesday, October 29, 2014 4:27 AM > Subject: 答复: Could JStorm project collaborate with Storm? > > Thanks for your suggestion. > > @dev, Does anyone has proposal? please feel free to raise. > > > Thanks > Longda > > -----邮件原件----- > 发件人: Ted Dunning [mailto:tdunn...@maprtech.com] > 发送时间: 2014年10月25日 7:27 > 收件人: dev@storm.apache.org > 主题: Re: Could JStorm project collaborate with Storm? > > I just moved all of the non-Storm-dev addresses on this thread to bcc. If > anybody is not on the Storm dev list and would like to continue to follow > this conversation, you should subscribe to the storm dev list. We should not > be spamming everybody else. > > > There is a standard Apache process to deal with this. JStorm can become an > incubator project. Once JStorm demonstrates the ability to function as an > Apache project, it can graduate either to a top-level project or to become > part of a project like Storm. > [Longda]This will be a feasible solution. From the Apache side, it had better > Jstorm can collaborate with Storm. > > A second mechanism is that if the Storm group is interested, they can > directly adopt the code as it is, possibly asking one or more of the > contributors to become committers to help support this new code. That is a > much harder thing to do than incubation since it is unusual for a project to > be enthusiastic about supporting a large new codebase. > [Longda]I agree. It is a hard job. > > > > > > > On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <basti...@alibaba-inc.com> wrote: > >> Hi Andy, >> >> >> >> Thanks for your quick response. What you concern is reasonable and >> understandable. >> >> >> >> But maybe there is some misunderstanding on our proposal. According to >> the current development situation in China, java core is indeed to >> bring more contributors here. >> >> That is reason why we begin to think if it is also a way for us to >> provide our contribution to Storm project. It is true that it is a >> risk to switch the core in a short period. >> >> Actually, if possible, maybe we can consider to take JStorm as a >> subproject of Storm(A branch or some other ways…). We will be >> responsible for the maintain of it and try to propagate the features >> from Storm to JStorm, and continue to develop our features and >> improvements on this. You guys can try the new core and take a long >> time to estimate if it is worth doing the switch in the future or >> taking JStorm as official optional core of Storm for user. >> >> >> >> Regards >> >> Basti >> >> >> >> *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com>] >> *发送时间:* 2014年10月24日 0:24 >> *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca; >> f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com; >> cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org; >> d...@hortonworks.com; m.ben.frank...@gmail.com; >> benjamin.hind...@gmail.com >> *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com; >> mlc...@iflytek.com; z...@tencent.com >> *主题:* Re: 答复: Could JStorm project collaborate with Storm? >> >> >> >> Zhongyan: >> >> >> >> Yahoo almost took your path 2 years ago. After some discussion with >> Nathan, we decided to work with community on Storm, and some of us >> learned Clojure quickly. We are very glad that we did that. >> >> >> >> Under the assumption that Storm provides all features of JStorm, my >> personal vote will be NO for your proposal. >> >> - Storm has enjoyed its current success before Nathan built a very >> solid core in Clojure. We should not replace that core until we are 100% >> sure that alternative implementation is at least as good as our Clojure >> implementation. >> - Clojure has not prevent Storm attract contributors (currently 108). >> We have contributors who write code in Clojure and Java. It doesn’t take >> much time for one to be able to understand Clojure code. >> - The convergence of streaming processing and batch processing will >> occur at higher level of abstractions. Trident API, for example, is very >> much similar to batch API such as Pig or Cascading. >> - Including Jstorm under Storm will only create confusion to our user >> community at this stage. >> >> >> >> Why don’t you have discussion with 徐明明 to figure out an alternative >> path for JStorm? >> >> >> >> Thanks, >> >> >> >> Andy >> >> >> >> *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com> >> *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com> >> *Date: *Wednesday, October 22, 2014 at 11:37 PM >> *To: *Andy Feng <af...@yahoo-inc.com>, "nat...@nathanmarz.com" < >> nat...@nathanmarz.com>, "徐明明(护城)" <mingming.x...@alibaba-inc.com>, " >> ja...@cvk.ca" <ja...@cvk.ca>, "f...@infochimps.com" >> <f...@infochimps.com>, "david...@microsoft.com" >> <david...@microsoft.com>, "ptgo...@gmail.com" < ptgo...@gmail.com>, Doug >> Cutting <cutt...@apache.org>, " >> tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" < >> arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, " >> m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, " >> benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com> >> *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "scott.z...@vipshop.com" >> <scott.z...@vipshop.com>, "yannia...@tencent.com" >> <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>, >> "z...@tencent.com" < z...@tencent.com> >> *Subject: *答复: Could JStorm project collaborate with Storm? >> >> >> >> Andrew, Thanks for suggest. >> >> >> >> The problem is that the core of Storm is implemented by Clojure, if it >> is java, we are glad to merge all our commit into the Strom Trunk. >> >> If the core of Storm is implemented by Java, I think the contributor >> of Storm would be double. >> >> >> >> >> >> May I put forward 2 proposal: >> >> 1. Could Jstorm be the son project of Storm? >> >> 2. Task a long time to change Storm-core as Jstorm-core. >> >> a) If the core of Storm is implemented by Java, I think the >> contributor of Storm would be double. Once one user found one bug, >> maybe he is able to fix it by himself, in this method, Storm resolve >> problem will be double >> >> b) In the next one or two years, batch handling and stream handling >> will merge into one solution, Spark/Flink are doing this job. If we >> still use clojure, I am a little afraid that we can’t follow the steps >> of other community and this direction. >> >> >> >> >> >> Thanks >> >> Longda >> >> >> >> *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com>] >> *发送时间:* 2014年10月23日 13:31 >> *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca; >> f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com; >> cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org; >> d...@hortonworks.com; m.ben.frank...@gmail.com; >> benjamin.hind...@gmail.com >> *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com; >> mlc...@iflytek.com; z...@tencent.com >> *主题:* Re: Could JStorm project collaborate with Storm? >> >> >> >> Zhongyan: >> >> >> >> I would suggest you identify a collection of key features in Jstorm, >> and then propose pull requests to Storm. >> >> Once these pull requests are accepted, you could then simply adopt Storm. >> >> >> >> Andy >> >> >> >> *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com> >> *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com> >> *Date: *Wednesday, October 22, 2014 at 8:23 PM >> *To: *"nat...@nathanmarz.com" <nat...@nathanmarz.com>, "徐明明(护城)" < >> mingming.x...@alibaba-inc.com>, "ja...@cvk.ca" <ja...@cvk.ca>, Andy >> Feng < af...@yahoo-inc.com>, "f...@infochimps.com" <f...@infochimps.com>, " >> david...@microsoft.com" <david...@microsoft.com>, "ptgo...@gmail.com" >> < ptgo...@gmail.com>, Doug Cutting <cutt...@apache.org>, " >> tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" < >> arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, " >> m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, " >> benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com> >> *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "scott.z...@vipshop.com" >> <scott.z...@vipshop.com>, "yannia...@tencent.com" >> <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>, >> "z...@tencent.com" < z...@tencent.com> >> *Subject: *Could JStorm project collaborate with Storm? >> >> >> >> Hi all, >> >> >> >> Firstly, congratulate Storm becoming the top project of Apache. Since >> the first day of Storm opened source, it has become one of hottest >> technologies in the BigData field and brought a new perspective to the >> world, which has helped many companies resolve hundreds of requirements. >> >> >> >> Since 2012/4, we have forked a project named "JStorm". At the >> beginning time, we just wanted to implement Storm with Java. As time >> goes on, more and more features have been added to JStorm. In 2013/9, >> JStorm changed to be an open source project; from then on, more and >> more people and companies join this project. >> >> >> >> As JStorm is derived from Storm, and born in a great company named >> Alibaba, which just successfully listed on the New York Stock Exchange >> on Sept. 19, is a very open company. We also want to share our >> technology with more people, and favor rapid evolution of streaming >> technology. >> >> >> >> *So, could JStorm project collaborate with Storm? * >> >> >> >> A little introduction for JStorm as follows: >> >> JStorm is a distributed and fault-tolerant real-time computation system. >> Inspired by Apache Storm, JStorm has been completely rewritten in Java >> and provides more enhanced features. JStorm has been widely used in >> many enterprise environments and proved robust and stable. >> >> >> >> Begun from 2012/2, we have released 11 versions till 2014/09/28. More >> than >> 10 companies are using it, including several leading companies such as >> Alibaba, Tencent, Baidu, Huawei, ZTE in China. >> >> >> >> Please refer to https://github.com/alibaba/jstorm for more details >> >> >> >> >> >> Regards >> >> Longda >> > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
