Sean, Thanks for your response.
Could you please leave a single phone call number to me in a single email, I will contact to you later. Regards Longda 发件人: Sean Zhong [mailto:clock...@gmail.com] 发送时间: 2014年10月30日 13:27 收件人: dev@storm.apache.org 抄送: 封仲淹(纪君祥); jstorm_...@googlegroups.com 主题: Re: Could JStorm project collaborate with Storm? Resend this, I missed some recipients in last mail I love this discussion. Let's list the facts first: 1. JStorm and Storm have different feature set, but we both target on the same goals. 2. We have two community and doing things in duplicated fashion. 3. Both realize that It will take huge efforts to absort the common and good things from the other with different source repo. 4. Storm doesn't preclude java, Clojure and Java works pretty well together in storm. As Talyor said 66% of code is java. 5. Both want to do thing in a open way in apache that can everyone around the world is comfortable, and be benefited from this process. So, IMHO, based on the above facts, we should merge these two together under the name of storm apache project. For the difference of these two projects, here is my suggestions: 1. Review the change at the feature level. Instead of saying whether we should replace the core from clojure to java as a whole, let's think in the view of single feature. What is the best decision for this feature. In my opinion, storm are open to the changes, including changing some part of clojure to java, but only when necessary. 2. Make the new feature plugable, and general. If we can make some plug point in storm clojure core, then the implementation can be either in java or clojure. For example, like the JStorm scheduler feature, it is more powerful than storm. Can we stick to the java code JStorm scheduler, and make a plug point in storm? They are good examples for this practice in storm already. Like the storm messaging layer. The netty implementation is in java, the plugin point is in clojure. By following this process, it can largely reduce the refactory efforts we need to make. 3. Colloborate to solve the gap of clojure language, across Jstorm and Storm community. Though I personally think Clojure is not that hard to learn, but it do requires a steep learning curve. Insteading of letting JStorm to program with a new language from zero, I suggest we stick to our most productive language in the begining, and do things colloborately in community. Java guys and Clojure guys from two community can decide the interface(plug point) first, works seperately in their own favorable language. 4. More communication channels beyond mail to negotiate the difference and make sure we have same understanding. We also need some online chat, face to face meetup to do a more effiicient and fast loop communication and negotiation. It is important to establish common understanding. Since JStorm guys are mostly in China. There are two guys of storm commitor in China, Xu Mingming and me(located @Intel Shanghai). I would love to volunteer to be one additional channel for early stage discussion, so that we can minimize the differences faster. By then, we will have a further considerate proposal that are acceptable for both community. On Thu, Oct 30, 2014 at 4:17 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote: After taking some time to review the JStorm code and history of both projects, here are my thoughts: Apache Storm is largely implemented in Java (~68%) and Clojure (~18%). The disparity in LOC is primarily due to the conciseness of Clojure compared to Java. Apache Storm's Clojure code consists mainly of implementations of interfaces defined in Java. This fact makes it possible to provide an implementation in Java itself, which is exactly what the JStorm project seems to have done. JStorm is a copy of the Apache Storm codebase, with the Clojure code being replaced with a Java implementation, with some additional changes. The justification of the project centers around a distaste for Clojure, and a seeming lack of developers with Clojure skills [1]: "Storm is wonderful product, but it is implemented with Clojure. We don't like clojure. it isn't a populate language, so in Alibaba, few people can fix the bug of storm. We are the first users of Storm since it has been open source. During using, we found several problem such as zeromq, zookeeper, performance.” While I agree that to someone experienced with Java development, the first exposure to Clojure can be a bit jarring, I don’t think it is much of a barrier to entry. In my opinion, learning Clojure is a valuable investment in making oneself a more well-rounded developer. The technology ecosystem to which Storm belongs is becoming increasingly polyglot with respect to JVM languages (Java, Scala, Clojure, etc.), and developers need to adapt to that fact. As an example (perhaps not the best example), lets look at Kafka. If I didn’t like or want to learn Scala, would it make sense to write a clone in Java and attempt to keep feature parity and compatibility, or would it be better to bite the bullet and learn Scala? (Yes, I know it’s not the best example since Storm’s Clojure code is a subset of the codebase, and the way it’s designed makes it possible for alternative implementations). And in reality, someone from Alibaba had to at least learn enough Clojure to be able to replicate Storm’s Clojure implementation in java. Looking at the history of JStorm [2][3], it appears a lot of work has gone into keeping feature parity with Apache Storm by copying in new code or reimplementing it. For example a kafka spout was added about two weeks ago [4], and Yahoo’s storm-on-yarn project was also recently copied in [5]. To me, this kind of work is rather counter-productive. If the Apache Storm community feels strongly that Storm should support multiple “core” implementations (i.e. one implemented in Clojure, and one implemented in Java), then it could be accommodated by making them pluggable and rearranging the codebase a bit. That being said, the Apache Storm community has invested a significant amount of time and effort maintaining and improving Storm’s Clojure “core.” One place this is most apparent, as Bobby mentioned, is the upcoming security release. To maintain feature parity with that release a “JStorm core” would have to be updated to implement that functionality, which is not a trivial undertaking. At some point as developers we need to ask ourselves “what's the point of that work?” There are a lot more people using Storm than developing internals. We need to make sure that both groups gain value from whatever we decide, and that we don’t sacrifice any value in the process. So to answer the question in the subject line of the email thread: Could the JStorm project collaborate with Apache Storm? My answer would be a resounding “yes.” Though the JStorm community is small in comparison to that of Apache Storm, it’s clear that the JStorm contributors have a solid grasp of Storm’s internals and would be welcomed by the Storm community. Reimplementing Storm’s core in Java no doubt provided insight into areas that might be improved. That being said, replacing Storm’s Clojure-based core is a decision that should not be taken lightly, and would require the support of the Apache Storm PMC and entire community. A “dual-core” setup could be an option, but again could become a maintenance nightmare with little value added for end users. I’m open to discussing options further, and welcome input from other community members. In short, I would rather see us work together than apart. -Taylor [1] https://github.com/alibaba/jstorm/commit/903ca941023dd37a54d9574e2a554883111f0b4a [2] https://github.com/alibaba/jstorm/commits/master [3] https://github.com/alibaba/jstorm/blob/master/history.md [4] https://github.com/alibaba/jstorm/tree/master/jstorm-utility/jstorm-kafka [5] https://github.com/alibaba/jstorm/tree/master/jstorm-on-yarn On Oct 29, 2014, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote: Sorry I am coming late to the conversation on this. Looking at the code for jstorm I am willing to entertain the idea of merging the two projects. I understand that by using java we can make the project more accessible to a larger number of developers, and that does seem to fit with the apache model of community over code. But I don't really want change just for change's sake. Look at kill_topology in both java:https://github.com/alibaba/jstorm/blob/master/jstorm-client/src/main/java/backtype/storm/command/kill_topology.java and in clojure:https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/command/kill_topology.clj The java version is 61 lines of code and is not actually compatible with the clojure version because it just counts argument for the wait parameter instead of using a -w. The clojure version is 29, but the 15 of which are the apache header, so it is closer to 14 lines. Granted, to someone who does not know you have to work a bit to read it, but not that much. My concerns would be two fold. First is the effort involved in maintaining two separate code bases, sub project or otherwise, is a lot. I don't want to do it, and I assume you don't want to either, that is why you are talking about merging. We would need to decide which code to keep and which code to drop, and in all likelihood some of it will still be clojure. I also would not want to restrict people from using clojure for new work if they want to, just like we don't restrict java usage for new work. Second is true feature parity. Storm is a moving target, I would not want to replace any code, even if it is faster, without verifying that it has feature parity, or it is a conscious decision that the missing features are not needed. Especially around the security branch, which is scheduled for the 0.10.0 release, but is not a part of jstorm at all. - Bobby From: 封仲淹(纪君祥) <zhongyan.f...@alibaba-inc.com> To: dev@storm.apache.org; tdunn...@maprtech.com Cc: jstorm_...@googlegroups.com Sent: Wednesday, October 29, 2014 4:27 AM Subject: 答复: Could JStorm project collaborate with Storm? Thanks for your suggestion. @dev, Does anyone has proposal? please feel free to raise. Thanks Longda -----邮件原件----- 发件人: Ted Dunning [mailto:tdunn...@maprtech.com] 发送时间: 2014年10月25日 7:27 收件人: dev@storm.apache.org 主题: Re: Could JStorm project collaborate with Storm? I just moved all of the non-Storm-dev addresses on this thread to bcc. If anybody is not on the Storm dev list and would like to continue to follow this conversation, you should subscribe to the storm dev list. We should not be spamming everybody else. There is a standard Apache process to deal with this. JStorm can become an incubator project. Once JStorm demonstrates the ability to function as an Apache project, it can graduate either to a top-level project or to become part of a project like Storm. [Longda]This will be a feasible solution. From the Apache side, it had better Jstorm can collaborate with Storm. A second mechanism is that if the Storm group is interested, they can directly adopt the code as it is, possibly asking one or more of the contributors to become committers to help support this new code. That is a much harder thing to do than incubation since it is unusual for a project to be enthusiastic about supporting a large new codebase. [Longda]I agree. It is a hard job. On Thu, Oct 23, 2014 at 8:19 PM, 刘键(夏蒅) <basti...@alibaba-inc.com> wrote: Hi Andy, Thanks for your quick response. What you concern is reasonable and understandable. But maybe there is some misunderstanding on our proposal. According to the current development situation in China, java core is indeed to bring more contributors here. That is reason why we begin to think if it is also a way for us to provide our contribution to Storm project. It is true that it is a risk to switch the core in a short period. Actually, if possible, maybe we can consider to take JStorm as a subproject of Storm(A branch or some other ways…). We will be responsible for the maintain of it and try to propagate the features from Storm to JStorm, and continue to develop our features and improvements on this. You guys can try the new core and take a long time to estimate if it is worth doing the switch in the future or taking JStorm as official optional core of Storm for user. Regards Basti *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com>] *发送时间:* 2014年10月24日 0:24 *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca; f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com; cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org; d...@hortonworks.com; m.ben.frank...@gmail.com; benjamin.hind...@gmail.com *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com; mlc...@iflytek.com; z...@tencent.com *主题:* Re: 答复: Could JStorm project collaborate with Storm? Zhongyan: Yahoo almost took your path 2 years ago. After some discussion with Nathan, we decided to work with community on Storm, and some of us learned Clojure quickly. We are very glad that we did that. Under the assumption that Storm provides all features of JStorm, my personal vote will be NO for your proposal. - Storm has enjoyed its current success before Nathan built a very solid core in Clojure. We should not replace that core until we are 100% sure that alternative implementation is at least as good as our Clojure implementation. - Clojure has not prevent Storm attract contributors (currently 108). We have contributors who write code in Clojure and Java. It doesn’t take much time for one to be able to understand Clojure code. - The convergence of streaming processing and batch processing will occur at higher level of abstractions. Trident API, for example, is very much similar to batch API such as Pig or Cascading. - Including Jstorm under Storm will only create confusion to our user community at this stage. Why don’t you have discussion with 徐明明 to figure out an alternative path for JStorm? Thanks, Andy *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com> *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com> *Date: *Wednesday, October 22, 2014 at 11:37 PM *To: *Andy Feng <af...@yahoo-inc.com>, "nat...@nathanmarz.com" < nat...@nathanmarz.com>, "徐明明(护城)" <mingming.x...@alibaba-inc.com>, " ja...@cvk.ca" <ja...@cvk.ca>, "f...@infochimps.com" <f...@infochimps.com>, "david...@microsoft.com" <david...@microsoft.com>, "ptgo...@gmail.com" < ptgo...@gmail.com>, Doug Cutting <cutt...@apache.org>, " tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" < arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, " m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, " benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com> *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "scott.z...@vipshop.com" <scott.z...@vipshop.com>, "yannia...@tencent.com" <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>, "z...@tencent.com" < z...@tencent.com> *Subject: *答复: Could JStorm project collaborate with Storm? Andrew, Thanks for suggest. The problem is that the core of Storm is implemented by Clojure, if it is java, we are glad to merge all our commit into the Strom Trunk. If the core of Storm is implemented by Java, I think the contributor of Storm would be double. May I put forward 2 proposal: 1. Could Jstorm be the son project of Storm? 2. Task a long time to change Storm-core as Jstorm-core. a) If the core of Storm is implemented by Java, I think the contributor of Storm would be double. Once one user found one bug, maybe he is able to fix it by himself, in this method, Storm resolve problem will be double b) In the next one or two years, batch handling and stream handling will merge into one solution, Spark/Flink are doing this job. If we still use clojure, I am a little afraid that we can’t follow the steps of other community and this direction. Thanks Longda *发件人:* Andrew Feng [mailto:af...@yahoo-inc.com <af...@yahoo-inc.com>] *发送时间:* 2014年10月23日 13:31 *收件人:* 封仲淹(纪君祥); nat...@nathanmarz.com; 徐明明(护城); ja...@cvk.ca; f...@infochimps.com; david...@microsoft.com; ptgo...@gmail.com; cutt...@apache.org; tdunn...@maprtech.com; arv...@apache.org; d...@hortonworks.com; m.ben.frank...@gmail.com; benjamin.hind...@gmail.com *抄送:* aloha-dev; scott.z...@vipshop.com; yannia...@tencent.com; mlc...@iflytek.com; z...@tencent.com *主题:* Re: Could JStorm project collaborate with Storm? Zhongyan: I would suggest you identify a collection of key features in Jstorm, and then propose pull requests to Storm. Once these pull requests are accepted, you could then simply adopt Storm. Andy *From: *"封仲淹 (纪君祥)" <zhongyan.f...@alibaba-inc.com> *Reply-To: *"封仲淹(纪君祥)" <zhongyan.f...@alibaba-inc.com> *Date: *Wednesday, October 22, 2014 at 8:23 PM *To: *"nat...@nathanmarz.com" <nat...@nathanmarz.com>, "徐明明(护城)" < mingming.x...@alibaba-inc.com>, "ja...@cvk.ca" <ja...@cvk.ca>, Andy Feng < af...@yahoo-inc.com>, "f...@infochimps.com" <f...@infochimps.com>, " david...@microsoft.com" <david...@microsoft.com>, "ptgo...@gmail.com" < ptgo...@gmail.com>, Doug Cutting <cutt...@apache.org>, " tdunn...@maprtech.com" <tdunn...@maprtech.com>, "arv...@apache.org" < arv...@apache.org>, "d...@hortonworks.com" <d...@hortonworks.com>, " m.ben.frank...@gmail.com" <m.ben.frank...@gmail.com>, " benjamin.hind...@gmail.com" <benjamin.hind...@gmail.com> *Cc: *aloha-dev <aloha-...@list.alibaba-inc.com>, "scott.z...@vipshop.com" <scott.z...@vipshop.com>, "yannia...@tencent.com" <yannia...@tencent.com>, "mlc...@iflytek.com" <mlc...@iflytek.com>, "z...@tencent.com" < z...@tencent.com> *Subject: *Could JStorm project collaborate with Storm? Hi all, Firstly, congratulate Storm becoming the top project of Apache. Since the first day of Storm opened source, it has become one of hottest technologies in the BigData field and brought a new perspective to the world, which has helped many companies resolve hundreds of requirements. Since 2012/4, we have forked a project named "JStorm". At the beginning time, we just wanted to implement Storm with Java. As time goes on, more and more features have been added to JStorm. In 2013/9, JStorm changed to be an open source project; from then on, more and more people and companies join this project. As JStorm is derived from Storm, and born in a great company named Alibaba, which just successfully listed on the New York Stock Exchange on Sept. 19, is a very open company. We also want to share our technology with more people, and favor rapid evolution of streaming technology. *So, could JStorm project collaborate with Storm? * A little introduction for JStorm as follows: JStorm is a distributed and fault-tolerant real-time computation system. Inspired by Apache Storm, JStorm has been completely rewritten in Java and provides more enhanced features. JStorm has been widely used in many enterprise environments and proved robust and stable. Begun from 2012/2, we have released 11 versions till 2014/09/28. More than 10 companies are using it, including several leading companies such as Alibaba, Tencent, Baidu, Huawei, ZTE in China. Please refer to https://github.com/alibaba/jstorm for more details Regards Longda
