hi, I am in agreement with Taylor and believe I understand his intent. An incredible tool/framework/application like Storm is only enhanced and gains value from the number of well maintained and vetted modules that can be used for integration and adding further functionality.
I am relatively new to the Storm community but have spent quite some time reviewing contributing modules out there, reviewing various duplicates and running into some version incompatibilities. I understand the need to keep Storm itself pure, but do think there needs to be some structure and governance added to the contributing modules. Look at the benefit a tool like npm brings to the node community. I like the idea of sponsorship, vetting and a community vote. I, as sure many would be, am willing to offer support and time to working through how to set this up and helping with the implementation if it is decided to pursue some solution. I hope these views are taken in the sprit they are made, to make this incredible system even better along with the surrounding eco-system. Thanks, Brian On Tue, Feb 25, 2014 at 9:36 PM, P. Taylor Goetz <[email protected]> wrote: > Just to be clear (and play a little Devil's advocate :) ), I'm not > suggesting that whatever a "contrib" project/module/subproject might > become, be a clearinghouse for anything Storm-related. > > I see it as something that is well-vetted by the Storm community, subject > to PPMC review, vote, etc. Entry would require community review, PPMC > review, and in some cases ASF IP clearance/legal review. Anything added > would require some level of commitment from the PPMC/committers to provide > some level of support. > > In other words, nothing "willy-nilly". > > One option could be that any module added require (X > 0) number of > committers to volunteer as "sponsor"s for the module, and commit to > maintaining it. > > That being said, I don't see storm-kafka being any different from anything > else that provides integration points for Storm. > > -Taylor > > > On Feb 25, 2014, at 7:53 PM, Nathan Marz <[email protected]> wrote: > > I'm only +1 for pulling in storm-kafka and updating it. Other projects put > these contrib modules in a "contrib" folder and keep them managed as > completely separate codebases. As it's not actually a "module" necessary > for Storm, there's an argument there for doing it that way rather than via > the multi-module route. > > > On Tue, Feb 25, 2014 at 4:39 PM, Milinda Pathirage > <[email protected]>wrote: > >> Hi Taylor, >> >> I'm +1 for pulling these external libraries into Apache codebase. This >> will certainly benifit Strom community. I also like to contribute to >> this process. >> >> Thanks >> Milinda >> >> On Tue, Feb 25, 2014 at 5:28 PM, P. Taylor Goetz <[email protected]> >> wrote: >> > A while back I opened STORM-206 [1] to capture ideas for pulling in >> > "contrib" modules to the Apache codebase. >> > >> > In the past, we had the storm-contrib github project [2] which >> subsequently >> > got broken up into individual projects hosted on the stormprocessor >> github >> > group [3] and elsewhere. >> > >> > The problem with this approach is that in certain cases it led to code >> rot >> > (modules not being updated in step with Storm's API), fragmentation >> > (multiple similar modules with the same name), and confusion. >> > >> > A good example of this is the storm-kafka module [4], since it is a >> widely >> > used component. Because storm-contrib wasn't being tagged in github, a >> lot >> > of users had trouble reconciling with which versions of storm it was >> > compatible. Some users built off specific commit hashes, some forked, >> and a >> > few even pushed custom builds to repositories such as clojars. With >> kafka >> > 0.8 now available, there are two main storm-kafka projects, the original >> > (compatible with kafka 0.7) and an updated fork [5] (compatible with >> kafka >> > 0.8). >> > >> > My intention is not to find fault in any way, but rather to point out >> the >> > resulting pain, and work toward a better solution. >> > >> > I think it would be beneficial to the Storm user community to have >> certain >> > commonly used modules like storm-kafka brought into the Apache Storm >> > project. Another benefit worth considering is the licensing/legal >> oversight >> > that the ASF provides, which is important to many users. >> > >> > If this is something we want to do, then the big question becomes what >> sort >> > governance process needs to be established to ensure that such things >> are >> > properly maintained. >> > >> > Some random thoughts, questions, etc. that jump to mind include: >> > >> > What to call these things: "contib modules", "connectors", "integration >> > modules", etc.? >> > Build integration: I imagine they would be a multi-module submodule of >> the >> > main maven build. Probably turned off by default and enabled by a maven >> > profile. >> > Governance: Have one or more committer volunteers responsible for >> > maintenance, merging patches, etc.? Proposal process for pulling new >> > modules? >> > >> > >> > I look forward to hearing others' opinions. >> > >> > - Taylor >> > >> > >> > [1] https://issues.apache.org/jira/browse/STORM-206 >> > [2] https://github.com/nathanmarz/storm-contrib >> > [3] https://github.com/stormprocessor >> > [4] https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka >> > [5] https://github.com/wurstmeister/storm-kafka-0.8-plus >> >> >> >> -- >> Milinda Pathirage >> >> PhD Student | Research Assistant >> School of Informatics and Computing | Data to Insight Center >> Indiana University >> >> twitter: milindalakmal >> skype: milinda.pathirage >> blog: http://milinda.pathirage.org >> > > > > -- > Twitter: @nathanmarz > http://nathanmarz.com > > >
