I think this idea has merits. I know I am much more comfortable in coding to the core kafka producer code and implementing kafka consumes but want to spend more time in the log code and a few other places (time always a variable)
I think we could get a lot more lean on things if we went with this type of approach though I think the wiring protocol has to be something that then bubbles up to either side producer/consumer ( on either fetch or response) I think the ability and desire from the comunity to keep building implementations of producer and consumer code also needs to keep a clean and healthy growth On Thu, Oct 4, 2012 at 4:13 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey guys, > > The number of developers and code base size for Kafka is getting larger. > > One way to help scale things gracefully would be to have an official idea > of "subsystems" and have official maintainers for those. The duties of a > maintainer would be > 1. Be the final word on API design in that area > 2. Ensure sufficient documentation and test coverage for that subsystem > 3. Review all code changes in that sub-system area > 4. Ensure that patches in that area get applied in a timely fashion > > In particular I think we could do a better job of getting patches in in a > timely manner. > > Here are what I see as logically distinct systems or areas: > > - Producer (java and scala) > - Consumer (java and scala) > - Network layer (kafka.network.*) > - Log (kafka.log.*) > - Replication (controller, fetcher threads, hw mark stuff, etc) > - Kafka API impl (basically just KafkaApi.scala) > - Hadoop stuff > - Perf tools and system tests > - Misc other small things: metrics, utils, etc. > > Obviously many features will cut across these layers, but the idea is that > by having a real owner that is responsible for that area we will get higher > quality. > > I think we are doing this informally already, but making it formal would > help ensure you knew the right people to get input from. I think it > probably wouldn't make sense to start doing this until post-0.8 since we > are in the middle of so many things right now, but I wanted to see what > people thought...? > > -Jay > -- /* Joe Stein http://www.linkedin.com/in/charmalloc Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> */
