@guozhang re: library vs framework - Nothing critical missing. But I think KIPs will serve as a sort of changelog since ones like this https://archive.apache.org/dist/kafka/0.8.2.0/RELEASE_NOTES.html are not all that helpful to end users. A KIP like this is a major new addition, so laying out all the benefits and tradeoffs wrt frameworks is important for users. Docs will cover that too, but Kafka tends to lag on documentation updates and at worst, the docs are just bootstrapped from very clear KIP text.
In terms of concrete changes, I'd love to see a few examples of how you would take advantage of it being in library form. There's the obvious benefit that you can easily integrate with whatever cluster/process management you want instead of requiring YARN/Mesos/etc, but having a library rather than framework also allows you to integrate with other code in the same process. The Copycat KIP showed how you might integrate Copycat and a stream processing library very generically, but maybe one or two real examples would be useful? What else is running in the process besides stream processing? All my comments were only meant as suggestions for how the KIP document might be improved. I'm already convinced that stream processing frameworks today are far too heavyweight and there's a gap to be filled between those frameworks and using producer/consumer directly. -Ewen On Fri, Jul 24, 2015 at 9:48 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Ewen, > > Replies inlined. > > On Thu, Jul 23, 2015 at 10:25 PM, Ewen Cheslack-Postava <e...@confluent.io > > > wrote: > > > Just some notes on the KIP doc itself: > > > > * It'd be useful to clarify at what point the plain consumer + custom > code > > + producer breaks down. I think trivial filtering and aggregation on a > > single stream usually work fine with this model. Anything where you need > > more complex joins, windowing, etc. are where it breaks down. I think > most > > interesting applications require that functionality, but it's helpful to > > make this really clear in the motivation -- right now, Kafka only > provides > > the lowest level plumbing for stream processing applications, so most > > interesting apps require very heavyweight frameworks. > > > > I think for users to efficiently express complex logic like joins > windowing, etc, a higher-level programming interface beyond the process() > interface would definitely be better, but that does not necessarily require > a "heavyweight" frameworks, which usually includes more than just the > high-level functional programming model. I would argue that an alternative > solution would better be provided for users who want some high-level > programming interface but not a heavyweight stream processing framework > that include the processor library plus another DSL layer on top of it. > > > > > * I think the feature comparison of plain producer/consumer, stream > > processing frameworks, and this new library is a good start, but we might > > want something more thorough and structured, like a feature matrix. Right > > now it's hard to figure out exactly how they relate to each other. > > > > Cool, I can do that. > > > > * I'd personally push the library vs. framework story very strongly -- > the > > total buy-in and weak integration story of stream processing frameworks > is > > a big downside and makes a library a really compelling (and currently > > unavailable, as far as I am aware) alternative. > > > > Are you suggesting there are still some content missing about the > motivations of adding the proposed library in the wiki page? > > > > * Comment about in-memory storage of other frameworks is interesting -- > it > > is specific to the framework, but is supposed to also give performance > > benefits. The high-level functional processing interface would allow for > > combining multiple operations when there's no shuffle, but when there is > a > > shuffle, we'll always be writing to Kafka, right? Spark (and presumably > > spark streaming) is supposed to get a big win by handling shuffles such > > that the data just stays in cache and never actually hits disk, or at > least > > hits disk in the background. Will we take a hit because we always write > to > > Kafka? > > > > I agree with Neha's comments here. One more point I want to make is > materializing to Kafka is not necessarily much worse than keeping data in > memory if the downstream consumption is caught up such that most of the > reads will be hitting file cache. I remember Samza has illustrated that > under such scenarios its throughput is actually quite comparable to Spark > Streaming / Storm. > > > > * I really struggled with the structure of the KIP template with Copycat > > because the flow doesn't work well for proposals like this. They aren't > as > > concrete changes as the KIP template was designed for. I'd completely > > ignore that template in favor of optimizing for clarity if I were you. > > > > -Ewen > > > > On Thu, Jul 23, 2015 at 5:59 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hi all, > > > > > > I just posted KIP-28: Add a transform client for data processing > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+transform+client+for+data+processing > > > > > > > . > > > > > > The wiki page does not yet have the full design / implementation > details, > > > and this email is to kick-off the conversation on whether we should add > > > this new client with the described motivations, and if yes what > features > > / > > > functionalities should be included. > > > > > > Looking forward to your feedback! > > > > > > -- Guozhang > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > -- Guozhang > -- Thanks, Ewen
