There is also another approach (harder to design, but may be easier to use and maintain), which is to make Kafka handle large messages better and allow users to set higher limits - for example, can Kafka provide really high throughput with 1GB messages? Some systems do it well.
I don't know where the slowdowns happen, but perhaps it is one of these? 1. Java GC used to be a problem, but maybe we didn't try with newer GC and simple tuning will solve it? 2. We have head-of-line blocking issue on the queue. There are approaches to solve that too. I'd love to see more exploration on what exactly is the problem we are facing (and is it still an issue? Becket's work is a few years old now.) On Thu, Sep 10, 2020 at 12:21 AM Alexander Sibiryakov < sibirya...@scrapinghub.com> wrote: > Hey Ben, thanks for the link. My proposal is partially based on Becket's > ideas, but I haven't reached out to him directly. > > +Becket > Hi Becket, would you mind to have a look at my proposal (link is in the > first message) ? > > A. > > On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford <b...@confluent.io> wrote: > > > LinkedIn had something like this. Becket did a talk on it a few years > ago. > > It would be interesting to know what became of it and if there were > lessons > > learned. > > https://www.youtube.com/watch?v=ZrbaXDYUZY8 > > > > On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov < > > sibirya...@scrapinghub.com> wrote: > > > > > Hello, > > > > > > I would like to get your opinions on this KIP idea. > > > > > > In short it will allow to transfer messages of bigger size than allowed > > by > > > the broker. > > > > > > > > > > > > https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing > > > > > > If all that makes sense, I'll create a full fledged KIP document and > > expand > > > the idea. > > > > > > Thanks, > > > A. > > > > > > > > > -- > > > > Ben Stopford > > > -- Gwen Shapira Engineering Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
