Hello, I don't have an answer for this. Ideally, the bigger the better. For our use case: working with Web data, hundreds of megabytes should be enough. But there could be other use cases, such as transferring of binary or media files.
There is another dimension to consider: amount of large messages in the stream. In our case it is less than 0.5%. If this number is high enough, Kafka will be out of consideration, I assume. A. On Thu, Sep 10, 2020 at 7:02 PM Ismael Juma <ism...@juma.me.uk> wrote: > Thanks for the KIP. I think the main question is what's the upper bound for > message size you are looking to support. Is it hundreds of MBs, GBs, tens > of GBs, something else? That would inform the options. > > Ismael > > On Thu, Sep 10, 2020 at 8:03 AM Gwen Shapira <g...@confluent.io> wrote: > > > There is another option of doing the splitting on the server and hiding > > this from the clients. My personal (and highly controversial) take is > that > > Kafka clients could use less complexity rather than more. They are > > incredibly difficult to reason about as is. But maybe this > > splitting/merging won't be that bad - multi-part messages are well > > understood in general. > > > > On Thu, Sep 10, 2020 at 7:51 AM Gwen Shapira <g...@confluent.io> wrote: > > > > > There is also another approach (harder to design, but may be easier to > > use > > > and maintain), which is to make Kafka handle large messages better and > > > allow users to set higher limits - for example, can Kafka provide > really > > > high throughput with 1GB messages? Some systems do it well. > > > > > > I don't know where the slowdowns happen, but perhaps it is one of > these? > > > 1. Java GC used to be a problem, but maybe we didn't try with newer GC > > and > > > simple tuning will solve it? > > > 2. We have head-of-line blocking issue on the queue. There are > approaches > > > to solve that too. > > > > > > I'd love to see more exploration on what exactly is the problem we are > > > facing (and is it still an issue? Becket's work is a few years old > now.) > > > > > > On Thu, Sep 10, 2020 at 12:21 AM Alexander Sibiryakov < > > > sibirya...@scrapinghub.com> wrote: > > > > > >> Hey Ben, thanks for the link. My proposal is partially based on > Becket's > > >> ideas, but I haven't reached out to him directly. > > >> > > >> +Becket > > >> Hi Becket, would you mind to have a look at my proposal (link is in > the > > >> first message) ? > > >> > > >> A. > > >> > > >> On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford <b...@confluent.io> > wrote: > > >> > > >> > LinkedIn had something like this. Becket did a talk on it a few > years > > >> ago. > > >> > It would be interesting to know what became of it and if there were > > >> lessons > > >> > learned. > > >> > https://www.youtube.com/watch?v=ZrbaXDYUZY8 > > >> > > > >> > On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov < > > >> > sibirya...@scrapinghub.com> wrote: > > >> > > > >> > > Hello, > > >> > > > > >> > > I would like to get your opinions on this KIP idea. > > >> > > > > >> > > In short it will allow to transfer messages of bigger size than > > >> allowed > > >> > by > > >> > > the broker. > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing > > >> > > > > >> > > If all that makes sense, I'll create a full fledged KIP document > and > > >> > expand > > >> > > the idea. > > >> > > > > >> > > Thanks, > > >> > > A. > > >> > > > > >> > > > >> > > > >> > -- > > >> > > > >> > Ben Stopford > > >> > > > >> > > > > > > > > > -- > > > Gwen Shapira > > > Engineering Manager | Confluent > > > 650.450.2760 | @gwenshap > > > Follow us: Twitter | blog > > > > > > > > > -- > > Gwen Shapira > > Engineering Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > >
