Hi Kirk, Yes, the pressure in broker comes from the message format down conversion.
Luke Kirk True <k...@kirktrue.pro> 於 2023年5月13日 週六 上午1:30 寫道： > Hi David, > > For my own edification, when you refer to this change possibly putting > "more pressure on the brokers," is that from the downconversion of the > message format, specifically, or something else? > > Thanks, > Kirk > > On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote: > > Hi David, > > > > I know what you mean. > > Let's hear what others' thoughts about it. :) > > > > Luke > > > > On Fri, May 12, 2023 at 4:53 PM David Jacot <dja...@confluent.io.invalid > > > > wrote: > > > > > Thanks, Luke. > > > > > > > But if the producers and consumers all existed in the same > organization, > > > which means upgrading producers/consumers for the org's cost saving, > should > > > be a reasonable motivation. > > > > > > Yeah, that works in this case. However, Kafka is often used as a > service > > > (on premise or in cloud) nowadays and in this case the > producers/consumers > > > versions are completely out of control thus my concern. > > > > > > BR, > > > David > > > > > > On Fri, May 12, 2023 at 10:47 AM Luke Chen <show...@gmail.com> wrote: > > > > > > > Hi David, > > > > > > > > Yes, you're right. I've bumped the version of record batch, and > describe > > > > the down-conversion will happen like what we do for message format > v1 now > > > > when old consumers consuming records. > > > > > > > > > Overall, I wonder if the bandwidth saving is worth this change > given > > > that > > > > it will put more pressure on the brokers. > > > > Actually, I'm not 100% sure. So I'd also like to hear what the > community > > > > thought about it. > > > > But if the producers and consumers all existed in the same > organization, > > > > which means upgrading producers/consumers for the org's cost saving, > > > should > > > > be a reasonable motivation. > > > > > > > > Thanks. > > > > Luke > > > > > > > > > > > > On Fri, May 12, 2023 at 3:43 PM David Jacot > <dja...@confluent.io.invalid > > > > > > > > wrote: > > > > > > > > > Hi Luke, > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > What do we do in the case where a batch is written with > > > > > `ignoreMessageAttributes` set to 1, which means that messages won't > > > have > > > > > the `attributes`, and is consumed by a consumer which does not > > > understand > > > > > this new format? I suppose that we would need to introduce a new > > > version > > > > > for the message format (v3) and that we will have to downconvert > > > records > > > > > from the new format version to v2 in this case. This is not clear > in > > > the > > > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if > the > > > > > bandwidth saving is worth this change given that it will put more > > > > pressure > > > > > on the brokers. > > > > > > > > > > Best, > > > > > David > > > > > > > > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <show...@gmail.com> > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore > unused > > > > > > message attribute field. This KIP is to add a flag in the batch > > > header > > > > to > > > > > > indicate if messages inside the batch have attribute field or > not, to > > > > > > reduce the message size, thus, save network traffic and storage > size > > > > (and > > > > > > money, of course). > > > > > > > > > > > > Please check the link for more detail: > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field > > > > > > > > > > > > Any feedback is welcome. > > > > > > > > > > > > Thank you. > > > > > > Luke > > > > > > > > > > > > > > > > > > > > >
