Hi PoAn, Thank you, I created the Jira and updated the link in the KIP.
Best, Ivan On Wed, Apr 23, 2025, at 10:57, PoAn Yang wrote: > Hi Ivan, > > Thanks for the KIP. The Jira link for this KIP is KAFKA-1. > Could you update to a correct Jira number? > > Best, > PoAn > > > On Mar 30, 2025, at 2:13 AM, Ivan Yurchenko <i...@ivanyu.me> wrote: > > > > Hi Federico, > > > > There's a sentence about reduced latency in the Motivation section. Do you > > think that'd be enough? > > > > Best, > > Ivan > > > > > > On Tue, Mar 25, 2025, at 09:23, Federico Valeri wrote: > >> Hi, thanks for the KIP and sorry for the late reply. Should we also > >> highlight the reduced latency in the motivation section? > >> > >> > >> On Mon, Mar 10, 2025 at 9:24 PM Ivan Yurchenko <i...@ivanyu.me> wrote: > >>> > >>> Hi all, > >>> > >>> I took the pause with this KIP while Kafka 4.0 was in making to not > >>> distract the folks. Now let's continue the discussion! > >>> > >>> Thank you for the comments, Luke! I've applied your suggestions. > >>> > >>> Best, > >>> Ivan > >>> > >>> On Mon, Dec 23, 2024, at 03:23, Luke Chen wrote: > >>>> Hi Ivan, > >>>> > >>>> Thanks for the KIP! > >>>> This is a great improvement from the cost and latency perspective! > >>>> > >>>> Some comments: > >>>> 1. In the description of `partitioner.rack.aware` config, it'd be better > >>>> to > >>>> make it clear that this setting has no effect if a custom partitioner is > >>>> used. > >>>> > >>>> 2. "Select the next partition from all partitions following the current > >>>> algorithm in the following cases:" > >>>> I think there should be one more case that "If the > >>>> "partitioner.rack.aware" > >>>> is false; > >>>> > >>>> 3. "If the automatic partitioning is needed (i.e. no record partition or > >>>> key is specified):" > >>>> I think we should also add the case: "key is provided but > >>>> `partitioner.ignore.keys` > >>>> is enabled" > >>>> > >>>> Thank you. > >>>> Luke > >>>> > >>>> > >>>> On Sat, Dec 21, 2024 at 2:32 AM Stanislav Kozlovski < > >>>> stanislavkozlov...@apache.org> wrote: > >>>> > >>>>> Wow, I am super happy to see this KIP! Thanks for publishing it! > >>>>> > >>>>> I threw the idea out there last week in an article of mine about > >>>>> calculating Kafka costs[1] > >>>>> > >>>>>> [FUTURE KIP] - a Produce to Local Leader KIP, similar to KIP-392, can > >>>>>> be > >>>>> introduced to eliminate producer inter-AZ network costs for topics that > >>>>> do > >>>>> not have keys. > >>>>>> there is no fundamental reason that a topic without ordering guarantees > >>>>> needs to produce to a specific partition - why not just choose the > >>>>> broker > >>>>> in the closest zone? > >>>>>> if all of your traffic is unkeyed, then this can further reduce Kafka’s > >>>>> network cost by 25%. > >>>>>> it sounds like a change that wouldn’t be too complicated, maybe even > >>>>> achievable today through the Producer’s partitioner. > >>>>> > >>>>> I don't know if you saw it from there, but I'm super happy to see it > >>>>> come > >>>>> to fruition! It's even easier than I thought - I didn't realize we had > >>>>> the > >>>>> node/rack information in the partitioner already. > >>>>> > >>>>> I think it will be very impactful. > >>>>> We've seen the strong trend in the industry of trading off latency for > >>>>> cost reduction. Namely - almost every vendor has introduced some sort of > >>>>> leaderless Kafka API model that outsources replication to a remote store > >>>>> cost[2][3][4][5]. This in turn allows them to reduce cross-zone > >>>>> networking > >>>>> costs to literally zero. In certain optimized deployments the networking > >>>>> cost can be up to 80-90% of the total cost![6] KIP-392 allows us to > >>>>> eliminate the consumer-side traffic cost, but there is great motivation > >>>>> to > >>>>> enable users to do the same for producers that don't depend on ordering. > >>>>> > >>>>> I am +1 the KIP as is. > >>>>> > >>>>> One may make an argument to have a way to enable it server-side via the > >>>>> broker, but I'd like to hear a good reason for that. I believe the > >>>>> simplicity in the current state is preferred, since clients already have > >>>>> freedom to produce to any partition they explicitly choose. > >>>>> > >>>>> Best, > >>>>> Stan > >>>>> > >>>>> [1] > >>>>> https://bigdata.2minutestreaming.com/p/the-brutal-truth-about-apache-kafka-cost-calculators > >>>>> [2] WarpStream and its $220m acquisition > >>>>> https://www.linkedin.com/pulse/how-confluent-acquired-warpstream-220m-after-just-13-months-hxgyf/ > >>>>> [3] Confluent Freight > >>>>> https://www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/ > >>>>> [4] RedPanda Cloud Topics > >>>>> https://www.redpanda.com/blog/cloud-topics-streaming-data-object-storage > >>>>> [5] BufStream https://buf.build/product/bufstream > >>>>> [6] calculator https://akalculator.com/ > >>>>> > >>>>> On 2024/12/20 11:35:28 Ivan Yurchenko wrote: > >>>>>> Hello all, > >>>>>> > >>>>>> I'd like to propose a new KIP to discuss: KIP-1123: Rack-aware > >>>>> partitioning for Kafka Producer [1]. > >>>>>> > >>>>>> Best, > >>>>>> Ivan Yurchenko > >>>>>> > >>>>>> [1] > >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123%3A+Rack-aware+partitioning+for+Kafka+Producer > >>>>>> > >>>>> > >>>> > >> > >
