[jira] [Created] (KAFKA-13707) Refactor - unify the naming convention of in-flight
Arvin Zheng created KAFKA-13707: --- Summary: Refactor - unify the naming convention of in-flight Key: KAFKA-13707 URL: https://issues.apache.org/jira/browse/KAFKA-13707 Project: Kafka Issue Type: Improvement Components: clients Reporter: Arvin Zheng Methods and variables that have in-flight in them are named in different styles, some treats in-flight as a single word while some doesn't, e.g. inFlight vs inflight or InFlight vs Inflight. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] KIP-637 Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events
Thanks for the comment. Yes, it's indeed a downside that this will increase the size of the metadata response, I was thinking if it's worth providing this information to the Producer conditionally, e.g. add a config to Producer to allow people choose whether to include this information in the MetadataResponse or not. What do you think? Br, Arvin Colin McCabe 于2020年7月7日周二 上午7:40写道： > Hi Arvin, > > Thanks for the KIP. > > Unfortunately, I don't think this makes sense since it would increase the > amount of data we send back in the metadata response, which is pretty bad > for scalability. In general we probably want to avoid the case where we > don't have the appropriate number of in-sync replicas, not try to optimize > for it. > > best, > Colin > > On Mon, Jul 6, 2020, at 10:38, Arvin Zheng wrote: > > Hi everyone, > > > > I would like to start the discussion for KIP-637 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-637%3A+Include+min.insync.replicas+in+MetadataResponse+to+make+Producer+smarter+in+partitioning+events > > > > Looking forward to your feedback. > > > > Thanks, > > Arvin > > >
[DISCUSS] KIP-637 Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events
Updated the subject to add KIP number Arvin Zheng 于2020年7月6日周一 上午10:38写道： > Hi everyone, > > I would like to start the discussion for KIP-637 > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-637%3A+Include+min.insync.replicas+in+MetadataResponse+to+make+Producer+smarter+in+partitioning+events > > Looking forward to your feedback. > > Thanks, > Arvin >
Re: Potential Improvement for Kafka Producer
Thanks to Matthias for the response, I've created a KIP and started another email thread to discuss this, check following. [DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events Matthias J. Sax 于2020年7月6日周一 下午12:30写道： > Arvin, > > thanks for your email. This is definitely the right channel. I am > personally not familiar enough with the producer code, but what you say > makes sense to me from a high level. > > Maybe it would be best if you would file a Jira to improve the producer > accordingly? I guess, this change would require a KIP. > > Of course, if you are interested, feel free to pick it up yourself. > > > -Matthias > > > On 6/28/20 8:53 AM, Arvin Zheng wrote: > > Hi All, > > > > Not sure if this is the right channel and thread to ask, but would like > to > > discuss a potential improvement to Java Kafka Producer. > > > > ``` > > Currently the Kafka Producer is able to identify unavailable partitions > and > > avoid routing messages to them, but the definition of an unavailable > > partitions is - the leader of the partition is not available. > > From Producer point of view, acks for sending messages can be [all, -1, > 0, > > 1] > > 1. When acks is set to either 0 or 1, leader availability is good enough > to > > determine whether we should route messages to that partition. > > 2. When acks is set to -1 or all, leader available doesn't mean we are > able > > to persist messages to that partition successfully, instead, we need to > > make sure > > a. leader is available. > > b. at least min.insync.replicas number of replicas are available > > ``` > > > > To achieve 2, what we need is to carry min.insync.replicas information > of a > > topic to the metadata response, so that Producer is able to determine if > it > > should route messages to that partition when there's no enough replicas > > available and it's acks is set to -1 or all. > > > > Advantages that I can think of > > 1. Avoid exhausting the entire Producer cache when a partition is not > > available for a long time and > > a. retries is set to a large value > > b. acks is set to all > > 2. Avoid unnecessary network tries > > > > Not sure if this is a valid case but would like to hear any opinions. > > > > Br, > > Arvin > > > >
[DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events
Hi everyone, I would like to start the discussion for KIP-637 https://cwiki.apache.org/confluence/display/KAFKA/KIP-637%3A+Include+min.insync.replicas+in+MetadataResponse+to+make+Producer+smarter+in+partitioning+events Looking forward to your feedback. Thanks, Arvin
[jira] [Created] (KAFKA-10230) Include min.insync.replicas in MetadataResponse to make Producer smarter in determining unavailable partitions
Arvin Zheng created KAFKA-10230: --- Summary: Include min.insync.replicas in MetadataResponse to make Producer smarter in determining unavailable partitions Key: KAFKA-10230 URL: https://issues.apache.org/jira/browse/KAFKA-10230 Project: Kafka Issue Type: Improvement Components: clients, core Reporter: Arvin Zheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
Permission to create KIPs
Hi, Could someone please help grant permission to create KIPs to user arvin.zheng? Thanks, Arvin
Potential Improvement for Kafka Producer
Hi All, Not sure if this is the right channel and thread to ask, but would like to discuss a potential improvement to Java Kafka Producer. ``` Currently the Kafka Producer is able to identify unavailable partitions and avoid routing messages to them, but the definition of an unavailable partitions is - the leader of the partition is not available. >From Producer point of view, acks for sending messages can be [all, -1, 0, 1] 1. When acks is set to either 0 or 1, leader availability is good enough to determine whether we should route messages to that partition. 2. When acks is set to -1 or all, leader available doesn't mean we are able to persist messages to that partition successfully, instead, we need to make sure a. leader is available. b. at least min.insync.replicas number of replicas are available ``` To achieve 2, what we need is to carry min.insync.replicas information of a topic to the metadata response, so that Producer is able to determine if it should route messages to that partition when there's no enough replicas available and it's acks is set to -1 or all. Advantages that I can think of 1. Avoid exhausting the entire Producer cache when a partition is not available for a long time and a. retries is set to a large value b. acks is set to all 2. Avoid unnecessary network tries Not sure if this is a valid case but would like to hear any opinions. Br, Arvin
