Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Thanks Stanislav for the update! One suggestion I have is that it would be helpful to put your reasoning on deciding the current default value. For example, in certain use cases at Pinterest we are very likely to have more consumers than 250 when we configure 8 stream instances with 32

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey everybody, It's been a week since this KIP and not much discussion has been made. I assume that this is a straight forward change and I will open a voting thread in the next couple of days if nobody has anything to suggest. Best, Stanislav On Thu, Nov 22, 2018 at 12:56 PM Stanislav

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Another wrinkle to consider is KIP-320. If you are planning to replicate __consumer_offsets directly, then you will have to account for leader epoch information which is stored with the committed offsets. But I cannot think how it would be possible to replicate the leader epoch information in

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Hey Guozhang, Thanks for the comments. Responses below: 0. The new API is used between brokers, so we govern its usage using `inter.broker.protocol.version`. If the other broker hasn't upgraded, we will just fallback to the old logic, which is to accept the write. This is similar to how we

Build failed in Jenkins: kafka-trunk-jdk8 #3219

See Changes: [cmccabe] Trogdor: Add Task State filter to /coordinator/tasks endpoint (#5907) -- [...truncated 2.38 MB...]

[DISCUSS] KIP-394: Require member.id for initial join group request

Hey friends, I would like to start a discussion thread for KIP-394 which is trying to mitigate broker cache bursting issue due to anonymous join group requests: https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request Thanks! Boyang

Jenkins build is back to normal : kafka-trunk-jdk11 #116

See

Re: [VOTE] - KIP-213 Support non-key joining in KTable

On 07.11.2018 22:24, Adam Bellemare wrote: > Bumping this thread, as per convention - 1 > > On Fri, Nov 2, 2018 at 8:22 AM Adam Bellemare > wrote: > >> As expected :) But still, thanks none-the-less! >> >> On Fri, Nov 2, 2018 at 3:36 AM Jan Filipiak >> wrote: >> >>> reminder >>> >>> On

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Thanks Mayuresh and Jason for your follow-ups! Let me try to answer both in this reply. >1. Do you intend to have member.id is a static config like member.name >after KIP-345 and KIP-394? No, we shall only rely on broker to allocate member.id for the consumer instances. FYI, I already

[jira] [Created] (KAFKA-7673) Upgrade RocksDB to include fix for WinEnvIO::GetSectorSize

Yanick Salzmann created KAFKA-7673: -- Summary: Upgrade RocksDB to include fix for WinEnvIO::GetSectorSize Key: KAFKA-7673 URL: https://issues.apache.org/jira/browse/KAFKA-7673 Project: Kafka

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

a few questions: 1. how do you handle possible duplications caused by the "special" producer timing-out/retrying? are you explicitely relying on the "exactly once" sequencing? 2. what about the combination of log compacted topics + replicator downtime? by the time the replicator comes back up

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Do you have a discuss thread for KIP-394 that you mentioned here ? Thanks, Mayuresh On Mon, Nov 26, 2018 at 4:52 AM Boyang Chen wrote: > Hey Dong, thanks for the follow-up here! > > > 1) It is not very clear to the user what is the difference between > member.name and client.id as

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

Hi All, I'm bumping this thread as it has been a couple weeks with no activity. The proposed changes in this KIP are minor, but are extremely helpful for operators to immediately identify partitions under min ISR. Please take a couple minutes to review and provide a vote. Thanks~ Regards,

[jira] [Created] (KAFKA-7674) Quick Start Guide steps for managing topics do not work

Nathan Long created KAFKA-7674: -- Summary: Quick Start Guide steps for managing topics do not work Key: KAFKA-7674 URL: https://issues.apache.org/jira/browse/KAFKA-7674 Project: Kafka Issue

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hi Stanislav, Thanks for the KIP. Can you clarify the compatibility impact here? What will happen to groups that are already larger than the max size? Also, just to be clear, the resource we are trying to conserve here is what? Memory? -Jason On Mon, Nov 26, 2018 at 2:44 AM Boyang Chen wrote:

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Dong, thanks for the follow-up here! 1) It is not very clear to the user what is the difference between member.name and client.id as both seems to be used to identify the consumer. I am wondering if it would be more intuitive to name it group.member.name (preferred choice since it matches

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

+1 (non-binding) Thanks for the KIP! On Mon, Nov 26, 2018 at 4:32 PM Kevin Lu wrote: > > Hi All, > > I'm bumping this thread as it has been a couple weeks with no activity. > > The proposed changes in this KIP are minor, but are extremely helpful for > operators to immediately identify partitions

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks a lot for replying to all the queries and discussions here, so patiently. Really appreciate it. Had a few questions and suggestions after rereading the current version of the KIP : 1. Do you intend to have member.id is a static config like member.name after KIP-345 and

[jira] [Created] (KAFKA-7675) Trogdor CLI - Support filter arguments in the '--show-tasks' command

Stanislav Kozlovski created KAFKA-7675: -- Summary: Trogdor CLI - Support filter arguments in the '--show-tasks' command Key: KAFKA-7675 URL: https://issues.apache.org/jira/browse/KAFKA-7675

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Edoardo, Thanks a lot for the KIP. I have a few questions/suggestions in addition to what Radai has mentioned above : 1. Is this meant only for 1:1 replication, for example one Kafka cluster replicating to other, instead of having multiple Kafka clusters mirroring into one Kafka

[jira] [Created] (KAFKA-7676) Kafka compile failed with Gradle 5.0

Jiangtao Liu created KAFKA-7676: --- Summary: Kafka compile failed with Gradle 5.0 Key: KAFKA-7676 URL: https://issues.apache.org/jira/browse/KAFKA-7676 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-354 Time-based log compaction policy

Thanks for binding and non-binding votes. Can I get one more binding vote? Thanks in advance! Xiongqi (Wesley) Wu On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer wrote: > I'm a +1 (non-binding) — This looks like it would have saved us a lot of > pain in an issue we had to debug recently. I can't

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for the updates. Looks like we're headed in the right direction and clearly the interest that this KIP is receiving shows how strong the motivation is! I have a few questions: 1. This may be the same thing that Mayuresh is asking about. I think the suggestion in the KIP is