Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #40

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #13

See

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

1) Once the short-circuit is triggered, the member will downgrade to the EAGER protocol, but won't necessarily try to rejoin the group right away. In the "happy path", the user has implemented #onPartitionsLost correctly and will not attempt to commit partitions that are lost. And since these

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #12

See

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Thanks for the KIP, Sophie. I think this is a great approach, and it makes perfect sense in the 3.0 release as well. -John On Wed, Apr 14, 2021, at 01:17, Bruno Cadonna wrote: > Hi Sophie, > > the KIP makes sense to me! Thank you for writing it! > > Best, > Bruno > > On 14.04.21 04:52,

[jira] [Resolved] (KAFKA-12664) Mirrormaker 2.0 infinite rebalance loop when dealing with more than 2 clusters in standalone mode

[ https://issues.apache.org/jira/browse/KAFKA-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Urban resolved KAFKA-12664. -- Resolution: Not A Bug > Mirrormaker 2.0 infinite rebalance loop when dealing with more than 2

[VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

Hi all, I would like to start a vote on KIP-718, which proposes to make KTable join on foreign key unopinionated in terms of persistence method (currently it forces RocksDB usage for subscription store and no other option is available).

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

Hi Cyrus, Thanks for the KIP. A few questions: 1. What status code does the /plugins/{plugin-type} endpoint return when an unknown type is given? 2. The result of /connector-plugins is a list of objects with 'class', 'type' and 'version' properties. Presumably /plugins/connector is the same, but

Re: [VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

I’m +1 (binding) Thanks, Marco! -John On Wed, Apr 14, 2021, at 07:59, Marco Aurélio Lotz wrote: > Hi all, > > I would like to start a vote on KIP-718, which proposes to make KTable join > on foreign key unopinionated in terms of persistence method (currently it > forces RocksDB usage for

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Thanks Sophie. This makes sense to me. One question: do we want to be a bit clearer about the removal plans? That is, can we say that the deprecated configs will be removed in 4.0 (instead of likely to be removed)? The implication would be that exactly-once would only work with 2.5+ while at least

[jira] [Created] (KAFKA-12668) MockScheduler is not safe to use in concurrent code.

Jose Armando Garcia Sancio created KAFKA-12668: -- Summary: MockScheduler is not safe to use in concurrent code. Key: KAFKA-12668 URL: https://issues.apache.org/jira/browse/KAFKA-12668

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

Hi Matthias, Thanks for reviewing it. That's a nice question. I can provide you the answer as a user of the API - but of course it very much depends on what the original intention towards the user is. The only reason I found out it was opinionated was when my tests using in-memory materialization

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Hi Guozhang, That was my original suggestion, so I am naturally +1 :) Ismael On Wed, Apr 14, 2021 at 11:44 AM Guozhang Wang wrote: > Hi David, > > Just putting my paranoid hat here :) Could we name the req/resp name as > "AllocateProducerIds" instead of "AllocateProducerId"? Otherwise, LGTM!

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hey Gary, thanks for the suggestion. My impression is that the tech debt burden of maintaining the eos-alpha style code in the Producer is significantly less than what we have to do in Streams, since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2 in which only a single

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

I don't think we should deprecate the producer method, because there are also "producer only" application that may use this method. For example, source connectors. -Matthias On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote: > Hey Gary, thanks for the suggestion. My impression is that the tech

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #39

See

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

There is also all that downgrade stuff in the TxnOffsetCommitRequest​ that will downgrade to "alpha" if the broker is < 2.5 (if ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT​ is set). If the intention is to eventually pull support for alpha/v1 altogether then why not remove all that too?

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Ah. My bad. For a "producer only" app, you won't call either of both... As you don't have any associated consumer, you won't need to add offsets to a TX to begin with. Calling `sendOffsetsToTransaction` to optional after all. I withdraw my objection :) -Matthias On 4/14/21 2:22 PM, Sophie

[jira] [Resolved] (KAFKA-12667) Incorrect error log on StateDirectory close

[ https://issues.apache.org/jira/browse/KAFKA-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-12667. Resolution: Fixed > Incorrect error log on StateDirectory close >

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hey John, Yes, we should definitely do that. It's meant to be the standard procedure for configs that are deprecated. Ismael On Wed, Apr 14, 2021 at 12:02 PM John Roesler wrote: > Hello again, Sophie, > > I've just had a late-breaking thought about your KIP. It > might be worthwhile also

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Thanks all. I've updated the KIP to clarify these will be removed in 4.0, and added a note about logging a warning as well. I agree that some users may set the config value directly and could miss the deprecation notice. I'll call for a vote today since there seems to be consensus, but please let

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Sophie, Why not also deprecate Producer.sendOffsetsToTransaction​ method variant with the consumerGroupId​ parameter? This would encourage non-Streams EOS users to move to beta/v2 implementations, and the fallback to alpha/v1 mode code can also be removed in 4.0.

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Well, the method being deprecated has a consumerGroupId parameter... if it's a "producer only" application then do they just set this as null, or empty string? Btw: this method actually just invokes the other overload, which accepts a ConsumerGroupMetadata instead of a String consumerGroupId. So

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

If you argue this way, would the consequence not be that we would need to add even more overloads that allow to only use Materialized for the subscription store? -> I only want to switch to in-memory for the subscription store: Why do I need to set Materialized for the main store? Btw: the above

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Hi David, Just putting my paranoid hat here :) Could we name the req/resp name as "AllocateProducerIds" instead of "AllocateProducerId"? Otherwise, LGTM! Guozhang On Thu, Apr 8, 2021 at 2:23 PM Ron Dagostino wrote: > Hi David. I'm wondering if it might be a good idea to have the broker >

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

I just re-read the KIP, and I am wondering why we would *only add* new methods. Is it an expected use case to only change the main stores but not the subscription stores? Wondering if we should deprecate the existing methods that only accept a single `Materialized` parameter? -Matthias On

Re: [kafka-clients] Re: Subject: [VOTE] 2.8.0 RC1

Thanks for the feedback, all. I am now closing this vote thread in favor of the vote on 2.8.0 RC2. Thanks, John On Mon, 2021-04-12 at 21:24 -0400, Israel Ekpo wrote: > No problem, I will assign in to you shortly. > > https://issues.apache.org/jira/browse/KAFKA-12658 > > On Mon, Apr 12, 2021

Re: [VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

Thanks Marco, +1 from me too. Guozhang On Wed, Apr 14, 2021 at 8:15 AM John Roesler wrote: > I’m +1 (binding) > > Thanks, Marco! > -John > > On Wed, Apr 14, 2021, at 07:59, Marco Aurélio Lotz wrote: > > Hi all, > > > > I would like to start a vote on KIP-718, which proposes to make KTable >

[jira] [Created] (KAFKA-12669) Add deleteRange to WindowStore / KeyValueStore interfaces

Guozhang Wang created KAFKA-12669: - Summary: Add deleteRange to WindowStore / KeyValueStore interfaces Key: KAFKA-12669 URL: https://issues.apache.org/jira/browse/KAFKA-12669 Project: Kafka

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hello again, Sophie, I've just had a late-breaking thought about your KIP. It might be worthwhile also printing a WARN log when the deprecated configs are used. People may not always be referencing the StreamsConfig fields in their source code when they set these configs, so it would be good to

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hi Sophie, Thanks for the KIP, it makes sense to me. I'm a +1 on the proposal. I'm also a +1 to Ismael's suggestion for stating the removal of the deprecated configs in 4.0 -Bill On Wed, Apr 14, 2021 at 3:02 PM John Roesler wrote: > Hello again, Sophie, > > I've just had a late-breaking

[VOTE] 2.8.0 RC2

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 2.8.0.This is a major release that includes many new features, including: * Early-access release of replacing Zookeeper with a self-managed quorum * Add Describe Cluster API *

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Thanks Sophie for writing the KIP! I'm +1 on the proposal. On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma wrote: > Thanks Sophie. This makes sense to me. One question: do we want to be a bit > clearer about the removal plans? That is, can we say that the deprecated > configs will be removed in 4.0

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

Hello Sophie, Thanks for the detailed explanation, a few clarifying questions: 1) when the short-circuit is triggered, what would happen next? Would the consumers switch back to EAGER, and try to re-join the group, and then upon succeeding the next rebalance reset the flag to allow committing?

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

Thanks Matthias. I'm +1 (binding) -Sophie On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax wrote: > Hi, > > Because this KIP is rather small, I would like to skip a dedicated > discussion thread and call for a vote right way. If there are any > concerns, we can just discuss on this vote thread:

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Thanks all, I added a note to the KIP about deprecating the old Producer#sendOffsetsToTransaction method. I think this is sufficient for now, and keeps this KIP well scoped. If we have reason to do a more aggressive cleanup that doesn't violate any compatibility policy, someone can pick that up in

Re: [VOTE] 2.6.2 RC1

I verified the signatures and checksums, ran the tests, and spot-checked the license. I'm +1 (binding). Thanks for driving this, Sophie! -John On Thu, 2021-04-08 at 16:19 -0700, Guozhang Wang wrote: > Looked over the javadocs and web docs again. Download the jars and check > the license files

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

Thanks for the KIP Matthias. +1 (binding) -Bill On Wed, Apr 14, 2021 at 7:06 PM Sophie Blee-Goldman wrote: > Thanks Matthias. I'm +1 (binding) > > -Sophie > > On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax wrote: > > > Hi, > > > > Because this KIP is rather small, I would like to skip a

Re: [VOTE] 2.7.1 RC2

Hi Mickael, I verified the signatures and checksums, ran the tests, and spot-checked the license. I'm +1 (binding). Thanks for driving this release! -John On Fri, 2021-04-09 at 11:07 +0100, Mickael Maison wrote: > Hi, > > Here is a successful build of 2.7.1 RC2: >

[jira] [Created] (KAFKA-12670) KRaft support for unclean.leader.election.enable

Colin McCabe created KAFKA-12670: Summary: KRaft support for unclean.leader.election.enable Key: KAFKA-12670 URL: https://issues.apache.org/jira/browse/KAFKA-12670 Project: Kafka Issue Type:

[VOTE] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hey all, I'd like to kick off the vote on KIP-732, to deprecate eos-alpha in Kafka Streams and migrate away from the "eos-beta" terminology by replacing it with "eos-v2" to shore up user confidence in this feature.

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Kafka protocol support has a different compatibility policy. For example, we still support 0.8.x protocol versions to this day. Ismael On Wed, Apr 14, 2021 at 2:43 PM Gary Russell wrote: > There is also all that downgrade stuff in the TxnOffsetCommitRequest​ that > will downgrade to "alpha" if

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #11

See

[VOTE} KIP-733: change Kafka Streams default replication factor config

Hi, Because this KIP is rather small, I would like to skip a dedicated discussion thread and call for a vote right way. If there are any concerns, we can just discuss on this vote thread:

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

Hi Sophie, the KIP makes sense to me! Thank you for writing it! Best, Bruno On 14.04.21 04:52, Sophie Blee-Goldman wrote: Hey all, I'd like to kick off discussion on a small KIP to move towards a unified EOS and clean up the current options. Please give it a pass and let me know what you

[jira] [Resolved] (KAFKA-12598) Remove deprecated --zookeeper in ConfigCommand

[ https://issues.apache.org/jira/browse/KAFKA-12598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-12598. --- Fix Version/s: (was: 3.0.0) Resolution: Won't Fix > Remove deprecated --zookeeper in

[jira] [Created] (KAFKA-12667) Incorrect error log on StateDirectory close

Jan Justesen created KAFKA-12667: Summary: Incorrect error log on StateDirectory close Key: KAFKA-12667 URL: https://issues.apache.org/jira/browse/KAFKA-12667 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

Hey all, To revive this old KIP, Sarwar Bhuiyan has volunteered to take over ownership. He will continue to drive this KIP to approval and completion - I understand that he will re-start the discussion with a new [DISCUSS] or [VOTE] thread. Thank you Sarwar! Best, Stanislav On Fri, Jan 10,

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

Forgive me while I find my feet, but since this reached a VOTE stage, is it really required to go through a DISCUSS thread again or can we continue to address the issues on this thread since the issues were raised on this thread? If I do have to create a new DISCUSS thread, do I just copy the