Re: [DISCUSS] Kafka 3.0

2021-02-22 Thread Guozhang Wang
+1 on getting to 3.0 for the June release this year too. Guozhang On Mon, Feb 22, 2021 at 6:54 PM Matthias J. Sax wrote: > To move this forward, I took the liberty to create a PR to bump the > version to 3.0.0-SNAPSHOT > > https://github.com/apache/kafka/pull/10186 > > Please let us know if

Re: [DISCUSS] Kafka 3.0

2021-02-22 Thread Matthias J. Sax
To move this forward, I took the liberty to create a PR to bump the version to 3.0.0-SNAPSHOT https://github.com/apache/kafka/pull/10186 Please let us know if there are any concerns. -Matthias On 2/16/21 5:18 PM, Ismael Juma wrote: > I'm +1 on 3.0 for the mid year release. > > On Tue, Feb

Re: [DISCUSS] Kafka 3.0

2021-02-16 Thread Ismael Juma
I'm +1 on 3.0 for the mid year release. On Tue, Feb 16, 2021 at 5:08 PM Matthias J. Sax wrote: > Hi, > > given that we passed 2.8 feature freeze, I wanted to restart this > thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if > the decision for the 3.0 release is final and if

Re: [DISCUSS] Kafka 3.0

2021-02-16 Thread Matthias J. Sax
Hi, given that we passed 2.8 feature freeze, I wanted to restart this thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if the decision for the 3.0 release is final and if we should bump the version number? I am asking particularly because there a many Jiras with a 3.0 target

Re: [DISCUSS] Kafka 3.0

2020-10-15 Thread Matthias J. Sax
Thanks for clarifying Colin. Works for me. Overall, 3.0 should be guided by the ZK removal progress and if we are not there yet, it's better to have a 2.8 first. -Matthias On 10/15/20 2:41 PM, Colin McCabe wrote: > Hi all, > > Just to follow up on this... since we're not quite ready for 3.0

Re: [DISCUSS] Kafka 3.0

2020-10-15 Thread Colin McCabe
Hi all, Just to follow up on this... since we're not quite ready for 3.0 yet, it's probably best if we release a 2.8 next, and then go to 3.0 after that. Sorry for any confusion. best, Colin On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: > Did we reach any conclusion on the subject?

Re: [DISCUSS] Kafka 3.0

2020-07-20 Thread Matthias J. Sax
Did we reach any conclusion on the subject? It seems we are aiming for 2.7 after 2.6 and plan the major version bump to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?) -Matthias On 5/18/20 1:11 PM, Boyang Chen wrote: > One more thing I would like to see deprecated

Re: [DISCUSS] Kafka 3.0

2020-05-18 Thread Boyang Chen
One more thing I would like to see deprecated (hopefully no one mentioned before) is the zk based consumer offset support. On Mon, May 11, 2020 at 2:15 PM Colin McCabe wrote: > Hi Michael, > > It would be better to discuss the background behind KIP-500 in a separate > thread, since this thread

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Colin McCabe
Hi Michael, It would be better to discuss the background behind KIP-500 in a separate thread, since this thread is about the Kafka 3.0 release. As others have said, your questions are answered in the KIP. For example, "what is the actual goal?" is addressed in the motivation section. I

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Andrew Otto
> Before we deprecate server side auto topic creation, we should have client side auto topic creation for the producer: Deprecating sounds fine, but before disabling it, it might be worthwhile to wait long enough for non Java clients to catch up to this too. :) On Mon, May 11, 2020 at 4:45 PM

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Ismael Juma
Before we deprecate server side auto topic creation, we should have client side auto topic creation for the producer: https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer Ismael On Mon, May 11, 2020 at 1:41 PM Colin McCabe wrote: > On

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Colin McCabe
On Mon, May 11, 2020, at 01:19, David Jacot wrote: > Hi all, > > First, I agree with what has been discussed. Having 3.x as the bridge > releases and entirely removing ZK in 4.0 makes total sense. > > Second, what would you think about removing the auto topics creation > in 3.0? It is not

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread Guozhang Wang
I'm +1 on deprecating the auto.topic.creation in 2.6 as well. Guozhang On Mon, May 11, 2020 at 1:20 AM David Jacot wrote: > Hi all, > > First, I agree with what has been discussed. Having 3.x as the bridge > releases and entirely removing ZK in 4.0 makes total sense. > > Second, what would

Re: [DISCUSS] Kafka 3.0

2020-05-11 Thread David Jacot
Hi all, First, I agree with what has been discussed. Having 3.x as the bridge releases and entirely removing ZK in 4.0 makes total sense. Second, what would you think about removing the auto topics creation in 3.0? It is not recommended to use it anymore and that could simplify a bit our path

Re: [DISCUSS] Kafka 3.0

2020-05-10 Thread Michael K. Edwards
Yes, I've read the KIP. But all it really says to me is "we have never gotten around to using ZooKeeper properly." To the extent that any of the distributed-state-maintenance problems discussed in "Metadata as an Event Log" can be solved — and some of them intrinsically can't, because CAP

Re: [DISCUSS] Kafka 3.0

2020-05-10 Thread Ron Dagostino
Hi Michael. This is discussed in the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation Ron > On May 10, 2020, at 1:35 PM, Michael K. Edwards wrote: > >

Re: [DISCUSS] Kafka 3.0

2020-05-10 Thread Michael K. Edwards
What is the actual goal of removing the ZooKeeper dependency? In my experience, if ZooKeeper is properly provisioned and deployed, it's largely trouble-free. (You do need to know how to use observers properly.) There are some subtleties about timeouts and leadership changes, but they're pretty

Re: [DISCUSS] Kafka 3.0

2020-05-08 Thread Matthias J. Sax
Sure, we can compile a list for Kafka Streams. But the KIP would be for 3.0, so I don't think it's urgent to do it now? -Matthias On 5/8/20 3:47 PM, Colin McCabe wrote: > Thanks, Guozhang-- sounds like a good plan. > > I think it would be good to have a list of deprecated streams APIs that we

Re: [DISCUSS] Kafka 3.0

2020-05-08 Thread Colin McCabe
Thanks, Guozhang-- sounds like a good plan. I think it would be good to have a list of deprecated streams APIs that we want to remove in 3.0. Maybe it's easiest to do that as its own KIP? For MirrorMaker 1, we should have a KIP to deprecate its use in 2.6 if we want to remove it in 3.0. I

Re: [DISCUSS] Kafka 3.0

2020-05-08 Thread Ryanne Dolan
Colin, Guozhang, Matthias, all makes sense to me, thanks. Ryanne On Fri, May 8, 2020, 1:48 PM Matthias J. Sax wrote: > I agree with what was discussed. Having a (breaking) 3.0 release to > start the "bridge release" series and completing it with a 4.0 release > sounds intuitive to me. > > For

Re: [DISCUSS] Kafka 3.0

2020-05-08 Thread Matthias J. Sax
I agree with what was discussed. Having a (breaking) 3.0 release to start the "bridge release" series and completing it with a 4.0 release sounds intuitive to me. For old MirrorMaker: In general I am also in favor of removing it, but there should be a reasonable deprecation period. Deprecating it

Re: [DISCUSS] Kafka 3.0

2020-05-07 Thread Guozhang Wang
Hey folks, Sorry for stating that the bridge release would not break any compatibility before, which is incorrect and confused many people. I think one way to think about the versioning is that: 0) In a 2.x version moving ahead we would deprecate the ZK-dependent tools such as --zookeeper flags

Re: [DISCUSS] Kafka 3.0

2020-05-07 Thread Colin McCabe
On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > In fact, we know that the bridge release will involve at least one > > incompatible change. We will need to drop support for the --zookeeper > > flags in the command-line tools. > > If the bridge release(s) and the subsequent post-ZK release

Re: [DISCUSS] Kafka 3.0

2020-05-07 Thread Colin McCabe
On Wed, May 6, 2020, at 21:40, Ryanne Dolan wrote: > > This will allow us to get an "alpha" version of the KIP-500 mode out > > early for people to experiment with > > I think this is a non-sequitur. It's not a requirement that KIP-500 be > merged to master and released in order for people to

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Ryanne Dolan
> This will allow us to get an "alpha" version of the KIP-500 mode out early for people to experiment with I think this is a non-sequitur. It's not a requirement that KIP-500 be merged to master and released in order for people to experiment with it. Nor does it sound great to call for a major

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Ryanne Dolan
> In fact, we know that the bridge release will involve at least one > incompatible change. We will need to drop support for the --zookeeper > flags in the command-line tools. If the bridge release(s) and the subsequent post-ZK release are _both_ breaking changes, I think we only have one

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
On Tue, May 5, 2020, at 12:36, Ryanne Dolan wrote: > > In 3.0 it sounds like nothing is breaking and our big change won't be > > complete... so, what's the motivation for the major release? > > Exactly. Why would 3.1 be the breaking release? No one would expect > everything to break going from

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
On Mon, May 4, 2020, at 17:33, Gwen Shapira wrote: > +1 for removing MM 1.0 when we cut a breaking release. It is sad to see > that we are still investing in it (I just saw a KIP toward improving its > reset policy). > > My understanding was that KIP-590 is not breaking compatibility, I think >

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Colin McCabe
On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > Hey Colin, I think we should wait until after KIP-500's "bridge > release" so there is a clean break from Zookeeper after 3.0. The > bridge release by definition is an attempt to not break anything, so > it theoretically doesn't warrant a major

RE: [DISCUSS] Kafka 3.0

2020-05-06 Thread Edoardo Comar
Ryanne Dolan wrote on 05/05/2020 20:36:49: > Exactly. Why would 3.1 be the breaking release? No one would expect > everything to break going from 3.0 to 3.1 Agree completely > > Ryanne > > On Tue, May 5, 2020 at 2:34 PM Gwen Shapira wrote: > > > It sounds like the decision to make the next

Re: [DISCUSS] Kafka 3.0

2020-05-06 Thread Andrew Schofield
That's my view here. I think there are two ways this could work: 1) 2.x is the bridge release, and 3.0 is the one that completes the ZK removal. 2) 3.0 is the bridge release, and 4.0 is the one that completes the ZK removal. On 06/05/2020, 05:15, "Jeff Widman" wrote: IMO a bridge release,

Re: [DISCUSS] Kafka 3.0

2020-05-05 Thread Jeff Widman
IMO a bridge release, or one that you have to upgrade to before upgrading to the breaking release should be numbered as the last of the 2.x series... In other words, it's acceptable to say "before upgrading to 3.0, first upgrade to 2.9" but it's very unexpected to say "before upgrading to 3.1,

Re: [DISCUSS] Kafka 3.0

2020-05-05 Thread Ryanne Dolan
> In 3.0 it sounds like nothing is breaking and our big change won't be > complete... so, what's the motivation for the major release? Exactly. Why would 3.1 be the breaking release? No one would expect everything to break going from 3.0 to 3.1 Ryanne On Tue, May 5, 2020 at 2:34 PM Gwen Shapira

Re: [DISCUSS] Kafka 3.0

2020-05-05 Thread Gwen Shapira
It sounds like the decision to make the next release 3.0 is a bit arbitrary then? With Exactly Once, we announced 1.0 as one release after the one where EOS shipped, when we felt it was "ready" (little did we know... but that's another story). 2.0 was breaking due to us dropping Java 7. In 3.0

Re: [DISCUSS] Kafka 3.0

2020-05-05 Thread Guozhang Wang
I think there's a confusion regarding the "bridge release" proposed in KIP-500: should it be release "3.0" or be release "2.X" (i.e. the last minor release before 3.0). My understanding is that "3.0" would be the bridge release, i.e. it would not break any compatibility, but 3.1 potentially

Re: [DISCUSS] Kafka 3.0

2020-05-04 Thread Gwen Shapira
+1 for removing MM 1.0 when we cut a breaking release. It is sad to see that we are still investing in it (I just saw a KIP toward improving its reset policy). My understanding was that KIP-590 is not breaking compatibility, I think Guozhang said that in response to my question on the discussion

Re: [DISCUSS] Kafka 3.0

2020-05-04 Thread Ryanne Dolan
Hey Colin, I think we should wait until after KIP-500's "bridge release" so there is a clean break from Zookeeper after 3.0. The bridge release by definition is an attempt to not break anything, so it theoretically doesn't warrant a major release. If that's not the case (i.e. if a single "bridge

[DISCUSS] Kafka 3.0

2020-05-04 Thread Colin McCabe
Hi all, We've had a few proposals recently for incompatible changes. One of them is my KIP-604: Remove ZooKeeper Flags from the Administrative Tools. The other is Boyang's KIP-590: Redirect ZK Mutation Protocols to the Controller. I think it's time to start thinking about Kafka 3.0.