To move this forward, I took the liberty to create a PR to bump the version to 3.0.0-SNAPSHOT
https://github.com/apache/kafka/pull/10186 Please let us know if there are any concerns. -Matthias On 2/16/21 5:18 PM, Ismael Juma wrote: > I'm +1 on 3.0 for the mid year release. > > On Tue, Feb 16, 2021 at 5:08 PM Matthias J. Sax <mj...@apache.org> wrote: > >> Hi, >> >> given that we passed 2.8 feature freeze, I wanted to restart this >> thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if >> the decision for the 3.0 release is final and if we should bump the >> version number? >> >> I am asking particularly because there a many Jiras with a 3.0 target >> release version for breaking changes and we should ensure that we have >> enough time to work on those tickets. -- As long as we don't agree that >> the next release will indeed be 3.0, those tickets are effectively >> blocked/pending. >> >> Thoughts? >> >> >> -Matthias >> >> >> On 10/15/20 4:28 PM, Matthias J. Sax wrote: >>> Thanks for clarifying Colin. Works for me. Overall, 3.0 should be guided >>> by the ZK removal progress and if we are not there yet, it's better to >>> have a 2.8 first. >>> >>> >>> -Matthias >>> >>> >>> On 10/15/20 2:41 PM, Colin McCabe wrote: >>>> Hi all, >>>> >>>> Just to follow up on this... since we're not quite ready for 3.0 yet, >> it's probably best if we release a 2.8 next, and then go to 3.0 after >> that. Sorry for any confusion. >>>> >>>> best, >>>> Colin >>>> >>>> >>>> On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: >>>>> Did we reach any conclusion on the subject? >>>>> >>>>> It seems we are aiming for 2.7 after 2.6 and plan the major version >> bump >>>>> to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?) >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> >>>>> On 5/18/20 1:11 PM, Boyang Chen wrote: >>>>>> One more thing I would like to see deprecated (hopefully no one >> mentioned >>>>>> before) is the zk based consumer offset support. >>>>>> >>>>>> On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> >> wrote: >>>>>> >>>>>>> Hi Michael, >>>>>>> >>>>>>> It would be better to discuss the background behind KIP-500 in a >> separate >>>>>>> thread, since this thread is about the Kafka 3.0 release. As others >> have >>>>>>> said, your questions are answered in the KIP. For example, "what is >> the >>>>>>> actual goal?" is addressed in the motivation section. >>>>>>> >>>>>>> I agree that Kafka's usage of Apache ZooKeeper could be optimized. >> But >>>>>>> there are fundamental limitations to this approach compared to >> storing our >>>>>>> metadata internally. For example, having to contact a remote server >> to >>>>>>> reload all your metadata on a controller failover simply doesn't >> scale past >>>>>>> a certain point. >>>>>>> >>>>>>> Apache Curator is a nice API, and if we were starting again today we >> would >>>>>>> certainly consider using it. But it doesn't allow us to do anything >> more >>>>>>> efficiently than ZooKeeper could already do it. >>>>>>> >>>>>>> Finally, Kafka's core competence is logs. While our replication >> protocol >>>>>>> is not Raft, it shares many similarities with that protocol. So I >> think >>>>>>> it's a bit unfair to say that it is "catastrophic hubris" to believe >> we can >>>>>>> implement the protocol. >>>>>>> >>>>>>> best, >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: >>>>>>>> Yes, I've read the KIP. But all it really says to me is "we have >> never >>>>>>>> gotten around to using ZooKeeper properly." To the extent that any >> of >>>>>>> the >>>>>>>> distributed-state-maintenance problems discussed in "Metadata as an >> Event >>>>>>>> Log" can be solved — and some of them intrinsically can't, because >> CAP >>>>>>>> theorem — most of them are already implemented very effectively in >>>>>>> Curator >>>>>>>> recipes. (For instance, Curator's Tree Cache >>>>>>>> https://curator.apache.org/curator-recipes/tree-cache.html is a >> good >>>>>>> fit to >>>>>>>> some of the state-maintenance needs.) >>>>>>>> >>>>>>>> Kafka does have some usage patterns that don't map neatly onto >> existing >>>>>>>> Curator recipes. For instance, neither LeaderSelector nor >> LeaderLatch >>>>>>>> implements leader preference in the way that the existing Kafka >> partition >>>>>>>> leadership election procedure does. But why not handle that by >> improving >>>>>>>> and extending Curator? That way, other Curator users benefit, and >> we get >>>>>>>> additional highly experienced reviewers' eyes on the distributed >>>>>>>> algorithms, which are very very tricky to get right. >>>>>>>> >>>>>>>> >>>>>>>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Michael. This is discussed in the KIP. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation >>>>>>>>> >>>>>>>>> Ron >>>>>>>>> >>>>>>>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < >>>>>>> m.k.edwa...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> What is the actual goal of removing the ZooKeeper dependency? In >> my >>>>>>>>>> experience, if ZooKeeper is properly provisioned and deployed, >> it's >>>>>>>>> largely >>>>>>>>>> trouble-free. (You do need to know how to use observers >> properly.) >>>>>>>>> There >>>>>>>>>> are some subtleties about timeouts and leadership changes, but >>>>>>> they're >>>>>>>>>> pretty small stuff. Why go to all the trouble of building a new >>>>>>>>>> distributed-consensus system that's going to have catastrophic >> bugs >>>>>>> for >>>>>>>>>> years to come? It seems like such an act of hubris to me, as well >>>>>>> as a >>>>>>>>>> massive waste of engineering effort. What is there to be gained? >>>>>>>>>> >>>>>>>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org >>> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Sure, we can compile a list for Kafka Streams. But the KIP would >> be >>>>>>> for >>>>>>>>>>> 3.0, so I don't think it's urgent to do it now? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -Matthias >>>>>>>>>>> >>>>>>>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: >>>>>>>>>>>> Thanks, Guozhang-- sounds like a good plan. >>>>>>>>>>>> >>>>>>>>>>>> I think it would be good to have a list of deprecated streams >> APIs >>>>>>> that >>>>>>>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its >> own >>>>>>> KIP? >>>>>>>>>>>> >>>>>>>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use in >>>>>>> 2.6 if >>>>>>>>>>> we want to remove it in 3.0. I don't have a good sense of how >>>>>>>>> practical it >>>>>>>>>>> is to deprecate this now, so I will defer to others here. But >> the >>>>>>> KIP >>>>>>>>>>> freeze for 2.6 is coming soon, so if we want to make the case, >> now >>>>>>> is >>>>>>>>> the >>>>>>>>>>> time. >>>>>>>>>>>> >>>>>>>>>>>> best, >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: >>>>>>>>>>>>> Hey folks, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for stating that the bridge release would not break any >>>>>>>>>>> compatibility >>>>>>>>>>>>> before, which is incorrect and confused many people. >>>>>>>>>>>>> >>>>>>>>>>>>> I think one way to think about the versioning is that: >>>>>>>>>>>>> >>>>>>>>>>>>> 0) In a 2.x version moving ahead we would deprecate the >>>>>>> ZK-dependent >>>>>>>>>>> tools >>>>>>>>>>>>> such as --zookeeper flags from various scripts (KIP-555) >>>>>>>>>>>>> >>>>>>>>>>>>> 1) In 3.0 we would at least make one incompatible change for >>>>>>> example >>>>>>>>> to >>>>>>>>>>>>> remove the deprecated ZK flags. >>>>>>>>>>>>> >>>>>>>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK >> entirely, >>>>>>>>>>>>> including usages such as security credentials / broker >>>>>>> registration / >>>>>>>>>>> etc >>>>>>>>>>>>> which are via ZK today as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> For 1), I'd love to add a few more incompatibility changes in >> 3.0 >>>>>>> from >>>>>>>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating and >>>>>>> then >>>>>>>>>>> remove >>>>>>>>>>>>> in major releases, and since 2.0 we've accumulated quite a few >>>>>>>>>>> deprecated >>>>>>>>>>>>> APIs, and I can compile a list of KIPs that contain those if >>>>>>> people >>>>>>>>> are >>>>>>>>>>>>> interested. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Guozhang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe < >> cmcc...@apache.org> >>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: >>>>>>>>>>>>>>>> In fact, we know that the bridge release will involve at >> least >>>>>>> one >>>>>>>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>>>>>>> --zookeeper >>>>>>>>>>>>>>>> flags in the command-line tools. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release >> are >>>>>>>>> _both_ >>>>>>>>>>>>>>> breaking changes, I think we only have one option: the 3.x >> line >>>>>>> are >>>>>>>>>>> the >>>>>>>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by >>>>>>> Andrew >>>>>>>>>>>>>>> Schofield. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Specifically: >>>>>>>>>>>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper >>>>>>> args, >>>>>>>>> we >>>>>>>>>>>>>> will >>>>>>>>>>>>>>> need a major release. >>>>>>>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a >> bunch of >>>>>>>>>>> external >>>>>>>>>>>>>>> tooling like Cruise Control), we will need a major release. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I count two major releases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Ryanne, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I agree that dropping ZK completely will need a new major >> release >>>>>>>>> after >>>>>>>>>>>>>> 3.0. I think that's OK and in keeping with how we've handled >>>>>>>>>>> deprecation >>>>>>>>>>>>>> and removal in the past. It's important for users to have a >>>>>>> smooth >>>>>>>>>>> upgrade >>>>>>>>>>>>>> path. >>>>>>>>>>>>>> >>>>>>>>>>>>>> best, >>>>>>>>>>>>>> Colin >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ryanne >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < >>>>>>> cmcc...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: >>>>>>>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's >>>>>>> "bridge >>>>>>>>>>>>>>>>> release" so there is a clean break from Zookeeper after >> 3.0. >>>>>>> The >>>>>>>>>>>>>>>>> bridge release by definition is an attempt to not break >>>>>>> anything, >>>>>>>>> so >>>>>>>>>>>>>>>>> it theoretically doesn't warrant a major release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ryanne, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think it's important to clarify this a little bit. The >>>>>>> bridge >>>>>>>>>>>>>> release >>>>>>>>>>>>>>>> (really, releases, plural) allow you to upgrade from a >> cluster >>>>>>> that >>>>>>>>>>> is >>>>>>>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, >> that >>>>>>>>>>> doesn't >>>>>>>>>>>>>>>> imply that the bridge release itself doesn't break anything. >>>>>>>>>>> Upgrading >>>>>>>>>>>>>>>> to the bridge release itself might involve some minor >>>>>>>>>>> incompatibility. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Kafka does occasionally have incompatible changes. In those >>>>>>> cases, >>>>>>>>>>> we >>>>>>>>>>>>>>>> bump the major version number. One example is that when we >>>>>>> went >>>>>>>>> from >>>>>>>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This >> is >>>>>>> an >>>>>>>>>>>>>>>> incompatible change. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In fact, we know that the bridge release will involve at >> least >>>>>>> one >>>>>>>>>>>>>>>> incompatible change. We will need to drop support for the >>>>>>>>>>> --zookeeper >>>>>>>>>>>>>>>> flags in the command-line tools. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We've been preparing for this change for a long time. >> People >>>>>>> have >>>>>>>>>>>>>> spent >>>>>>>>>>>>>>>> a lot of effort designing new APIs that can be used instead >> of >>>>>>> the >>>>>>>>>>> old >>>>>>>>>>>>>>>> zookeeper-based code that some of the command-line tools >>>>>>> used. We >>>>>>>>>>> have >>>>>>>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of >> the >>>>>>> day, >>>>>>>>>>> it >>>>>>>>>>>>>>>> is still an incompatible change. So it's unfortunately not >>>>>>>>> possible >>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> bridge release to be a 2.x release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" >>>>>>> turns >>>>>>>>> out >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> be impractical), we should consider forking 3.0 while >>>>>>> maintaining >>>>>>>>> a >>>>>>>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can >>>>>>> evolve >>>>>>>>>>>>>>>>> dramatically without breaking the 2.x line. In particular, >>>>>>>>> anything >>>>>>>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while >>>>>>> every >>>>>>>>>>> other >>>>>>>>>>>>>>>>> feature targets 2.6. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Just to be super clear about this, what we want to do here >> is >>>>>>>>> support >>>>>>>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a >>>>>>> while. >>>>>>>>> So >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> same branch will have support for both the old way and the >> new >>>>>>> way >>>>>>>>> of >>>>>>>>>>>>>>>> managing metadata. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This will allow us to get an "alpha" version of the KIP-500 >>>>>>> mode >>>>>>>>> out >>>>>>>>>>>>>> early >>>>>>>>>>>>>>>> for people to experiment with. It also greatly reduces the >>>>>>> number >>>>>>>>> of >>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>> releases we have to make, and the amount of backporting we >>>>>>> have to >>>>>>>>>>> do. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", I >>>>>>> think >>>>>>>>>>> this >>>>>>>>>>>>>>>>> is premature given Kafka's time-based release schedule. If >> the >>>>>>>>>>> bridge >>>>>>>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, >> then >>>>>>>>> sure >>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> let's make that the bridge release in retrospect. And if we >>>>>>> get >>>>>>>>> all >>>>>>>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard >>>>>>> with >>>>>>>>>>>>>> naming >>>>>>>>>>>>>>>>> it "3.0" instead. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker >> before >>>>>>> 3.0 >>>>>>>>> as >>>>>>>>>>>>>>>>> well. I'm happy to drive that additional breaking change. >>>>>>> Maybe >>>>>>>>> 2.6 >>>>>>>>>>>>>>>>> can be the "bridge" for MM2 as well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I don't have a strong opinion either way about this, but if >> we >>>>>>> want >>>>>>>>>>> to >>>>>>>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it >> first, >>>>>>>>>>>>>> right? Are >>>>>>>>>>>>>>>> we ready to do that? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> best, >>>>>>>>>>>>>>>> Colin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ryanne >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe < >> cmcc...@apache.org >>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We've had a few proposals recently for incompatible >>>>>>> changes. One >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the >>>>>>>>> Administrative >>>>>>>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK >> Mutation >>>>>>>>>>>>>>>>>> Protocols to the Controller. I think it's time to start >>>>>>> thinking >>>>>>>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to >> 3.0 >>>>>>>>> after >>>>>>>>>>>>>>>>>> the 2.6 release. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to >>>>>>> make >>>>>>>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the >>>>>>>>>>>>>> default.) >>>>>>>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as >> well as >>>>>>>>> some >>>>>>>>>>>>>>>>>> of the other KIPs we've done recently. Since it will take >>>>>>> some >>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will >> hide >>>>>>> it >>>>>>>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing >>>>>>> this all >>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> detail soon.) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up >> next >>>>>>>>> after >>>>>>>>>>>>>>>>>> 2.6? Are there any other things we should change in the >> 2.6 >>>>>>> -> >>>>>>>>> 3.0 >>>>>>>>>>>>>>>>>> transition? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> best, Colin >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> -- Guozhang >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> Attachments: >>>>> * signature.asc >> >
