+1 on getting to 3.0 for the June release this year too. Guozhang
On Mon, Feb 22, 2021 at 6:54 PM Matthias J. Sax <mj...@apache.org> wrote: > To move this forward, I took the liberty to create a PR to bump the > version to 3.0.0-SNAPSHOT > > https://github.com/apache/kafka/pull/10186 > > Please let us know if there are any concerns. > > > -Matthias > > On 2/16/21 5:18 PM, Ismael Juma wrote: > > I'm +1 on 3.0 for the mid year release. > > > > On Tue, Feb 16, 2021 at 5:08 PM Matthias J. Sax <mj...@apache.org> > wrote: > > > >> Hi, > >> > >> given that we passed 2.8 feature freeze, I wanted to restart this > >> thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if > >> the decision for the 3.0 release is final and if we should bump the > >> version number? > >> > >> I am asking particularly because there a many Jiras with a 3.0 target > >> release version for breaking changes and we should ensure that we have > >> enough time to work on those tickets. -- As long as we don't agree that > >> the next release will indeed be 3.0, those tickets are effectively > >> blocked/pending. > >> > >> Thoughts? > >> > >> > >> -Matthias > >> > >> > >> On 10/15/20 4:28 PM, Matthias J. Sax wrote: > >>> Thanks for clarifying Colin. Works for me. Overall, 3.0 should be > guided > >>> by the ZK removal progress and if we are not there yet, it's better to > >>> have a 2.8 first. > >>> > >>> > >>> -Matthias > >>> > >>> > >>> On 10/15/20 2:41 PM, Colin McCabe wrote: > >>>> Hi all, > >>>> > >>>> Just to follow up on this... since we're not quite ready for 3.0 yet, > >> it's probably best if we release a 2.8 next, and then go to 3.0 after > >> that. Sorry for any confusion. > >>>> > >>>> best, > >>>> Colin > >>>> > >>>> > >>>> On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: > >>>>> Did we reach any conclusion on the subject? > >>>>> > >>>>> It seems we are aiming for 2.7 after 2.6 and plan the major version > >> bump > >>>>> to 3.0 after 2.7 (assuming we make progress on ZK removal as > planned?) > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 5/18/20 1:11 PM, Boyang Chen wrote: > >>>>>> One more thing I would like to see deprecated (hopefully no one > >> mentioned > >>>>>> before) is the zk based consumer offset support. > >>>>>> > >>>>>> On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> > >> wrote: > >>>>>> > >>>>>>> Hi Michael, > >>>>>>> > >>>>>>> It would be better to discuss the background behind KIP-500 in a > >> separate > >>>>>>> thread, since this thread is about the Kafka 3.0 release. As > others > >> have > >>>>>>> said, your questions are answered in the KIP. For example, "what > is > >> the > >>>>>>> actual goal?" is addressed in the motivation section. > >>>>>>> > >>>>>>> I agree that Kafka's usage of Apache ZooKeeper could be optimized. > >> But > >>>>>>> there are fundamental limitations to this approach compared to > >> storing our > >>>>>>> metadata internally. For example, having to contact a remote > server > >> to > >>>>>>> reload all your metadata on a controller failover simply doesn't > >> scale past > >>>>>>> a certain point. > >>>>>>> > >>>>>>> Apache Curator is a nice API, and if we were starting again today > we > >> would > >>>>>>> certainly consider using it. But it doesn't allow us to do > anything > >> more > >>>>>>> efficiently than ZooKeeper could already do it. > >>>>>>> > >>>>>>> Finally, Kafka's core competence is logs. While our replication > >> protocol > >>>>>>> is not Raft, it shares many similarities with that protocol. So I > >> think > >>>>>>> it's a bit unfair to say that it is "catastrophic hubris" to > believe > >> we can > >>>>>>> implement the protocol. > >>>>>>> > >>>>>>> best, > >>>>>>> Colin > >>>>>>> > >>>>>>> > >>>>>>> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: > >>>>>>>> Yes, I've read the KIP. But all it really says to me is "we have > >> never > >>>>>>>> gotten around to using ZooKeeper properly." To the extent that > any > >> of > >>>>>>> the > >>>>>>>> distributed-state-maintenance problems discussed in "Metadata as > an > >> Event > >>>>>>>> Log" can be solved — and some of them intrinsically can't, because > >> CAP > >>>>>>>> theorem — most of them are already implemented very effectively in > >>>>>>> Curator > >>>>>>>> recipes. (For instance, Curator's Tree Cache > >>>>>>>> https://curator.apache.org/curator-recipes/tree-cache.html is a > >> good > >>>>>>> fit to > >>>>>>>> some of the state-maintenance needs.) > >>>>>>>> > >>>>>>>> Kafka does have some usage patterns that don't map neatly onto > >> existing > >>>>>>>> Curator recipes. For instance, neither LeaderSelector nor > >> LeaderLatch > >>>>>>>> implements leader preference in the way that the existing Kafka > >> partition > >>>>>>>> leadership election procedure does. But why not handle that by > >> improving > >>>>>>>> and extending Curator? That way, other Curator users benefit, and > >> we get > >>>>>>>> additional highly experienced reviewers' eyes on the distributed > >>>>>>>> algorithms, which are very very tricky to get right. > >>>>>>>> > >>>>>>>> > >>>>>>>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com > > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Michael. This is discussed in the KIP. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation > >>>>>>>>> > >>>>>>>>> Ron > >>>>>>>>> > >>>>>>>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < > >>>>>>> m.k.edwa...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> What is the actual goal of removing the ZooKeeper dependency? > In > >> my > >>>>>>>>>> experience, if ZooKeeper is properly provisioned and deployed, > >> it's > >>>>>>>>> largely > >>>>>>>>>> trouble-free. (You do need to know how to use observers > >> properly.) > >>>>>>>>> There > >>>>>>>>>> are some subtleties about timeouts and leadership changes, but > >>>>>>> they're > >>>>>>>>>> pretty small stuff. Why go to all the trouble of building a new > >>>>>>>>>> distributed-consensus system that's going to have catastrophic > >> bugs > >>>>>>> for > >>>>>>>>>> years to come? It seems like such an act of hubris to me, as > well > >>>>>>> as a > >>>>>>>>>> massive waste of engineering effort. What is there to be > gained? > >>>>>>>>>> > >>>>>>>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax < > mj...@apache.org > >>> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Sure, we can compile a list for Kafka Streams. But the KIP > would > >> be > >>>>>>> for > >>>>>>>>>>> 3.0, so I don't think it's urgent to do it now? > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -Matthias > >>>>>>>>>>> > >>>>>>>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: > >>>>>>>>>>>> Thanks, Guozhang-- sounds like a good plan. > >>>>>>>>>>>> > >>>>>>>>>>>> I think it would be good to have a list of deprecated streams > >> APIs > >>>>>>> that > >>>>>>>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its > >> own > >>>>>>> KIP? > >>>>>>>>>>>> > >>>>>>>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use > in > >>>>>>> 2.6 if > >>>>>>>>>>> we want to remove it in 3.0. I don't have a good sense of how > >>>>>>>>> practical it > >>>>>>>>>>> is to deprecate this now, so I will defer to others here. But > >> the > >>>>>>> KIP > >>>>>>>>>>> freeze for 2.6 is coming soon, so if we want to make the case, > >> now > >>>>>>> is > >>>>>>>>> the > >>>>>>>>>>> time. > >>>>>>>>>>>> > >>>>>>>>>>>> best, > >>>>>>>>>>>> Colin > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > >>>>>>>>>>>>> Hey folks, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Sorry for stating that the bridge release would not break any > >>>>>>>>>>> compatibility > >>>>>>>>>>>>> before, which is incorrect and confused many people. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I think one way to think about the versioning is that: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 0) In a 2.x version moving ahead we would deprecate the > >>>>>>> ZK-dependent > >>>>>>>>>>> tools > >>>>>>>>>>>>> such as --zookeeper flags from various scripts (KIP-555) > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) In 3.0 we would at least make one incompatible change for > >>>>>>> example > >>>>>>>>> to > >>>>>>>>>>>>> remove the deprecated ZK flags. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK > >> entirely, > >>>>>>>>>>>>> including usages such as security credentials / broker > >>>>>>> registration / > >>>>>>>>>>> etc > >>>>>>>>>>>>> which are via ZK today as well. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> For 1), I'd love to add a few more incompatibility changes in > >> 3.0 > >>>>>>> from > >>>>>>>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating > and > >>>>>>> then > >>>>>>>>>>> remove > >>>>>>>>>>>>> in major releases, and since 2.0 we've accumulated quite a > few > >>>>>>>>>>> deprecated > >>>>>>>>>>>>> APIs, and I can compile a list of KIPs that contain those if > >>>>>>> people > >>>>>>>>> are > >>>>>>>>>>>>> interested. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Guozhang > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe < > >> cmcc...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > >>>>>>>>>>>>>>>> In fact, we know that the bridge release will involve at > >> least > >>>>>>> one > >>>>>>>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>>>>>>> --zookeeper > >>>>>>>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release > >> are > >>>>>>>>> _both_ > >>>>>>>>>>>>>>> breaking changes, I think we only have one option: the 3.x > >> line > >>>>>>> are > >>>>>>>>>>> the > >>>>>>>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested > by > >>>>>>> Andrew > >>>>>>>>>>>>>>> Schofield. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Specifically: > >>>>>>>>>>>>>>> - in order to _remove_ (not merely deprecate) the > --zookeeper > >>>>>>> args, > >>>>>>>>> we > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>>> need a major release. > >>>>>>>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a > >> bunch of > >>>>>>>>>>> external > >>>>>>>>>>>>>>> tooling like Cruise Control), we will need a major release. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I count two major releases. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Ryanne, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I agree that dropping ZK completely will need a new major > >> release > >>>>>>>>> after > >>>>>>>>>>>>>> 3.0. I think that's OK and in keeping with how we've > handled > >>>>>>>>>>> deprecation > >>>>>>>>>>>>>> and removal in the past. It's important for users to have a > >>>>>>> smooth > >>>>>>>>>>> upgrade > >>>>>>>>>>>>>> path. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> best, > >>>>>>>>>>>>>> Colin > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ryanne > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < > >>>>>>> cmcc...@apache.org> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > >>>>>>>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's > >>>>>>> "bridge > >>>>>>>>>>>>>>>>> release" so there is a clean break from Zookeeper after > >> 3.0. > >>>>>>> The > >>>>>>>>>>>>>>>>> bridge release by definition is an attempt to not break > >>>>>>> anything, > >>>>>>>>> so > >>>>>>>>>>>>>>>>> it theoretically doesn't warrant a major release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Ryanne, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I think it's important to clarify this a little bit. The > >>>>>>> bridge > >>>>>>>>>>>>>> release > >>>>>>>>>>>>>>>> (really, releases, plural) allow you to upgrade from a > >> cluster > >>>>>>> that > >>>>>>>>>>> is > >>>>>>>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, > >> that > >>>>>>>>>>> doesn't > >>>>>>>>>>>>>>>> imply that the bridge release itself doesn't break > anything. > >>>>>>>>>>> Upgrading > >>>>>>>>>>>>>>>> to the bridge release itself might involve some minor > >>>>>>>>>>> incompatibility. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Kafka does occasionally have incompatible changes. In > those > >>>>>>> cases, > >>>>>>>>>>> we > >>>>>>>>>>>>>>>> bump the major version number. One example is that when > we > >>>>>>> went > >>>>>>>>> from > >>>>>>>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This > >> is > >>>>>>> an > >>>>>>>>>>>>>>>> incompatible change. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> In fact, we know that the bridge release will involve at > >> least > >>>>>>> one > >>>>>>>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>>>>>>> --zookeeper > >>>>>>>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We've been preparing for this change for a long time. > >> People > >>>>>>> have > >>>>>>>>>>>>>> spent > >>>>>>>>>>>>>>>> a lot of effort designing new APIs that can be used > instead > >> of > >>>>>>> the > >>>>>>>>>>> old > >>>>>>>>>>>>>>>> zookeeper-based code that some of the command-line tools > >>>>>>> used. We > >>>>>>>>>>> have > >>>>>>>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of > >> the > >>>>>>> day, > >>>>>>>>>>> it > >>>>>>>>>>>>>>>> is still an incompatible change. So it's unfortunately > not > >>>>>>>>> possible > >>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> bridge release to be a 2.x release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" > >>>>>>> turns > >>>>>>>>> out > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> be impractical), we should consider forking 3.0 while > >>>>>>> maintaining > >>>>>>>>> a > >>>>>>>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x > can > >>>>>>> evolve > >>>>>>>>>>>>>>>>> dramatically without breaking the 2.x line. In > particular, > >>>>>>>>> anything > >>>>>>>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while > >>>>>>> every > >>>>>>>>>>> other > >>>>>>>>>>>>>>>>> feature targets 2.6. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Just to be super clear about this, what we want to do here > >> is > >>>>>>>>> support > >>>>>>>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a > >>>>>>> while. > >>>>>>>>> So > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> same branch will have support for both the old way and the > >> new > >>>>>>> way > >>>>>>>>> of > >>>>>>>>>>>>>>>> managing metadata. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This will allow us to get an "alpha" version of the > KIP-500 > >>>>>>> mode > >>>>>>>>> out > >>>>>>>>>>>>>> early > >>>>>>>>>>>>>>>> for people to experiment with. It also greatly reduces > the > >>>>>>> number > >>>>>>>>> of > >>>>>>>>>>>>>> Kafka > >>>>>>>>>>>>>>>> releases we have to make, and the amount of backporting we > >>>>>>> have to > >>>>>>>>>>> do. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", > I > >>>>>>> think > >>>>>>>>>>> this > >>>>>>>>>>>>>>>>> is premature given Kafka's time-based release schedule. > If > >> the > >>>>>>>>>>> bridge > >>>>>>>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, > >> then > >>>>>>>>> sure > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> let's make that the bridge release in retrospect. And if > we > >>>>>>> get > >>>>>>>>> all > >>>>>>>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm > onboard > >>>>>>> with > >>>>>>>>>>>>>> naming > >>>>>>>>>>>>>>>>> it "3.0" instead. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker > >> before > >>>>>>> 3.0 > >>>>>>>>> as > >>>>>>>>>>>>>>>>> well. I'm happy to drive that additional breaking change. > >>>>>>> Maybe > >>>>>>>>> 2.6 > >>>>>>>>>>>>>>>>> can be the "bridge" for MM2 as well. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I don't have a strong opinion either way about this, but > if > >> we > >>>>>>> want > >>>>>>>>>>> to > >>>>>>>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it > >> first, > >>>>>>>>>>>>>> right? Are > >>>>>>>>>>>>>>>> we ready to do that? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> best, > >>>>>>>>>>>>>>>> Colin > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Ryanne > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe < > >> cmcc...@apache.org > >>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> We've had a few proposals recently for incompatible > >>>>>>> changes. One > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the > >>>>>>>>> Administrative > >>>>>>>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK > >> Mutation > >>>>>>>>>>>>>>>>>> Protocols to the Controller. I think it's time to start > >>>>>>> thinking > >>>>>>>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to > >> 3.0 > >>>>>>>>> after > >>>>>>>>>>>>>>>>>> the 2.6 release. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like > to > >>>>>>> make > >>>>>>>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet > the > >>>>>>>>>>>>>> default.) > >>>>>>>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as > >> well as > >>>>>>>>> some > >>>>>>>>>>>>>>>>>> of the other KIPs we've done recently. Since it will > take > >>>>>>> some > >>>>>>>>>>>>>> time > >>>>>>>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will > >> hide > >>>>>>> it > >>>>>>>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing > >>>>>>> this all > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> detail soon.) > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up > >> next > >>>>>>>>> after > >>>>>>>>>>>>>>>>>> 2.6? Are there any other things we should change in the > >> 2.6 > >>>>>>> -> > >>>>>>>>> 3.0 > >>>>>>>>>>>>>>>>>> transition? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> best, Colin > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> -- Guozhang > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> Attachments: > >>>>> * signature.asc > >> > > > -- -- Guozhang
