Hi Michael. This is discussed in the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation
Ron > On May 10, 2020, at 1:35 PM, Michael K. Edwards <m.k.edwa...@gmail.com> wrote: > > What is the actual goal of removing the ZooKeeper dependency? In my > experience, if ZooKeeper is properly provisioned and deployed, it's largely > trouble-free. (You do need to know how to use observers properly.) There > are some subtleties about timeouts and leadership changes, but they're > pretty small stuff. Why go to all the trouble of building a new > distributed-consensus system that's going to have catastrophic bugs for > years to come? It seems like such an act of hubris to me, as well as a > massive waste of engineering effort. What is there to be gained? > >> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org> wrote: >> >> Sure, we can compile a list for Kafka Streams. But the KIP would be for >> 3.0, so I don't think it's urgent to do it now? >> >> >> -Matthias >> >>> On 5/8/20 3:47 PM, Colin McCabe wrote: >>> Thanks, Guozhang-- sounds like a good plan. >>> >>> I think it would be good to have a list of deprecated streams APIs that >> we want to remove in 3.0. Maybe it's easiest to do that as its own KIP? >>> >>> For MirrorMaker 1, we should have a KIP to deprecate its use in 2.6 if >> we want to remove it in 3.0. I don't have a good sense of how practical it >> is to deprecate this now, so I will defer to others here. But the KIP >> freeze for 2.6 is coming soon, so if we want to make the case, now is the >> time. >>> >>> best, >>> Colin >>> >>> >>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: >>>> Hey folks, >>>> >>>> Sorry for stating that the bridge release would not break any >> compatibility >>>> before, which is incorrect and confused many people. >>>> >>>> I think one way to think about the versioning is that: >>>> >>>> 0) In a 2.x version moving ahead we would deprecate the ZK-dependent >> tools >>>> such as --zookeeper flags from various scripts (KIP-555) >>>> >>>> 1) In 3.0 we would at least make one incompatible change for example to >>>> remove the deprecated ZK flags. >>>> >>>> 2) In a future major version (e.g. 4.0) we would drop ZK entirely, >>>> including usages such as security credentials / broker registration / >> etc >>>> which are via ZK today as well. >>>> >>>> Then for the bridge release(s), it can be any or all of 3.x. >>>> >>>> >>>> For 1), I'd love to add a few more incompatibility changes in 3.0 from >>>> Kafka Streams: we evolve Streams public APIs by deprecating and then >> remove >>>> in major releases, and since 2.0 we've accumulated quite a few >> deprecated >>>> APIs, and I can compile a list of KIPs that contain those if people are >>>> interested. >>>> >>>> >>>> Guozhang >>>> >>>> >>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> wrote: >>>>> >>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: >>>>>>> In fact, we know that the bridge release will involve at least one >>>>>>> incompatible change. We will need to drop support for the >> --zookeeper >>>>>>> flags in the command-line tools. >>>>>> >>>>>> If the bridge release(s) and the subsequent post-ZK release are _both_ >>>>>> breaking changes, I think we only have one option: the 3.x line are >> the >>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by Andrew >>>>>> Schofield. >>>>>> >>>>>> Specifically: >>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper args, we >>>>> will >>>>>> need a major release. >>>>>> - in oder to drop support for ZK entirely (e.g. break a bunch of >> external >>>>>> tooling like Cruise Control), we will need a major release. >>>>>> >>>>>> I count two major releases. >>>>> >>>>> Hi Ryanne, >>>>> >>>>> I agree that dropping ZK completely will need a new major release after >>>>> 3.0. I think that's OK and in keeping with how we've handled >> deprecation >>>>> and removal in the past. It's important for users to have a smooth >> upgrade >>>>> path. >>>>> >>>>> best, >>>>> Colin >>>>> >>>>>> >>>>>> Ryanne >>>>>> >>>>>> - >>>>>> >>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> >> wrote: >>>>>> >>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: >>>>>>>> Hey Colin, I think we should wait until after KIP-500's "bridge >>>>>>>> release" so there is a clean break from Zookeeper after 3.0. The >>>>>>>> bridge release by definition is an attempt to not break anything, so >>>>>>>> it theoretically doesn't warrant a major release. >>>>>>> >>>>>>> Hi Ryanne, >>>>>>> >>>>>>> I think it's important to clarify this a little bit. The bridge >>>>> release >>>>>>> (really, releases, plural) allow you to upgrade from a cluster that >> is >>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, that >> doesn't >>>>>>> imply that the bridge release itself doesn't break anything. >> Upgrading >>>>>>> to the bridge release itself might involve some minor >> incompatibility. >>>>>>> >>>>>>> Kafka does occasionally have incompatible changes. In those cases, >> we >>>>>>> bump the major version number. One example is that when we went from >>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an >>>>>>> incompatible change. >>>>>>> >>>>>>> In fact, we know that the bridge release will involve at least one >>>>>>> incompatible change. We will need to drop support for the >> --zookeeper >>>>>>> flags in the command-line tools. >>>>>>> >>>>>>> We've been preparing for this change for a long time. People have >>>>> spent >>>>>>> a lot of effort designing new APIs that can be used instead of the >> old >>>>>>> zookeeper-based code that some of the command-line tools used. We >> have >>>>>>> also deprecated the old ZK-based flags. But at the end of the day, >> it >>>>>>> is still an incompatible change. So it's unfortunately not possible >>>>> for >>>>>>> the >>>>>>> bridge release to be a 2.x release. >>>>>>> >>>>>>>> If that's not the case (i.e. if a single "bridge release" turns out >>>>> to >>>>>>>> be impractical), we should consider forking 3.0 while maintaining a >>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve >>>>>>>> dramatically without breaking the 2.x line. In particular, anything >>>>>>>> related to removing Zookeeper could land in pre-3.0 while every >> other >>>>>>>> feature targets 2.6. >>>>>>> >>>>>>> Just to be super clear about this, what we want to do here is support >>>>>>> operating in __either__ KIP-500 mode and legacy mode for a while. So >>>>> the >>>>>>> same branch will have support for both the old way and the new way of >>>>>>> managing metadata. >>>>>>> >>>>>>> This will allow us to get an "alpha" version of the KIP-500 mode out >>>>> early >>>>>>> for people to experiment with. It also greatly reduces the number of >>>>> Kafka >>>>>>> releases we have to make, and the amount of backporting we have to >> do. >>>>>>> >>>>>>>> >>>>>>>> If you are proposing 2.6 should be the "bridge release", I think >> this >>>>>>>> is premature given Kafka's time-based release schedule. If the >> bridge >>>>>>>> features happen to be merged before 2.6's feature freeze, then sure >>>>> -- >>>>>>>> let's make that the bridge release in retrospect. And if we get all >>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard with >>>>> naming >>>>>>>> it "3.0" instead. >>>>>>>> >>>>>>>> That said, we should aim to remove legacy MirrorMaker before 3.0 as >>>>>>>> well. I'm happy to drive that additional breaking change. Maybe 2.6 >>>>>>>> can be the "bridge" for MM2 as well. >>>>>>> >>>>>>> I don't have a strong opinion either way about this, but if we want >> to >>>>>>> remove the original MirrorMaker, we have to deprecate it first, >>>>> right? Are >>>>>>> we ready to do that? >>>>>>> >>>>>>> best, >>>>>>> Colin >>>>>>> >>>>>>>> >>>>>>>> Ryanne >>>>>>>> >>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> We've had a few proposals recently for incompatible changes. One >>>>> of >>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the Administrative >>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation >>>>>>>>> Protocols to the Controller. I think it's time to start thinking >>>>>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 after >>>>>>>>> the 2.6 release. >>>>>>>>> >>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to make >>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the >>>>> default.) >>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as well as some >>>>>>>>> of the other KIPs we've done recently. Since it will take some >>>>> time >>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide it >>>>>>>>> behind an option initially. (We'll have a KIP describing this all >>>>> in >>>>>>>>> detail soon.) >>>>>>>>> >>>>>>>>> What does everyone think about having Kafka 3.0 come up next after >>>>>>>>> 2.6? Are there any other things we should change in the 2.6 -> 3.0 >>>>>>>>> transition? >>>>>>>>> >>>>>>>>> best, Colin >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> -- Guozhang >>>> >> >>
