Colin, Guozhang, Matthias, all makes sense to me, thanks. Ryanne
On Fri, May 8, 2020, 1:48 PM Matthias J. Sax <mj...@apache.org> wrote: > I agree with what was discussed. Having a (breaking) 3.0 release to > start the "bridge release" series and completing it with a 4.0 release > sounds intuitive to me. > > For old MirrorMaker: In general I am also in favor of removing it, but > there should be a reasonable deprecation period. Deprecating it in 2.6 > and removing it in 3.0 seems a little bit short to me? However, given > that 4.0 would not come too much after 3.0 (not sure how many release we > need to finalize ZK removal) it might be ok to have it a little longer? > > And yes, for Kafka Stream we would use the 3.0 release to remove a bunch > of previously deprecated APIs that we accumulated since the 2.0 release. > > > -Matthias > > On 5/7/20 4:28 PM, Guozhang Wang wrote: > > Hey folks, > > > > Sorry for stating that the bridge release would not break any > compatibility > > before, which is incorrect and confused many people. > > > > I think one way to think about the versioning is that: > > > > 0) In a 2.x version moving ahead we would deprecate the ZK-dependent > tools > > such as --zookeeper flags from various scripts (KIP-555) > > > > 1) In 3.0 we would at least make one incompatible change for example to > > remove the deprecated ZK flags. > > > > 2) In a future major version (e.g. 4.0) we would drop ZK entirely, > > including usages such as security credentials / broker registration / etc > > which are via ZK today as well. > > > > Then for the bridge release(s), it can be any or all of 3.x. > > > > > > For 1), I'd love to add a few more incompatibility changes in 3.0 from > > Kafka Streams: we evolve Streams public APIs by deprecating and then > remove > > in major releases, and since 2.0 we've accumulated quite a few deprecated > > APIs, and I can compile a list of KIPs that contain those if people are > > interested. > > > > > > Guozhang > > > > > > On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> wrote: > > > >> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > >>>> In fact, we know that the bridge release will involve at least one > >>>> incompatible change. We will need to drop support for the --zookeeper > >>>> flags in the command-line tools. > >>> > >>> If the bridge release(s) and the subsequent post-ZK release are _both_ > >>> breaking changes, I think we only have one option: the 3.x line are the > >>> bridge release(s), and ZK is removed in 4.0, as suggested by Andrew > >>> Schofield. > >>> > >>> Specifically: > >>> - in order to _remove_ (not merely deprecate) the --zookeeper args, we > >> will > >>> need a major release. > >>> - in oder to drop support for ZK entirely (e.g. break a bunch of > external > >>> tooling like Cruise Control), we will need a major release. > >>> > >>> I count two major releases. > >> > >> Hi Ryanne, > >> > >> I agree that dropping ZK completely will need a new major release after > >> 3.0. I think that's OK and in keeping with how we've handled > deprecation > >> and removal in the past. It's important for users to have a smooth > upgrade > >> path. > >> > >> best, > >> Colin > >> > >>> > >>> Ryanne > >>> > >>> - > >>> > >>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> > wrote: > >>> > >>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > >>>>> Hey Colin, I think we should wait until after KIP-500's "bridge > >>>>> release" so there is a clean break from Zookeeper after 3.0. The > >>>>> bridge release by definition is an attempt to not break anything, so > >>>>> it theoretically doesn't warrant a major release. > >>>> > >>>> Hi Ryanne, > >>>> > >>>> I think it's important to clarify this a little bit. The bridge > >> release > >>>> (really, releases, plural) allow you to upgrade from a cluster that is > >>>> using ZooKeeper to one that is not using ZooKeeper. But, that doesn't > >>>> imply that the bridge release itself doesn't break anything. > Upgrading > >>>> to the bridge release itself might involve some minor incompatibility. > >>>> > >>>> Kafka does occasionally have incompatible changes. In those cases, we > >>>> bump the major version number. One example is that when we went from > >>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an > >>>> incompatible change. > >>>> > >>>> In fact, we know that the bridge release will involve at least one > >>>> incompatible change. We will need to drop support for the --zookeeper > >>>> flags in the command-line tools. > >>>> > >>>> We've been preparing for this change for a long time. People have > >> spent > >>>> a lot of effort designing new APIs that can be used instead of the old > >>>> zookeeper-based code that some of the command-line tools used. We > have > >>>> also deprecated the old ZK-based flags. But at the end of the day, it > >>>> is still an incompatible change. So it's unfortunately not possible > >> for > >>>> the > >>>> bridge release to be a 2.x release. > >>>> > >>>>> If that's not the case (i.e. if a single "bridge release" turns out > >> to > >>>>> be impractical), we should consider forking 3.0 while maintaining a > >>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve > >>>>> dramatically without breaking the 2.x line. In particular, anything > >>>>> related to removing Zookeeper could land in pre-3.0 while every other > >>>>> feature targets 2.6. > >>>> > >>>> Just to be super clear about this, what we want to do here is support > >>>> operating in __either__ KIP-500 mode and legacy mode for a while. So > >> the > >>>> same branch will have support for both the old way and the new way of > >>>> managing metadata. > >>>> > >>>> This will allow us to get an "alpha" version of the KIP-500 mode out > >> early > >>>> for people to experiment with. It also greatly reduces the number of > >> Kafka > >>>> releases we have to make, and the amount of backporting we have to do. > >>>> > >>>>> > >>>>> If you are proposing 2.6 should be the "bridge release", I think this > >>>>> is premature given Kafka's time-based release schedule. If the bridge > >>>>> features happen to be merged before 2.6's feature freeze, then sure > >> -- > >>>>> let's make that the bridge release in retrospect. And if we get all > >>>>> the post-Zookeeper features merged before 2.7, I'm onboard with > >> naming > >>>>> it "3.0" instead. > >>>>> > >>>>> That said, we should aim to remove legacy MirrorMaker before 3.0 as > >>>>> well. I'm happy to drive that additional breaking change. Maybe 2.6 > >>>>> can be the "bridge" for MM2 as well. > >>>> > >>>> I don't have a strong opinion either way about this, but if we want to > >>>> remove the original MirrorMaker, we have to deprecate it first, > >> right? Are > >>>> we ready to do that? > >>>> > >>>> best, > >>>> Colin > >>>> > >>>>> > >>>>> Ryanne > >>>>> > >>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> > >> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> We've had a few proposals recently for incompatible changes. One > >> of > >>>>>> them is my KIP-604: Remove ZooKeeper Flags from the Administrative > >>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > >>>>>> Protocols to the Controller. I think it's time to start thinking > >>>>>> about Kafka 3.0. Specifically, I think we should move to 3.0 after > >>>>>> the 2.6 release. > >>>>>> > >>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to make > >>>>>> running in a ZooKeeper-less mode possible (but not yet the > >> default.) > >>>>>> This is the motivation behind KIP-590 and KIP-604, as well as some > >>>>>> of the other KIPs we've done recently. Since it will take some > >> time > >>>>>> to stabilize the new ZooKeeper-free Kafka code, we will hide it > >>>>>> behind an option initially. (We'll have a KIP describing this all > >> in > >>>>>> detail soon.) > >>>>>> > >>>>>> What does everyone think about having Kafka 3.0 come up next after > >>>>>> 2.6? Are there any other things we should change in the 2.6 -> 3.0 > >>>>>> transition? > >>>>>> > >>>>>> best, Colin > >>>>>> > >>>>> > >>>> > >>> > >> > > > > > >
