Hey everyone. Regarding the status of KIP-125, just a heads up: I have an implementation of KIP-125 (KAFKA-4513) here: https://github.com/onurkaraman/kafka/commit/3b5448006ab70ba2b0b5e177853d191d0f777452
The code might need to be rebased. The steps described in the KIP are a bit involved. Other than that, the implementation might have a bug with respect to converting arbitrary blacklist regexes to whitelist regexes since the new consumer only accepts whitelists. On Fri, Nov 10, 2017 at 11:36 AM, Jeff Widman <j...@jeffwidman.com> wrote: > Re: migrating offsets for old Scala consumers. > > I work in the python world, so haven't directly used the old high level > consumer, but from what I understand the underlying problem remains the > migration of zookeeper offsets to the __consumer_offsets topic. > > We've used a slightly modified version of Grant Henke's script for > migrating offsets here: https://github.com/apache/kafka/pull/2615 > It doesn't support rolling upgrades, but other than that it's great... I've > used it for multiple migrations, and very thankful for the time Grant put > into it. > > I don't know that it's worth pulling this into core, it might be, it might > not be. But it probably is worth documenting the procedure at least > somewhere. > > Personally, I suspect that those who absolutely need a rolling migration > and cannot handle a short period of downtime while doing a migration > probably have in-house experts on Kafka who are familiar with the issues > and willing to figure out a solution. The rest of the world can generally > handle a short maintenance window. > > > > > On Fri, Nov 10, 2017 at 10:46 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Gwen, > > > > A KIP has been proposed, but it is stalled: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+ > > ZookeeperConsumerConnector+to+KafkaConsumer+Migration+and+Rollback > > > > Unless the interested parties pick that up, we would drop support > without a > > rolling upgrade path. Users would be able to use the old consumers from > > 1.1.x for a long time. The old Scala clients don't support the message > > format introduced in 0.11.0, so the feature set is pretty much frozen and > > there's little benefit in upgrading. But there is a cost in keeping them > in > > the codebase. > > > > Ismael > > > > On Fri, Nov 10, 2017 at 6:02 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > Last time we tried deprecating the Scala consumer, there were concerns > > > about a lack of upgrade path. There is no rolling upgrade, and > migrating > > > offsets is not trivial (and not documented). > > > > > > Did anything change in that regard? Or are we planning on dropping > > support > > > without an upgrade path? > > > > > > > > > On Fri, Nov 10, 2017 at 5:37 PM Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > Thanks Ismael, the proposal looks good to me. > > > > > > > > A side note regarding: https://issues.apache.org/ > > jira/browse/KAFKA-5637, > > > > could we resolve this ticket sooner than later to make clear about > the > > > code > > > > deprecation and support duration when moving from 1.0.x to 2.0.x? > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma <ism...@juma.me.uk> > > wrote: > > > > > > > > > Features for 2.0.0 will be known after 1.1.0 is released in > February > > > > 2018. > > > > > We are still doing the usual time-based release process[1]. > > > > > > > > > > I am raising this well ahead of time because of the potential > impact > > of > > > > > removing the old Scala clients (particularly the old high-level > > > consumer) > > > > > and dropping support for Java 7. Hopefully users can then plan > > > > accordingly. > > > > > We would do these changes in trunk soon after 1.1.0 is released > > (around > > > > > February). > > > > > > > > > > I think it makes sense to complete some of the work that was not > > ready > > > in > > > > > time for 1.0.0 (Controller improvements and JBOD are two that come > to > > > > mind) > > > > > in 1.1.0 (January 2018) and combined with the desire to give > advance > > > > > notice, June 2018 was the logical choice. > > > > > > > > > > There is no plan to support a particular release for longer. 1.x > > versus > > > > 2.x > > > > > is no different than 0.10.x versus 0.11.x from the perspective of > > > > > supporting older releases. > > > > > > > > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/Time+ > > > > > Based+Release+Plan > > > > > > > > > > On Fri, Nov 10, 2017 at 11:21 AM, Jaikiran Pai < > > > jai.forums2...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hi Ismael, > > > > > > > > > > > > Are there any new features other than the language specific > changes > > > > that > > > > > > are being planned for 2.0.0? Also, when 2.x gets released, will > the > > > 1.x > > > > > > series see continued bug fixes and releases in the community or > is > > > the > > > > > plan > > > > > > to have one single main version that gets continuous updates and > > > > > releases? > > > > > > > > > > > > By the way, why June 2018? :) > > > > > > > > > > > > -Jaikiran > > > > > > > > > > > > > > > > > > > > > > > > On 09/11/17 3:14 PM, Ismael Juma wrote: > > > > > > > > > > > >> Hi all, > > > > > >> > > > > > >> I'm starting this discussion early because of the potential > > impact. > > > > > >> > > > > > >> Kafka 1.0.0 was just released and the focus was on achieving the > > > > > original > > > > > >> project vision in terms of features provided while maintaining > > > > > >> compatibility for the most part (i.e. we did not remove > deprecated > > > > > >> components like the Scala clients). > > > > > >> > > > > > >> This was the right decision, in my opinion, but it's time to > start > > > > > >> thinking > > > > > >> about 2.0.0, which is an opportunity for us to remove major > > > deprecated > > > > > >> components and to benefit from Java 8 language enhancements (so > > that > > > > we > > > > > >> can > > > > > >> move faster). So, I propose the following for Kafka 2.0.0: > > > > > >> > > > > > >> 1. It should be released in June 2018 > > > > > >> 2. The Scala clients (Consumer, SimpleConsumer, Producer, > > > > SyncProducer) > > > > > >> will be removed > > > > > >> 3. Java 8 or higher will be required, i.e. support for Java 7 > will > > > be > > > > > >> dropped. > > > > > >> > > > > > >> Thoughts? > > > > > >> > > > > > >> Ismael > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > -- > > *Jeff Widman* > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) > <>< >
