Re: [ANNOUNCE] Apache Kafka 3.1.0

[Gwen Shapira]

Exciting! Thanks for driving the release, David.

On Mon, Jan 24, 2022 at 9:04 AM David Jacot  wrote:
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.1.0.
>
> It is a major release that includes many new features, including:
>
> * Apache Kafka supports Java 17
> * The FetchRequest supports Topic IDs (KIP-516)
> * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> * Add broker count metrics (KIP-748)
> * Differentiate consistently metric latency measured in millis and
> nanos (KIP-773)
> * The eager rebalance protocol is deprecated (KAFKA-13439)
> * Add TaskId field to StreamsException (KIP-783)
> * Custom partitioners in foreign-key joins (KIP-775)
> * Fetch/findSessions queries with open endpoints for
> SessionStore/WindowStore (KIP-766)
> * Range queries with open endpoints (KIP-763)
> * Add total blocked time metric to Streams (KIP-761)
> * Add additional configuration to control MirrorMaker2 internal topics
> naming convention (KIP-690)
>
> You may read a more detailed list of features in the 3.1.0 blog post:
> https://blogs.apache.org/kafka/
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#3.1.0
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 114 contributors to this release!
>
> A. Sophie Blee-Goldman, Alexander Iskuskov, Alexander Stohr, Almog
> Gavra, Andras Katona, Andrew Patterson, Andy Chambers, Andy Lapidas,
> Anna Sophie Blee-Goldman, Antony Stubbs, Arjun Satish, Bill Bejeck,
> Boyang Chen, Bruno Cadonna, CHUN-HAO TANG, Cheng Tan, Chia-Ping Tsai,
> Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel
> Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo
> Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez,
> Ismael Juma, Israel Ekpo, Ivan Ponomarev, Jakub Scholz, James Galasyn,
> Jason Gustafson, Jeff Kim, Jim Galasyn, JoeCqupt, Joel Hamill, John
> Gray, John Roesler, Jongho Jeon, Jorge Esteban Quilcate Otoya, Jose
> Sancio, Josep Prat, José Armando García Sancio, Jun Rao, Justine
> Olshan, Kalpesh Patel, Kamal Chandraprakash, Kevin Zhang, Kirk True,
> Konstantine Karantasis, Kowshik Prakasam, Leah Thomas, Lee Dongjin,
> Lucas Bradstreet, Luke Chen, Manikumar Reddy, Matthew Wong, Matthias
> J. Sax, Michael Carter, Mickael Maison, Nigel Liang, Niket, Niket
> Goel, Oliver Hutchison, Omnia G H Ibrahim, Patrick Stuedi, Phil
> Hardwick, Prateek Agarwal, Rajini Sivaram, Randall Hauch, René Kerner,
> Richard Yu, Rohan, Ron Dagostino, Ryan Dielhenn, Sanjana Kaundinya,
> Satish Duggana, Sergio Peña, Sherzod Mamadaliev, Stanislav Vodetskyi,
> Ted Yu, Tom Bentley, Tomas Forsman, Tomer Wizman, Uwe Eisele, Victoria
> Xia, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Weisheng
> Yang, Xavier Léauté, Yanwen(Jason) Lin, Yi Ding, Zara Lim, andy0x01,
> dengziming, feyman2016, ik, ik.lim, jem, jiangyuan, kpatelatwork,
> leah, loboya~, lujiefsi, sebbASF, singingMan, vamossagar12,
> wenbingshen
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> David

[ANNOUNCE] New Kafka PMC member: David Jacot

[Gwen Shapira]

Hi everyone,

David Jacot has been an Apache Kafka committer since Oct 2020 and has been 
contributing to the community consistently this entire time - especially 
notable the fact that he reviewed around 150 PRs in the last year. It is my 
pleasure to announce that David agreed to join the Kafka PMC.

Congratulations, David!

Gwen Shapira, on behalf of Apache Kafka PMC

Re: [VOTE] KIP-800: Add reason to LeaveGroupRequest

[Gwen Shapira]

+1

Thanks for driving David. Super useful.

On Wed, Nov 24, 2021 at 8:53 AM David Jacot 
wrote:

> Hi folks,
>
> I'd like to start a vote on KIP-800: Add reason to LeaveGroupRequest.
>
> KIP: https://cwiki.apache.org/confluence/x/eYyqCw
>
> Please let me know what you think.
>
> Best,
> David
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-714: Client metrics and observability

[Gwen Shapira]

Hey,

I noticed that there was no discussion for the last 10 days, but I couldn't
find the vote thread. Is there one that I'm missing?

Gwen

On Wed, Sep 22, 2021 at 4:58 AM Magnus Edenhill  wrote:

> Den tis 21 sep. 2021 kl 06:58 skrev Colin McCabe :
>
> > On Mon, Sep 20, 2021, at 17:35, Feng Min wrote:
> > > Thanks Magnus & Colin for the discussion.
> > >
> > > Based on KIP-714's stateless design, Client can pretty much use any
> > > connection to any broker to send metrics. We are not associating
> > connection
> > > with client metric state. Is my understanding correct? If yes,  how
> about
> > > the following two scenarios
> > >
> > > 1) One Client (Client-ID) registers two different client instance id
> via
> > > separate registration. Is it permitted? If OK, how to distinguish them
> > from
> > > the case 2 below.
> > >
> >
> > Hi Feng,
> >
> > My understanding, which Magnus can clarify I guess, is that you could
> have
> > something like two Producer instances running with the same client.id
> > (perhaps because they're using the same config file, for example). They
> > could even be in the same process. But they would get separate UUIDs.
> >
> > I believe Magnus used the term client to mean "Producer or Consumer". So
> > if you have both a Producer and a Consumer in your application I would
> > expect you'd get separate UUIDs for both. Again Magnus can chime in
> here, I
> > guess.
> >
>
> That's correct.
>
>
> >
> > > 2) How about the client restarting? What's the expectation? Should the
> > > server expect the client to carry a persisted client instance id or
> > should
> > > the client be treated as a new instance?
> >
> > The KIP doesn't describe any mechanism for persistence, so I would assume
> > that when you restart the client you get a new UUID. I agree that it
> would
> > be good to spell this out.
> >
> >
> Right, it will not be persisted since a client instance can't be restarted.
>
> Will update the KIP to make this clearer.
>
> /Magnus
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Gwen Shapira]

+1 (binding). Thank you for your patience and clear explanations, Omnia.

On Mon, Jul 26, 2021 at 3:39 PM Omnia Ibrahim 
wrote:

> Bumping up this voting thread.
>
> On Fri, Jul 16, 2021 at 1:57 PM Omnia Ibrahim 
> wrote:
>
> > Hi,
> > Can I get 2 more +1 binding for this KIP?
> > Thanks
> >
> > On Fri, Jul 2, 2021 at 5:14 PM Omnia Ibrahim 
> > wrote:
> >
> >> Hi All,
> >>
> >> Just thought of bumping this voting thread again to see if we can form a
> >> consensus around this.
> >>
> >> Thanks
> >>
> >> On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> +1 (binding)
> >>> Thanks for the KIP!
> >>>
> >>> On Tue, May 4, 2021 at 3:23 PM Igor Soarez 
> >>> wrote:
> >>> >
> >>> > Another +1 here, also non-binding.
> >>> >
> >>> > Thank you Omnia!
> >>> >
> >>> > --
> >>> > Igor
> >>> >
> >>> >
> >>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
> >>> > > +1 (non-binding), thanks!
> >>> > >
> >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> >>> wrote:
> >>> > >
> >>> > >> Hi
> >>> > >> Can I get a vote on this, please?
> >>> > >>
> >>> > >> Best
> >>> > >> Omnia
> >>> > >>
> >>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
> >>> o.g.h.ibra...@gmail.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >>> If anyone interested in reading the discussions you can find it
> >>> here
> >>> > >>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
> >>> > >>>
> >>> > >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim <
> >>> o.g.h.ibra...@gmail.com>
> >>> > >>> wrote:
> >>> > >>>
> >>> > >>>> Hi everyone,
> >>> > >>>> I’m proposing a new KIP for MirrorMaker 2 to add the ability to
> >>> control
> >>> > >>>> internal topics naming convention. The proposal details are here
> >>> > >>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >>> > >>>>
> >>> > >>>> Please vote in this thread.
> >>> > >>>> Thanks
> >>> > >>>> Omnia
> >>> > >>>>
> >>> > >>>
> >>> > >
> >>>
> >>
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-618: Exactly-once support for source connectors

[Gwen Shapira]

I'm supportive of the feature and the interface details discussed in the
discussion thread. I just want to clarify that I'm voting for the last
version discussed in the thread - that includes two phase upgrade and no
breaking changes in 3.0.

+1 (binding)


On Wed, Jun 9, 2021, 5:32 AM Chris Egerton 
wrote:

> Hi all,
>
> Friendly reminder that the KIP freeze is today; please cast your votes if
> you'd like to see this feature introduced in time for 3.0.
>
> Cheers,
>
> Chris
>
> On Mon, Jun 7, 2021 at 1:49 AM Dongjin Lee  wrote:
>
> > +1 (non-binding).
> >
> > Thanks,
> > Dongjin
> >
> > On Fri, Jun 4, 2021 at 2:35 PM Ryanne Dolan 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Thu, Jun 3, 2021, 10:23 AM Chris Egerton
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-618, which adds support for
> > > exactly-once
> > > > delivery guarantees for source connectors in the Kafka Connect
> > framework.
> > > >
> > > > I suspect there might be a little more discussion to be had but with
> > the
> > > > KIP freeze deadline approaching, I wanted to give anyone following
> > along
> > > > the chance to cast a +1 as soon as they feel that we've gotten to a
> > > > reasonable state.
> > > >
> > > > The KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
> > > >
> > > > The discussion thread:
> > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCAMdOrUX-VK5OSB3OOdJNXW_YKEJH9FjQZ4eyzr2GXUhSeDnF3Q%40mail.gmail.com%3E
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[Gwen Shapira]

+1 (binding)

Exciting to see this.

On Wed, Jun 9, 2021, 11:29 AM Ismael Juma  wrote:

> Hi all,
>
> Consensus was reached in the discussion thread and part of what is proposed
> has to happen by 3.0, so starting the vote for KIP-724:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
>
> If there are concerns or objections, feel free to point them out in this
> thread or the discuss thread.
>
> Ismael
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Gwen Shapira]

+1 (binding)

On Wed, Jun 9, 2021 at 8:28 AM Dongjin Lee  wrote:

> Bumping up the voting thread.
>
> Please note that today is the KIP freeze day. +1 non-binding until now.
>
> Thanks,
> Dongjin
>
> On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:
>
> > Hi Dongjin,
> > Thanks for the KIP.
> > +1 (non-binding)
> >
> > Thanks.
> > Luke
> >
> > On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > > ReplicaVerificationTool:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  <http://goog_969573159/>github.com/dongjinleekr
> > > <https://github.com/dongjinleekr>keybase:
> > https://keybase.io/dongjinleekr
> > > <https://keybase.io/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > speakerdeck.com/dongjin
> > > <https://speakerdeck.com/dongjin>*
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr
> <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Gwen Shapira]

ents, "offsetSyncTopic()"
> >>>>>>>> takes the
> >>>>>>>> >>> target cluster alias and "checkpointTopic()" takes
> >>>>>>>> "clusterAlias"
> >>>>>>>> >>> (which one is it? source or target?). As the interface extends
> >>>>>>>> >>> Configurable, maybe we could get rid of all the arguments and
> >>>>>>>> use the
> >>>>>>>> >>> config to find the cluster aliases. WDYT?
> >>>>>>>> >>>
> >>>>>>>> >>> These are minor concerns, just making sure I fully understand
> >>>>>>>> how the
> >>>>>>>> >>> API is expected to be used. Once these are cleared, I'll be
> >>>>>>>> happy to
> >>>>>>>> >>> vote for this KIP.
> >>>>>>>> >>>
> >>>>>>>> >>> Thanks
> >>>>>>>> >>>
> >>>>>>>> >>> On Fri, Jan 8, 2021 at 12:06 PM Omnia Ibrahim <
> >>>>>>>> o.g.h.ibra...@gmail.com> wrote:
> >>>>>>>> >>> >
> >>>>>>>> >>> > Hi Mickael,
> >>>>>>>> >>> > Did you get time to review the changes to the KIP? If you
> >>>>>>>> okay with it could you vote for the KIP here ttps://
> >>>>>>>> www.mail-archive.com/dev@kafka.apache.org/msg113575.html?
> >>>>>>>> >>> > Thanks
> >>>>>>>> >>> >
> >>>>>>>> >>> > On Thu, Dec 10, 2020 at 2:19 PM Omnia Ibrahim <
> >>>>>>>> o.g.h.ibra...@gmail.com> wrote:
> >>>>>>>> >>> >>
> >>>>>>>> >>> >> Hi Mickael,
> >>>>>>>> >>> >> 1) That's right the interface and default implementation
> >>>>>>>> will in mirror-connect
> >>>>>>>> >>> >> 2) Renaming the interface should be fine too especially if
> >>>>>>>> you planning to move other functionality related to the creation
> there, I
> >>>>>>>> can edit this
> >>>>>>>> >>> >>
> >>>>>>>> >>> >> if you are okay with that please vote for the KIP here
> >>>>>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html
> >>>>>>>> >>> >>
> >>>>>>>> >>> >>
> >>>>>>>> >>> >> Thanks
> >>>>>>>> >>> >> Omnia
> >>>>>>>> >>> >> On Thu, Dec 10, 2020 at 12:58 PM Mickael Maison <
> >>>>>>>> mickael.mai...@gmail.com> wrote:
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> Hi Omnia,
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> Thank you for the reply, it makes sense.
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> A couple more comments:
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> 1) I'm assuming the new interface and default
> >>>>>>>> implementation will be
> >>>>>>>> >>> >>> in the mirror-client project? as the names of some of
> these
> >>>>>>>> topics are
> >>>>>>>> >>> >>> needed by RemoteClusterUtils on the client-side.
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> 2) I'm about to open a KIP to specify where the
> >>>>>>>> offset-syncs topic is
> >>>>>>>> >>> >>> created by MM2. In restricted environments, we'd prefer
> MM2
> >>>>>>>> to only
> >>>>>>>> >>> >>> have read access to the source cluster and have the
> >>>>>>>> offset-syncs on
> >>>>>>>> >>> >>> the target cluster. I think allowing to specify the
> cluster
> >>>>>>>> where to
> >>>>>>>> >>> >>> create that topic would be a natural extension of the
> >>>>>>>> interface you
> >>>>>>>> >>> >>> propose here.
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> So I wonder if your interface could be named
> >>>>>>>> InternalTopicsPolicy?
> >>>>>>>> >>> >>> That's a bit more generic than InternalTopicNamingPolicy.
> >>>>>>>> That would
> >>>>>>>> >>> >>> also match the configuration setting,
> >>>>>>>> internal.topics.policy.class,
> >>>>>>>> >>> >>> you're proposing.
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> Thanks
> >>>>>>>> >>> >>>
> >>>>>>>> >>> >>> On Thu, Dec 3, 2020 at 10:15 PM Omnia Ibrahim <
> >>>>>>>> o.g.h.ibra...@gmail.com> wrote:
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > Hi Mickael,
> >>>>>>>> >>> >>> > Thanks for your feedback!
> >>>>>>>> >>> >>> > Regards your question about having more configurations,
> I
> >>>>>>>> considered adding
> >>>>>>>> >>> >>> > configuration per each topic however this meant adding
> >>>>>>>> more configurations
> >>>>>>>> >>> >>> > for MM2 which already have so many, also the more
> >>>>>>>> complicated and advanced
> >>>>>>>> >>> >>> > replication pattern you have between clusters the more
> >>>>>>>> configuration lines
> >>>>>>>> >>> >>> > will be added to your MM2 config which isn't going to be
> >>>>>>>> pretty if you
> >>>>>>>> >>> >>> > don't have the same topics names across your clusters.
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > Also, it added more complexity to the implementation as
> >>>>>>>> MM2 need to
> >>>>>>>> >>> >>> > 1- identify if a topic is checkpoints so we could list
> >>>>>>>> the checkpoints
> >>>>>>>> >>> >>> > topics in MirrorMaker 2 utils as one cluster could have
> X
> >>>>>>>> numbers
> >>>>>>>> >>> >>> > checkpoints topics if it's connected to X clusters, this
> >>>>>>>> is done right now
> >>>>>>>> >>> >>> > by listing any topic with suffix
> `.checkpoints.internal`.
> >>>>>>>> This could be
> >>>>>>>> >>> >>> > done by add `checkpoints.topic.suffix` config but this
> >>>>>>>> would make an
> >>>>>>>> >>> >>> > assumption that checkpoints will always have a suffix
> >>>>>>>> also having a suffix
> >>>>>>>> >>> >>> > means that we may need a separator as well to
> concatenate
> >>>>>>>> this suffix with
> >>>>>>>> >>> >>> > a prefix to identify source cluster name.
> >>>>>>>> >>> >>> > 2- identify if a topic is internal, so it shouldn't be
> >>>>>>>> replicated or track
> >>>>>>>> >>> >>> > checkpoints for it, right now this is relaying on
> >>>>>>>> disallow topics with
> >>>>>>>> >>> >>> > `.internal` suffix to be not replicated and not tracked
> >>>>>>>> in checkpoints but
> >>>>>>>> >>> >>> > with making topics configurable we need a way to define
> >>>>>>>> what is an internal
> >>>>>>>> >>> >>> > topic. This could be done by making using a list of all
> >>>>>>>> internal topics
> >>>>>>>> >>> >>> > have been entered to the configuration.
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > So having an interface seemed easier and also give more
> >>>>>>>> flexibility for
> >>>>>>>> >>> >>> > users to define their own topics name, define what is
> >>>>>>>> internal topic means,
> >>>>>>>> >>> >>> > how to find checkpoints topics and it will be one line
> >>>>>>>> config for each
> >>>>>>>> >>> >>> > herder, also it more consistence with MM2 code as MM2
> >>>>>>>> config have
> >>>>>>>> >>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as
> >>>>>>>> interface and they can
> >>>>>>>> >>> >>> > be overridden by providing a custom implementation for
> >>>>>>>> them or have some
> >>>>>>>> >>> >>> > config that change their default implementations.
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > Hope this answer your question. I also updated the KIP
> to
> >>>>>>>> add this to the
> >>>>>>>> >>> >>> > rejected solutions.
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
> >>>>>>>> mickael.mai...@gmail.com>
> >>>>>>>> >>> >>> > wrote:
> >>>>>>>> >>> >>> >
> >>>>>>>> >>> >>> > > Hi Omnia,
> >>>>>>>> >>> >>> > >
> >>>>>>>> >>> >>> > > Thanks for the KIP. Indeed being able to configure
> >>>>>>>> MM2's internal
> >>>>>>>> >>> >>> > > topic names would be a nice improvement.
> >>>>>>>> >>> >>> > >
> >>>>>>>> >>> >>> > > Looking at the KIP, I was surprised you propose an
> >>>>>>>> interface to allow
> >>>>>>>> >>> >>> > > users to specify names. Have you considered making
> >>>>>>>> names changeable
> >>>>>>>> >>> >>> > > via configurations? If so, we should definitely
> mention
> >>>>>>>> it in the
> >>>>>>>> >>> >>> > > rejected alternatives as it's the first method that
> >>>>>>>> comes to mind.
> >>>>>>>> >>> >>> > >
> >>>>>>>> >>> >>> > > I understand an interface gives a lot of flexibility
> >>>>>>>> but I'd expect
> >>>>>>>> >>> >>> > > topic names to be relatively simple and known in
> >>>>>>>> advance in most
> >>>>>>>> >>> >>> > > cases.
> >>>>>>>> >>> >>> > >
> >>>>>>>> >>> >>> > > I've not checked all use cases but something like
> below
> >>>>>>>> felt appropriate:
> >>>>>>>> >>> >>> > > clusters = primary,backup
> >>>>>>>> >>> >>> > >
> >>>>>>>> primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
> >>>>>>>> >>> >>> > >
> >>>>>>>> >>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
> >>>>>>>> o.g.h.ibra...@gmail.com>
> >>>>>>>> >>> >>> > > wrote:
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > > > Hey everyone,
> >>>>>>>> >>> >>> > > > Please take a look at KIP-690:
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > >
> >>>>>>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > > > Thanks for your feedback and support.
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > > > Omnia
> >>>>>>>> >>> >>> > > >
> >>>>>>>> >>> >>> > >
> >>>>>>>>
> >>>>>>>
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-735: Increase default consumer session timeout

[Gwen Shapira]

I love this improvement.

+1 (binding)

On Wed, Apr 28, 2021 at 10:46 AM Jason Gustafson 
wrote:

> Hi All,
>
> I'd like to start a vote on KIP-735:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout
> .
> +1
> from myself obviously
>
> -Jason
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Created] (KAFKA-12674) Client failover takes 2-4 seconds on clean broker shutdown

[Gwen Shapira (Jira)]

Gwen Shapira created KAFKA-12674:


 Summary: Client failover takes 2-4 seconds on clean broker shutdown
 Key: KAFKA-12674
 URL: https://issues.apache.org/jira/browse/KAFKA-12674
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Gwen Shapira


I ran two perf-producer clients against a 4-broker cluster running AWS, behind 
ELB. And then did a rolling restart, taking down one broker at a time using 
controlled shutdown.

I got the following errors on every broker shutdown:

{{[2021-04-16 01:31:39,846] WARN [Producer clientId=producer-1] Received 
invalid metadata error in produce request on partition perf-test-3 due to 
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests 
intended only for the leader, this error indicates that the broker is not the 
current leader. For requests intended for any replica, this error indicates 
that the broker is not a replica of the topic partition.. Going to request 
metadata update now (org.apache.kafka.clients.producer.internals.Sender)}}
 {{[2021-04-16 01:44:22,691] WARN [Producer clientId=producer-1] Connection to 
node 0 (b0-pkc-7yrmj.us-east-2.aws.confluent.cloud/3.140.123.43:9092) 
terminated during authentication. This may happen due to any of the following 
reasons: (1) Authentication failed due to invalid credentials with brokers 
older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow 
HTTPS traffic), (3) Transient network issue. 
(org.apache.kafka.clients.NetworkClient)}}

 The "Connection to node... terminated" error continued for 2-4 seconds. 

It looks like the metadata request was repeatedly sent to the node that just 
went down. I'd expect it to go on an existing connection to one of the live 
nodes.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] 2.8.0 RC2

[Gwen Shapira]

Historically, "beta" features were not added to the documentation. I
complained that MirrorMaker 2 was missing and I was pointed to the readme...

I can't say I love this, but it seems to be The Kafka Way of doing things.
So not a release blocker IMO.

On Thu, Apr 15, 2021 at 1:46 PM Israel Ekpo  wrote:

> I just checked the documentation page and could not find any reference to
> the KIP-631 configurations like process.roles, node.id and any of the
> controller.quorum.* configs
> https://kafka.apache.org/28/documentation.html
>
> Were these left out on purpose or should there be included in the 2.8
> documentation page as well?
>
> In the README.md file for KIP-500 (KRaft) mode, I see these configs:
> https://github.com/apache/kafka/blob/2.8/config/kraft/README.md
>
> However, I don't see them in the documentation page where most users will
> check first
> https://kafka.apache.org/28/documentation.html
>
> A lot of users looking to try out KRaft mode are going to be very confused
> on how to set this up without Zookeeper to check things out.
>
> When you have a moment, please share your thoughts regarding if these
> configs/settings need to be in this 2.8 documentation for this release.
>
> Thanks.
>
>
> Additional References:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-631%3A+The+Quorum-based+Kafka+Controller#KIP631:TheQuorumbasedKafkaController-Configurations
>
>
> https://github.com/apache/kafka/blob/2.8/raft/src/main/java/org/apache/kafka/raft/RaftConfig.java#L57
>
>
> https://github.com/apache/kafka/blob/2.8/core/src/main/scala/kafka/server/KafkaConfig.scala#L277
>
>
> https://github.com/apache/kafka/blob/2.8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
>
> On Thu, Apr 15, 2021 at 12:30 PM Bill Bejeck  wrote:
>
> > Hi John,
> >
> > Validation steps taken:
> >
> >1. Validated the signatures and checksums,
> >2. Built the project from the src tgz file, and ran the unit tests
> >3. Went through the quickstart and the Kafka Streams quickstart
> >4. Ran the quickstart for the KRaft module
> >   1. Created a topic
> >   2. Produced and consumed messages
> >   3. Ran the Metadata shell
> >
> > +1 (binding)
> >
> > Thanks for running this release!
> >
> > Bill
> >
> >
> > On Wed, Apr 14, 2021 at 4:03 PM John Roesler  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka
> > > 2.8.0.This is a major release that includes many new
> > > features, including:
> > >
> > >  * Early-access release of replacing Zookeeper with a
> > >self-managed quorum
> > >  * Add Describe Cluster API
> > >  * Support mutual TLS authentication on SASL_SSL listeners
> > >  * Ergonomic improvements to Streams TopologyTestDriver
> > >  * Logger API improvement to respect the hierarchy
> > >  * Request/response trace logs are now JSON-formatted
> > >  * New API to add and remove Streams threads while running
> > >  * New REST API to expose Connect task configurations
> > >  * Fixed the TimeWindowDeserializer to be able to
> > >deserialize
> > >keys outside of Streams (such as in the console consumer)
> > >  * Streams resilient improvement: new uncaught exception
> > >handler
> > >  * Streams resilience improvement: automatically recover
> > >from transient timeout exceptions
> > >
> > > Release notes for the 2.8.0 release:
> > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by 19 April 2021 ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the
> > > release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 2.8 branch) is the 2.8.0 tag:
> > > https://github.com/apache/kafka/releases/tag/2.8.0-rc2
> > >
> > > * Documentation:
> > > https://kafka.apache.org/28/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/28/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.8 branch:
> > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/
> > > (still flaky)
> > >
> > > System tests:
> > >
> > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/6
> > > 0/
> > >
> > >
> > >
> >
> http://confluent-kafka-2-8-system-test-results.s3-us-west-2.amazonaws.com/2021-04-14--001.1618407001--confluentinc--2.8--1b61272d45/report.html
> > >
> > > /**
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> >
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: New Jenkins job for master and release branches

[Gwen Shapira]

W00t! This is super awesome. Thank you so much!!!

On Sun, Apr 4, 2021 at 2:22 PM Ismael Juma  wrote:

> Hi all,
>
> As part of KAFKA-12614 <https://issues.apache.org/jira/browse/KAFKA-12614
> >,
> I have created a Jenkinsfile-based job for trunk, 2.8 and future release
> branches:
>
> https://ci-builds.apache.org/job/Kafka/job/kafka/
>
> This has several advantages (many of these are already the case for PR
> builds):
>
>1. The configuration is in source control.
>2. PR and branch build configuration share most of the logic.
>3. Unstable (tests failed) and unsuccessful (compile, checkstyle, etc.
>failed) builds are given different status and color (red vs amber).
>4. Reporting within a build is improved (each stage is shown as part of
>the build graph)
>5. Improved parallelism (each JDK+Scala combination is built in
> parallel)
>6. Release branches get better JDK version coverage (instead of only JDK
>8 as it used to be)
>7. Instead of creating a new job for each new release, we can adjust the
>configuration to allow the new release branch.
>
> There is currently an open PR <https://github.com/apache/kafka/pull/10473>
> to
> extend the Jenkinsfile with functionality desired for branch builds. Once
> that is merged and has been shown to work correctly, I will delete legacy
> Jenkins jobs like:
>
>- https://ci-builds.apache.org/job/Kafka/job/kafka-2.8-jdk8/
>- https://ci-builds.apache.org/job/Kafka/job/kafka-trunk-jdk11/
>
> Let me know if you have questions or comments.
>
> Ismael
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-720 Deprecate MirrorMaker v1

[Gwen Shapira]

Woot!
+1

On Sat, Mar 20, 2021, 10:41 AM Ryanne Dolan  wrote:

> Hey y'all, I'm starting the vote on KIP-720, which proposes to deprecate
> the original MirrorMaker in the upcoming 3.0 major release.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-720%3A+Deprecate+MirrorMaker+v1
>
> Thanks!
> Ryanne
>

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Gwen Shapira]

Hey Chris,

Thank you for the proposal. Few questions / comments:

0. It may be worth pointing out in the motivation section that
source-connector exactly once is more important than sink connector
exactly once, since many target systems will have unique key
constraint mechanisms that will prevent duplicates. Kafka does not
have any such constraints, so without this KIP-618, exactly once won't
be possible.
1. I am not clear why we need the worker to async copy offsets from
the connector-specific offset topic to a global offsets topic
2. While the reasoning you have for offset topic per connector appears
sound, it doesn't add up with the use of transactions in KafkaStreams.
My understanding is that KafkaStreams uses shared offsets topic with
all the other consumers, and (apparently) corrupting data and delays
by other tenants is a non-issue. Perhaps you can comment on how
Connect is different? In general much of the complexity in the KIP is
related to the separate offset topic, and I'm wondering if this can be
avoided. The migration use-case is interesting, but not related to
exactly-once and can be handled separately.
3. Allowing users to override the isolation level for the offset
reader, even when exactly-once is enabled, thereby disabling
exactly-once in a non-obvious way. I get that connect usually allows
users to shoot themselves in the foot, but are there any actual
benefits for allowing it in this case? Maybe it is better if we don't?
I don't find the argument that we always did this to be particularly
compelling.
4. It isn't stated explicitly, but it sounds like connect or source
connectors already have some batching mechanism, and that transaction
boundaries will match the batches (i.e. each batch will be a
transaction?). If so, worth being explicit.
5. "When a rebalance is triggered, before (re-)joining the cluster
group, all workers will preemptively stop all tasks of all source
connectors for which task configurations are present in the config
topic after the latest task count record" - how will this play with
the incremental rebalances? isn't this exactly the stop-the-world
rebalance we want to avoid?
6. "the worker will instantiate a transactional producer whose
transactional ID is, by default, the group ID of the cluster (but may
be overwritten by users using the transactional.id worker property)" -
If users change transactional.id property, zombie leaders won't get
fenced (since they will have an older and different transactional id)

Thanks,

Gwen

On Thu, May 21, 2020 at 11:21 PM Chris Egerton  wrote:
>
> Hi all,
>
> I know it's a busy time with the upcoming 2.6 release and I don't expect
> this to get a lot of traction until that's done, but I've published a KIP
> for allowing atomic commit of offsets and records for source connectors and
> would appreciate your feedback:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Atomic+commit+of+source+connector+records+and+offsets
>
> This feature should make it possible to implement source connectors with
> exactly-once delivery guarantees, and even allow a wide range of existing
> source connectors to provide exactly-once delivery guarantees with no
> changes required.
>
> Cheers,
>
> Chris



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Gwen Shapira]

I realize that I'm very very late to a pretty big party, but I was
just looking at the new code (super readable, btw), and noticed that
the inverse API for registerBroker() is called decomissionBroker().
It would be way clearer that we are undoing a registration is we
rename it to unregisterBroker() (and the AdminClient API that matches,
to avoid more confusion).

Any objections?

On Fri, Dec 18, 2020 at 10:08 AM Jun Rao  wrote:
>
> Hi, Colin,
>
> Thanks for the reply. The KIP looks good to me now. Thanks for your
> diligence.
>
> Jun
>
> On Thu, Dec 17, 2020 at 5:15 PM Colin McCabe  wrote:
>
> > On Thu, Dec 17, 2020, at 17:02, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the reply. Sounds good. One more comment.
> > >
> > > 231. Currently, we have sasl.mechanism.inter.broker.protocol for inter
> > > broker communication. It seems that we need a similar config for
> > specifying
> > > the sasl mechanism for the communication between the broker and the
> > > controller.
> > >
> >
> > Hi Jun,
> >
> > Yeah... sounds like we could use a new configuration key for this.  I
> > added sasl.mechanism.controller.protocol for this.
> >
> > regards,
> > Colin
> >
> > > Jun
> > >
> > > On Thu, Dec 17, 2020 at 2:29 PM Colin McCabe  wrote:
> > >
> > > > On Thu, Dec 17, 2020, at 10:19, Jun Rao wrote:
> > > > > Hi, Colin,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 211. Hmm, I still don't see a clear benefit of registering a broker
> > > > before
> > > > > recovery. It's possible for the recovery to take time. However,
> > during
> > > > the
> > > > > recovery mode, it seems the broker will still be in the fenced mode
> > and
> > > > > won't be able to do work for the clients. So, registering and
> > > > heartbeating
> > > > > early seems to only add unnecessary overhead. For your point on topic
> > > > > creation, I thought we now allow replicas to be created on
> > > > > unregistered brokers.
> > > > >
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks again for the reviews.
> > > >
> > > > Topics cannot be created on unregistered brokers.  They can be created
> > on
> > > > registered but fenced brokers.  So for that reason I think it makes
> > sense
> > > > to register as early as possible.
> > > >
> > > > > 230. Currently, we do have a ControllerId field in MetadataResponse.
> > In
> > > > the
> > > > > early discussion, I thought that we want to expose the controller for
> > > > > debugging purposes, but not used by the client library.
> > > > >
> > > >
> > > > The current plan is that we will expose the controller node ID, but the
> > > > controller will not be included in the list of nodes in the metadata
> > > > response.
> > > >
> > > > It's not really possible to include the controller in that list of
> > nodes
> > > > because the controller may not share the same set of listeners as the
> > > > broker.  So, for example, maybe the controller endpoint is using a
> > > > different type of security than the broker.  So while we could pass
> > back a
> > > > hostname and port, the client would have no way to connect since it
> > doesn't
> > > > know what security settings to use.
> > > >
> > > > regards,
> > > > Colin
> > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Dec 16, 2020 at 9:13 PM Colin McCabe 
> > wrote:
> > > > >
> > > > > > On Wed, Dec 16, 2020, at 18:13, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the reply. Just a couple of more comments.
> > > > > > >
> > > > > > > 211. Currently, the broker only registers itself in ZK after log
> > > > > > recovery.
> > > > > > > Is there any benefit to change that? As you mentioned, the broker
> > > > can't
> > > > > > do
> > > > > > > much before completing log recovery.
> > > > > > >
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Previously, it wasn't possible to register in ZK without
> > immediately
> > > > > > getting added to the MetadataResponse.  So I think that's the main
> > > > reason
> > > > > > why registration was delayed until after log recovery.  Since that
> > > > > > constraint doesn't exist any more, there seems to be no reason to
> > delay
> > > > > > registration.
> > > > > >
> > > > > > I think delaying registration would have some major downsides.  If
> > log
> > > > > > recovery takes a while, that's a longer window during which someone
> > > > else
> > > > > > could register a broker with the same ID.  Having parts of the
> > cluster
> > > > > > missing for a while gives up some of the benefit of separating
> > > > registration
> > > > > > from fencing.  For example, if a broker somehow gets unregistered
> > and
> > > > we
> > > > > > want to re-register it, but we have to wait for a 10 minute log
> > > > recovery to
> > > > > > do that, that could be a window during which topics can't be
> > created
> > > > that
> > > > > > need to be on that broker, and so forth.
> > > > > >
> > > > > > > 230. Regarding MetadataResponse, there is a slight awkwardness.
> > We
> > > > 

Re: Kafka Advisory Topic

[Gwen Shapira]

Agree that this sounds like a good idea.

Would be good to have a more formal proposal (i.e. a KIP) with the details.
I can think of about 100 different questions (will there be "levels"
like in logs, what type of events are in or out of scope, rate
limiting, data formats, etc).
I am also curious on whether the notifications are intended for
humans, automated processes or even the Kafka client applications
themselves. I hope the proposal can include a few example scenarios to
help us reason about the experience.

Knowlton, is this something you want to pick up?

Gwen

On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
 wrote:
>
> Hi,
>
> I am on the ActiveMQ PMC and I think this is a very good idea to have a way
> to do advisories/notifications/events (whatever you want to call it). In
> ActiveMQ classic you have advisories and in Artemis you have notifications.
> Having management messages that can be subscribed to in real time is
> actually a major feature that is missing from Kafka that many other brokers
> have.
>
> The idea here would be to publish notifications of different configurable
> events when something important happens so a consumer can listen in on
> things it cares about and be able to do something instead of having to poll
> the admin API. There are many events that happen in a broker that would be
> useful to be notified about. Events such as new connections to the cluster,
> new topics created or destroyed, consumer group creation, authorization
> errors, new leader election, etc. The list is pretty much endless.
>
> The metadata topic that will exist is probably not going to have all of
> this information so some other mechanism would be needed to handle
> publishing these messages to a specific management topic that would be
> useful for a consumer.
>
> Chris
>
>
> On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen 
> wrote:
>
> > Hey Knowles,
> >
> > in Kafka people normally use admin clients to get those metadata. I'm not
> > sure why you mentioned specifically that having a topic to manage these
> > information is useful, but a good news is that in KIP-500
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > >
> > we
> > are trying to deprecate Zookeeper and migrate to a self-managed metadata
> > topic quorum. At the time this feature is fully done, you should be able to
> > use consumers to pull the metadata log.
> >
> > Best,
> > Boyang
> >
> > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > katchiso...@gmail.com>
> > wrote:
> >
> > > Good afternoon all,
> > >
> > > In our Kafka clusters we have a need to know when certain activities are
> > > performed, mainly topics being created, but brokers coming up/down is
> > also
> > > useful. This would be akin to what ActiveMQ does via advisory messages (
> > > https://activemq.apache.org/advisory-message).
> > >
> > > Since there did not appear to be anything in the ecosystem currently, I
> > > wrote a standalone Java program that watches the various ZooKeeper
> > > locations that the Kafka broker writes to and deltas can tell us
> > > topic/broker actions etc... and writes to a kafka topic for downstream
> > > consumption.
> > >
> > > Ideally, we would rather have the broker handle this internally rather
> > > than yet another service stood up in our systems. I began digging through
> > > the broker source (my Scala is basically hello world level) and there
> > does
> > > not appear to be any mechanism in which this could be easily patched into
> > > the broker.
> > >
> > > Specifically, a producer or consumer acting upon an nonexistent topic or
> > a
> > > manual CreateTopic would trigger a Produce to this advisory topic and the
> > > KafkaApis framework would handle it like any other request. However, by
> > the
> > > time we are inside the getTopicMetadata call there doesn't seem to be a
> > > clean way to fire off another message that would make its way through
> > > KafkaApis. Perhaps another XManager type object is required?
> > >
> > > Looking for alternative ideas or guidance (or I missed something in the
> > > broker).
> > >
> > > Thank you.
> > >
> > > Knowles
> > >
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-697: Stricter parsing of addresses in configs

[Gwen Shapira]

Sorry for being late, I just saw this. I have a concern about the
compatibility story:

1. Do we know how common use of protocol is? While this is an
improvement, the benefits are a bit small (IMO) and if this will break
a large number of installations (or will make the upgrade to 3.0 more
painful and therefore less likely to happen) - maybe it is't worth it.
2. Should we add a PR to 2.8 that will print deprecation warnings if
protocol is used? This way people will at least know what is coming.

Gwen

On Mon, Jan 4, 2021 at 1:21 AM Tom Bentley  wrote:
>
> Hi,
>
> If there are no comments about this minor change in the next day or two I
> will start a vote.
>
> Kind regards,
>
> Tom
>
> On Wed, Dec 9, 2020 at 6:10 PM Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to start a discussion on a small KIP which proposes stricter
> > parsing of host:port addresses in various configs for Kafka 3.0:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-697%3A+Stricter+parsing+of+addresses+in+configs
> >
> > I'd be grateful for any feedback people may have.
> >
> > Kind regards,
> >
> > Tom
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-687: Automatic Reloading of Security Store

[Gwen Shapira]

+1, Thank you!

On Wed, Jan 6, 2021 at 12:27 PM Boyang Chen  wrote:
>
> Hey folks,
>
> just bumping up this thread and see if you have further comments.
>
> On Wed, Dec 16, 2020 at 3:25 PM Boyang Chen 
> wrote:
>
> > Hey there,
> >
> > I would like to start the voting for KIP-687:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-687
> > %3A+Automatic+Reloading+of+Security+Store to make the security store
> > reloading automated.
> >
> > Best,
> > Boyang
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Apache Kafka 2.8.0 release

[Gwen Shapira]

+1, Thank you!!!

On Wed, Jan 6, 2021 at 9:29 PM John Roesler  wrote:
>
> Hello All,
>
> I'd like to volunteer to be the release manager for our next
> feature release, 2.8.0. If there are no objections, I'll
> send out the release plan soon.
>
> Thanks,
> John Roesler
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-700: Add Describe Cluster API

[Gwen Shapira]

+1 (binding). Thank you for driving this, David.

On Wed, Jan 6, 2021 at 5:54 AM David Jacot  wrote:
>
> Hi all,
>
> I'd like to start the vote on KIP-700: Add Describe Cluster API. This KIP
> is here:
> https://cwiki.apache.org/confluence/x/jQ4mCg
>
> Please take a look and vote if you can.
>
> Best,
> David



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] Apache Kafka 2.7.0

[Gwen Shapira]

woooh!!!

Great job on the release Bill and everyone!

On Mon, Dec 21, 2020 at 8:01 AM Bill Bejeck  wrote:
>
> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.7.0
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.7.0/RELEASE_NOTES.html
>
> You can read about some of the more prominent changes in the Apache Kafka
> blog:
> https://blogs.apache.org/kafka/entry/what-s-new-in-apache4
>
> You can download the source and binary release (Scala 2.12, 2.13) from:
> https://kafka.apache.org/downloads#2.7.0
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 117 contributors to this release!
>
> A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Adem Efe Gencer,
> albert02lowis, Alex Diachenko, Andras Katona, Andre Araujo, Andrew Choi,
> Andrew Egelhofer, Andy Coates, Ankit Kumar, Anna Povzner, Antony Stubbs,
> Arjun Satish, Ashish Roy, Auston, Badai Aqrandista, Benoit Maggi, bill,
> Bill Bejeck, Bob Barrett, Boyang Chen, Brian Byrne, Bruno Cadonna, Can
> Cecen, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> David Arthur, David Jacot, David Mao, Dhruvil Shah, Dima Reznik, Edoardo
> Comar, Ego, Evelyn Bayes, feyman2016, Gal Margalit, gnkoshelev, Gokul
> Srinivas, Gonzalo Muñoz, Greg Harris, Guozhang Wang, high.lee, huangyiming,
> huxi, Igor Soarez, Ismael Juma, Ivan Yurchenko, Jason Gustafson, Jeff Kim,
> jeff kim, Jesse Gorzinski, jiameixie, Jim Galasyn, JoelWee, John Roesler,
> John Thomas, Jorge Esteban Quilcate Otoya, Julien Jean Paul Sirocchi,
> Justine Olshan, khairy, Konstantine Karantasis, Kowshik Prakasam, leah, Lee
> Dongjin, Leonard Ge, Levani Kokhreidze, Lucas Bradstreet, Lucent-Wong, Luke
> Chen, Mandar Tillu, manijndl7, Manikumar Reddy, Mario Molina, Matthias J.
> Sax, Micah Paul Ramos, Michael Bingham, Mickael Maison, Navina Ramesh,
> Nikhil Bhatia, Nikolay, Nikolay Izhikov, Ning Zhang, Nitesh Mor, Noa
> Resare, Rajini Sivaram, Raman Verma, Randall Hauch, Rens Groothuijsen,
> Richard Fussenegger, Rob Meng, Rohan, Ron Dagostino, Sanjana Kaundinya,
> Sasaki Toru, sbellapu, serjchebotarev, Shaik Zakir Hussain, Shailesh
> Panwar, Sharath Bhat, showuon, Stanislav Kozlovski, Thorsten Hake, Tom
> Bentley, tswstarplanet, vamossagar12, Vikas Singh, vinoth chandar, Vito
> Jeng, voffcheg109, xakassi, Xavier Léauté, Yuriy Badalyantc, Zach Zhang
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> Bill Bejeck



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-700: Add Describe Cluster API

[Gwen Shapira]

On Fri, Dec 18, 2020, 12:48 PM Colin McCabe  wrote:

> On Fri, Dec 18, 2020, at 08:32, Gwen Shapira wrote:
> > Agree. Once we have the basic API we can decide what belongs and what
> > doesn't. I remember requests for broker status, version, and various
> > other things. Considering all those options together will be too much
> > at this point.
> >
>
> Hi Gwen,
>
> I agree that there are a lot of potential improvements, and we want to
> avoid putting too much in this KIP.  But I still think it would be helpful
> to have one or two new user-visible things in this KIP, just to demonstrate
> why we need a new RPC.  Or if that's too much, can we identify a few things
> in a "future work" section?  I think it adds a lot to the motivation for
> this KIP.  After all, if you don't think we're going to add anything more
> to describeCluster, there is not a strong case for a new RPC.
>

I think the complexity of MetadataResponse alone justifies a cleaner API,
but if adding JMX port (or broker version, or broker status) is a must-have
in your mind, I have no objection - I have few use-cases for the
broker-status response.


> > One thing that may be worth considering is whether you want to include
> > not just the controller ID but also its epoch. This will allow
> > resolving cases where brokers respond with outdated information. I
> > haven't thought it through, so maybe it doesn't make sense here, but I
> > was wondering if this was considered.
>
> I think we should hold off on adding more epochs right now until we have a
> more general solution.  With KIP-500 we can potentially put an epoch on the
> whole metadata response, which would allow us to have read-after-write
> consistency for metadata-- something people have wanted for a while.
>

Oh, this is wonderful news! We can add the metadata epoch separately (I did
not suggest a new epoch, I suggested publishing the existing broker and
controller epochs in the response, but we can add this later... as I just
argued above).


>
> best,
> Colin
>
>
> >
> > On Fri, Dec 18, 2020 at 4:51 AM David Jacot  wrote:
> > >
> > > Hi Ryan,
> > >
> > > Thanks for your feedback. That's an interesting idea but that raises
> more
> > > questions. At the moment, `describeCluster` only requires to be
> > > authenticated
> > > (if authentication is enabled) to get the basic information about the
> > > cluster.
> > > If we add the JMX port, is it something that we would provide without
> > > requiring
> > > more permissions? I am not sure about this.
> > >
> > > I lean towards keeping this KIP focused on introducing the new API and
> to
> > > add new information with separate KIPs. There might be more information
> > > that we want to add as part of KIP-500.
> > >
> > > I will be happy to hear what other members of the community think about
> > > this.
> > >
> > > Best,
> > > David
> > >
> > > On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe 
> wrote:
> > >
> > > > Hi David,
> > > >
> > > > This seems reasonable.  It would be nice to have an API specifically
> for
> > > > describeCluster, so that we could extend this API without adding more
> > > > fields to the already large MetadataRequest.
> > > >
> > > > As you mention in the KIP, KIP-700 would allow us to deprecate
> > > > MetadataRequest#ClusterAuthorizedOperations.  So it seems like this
> KIP
> > > > should specify a new version of MetadataRequest where the related
> fields
> > > > are absent, right?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > > > > Hi all,
> > > > >
> > > > > I'd like to propose a small KIP:
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> > > > >
> > > > > Please let me know what you think.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > >
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: [DISCUSS] KIP-700: Add Describe Cluster API

[Gwen Shapira]

Agree. Once we have the basic API we can decide what belongs and what
doesn't. I remember requests for broker status, version, and various
other things. Considering all those options together will be too much
at this point.

One thing that may be worth considering is whether you want to include
not just the controller ID but also its epoch. This will allow
resolving cases where brokers respond with outdated information. I
haven't thought it through, so maybe it doesn't make sense here, but I
was wondering if this was considered.

On Fri, Dec 18, 2020 at 4:51 AM David Jacot  wrote:
>
> Hi Ryan,
>
> Thanks for your feedback. That's an interesting idea but that raises more
> questions. At the moment, `describeCluster` only requires to be
> authenticated
> (if authentication is enabled) to get the basic information about the
> cluster.
> If we add the JMX port, is it something that we would provide without
> requiring
> more permissions? I am not sure about this.
>
> I lean towards keeping this KIP focused on introducing the new API and to
> add new information with separate KIPs. There might be more information
> that we want to add as part of KIP-500.
>
> I will be happy to hear what other members of the community think about
> this.
>
> Best,
> David
>
> On Thu, Dec 17, 2020 at 5:57 AM Colin McCabe  wrote:
>
> > Hi David,
> >
> > This seems reasonable.  It would be nice to have an API specifically for
> > describeCluster, so that we could extend this API without adding more
> > fields to the already large MetadataRequest.
> >
> > As you mention in the KIP, KIP-700 would allow us to deprecate
> > MetadataRequest#ClusterAuthorizedOperations.  So it seems like this KIP
> > should specify a new version of MetadataRequest where the related fields
> > are absent, right?
> >
> > best,
> > Colin
> >
> >
> > On Mon, Dec 14, 2020, at 08:10, David Jacot wrote:
> > > Hi all,
> > >
> > > I'd like to propose a small KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-700%3A+Add+Describe+Cluster+API
> > >
> > > Please let me know what you think.
> > >
> > > Best,
> > > David
> > >
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.7.0 RC6

[Gwen Shapira]

+1 (binding)

Thank you for the release, Bill!
I validated signatures, built from source package and ran the perf
producer/consumer to validate.

On Wed, Dec 16, 2020 at 6:53 AM Bill Bejeck  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the seventh candidate for release of Apache Kafka 2.7.0.
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> *** Please download, test and vote by Monday, December 21, 12 PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc6
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-2.7-jdk8/detail/kafka-2.7-jdk8/81/
>
> Thanks,
> Bill



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Resolved] (KAFKA-7819) Trogdor - Improve RoundTripWorker

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-7819.
-
Fix Version/s: 2.2.0
   Resolution: Fixed

Closing since I noticed the PR was merged.

> Trogdor - Improve RoundTripWorker
> -
>
> Key: KAFKA-7819
> URL: https://issues.apache.org/jira/browse/KAFKA-7819
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Stanislav Kozlovski
>Assignee: Stanislav Kozlovski
>Priority: Minor
> Fix For: 2.2.0
>
>
> Trogdor's RoundTripWorker task has a couple of shortcomings:
>  * Consumer GroupID is hardcoded and consumers use `KafkaConsumer#assign()`: 
> [https://github.com/apache/kafka/blob/12947f4f944955240fd14ce8b75fab5464ea6808/tools/src/main/java/org/apache/kafka/trogdor/workload/RoundTripWorker.java#L314]
> Leaving you unable to run two separate instances of this worker on the same 
> partition in the same cluster, as the consumers would overwrite each other's 
> commits. It's probably better to add the task ID to the consumer group
>  * the task spec's `maxMessages` [is an 
> integer|https://github.com/apache/kafka/blob/12947f4f944955240fd14ce8b75fab5464ea6808/tools/src/main/java/org/apache/kafka/trogdor/workload/RoundTripWorkloadSpec.java#L39],
>  leaving you unable to schedule long-winded tasks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-692: Make AdminClient value object constructors public

[Gwen Shapira]

Sounds like a good idea :)

At some point we may want to switch to builders - it gives you more
flexibility in how you construct all those objects. But those two
changes are unrelated.

On Thu, Dec 3, 2020 at 9:30 AM Noa Resare  wrote:
>
> Hi there,
>
> I finally got around to write up the KIP for a change I have wanted for some 
> time:
>
> KIP-692: Make AdminClient value object constructors public 
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-692:+Make+AdminClient+value+object+constructors+public>
>
> In essence: Let us encourage better patterns for testing by making it easier 
> to mock AdminService.
>
> I am looking forward to hearing your thoughts on this.
>
> Cheers
> noa



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Gwen Shapira]

+1 (binding). Awesome suggestion, Cheng.

On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  wrote:
>
> Hi all,
>
> I’m proposing a new KIP for enabling the strongest delivery guarantee by 
> default. Today Kafka support EOS and N-1 concurrent failure tolerance but the 
> default settings haven’t bring them out of the box. The proposal changes 
> include the producer defaults change to `ack=all` and 
> `enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE 
> will be deprecated. If a producer has WRITE permission to any topic, it will 
> be able to request a producer id and perform idempotent produce.
>
> KIP here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>  
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default>
>
> Please vote in this mail thread.
>
> Thanks
>
> - Cheng Tan



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

[Gwen Shapira]

Agree with Igor. IIRC, we also encountered cases where filewatch was
not triggered as expected. An interval will give us a better
worse-case scenario that is easily controlled by the Kafka admin.

Gwen

On Sun, Dec 6, 2020 at 8:17 AM Igor Soarez  wrote:
>
>
> > > The proposed change relies on a file watch, why not also have a polling
> > > interval to check the file for changes?
> > >
> > > The periodical check could work, the slight downside is that we need
> > additional configurations to schedule the interval. Do you think the
> > file-watch approach has any extra overhead than the interval based solution?
>
> I don't think so. The reason I'm asking this is the KIP currently includes:
>
>   "When the file watch does not work for unknown reason, user could still try 
> to change the store path in an explicit AlterConfig call in the worst case."
>
> Having the interval in addition to the file watch could result in a better 
> worst case scenario.
> I understand it would require introducing at least one new configuration for 
> the interval, so maybe this doesn't have to solved in this KIP.
>
> --
> Igor
>
> On Fri, Dec 4, 2020, at 5:14 PM, Boyang Chen wrote:
> > Hey Igor, thanks for the feedback.
> >
> > On Fri, Dec 4, 2020 at 5:24 AM Igor Soarez  wrote:
> >
> > > Hi Boyang,
> > >
> >
> >
> > > What happens if the file is changed into an invalid store? Does the
> > > previous store stay in use?
> > >
> > > If the reload fails, the previous store should be effective. I will state
> > that in the KIP.
> >
> >
> > > Thanks,
> > >
> > > --
> > > Igor
> > >
> > > On Fri, Dec 4, 2020, at 1:28 AM, Boyang Chen wrote:
> > > > Hey there,
> > > >
> > > > I would like to start the discussion thread for KIP-687:
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
> > > >
> > > > This KIP is trying to deprecate the AlterConfigs API support of updating
> > > > the security store by reloading path in-place, and replace with a
> > > > file-watch mechanism inside the broker. Let me know what you think.
> > > >
> > > > Best,
> > > > Boyang
> > > >
> > >
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.7.0 RC3

[Gwen Shapira]

+1 (binding) - assuming we get a successful Jenkins build for the branch.

I built from sources, tested resulting binaries locally, verified
signature and checksums.

Thank you for the release, Bill.

On Wed, Nov 25, 2020 at 7:31 AM Bill Bejeck  wrote:
>
> This is the fourth candidate for the release of Apache Kafka 2.7.0.
>
> This is a major release that includes many new features, including:
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 84
> bug fixes.
>
> Release notes for the 2.7.0 release:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday, December 2, 12PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc3
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests: (link to follow)
> System tests: (link to follow)
>
> Thanks,
> Bill



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config

[Gwen Shapira]

Harsha,

I understand the concern, but the solution of using this flag is a bit weird.

First, I can cause similar damage by setting retention.ms=1 on any
topic. We also have APIs to delete data from topics directly. All
without a "gating" flag. Because as a system we rely on access
controls to make sure unauthorized users can't cause excessive damage.

Second, I can think of no other system with a global flag for
enabling/disabling deletes - no OS does that (deleting files from
Linux is also a potential disaster) and no database. The fact that
Kafka is rather unique in this makes the flag and its default seem
like a trap for new users and an endless source of confusion.

Keeping a feature flag around for deletion seems to me to have more
downsides than upsides. If we want to add soft deletes to Kafka, we
can discuss that proposal - at least it is something users will be
familiar with.

On Wed, Nov 25, 2020 at 11:54 AM Harsha Ch  wrote:
>
> Hi Gwen,
>  Agree with you ACLs would be a better option. But not
> all installations will be able to have secure setup and authorizer.
> In a hosted service where they may not be ACLs or security setup this flag
> helps prevent accidental topic deletion.
> This config acted as a good gatekeeper to prevent accidents in deletion as
> there is no confirmation or there is time delay in execution of the
> deletion of a topic or a soft delete etc...
> It's pretty much a catastrophic event in case of accidental deletion. We
> also need to consider that not all installations will be secure and we need
> to
> provide a simpler way to gate keep certain functionalities such as delete
> topic.
>
> Thanks,
> Harsha
>
>
> On Wed, Nov 25, 2020 at 10:25 AM Gwen Shapira  wrote:
>
> > Thanks for bringing this up!
> >
> > I disagree that " there is a valid use-case to keep the flag
> > "delete.topic.enable" as "false" in server.properties configs, so that
> > someone doesn't accidentally delete a topic,".
> > The flag was added and set to "false" back in the days when topic
> > deletion was not a stable feature, and enabling it meant a risk of
> > encountering serious bugs. Things changed in the last 4-5 years:
> > - Topic deletion is now a stable and reliable feature. We have it
> > enabled by default in Confluent Cloud and over a few years and
> > hundreds of clusters, there was maybe one issue related to deletion of
> > large number of partitions, and that was fixed.
> > - We have authorization and ACLs, so the permissions to delete topics
> > can be granted on an individual basis. This is the correct way to
> > handle concerns about someone accidentally deleting data valuable
> > data. Note that ACLs are dynamic.
> >
> > IMO, the best solution is to use the upcoming major release (3.0) to
> > enable topic deletion by default and remove the flag entirely.
> >
> > Gwen
> >
> > On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal 
> > wrote:
> > >
> > > Hi,
> > >
> > > I've created KIP-688 to enable support for dynamic update of
> > > delete.topic.enable config.
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks
> > > --
> > > Prateek Agarwal
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config

[Gwen Shapira]

Thanks for bringing this up!

I disagree that " there is a valid use-case to keep the flag
"delete.topic.enable" as "false" in server.properties configs, so that
someone doesn't accidentally delete a topic,".
The flag was added and set to "false" back in the days when topic
deletion was not a stable feature, and enabling it meant a risk of
encountering serious bugs. Things changed in the last 4-5 years:
- Topic deletion is now a stable and reliable feature. We have it
enabled by default in Confluent Cloud and over a few years and
hundreds of clusters, there was maybe one issue related to deletion of
large number of partitions, and that was fixed.
- We have authorization and ACLs, so the permissions to delete topics
can be granted on an individual basis. This is the correct way to
handle concerns about someone accidentally deleting data valuable
data. Note that ACLs are dynamic.

IMO, the best solution is to use the upcoming major release (3.0) to
enable topic deletion by default and remove the flag entirely.

Gwen

On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal  wrote:
>
> Hi,
>
> I've created KIP-688 to enable support for dynamic update of
> delete.topic.enable config.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
>
> Please take a look and let me know if you have any feedback.
>
> Thanks
> --
> Prateek Agarwal



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

[Gwen Shapira]

Michael,

I talked a bit to Colin and Ismael offline and got some clarity about
KIP-500. Basically, replica placement requires an entire metadata map
- all the placements of all replicas. Since one of the goals of
KIP-500 is to go to 1M or even 10M partitions on the cluster, this
will be a rather large map. Since brokers normally don't need to know
all the placements (they just need to know about the subset of
replicas that they lead or follow), the idea is to keep this map on
the controller only. Which means that the replica placement plugin
will ideally be on the controller too. This also has a nice side
benefit - since we will be able to run the controller quorum on a
separate set of machines, we'll be able to replace the replica
placement plugin by updating 3-5 controller nodes, not the entire
cluster.

Gwen

On Thu, Nov 19, 2020 at 4:54 AM Mickael Maison  wrote:
>
> Hi David,
>
> I think using updateClusterMetadata() like in ClientQuotaCallback is a
> great idea. I've updated the KIP accordingly.
>
> As mentioned, computing assignments for the full batch of
> topics/partitions was considered but it made the logic in AdminManager
> really complicated. For the initial use cases this KIP is targetting,
> it felt simpler and acceptable to compute assignments one topic at a
> time.
>
> Cluster is already used in other APIs, like ClientQuotaCallback so I
> think it makes sense to reuse it here.
>
> I'm not fully up to date with the latest advances on KIP-500, but like
> Gwen, I'm not sure we'd want to move that logic into the Controller.
> This KIP is keeping the metadata creation in AdminManager and just
> making the logic pluggable.
>
> Thanks
>
> On Mon, Nov 16, 2020 at 3:56 PM Gwen Shapira  wrote:
> >
> > Why would the replica placement logic run in the controller rather than in
> > the AdminManager?
> >
> > My understanding, and correct me if I got it wrong, is that we are aiming
> > at better separation of concerns. The controller job will be managing
> > consensus and consistency of metadata, but creating the metadata itself
> > will be in the AdminManager.
> >
> > On Mon, Nov 16, 2020, 5:31 AM David Jacot  wrote:
> >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP. It is an interesting one.
> > >
> > > I imagine that custom assignors may use a rather complex model of the
> > > cluster in order
> > > to be able to allocate partitions in smarter ways. For instance, one may
> > > use the distribution
> > > of leaders in the cluster to allocate the new leaders. With the current
> > > interface, that
> > > means computing the distribution based on the Cluster received for every
> > > assignment
> > > request. That could become pretty inefficient in clusters with a large
> > > number of nodes
> > > and/or partitions. That could become even worse if the model is more
> > > complicated.
> > >
> > > I wonder if you have thought about this or experienced this with your
> > > prototype?
> > >
> > > Have you considered going with an approach à la ClientQuotaCallback where
> > > the plugin
> > > is updated when the Cluster has changed? That would allow to keep an
> > > internal model
> > > ready. Another way would be to use batching as suggested as it would allow
> > > to amortize
> > > the cost of building a model for the current request/user.
> > >
> > > I also wonder if using Cluster is a good idea here. With KIP-500, I can
> > > imagine that this
> > > plugin will run in the controller directly instead of running in the
> > > AdminManager as today.
> > > In this case, we could obviously continue to build that Cluster object but
> > > we may have
> > > better ways. Therefore, I wonder if having an interface to represent the
> > > cluster may be
> > > better from an evolution perspective. Have you considered this?
> > >
> > > Best,
> > > David
> > >
> > > On Mon, Nov 16, 2020 at 12:10 PM Mickael Maison 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > If I don't see additional feedback in the next few days, I'll start a
> > > vote.
> > > > Thanks
> > > >
> > > > On Thu, Nov 5, 2020 at 6:29 PM Mickael Maison 
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I've updated the KIP to reflect the latest discussions.
> > > > >
> > > > > Tom,
> > > > > 2) Updated
> > > >

Re: [VOTE] KIP-681: Rename master key in delegation token feature

[Gwen Shapira]

+1
Thank you for catching this.


On Mon, Nov 16, 2020, 3:22 AM Tom Bentley  wrote:

> Hi,
>
> I'd like to start a vote on KIP-681 which proposes to rename a broker
> config to use a more racially neutral term:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-681
> %3A+Rename+master+key+in+delegation+token+feature
>
> Kind regards,
>
> Tom
>

Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor

[Gwen Shapira]

Why would the replica placement logic run in the controller rather than in
the AdminManager?

My understanding, and correct me if I got it wrong, is that we are aiming
at better separation of concerns. The controller job will be managing
consensus and consistency of metadata, but creating the metadata itself
will be in the AdminManager.

On Mon, Nov 16, 2020, 5:31 AM David Jacot  wrote:

> Hi Mickael,
>
> Thanks for the KIP. It is an interesting one.
>
> I imagine that custom assignors may use a rather complex model of the
> cluster in order
> to be able to allocate partitions in smarter ways. For instance, one may
> use the distribution
> of leaders in the cluster to allocate the new leaders. With the current
> interface, that
> means computing the distribution based on the Cluster received for every
> assignment
> request. That could become pretty inefficient in clusters with a large
> number of nodes
> and/or partitions. That could become even worse if the model is more
> complicated.
>
> I wonder if you have thought about this or experienced this with your
> prototype?
>
> Have you considered going with an approach à la ClientQuotaCallback where
> the plugin
> is updated when the Cluster has changed? That would allow to keep an
> internal model
> ready. Another way would be to use batching as suggested as it would allow
> to amortize
> the cost of building a model for the current request/user.
>
> I also wonder if using Cluster is a good idea here. With KIP-500, I can
> imagine that this
> plugin will run in the controller directly instead of running in the
> AdminManager as today.
> In this case, we could obviously continue to build that Cluster object but
> we may have
> better ways. Therefore, I wonder if having an interface to represent the
> cluster may be
> better from an evolution perspective. Have you considered this?
>
> Best,
> David
>
> On Mon, Nov 16, 2020 at 12:10 PM Mickael Maison 
> wrote:
>
> > Hi all,
> >
> > If I don't see additional feedback in the next few days, I'll start a
> vote.
> > Thanks
> >
> > On Thu, Nov 5, 2020 at 6:29 PM Mickael Maison 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I've updated the KIP to reflect the latest discussions.
> > >
> > > Tom,
> > > 2) Updated
> > > 4) I decided against switching to a "batch interface" and added the
> > > reasons in the Rejected Alternatives section
> > >
> > > Please take a look and let me know if you have any feedback.
> > > Thanks
> > >
> > > On Tue, Oct 6, 2020 at 9:43 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> > > >
> > > > Hi Efe,
> > > >
> > > > Thanks for the feedback.
> > > > We also need to assign replicas when adding partitions to an existing
> > > > topic. This is why I choose to use a list of partition ids. Otherwise
> > > > we'd need the number of partitions and the starting partition id.
> > > >
> > > > Let me know if you have more questions
> > > >
> > > > On Tue, Oct 6, 2020 at 2:16 AM Efe Gencer
> 
> > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP!
> > > > > A call to an external system, e.g. Cruise Control, in the
> > implementation of the provided interface can indeed help with the initial
> > assignment of partitions.
> > > > >
> > > > > I am curious why the proposed
> > `ReplicaAssignor#assignReplicasToBrokers` receives a list of partition
> ids
> > as opposed to the number of partitions to create the topic with?
> > > > >
> > > > > Would you clarify if this API is expected to be used (1) only for
> > new topics or (2) also for existing topics?
> > > > >
> > > > > Best,
> > > > > Efe
> > > > > 
> > > > > From: Mickael Maison 
> > > > > Sent: Thursday, October 1, 2020 9:43 AM
> > > > > To: dev 
> > > > > Subject: Re: [DISCUSS] KIP-660: Pluggable ReplicaAssignor
> > > > >
> > > > > Thanks Tom for the feedback!
> > > > >
> > > > > 1. If the data returned by the ReplicaAssignor implementation does
> > not
> > > > > match that was requested, we'll also throw a
> ReplicaAssignorException
> > > > >
> > > > > 2. Good point, I'll update the KIP
> > > > >
> > > > > 3. The KIP mentions an error code associated with
> > > > > ReplicaAssignorException: REPLICA_ASSIGNOR_FAILED
> > > > >
> > > > > 4. (I'm naming your last question 4.) I spent some time looking at
> > it.
> > > > > Initially I wanted to follow the model from the topic policies. But
> > as
> > > > > you said, computing assignments for the whole batch may be more
> > > > > desirable and also avoids incrementally updating the cluster state.
> > > > > The logic in AdminManager is very much centered around doing 1
> topic
> > > > > at a time but as far as I can tell we should be able to update it
> to
> > > > > compute assignments for the whole batch.
> > > > >
> > > > > I'll play a bit more with 4. and I'll update the KIP in the next
> few
> > days
> > > > >
> > > > > On Mon, Sep 21, 2020 at 10:29 AM Tom Bentley 
> > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > 

Re: [DISCUSS] KIP-685: Loosen permission for listing reassignments

[Gwen Shapira]

Makes sense to me. Thanks David.

On Tue, Nov 10, 2020 at 6:57 AM David Jacot  wrote:
>
> Hi all,
>
> I have submitted a small KIP to address KAFKA-10216. This KIP proposes to
> change
> the permission of the ListPartitionReassignments API to Topic Describe
> instead of Cluster Describe.
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-685%3A+Loosen+permission+for+listing+reassignments
>
> Please, let me know what you think.
>
> Best,
> David



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Checkstyle Import Order

[Gwen Shapira]

I don't have any specific preference on the style. But I am glad you
are bringing it up. Every other project I worked on had a specific
import style, and the random import changes in PRs are pretty
annoying.

On Wed, Oct 14, 2020 at 10:36 PM Dongjin Lee  wrote:
>
> Hello. I hope to open a discussion about the import order in Java code.
>
> As Nikolay stated recently[^1], Kafka uses a relatively strict code style
> for Java code. However, it misses any rule on import order. For this
> reason, the code formatting settings of every local dev environment are
> different from person to person, resulting in the countless meaningless
> import order changes in the PR.
>
> For example, in `NamedCache.java` in the streams module, the `java.*`
> imports are split into two chunks, embracing the other imports between
> them. So, I propose to define an import order to prevent these kinds of
> cases in the future.
>
> To define the import order, we have to regard the following three
> orthogonal issues beforehand:
>
> a. How to group the type imports?
> b. Whether to sort the imports alphabetically?
> c. Where to place static imports: above the type imports, or below them.
>
> Since b and c seem relatively straightforward (sort the imports
> alphabetically and place the static imports below the type imports), I hope
> to focus the available alternatives on the problem a.
>
> I evaluated the following alternatives and checked how many files are get
> effected for each case. (based on commit 1457cc652) And here are the
> results:
>
> *1. kafka, org.apache.kafka, *, javax, java (5 groups): 1222 files.*
>
> ```
> 
>value="kafka,/^org\.apache\.kafka.*$/,*,javax,java"/>
>   
>   
>   
>   
> 
> ```
>
> *2. (kafka|org.apache.kafka), *, javax? (3 groups): 968 files.*
>
> ```
> 
>   
>   
>   
>   
>   
> 
> ```
>
> *3. (kafka|org.apache.kafka), *, javax, java (4 groups): 533 files.*
>
> ```
> 
>value="(kafka|org\.apache\.kafka),*,javax,java"/>
>   
>   
>   
>   
> 
> ```
>
> *4. *, javax? (2 groups): 707 files.*
>
> ```
> 
>   
>   
>   
>   
>   
> 
> ```
>
> *5. javax?, * (2 groups): 1822 files.*
>
> ```
> 
>   
>   
>   
>   
>   
> 
> ```
>
> *6. java, javax, * (3 groups): 1809 files.*
>
> ```
> 
>   
>   
>   
>   
>   
> 
> ```
>
> I hope to get some feedback on this issue here.
>
> For the WIP PR, please refer here: https://github.com/apache/kafka/pull/8404
>
> Best,
> Dongjin
>
> [^1]:
> https://lists.apache.org/thread.html/r2bbee24b8a459842a0fc840c6e40958e7158d29f3f2d6c0d223be80b%40%3Cdev.kafka.apache.org%3E
> [^2]:
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/NamedCache.java
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
>
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr
> <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

[Gwen Shapira]

Congratulations, Sophie!

On Mon, Oct 19, 2020 at 9:41 AM Matthias J. Sax  wrote:
>
> Hi all,
>
> I am excited to announce that A. Sophie Blee-Goldman has accepted her
> invitation to become an Apache Kafka committer.
>
> Sophie is actively contributing to Kafka since Feb 2019 and has
> accumulated 140 commits. She authored 4 KIPs in the lead
>
>  - KIP-453: Add close() method to RocksDBConfigSetter
>  - KIP-445: In-memory Session Store
>  - KIP-428: Add in-memory window store
>  - KIP-613: Add end-to-end latency metrics to Streams
>
> and helped to implement two critical KIPs, 429 (incremental rebalancing)
> and 441 (smooth auto-scaling; not just implementation but also design).
>
> In addition, she participates in basically every Kafka Streams related
> KIP discussion, reviewed 142 PRs, and is active on the user mailing list.
>
> Thanks for all the contributions, Sophie!
>
>
> Please join me to congratulate her!
>  -Matthias
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-661: Expose task configurations in Connect REST API

[Gwen Shapira]

I definitely needed this capability a few times before. Thank you.

+1 (binding)

On Thu, Sep 24, 2020 at 7:54 AM Mickael Maison  wrote:
>
> Hi,
>
> I'd like to start a vote on KIP-661:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API
>
> Thanks



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[ANNOUNCE] New committer: David Jacot

[Gwen Shapira]

The PMC for Apache Kafka has invited David Jacot as a committer, and
we are excited to say that he accepted!

David Jacot has been contributing to Apache Kafka since July 2015 (!)
and has been very active since August 2019. He contributed several
notable KIPs:

KIP-511: Collect and Expose Client Name and Version in Brokers
KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies:
KIP-570: Add leader epoch in StopReplicaReques
KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations
KIP-496 Added an API for the deletion of consumer offsets

In addition, David Jacot reviewed many community contributions and
showed great technical and architectural taste. Great reviews are hard
and often thankless work - but this is what makes Kafka a great
product and helps us grow our community.

Thanks for all the contributions, David! Looking forward to more
collaboration in the Apache Kafka community.

-- 
Gwen Shapira

Re: [VOTE] KIP-676: Respect the logging hierarchy

[Gwen Shapira]

+1 (binding)

Makes sense, thank you.

On Fri, Oct 9, 2020, 2:50 AM Tom Bentley  wrote:

> Hi all,
>
> KIP-676 is pretty trivial and the comments on the discussion thread seem to
> be favourable, so I'd like to start a vote on it.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-676%3A+Respect+logging+hierarchy
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-676%3A+Respect+logging+hierarchy?moved=true
> >
>
> Please take a look if you have time.
>
> Many thanks,
>
> Tom
>

Re: New release branch 2.7

[Gwen Shapira]

Hey Bill!

Thank you for driving this release!

We are only now starting to work on KIP-629, so we missed feature
freeze. However the KIP is a collection of small changes (either a
single method or literally just renames). We are wondering how you
feel about us cherry-picking the PRs into 2.7 branch?
For reference: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase

Gwen

On Thu, Oct 8, 2020 at 5:27 AM Bill Bejeck  wrote:
>
> Hello Kafka developers and friends,
>
> As promised, we now have a release branch for the 2.7 release (with 2.7.0
> as the version).
> Trunk has been bumped to 2.8.0-SNAPSHOT.
>
> I'll be going over the JIRAs to move every non-blocker from this release to
> the next release.
>
> From this point, most changes should go to trunk.
> *Blockers (existing and new that we discover while testing the release)
> will be double-committed. *Please discuss with your reviewer whether your
> PR should go to trunk or to trunk+release so they can merge accordingly.
>
> *Please help us test the release! *
>
> Thanks!
>
> Bill Bejeck



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-653: Upgrade log4j to log4j2

[Gwen Shapira]

+1 (binding)

A very welcome update :)

On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee  wrote:
>
> Hi devs,
>
> Here I open the vote for KIP-653: Upgrade log4j to log4j2. It replaces the
> obsolete log4j logging library into the current standard, log4j2, with
> maintaining backward-compatibility.
>
> Thanks,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
>
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr
> <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema

[Gwen Shapira]

Thank you, this will be quite helpful.

+1 (binding)

On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela  wrote:
>
> Hi everyone,
>
> Thanks again for the discussion. I'd like to start the vote for this KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
>
> Thanks,
> Anastasia



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP idea: Support of multipart messages

[Gwen Shapira]

There is another option of doing the splitting on the server and hiding
this from the clients. My personal (and highly controversial) take is that
Kafka clients could use less complexity rather than more. They are
incredibly difficult to reason about as is.  But maybe this
splitting/merging won't be that bad - multi-part messages are well
understood in general.

On Thu, Sep 10, 2020 at 7:51 AM Gwen Shapira  wrote:

> There is also another approach (harder to design, but may be easier to use
> and maintain), which is to make Kafka handle large messages better and
> allow users to set higher limits - for example, can Kafka provide really
> high throughput with 1GB messages? Some systems do it well.
>
> I don't know where the slowdowns happen, but perhaps it is one of these?
> 1. Java GC used to be a problem, but maybe we didn't try with newer GC and
> simple tuning will solve it?
> 2. We have head-of-line blocking issue on the queue. There are approaches
> to solve that too.
>
> I'd love to see more exploration on what exactly is the problem we are
> facing (and is it still an issue? Becket's work is a few years old now.)
>
> On Thu, Sep 10, 2020 at 12:21 AM Alexander Sibiryakov <
> sibirya...@scrapinghub.com> wrote:
>
>> Hey Ben, thanks for the link. My proposal is partially based on Becket's
>> ideas, but I haven't reached out to him directly.
>>
>> +Becket
>> Hi Becket, would you mind to have a look at my proposal (link is in the
>> first message) ?
>>
>> A.
>>
>> On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford  wrote:
>>
>> > LinkedIn had something like this. Becket did a talk on it a few years
>> ago.
>> > It would be interesting to know what became of it and if there were
>> lessons
>> > learned.
>> > https://www.youtube.com/watch?v=ZrbaXDYUZY8
>> >
>> > On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov <
>> > sibirya...@scrapinghub.com> wrote:
>> >
>> > > Hello,
>> > >
>> > > I would like to get your opinions on this KIP idea.
>> > >
>> > > In short it will allow to transfer messages of bigger size than
>> allowed
>> > by
>> > > the broker.
>> > >
>> > >
>> > >
>> >
>> https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing
>> > >
>> > > If all that makes sense, I'll create a full fledged KIP document and
>> > expand
>> > > the idea.
>> > >
>> > > Thanks,
>> > > A.
>> > >
>> >
>> >
>> > --
>> >
>> > Ben Stopford
>> >
>>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP idea: Support of multipart messages

[Gwen Shapira]

There is also another approach (harder to design, but may be easier to use
and maintain), which is to make Kafka handle large messages better and
allow users to set higher limits - for example, can Kafka provide really
high throughput with 1GB messages? Some systems do it well.

I don't know where the slowdowns happen, but perhaps it is one of these?
1. Java GC used to be a problem, but maybe we didn't try with newer GC and
simple tuning will solve it?
2. We have head-of-line blocking issue on the queue. There are approaches
to solve that too.

I'd love to see more exploration on what exactly is the problem we are
facing (and is it still an issue? Becket's work is a few years old now.)

On Thu, Sep 10, 2020 at 12:21 AM Alexander Sibiryakov <
sibirya...@scrapinghub.com> wrote:

> Hey Ben, thanks for the link. My proposal is partially based on Becket's
> ideas, but I haven't reached out to him directly.
>
> +Becket
> Hi Becket, would you mind to have a look at my proposal (link is in the
> first message) ?
>
> A.
>
> On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford  wrote:
>
> > LinkedIn had something like this. Becket did a talk on it a few years
> ago.
> > It would be interesting to know what became of it and if there were
> lessons
> > learned.
> > https://www.youtube.com/watch?v=ZrbaXDYUZY8
> >
> > On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov <
> > sibirya...@scrapinghub.com> wrote:
> >
> > > Hello,
> > >
> > > I would like to get your opinions on this KIP idea.
> > >
> > > In short it will allow to transfer messages of bigger size than allowed
> > by
> > > the broker.
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing
> > >
> > > If all that makes sense, I'll create a full fledged KIP document and
> > expand
> > > the idea.
> > >
> > > Thanks,
> > > A.
> > >
> >
> >
> > --
> >
> > Ben Stopford
> >
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New Kafka PMC Member: John Roesler

[Gwen Shapira]

Congratulations, John!

On Mon, Aug 10, 2020, 1:11 PM Jun Rao  wrote:

> Hi, Everyone,
>
> John Roesler has been a Kafka committer since Nov. 5, 2019. He has remained
> active in the community since becoming a committer. It's my pleasure to
> announce that John is now a member of Kafka PMC.
>
> Congratulations John!
>
> Jun
> on behalf of Apache Kafka PMC
>

Re: [ANNOUNCE] Apache Kafka 2.6.0

[Gwen Shapira]

Yay!

Congratulations everyone :) And thank you for the release, Randall!

On Thu, Aug 6, 2020 at 7:21 AM Randall Hauch  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.0
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Significant performance improvements, especially when the broker has
> large numbers of partitions
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes other features, 74 improvements, 175 bug fixes,
> plus other changes.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 127 contributors to this release!
>
> 17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
> Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
> Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
> Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
> Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
> Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
> Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
> Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
> feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
> Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
> Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
> Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
> Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
> Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
> Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
> Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
> maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
> Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
> Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
> Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
> Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
> Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
> Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
> Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
> Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
>
> Randall Hauch
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-651 - Support PEM format for SSL certificates and private key

[Gwen Shapira]

+1 (binding)
Thank you for driving this, Rajini

On Thu, Aug 6, 2020 at 10:43 AM Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start vote on KIP-651 to support SSL key stores and trust
> stores in PEM format:
>
>-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
>
>
> Thank you...
>
> Regards,
>
> Rajini
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 2.6.0 RC2

[Gwen Shapira]

Thank you, Randall for driving this release.

+1 (binding) after verifying signatures and hashes, building from sources,
running unit/integration tests and some manual tests with the 2.13 build.

Two minor things:
1. There were two sitedoc files - 2.12 and 2.13, we don't really need two
sitedocs generated. Not a big deal, but maybe worth tracking and fixing.
2. I got one test failure locally:

org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus failed, log
available in
/Users/gwenshap/releases/2.6.0-rc2/kafka-2.6.0-src/tools/build/reports/testOutput/org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus.test.stdout

org.apache.kafka.trogdor.agent.AgentTest > testAgentGetStatus FAILED
java.lang.RuntimeException:
at
org.apache.kafka.trogdor.rest.RestExceptionMapper.toException(RestExceptionMapper.java:69)
at
org.apache.kafka.trogdor.rest.JsonRestServer$HttpResponse.body(JsonRestServer.java:285)
at
org.apache.kafka.trogdor.agent.AgentClient.status(AgentClient.java:130)
at
org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus(AgentTest.java:115)

Gwen

On Tue, Jul 28, 2020 at 2:50 PM Randall Hauch  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 2.6.0. This is a
> major release that includes many new features, including:
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes a few other features, 74 improvements, 175 bug
> fixes, plus other fixes.
>
> Release notes for the 2.6.0 release:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, August 3, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> https://github.com/apache/kafka/releases/tag/2.6.0-rc2
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/101/
> System tests: (link to follow)
>
>
> Thanks,
> Randall Hauch
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [ANNOUNCE] New Kafka PMC Member: Mickael Maison

[Gwen Shapira]

Congratulations Mickael. Thrilled to have you as part of the PMC :)

On Fri, Jul 31, 2020 at 8:14 AM Jun Rao  wrote:

> Hi, Everyone,
>
> Mickael Maison has been a Kafka committer since Nov. 5, 2019. He has
> remained active in the community since becoming a committer. It's my
> pleasure to announce that Mickael is now a member of Kafka PMC.
>
> Congratulations Mickael!
>
> Jun
> on behalf of Apache Kafka PMC
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-627: Expose Trogdor-specific JMX Metrics for Tasks and Agents

[Gwen Shapira]

+1 (binding)

Thank you, Sam. It is great to see Trogdor getting the care it deserves.

On Mon, Jun 22, 2020, 1:46 PM Sam Pal  wrote:

> Hi all,
>
> I would like to start a vote for KIP-627, which adds metrics about active
> agents and the number of created, running, and done tasks in a Trogdor
> cluster:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-627%3A+Expose+Trogdor-specific+JMX+Metrics+for+Tasks+and+Agents
>
> Looking forward to hearing from you all!
>
> Best,
> Sam
>
>

Re: [VOTE] KIP-629: Use racially neutral terms in our codebase

[Gwen Shapira]

+1 (binding)

Thank you Xavier!

On Thu, Jun 25, 2020, 3:44 PM Xavier Léauté  wrote:

> Hi Everyone,
>
> I would like to initiate the voting process for KIP-629.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase
>
> Thank you,
> Xavier
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Gwen Shapira]

Kafka follows a "fully compatible model" and new features are always
optional. We have extensive compatibility testing to make sure rolling
upgrades are safe and painless.
This investment in compatibility allows us to reduce the effort needed in
maintaining old versions (Security CVE is the obvious exception, as well as
data-loss bugs).

Not every OSS project has the same model, but it has worked for Kafka for
years now (since the awful upgrade from 0.7 to 0.8 in 2014).

Gwen


On Wed, Jun 24, 2020 at 10:40 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> I agree that is an option, but is there any reason to not have a 2.3.2 and
> 2.4.2 released? If so, it would be nice to know about these reasons.
> Appreciate your time on this.
>
> On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma  wrote:
>
> > I think there's some confusion here. You can upgrade to AK 2.5.0 and
> > completely ignore ZK and TLS. It's completely optional.
> >
> > Ismael
> >
> > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian <
> > gokul24...@gmail.com> wrote:
> >
> > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating
> > > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it
> > would
> > > be nice to have a bug-free version of 2.3 and 2.4.
> > >
> > > Is there a technical reason why we hesitate to get these versions out?
> Or
> > > is it that no one has got around to it?
> > >
> > > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia <
> > sankalpbhati...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Ismael for the response.
> > > >
> > > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which
> had
> > > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> > > > introduces some major changes like support for ZK encryption and a
> few
> > > > others, I was wondering if we should choose a smaller upgrade in such
> > > cases
> > > > as we don't really require the new features in 2.5 and above right
> now.
> > > >
> > > > -
> > > > Sankalp
> > > >
> > > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:
> > > >
> > > > > Hi Sankalp,
> > > > >
> > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0
> > instead?
> > > > We
> > > > > are working on the 2.5.1 release, which would be the recommended
> > > release.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> > > > sankalpbhati...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I would like to know if there are any plans to release a 2.3.2
> and
> > > > 2.4.2
> > > > > > versions for Apache Kafka in the near future. I see there are
> some
> > > > issues
> > > > > > marked as fixed in these two versions
> > > > > > (https://tinyurl.com/ycdpz5cb).
> > > > > > However, I could not find a branch/tag corresponding to these
> > > versions
> > > > in
> > > > > > the github repository (https://github.com/apache/kafka).
> > > > > >
> > > > > > Also, It would be great if someone can help me understand or
> share
> > > any
> > > > > > documentation around the release processes (specifically on when
> we
> > > > > decide
> > > > > > to release a new bug fix version like the ones mentioned above.)
> > > > > >
> > > > > > Thanks,
> > > > > > Sankalp
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

[Gwen Shapira]

Thank you so much for this initiative. Small change, but it makes our
community more inclusive.

Gwen

On Fri, Jun 19, 2020, 6:02 PM Xavier Léauté  wrote:

> Hi Everyone,
>
> There are a number of places in our codebase that use racially charged
> terms. I am proposing we update them to use more neutral terms.
>
> The KIP lists the ones I have found and proposes alternatives. If you see
> any I missed or did not consider, please reply and I'll add them.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase
>
> Thank you,
> Xavier
>

Re: First time patch submitter advice

[Gwen Shapira]

Hi,

1. Unfortunately, you need to get a committer to approve running the tests.
I just gave the green-light on your PR.
2. You can hope that committers will see your PR, but sometimes things get
lost. If you know someone who is familiar with that area of the code, it is
a good idea to ping them.
3. We do have some flaky tests. You can see that Jenkins will run 3
parallel builds, if some of them pass and the committer confirms that
failures are not related to your code, we are ok to merge. Obviously, if
you end up tracking them down and fixing, everyone will be very grateful.

Hope this helps,

Gwen

On Sun, Jun 14, 2020 at 5:52 PM Michael Carter <
michael.car...@instaclustr.com> wrote:

> Hi all,
>
> I’ve submitted a patch for the first time(
> https://github.com/apache/kafka/pull/8844 <
> https://github.com/apache/kafka/pull/8844>), and I have a couple of
> questions that I’m hoping someone can help me answer.
>
> I’m a little unclear what happens after that patch has been submitted. The
> coding guidelines say Jenkins will run tests automatically, but I don’t see
> any results anywhere. Have I misunderstood what should happen, or do I just
> not know where to look?
> Should I be attempting to find reviewers for the change myself, or is that
> done independently of the patch submitter?
>
> Also, in resolving a couple of conflicts that have arisen after the patch
> was first submitted, I noticed that there are now failing unit tests that
> have nothing to do with my change. Is there a convention on how to deal
> with these? Should it be something that I try to fix on my branch?
>
> Any thoughts are appreciated.
>
> Thanks,
> Michael



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-601: Configurable socket connection timeout in NetworkClient

[Gwen Shapira]

+1 (binding)

Thank you for the contribution.

On Wed, Jun 3, 2020 at 12:53 AM Cheng Tan  wrote:

> Dear Rajini,
>
> Thanks for the feedback.
>
> 1)
> Because "request.timeout.ms" only affects in-flight requests, after the
> API NetworkClient.ready() is invoked, the connection won't get closed after
> "request.timeout.ms” hits. Before
> a) the SocketChannel is connected
> b) ssl handshake finished
> c) authentication has finished (sasl)
> clients cannot invoke NetworkClient.send() to send any request, which
> means no in-flight request targeting to the connection will be added.
>
>
> 2)
> I think a default value of 127 seconds make sense, which meets the timeout
> indirectly specified by the default value of “tcp.syn.retries”. I’ve added
> this into the KIP proposal.
>
>
> 3)
> Every time the timeout hits, the timeout value of the next connection try
> will increase.
>
> The timeout will hit iff a connection stays at the `connecting` state
> longer than the timeout value, as indicated by
> ClusterConnectionStates.NodeConnectionState. The connection state of a node
> will change iff `SelectionKey.OP_CONNECT` is detected by
> `nioSelector.Select()`. The connection state may transit from `connecting`
> to
>
> a) `disconnected` when SocketChannel.finishConnect() throws
> IOException.
> b) `connected` when SocketChannel.finishConnect() return TRUE.
>
> In other words, the timeout will hit and increase iff the interested
> SelectionKey.OP_CONNECT doesn't trigger before the timeout arrives, which
> means, for example, network congestion, failure of the ARP request, packet
> filtering, routing error, or a silent discard may happen. (I didn’t read
> the Java NIO source code. Please correct me the case when OP_CONNECT won’t
> get triggered if I’m wrong)
>
>
> 4)
>
> A) Connection timeout dominates both request timeout and API timeout
>
> When connection timeout hits, the connection will be closed. The client
> will be notified either by the responses constructed by NetworkClient or
> the callbacks attached to the request. As a result, the request failure
> will be handled before either connection timeout or API timeout arrives.
>
>
> B) Neither request timeout or API timeout dominates connection timeout
>
> i) Request timeout: Because request timeout only affects in-flight
> requests, after the API NetworkClient.ready() is invoked, the connection
> won't get closed after "request.timeout.ms” hits. Before
> 1. the SocketChannel is connected
> 2. SSL handshake finished
> 3. authentication has finished (SASL)
> , clients won't be able to invoke NetworkClient.send() to send any
> request, which means no in-flight request targeting to the connection will
> be added.
>
> ii) API timeout: In AdminClient, API timeout acts by putting a smaller and
> smaller timeout value to the chain of requests in a same API. After the API
> timeout hits, the retry logic won't close any connection. In consumer, API
> timeout acts as a whole by putting a limit to the code block executing
> time. The retry logic won't close any connection as well.
>
>
> Conclusion:
>
> Thanks again for the long feedback and I’m always enjoying them. I’ve
> supplement the above discussion into the KIP proposal. Please let me know
> what you think.
>
>
> Best, - Cheng Tan
>
>
> > On Jun 2, 2020, at 3:01 AM, Rajini Sivaram 
> wrote:
> >
> > Hi Cheng,
> >
> > Not sure if the discussion should move back to the DISCUSS thread. I
> have a
> > few questions:
> >
> > 1) The KIP motivation says that in some cases `request.timeout.ms`
> doesn't
> > timeout connections properly and as a result it takes 127s to detect a
> > connection failure. This sounds like a bug rather than a limitation of
> the
> > current approach. Can you explain the scenarios where this occurs?
> >
> > 2) I think the current proposal is to use non-exponential 10s connection
> > timeout as default with the option to use exponential timeout. So
> > connection timeouts for every connection attempt will be between 8s and
> 12s
> > by default. Is that correct? Should we use a default max timeout to
> enable
> > exponential timeout by default since 8s seems rather small?
> >
> > 3) What is the scope of `failures` used to determine connection timeout
> > with exponential timeouts? Will we always use 10s followed by 20s every
> > time a connection is attempted?
> >
> > 4) It will be good if we can include two flows with the relationship
> > between various timeouts in the KIP. One with a fixed node like a typic

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

[Gwen Shapira]

+1 (binding)

Looks great. Thank you for the in-depth design and discussion.

On Fri, May 29, 2020 at 7:58 AM David Jacot  wrote:

> Hi folks,
>
> I'd like to start the vote for KIP-599 which proposes a new quota to
> throttle create topic, create partition, and delete topics operations to
> protect the Kafka controller:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations
>
> Please, let me know what you think.
>
> Cheers,
> David
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Resolved] (KAFKA-3034) kafka.api.PlaintextConsumerTest > testSeek FAILED

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3034.
-
Resolution: Fixed

> kafka.api.PlaintextConsumerTest > testSeek FAILED
> -
>
> Key: KAFKA-3034
> URL: https://issues.apache.org/jira/browse/KAFKA-3034
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Gwen Shapira
>Priority: Major
>  Labels: transient-unit-test-failure
>
> See for example:
> https://builds.apache.org/job/kafka-trunk-jdk7/921/console
> It doesn't fail consistently, but happens fairly often.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-3035) Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3035.
-
Resolution: Fixed

> Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED
> ---
>
> Key: KAFKA-3035
> URL: https://issues.apache.org/jira/browse/KAFKA-3035
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Gwen Shapira
>Assignee: Liquan Pei
>Priority: Major
>  Labels: transient-unit-test-failure
>
> See: https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/1868/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-5034) Connect workers don't discover new Connector Plugins without Restart

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-5034.
-
Resolution: Won't Do

Assuming this was fixed since there were many iterations on connector loading.

> Connect workers don't discover new Connector Plugins without Restart
> 
>
> Key: KAFKA-5034
> URL: https://issues.apache.org/jira/browse/KAFKA-5034
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>    Reporter: Gwen Shapira
>Priority: Major
>
> If I want to add a new Connector Plugin to a running distributed Connect 
> cluster, I need to copy the JAR to the classpath and then restart all the 
> workers so they will pick up the new plugin before I can create a connector.
> This is both un-intuitive (most modern up can pick up changes dynamically) 
> and can get difficult when a connect cluster is shared between teams.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-5000) Framework should log some progress information regularly to assist in troubleshooting

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-5000.
-
Resolution: Fixed

Metrics seem like a better indicator than logs.

> Framework should log some progress information regularly to assist in 
> troubleshooting
> -
>
> Key: KAFKA-5000
> URL: https://issues.apache.org/jira/browse/KAFKA-5000
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Gwen Shapira
>Priority: Major
>
> We get many questions of the type:
> "I started a connector, it doesn't seem to make any progress, I don't know 
> what to do"
> I think that periodic "progress reports" on the worker logs may help. 
> We have the offset commit message: "INFO 
> WorkerSinkTask{id=cassandra-sink-connector-0} Committing offsets 
> (org.apache.kafka.connect.runtime.WorkerSinkTask:244)"
> But I think we'd also want to know: topic, partition, offsets, how many rows 
> were read from source/kafka and how many were successfully written.
> This will help determine if there is any progress being made and whether some 
> partitions are "stuck" for some reason.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-3321) KafkaConfigStorage should never encounter commit without config data

[Gwen Shapira (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3321.
-
Resolution: Won't Do

Let's assume this was already resolved or at least the root cause isn't 
interesting any more.

> KafkaConfigStorage should never encounter commit without config data
> 
>
> Key: KAFKA-3321
> URL: https://issues.apache.org/jira/browse/KAFKA-3321
> Project: Kafka
>  Issue Type: Bug
>        Reporter: Gwen Shapira
>Priority: Major
>
> After fixing  KAFKA-2994, KafkaConfigStorage should no longer blow up with 
> surprise NPEs, but we did not figure out what causes it to see commit 
> messages before seeing the configurations that are committed. 
> This JIRA is to track down the root cause of the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-602: Change default value for client.dns.lookup

[Gwen Shapira]

+1 (binding)

Thank you!

On Fri, May 22, 2020 at 1:40 AM Badai Aqrandista  wrote:

> Hi All,
>
> I would like to start the vote on KIP-602: Change default value for
> client.dns.lookup
>
> For reference, here is the KIP wiki:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-602%3A+Change+default+value+for+client.dns.lookup
>
> And discussion thread:
>
>
> https://lists.apache.org/thread.html/r0e70d3757267c4158f12c05a4e5ac9eb33f2d11ce99d5878b3b4b3f7%40%3Cdev.kafka.apache.org%3E
>
> --
> Thanks,
> Badai
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers

[Gwen Shapira]

+1 (binding)

Thank you for driving this, Anna

On Mon, May 18, 2020 at 4:55 PM Anna Povzner  wrote:

> Hi All,
>
> I would like to start the vote on KIP-612: Ability to limit connection
> creation rate on brokers.
>
> For reference, here is the KIP wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers
>
> And discussion thread:
>
> https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E
>
> Thanks,
>
> Anna
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-586: Deprecate commit records without record metadata

[Gwen Shapira]

+1 (binding)

Makes a lot of sense :) Thanks, Mario.

On Mon, Apr 27, 2020 at 12:05 PM Mario Molina  wrote:

> Hi all,
>
> I'd like to start a vote for KIP-586. You can find the link for this KIP
> here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-586%3A+Deprecate+commit+records+without+record+metadata
>
> Thanks!
> Mario
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-606: Add Metadata Context to MetricsReporter

[Gwen Shapira]

+1 (binding)
Thanks for the proposal, Xavier.

On Wed, May 13, 2020 at 11:54 AM Xavier Léauté  wrote:

> Hi everyone,
>
> Folks seem happy with the state of the KIP, so I'd like to start the vote
> for KIP-606
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter
>
> - Xavier
>


-- 
Gwen Shapira

Re: [DISCUSS] KIP-592: MirrorMaker should replicate topics from earliest

[Gwen Shapira]

Same.

There is a  backward compatibility risk (we are changing the behavior, and
the assessment that it won't affect anyone may be correct - but there is a
risk that people depend on existing behaviors in ways we didn't consider)

Remember that users are yelling on twitter even when we do small and very
reasonable changes:
https://twitter.com/jessetanderson/status/1250095104779378690

Since we want to encourage users to move to a significantly better and
safer alternative, I just don't see how the compatibility risk is
justified.

Gwen


On Fri, May 8, 2020 at 4:08 PM Matthias J. Sax  wrote:

> With the discussion about a 3.0 release and deprecating the old MM, I am
> wondering if it's worth to do anything.
>
> Please should just switch to MM2 that has a better default.
>
> Thoughts?
>
>
> On 4/27/20 10:48 AM, Ryanne Dolan wrote:
> > Conversely, we could consider making MM2 use "latest" in "legacy mode",
> and
> > leave MM1 as it is? (Just thinking out loud.)
> >
> > Ryanne
> >
> > On Mon, Apr 27, 2020 at 12:39 PM Jeff Widman 
> wrote:
> >
> >> Good questions:
> >>
> >>
> >> *I agree that `auto.offset.reset="earliest"` would be a better default.
> >> However, I am a little worried about backwardcompatibility. *
> >>
> >> Keep in mind that existing mirrormaker instances will *not* be affected
> for
> >> topics they are currently consuming because they will already have saved
> >> offsets. This will only affect mirrormakers that start consuming new
> >> topics, for which they don't have a saved offset. In those cases, they
> will
> >> stop seeing data loss when they first start consuming. My guess is the
> >> majority of those new topics are going to be newly-created topics
> anyway,
> >> so most of the time starting from the earliest simply prevents skipping
> the
> >> first few seconds/minutes of data written to the topic.
> >>
> >> *What I am also wondering thought is, does this only affect MirrorMaker
> or
> >> also MirrorMaker 2? *
> >>
> >> I checked and MM2 already sets `auto.offset.reset = 'earliest'`
> >> <
> >>
> https://github.com/apache/kafka/blob/d63e0181bb7b9b4f5ed088abc00d7b32aeb0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L233
> >>>
> >> .
> >>
> >> *Also, is it worth to change MirrorMaker now that **MirrorMaker 2 is
> >> available?*
> >>
> >> Given that it's 1-line of code, doesn't affect existing instances, and
> >> prevents data loss on new regex subscriptions, I think it's worth
> >> setting... I basically view it as a bugfix rather than a feature change.
> >>
> >> I realize MM1 is deprecated, but there's still a lot of old mirrormakers
> >> running, so flipping this now will ease the future transition to MM2
> >> because it brings the behavior of MM1 in line with MM2.
> >>
> >> Thoughts?
> >>
> >>
> >>
> >> On Sat, Apr 11, 2020 at 11:59 AM Matthias J. Sax 
> wrote:
> >>
> >>> Jeff,
> >>>
> >>> thanks for the KIP. I agree that `auto.offset.reset="earliest"` would
> be
> >>> a better default. However, I am a little worried about backward
> >>> compatibility. And even if the current default is not idea, users can
> >>> still change it.
> >>>
> >>> What I am also wondering thought is, does this only affect MirrorMaker
> >>> or also MirrorMaker 2? Also, is it worth to change MirrorMaker now that
> >>> MirrorMaker 2 is available?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 4/10/20 9:56 PM, Jeff Widman wrote:
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-592%3A+MirrorMaker+should+replicate+topics+from+earliest
> >>>>
> >>>> It's a relatively minor change, only one line of code. :-D
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >> --
> >>
> >> *Jeff Widman*
> >> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> >> <><
> >>
> >
>
>

-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

[Gwen Shapira]

This would be useful, thank you :)

On Tue, May 5, 2020 at 4:58 PM Xavier Léauté  wrote:

> Hi Everyone,
>
> I've published a KIP to address some shortcoming of our current metrics
> reporter interface. Would appreciate feedback.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter
>
> Thank you,
> Xavier
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Kafka 3.0

[Gwen Shapira]

It sounds like the decision to make the next release 3.0 is a bit arbitrary
then?

With Exactly Once, we announced 1.0 as one release after the one where EOS
shipped, when we felt it was "ready" (little did we know... but that's
another story).
2.0 was breaking due to us dropping Java 7.

In 3.0 it sounds like nothing is breaking and our big change won't be
complete... so, what's the motivation for the major release?

On Tue, May 5, 2020 at 12:12 PM Guozhang Wang  wrote:

> I think there's a confusion regarding the "bridge release" proposed in
> KIP-500: should it be release "3.0" or be release "2.X" (i.e. the last
> minor release before 3.0).
>
> My understanding is that "3.0" would be the bridge release, i.e. it would
> not break any compatibility, but 3.1 potentially would, so an upgrade from
> 2.5 to 3.1 would need to first upgrade to 3.0, and then to 3.1. For
> clients, all broker-client compatibility are still maintained 3.1+ so that
> 2.x producer / consumer clients could still talk to 3.1+ brokers, only
> those old versioned scripts with on "--zookeeper" would not work with 3.1+
> brokers anymore since there are no zookeepers.
>
>
> Guozhang
>
> On Mon, May 4, 2020 at 5:33 PM Gwen Shapira  wrote:
>
> > +1 for removing MM 1.0 when we cut a breaking release. It is sad to see
> > that we are still investing in it (I just saw a KIP toward improving its
> > reset policy).
> >
> > My understanding was that KIP-590 is not breaking compatibility, I think
> > Guozhang said that in response to my question on the discussion thread.
> >
> > Overall, since Kafka has time-based releases, we can make the call on 3.0
> > vs 2.7 when we are at "KIP freeze date" and can see which features are
> > likely to make it.
> >
> >
> > On Mon, May 4, 2020 at 5:13 PM Ryanne Dolan 
> wrote:
> >
> > > Hey Colin, I think we should wait until after KIP-500's "bridge
> release"
> > so
> > > there is a clean break from Zookeeper after 3.0. The bridge release by
> > > definition is an attempt to not break anything, so it theoretically
> > doesn't
> > > warrant a major release. If that's not the case (i.e. if a single
> "bridge
> > > release" turns out to be impractical), we should consider forking 3.0
> > while
> > > maintaining a line of Zookeeper-dependent Kafka in 2.x. That way 3.x
> can
> > > evolve dramatically without breaking the 2.x line. In particular,
> > anything
> > > related to removing Zookeeper could land in pre-3.0 while every other
> > > feature targets 2.6.
> > >
> > > If you are proposing 2.6 should be the "bridge release", I think this
> is
> > > premature given Kafka's time-based release schedule. If the bridge
> > features
> > > happen to be merged before 2.6's feature freeze, then sure -- let's
> make
> > > that the bridge release in retrospect. And if we get all the
> > post-Zookeeper
> > > features merged before 2.7, I'm onboard with naming it "3.0" instead.
> > >
> > > That said, we should aim to remove legacy MirrorMaker before 3.0 as
> well.
> > > I'm happy to drive that additional breaking change. Maybe 2.6 can be
> the
> > > "bridge" for MM2 as well.
> > >
> > > Ryanne
> > >
> > > On Mon, May 4, 2020, 5:05 PM Colin McCabe  wrote:
> > >
> > > > Hi all,
> > > >
> > > > We've had a few proposals recently for incompatible changes.  One of
> > them
> > > > is my KIP-604: Remove ZooKeeper Flags from the Administrative Tools.
> > The
> > > > other is Boyang's KIP-590: Redirect ZK Mutation Protocols to the
> > > > Controller.  I think it's time to start thinking about Kafka 3.0.
> > > > Specifically, I think we should move to 3.0 after the 2.6 release.
> > > >
> > > > From the perspective of KIP-500, in Kafka 3.x we'd like to make
> running
> > > in
> > > > a ZooKeeper-less mode possible (but not yet the default.)  This is
> the
> > > > motivation behind KIP-590 and KIP-604, as well as some of the other
> > KIPs
> > > > we've done recently.  Since it will take some time to stabilize the
> new
> > > > ZooKeeper-free Kafka code, we will hide it behind an option
> initially.
> > > > (We'll have a KIP describing this all in detail soon.)
> > > >
> > > > What does everyone think about having Kafka 3.0 come up next after
> 2.6?
> > > > Are there any other things we should change in the 2.6 -> 3.0
> > transition?
> > > >
> > > > best,
> > > > Colin
> > > >
> > >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
> --
> -- Guozhang
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Kafka 3.0

[Gwen Shapira]

+1 for removing MM 1.0 when we cut a breaking release. It is sad to see
that we are still investing in it (I just saw a KIP toward improving its
reset policy).

My understanding was that KIP-590 is not breaking compatibility, I think
Guozhang said that in response to my question on the discussion thread.

Overall, since Kafka has time-based releases, we can make the call on 3.0
vs 2.7 when we are at "KIP freeze date" and can see which features are
likely to make it.


On Mon, May 4, 2020 at 5:13 PM Ryanne Dolan  wrote:

> Hey Colin, I think we should wait until after KIP-500's "bridge release" so
> there is a clean break from Zookeeper after 3.0. The bridge release by
> definition is an attempt to not break anything, so it theoretically doesn't
> warrant a major release. If that's not the case (i.e. if a single "bridge
> release" turns out to be impractical), we should consider forking 3.0 while
> maintaining a line of Zookeeper-dependent Kafka in 2.x. That way 3.x can
> evolve dramatically without breaking the 2.x line. In particular, anything
> related to removing Zookeeper could land in pre-3.0 while every other
> feature targets 2.6.
>
> If you are proposing 2.6 should be the "bridge release", I think this is
> premature given Kafka's time-based release schedule. If the bridge features
> happen to be merged before 2.6's feature freeze, then sure -- let's make
> that the bridge release in retrospect. And if we get all the post-Zookeeper
> features merged before 2.7, I'm onboard with naming it "3.0" instead.
>
> That said, we should aim to remove legacy MirrorMaker before 3.0 as well.
> I'm happy to drive that additional breaking change. Maybe 2.6 can be the
> "bridge" for MM2 as well.
>
> Ryanne
>
> On Mon, May 4, 2020, 5:05 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > We've had a few proposals recently for incompatible changes.  One of them
> > is my KIP-604: Remove ZooKeeper Flags from the Administrative Tools.  The
> > other is Boyang's KIP-590: Redirect ZK Mutation Protocols to the
> > Controller.  I think it's time to start thinking about Kafka 3.0.
> > Specifically, I think we should move to 3.0 after the 2.6 release.
> >
> > From the perspective of KIP-500, in Kafka 3.x we'd like to make running
> in
> > a ZooKeeper-less mode possible (but not yet the default.)  This is the
> > motivation behind KIP-590 and KIP-604, as well as some of the other KIPs
> > we've done recently.  Since it will take some time to stabilize the new
> > ZooKeeper-free Kafka code, we will hide it behind an option initially.
> > (We'll have a KIP describing this all in detail soon.)
> >
> > What does everyone think about having Kafka 3.0 come up next after 2.6?
> > Are there any other things we should change in the 2.6 -> 3.0 transition?
> >
> > best,
> > Colin
> >
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Gwen Shapira]

Hey Boyang,

Sorry if this was already discussed, but I didn't see this as rejected
alternative:

Until now, we always did client side routing - the client itself found the
controller via metadata and directed requests accordingly. Brokers that
were not the controller, rejected those requests.

Why did we decide to move to broker side routing? Was the client-side
option discussed and rejected somewhere and I missed it?

Gwen

On Fri, Apr 3, 2020, 4:45 PM Boyang Chen  wrote:

> Hey all,
>
> I would like to start off the discussion for KIP-590, a follow-up
> initiative after KIP-500:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller
>
> This KIP proposes to migrate existing Zookeeper mutation paths, including
> configuration, security and quota changes, to controller-only by always
> routing these alterations to the controller.
>
> Let me know your thoughts!
>
> Best,
> Boyang
>

Re: Retention policies question

[Gwen Shapira]

Kafka purges old messages from both leaders and replicas.

If there was a mistake in the book, can you tell me which chapter and page?
We'll fix it.

Gwen

On Wed, Apr 22, 2020, 7:51 AM Alex Bull  wrote:

> Hi, Dear Kafka Developers,
>
> I've read 'Kafka: The Definitive Guide' by Narkhede and others and I have a
> following question.
> On what side topic retention policies (delete or compact) are performed?
> I have a guess that they work only on  brokers that hold leader replica of
> partition.
> Or am I  wrong ?
>
> With best regards,
> Alex.
>

Re: [KAFKA-9861] Process Simplification - Community Validation of Kafka Release Candidates

[Gwen Shapira]

Hi Israel,

I don't think a KIP is required. We evolved our release process few times 
without KIPs, IIRC.

Subtasks in JIRA seem like a good place to track this right now.

Regarding the specific tasks:

- Most of the sample scripts and environment setup can/should be completely 
decoupled from the release itself. There is a release.py ( http://release.py/ ) 
script that generates the "community email", it can be modified to link to 
these instructions and scripts.
- Things like documenting versions should be updated every release. We can add 
them to the docs and add a step to revisit before releasing new doc version to 
the release process wiki.

Gwen

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Mon, Apr 13, 2020 at 2:59 PM, Israel Ekpo < israele...@gmail.com > wrote:

> 
> Hi Everyone,
> 
> 
> I have created [KAFKA-9861] as a process improvement and I have started
> work on it.
> https:/ / issues. apache. org/ jira/ browse/ KAFKA-9861 (
> https://issues.apache.org/jira/browse/KAFKA-9861 )
> 
> 
> I understand that KIPs are generally reserved for code improvements or
> changes but I wanted to find out if something like this is better as a KIP
> rather than a JIRA issue.
> 
> 
> When you have a moment, please let me know and I can either submit the KIP
> or add subtasks or related tasks to this current issue.
> 
> 
> Thanks in advance for any feedback or guidance.
> 
> 
> 
> On Tue, Mar 17, 2020 at 11:00 AM David Arthur < mumrah@ gmail. com (
> mum...@gmail.com ) > wrote:
> 
> 
>> Thanks, Israel. I agree with Gwen, this is a great list and would be
>> useful to add to our release candidate boilerplate. Since we found a
>> blocker bug on RC1, I'll go ahead and close voting. RC2 will be announced
>> shortly.
>> 
>> 
>> -David
>> 
>> On Tue, Mar 17, 2020 at 10:46 AM Israel Ekpo < israelekpo@ gmail. com (
>> israele...@gmail.com ) > wrote:
>> 
>> 
>>> Thanks for the feedback, Gwen. I will create JIRA tasks to track the items
>>> 
>>> shortly.
>>> 
>>> The JIRA tasks will document the goals, expectations and relevant Kafka
>>> versions for each resource.
>>> 
>>> I will volunteer for some of them and update the JIRA tasks accordingly.
>>> 
>>> 
>>> On Tue, Mar 17, 2020 at 12:51 AM Gwen Shapira < gwen@ confluent. io (
>>> g...@confluent.io ) > wrote:
>>> 
>>> > Oh wow, I love this checklist. I don't think we'll have time to create
>>> one
>>> > for this release, but will be great to track this via JIRA and see if we
>>> 
>>> > can get all those contributed before 2.6...
>>> >
>>> > Gwen Shapira
>>> > Engineering Manager | Confluent
>>> > 650.450.2760 | @gwenshap
>>> > Follow us: Twitter | blog
>>> >
>>> > On Mon, Mar 16, 2020 at 3:02 PM, Israel Ekpo < israelekpo@ gmail. com (
>>> israele...@gmail.com ) >
>>> > wrote:
>>> >
>>> > >
>>> > >
>>> > >
>>> > > - Download artifacts successfully
>>> > > - Verified signatures successfully
>>> > > - All tests have passed so far for Scala 2.12. Have not run it on 2.13
>>> 
>>> > yet
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > +1 (non-binding) for the release
>>> > >
>>> > >
>>> > >
>>> > > I do have some feedback so I think we should include in the RC
>>> > > announcement a link for how the community should test and include
>>> > > information like:
>>> > >
>>> > >
>>> > >
>>> > > - How to set up test environment for unit and functional tests
>>> > > - Java version(s) needed for the tests
>>> > > - Scala version(s) needed for the tests
>>> > > - Gradle version needed
>>> > > - Sample script for running sanity checks and unit tests
>>> > > - Sample Helm Charts for running all the basic components on a
>>> Kubernetes
>>> > > - Sample Ansible Script for running all the basic components on
>>> Virtual
>>> > > Machines
>>> > >
>>> > >
>>> > >
>>> > > It takes a bit of time for newcomers to investigate why the tests are
>>> not
>>> > > running successfully in the beginning and providing guidance for these
>>>

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

[Gwen Shapira]

Vote is still good. We can always add things later if needed.

On Fri, Mar 27, 2020, 8:15 AM Aneel Nazareth  wrote:

> Hi Colin and Gwen,
>
> I wanted to double-check that your +1 votes were still good, since the
> proposal has been simplified (removing the --delete-config-file option
> and taking out the support for reading from STDIN).
>
> Thanks!
>
> On Fri, Mar 27, 2020 at 10:01 AM Rajini Sivaram 
> wrote:
> >
> > +1 (binding)
> >
> > Thanks for the KIP, Aneel!
> >
> > Regards,
> >
> > Rajini
> >
> > On Wed, Mar 25, 2020 at 5:06 AM Gwen Shapira  wrote:
> >
> > > +1 (binding), thank you
> > >
> > > On Wed, Mar 18, 2020, 2:26 PM Colin McCabe  wrote:
> > >
> > > > +1 (binding).
> > > >
> > > > Thanks, Aneel.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Mar 18, 2020, at 00:04, David Jacot wrote:
> > > > > +1 (non-binding), thanks for the KIP!
> > > > >
> > > > > David
> > > > >
> > > > > On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth  >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > Thanks to the folks who have given feedback. I've incorporated
> the
> > > > > > suggestions, and think that this is now ready for a vote:
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-574%3A+CLI+Dynamic+Configuration+with+file+input
> > > > > >
> > > > > > Thank you,
> > > > > > Aneel
> > > > > >
> > > > >
> > > >
> > >
>

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

[Gwen Shapira]

+1 (binding), thank you

On Wed, Mar 18, 2020, 2:26 PM Colin McCabe  wrote:

> +1 (binding).
>
> Thanks, Aneel.
>
> best,
> Colin
>
> On Wed, Mar 18, 2020, at 00:04, David Jacot wrote:
> > +1 (non-binding), thanks for the KIP!
> >
> > David
> >
> > On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth 
> wrote:
> >
> > > Hello all,
> > >
> > > Thanks to the folks who have given feedback. I've incorporated the
> > > suggestions, and think that this is now ready for a vote:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-574%3A+CLI+Dynamic+Configuration+with+file+input
> > >
> > > Thank you,
> > > Aneel
> > >
> >
>

Re: [VOTE] KIP-580: Exponential Backoff for Kafka Clients

[Gwen Shapira]

+1 (binding) - thank you

On Mon, Mar 23, 2020, 10:50 AM Sanjana Kaundinya 
wrote:

> Hi Everyone,
>
> I’d like to start a vote for KIP-580: Exponential Backoff for Kafka
> Clients. The link to the KIP can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-580%3A+Exponential+Backoff+for+Kafka+Clients
> .
>
> Thanks,
> Sanjana
>
>

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC1

[Gwen Shapira]

Oh wow, I love this checklist. I don't think we'll have time to create one for 
this release, but will be great to track this via JIRA and see if we can get 
all those contributed before 2.6...

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Mon, Mar 16, 2020 at 3:02 PM, Israel Ekpo < israele...@gmail.com > wrote:

> 
> 
> 
> - Download artifacts successfully
> - Verified signatures successfully
> - All tests have passed so far for Scala 2.12. Have not run it on 2.13 yet
> 
> 
> 
> 
> +1 (non-binding) for the release
> 
> 
> 
> I do have some feedback so I think we should include in the RC
> announcement a link for how the community should test and include
> information like:
> 
> 
> 
> - How to set up test environment for unit and functional tests
> - Java version(s) needed for the tests
> - Scala version(s) needed for the tests
> - Gradle version needed
> - Sample script for running sanity checks and unit tests
> - Sample Helm Charts for running all the basic components on a Kubernetes
> - Sample Ansible Script for running all the basic components on Virtual
> Machines
> 
> 
> 
> It takes a bit of time for newcomers to investigate why the tests are not
> running successfully in the beginning and providing guidance for these
> categories of contributors will be great. If I did not know where to look
> (kafka-2.5.0-src/gradle/dependencies.gradle) it would take longer to
> figure out why the tests are not working/running
> 
> 
> 
> Thanks.
> 
> 
> 
> On Thu, Mar 12, 2020 at 11:21 AM Bill Bejeck < bbejeck@ gmail. com (
> bbej...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi David,
>> 
>> 
>> 
>> 1. Scanned the Javadoc, looks good
>> 2. Downloaded kafka_2.12-2.5.0 and ran the quickstart and streams
>> quickstart
>> 3. Verified the signatures
>> 
>> 
>> 
>> +1 (non-binding)
>> 
>> 
>> 
>> Thanks for running the release David!
>> 
>> 
>> 
>> -Bill
>> 
>> 
>> 
>> On Tue, Mar 10, 2020 at 4:01 PM David Arthur < david. arthur@ confluent. io
>> ( david.art...@confluent.io ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Thanks for the test failure reports, Tom. Tracking (and fixing) these is
>>> important and will make future release managers have an easier time :)
>>> 
>>> 
>>> 
>>> -David
>>> 
>>> 
>>> 
>>> On Tue, Mar 10, 2020 at 10:16 AM Tom Bentley < tbentley@ redhat. com (
>>> tbent...@redhat.com ) > wrote:
>>> 
>>> 
>>>> 
>>>> 
>>>> Hi David,
>>>> 
>>>> 
>>>> 
>>>> I verified signatures, built the tagged branch and ran unit and
>>>> integration
>>>> tests. I found some flaky tests, as follows:
>>>> 
>>>> 
>>>> 
>>>> https:/ / issues. apache. org/ jira/ browse/ KAFKA-9691 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9691 ) (new) https:/ / issues.
>>>> apache. org/ jira/ browse/ KAFKA-9692 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9692 ) (new) https:/ / issues.
>>>> apache. org/ jira/ browse/ KAFKA-9283 (
>>>> https://issues.apache.org/jira/browse/KAFKA-9283 ) (already reported)
>>>> 
>>>> 
>>>> 
>>>> Many thanks,
>>>> 
>>>> 
>>>> 
>>>> Tom
>>>> 
>>>> 
>>>> 
>>>> On Tue, Mar 10, 2020 at 3:28 AM David Arthur < mumrah@ gmail. com (
>>>> mum...@gmail.com ) > wrote:
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> Hello Kafka users, developers and client-developers,
>>>>> 
>>>>> 
>>>>> 
>>>>> This is the second candidate for release of Apache Kafka 2.5.0. The
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> first
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> release candidate included an erroneous NOTICE file, so another RC was
>>>>> needed to fix that.
>>>>> 
>>>>> 
>>>>> 
>>>>> This is a major release of Kafka which includes many new features,
>>>>> improvements, and bug fixes including:
>>>>> 
>>>>> 
>>>>> 
>>>>> * TLS 1.3 support (1.2 is now the

Re: [Vote] KIP-569: Update DescribeConfigsResponse to include additional metadata information

[Gwen Shapira]

+1
Looks great. Thanks for the proposal, Shailesh.

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Mon, Mar 09, 2020 at 6:00 AM, Shailesh Panwar < span...@confluent.io > wrote:

> 
> 
> 
> Hi All,
> I would like to start a vote on KIP-569: Update
> DescribeConfigsResponse to include additional metadata information
> 
> 
> 
> The KIP is here:
> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ 
> KIP-569%3A+DescribeConfigsResponse+-+Update+the+schema+to+include+additional+metadata+information+of+the+field
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-569%3A+DescribeConfigsResponse+-+Update+the+schema+to+include+additional+metadata+information+of+the+field
> )
> 
> 
> 
> Thanks,
> Shailesh
> 
> 
>

Re: Subject: [VOTE] 2.4.1 RC0

[Gwen Shapira]

+1 (binding)

Verified signatures, built jars from source, quickstart passed and local unit 
tests all passed.

Thank you for the release Bill!

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Sat, Mar 07, 2020 at 8:15 PM, Vahid Hashemian < vahid.hashem...@gmail.com > 
wrote:

> 
> 
> 
> +1 (binding)
> 
> 
> 
> Verified signature, built from source, and ran quickstart successfully
> (using openjdk version "11.0.6"). I also ran unit tests locally which
> resulted in a few flaky tests for which there are already open Jiras:
> 
> 
> 
> ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker
> ConsumerBounceTest.testCloseDuringRebalance
> 
> 
> 
> ConsumerBounceTest.testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
> PlaintextEndToEndAuthorizationTest.testNoConsumeWithDescribeAclViaAssign
> 
> 
> 
> SaslClientsWithInvalidCredentialsTest.testManualAssignmentConsumerWithAuthenticationFailure
> SaslMultiMechanismConsumerTest.testCoordinatorFailover
> 
> 
> 
> Thanks for running the release Bill.
> 
> 
> 
> Regards,
> --Vahid
> 
> 
> 
> On Fri, Mar 6, 2020 at 9:20 AM Colin McCabe < cmccabe@ apache. org (
> cmcc...@apache.org ) > wrote:
> 
> 
>> 
>> 
>> +1 (binding)
>> 
>> 
>> 
>> Checked the git hash and branch, looked at the docs a bit. Ran quickstart
>> (although not the connect or streams parts). Looks good.
>> 
>> 
>> 
>> best,
>> Colin
>> 
>> 
>> 
>> On Fri, Mar 6, 2020, at 07:31, David Arthur wrote:
>> 
>> 
>>> 
>>> 
>>> +1 (binding)
>>> 
>>> 
>>> 
>>> Download kafka_2.13-2.4.1 and verified signature, ran quickstart,
>>> everything looks good.
>>> 
>>> 
>>> 
>>> Thanks for running this release, Bill!
>>> 
>>> 
>>> 
>>> -David
>>> 
>>> 
>>> 
>>> On Wed, Mar 4, 2020 at 6:06 AM Eno Thereska < eno. thereska@ gmail. com (
>>> eno.there...@gmail.com ) >
>>> 
>>> 
>> 
>> 
>> 
>> wrote:
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> Hi Bill,
>>>> 
>>>> 
>>>> 
>>>> I built from source and ran unit and integration tests. They passed. There
>>>> was a large number of skipped tests, but I'm assuming that is intentional.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Cheers
>>>> Eno
>>>> 
>>>> 
>>>> 
>>>> On Tue, Mar 3, 2020 at 8:42 PM Eric Lalonde < eric@ autonomic. ai (
>>>> e...@autonomic.ai ) > wrote:
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> 
>>>>> 
>>>>> I ran:
>>>>> $
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> https:/ / github. com/ elalonde/ kafka/ blob/ master/ bin/ verify-kafka-rc.
>> sh ( https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh )
>> 
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> < https:/ / github. com/ elalonde/ kafka/ blob/ master/ bin/ 
>>>> verify-kafka-rc.
>>>> sh ( https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh )
>>>> >
>>>> 2.4.1 https:/ / home. apache. org/ ~bbejeck/ kafka-2. 4. 1-rc0 (
>>>> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 ) < https:/ / home. 
>>>> apache.
>>>> org/ ~bbejeck/ kafka-2. 4. 1-rc0 (
>>>> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 ) >
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> All checksums and signatures are good and all unit and integration
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> tests
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> that were executed passed successfully.
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> - Eric
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mar 2, 2020, at 6:39 PM, Bill Bejeck < bbejeck@ gmail. com (
>>>>>> bbej...@gmail.com ) &

Re: [VOTE] KIP-570: Add leader epoch in StopReplicaRequest

[Gwen Shapira]

+1

On Mon, Feb 24, 2020, 2:16 AM David Jacot  wrote:

> Hi all,
>
> I would like to start a vote on KIP-570: Add leader epoch in
> StopReplicaRequest
>
> The KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-570%3A+Add+leader+epoch+in+StopReplicaRequest
>
> Thanks,
> David
>

Re: [VOTE] KIP-518: Allow listing consumer groups per state

[Gwen Shapira]

+1 (binding)

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Mon, Mar 02, 2020 at 8:32 AM, David Jacot < dja...@confluent.io > wrote:

> 
> 
> 
> +1 (non-binding). Thanks for the KIP!
> 
> 
> 
> David
> 
> 
> 
> On Thu, Feb 6, 2020 at 10:45 PM Colin McCabe < cmccabe@ apache. org (
> cmcc...@apache.org ) > wrote:
> 
> 
>> 
>> 
>> Hi Mickael,
>> 
>> 
>> 
>> Thanks for the KIP. I left a comment on the DISCUSS thread as well.
>> 
>> 
>> 
>> best,
>> Colin
>> 
>> 
>> 
>> On Thu, Feb 6, 2020, at 08:58, Mickael Maison wrote:
>> 
>> 
>>> 
>>> 
>>> Hi Manikumar,
>>> 
>>> 
>>> 
>>> I believe I've answered David's comments in the DISCUSS thread. Thanks
>>> 
>>> 
>>> 
>>> On Wed, Jan 15, 2020 at 10:15 AM Manikumar < manikumar. reddy@ gmail. com (
>>> manikumar.re...@gmail.com ) >
>>> 
>>> 
>> 
>> 
>> 
>> wrote:
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> Hi Mickael,
>>>> 
>>>> 
>>>> 
>>>> Thanks for the KIP. Can you respond to the comments from David on
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> discuss
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> thread?
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 
>

[jira] [Created] (KAFKA-9631) MockAdminClient doesn't handle CreateTopics optional fields

[Gwen Shapira (Jira)]

Gwen Shapira created KAFKA-9631:
---

 Summary: MockAdminClient doesn't handle CreateTopics optional 
fields
 Key: KAFKA-9631
 URL: https://issues.apache.org/jira/browse/KAFKA-9631
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


AdminClient's {{createTopics()}} method has a variant with two optional fields. 
So I'd expect the following code to work correctly:
 {{admin.createTopics(Collections.singletonList(new NewTopic(TOPIC_NAME, 
Optional.empty(), Optional.empty(}}

Indeed it works great, as long as we are using the real KafkaAdminClient. 
MockKafkaAdminClient tries to get number of replicas without checking that the 
values make sense , and therefore it fails with:

{{java.lang.IllegalArgumentException: Illegal Capacity: -1}}{{at 
java.base/java.util.ArrayList.(ArrayList.java:158)}}
{{ at 
org.apache.kafka.clients.admin.MockAdminClient.createTopics(MockAdminClient.java:183)}}
{{ at org.apache.kafka.clients.admin.Admin.createTopics(Admin.java:125)}}

Making a mockery of the mock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Gwen Shapira]

Ah, got it! I am indeed curious why they do this :)

Maybe John can shed more light. But if we can't find a better fix, perhaps the 
nice thing to do is really a separate logger, so users who are not worried 
about shooting themselves in the foot can make those warnings go away.

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Fri, Feb 07, 2020 at 4:13 AM, Patrik Kleindl < pklei...@gmail.com > wrote:

> 
> 
> 
> Hi Gwen
> 
> 
> 
> Kafka Streams is not a third party library and produces a lot of these
> warnings, e.g.
> 
> 
> 
> *The configuration 'main.consumer.max.poll.records' was supplied but isn't
> a known config.*
> *The configuration 'admin.retries' was supplied but isn't a known config.*
> and various others if you try to fine-tune the restoration consumer or
> inject parameters for state stores.
> This results in a lot of false positives and only makes new people worried
> and then ignore the warnings altogether.
> 
> 
> 
> Unless this is taken care of at least the Kafka Streams users will
> probably be better off having this on debug level.
> 
> 
> 
> Best regards
> 
> 
> 
> Patrik
> 
> 
> 
> On Thu, 6 Feb 2020 at 16:55, Gwen Shapira < gwen@ confluent. io (
> g...@confluent.io ) > wrote:
> 
> 
>> 
>> 
>> INFO is the default log level, and while it looks less "alarming" than
>> WARN, users will still see it and in my experience, they will worry that
>> something is wrong anyway. Or if INFO isn't the default, users won't see
>> it, so it is no different from debug and we are left with no way of
>> warning users that they misconfigured something.
>> 
>> 
>> 
>> The point is that "known configs" exist in Kafka as a validation step. It
>> is there to protect users. So anything that makes the concerns about
>> unknown configs invisible to users, makes the validation step useless and
>> we may as well remove it. I'm against that - I think users should be made
>> aware of misconfigs as much as possible - especially since if you misspell
>> 
>> "retention", you will lose data.
>> 
>> 
>> 
>> If we look away from the symptom and go back to the actual cause
>> 
>> 
>> 
>> I think Kafka had a way (and maybe it still does) for 3rd party developers
>> who create client plugins (mostly interceptors) to make their configs
>> "known". 3rd party developers should be responsible for the good
>> experience of their users. Now it is possible that you'll pick a 3rd party
>> library that didn't do it and have a worse user experience, but I am not
>> sure it is the job of Apache Kafka to protect users from their choice of
>> libraries
>> (and as long as those libraries are OSS, users can fix them). Especially
>> not at the expense of someone who doesn't use 3rd party libs.
>> 
>> 
>> 
>> Gwen
>> 
>> 
>> 
>> Gwen Shapira
>> Engineering Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>> 
>> 
>> 
>> On Thu, Feb 06, 2020 at 2:06 AM, Artur Burtsev < artjock@ gmail. com (
>> artj...@gmail.com ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi John,
>>> 
>>> 
>>> 
>>> In out case it wont help, since we are running instance per partition and
>>> even with summary only we get 32 warnings per rollout.
>>> 
>>> 
>>> 
>>> Hi Gwen,
>>> 
>>> 
>>> 
>>> Thanks for you reply, I understand and share your concern, I also
>>> mentioned it earlier in the thread. Do you think it will work if we
>>> 
>>> 
>> 
>> 
>> 
>> change
>> 
>> 
>>> 
>>> 
>>> DEBUG to INFO?
>>> 
>>> 
>>> 
>>> Thanks,
>>> Artur
>>> 
>>> 
>>> 
>>> On Thu, Feb 6, 2020 at 4:21 AM Gwen Shapira < gwen@ confluent. io ( gwen@ 
>>> confluent.
>>> io ( g...@confluent.io ) ) > wrote:
>>> 
>>> 
>>>> 
>>>> 
>>>> Sorry for late response. The reason that unused configs is in WARN is
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> that
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> if you misspell a config, it means that it will not apply. In some cases
>>>> (default retention) you want know until too late. We wanted to warn
>>>> 
>>

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Gwen Shapira]

INFO is the default log level, and while it looks less "alarming" than WARN, 
users will still see it and in my experience, they will worry that something is 
wrong anyway.  Or if INFO isn't the default, users won't see it, so it is no 
different from debug and we are left with no way of warning users that they 
misconfigured something.

The point is that "known configs" exist in Kafka as a validation step. It is 
there to protect users. So anything that makes the concerns about unknown 
configs invisible to users, makes the validation step useless and we may as 
well remove it. I'm against that - I think users should be made aware of 
misconfigs as much as possible - especially since if you misspell "retention", 
you will lose data.

If we look away from the symptom and go back to the actual cause  

I think Kafka had a way (and maybe it still does) for 3rd party developers who 
create client plugins (mostly interceptors) to make their configs "known". 3rd 
party developers should be responsible for the good experience of their users.  
Now it is possible that you'll pick a 3rd party library that didn't do it and 
have a worse user experience, but I am not sure it is the job of Apache Kafka 
to protect users from their choice of libraries (and as long as those libraries 
are OSS, users can fix them). Especially not at the expense of someone who 
doesn't use 3rd party libs.

Gwen

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Thu, Feb 06, 2020 at 2:06 AM, Artur Burtsev < artj...@gmail.com > wrote:

> 
> 
> 
> Hi John,
> 
> 
> 
> In out case it wont help, since we are running instance per partition and
> even with summary only we get 32 warnings per rollout.
> 
> 
> 
> Hi Gwen,
> 
> 
> 
> Thanks for you reply, I understand and share your concern, I also
> mentioned it earlier in the thread. Do you think it will work if we change
> DEBUG to INFO?
> 
> 
> 
> Thanks,
> Artur
> 
> 
> 
> On Thu, Feb 6, 2020 at 4:21 AM Gwen Shapira < gwen@ confluent. io (
> g...@confluent.io ) > wrote:
> 
> 
>> 
>> 
>> Sorry for late response. The reason that unused configs is in WARN is that
>> if you misspell a config, it means that it will not apply. In some cases
>> (default retention) you want know until too late. We wanted to warn admins
>> about possible misconfigurations.
>> 
>> 
>> 
>> In the context of a company supporting Kafka - customers run logs at INFO
>> level normally, so if we suspect a misconfiguration, we don't want to ask
>> the customer to change level to DEBUG and bounce the broker. It is time
>> consuming and can be risky.
>> 
>> 
>> 
>> *Gwen Shapira*
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter ( https:/ / twitter. com/ ConfluentInc (
>> https://twitter.com/ConfluentInc ) ) | blog ( http:/ / www. confluent. io/
>> blog ( http://www.confluent.io/blog ) )
>> 
>> 
>> 
>> Sent via Superhuman ( https:/ / sprh. mn/ ?vip=gwen@ confluent. io (
>> https://sprh.mn/?vip=g...@confluent.io ) )
>> 
>> 
>> 
>> On Mon, Jan 06, 2020 at 4:21 AM, Stanislav Kozlovski < stanislav@ confluent.
>> io ( stanis...@confluent.io ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hey Artur,
>>> 
>>> 
>>> 
>>> Perhaps changing the log level to DEBUG is the simplest approach.
>>> 
>>> 
>>> 
>>> I wonder if other people know what the motivation behind the WARN log was?
>>> I'm struggling to think up of a scenario where I'd like to see unused
>>> values printed in anything above DEBUG.
>>> 
>>> 
>>> 
>>> Best,
>>> Stanislav
>>> 
>>> 
>>> 
>>> On Mon, Dec 30, 2019 at 12:52 PM Artur Burtsev < artjock@ gmail. com ( 
>>> artjock@
>>> gmail. com ( artj...@gmail.com ) ) > wrote:
>>> 
>>> 
>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> 
>>>> 
>>>> Indeed changing the log level for the whole AbstractConfig is not an
>>>> option, because logAll is extremely useful.
>>>> 
>>>> 
>>>> 
>>>> Grouping warnings into 1 (with the count of unused only) will not be a
>>>> good option for us either. It will still be pretty noisy. Imagine we have
>>>> 32 partitions and scaled up the application to 32 instances then we still
>>>> have 32 warnings per application (instead of 96 now) while we would like
>>>> to have 0 warnings

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Gwen Shapira]

Sorry for late response. The reason that unused configs is in WARN is that if 
you misspell a config, it means that it will not apply. In some cases (default 
retention) you want know until too late. We wanted to warn admins about 
possible misconfigurations.

In the context of a company supporting Kafka - customers run logs at INFO level 
normally, so if we suspect a misconfiguration, we don't want to ask the 
customer to change level to DEBUG and bounce the broker. It is time consuming 
and can be risky.

*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter ( https://twitter.com/ConfluentInc ) | blog ( 
http://www.confluent.io/blog )

Sent via Superhuman ( https://sprh.mn/?vip=g...@confluent.io )

On Mon, Jan 06, 2020 at 4:21 AM, Stanislav Kozlovski < stanis...@confluent.io > 
wrote:

> 
> 
> 
> Hey Artur,
> 
> 
> 
> Perhaps changing the log level to DEBUG is the simplest approach.
> 
> 
> 
> I wonder if other people know what the motivation behind the WARN log was?
> I'm struggling to think up of a scenario where I'd like to see unused
> values printed in anything above DEBUG.
> 
> 
> 
> Best,
> Stanislav
> 
> 
> 
> On Mon, Dec 30, 2019 at 12:52 PM Artur Burtsev < artjock@ gmail. com (
> artj...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi,
>> 
>> 
>> 
>> Indeed changing the log level for the whole AbstractConfig is not an
>> option, because logAll is extremely useful.
>> 
>> 
>> 
>> Grouping warnings into 1 (with the count of unused only) will not be a
>> good option for us either. It will still be pretty noisy. Imagine we have
>> 32 partitions and scaled up the application to 32 instances then we still
>> have 32 warnings per application (instead of 96 now) while we would like
>> to have 0 warnings because we are perfectly aware of using
>> schema.registry.url and its totally fine, and we don't have to be warned
>> every time we start the application. Now imagine we use more than one
>> consumer per application, then it will add another multiplication factor
>> to these grouped warnings and we still have a lot of those. So I would say
>> grouping doesn't help much.
>> 
>> 
>> 
>> I think adding extra logger like
>> "org.apache.kafka.clients.producer.ProducerConfig.unused" could be another
>> good option. That would leave the existing interface untouched and give
>> everyone an option to mute irrelevant warnings.
>> 
>> 
>> 
>> To summarize, I still can see 3 options with its pros and cons discussed
>> in the thread:
>> 1) extra config with interface to handle unused
>> 2) change unused warn to debug
>> 3) add extra logger for unused
>> 
>> 
>> 
>> Please let me know what do you think.
>> 
>> 
>> 
>> Thanks,
>> Artur
>> 
>> 
>> 
>> On Mon, Dec 30, 2019 at 11:07 AM Stanislav Kozlovski
>> < stanislav@ confluent. io ( stanis...@confluent.io ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi all,
>>> 
>>> 
>>> 
>>> Would printing all the unused configurations in one line, versus N lines,
>>> be more helpful? I know that it would greatly reduce the verbosity in log
>>> visualization tools like Kibana while still allowing us to see all the
>>> relevant information without the need for an explicit action (e.g changing
>>> the log level)
>>> 
>>> 
>>> 
>>> Best,
>>> Stanislav
>>> 
>>> 
>>> 
>>> On Sat, Dec 28, 2019 at 3:13 PM John Roesler < vvcephei@ apache. org (
>>> vvcep...@apache.org ) >
>>> 
>>> 
>> 
>> 
>> 
>> wrote:
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> Hi Artur,
>>>> 
>>>> 
>>>> 
>>>> That’s a good point.
>>>> 
>>>> 
>>>> 
>>>> One thing you can do is log a summary at WARN level, like “27
>>>> configurations were ignored. Ignored configurations are logged at DEBUG
>>>> level.”
>>>> 
>>>> 
>>>> 
>>>> I looked into the code a little, and these log messages are generated
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> in
>> 
>> 
>>> 
>>>> 
>>>> 
>>>> AbstractConfig (logAll and logUnused). They both use the logger
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> associat

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[Gwen Shapira]

Thank you for the KIP. Awesomely cloud-native improvement :)

+1 (binding)


On Tue, Jan 21, 2020, 9:35 AM David Jacot  wrote:

> Hi all,
>
> I would like to start a vote on KIP-559: Make the Kafka Protocol Friendlier
> with L7 Proxies.
>
> The KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
>
> Thanks,
> David
>

Re: [VOTE] KIP-515: Hardened TLS Configs to ZooKeeper

[Gwen Shapira]

+1 (binding), this has been an on-going concern. Thank you for
addressing this, Ron.

On Mon, Jan 20, 2020 at 5:22 AM Ron Dagostino  wrote:
>
> Hi everyone.  I would like to start a vote on KIP-515: Enable ZK
> client to use the new TLS supported authentication.
>
> The KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>
> The discussion thread is here:
> https://lists.apache.org/thread.html/519d07e607cf6598a8126139c964d31fa46f2c028b88b1d44086b8a9%40%3Cdev.kafka.apache.org%3E
>
> Thanks,
>
> Ron

Re: KIP-560 Discuss

[Gwen Shapira]

I am not sure I follow. Afaik:

1. Topics don't include client ID information
2. Even if you did, the same ID could be used for topics that are not Kafka
Streams input

The regex idea sounds doable, but I'm not sure it solves much?


On Sat, Jan 18, 2020, 7:12 AM Sang wn Lee  wrote:

> Thank you
> Gwen Shapira!
> We'll add a flag to clear all topics by clientId
> It is ‘reset-all-external-topics’
>
> I also want to use regex on the input topic flag to clear all matching
> topics.
>
> On 2020/01/17 19:29:09, Gwen Shapira  wrote:
> > Seem like a very nice improvement to me. But I have to admit that I
> > don't understand how this will how - how could you infer the input
> > topics?
> >
> > On Thu, Jan 16, 2020 at 10:03 AM Sang wn Lee 
> wrote:
> > >
> > > Hello,
> > >
> > > Starting this thread to discuss KIP-560:
> > > wiki link :
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool
> > >
> > > I'm newbie
> > > I would like to receive feedback on the following features!
> > >
> > >
> >
>

Re: KIP-560 Discuss

[Gwen Shapira]

Seem like a very nice improvement to me. But I have to admit that I
don't understand how this will how - how could you infer the input
topics?

On Thu, Jan 16, 2020 at 10:03 AM Sang wn Lee  wrote:
>
> Hello,
>
> Starting this thread to discuss KIP-560:
> wiki link :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-560%3A+Auto+infer+external+topic+partitions+in+stream+reset+tool
>
> I'm newbie
> I would like to receive feedback on the following features!
>
>

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

[Gwen Shapira]

+1 (binding)
Looks super useful. Thank you.




On Mon, Jan 13, 2020 at 8:16 AM Konstantine Karantasis
 wrote:
>
> Hi everyone.
>
> I hope y'all had a nice break. The discussion on KIP-158 seems to have
> wrapped up since last year, so I'd like to open the vote on this KIP.
>
> A reminder that this is an updated KIP-158 (that had also been approved in
> its earlier version) and it seems to be a highly anticipated feature for
> many of us. I hope we can get this in for the upcoming release.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics
>
> Best,
> Konstantine

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Gwen Shapira]

+1 (binding, re-vote)

On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe  wrote:
>
> Hi all,
>
> I'm reposting this since I've been informed that gmail mashed the original 
> VOTE thread into a different email thread.  Hopefully the different thread 
> title will prevent it doing that in this case.
>
> I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
> Kafka Administrative Tools.
>
> KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
>
> Discussion thread: 
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
>
> cheers,
> Colin

[ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Gwen Shapira]

Hi everyone,

I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
Reddy are now members of Apache Kafka PMC.

Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
2019. They all contributed many patches, code reviews and participated
in many KIP discussions. We appreciate their contributions and are
looking forward to many more to come.

Congrats Colin, Vahid and Manikumar!

Gwen, on behalf of Apache Kafka PMC

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Gwen Shapira]

+1, thanks for driving this

On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe  wrote:
>
> Hi all,
>
> I'd like to start the vote on KIP-551: Expose disk read and write metrics.
>
> KIP:  https://cwiki.apache.org/confluence/x/sotSC
>
> Discussion thread:
> https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
>
> cheers,
> Colin

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Gwen Shapira]

Sorry for the super late reply, but since the vote thread showed up,
I've read the KIP and have a concern.
The concern was raised by Matthias Sax earlier, but I didn't see it addressed.

Basically the current iteration of the KIP unifies deserialization
errors with corruption. As was pointed out, these are not the same
thing. Corruption means that the message itself (including envelope,
not just the payload) is broken. De-serialization errors mean that
either key or value serializers have a problem. It can even be a
temporary problem of connecting to schema registry, I believe. Corrupt
messages can only be skipped. De-serialization errors can (and
arguably should) be retried with a different serializer. Something
like Connect will need to skip corrupt messages, but messages with
SerDe issues should probably go into a dead-letter queue.

Anyway, IMO we need exceptions that will let us tell the difference.

Gwen

On Fri, Oct 11, 2019 at 10:05 AM Stanislav Kozlovski
 wrote:
>
> Thanks Jason. I've edited the KIP with the latest proposal.
>
> On Thu, Oct 10, 2019 at 2:00 AM Jason Gustafson  wrote:
>
> > Hi Stanislav,
> >
> > Sorry for the late comment. I'm happy with the current proposal. Just one
> > small request is to include an example which shows how a user could use
> > this to skip over a record.
> >
> > I'd suggest pushing this to a vote to see if anyone else has feedback.
> >
> > Thanks,
> > Jason
> >
> > On Sat, Jul 13, 2019 at 2:27 PM Stanislav Kozlovski <
> > stanis...@confluent.io>
> > wrote:
> >
> > > Hello,
> > >
> > > Could we restart the discussion here again?
> > >
> > > My last message was sent on the 3rd of June but I haven't received
> > replies
> > > since then.
> > >
> > > I'd like to get this KIP to a finished state, regardless of whether that
> > is
> > > merged-in or discarded. It has been almost one year since the publication
> > > of the KIP.
> > >
> > > Thanks,
> > > Stanislav
> > >
> > > On Mon, Jun 3, 2019 at 11:19 AM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > wrote:
> > >
> > > > Do people agree with the approach I outlined in my last reply?
> > > >
> > > > On Mon, May 6, 2019 at 2:12 PM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > >> Hey there Kamal,
> > > >>
> > > >> I'm sincerely sorry for missing your earlier message. As I open this
> > > >> thread up, I see I have an unsent draft message about resuming
> > > discussion
> > > >> from some time ago.
> > > >>
> > > >> In retrospect, I think I may have been too pedantic with the exception
> > > >> naming and hierarchy.
> > > >> I now believe a single exception type of
> > > `RecordDeserializationException`
> > > >> is enough. Let's go with that.
> > > >>
> > > >> On Mon, May 6, 2019 at 6:40 AM Kamal Chandraprakash <
> > > >> kamal.chandraprak...@gmail.com> wrote:
> > > >>
> > > >>> Matthias,
> > > >>>
> > > >>> We already have CorruptRecordException which doesn't extend the
> > > >>> SerializationException. So, we need an alternate
> > > >>> name suggestion for the corrupted record error if we decide to extend
> > > the
> > > >>> FaultyRecordException class.
> > > >>>
> > > >>> Stanislav,
> > > >>>
> > > >>> Our users are also facing this error. Could we bump up this
> > discussion?
> > > >>>
> > > >>> I think we can have a single exception type
> > > >>> FaultyRecordException/RecordDeserialization exception to capture both
> > > >>> the errors. We can add an additional enum field to differentiate the
> > > >>> errors
> > > >>> if required.
> > > >>>
> > > >>> Thanks,
> > > >>> Kamal Chandraprakash
> > > >>>
> > > >>> On Wed, Apr 24, 2019 at 1:49 PM Kamal Chandraprakash <
> > > >>> kamal.chandraprak...@gmail.com> wrote:
> > > >>>
> > > >>> > Stanislav,
> > > >>> >
> > > >>> > Any updates on this KIP? We have internal users who want to skip
> > the
> > > >>> > corrupted message while consuming the records.
> > > >>> >
> > > >>> >
> > > >>> > On Fri, Oct 19, 2018 at 11:34 PM Matthias J. Sax <
> > > >>> matth...@confluent.io>
> > > >>> > wrote:
> > > >>> >
> > > >>> >> I am not 100% familiar with the details of the consumer code,
> > > however
> > > >>> I
> > > >>> >> tend to disagree with:
> > > >>> >>
> > > >>> >> > There's no difference between the two cases -- if (and only if)
> > > the
> > > >>> >> message is corrupt, it can't be deserialized.  If (and only if) it
> > > >>> can't be
> > > >>> >> deserialized, it is corrupt.
> > > >>> >>
> > > >>> >> Assume that a user configures a JSON deserializer but a faulty
> > > >>> upstream
> > > >>> >> producer writes an Avro message. For this case, the message is not
> > > >>> >> corrupted, but still can't be deserialized. And I can imaging that
> > > >>> users
> > > >>> >> want to handle both cases differently.
> > > >>> >>
> > > >>> >> Thus, I think it makes sense to have two different exceptions
> > > >>> >> `RecordDeserializationException` and `CorruptedRecordException`
> > that
> > > >>> can
> > > >>> >> both extend 

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

[Gwen Shapira]

I think you linked to the wrong KIP? Here's a link that worked for me:
https://s.apache.org/ve15g

On Thu, Jan 9, 2020 at 4:24 PM Colin McCabe  wrote:
>
> Hi all,
>
> I wrote KIP about deprecating the --zookeeper flag in the administrative 
> tools.  Take a look if you get a chance:
>
> https://cwiki.apache.org/confluence/x/sotSC
>
> best,
> Colin

[jira] [Created] (KAFKA-9328) Move MirrorMaker 2.0 documentation to site-docs

[Gwen Shapira (Jira)]

Gwen Shapira created KAFKA-9328:
---

 Summary: Move MirrorMaker 2.0 documentation to site-docs
 Key: KAFKA-9328
 URL: https://issues.apache.org/jira/browse/KAFKA-9328
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira


Currently MirrorMaker 2.0 is documented here: 
https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md

It will be much easier for our users to find and use if we include it in the 
main docs, side by side with the original MirrorMaker (while clarifying that 
MirrorMaker 2.0 is the preferred option going forward and the original is on 
deprecation schedule).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: MirrorMaker 2.0 Documentation

[Gwen Shapira]

OK, awesome :)
I just wanted to make sure there are no objections before opening the
JIRA, since discussions are easier on the mailing list.

On Mon, Dec 23, 2019 at 5:01 PM Ryanne Dolan  wrote:
>
> Gwen, no one is hiding anything. Please follow up your JIRAs with PRs and
> I'm sure we can get the site updated.
>
> Ryanne
>
> On Mon, Dec 23, 2019, 8:38 AM Gwen Shapira  wrote:
>
> > There's nothing preventing us from updating the site-docs now to make them
> > match the 2.4 release better (I just opened few JIRA for additional docs).
> >
> > Is there a reason why we want to hide MirrorMaker 2.0 until next release?
> > For example, is it unstable? Do you expect interfaces or tools to
> > significantly evolve?
> >
> > On Mon, Dec 23, 2019, 4:33 PM Ryanne Dolan  wrote:
> >
> > > Hey Gwen, per the KIP, the depreciation plan will take a few releases.
> > > Copied here:
> > >
> > > Phase 1 (targeting next Apache Kafka release): All MirrorMaker 2.0 Java
> > > code is added to ./connect/mirror/.
> > >
> > > Phase 2 (subsequent release): Legacy MirrorMaker Scala code is
> > deprecated,
> > > but kept in place. Sample MM2 scripts and configuration files are added
> > to
> > > ./bin/ and ./config/.
> > >
> > > Phase 3 (subsequent release): Legacy MirrorMaker Scala code is removed
> > from
> > > Apache Kafka. A new ./bin/kafka-mirror-maker.sh script is provided which
> > > replaces and emulates the legacy script.
> > >
> > > Accordingly, let's plan to expand the documentation in the next release,
> > > and then remove the legacy documentation in Phase 3. At present, the
> > > existing documentation is still accurate.
> > >
> > > Ryanne
> > >
> > > On Mon, Dec 23, 2019, 2:25 AM Gwen Shapira  wrote:
> > >
> > > > Hey,
> > > >
> > > > I was surprised to discover that MirrorMaker 2.0 (part of 2.4
> > > > release), is not mentioned at all in our documentation
> > > > (https://kafka.apache.org/documentation/).
> > > >
> > > > Ryanne Dolan was kind enough to point me to the documentation for this
> > > > feature (
> > > > https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md)
> > > > and mentioned that the Operations docs will be updated when MM1 is
> > > > deprecated.
> > > >
> > > > IMO, this is a disservice to our users. If we already know that we
> > > > plan to deprecate MirrorMaker and replace it, we need to let users
> > > > know that an alternative exists, that they should start planning a
> > > > transition and how to use the cool new feature. And we need to do it
> > > > in the place where our users normally go to learn how to use our
> > > > product.
> > > >
> > > > As it stands now, the README doesn't even show up on Google search
> > > > results, so discovering how to use MirrorMaker 2.0 is nearly
> > > > impossible. How can we expect to deprecate 1.0 if users can't find out
> > > > about 2.0?
> > > >
> > > > In the past, we've documented new features before existing features
> > > > were deprecated (--broker-list and --bootstrap were documented before
> > > > --zookeeper was removed). I suggest we'll stick to this precedent.
> > > >
> > > > Gwen
> > > >
> > >
> >

Re: MirrorMaker 2.0 Documentation

[Gwen Shapira]

There's nothing preventing us from updating the site-docs now to make them
match the 2.4 release better (I just opened few JIRA for additional docs).

Is there a reason why we want to hide MirrorMaker 2.0 until next release?
For example, is it unstable? Do you expect interfaces or tools to
significantly evolve?

On Mon, Dec 23, 2019, 4:33 PM Ryanne Dolan  wrote:

> Hey Gwen, per the KIP, the depreciation plan will take a few releases.
> Copied here:
>
> Phase 1 (targeting next Apache Kafka release): All MirrorMaker 2.0 Java
> code is added to ./connect/mirror/.
>
> Phase 2 (subsequent release): Legacy MirrorMaker Scala code is deprecated,
> but kept in place. Sample MM2 scripts and configuration files are added to
> ./bin/ and ./config/.
>
> Phase 3 (subsequent release): Legacy MirrorMaker Scala code is removed from
> Apache Kafka. A new ./bin/kafka-mirror-maker.sh script is provided which
> replaces and emulates the legacy script.
>
> Accordingly, let's plan to expand the documentation in the next release,
> and then remove the legacy documentation in Phase 3. At present, the
> existing documentation is still accurate.
>
> Ryanne
>
> On Mon, Dec 23, 2019, 2:25 AM Gwen Shapira  wrote:
>
> > Hey,
> >
> > I was surprised to discover that MirrorMaker 2.0 (part of 2.4
> > release), is not mentioned at all in our documentation
> > (https://kafka.apache.org/documentation/).
> >
> > Ryanne Dolan was kind enough to point me to the documentation for this
> > feature (
> > https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md)
> > and mentioned that the Operations docs will be updated when MM1 is
> > deprecated.
> >
> > IMO, this is a disservice to our users. If we already know that we
> > plan to deprecate MirrorMaker and replace it, we need to let users
> > know that an alternative exists, that they should start planning a
> > transition and how to use the cool new feature. And we need to do it
> > in the place where our users normally go to learn how to use our
> > product.
> >
> > As it stands now, the README doesn't even show up on Google search
> > results, so discovering how to use MirrorMaker 2.0 is nearly
> > impossible. How can we expect to deprecate 1.0 if users can't find out
> > about 2.0?
> >
> > In the past, we've documented new features before existing features
> > were deprecated (--broker-list and --bootstrap were documented before
> > --zookeeper was removed). I suggest we'll stick to this precedent.
> >
> > Gwen
> >
>

[jira] [Created] (KAFKA-9327) GroupMetadata metrics are not documented

[Gwen Shapira (Jira)]

Gwen Shapira created KAFKA-9327:
---

 Summary: GroupMetadata metrics are not documented
 Key: KAFKA-9327
 URL: https://issues.apache.org/jira/browse/KAFKA-9327
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira


GroupMetadataManager includes quite a few gauges that look very useful! 
NumGroups, NumGroupsPreparingRebalance, NumGroupsStable, NumGroupsDead, 
NumGroupsEmpty.

I couldn't find them in our site-docs, but they seem quite useful and will be 
nice to include. 

LastStableOffsetLag is also missing, but I'm not sure if it is useful enough to 
include.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)