Re: [DISCUSS] KIP-1010: Topic Partition Quota

[Viktor Somogyi-Vass]

Hi Afshin,

We keep KIP discussions on d...@kafka.apache.org so please post this over
there too. I'll go over this later this week but devs usually monitor that
list more frequently and you'll have better chances of getting a reply
there.

Regards,
Viktor

On Wed, Jan 17, 2024 at 12:03 AM Afshin Moazami
 wrote:

> Hi folks,
> I am not sure what is the KIP life-cycle and how we can get more attention
> on them, so I just reply to this thread with the hope to get some
> discussion started.
>
> Thanks,
> Afshin
>
> On Mon, Dec 11, 2023 at 10:43 AM Afshin Moazami 
> wrote:
>
> > Hi folks,
> > I would like to propose a new feature to extend the quota management in
> > Kafka to support topic-partition based quotas. The following is the link
> to
> > the KIP
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota
> >
> >
> > Best,
> > Afshin Moazami
> >
>

Re: [PROPOSAL] Add commercial support page on website

[Viktor Somogyi-Vass]

Hi Romain,

Yes, you may be right that it's either everybody or nobody. Coming up with
any levels of entry is going to raise questions as we see :).
I wasn't aware that my idea might be against ASF guidelines as it is
somewhat based on its own sponsorship page (
https://www.apache.org/foundation/sponsors). Anyway, my point is moot if
it's not allowed by those guidelines.

Regarding your original suggestion, I'm not against it, although I am
worried about that it might get out of hand, however I also see that we
don't really have any means to anticipate if it does or not, so it might be
that the only way is to try it out if the community leans toward it.

Thanks,
Viktor


On Mon, Jan 15, 2024 at 4:15 PM Romain Manni-Bucau 
wrote:

> Hi Viktor,
>
> Having a support page is ASF friendly and quite common for the reasons
> mentionned in the ticket and these threads and aim at helping end users.
>
> I saw that Matthias was afraid of the maintenance of such a page but I
> guess it costs nothing to merge François PR - or alike, accept the coming
> proposals for a few months and see after some period if it stays something
> moving a lot and if it becomes something the community can't handle - other
> projects kind of show it is not the case and some projects are way more
> used and supported than Kafka in terms of potential entries count - please
> don't take it as an offense it is just an history and scope point - so
> means they would have had the issue which is not what happent AFAIK.
>
> However please note that trying to promote cloudera, for example, because
> it contributes often to Kafka and reject Yupiik because it just takes the
> support side for now would be quite unfair on one side and strictly
> speaking this would be against ASF policy (you can reread
> https://www.apache.org/foundation/policies/conduct for example) so the
> question is quite simple for me: it is everybody or nobody and if you pick
> nobody, do you accept to make it harder for users to embrace Kafka (even if
> it would be ok from an ASF standpoint, the spirit would be more to enable
> that strictly speaking cause support is just part of the real life of such
> a complex project). I really take the "can be hard to handle" point
> seriously but trying that for 6 months or so does not cost much to the
> community and if at the end it helps some users it is only a win IMHO.
>
> So overall my 2cts would be to try based on other projects model and if it
> turns out Kafka is specific and it does not work for any reason, the page
> can be dropped quite easily.
>
> Wdyt?
>
> Best,
>
> On 2024/01/15 14:35:15 Viktor Somogyi-Vass wrote:
> > Hi all,
> >
> > I think that making and updating such a list implies tricky questions
> that
> > we have to deal with. Good examples were given by Matthias, that was my
> > first thought as well before reading his response.
> >
> > On the other hand I think it would be a good alternative to create a list
> > of companies that are either regular or meaningful contributors to the
> > project, provide other help such as testing infra or sponsors of the
> > project in other ways. This may present those companies who provide
> support
> > too and I think it'd be some appreciation as well to display them.
> >
> > What do you all think?
> >
> > Best,
> > Viktor
> >
> >
> > On Mon, Jan 15, 2024 at 10:08 AM Francois Papon <
> > francois.pa...@openobject.fr> wrote:
> >
> > > Hi Tison,
> > >
> > > Publishing a dedicated website for that can be a good idea, however if
> > > the link of the website could not be mention in the official Apache
> > > Kafka website I'm afraid that it will not be relevant.
> > >
> > > BTW, as I understand after all the feedback of the Apache Kafka PMC and
> > > community, my proposal is not a good idea for the project so I will
> > > close the PR.
> > >
> > > Thanks all for the feedback.
> > >
> > > regards,
> > >
> > > François
> > >
> > > On 14/01/2024 12:56, tison wrote:
> > > > FWIW - even if it's rejected by the Kafka PMC, you can maintain your
> > > > own page for such information and provide your personal comments on
> > > > them. If the object is to provide information and help users to make
> > > > decisions, it should help. Although you should do the SEO by
> yourself,
> > > > if the information is somehow neutral and valuable, you can ask the
> > > > @apachekafka Twitter (X) account to propagate it and provide a blog
> > > > for Kafka blogs.
> > > >
> > > > This 

Re: [PROPOSAL] Add commercial support page on website

[Viktor Somogyi-Vass]

Hi all,

I think that making and updating such a list implies tricky questions that
we have to deal with. Good examples were given by Matthias, that was my
first thought as well before reading his response.

On the other hand I think it would be a good alternative to create a list
of companies that are either regular or meaningful contributors to the
project, provide other help such as testing infra or sponsors of the
project in other ways. This may present those companies who provide support
too and I think it'd be some appreciation as well to display them.

What do you all think?

Best,
Viktor


On Mon, Jan 15, 2024 at 10:08 AM Francois Papon <
francois.pa...@openobject.fr> wrote:

> Hi Tison,
>
> Publishing a dedicated website for that can be a good idea, however if
> the link of the website could not be mention in the official Apache
> Kafka website I'm afraid that it will not be relevant.
>
> BTW, as I understand after all the feedback of the Apache Kafka PMC and
> community, my proposal is not a good idea for the project so I will
> close the PR.
>
> Thanks all for the feedback.
>
> regards,
>
> François
>
> On 14/01/2024 12:56, tison wrote:
> > FWIW - even if it's rejected by the Kafka PMC, you can maintain your
> > own page for such information and provide your personal comments on
> > them. If the object is to provide information and help users to make
> > decisions, it should help. Although you should do the SEO by yourself,
> > if the information is somehow neutral and valuable, you can ask the
> > @apachekafka Twitter (X) account to propagate it and provide a blog
> > for Kafka blogs.
> >
> > This is the common way how third-party "evangelist" producing content
> > and get it promoted.
> >
> > Best,
> > tison.
> >
> > Matthias J. Sax  于2024年1月13日周六 07:35写道：
> >> François,
> >>
> >> thanks for starting this initiative. Personally, I don't think it's
> >> necessarily harmful for the project to add such a new page, however, I
> >> share the same concerns others raised already.
> >>
> >> I understand your motivation that people had issues finding commercial
> >> support, but I am not sure we can address this issue that way. I am also
> >> "worried" (for the lack of a better word) that the page might become
> >> long an unwieldy. In the end, any freelancer/consultant offering Kafka
> >> services would be able to get on the page, so we might get hundreds of
> >> entries, what also makes it impossible for users to find what they are
> >> looking for. Also, the services of different companies might vary
> >> drastically; should users read all these descriptions? I can also
> >> imagine that some companies offer their services only in some
> >> countries/regions making it even harder for user to find what they are
> >> looking for?
> >>
> >> Overall, it sounds more like a search optimization problem, and thus it
> >> seems out-of-scope what we can solve. As I said, I am not strictly
> >> against it, but I just don't see much value either.
> >>
> >>
> >> -Matthias
> >>
> >> On 1/11/24 12:55 PM, Francois Papon wrote:
> >>> Hi Justine,
> >>>
> >>> You're right, Kafka is a part of my business (training, consulting,
> >>> architecture design, sla...) and most of the time, users/customers said
> >>> that it was hard for them to find a commercial support (in France for
> my
> >>> case) after searching on the Kafka website (Google didn't help them).
> >>>
> >>> As an ASF member and PMC of several ASF projects, I know that this kind
> >>> of page exist so this is why I made this proposal for the Kafka project
> >>> because I really think that it can help users.
> >>>
> >>> As you suggest, I can submit a PR to be added on the "powered by" page.
> >>>
> >>> Thanks,
> >>>
> >>> François
> >>>
> >>> On 11/01/2024 21:00, Justine Olshan wrote:
>  Hey François,
> 
>  My point was that the companies on that page use kafka as part of
> their
>  business. If you use Kafka as part of your business feel free to
> submit a
>  PR to be added.
> 
>  I second Chris's point that other projects are not enough to require
>  Kafka
>  having such a support page.
> 
>  Justine
> 
>  On Thu, Jan 11, 2024 at 11:57 AM Chris Egerton <
> fearthecel...@gmail.com>
>  wrote:
> 
> > Hi François,
> >
> > Is it an official policy of the ASF that projects provide a listing
> of
> > commercial support options for themselves? I understand that other
> > projects
> > have chosen to provide one, but this doesn't necessarily imply that
> all
> > projects should do the same, and I can't say I find this point very
> > convincing as a rebuttal to some of the good-faith concerns raised by
> > the
> > PMC and members of the community so far. However, if there's an
> official
> > ASF stance on this topic, then I acknowledge that Apache Kafka should
> > align
> > with it.
> >
> > Best,
> >
> > Chris
> >
> >
> > On Thu, 

Re: [ANNOUNCE] Apache Kafka 3.6.0

[Viktor Somogyi-Vass]

Thanks for the release Satish! :)

On Wed, Oct 11, 2023, 09:30 Bruno Cadonna  wrote:

> Thanks for the release, Satish!
>
> Best,
> Bruno
>
> On 10/11/23 8:29 AM, Luke Chen wrote:
> > Thanks for running the release, Satish!
> >
> > BTW, 3.6.0 should be a major release, not a minor one. :)
> >
> > Luke
> >
> > On Wed, Oct 11, 2023 at 1:39 PM Satish Duggana 
> wrote:
> >
> >> The Apache Kafka community is pleased to announce the release for
> >> Apache Kafka 3.6.0
> >>
> >> This is a minor release and it includes fixes and improvements from 238
> >> JIRAs.
> >>
> >> All of the changes in this release can be found in the release notes:
> >> https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html
> >>
> >> An overview of the release can be found in our announcement blog post:
> >> https://kafka.apache.org/blog
> >>
> >> You can download the source and binary release (Scala 2.12 and Scala
> 2.13)
> >> from:
> >> https://kafka.apache.org/downloads#3.6.0
> >>
> >>
> >>
> ---
> >>
> >>
> >> Apache Kafka is a distributed streaming platform with four core APIs:
> >>
> >>
> >> ** The Producer API allows an application to publish a stream of
> records to
> >> one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output stream to one or more output topics, effectively transforming the
> >> input streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might
> >> capture every change to a table.
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react
> >> to the streams of data.
> >>
> >>
> >> Apache Kafka is in use at large and small companies worldwide, including
> >> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> >> Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >>
> >> A big thank you for the following 139 contributors to this release!
> >> (Please report an unintended omission)
> >>
> >> This was a community effort, so thank you to everyone who contributed
> >> to this release, including all our users and our 139 contributors:
> >> A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
> >> Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
> >> Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
> >> Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
> >> bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
> >> Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
> >> Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
> >> Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
> >> David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
> >> Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
> >> Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
> >> Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
> >> Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
> >> iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
> >> Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
> >> Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
> >> Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
> >> Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
> >> Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> >> Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax,
> >> maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab),
> >> Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada
> >> Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven
> >> Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall
> >> Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron
> >> Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana,
> >> sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay
> >> Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent
> >> Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng,
> >> 蓝士钦
> >>
> >> We welcome your help and feedback. For more information on 

Re: [ANNOUNCE] New committer: Josep Prat

[Viktor Somogyi-Vass]

Congrats Josep!

On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:

> Congrats!
>
> On 12/20/22 12:01 PM, Josep Prat wrote:
> > Thank you all!
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> >
> >> Congratulations Josep!
> >>
> >> -Bill
> >>
> >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> Congratulations Josep!
> >>>
> >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna 
> >> wrote:
> 
>  Congrats, Josep!
> 
>  Well deserved!
> 
>  Best,
>  Bruno
> 
>  On 20.12.22 18:40, Kirk True wrote:
> > Congrats Josep!
> >
> > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya wrote:
> >> Congrats Josep!!
> >>
> >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> >>  
> >> wrote:
> >>
> >>> Congratulations Josep!
> >>>
> >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> >>> fearthecel...@gmail.com>
> >>> wrote:
> >>>
>  Congrats Josep! Well-earned.
> 
>  On Tue, Dec 20, 2022, 12:26 Jun Rao 
> >>> wrote:
> 
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka
> >>> committer
>  Josep
> >Prat.
> >
> > Josep has been contributing to Kafka since May 2021. He
> >>> contributed 20
>  PRs
> > including the following 2 KIPs.
> >
> > KIP-773 Differentiate metric latency measured in ms and ns
> > KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface
> >>> with
> > internal implementation
> >
> > Congratulations, Josep!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
> >
> 
> >>>
> >>
> >
> >>>
> >>
> >
>

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Viktor Somogyi-Vass]

Thanks again everyone!

On Fri, Dec 16, 2022, 18:36 Bill Bejeck  wrote:

> Congratulations, Viktor!
>
> -Bill
>
> On Fri, Dec 16, 2022 at 12:32 PM Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 12/15/22 7:10 AM, Rajini Sivaram wrote:
> > > Congratulations, Viktor!
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Thu, Dec 15, 2022 at 11:41 AM Ron Dagostino 
> > wrote:
> > >
> > >> Congrats to you too, Victor!
> > >>
> > >> Ron
> > >>
> > >>> On Dec 15, 2022, at 4:59 AM, Viktor Somogyi-Vass <
> > >> viktor.somo...@cloudera.com.invalid> wrote:
> > >>>
> > >>> Thank you everyone! :)
> > >>>
> > >>>> On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison <
> > >> mickael.mai...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>> Congratulations Viktor!
> > >>>>
> > >>>>> On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
> > >>>>>  wrote:
> > >>>>>
> > >>>>> Congratulations, Viktor!
> > >>>>
> > >>
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Viktor Somogyi-Vass]

Congrats Sophie!

On Tue, Aug 2, 2022 at 8:51 AM Josep Prat 
wrote:

> Congrats Sophie!
>
> On Tue, Aug 2, 2022 at 5:36 AM pengyh  wrote:
>
> >
> > congrats!
> >
> > > Congrats Sophie! 
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Manager, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Immanuelkirchstraße 26, 10405 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Viktor Somogyi-Vass]

Congrats Chris!

On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
 wrote:

> Congratulations!
>
> --
> Matthew de Detrich
> Aiven Deutschland GmbH
> Immanuelkirchstraße 26, 10405 Berlin
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> m: +491603708037
> w: aiven.io e: matthew.dedetr...@aiven.io
> On 25. Jul 2022, 18:26 +0200, Mickael Maison , wrote:
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > we are excited to announce that he accepted!
> >
> > Chris has been contributing to Kafka since 2017. He has made over 80
> > commits mostly around Kafka Connect. His most notable contributions
> > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > Exactly-Once Support for Source Connectors.
> >
> > He has been an active participant in discussions and reviews on the
> > mailing lists and on Github.
> >
> > Thanks for all of your contributions Chris. Congratulations!
> >
> > -- Mickael, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New committer: Mickael Maison

[Viktor Somogyi-Vass]

Congrats Mickael!! :)

On Fri, Nov 8, 2019 at 1:24 PM Satish Duggana 
wrote:

> Congratulations Mickael!!
>
> On Fri, Nov 8, 2019 at 2:50 PM Rajini Sivaram 
> wrote:
> >
> > Congratulations, Mickael, well deserved!!
> >
> > Regards,
> >
> > Rajini
> >
> > On Fri, Nov 8, 2019 at 9:08 AM David Jacot  wrote:
> >
> > > Congrats Mickeal, well deserved!
> > >
> > > On Fri, Nov 8, 2019 at 8:56 AM Tom Bentley 
> wrote:
> > >
> > > > Congratulations Mickael!
> > > >
> > > > On Fri, Nov 8, 2019 at 6:41 AM Vahid Hashemian <
> > > vahid.hashem...@gmail.com>
> > > > wrote:
> > > >
> > > > > Congrats Mickael,
> > > > >
> > > > > Well deserved!
> > > > >
> > > > > --Vahid
> > > > >
> > > > > On Thu, Nov 7, 2019 at 9:10 PM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations Mickael!
> > > > > >
> > > > > > On Thu, Nov 7, 2019 at 8:27 PM Manikumar <
> manikumar.re...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats Mickeal!
> > > > > > >
> > > > > > > On Fri, Nov 8, 2019 at 9:05 AM Dong Lin 
> > > wrote:
> > > > > > >
> > > > > > > > Congratulations Mickael!
> > > > > > > >
> > > > > > > > On Thu, Nov 7, 2019 at 1:38 PM Jun Rao 
> wrote:
> > > > > > > >
> > > > > > > > > Hi, Everyone,
> > > > > > > > >
> > > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > > committer
> > > > > > > > > Mickael
> > > > > > > > > Maison.
> > > > > > > > >
> > > > > > > > > Mickael has been contributing to Kafka since 2016. He
> proposed
> > > > and
> > > > > > > > > implemented multiple KIPs. He has also been propomating
> Kafka
> > > > > through
> > > > > > > > blogs
> > > > > > > > > and public talks.
> > > > > > > > >
> > > > > > > > > Congratulations, Mickael!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > >
> > >
>

Re: [VOTE] 2.2.1 RC1

[Viktor Somogyi-Vass]

+1 (non-binding)

1. Ran unit tests
2. Ran some basic automatic end-to-end tests over plaintext and SSL too
3. Ran systests sanity checks

Viktor

On Thu, May 23, 2019 at 6:04 PM Harsha  wrote:

> +1 (binding)
>
> 1. Ran unit tests
> 2. System tests
> 3. 3 node cluster with few manual tests.
>
> Thanks,
> Harsha
>
> On Wed, May 22, 2019, at 8:09 PM, Vahid Hashemian wrote:
> > Bumping this thread to get some more votes, especially from committers,
> so
> > we can hopefully make a decision on this RC by the end of the week.
> >
> > Thanks,
> > --Vahid
> >
> > On Mon, May 13, 2019 at 8:15 PM Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 2.2.1.
> > >
> > > Compared to RC0, this release candidate also fixes the following
> issues:
> > >
> > >- [KAFKA-6789] - Add retry logic in AdminClient requests
> > >- [KAFKA-8348] - Document of kafkaStreams improvement
> > >- [KAFKA-7633] - Kafka Connect requires permission to create
> internal
> > >topics even if they exist
> > >- [KAFKA-8240] - Source.equals() can fail with NPE
> > >- [KAFKA-8335] - Log cleaner skips Transactional mark and batch
> > >record, causing unlimited growth of __consumer_offsets
> > >- [KAFKA-8352] - Connect System Tests are failing with 404
> > >
> > > Release notes for the 2.2.1 release:
> > > https://home.apache.org/~vahid/kafka-2.2.1-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Thursday, May 16, 9:00 pm PT.
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~vahid/kafka-2.2.1-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~vahid/kafka-2.2.1-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 2.2 branch) is the 2.2.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.2.1-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/22/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/22/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.2 branch:
> > > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.2-jdk8/115/
> > >
> > > Thanks!
> > > --Vahid
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >
>

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

[Viktor Somogyi-Vass]

Congrats Matthias!

On Tue, Apr 23, 2019 at 4:24 AM Becket Qin  wrote:

> Congrats, Matthias!
>
> On Sat, Apr 20, 2019 at 10:28 AM Matthias J. Sax 
> wrote:
>
> > Thank you all!
> >
> > -Matthias
> >
> >
> > On 4/19/19 3:58 PM, Lei Chen wrote:
> > > Congratulations Matthias! Well deserved!
> > >
> > > -Lei
> > >
> > > On Fri, Apr 19, 2019 at 2:55 PM James Cheng  > > > wrote:
> > >
> > > Congrats!!
> > >
> > > -James
> > >
> > > Sent from my iPhone
> > >
> > > > On Apr 18, 2019, at 2:35 PM, Guozhang Wang  > > > wrote:
> > > >
> > > > Hello Everyone,
> > > >
> > > > I'm glad to announce that Matthias J. Sax is now a member of
> Kafka
> > > PMC.
> > > >
> > > > Matthias has been a committer since Jan. 2018, and since then he
> > > continued
> > > > to be active in the community and made significant contributions
> > the
> > > > project.
> > > >
> > > >
> > > > Congratulations to Matthias!
> > > >
> > > > -- Guozhang
> > >
> >
> >
>

Re: [ANNOUNCE] New Committer: Randall Hauch

[Viktor Somogyi-Vass]

Congrats Randall! :)

On Fri, Feb 15, 2019 at 10:15 AM Satish Duggana 
wrote:

> Congratulations Randall!
>
> On Fri, Feb 15, 2019 at 1:51 PM Mickael Maison 
> wrote:
> >
> > Congrats Randall!
> >
> > On Fri, Feb 15, 2019 at 6:37 AM James Cheng 
> wrote:
> > >
> > > Congrats, Randall! Well deserved!
> > >
> > > -James
> > >
> > > Sent from my iPhone
> > >
> > > > On Feb 14, 2019, at 6:16 PM, Guozhang Wang 
> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > The PMC of Apache Kafka is happy to announce another new committer
> joining
> > > > the project today: we have invited Randall Hauch as a project
> committer and
> > > > he has accepted.
> > > >
> > > > Randall has been participating in the Kafka community for the past 3
> years,
> > > > and is well known as the founder of the Debezium project, a popular
> project
> > > > for database change-capture streams using Kafka (https://debezium.io).
> More
> > > > recently he has become the main person keeping Kafka Connect moving
> > > > forward, participated in nearly all KIP discussions and QAs on the
> mailing
> > > > list. He's authored 6 KIPs and authored 50 pull requests and
> conducted over
> > > > a hundred reviews around Kafka Connect, and has also been
> evangelizing
> > > > Kafka Connect at several Kafka Summit venues.
> > > >
> > > >
> > > > Thank you very much for your contributions to the Connect community
> Randall
> > > > ! And looking forward to many more :)
> > > >
> > > >
> > > > Guozhang, on behalf of the Apache Kafka PMC
>

Re: [VOTE] 2.1.1 RC1

[Viktor Somogyi-Vass]

Hi,

Ran the ducktapes but the streams upgrade tests failed because the dev
version was not updated. This will be fixed in
https://github.com/apache/kafka/pull/6217. Once it's merged I'll rerun them.

Viktor

On Wed, 30 Jan 2019, 22:19 Jonathan Santilli  Hello,
>
> I have downloaded the source code for tag *2.1.1-rc1* (667980043). Executed
> integration and unit tests:
>
> *BUILD SUCCESSFUL in 25m 34s*
> *136 actionable tasks: 133 executed, 3 up-to-date*
>
>
> Also, I have downloaded the artifact from
> http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/kafka_2.11-2.1.1.tgz and
> ran a cluster of 3 Brokers.
> Kept Kafka-monitor running for about 1 hour without any issues.
>
> +1
>
> Thanks for the release Colin.
> --
> Jonathan Santilli
>
>
> On Wed, Jan 30, 2019 at 8:18 PM Eno Thereska 
> wrote:
>
> > I couldn't repro locally, that was on an m3.large. And it's not happening
> > anymore. Might be a transient issue.
> >
> > Thanks,
> > Eno
> >
> > On Wed, Jan 30, 2019 at 6:46 PM Colin McCabe  wrote:
> >
> > > (+all lists)
> > >
> > > Hi Eno,
> > >
> > > Thanks for testing this.
> > >
> > > Those tests passed in the Jenkins build we did here:
> > > https://builds.apache.org/job/kafka-2.1-jdk8/118/
> > >
> > > Perhaps there is an environment issue at play here?  Do you get the
> same
> > > failures running those tests on the 2.1 release?
> > >
> > > Best,
> > > Colin
> > >
> > > On Wed, Jan 30, 2019, at 09:11, Eno Thereska wrote:
> > > > Hi Colin,
> > > >
> > > > I've been running the tests and so far I get the following failures.
> > Are
> > > > they known?
> > > >
> > > > kafka.server.ReplicaManagerQuotasTest >
> > > shouldGetBothMessagesIfQuotasAllow
> > > > FAILED
> > > > kafka.server.ReplicaManagerQuotasTest >
> > > > testCompleteInDelayedFetchWithReplicaThrottling FAILED
> > > > kafka.server.ReplicaManagerQuotasTest >
> > > > shouldExcludeSubsequentThrottledPartitions FAILED
> > > > kafka.server.ReplicaManagerQuotasTest >
> > > > shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions FAILED
> > > > kafka.server.ReplicaManagerQuotasTest >
> > > > shouldIncludeInSyncThrottledReplicas FAILED
> > > >
> > > > Thanks
> > > > Eno
> > > >
> > > > On Sun, Jan 27, 2019 at 9:46 PM Colin McCabe 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > This is the second candidate for release of Apache Kafka 2.1.1.
> This
> > > > > release includes many bug fixes for Apache Kafka 2.1.
> > > > >
> > > > > Compared to rc0, this release includes the following changes:
> > > > > * MINOR: Upgrade ducktape to 0.7.5 (#6197)
> > > > > * KAFKA-7837: Ensure offline partitions are picked up as soon as
> > > possible
> > > > > when shrinking ISR
> > > > > * tests/kafkatest/__init__.py now contains __version__ = '2.1.1'
> > rather
> > > > > than '2.1.1.dev0'
> > > > > * Maven artifacts should be properly staged this time
> > > > > * I have added my GPG key to https://kafka.apache.org/KEYS
> > > > >
> > > > > Check out the release notes here:
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/RELEASE_NOTES.html
> > > > >
> > > > > The vote will go until Friday, February 1st.
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/2.1.1-rc1
> > > > >
> > > > > * Successful Jenkins builds for the 2.1 branch:
> > > > > Unit/integration tests:
> > > https://builds.apache.org/job/kafka-2.1-jdk8/118/
> > > > >
> > > > > thanks,
> > > > > Colin
> > > > >
> > > >
> > >
> >
>
>
> --
> Santilli Jonathan
>

Re: [ANNOUNCE] New Committer: Manikumar Reddy

[Viktor Somogyi-Vass]

Congratulations Manikumar, well deserved!

On Fri, 12 Oct 2018, 06:30 Andras Beni, 
wrote:

> Congratulations, Manikumar!
>
> Srinivas Reddy  ezt írta (időpont: 2018. okt.
> 12., P 3:00):
>
> > Congratulations Mani. We'll deserved 
> >
> > -
> > Srinivas
> >
> > - Typed on tiny keys. pls ignore typos.{mobile app}
> >
> > On Fri 12 Oct, 2018, 01:39 Jason Gustafson,  wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Manikumar Reddy as a committer and
> > we
> > > are
> > > pleased to announce that he has accepted!
> > >
> > > Manikumar has contributed 134 commits including significant work to add
> > > support for delegation tokens in Kafka:
> > >
> > > KIP-48:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > KIP-249
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+KafkaKIP-249
> > >
> > > :
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
> > >
> > > He has broad experience working with many of the core components in
> Kafka
> > > and he has reviewed over 80 PRs. He has also made huge progress
> > addressing
> > > some of our technical debt.
> > >
> > > We appreciate the contributions and we are looking forward to more.
> > > Congrats Manikumar!
> > >
> > > Jason, on behalf of the Apache Kafka PMC
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Viktor Somogyi-Vass]

Congrats Dong! :)

On Tue, Aug 21, 2018 at 10:09 AM James Cheng  wrote:

> Congrats Dong!
>
> -James
>
> > On Aug 20, 2018, at 3:54 AM, Ismael Juma  wrote:
> >
> > Hi everyone,
> >
> > Dong Lin became a committer in March 2018. Since then, he has remained
> > active in the community and contributed a number of patches, reviewed
> > several pull requests and participated in numerous KIP discussions. I am
> > happy to announce that Dong is now a member of the
> > Apache Kafka PMC.
> >
> > Congratulation Dong! Looking forward to your future contributions.
> >
> > Ismael, on behalf of the Apache Kafka PMC
>
>

Re: 答复: [ANNOUNCE] New Committer: Dong Lin

[Viktor Somogyi]

Congrats Dong! :)

On Thu, Mar 29, 2018 at 2:12 PM, Satish Duggana 
wrote:

> Congratulations Dong!
>
>
> On Thu, Mar 29, 2018 at 5:12 PM, Sandor Murakozi 
> wrote:
>
> > Congrats, Dong!
> >
> >
> > On Thu, Mar 29, 2018 at 2:15 AM, Dong Lin  wrote:
> >
> > > Thanks everyone!!
> > >
> > > It is my great pleasure to be part of the Apache Kafka community and
> help
> > > make Apache Kafka more useful to its users. I am super excited to be a
> > > Kafka committer and I am hoping to contribute more to its design,
> > > implementation and review etc in the future.
> > >
> > > Thanks!
> > > Dong
> > >
> > > On Wed, Mar 28, 2018 at 4:04 PM, Hu Xi  wrote:
> > >
> > > > Congrats, Dong Lin!
> > > >
> > > >
> > > > 
> > > > 发件人: Matthias J. Sax 
> > > > 发送时间: 2018年3月29日 6:37
> > > > 收件人: users@kafka.apache.org; d...@kafka.apache.org
> > > > 主题: Re: [ANNOUNCE] New Committer: Dong Lin
> > > >
> > > > Congrats!
> > > >
> > > > On 3/28/18 1:16 PM, James Cheng wrote:
> > > > > Congrats, Dong!
> > > > >
> > > > > -James
> > > > >
> > > > >> On Mar 28, 2018, at 10:58 AM, Becket Qin 
> > > wrote:
> > > > >>
> > > > >> Hello everyone,
> > > > >>
> > > > >> The PMC of Apache Kafka is pleased to announce that Dong Lin has
> > > > accepted
> > > > >> our invitation to be a new Kafka committer.
> > > > >>
> > > > >> Dong started working on Kafka about four years ago, since which he
> > has
> > > > >> contributed numerous features and patches. His work on Kafka core
> > has
> > > > been
> > > > >> consistent and important. Among his contributions, most
> noticeably,
> > > Dong
> > > > >> developed JBOD (KIP-112, KIP-113) to handle disk failures and to
> > > reduce
> > > > >> overall cost, added deleteDataBefore() API (KIP-107) to allow
> users
> > > > >> actively remove old messages. Dong has also been active in the
> > > > community,
> > > > >> participating in KIP discussions and doing code reviews.
> > > > >>
> > > > >> Congratulations and looking forward to your future contribution,
> > Dong!
> > > > >>
> > > > >> Jiangjie (Becket) Qin, on behalf of Apache Kafka PMC
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Viktor Somogyi]

Congratulations! :)

On Thu, Jan 18, 2018 at 10:38 AM, Rajini Sivaram 
wrote:

> Thanks everyone!
>
> Regards,
>
> Rajini
>
> On Thu, Jan 18, 2018 at 8:53 AM, Damian Guy  wrote:
>
> > Congratulations Rajini!
> >
> > On Thu, 18 Jan 2018 at 00:57 Hu Xi  wrote:
> >
> > > Congratulations, Rajini Sivaram.  Very well deserved!
> > >
> > >
> > > 
> > > 发件人: Konstantine Karantasis 
> > > 发送时间: 2018年1月18日 6:23
> > > 收件人: d...@kafka.apache.org
> > > 抄送: users@kafka.apache.org
> > > 主题: Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
> > >
> > > Congrats Rajini!
> > >
> > > -Konstantine
> > >
> > > On Wed, Jan 17, 2018 at 2:18 PM, Becket Qin 
> > wrote:
> > >
> > > > Congratulations, Rajini!
> > > >
> > > > On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > Congratulations Rajini!
> > > > >
> > > > > On 17 Jan 2018 10:49 am, "Gwen Shapira"  wrote:
> > > > >
> > > > > Dear Kafka Developers, Users and Fans,
> > > > >
> > > > > Rajini Sivaram became a committer in April 2017.  Since then, she
> > > > remained
> > > > > active in the community and contributed major patches, reviews and
> > KIP
> > > > > discussions. I am glad to announce that Rajini is now a member of
> the
> > > > > Apache Kafka PMC.
> > > > >
> > > > > Congratulations, Rajini and looking forward to your future
> > > contributions.
> > > > >
> > > > > Gwen, on behalf of Apache Kafka PMC
> > > > >
> > > >
> > >
> >
>

Re: Kafka in virtualized environments

[Viktor Somogyi]

@Girish, wow, that could be a nice issue to debug. I was thinking about
exactly these kind of issues with virtualized environments.

@Wim, how did you overcome the problem?
Thinking about such issues my first thoughts are increasing the VM's memory
that can be utilized to read/write caching by the OS or using smaller
segments so it won't sync a big chunk of data at once (by possibly switching
to synchronized
<https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/>
from async) but more smaller ones.

On Fri, Dec 1, 2017 at 2:08 AM, Girish Aher <girisha...@gmail.com> wrote:

> I am no storage or ESX expert, what I was told by our storage folks is that
> they essentially created a dedicated storage pool in the SAN for zookeeper
> VMs plus other VMs that did not have a lot of IO activity (non DB VMs). I
> assume that implies dedicated physical disks in the SAN for that pool.
>
> I am not sure if a dedicated datastore was created in ESX for this pool, I
> am guessing they did.
> I have not seen the issue since then.
>
> Of course, the best solution is to have zookeeper on their own physicals
> and dedicated disks especially if you plan to use it for purposes in
> addition to Kafka.
>
> Also want to mention that a *temporary* solution around this problem is to
> increase the connection and session timeouts between Kafka and zookeeper.
>
>
> On Thu, Nov 30, 2017 at 2:33 PM, Sean Glover <sean.glo...@lightbend.com>
> wrote:
>
> > Giresh, I'm curious what your solution was.  Did you use locally attached
> > storage for your ZK ensemble?  Did you move it to static machines?
> >
> > On Thu, Nov 30, 2017 at 4:50 PM, John Yost <hokiege...@gmail.com> wrote:
> >
> > > Great point by Girish--its the delays of syncing with Zookeeper that
> are
> > > particularly problematic. Moreover, Zookeeper sync delays and session
> > > timeouts impact other systems as well such as Storm.
> > >
> > > --John
> > >
> > > On Thu, Nov 30, 2017 at 10:14 AM, Girish Aher <girisha...@gmail.com>
> > > wrote:
> > >
> > > > We did not face any problems with kafka application per se but we
> have
> > > > faced problems with zookeeper in virtualized environments due to
> > slowness
> > > > in fsyncs. We were using a shared SAN storage with shared pools with
> > > other
> > > > VMs. So every time, there was some kind of considerable storage
> > activity
> > > > like DB backup or something, our zookeeper fsyncs used to take tens
> of
> > > > seconds causing kafka-zookeeper sessions to timeout.
> > > >
> > > > On Nov 30, 2017 2:22 AM, "Viktor Somogyi" <viktorsomo...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Recently I bumped into an interesting question: using kafka in
> > > > virtualized
> > > > > environments, such as vmware. I'm not really familiar with
> > > virtualization
> > > > > in-depth (how disk virtualization works, what are the OS level
> > supports
> > > > > etc.), therefore I think this is an interesting discussion from
> > Kafka's
> > > > > point. As far as I know Kafka is designed for a non-virtualized
> > > > environment
> > > > > mainly (although I haven't seen it explicitly anywhere) but
> thinking
> > of
> > > > > it's hard reliance on disk optimization I always assumed this.
> > > > >
> > > > > Anyone has experiences with virtualized Kafka? Are you aware of any
> > > pain
> > > > > points that people should consider (or performance issues)?
> > > > > Are there any publications on this topic?
> > > > >
> > > > > Regards,
> > > > > Viktor
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Senior Software Engineer, Lightbend, Inc.
> >
> > <http://lightbend.com>
> >
> > @seg1o <https://twitter.com/seg1o>
> >
>

Kafka in virtualized environments

[Viktor Somogyi]

Hi folks,

Recently I bumped into an interesting question: using kafka in virtualized
environments, such as vmware. I'm not really familiar with virtualization
in-depth (how disk virtualization works, what are the OS level supports
etc.), therefore I think this is an interesting discussion from Kafka's
point. As far as I know Kafka is designed for a non-virtualized environment
mainly (although I haven't seen it explicitly anywhere) but thinking of
it's hard reliance on disk optimization I always assumed this.

Anyone has experiences with virtualized Kafka? Are you aware of any pain
points that people should consider (or performance issues)?
Are there any publications on this topic?

Regards,
Viktor

Re: What are reasonable limits for max number of consumer groups per partition and per broker?

[Viktor Somogyi]

Hi Jeff,

I think it's also worth considering that 1K consumer (implying that you
have at least 1 consumer per group) for 1 partition means 1K TCP connection
which means that they have to share the available bandwidth.
Why do you have so many consumer groups? Do you basically want to multicast?

Viktor


On Tue, Nov 14, 2017 at 4:18 PM, Arunkumar 
wrote:

> Hi Jeff
> Number of partition depends on number of consumers on that particular
> consumer group. So you may have to create your partitions based on that.
>
> ThanksArunkumar Pichaimuthu, PMP
>
> On Monday, November 13, 2017, 5:25:35 PM CST, Jeff Widman <
> j...@jeffwidman.com> wrote:
>
>  We're considering an architecture that would result in 5K-10K consumer
> groups consuming from a single topic that has one partition.
>
> What are the reasonable limits for the max number of consumer groups per
> partition and per broker?
>
> Can a single broker be the group coordinator for 1K+ consumer groups?
>
> --
>
> *Jeff Widman*
> jeffwidman.com  | 740-WIDMAN-J (943-6265)
> <><
>
>

Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

[Viktor Somogyi]

I'm happy that it's solved :)

On Thu, Nov 9, 2017 at 3:32 PM, John Yost <hokiege...@gmail.com> wrote:

> Excellent points Viktor! Also, the reason I mistakenly went > 8 GB memory
> heap was due to OOM errors that were being thrown when I upgraded from
> 0.9.0.1 to 0.10.0.0 and forgot to explicitly set the message format to
> 0.9.0.1 because we needed to support the older clients and the
> corresponding format. once I set the message format to 0.9.0.1, the memory
> requirements went WAY down, I reset the memory heap to 6 GB, and our Kafka
> cluster has been awesome since.
>
> --John
>
> On Thu, Nov 9, 2017 at 9:09 AM, Viktor Somogyi <viktorsomo...@gmail.com>
> wrote:
>
> > Hi Json.
> >
> > John might have a point. It is not reasonable to have more than 6-8GB of
> > heap provided for the JVM that's running Kafka. One of the reason is GC
> > time and the other is that Kafka relies heavily on the OS' disk
> read/write
> > in-memory caching.
> > Also there were a few synchronization bugs in 0.9 which caused similar
> > problems. I would recommend you to upgrade to 1.0.0 if that is feasible.
> >
> > Viktor
> >
> >
> > On Thu, Nov 9, 2017 at 2:59 PM, John Yost <hokiege...@gmail.com> wrote:
> >
> > > I've seen this before and it was due to long GC pauses due in large
> part
> > to
> > > a memory heap > 8 GB.
> > >
> > > --John
> > >
> > > On Thu, Nov 9, 2017 at 8:17 AM, Json Tu <kafka...@126.com> wrote:
> > >
> > > > Hi,
> > > > we have a kafka cluster which is made of 6 brokers,  with 8 cpu
> and
> > > > 16G memory on each broker’s machine, and we have about 1600 topics in
> > the
> > > > cluster，about 1700 partitions’ leader and 1600 partitions' replica on
> > > each
> > > > broker.
> > > > when we restart a normal broke,  we find that there are 500+
> > > > partitions shrink and expand frequently when restart the broker，
> > > > there are many logs as below.
> > > >
> > > >[2017-11-09 17:05:51,173] INFO Partition [Yelp,5] on broker
> 4759726:
> > > > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:06:22,047] INFO Partition [Yelp,5] on broker 4759726:
> > > > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:06:28,634] INFO Partition [Yelp,5] on broker 4759726:
> > > > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:06:44,658] INFO Partition [Yelp,5] on broker 4759726:
> > > > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:06:47,611] INFO Partition [Yelp,5] on broker 4759726:
> > > > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:07:19,703] INFO Partition [Yelp,5] on broker 4759726:
> > > > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > > > (kafka.cluster.Partition)
> > > > [2017-11-09 17:07:26,811] INFO Partition [Yelp,5] on broker 4759726:
> > > > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > > > (kafka.cluster.Partition)
> > > > …
> > > >
> > > >
> > > > and repeat shrink and expand after 30 minutes which is the
> default
> > > > value of leader.imbalance.check.interval.seconds, and at that time
> > > > we can find the log of controller’s auto rebalance，which can leads
> some
> > > > partition’s leader change to this restarted broker.
> > > > we have no shrink and expand when our cluster is running except
> > when
> > > > we restart it，so replica.fetch.thread.num is 1，and it seems enough.
> > > >
> > > > we can reproduce it at each restart，can someone give some
> > > suggestions.
> > > > thanks before.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

[Viktor Somogyi]

Hi Json.

John might have a point. It is not reasonable to have more than 6-8GB of
heap provided for the JVM that's running Kafka. One of the reason is GC
time and the other is that Kafka relies heavily on the OS' disk read/write
in-memory caching.
Also there were a few synchronization bugs in 0.9 which caused similar
problems. I would recommend you to upgrade to 1.0.0 if that is feasible.

Viktor


On Thu, Nov 9, 2017 at 2:59 PM, John Yost  wrote:

> I've seen this before and it was due to long GC pauses due in large part to
> a memory heap > 8 GB.
>
> --John
>
> On Thu, Nov 9, 2017 at 8:17 AM, Json Tu  wrote:
>
> > Hi,
> > we have a kafka cluster which is made of 6 brokers,  with 8 cpu and
> > 16G memory on each broker’s machine, and we have about 1600 topics in the
> > cluster，about 1700 partitions’ leader and 1600 partitions' replica on
> each
> > broker.
> > when we restart a normal broke,  we find that there are 500+
> > partitions shrink and expand frequently when restart the broker，
> > there are many logs as below.
> >
> >[2017-11-09 17:05:51,173] INFO Partition [Yelp,5] on broker 4759726:
> > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > (kafka.cluster.Partition)
> > [2017-11-09 17:06:22,047] INFO Partition [Yelp,5] on broker 4759726:
> > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > (kafka.cluster.Partition)
> > [2017-11-09 17:06:28,634] INFO Partition [Yelp,5] on broker 4759726:
> > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > (kafka.cluster.Partition)
> > [2017-11-09 17:06:44,658] INFO Partition [Yelp,5] on broker 4759726:
> > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > (kafka.cluster.Partition)
> > [2017-11-09 17:06:47,611] INFO Partition [Yelp,5] on broker 4759726:
> > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > (kafka.cluster.Partition)
> > [2017-11-09 17:07:19,703] INFO Partition [Yelp,5] on broker 4759726:
> > Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
> > (kafka.cluster.Partition)
> > [2017-11-09 17:07:26,811] INFO Partition [Yelp,5] on broker 4759726:
> > Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
> > (kafka.cluster.Partition)
> > …
> >
> >
> > and repeat shrink and expand after 30 minutes which is the default
> > value of leader.imbalance.check.interval.seconds, and at that time
> > we can find the log of controller’s auto rebalance，which can leads some
> > partition’s leader change to this restarted broker.
> > we have no shrink and expand when our cluster is running except when
> > we restart it，so replica.fetch.thread.num is 1，and it seems enough.
> >
> > we can reproduce it at each restart，can someone give some
> suggestions.
> > thanks before.
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: Metrics for Kafka Connect

[Viktor Somogyi]

Hi Jeff,

There is a KIP going on, would this answer your question?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

Viktor

On Tue, Sep 12, 2017 at 2:38 PM, Jeff Klukas  wrote:

> The Kafka docs on Monitoring don't mention anything specific for Kafka
> Connect. Are metrics for connectors limited to just the standard
> consumer/producer metrics? Do I understand correctly that the Connect API
> doesn't provide any hooks for custom connector-specific metrics? If not, is
> that something that's on the roadmap at all?
>

Re: Reduce Kafka Client logging

[Viktor Somogyi]

Hi Raghav,

I think it is enough to raise the logging level
of org.apache.kafka.clients.producer.ProducerConfig to WARN in log4j.
Also I'd like to mention that if possible, don't recreate the Kafka
producer each time. The protocol is designed for long-living connections
and recreating the connection each time puts pressure on the TCP layer (the
connection is expensive) and also on Kafka as well which may result in
broker failures (typically exceeding the maximum allowed number of file
descriptors).

HTH,
Viktor

On Thu, Sep 7, 2017 at 7:35 AM, Raghav  wrote:

> Due to the nature of code, I have to open a connection to a different Kafka
> broker each time, and send one message. We have several Kafka brokers. So
> my client log is full with the following logs. What log settings should I
> use in log4j just for Kafka producer logs ?
>
>
> 17/09/07 04:44:04 INFO producer.ProducerConfig:180 ProducerConfig values:
> acks = all
> batch.size = 16384
> block.on.buffer.full = false
> bootstrap.servers = [10.10.10.5:]
> buffer.memory = 33554432
> client.id =
> compression.type = none
> connections.max.idle.ms = 54
> interceptor.classes = null
> key.serializer = class
> org.apache.kafka.common.serialization.StringSerializer
> linger.ms = 1
> max.block.ms = 5000
> max.in.flight.requests.per.connection = 5
> max.request.size = 1048576
> metadata.fetch.timeout.ms = 6
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> receive.buffer.bytes = 32768
> reconnect.backoff.ms = 50
> request.timeout.ms = 5000
> retries = 0
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 6
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.mechanism = GSSAPI
> security.protocol = PLAINTEXT
> send.buffer.bytes = 131072
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.endpoint.identification.algorithm = null
> ssl.key.password = null
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.location = null
> ssl.keystore.password = null
> ssl.keystore.type = JKS
> ssl.protocol = TLS
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.location = null
> ssl.truststore.password = null
> ssl.truststore.type = JKS
> timeout.ms = 3
> value.serializer = class
> org.apache.kafka.common.serialization.StringSerializer
>
> On Wed, Sep 6, 2017 at 9:37 PM, Jaikiran Pai 
> wrote:
>
> > Can you post the exact log messages that you are seeing?
> >
> > -Jaikiran
> >
> >
> >
> > On 07/09/17 7:55 AM, Raghav wrote:
> >
> >> Hi
> >>
> >> My Java code produces Kafka config overtime it does a send which makes
> log
> >> very very verbose.
> >>
> >> How can I reduce the Kafka client (producer) logging in my java code ?
> >>
> >> Thanks for your help.
> >>
> >>
> >
>
>
> --
> Raghav
>

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

[Viktor Somogyi]

Hi Chao,

Apart from the actual request, the consumer sends metadata requests as
well. What is actually going through the network when you get these
latencies?
Try increasing the metadata max age with metadata.max.age.ms.

(Disclaimer: I myself never ran any benchmarking, just trying to help :). )

Viktor

On Mon, Aug 7, 2017 at 7:37 PM, Chao Wang  wrote:

> Thanks, David. I was trying to do Kafka pub/sub on a local, closed
> network. In my case, I observed microsecond-latency with bare-bone sockets,
> and I would like to know how to configure Kafka to achieve similar result;
> if it turned out to be infeasible, what might be the cause of the
> additional latency?
>
> Thanks,
>
> Chao
>
>
>
> On 08/07/2017 12:10 PM, David Garcia wrote:
>
>> You are not going to get that kind of latency (i.e. less than 100
>> microseconds).  In my experience, consumer->producer latency averages
>> around: 20 milliseconds (cluster is in AWS with enhanced networking).
>>
>> On 8/3/17, 2:32 PM, "Chao Wang"  wrote:
>>
>>  Hi,
>>   I observed that it took 2-6 milliseconds for a topic to be
>> received by a
>>  Kafka consumer from a Kafka producer, and I wonder what I might be
>>  missing or I was wrong in configuring Kafka for low latency
>> (targeting
>>  at < 100 microseconds). I did the following:
>>   1. On the broker, I tried to prevent frequent flush of data to
>> disk
>>  (log.flush.interval.messages=10)
>>   2. On the producer, I tried to reduce the delay by setting
>> batch.size=0,
>>  linger.ms=0, acks =0, and I invoked flush() right after send()
>>   3. On the consumer, I set poll(0) (i.e., fetch every data once
>> its
>>  available?)
>>   I got similar observation (millisecond latency) in varying
>> value size
>>  from 1 to 512B, and also similar results when either colocating
>>  producer/consumer or putting them on separate PCs (connecting by a
>>  switch). As a verification, I implemented simple C/C++ sockets for
>>  transmission and observed latencies no more than 100 microseconds.
>>   Thanks,
>>   Chao
>>
>>
>
>

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

[Viktor Somogyi]

Got it, thanks Hans!

On Sat, Jun 17, 2017 at 11:11 AM, Hans Jespersen <h...@confluent.io> wrote:

>
> Offset commit is something that is done in the act of consuming (or
> reading) Kafka messages.
> Yes technically it is a write to the Kafka consumer offset topic but it's
> much easier for
> administers to think of ACLs in terms of whether the user is allowed to
> write (Produce) or
> read (Consume) messages and not the lower level semantics that are that
> consuming is actually
> reading AND writing (albeit only to the offset topic).
>
> -hans
>
>
>
>
> > On Jun 17, 2017, at 10:59 AM, Viktor Somogyi <
> viktor.somo...@cloudera.com> wrote:
> >
> > Hi Vahid,
> >
> > +1 for OffsetFetch from me too.
> >
> > I also wanted to ask the strangeness of the permissions, like why is
> > OffsetCommit a Read operation instead of Write which would intuitively
> make
> > more sense to me. Perhaps any expert could shed some light on this? :)
> >
> > Viktor
> >
> > On Tue, Jun 13, 2017 at 2:38 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com <mailto:vahidhashem...@us.ibm.com>> wrote:
> >
> >> Hi Michal,
> >>
> >> Thanks a lot for your feedback.
> >>
> >> Your statement about Heartbeat is fair and makes sense. I'll update the
> >> KIP accordingly.
> >>
> >> --Vahid
> >>
> >>
> >>
> >>
> >> From:Michal Borowiecki <michal.borowie...@openbet.com>
> >> To:users@kafka.apache.org, Vahid S Hashemian <
> >> vahidhashem...@us.ibm.com>, d...@kafka.apache.org
> >> Date:06/13/2017 01:35 AM
> >> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL
> >> Permission of OffsetFetch
> >> --
> >>
> >>
> >>
> >> Hi Vahid,
> >>
> >> +1 wrt OffsetFetch.
> >>
> >> The "Additional Food for Thought" mentions Heartbeat as a non-mutating
> >> action. I don't think that's true as the GroupCoordinator updates the
> >> latestHeartbeat field for the member and adds a new object to the
> >> heartbeatPurgatory, see completeAndScheduleNextHeartbeatExpiration()
> >> called from handleHeartbeat()
> >>
> >> NB added dev mailing list back into CC as it seems to have been lost
> along
> >> the way.
> >>
> >> Cheers,
> >>
> >> Michał
> >>
> >>
> >> On 12/06/17 18:47, Vahid S Hashemian wrote:
> >> Hi Colin,
> >>
> >> Thanks for the feedback.
> >>
> >> To be honest, I'm not sure either why Read was selected instead of Write
> >> for mutating APIs in the initial design (I asked Ewen on the
> corresponding
> >> JIRA and he seemed unsure too).
> >> Perhaps someone who was involved in the design can clarify.
> >>
> >> Thanks.
> >> --Vahid
> >>
> >>
> >>
> >>
> >> From:   Colin McCabe *<cmcc...@apache.org <mailto:cmcc...@apache.org>>*
> <cmcc...@apache.org <mailto:cmcc...@apache.org>>
> >> To: *users@kafka.apache.org <mailto:users@kafka.apache.org>* <
> users@kafka.apache.org <mailto:users@kafka.apache.org>>
> >> Date:   06/12/2017 10:11 AM
> >> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL
> >> Permission of OffsetFetch
> >>
> >>
> >>
> >> Hi Vahid,
> >>
> >> I think you make a valid point that the ACLs controlling group
> >> operations are not very intuitive.
> >>
> >> This is probably a dumb question, but why are we using Read for mutating
> >> APIs?  Shouldn't that be Write?
> >>
> >> The distinction between Describe and Read makes a lot of sense for
> >> Topics.  A group isn't really something that you "read" from in the same
> >> way as a topic, so it always felt kind of weird there.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote:
> >>
> >> Hi all,
> >>
> >> I'm resending my earlier note hoping it would spark some conversation
> >> this
> >> time around :)
> >>
> >> Thanks.
> >> --Vahid
> >>
> >>
> >>
> >>
> >> From:   "Vahid S Hashemian" *<vahidhashem...@us.ibm.com  vahidhashem...@us.ibm.com>

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

[Viktor Somogyi]

Hi Vahid,

+1 for OffsetFetch from me too.

I also wanted to ask the strangeness of the permissions, like why is
OffsetCommit a Read operation instead of Write which would intuitively make
more sense to me. Perhaps any expert could shed some light on this? :)

Viktor

On Tue, Jun 13, 2017 at 2:38 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Michal,
>
> Thanks a lot for your feedback.
>
> Your statement about Heartbeat is fair and makes sense. I'll update the
> KIP accordingly.
>
> --Vahid
>
>
>
>
> From:Michal Borowiecki 
> To:users@kafka.apache.org, Vahid S Hashemian <
> vahidhashem...@us.ibm.com>, d...@kafka.apache.org
> Date:06/13/2017 01:35 AM
> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL
> Permission of OffsetFetch
> --
>
>
>
> Hi Vahid,
>
> +1 wrt OffsetFetch.
>
> The "Additional Food for Thought" mentions Heartbeat as a non-mutating
> action. I don't think that's true as the GroupCoordinator updates the
> latestHeartbeat field for the member and adds a new object to the
> heartbeatPurgatory, see completeAndScheduleNextHeartbeatExpiration()
> called from handleHeartbeat()
>
> NB added dev mailing list back into CC as it seems to have been lost along
> the way.
>
> Cheers,
>
> Michał
>
>
> On 12/06/17 18:47, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for the feedback.
>
> To be honest, I'm not sure either why Read was selected instead of Write
> for mutating APIs in the initial design (I asked Ewen on the corresponding
> JIRA and he seemed unsure too).
> Perhaps someone who was involved in the design can clarify.
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Colin McCabe ** 
> To: *users@kafka.apache.org* 
> Date:   06/12/2017 10:11 AM
> Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL
> Permission of OffsetFetch
>
>
>
> Hi Vahid,
>
> I think you make a valid point that the ACLs controlling group
> operations are not very intuitive.
>
> This is probably a dumb question, but why are we using Read for mutating
> APIs?  Shouldn't that be Write?
>
> The distinction between Describe and Read makes a lot of sense for
> Topics.  A group isn't really something that you "read" from in the same
> way as a topic, so it always felt kind of weird there.
>
> best,
> Colin
>
>
> On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote:
>
> Hi all,
>
> I'm resending my earlier note hoping it would spark some conversation
> this
> time around :)
>
> Thanks.
> --Vahid
>
>
>
>
> From:   "Vahid S Hashemian" **
> 
> To: dev ** , "Kafka User"
>
> ** 
>
> Date:   05/30/2017 08:33 AM
> Subject:KIP-163: Lower the Minimum Required ACL Permission of
> OffsetFetch
>
>
>
> Hi,
>
> I started a new KIP to improve the minimum required ACL permissions of
> some of the APIs:
>
>
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch*
> 
>
>
>
> The KIP is to address KAFKA-4585.
>
> Feedback and suggestions are welcome!
>
> Thanks.
> --Vahid
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>  *Michal Borowiecki*
> *Senior Software Engineer L4*
> *T: * +44 208 742 1600 <(208)%20742-1600>
> +44 203 249 8448 <(203)%20249-8448>
>
> *E: * *michal.borowie...@openbet.com* 
> *W: * *www.openbet.com* 
> *OpenBet Ltd*
> Chiswick Park Building 9
> 566 Chiswick High Rd
> London
> W4 5XT
> UK
> 
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> *postmas...@openbet.com* and delete it from your
> system as well as any copies. The content of e-mails as well as traffic
> data may be monitored by OpenBet for employment and security purposes. To
> protect the environment please do not print this e-mail unless necessary.
> OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High
> Road, London, W4 5XT, United Kingdom. A company registered in England and
> Wales. Registered no. 3134634. VAT no. GB927523612
>
>
>
>