[ANNOUNCE] Apache Kafka 3.6.2

[Manikumar]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.6.2

This is a bug fix release and it includes fixes and improvements from 28 JIRAs.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.6.2/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and Scala 2.13) from:
https://kafka.apache.org/downloads#3.6.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 40 contributors to this release!
(Please report an unintended omission)

Akhilesh Chaganti, Andrew Schofield, Anton Liauchuk, Bob Barrett,
Bruno Cadonna, Cheng-Kai, Zhang, Chia-Ping Tsai, Chris Egerton, Colin
P. McCabe, Colin Patrick McCabe, David Arthur, David Jacot, David Mao,
Divij Vaidya, Edoardo Comar, Emma Humber, Gaurav Narula, Greg Harris,
hudeqi, Ismael Juma, Jason Gustafson, Jim Galasyn, Joel Hamill, Johnny
Hsu, José Armando García Sancio, Justine Olshan, Luke Chen, Manikumar
Reddy, Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Mike
Lloyd, Paolo Patierno, PoAn Yang, Ron Dagostino, Sagar Rao, Stanislav
Kozlovski, Walker Carlson, Yash Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Manikumar
Release Manager for Apache Kafka 3.6.2

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Thanks all for voting.

I'm now closing the vote. The vote passes with
- 3 +1 bindings votes from Divij Vaidya, Justine, and Manikumar
- 3 +1 non-binding votes from Andrew, Jakub, and  Lianet
- 0 -1 votes

I'll go ahead and finish the release process.


Thanks,

On Thu, Apr 4, 2024 at 8:53 PM Manikumar  wrote:

> +1 (binding) from my side also.
>
> - verified the signatures, artifacts
> - ran the tests on the source archive
> - verified the quickstarts
>
>
> Thanks
>
> On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
> wrote:
>
>> Thanks for clarifying Manikumar!
>>
>> +1 (binding) from me
>>
>> Justine
>>
>> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
>> wrote:
>>
>> > Hi Manikumar
>> >
>> > I verified the following:
>> > - all non-minor commits in the branch are captured in release notes
>> > - signature on the artifact match the public signature of Manikumar
>> > - basic topic creation & produce / consumer works with JDK8 + ARM +
>> Kafka
>> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
>> >
>> > Things look good to me. We don't need another RC for fixing docs.
>> >
>> > +1 (binding) from me.
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
>> > wrote:
>> >
>> > > Hi Justine,
>> > >
>> > > Thanks for catching this. looks like we have missed updating
>> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
>> > >
>> > > I will make sure to use the correct version when updating docs for
>> 3.6.2
>> > > release.
>> > > I will also update 3.5 and 3.6 branches with the correct heading and
>> also
>> > > update the release wiki.
>> > >
>> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > we will remove the "-SNAPSHOT" suffix while updating the website
>> docs. we
>> > > may need to automate this in the release script.
>> > >
>> > >
>> > > [1]
>> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
>> > > [2]
>> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
>> > > > > >
>> > > wrote:
>> > >
>> > > > Thanks for clarifying!
>> > > > I took a look at the documentation.html file in there, and it said
>> 3.4.
>> > > Is
>> > > > that expected?
>> > > >
>> > > > There are some files that request fullDot version and that seemed
>> > closer
>> > > to
>> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > > The upgrade.html file also looked ok.
>> > > >
>> > > > Thanks for running the release and answering my questions!
>> > > > Justine
>> > > >
>> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar <
>> manikumar.re...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Justine,
>> > > > >
>> > > > > Yes, it is intended. For bug fix releases website docs will be
>> > updated
>> > > > > during the final release process.
>> > > > > We can verify the site-docs artifacts here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
>> > > > > These site-docs artifacts will be used to update website docs.
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
>> > > > > 
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Manikumar,
>> > > > > >
>> > > > > > I've verified the keys, scanned the artifacts, and other docs.
>> > > > > > I built from source and ran with a ZK cluster (since I saw that
>> we
>> > > > > updated
>> > > > > > ZK version in this release)
>> > > > >

Re: [VOTE] 3.6.2 RC2

[Manikumar]

+1 (binding) from my side also.

- verified the signatures, artifacts
- ran the tests on the source archive
- verified the quickstarts


Thanks

On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
wrote:

> Thanks for clarifying Manikumar!
>
> +1 (binding) from me
>
> Justine
>
> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
> wrote:
>
> > Hi Manikumar
> >
> > I verified the following:
> > - all non-minor commits in the branch are captured in release notes
> > - signature on the artifact match the public signature of Manikumar
> > - basic topic creation & produce / consumer works with JDK8 + ARM + Kafka
> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
> >
> > Things look good to me. We don't need another RC for fixing docs.
> >
> > +1 (binding) from me.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for catching this. looks like we have missed updating
> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
> > >
> > > I will make sure to use the correct version when updating docs for
> 3.6.2
> > > release.
> > > I will also update 3.5 and 3.6 branches with the correct heading and
> also
> > > update the release wiki.
> > >
> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > we will remove the "-SNAPSHOT" suffix while updating the website docs.
> we
> > > may need to automate this in the release script.
> > >
> > >
> > > [1]
> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
> > > [2]
> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
> > >
> > >
> > > Thanks,
> > >
> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
> >  > > >
> > > wrote:
> > >
> > > > Thanks for clarifying!
> > > > I took a look at the documentation.html file in there, and it said
> 3.4.
> > > Is
> > > > that expected?
> > > >
> > > > There are some files that request fullDot version and that seemed
> > closer
> > > to
> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > > The upgrade.html file also looked ok.
> > > >
> > > > Thanks for running the release and answering my questions!
> > > > Justine
> > > >
> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Yes, it is intended. For bug fix releases website docs will be
> > updated
> > > > > during the final release process.
> > > > > We can verify the site-docs artifacts here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > > > > These site-docs artifacts will be used to update website docs.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Manikumar,
> > > > > >
> > > > > > I've verified the keys, scanned the artifacts, and other docs.
> > > > > > I built from source and ran with a ZK cluster (since I saw that
> we
> > > > > updated
> > > > > > ZK version in this release)
> > > > > > I ran a few tests on this cluster.
> > > > > >
> > > > > > I also ran the 2.12 binary.
> > > > > >
> > > > > > I noticed the docs link (
> > > > https://kafka.apache.org/36/documentation.html)
> > > > > > mentions 3.6.1 as the latest. Is that intended?
> > > > > > I will give my final vote when we figure this out.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
> > wrote:
> > > > > >
> > > > > > > Hi Manikumar, I did the following checks:
> > > > > > >
> > > > > > > - downloaded and built from sr

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Gentle reminder. Please download, test and vote for the release.

Thanks,

On Fri, Mar 29, 2024 at 4:57 PM Manikumar  wrote:

> Hi All,
>
> System test runs are green. There were 13 test failures in the first run.
> All the failed tests passed in the second run.
>
> System test results:
> https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
>
> Thanks,
>
> On Thu, Mar 28, 2024 at 3:21 PM Manikumar  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate we are considering for the release of Apache
>> Kafka 3.6.2.
>>
>> This is a bugfix release with several fixes, including dependency
>> version bumps for CVEs.
>>
>> Release notes for the 3.6.2 release:
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by by Wednesday, April 3rd
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/
>>
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/
>>
>> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
>> https://github.com/apache/kafka/releases/tag/3.6.2-rc2
>>
>> * Documentation:
>> https://kafka.apache.org/36/documentation.html
>>
>> * Protocol:
>> https://kafka.apache.org/36/protocol.html
>>
>> * Successful Jenkins builds for the 3.6 branch:
>> Unit/integration tests:
>> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/167/ (test
>> failures passed in local runs)
>> System tests: I will update system test results soon.
>>
>>
>> Thanks,
>> Manikumar
>>
>

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Hi All,

System test runs are green. There were 13 test failures in the first run.
All the failed tests passed in the second run.

System test results:
https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f

Thanks,

On Thu, Mar 28, 2024 at 3:21 PM Manikumar  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate we are considering for the release of Apache
> Kafka 3.6.2.
>
> This is a bugfix release with several fixes, including dependency
> version bumps for CVEs.
>
> Release notes for the 3.6.2 release:
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by by Wednesday, April 3rd
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/
>
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> https://github.com/apache/kafka/releases/tag/3.6.2-rc2
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/167/ (test
> failures passed in local runs)
> System tests: I will update system test results soon.
>
>
> Thanks,
> Manikumar
>

[VOTE] 3.6.2 RC2

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the second candidate we are considering for the release of Apache
Kafka 3.6.2.

This is a bugfix release with several fixes, including dependency
version bumps for CVEs.

Release notes for the 3.6.2 release:
https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html

*** Please download, test and vote by by Wednesday, April 3rd

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-3.6.2-rc2/


* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
https://github.com/apache/kafka/releases/tag/3.6.2-rc2

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/167/ (test
failures passed in local runs)
System tests: I will update system test results soon.


Thanks,
Manikumar

Re: [VOTE] 3.6.2 RC1

[Manikumar]

Hi,

Thanks for letting me know. Pls let me know after merging the PR. I will
generate RC2.


Thanks

On Sat, Mar 23, 2024 at 1:58 AM Colin McCabe  wrote:

> Sorry but I have to vote -1
>
> I tried verifying that the migration quotas bug described in
> https://issues.apache.org/jira/browse/KAFKA-16222 was fixed, and it
> appears to still be an issue with 3.6.2 RC1. The quota on the default
> resource is still getting translated improperly.
>
> I am looking into what the issue is here.
>
> best,
> Colin
>
>
> On Thu, Mar 21, 2024, at 19:32, Chia-Ping Tsai wrote:
> > hi Manikumar
> >
> >> Pls let me know after merging the PR. I will generate RC2 later today.
> >
> > Sure. We will complete it ASAP
> >
> >
> >> Manikumar  於 2024年3月22日 上午9:26 寫道：
> >>
> >> Hi,
> >>
> >> Thanks. Since we have merged KAFKA-16342
> >> <https://issues.apache.org/jira/browse/KAFKA-16342>, it's probably
> better
> >> to take the PR for KAFKA-16341
> >> <https://issues.apache.org/jira/browse/KAFKA-16341> for consistency.
> >> I am canceling the RC1 in favour of including KAFKA-16341
> >> <https://issues.apache.org/jira/browse/KAFKA-16341>.
> >>
> >> Chia-Ping,
> >>
> >> Pls let me know after merging the PR. I will generate RC2 later today.
> >>
> >>
> >> Thank you,
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Mar 21, 2024 at 9:10 PM Chia-Ping Tsai 
> wrote:
> >>
> >>>> Is this a regression from the 3.5.0 release?
> >>>
> >>> I believe the bug is existent for a while, but it is not a true issue
> until
> >>> we allowed users to fetch offset of max timestamp.
> >>>
> >>>> Can we update the "Affects Version/s" field on JIRA?
> >>>
> >>> done. I attach the tags for active branches - 3.6.1 and 3.7.0
> >>>
> >>> Manikumar  於 2024年3月21日 週四 下午11:12寫道：
> >>>
> >>>> Hi Chia-Ping,
> >>>>
> >>>> Thanks for letting me know.
> >>>>
> >>>> Is this a regression from the 3.5.0 release?  Can we update the
> "Affects
> >>>> Version/s" field on JIRA?
> >>>>
> >>>> Thanks,
> >>>>
> >>>>
> >>>> On Thu, Mar 21, 2024 at 5:06 PM Chia-Ping Tsai 
> >>> wrote:
> >>>>
> >>>>> hi Manikumar,
> >>>>>
> >>>>> There is a bug fix which needs to be backport to 3.6 (
> >>>>> https://issues.apache.org/jira/browse/KAFKA-16341)
> >>>>>
> >>>>> It fixes the incorrect offset of max timestamp in non-compress path.
> >>> The
> >>>>> other paths are already fixed by
> >>>>> https://issues.apache.org/jira/browse/KAFKA-16342.
> >>>>>
> >>>>> Personally, we should backport both fixes for all paths, and we can
> >>>>> complete the backport today.
> >>>>>
> >>>>> Sorry for bring this news to RC1.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>> Manikumar  於 2024年3月21日 週四 下午6:11寫道：
> >>>>>
> >>>>>> Hello Kafka users, developers and client-developers,
> >>>>>>
> >>>>>> This is the first candidate for release of Apache Kafka 3.6.2.
> >>>>>>
> >>>>>> This is a bugfix release with several fixes, including dependency
> >>>>>> version bumps for CVEs.
> >>>>>>
> >>>>>> Release notes for the 3.6.2 release:
> >>>>>>
> >>> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
> >>>>>>
> >>>>>> *** Please download, test and vote by Tuesday, March 26th
> >>>>>>
> >>>>>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>>>>> https://kafka.apache.org/KEYS
> >>>>>>
> >>>>>> * Release artifacts to be voted upon (source and binary):
> >>>>>> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
> >>>>>>
> >>>>>>
> >>>>>> * Maven artifacts to be voted upon:
> >>>>>>
> >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>>>>>
> >>>>>> * Javadoc:
> >>>>>> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
> >>>>>>
> >>>>>> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> >>>>>> https://github.com/apache/kafka/releases/tag/3.6.2-rc1
> >>>>>>
> >>>>>> * Documentation:
> >>>>>> https://kafka.apache.org/36/documentation.html
> >>>>>>
> >>>>>> * Protocol:
> >>>>>> https://kafka.apache.org/36/protocol.html
> >>>>>>
> >>>>>> * Successful Jenkins builds for the 3.6 branch:
> >>>>>> Unit/integration tests:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> >>>>>> (with few flaky failures)
> >>>>>> System tests: I will update system test results
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Manikumar
> >>>>>>
> >>>>>
> >>>>
> >>>
>

[VOTE] 3.6.2 RC1

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.6.2.

This is a bugfix release with several fixes, including dependency
version bumps for CVEs.

Release notes for the 3.6.2 release:
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, March 26th

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/


* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
https://github.com/apache/kafka/releases/tag/3.6.2-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
Unit/integration tests:
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
(with few flaky failures)
System tests: I will update system test results

Thanks,
Manikumar

CVE-2023-25194: Apache Kafka: Possible RCE/Denial of service attack via SASL JAAS JndiLoginModule configuration using Kafka Connect

[Manikumar]

Severity: important

Description:

A possible security vulnerability has been identified in Apache Kafka
Connect. This requires access to a Kafka Connect worker,
and the ability to create/modify connectors on it with an arbitrary
Kafka client SASL JAAS config and a SASL-based security protocol,
which has been possible on Kafka Connect clusters since Apache Kafka
2.3.0. When configuring the connector via the Kafka Connect REST API,
an authenticated operator can set the `sasl.jaas.config` property for any
of the connector's Kafka clients to
"com.sun.security.auth.module.JndiLoginModule",
which can be done via the `producer.override.sasl.jaas.config`,
`consumer.override.sasl.jaas.config`, or
`admin.override.sasl.jaas.config` properties.

This will allow the server to connect to the attacker's LDAP server
and deserialize the LDAP response, which the attacker can use to
execute java deserialization gadget chains on the Kafka connect
server. Attackers can cause unrestricted deserialization of untrusted
data (or) RCE vulnerability when there are gadgets in the classpath.

Since Apache Kafka 3.0.0, users are allowed to specify these properties
in connector configurations for Kafka Connect clusters running with
out-of-the-box configurations. Before Apache Kafka 3.0.0, users may not
specify these properties unless the Kafka Connect cluster has been reconfigured
with a connector client override policy that permits them.

Since Apache Kafka 3.4.0, we have added a system property
("-Dorg.apache.kafka.disallowed.login.modules") to disable the
problematic login modules usage in SASL JAAS configuration. Also by
default "com.sun.security.auth.module.JndiLoginModule" is disabled
in Apache Kafka 3.4.0.

We advise the Kafka Connect users to validate connector configurations
and only allow trusted JNDI configurations. Also examine connector
dependencies for vulnerable versions and either upgrade their
connectors, upgrading that specific dependency, or removing the
connectors as options for remediation. Finally, in addition to leveraging the
"org.apache.kafka.disallowed.login.modules" system property, Kafka Connect users
can also implement their own connector client config override policy, which can
be used to control which Kafka client properties can be overridden directly
in a connector config and which cannot.

Credit:

Apache Kafka would like to thank Jari Jääskelä
(https://hackerone.com/reports/1529790)
and 4ra1n and Y4tacker (they found vulnerabilities in other Apache projects.
After discussion between PMC of the two projects, it was finally
confirmed that it was the vulnerability of Kafka then they reported it to us)


References:

https://kafka.apache.org/cve-list
https://kafka.apache.org/
https://www.cve.org/CVERecord?id=CVE-2023-25194

CVE-2022-34917: Unauthenticated clients may cause OutOfMemoryError on Apache Kafka Brokers

[Manikumar]

Severity: High

Description:

A security vulnerability has been identified in Apache Kafka. It
affects all releases since 2.8.0. The vulnerability allows malicious
unauthenticated clients to allocate large amounts of memory on
brokers. This can lead to brokers hitting OutOfMemoryException and
causing denial of service.

Example scenarios:
- Kafka cluster without authentication: Any clients able to establish
a network connection to a broker can trigger the issue.
- Kafka cluster with SASL authentication: Any clients able to
establish a network connection to a broker, without the need for valid
SASL credentials, can trigger the issue.
- Kafka cluster with TLS authentication: Only clients able to
successfully authenticate via TLS can trigger the issue.

We advise the users to upgrade the Kafka installations to one of the
3.2.3, 3.1.2, 3.0.2, 2.8.2 versions.

Credit:

Apache Kafka would like to thank Mickael Maison, Tom Bentley and
Daniel Collins for reporting this issue.

References:

https://kafka.apache.org/cve-list

[ANNOUNCE] Apache Kafka 3.2.3

[Manikumar]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.2.3

Apache Kafka 3.2.3 is a bugfix release and it contains important
security fixes. This also fixes 7 issues since the 3.2.1
release. Please see the release notes for more information.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.2.3/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#3.2.3

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 12 contributors to this release!

Andrew Borley, Andrew Dean, Colin Patrick McCabe, David Arthur, Derek
Troy-West, Divij Vaidya, Jason Gustafson, Manikumar Reddy, Mickael
Maison, Philip Nee, Thomas Cooper, Tom Bentley

Thanks to Mickael Maison and Tom Bentley for driving this release.

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Apache Kafka PMC

[ANNOUNCE] Apache Kafka 3.1.2

[Manikumar]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.1.2

Apache Kafka 3.1.2 is a bugfix release and it contains important
security fixes. It also fixes 4 issues since the 3.1.1
release. Please see the release notes for more information.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.1.2/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#3.1.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 16 contributors to this release!

Andrew Borley, Bruno Cadonna, Colin Patrick McCabe, David Jacot, Derek
Troy-West, Divij Vaidya, Guozhang Wang, Ismael Juma, Jason Gustafson,
Kirk True, Lucas Bradstreet, Manikumar Reddy, Mickael Maison,
nicolasguyomar, Niket, Tom Bentley

Thanks to Mickael Maison and Tom Bentley for driving this release.

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Apache Kafka PMC

[ANNOUNCE] Apache Kafka 3.0.2

[Manikumar]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 3.0.2

Apache Kafka 3.0.2 is a bugfix release and it contains important
security fixes. It also fixes 10 issues since the 3.0.1
release. Please see the release notes for more information.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.0.2/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#3.0.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 22 contributors to this release!

Andrew Borley, Bounkong Khamphousone, Colin Patrick McCabe, David Jacot,
Derek Troy-West, Ismael Juma, Jason Gustafson, Jules Ivanic, Justine
Olshan, Konstantine Karantasis, Lucas Bradstreet, Manikumar Reddy, Mickael
Maison, nicolasguyomar, Niket, Philip Nee, Randall Hauch, Stanislav
Vodetskyi, Tom Bentley, Vincent Jiang, Xiaoyue Xue, Yang Yu

Thanks to Tom Bentley for driving this release.

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Apache Kafka PMC

[ANNOUNCE] Apache Kafka 2.8.2

[Manikumar]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.8.2

Apache Kafka 2.8.2 is a bugfix release and it contains important
security fixes. It also fixes 11 issues since the 2.8.1
release. Please see the release notes for more information.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.8.2/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#2.8.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 17 contributors to this release!

A. Sophie Blee-Goldman, Andrew Borley, Bruno Cadonna, Colin Patrick McCabe,
David Jacot, Jason Gustafson, jiangyuan, Justine Olshan, Luke Chen,
Manikumar Reddy, Matthias J. Sax, Oliver Hutchison, Philip Nee, Prateek
Agarwal, prince-mahajan, Randall Hauch, Stanislav Vodetskyi

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Manikumar

Re: [kafka-clients] [VOTE] 2.7.2 RC0

[Manikumar]

Hi,

+1 (binding)

- verified the signatures
- verified the quickstart with binary

Thanks for running the release!

Thanks,
Manikumar

On Tue, Nov 2, 2021 at 11:16 PM Mickael Maison  wrote:

> Bumping the thread.
>
> Contributors, committers and PMC, please take some time to test this
> release candidate and vote.
>
> Thanks,
> Mickael
>
> On Tue, Oct 26, 2021 at 7:38 PM Israel Ekpo  wrote:
> >
> > Thanks Bill. That is greatly appreciated :)
> >
> > We need more PMC members with binding votes to participate.
> >
> > You can do it!
> >
> > On Tue, Oct 26, 2021 at 1:25 PM Bill Bejeck  wrote:
> >>
> >> Hi Mickael,
> >>
> >> Thanks for running the release.
> >>
> >> Steps taken
> >>
> >> Validated checksums
> >> Validated signatures
> >> Built from source
> >> Ran all the unit tests
> >> Spot checked various JavaDocs
> >>
> >>
> >> +1(binding)
> >>
> >> On Tue, Oct 26, 2021 at 4:43 AM Luke Chen  wrote:
> >>>
> >>> Hi Mickael,
> >>>
> >>> Thanks for the release. I did:
> >>> 1. Verified checksums and signatures
> >>> 2. Run quick start steps
> >>> 3. Verified the CVE-2021-38153 is indeed fixed in kafka-2.7.2-src.tgz
> >>> <https://home.apache.org/~mimaison/kafka-2.7.2-rc0/kafka-2.7.2-src.tgz
> >.
> >>>
> >>> +1 (non-binding)
> >>>
> >>> Thank you.
> >>> Luke
> >>>
> >>> On Tue, Oct 26, 2021 at 3:41 PM Tom Bentley 
> wrote:
> >>>
> >>> > Hi Mickael,
> >>> >
> >>> > As with 2.6.3 RC0, I have:
> >>> >
> >>> > * Verified checksums and signatures
> >>> > * Built jars and docs from the source jar
> >>> > * Run the unit and integration tests
> >>> >
> >>> > +1 non-binding
> >>> >
> >>> > Kind regards,
> >>> >
> >>> > Tom
> >>> >
> >>> > On Sun, Oct 24, 2021 at 3:05 PM Israel Ekpo 
> wrote:
> >>> >
> >>> > > Mickael,
> >>> > >
> >>> > > Do we need to do another RC? Were there issues with this release?
> >>> > >
> >>> > > What happens next?
> >>> > >
> >>> > >
> >>> > > On Sat, Oct 16, 2021 at 8:11 PM Israel Ekpo 
> >>> > wrote:
> >>> > >
> >>> > > >
> >>> > > > I have performed the following checks
> >>> > > >
> >>> > > > Validation of Release Artifacts Cryptographic Hashes (ASC MD5
> SHA1
> >>> > > SHA512)
> >>> > > > PGP Signatures used to sign the release artifacts
> >>> > > > Javadocs check
> >>> > > > Site docs check was not necessary
> >>> > > > Jenkins build was successful.
> >>> > > >
> >>> > > > I used the steps here for the first two checks
> >>> > > > https://github.com/izzyacademy/apache-kafka-release-party
> >>> > > >
> >>> > > > I vote +1 on this RC
> >>> > > >
> >>> > > >
> >>> > > > On Fri, Oct 15, 2021 at 12:11 PM Israel Ekpo <
> israele...@gmail.com>
> >>> > > wrote:
> >>> > > >
> >>> > > >> Hi Mickael,
> >>> > > >>
> >>> > > >> I am pretty surprised that there are no votes so far on the RCs
> and
> >>> > the
> >>> > > >> deadline has already passed.
> >>> > > >>
> >>> > > >> I am running my checks right now using the process outlined here
> >>> > > >>
> >>> > > >>
> >>> > > >>
> >>> > >
> >>> >
> https://github.com/izzyacademy/apache-kafka-release-party#how-to-validate-apache-kafka-release-candidates
> >>> > > >>
> >>> > > >> I will post my results and vote as soon as they are completed.
> >>> > > >>
> >>> > > >> On Fri, Oct 15, 2021 at 9:52 AM Mickael Maison <
> mimai...@apache.org>
> >>> > > >> wrote:
> >>> > > >>
> >>> > > 

Re: [VOTE] 2.8.1 RC1

[Manikumar]

Hi,

+1 (binding)

- verified the signatures
- ran the tests on the source archive
- verified the quickstart with binary
- verified the artifacts, javadoc

Thanks for running the release!

Thanks,
Manikumar

On Tue, Sep 14, 2021 at 10:36 PM Chris Egerton 
wrote:

> Hi David,
>
> I took a look at the Javadocs and noticed a small issue. Using the search
> bar at the top of the landing page (
> https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/), I entered
> "KafkaProducer" and clicked on the search item that came up. This brought
> me to
>
> https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/undefined/org/apache/kafka/clients/producer/KafkaProducer.html
> ,
> which displayed a 404 "Not Found" error.
>
> This doesn't appear to be a regression as the same behavior occurs for me
> when browsing the 2.8.0 Javadocs (
> https://kafka.apache.org/28/javadoc/index.html?overview-summary.html), but
> I wanted to bring it up just in case. Normally I don't use the search bar
> but with the removal of sidebar frames from Javadocs I've had to start
> relying on it a little bit more.
>
> Cheers,
>
> Chris
>
> On Tue, Sep 14, 2021 at 12:56 PM Randall Hauch  wrote:
>
> > Thanks for generating a new RC1 with the corrected site docs, David.
> >
> > I was able to successfully complete the following:
> >
> > - Installed 2.8.1 RC0 and performed quickstart for broker and Connect
> > - Verified signatures and checksums
> > - Verified the tag
> > - Manually compared the release notes to JIRA
> > - Build release archive from the tag, ran Connect tests, installed
> locally,
> > and ran a portion of quickstart
> > - Manually spotchecked the Javadocs
> > - Verified the site docs at
> https://kafka.apache.org/28/documentation.html
> > has corrected version references, except for the tar and cd commands in
> > step 1 of the https://kafka.apache.org/28/documentation.html#quickstart.
> >
> > I think that last issue is minor and not worth another RC, since the
> other
> > version references in https://kafka.apache.org/28/documentation.html do
> > reference 2.8.1 and we can easily fix it on the website, optionally as
> part
> > of the other post-vote changes to the site.
> >
> > So I'm +1 (binding)
> >
> > Best regards,
> >
> > Randall
> >
> > On Tue, Sep 14, 2021 at 8:39 AM David Jacot  >
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 2.8.1.
> > >
> > > Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the
> > 2.8.0
> > > release. Please see the release notes for more information.
> > >
> > > Release notes for the 2.8.1 release:
> > > https://home.apache.org/~dajac/kafka-2.8.1-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday, September 17, 9am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~dajac/kafka-2.8.1-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.8.1-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/28/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/28/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.8 branch:
> > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/
> > >
> > > /**
> > >
> > > Thanks,
> > > David
> > >
> >
>

Re: [VOTE] 2.6.2 RC0

[Manikumar]

Hi,

+1 (binding)

- verified the signatures
- ran the tests on the source archive with Scala 2.13
- verified the quickstart with Scala 2.12 binary
- verified the artifacts, javadoc

Thanks for running the release!

Thanks,
Manikumar

On Tue, Mar 16, 2021 at 9:41 AM Sophie Blee-Goldman
 wrote:

> Thanks Luke. I'll make sure to bump this in the kafka-site repo once the
> release is finalized.
>
> On Mon, Mar 15, 2021 at 8:58 PM Luke Chen  wrote:
>
> > Hi Sophie,
> > A small doc update for 2.6.2. I think we missed it.
> > https://github.com/apache/kafka/pull/10328
> >
> > Thanks.
> > Luke
> >
> > On Tue, Mar 16, 2021 at 11:12 AM Guozhang Wang 
> wrote:
> >
> > > Hi Sophie,
> > >
> > > I've reviewed the javadocs / release notes / documentations, and they
> > LGTM.
> > >
> > > +1.
> > >
> > >
> > > Guozhang
> > >
> > > On Fri, Mar 12, 2021 at 10:48 AM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 2.6.2.
> > > >
> > > > Apache Kafka 2.6.2 is a bugfix release and fixes 30 issues since the
> > > 2.6.1
> > > > release. Please see the release notes for more information.
> > > >
> > > > Release notes for the 2.6.2 release:
> > > >
> > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday, March 19th, 9am PST
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 2.6 branch) is the 2.6.2 tag:
> > > > https://github.com/apache/kafka/releases/tag/2.6.2-rc0
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/26/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/26/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 2.6 branch:
> > > > Unit/integration tests:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/105/
> > > > System tests:
> > > >
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4435/
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > > Sophie
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: Apache Kafka project CLA requirement

[Manikumar]

Hi,

Apache Software Foundation does not require code contributors
(non-committers) to sign CLA.
Contributions are considered to be under the Apache License. Committers
need to sign a CLA.

Thanks,
Manikumar



On Wed, Jan 20, 2021 at 7:53 PM Radoslava Zheleva 
wrote:

> Hello Team,
>
>
>
> May you please advise if the Apache Kafka project requires contributors to
> sign the ASF CLA?
>
> I have been going through the project’s GitHub repo and the webpage and I
> am not able to confirm if signing of the ASF CLA is require or not.
>
> May you please advise? In case you are requiring contributors to sign the
> ASF CLA, may you please provide me with an URL to the documentation?
>
>
>
>
>
> Thanks,
>
>
>
> Radi
>
>
>
> *From: *Radoslava Zheleva 
> *Date: *Monday, January 18, 2021 at 6:03 PM
> *To: *"users@kafka.apache.org" 
> *Cc: *Lauren Britton 
> *Subject: *Re: Apache Kafka project CLA requirement
>
>
>
> Hello Team,
>
>
>
> May you please look at my question below and advise?
>
>
>
> Thanks,
>
>
>
> Radi
>
>
>
> *From: *Radoslava Zheleva 
> *Date: *Thursday, January 14, 2021 at 4:33 PM
> *To: *"users@kafka.apache.org" 
> *Cc: *Lauren Britton 
> *Subject: *Apache Kafka project CLA requirement
>
>
>
> Hello Team,
>
>
>
> May you please advise if the Apache Kafka project requires contributors to
> sign the ASF CLA?
>
> I have been going through the project’s GitHub repo and the webpage and I
> am not able to confirm if signing of the ASF CLA is require or not.
>
> May you please advise? In case you are requiring contributors to sign the
> ASF CLA, may you please provide me with an URL to the documentation?
>
>
>
> Thank you!
>
>
>
>
>
> *Kind regards,*
>
>
>
>
> *Radoslava Zheleva *Open Source Compliance Program Manager
> rzhel...@vmware.com
> +359 (2) 8041407  <+359%20(2)%208041407>
>
> +359 (88) 8737439
>
> *2 Donka Ushlinova Str.*
>
> *Garitage Park, Building 3*
>
> Sofia, 1766,
> Bulgaria
>
>
>
> [image: VMware] <http://www.vmware.com/>
>
>
>

Re: [VOTE] 2.6.1 RC3

[Manikumar]

Hi,

+1 (binding)
- verified signatures
- ran the tests on the source archive with Scala 2.13
- verified the core/connect/streams quickstart with Scala 2.13 binary
archive.
- verified the artifacts, javadoc

Thanks for running the release!


Thanks,
Manikumar

On Tue, Dec 15, 2020 at 9:01 PM Rajini Sivaram  wrote:

> +1 (binding)
>
> Verified signatures, ran tests from source build (one flaky test failed but
> passed on rerun), ran Kafka quick start with the binary with both Scala
> 2.12 and Scala 2.13.
>
> Thanks for running the release, Mickael!
>
> Regards,
>
> Rajini
>
> On Fri, Dec 11, 2020 at 3:23 PM Mickael Maison 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the fourth candidate for release of Apache Kafka 2.6.1.
> >
> > Since RC2, the following JIRAs have been fixed: KAFKA-10811, KAFKA-10802
> >
> > Release notes for the 2.6.1 release:
> > https://home.apache.org/~mimaison/kafka-2.6.1-rc3/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday, December 18, 12 PM ET ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~mimaison/kafka-2.6.1-rc3/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~mimaison/kafka-2.6.1-rc3/javadoc/
> >
> > * Tag to be voted upon (off 2.6 branch) is the 2.6.1 tag:
> > https://github.com/apache/kafka/releases/tag/2.6.1-rc3
> >
> > * Documentation:
> > https://kafka.apache.org/26/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/26/protocol.html
> >
> > * Successful Jenkins builds for the 2.6 branch:
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/62/
> >
> > /**
> >
> > Thanks,
> > Mickael
> >
>

Re: [VOTE] 2.5.1 RC0

[Manikumar]

Hi John,

+1 (binding)

- Verified signatures, artifacts,  Release notes
- Built from sources, ran tests
- Ran core/connect/streams quick start for Scala 2.13 release, ran few
manual tests

Thanks for driving the release.

Thanks,
Manikumar

On Thu, Jul 30, 2020 at 10:32 PM Ismael Juma  wrote:

> +1 (binding), verified signatures, ran the tests on the source archive with
> Scala 2.13 and Java 11 and verified the quickstart with the source archive
> and Scala 2.13 binary archive.
>
> Ismael
>
> On Thu, Jul 23, 2020 at 7:39 PM John Roesler  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 2.5.1.
> >
> > Apache Kafka 2.5.1 is a bugfix release and fixes 72 issues since the
> 2.5.0
> > release. Please see the release notes for more information.
> >
> > Release notes for the 2.5.1 release:
> > https://home.apache.org/~vvcephei/kafka-2.5.1-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, 28 July 2020, 5pm Pacific
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~vvcephei/kafka-2.5.1-rc0/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~vvcephei/kafka-2.5.1-rc0/javadoc/
> >
> > * Tag to be voted upon (off 2.5 branch) is the 2.5.1 tag:
> > https://github.com/apache/kafka/releases/tag/2.5.1-rc0
> >
> > * Documentation:
> > https://kafka.apache.org/25/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/25/protocol.html
> >
> > * Successful Jenkins builds for the 2.5 branch:
> > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.5-jdk8/170/
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4066
> > System test results:
> >
> http://confluent-kafka-2-5-system-test-results.s3-us-west-2.amazonaws.com/2020-07-22--001.1595474441--apache--2.5--7f9187fe3/report.html
> >
> > Thanks,
> > John
> >
>

Re: Request for adding into contributors list

[Manikumar]

Hi,

Thanks for your interest. I just added you to the the contributors list.

Thanks,


On Mon, Apr 13, 2020 at 6:45 PM Wong Lucent  wrote:

> Hi,
>
>
> May I add my JIRA ID into the contributors list of Apache kafka.
>
> Here is my JIRA profile:
>
> Username: LucentWong
> Full name: Yu Wang
>
>
> Thanks and Best Regards
> Lucent Wong
>

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC3

[Manikumar]

Hi David,

+1 (binding)

- Verified signatures, artifacts,  Release notes
- Built from sources, ran tests
- Ran core/streams quick start for Scala 2.13 binary, ran few manual tests
- Verified docs

As discussed offline, we need to add upgrade instructions to 2.5 docs.

Thanks for driving the release.

Thanks,
Manikumar

On Fri, Apr 10, 2020 at 7:53 PM Bill Bejeck  wrote:

> Hi David,
>
> +1 (non-binding) Verified signatures, built jars from source, ran unit and
> integration tests, all passed.
>
> Thanks for running the release.
>
> -Bill
>
> On Wed, Apr 8, 2020 at 10:10 AM David Arthur  wrote:
>
> > Passing Jenkins build on 2.5 branch:
> > https://builds.apache.org/job/kafka-2.5-jdk8/90/
> >
> > On Wed, Apr 8, 2020 at 12:03 AM David Arthur  wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the forth candidate for release of Apache Kafka 2.5.0.
> >>
> >> * TLS 1.3 support (1.2 is now the default)
> >> * Co-groups for Kafka Streams
> >> * Incremental rebalance for Kafka Consumer
> >> * New metrics for better operational insight
> >> * Upgrade Zookeeper to 3.5.7
> >> * Deprecate support for Scala 2.11
> >>
> >> Release notes for the 2.5.0 release:
> >> https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Friday April 10th 5pm PT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/javadoc/
> >>
> >> * Tag to be voted upon (off 2.5 branch) is the 2.5.0 tag:
> >> https://github.com/apache/kafka/releases/tag/2.5.0-rc3
> >>
> >> * Documentation:
> >> https://kafka.apache.org/25/documentation.html
> >>
> >> * Protocol:
> >> https://kafka.apache.org/25/protocol.html
> >>
> >> Successful Jenkins builds to follow
> >>
> >> Thanks!
> >> David
> >>
> >
> >
> > --
> > David Arthur
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/kafka-clients/CA%2B0Ze6oy4_Vw6B4M%3DoFtLvfk0OZAnioQW2u1xjgqe9r%3D3sC%2B5A%40mail.gmail.com
> > <
> https://groups.google.com/d/msgid/kafka-clients/CA%2B0Ze6oy4_Vw6B4M%3DoFtLvfk0OZAnioQW2u1xjgqe9r%3D3sC%2B5A%40mail.gmail.com?utm_medium=email_source=footer
> >
> > .
> >
>

Re: [ANNOUNCE] New committer: Konstantine Karantasis

[Manikumar]

Congrats Konstantine!

On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Congrats!
>
> On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > Congrats Konstantine!
> >
> > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck 
> > wrote:
> >>
> >> Congratulations Konstantine! Well deserved.
> >>
> >> -Bill
> >>
> >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> >>  wrote:
> >>
> >>> The PMC for Apache Kafka has invited Konstantine Karantasis as
> >>> a committer and we are pleased to announce that he has
> >>> accepted!
> >>>
> >>> Konstantine has contributed 56 patches and helped to review
> >>> even more. His recent work includes a major overhaul of the
> >>> Connect task management system in order to support incremental
> >>> rebalancing. In addition to code contributions, Konstantine
> >>> helps the community in many other ways including talks at
> >>> meetups and at Kafka Summit and answering questions on
> >>> stackoverflow. He consistently shows good judgement in design
> >>> and a careful attention to details when it comes to code.
> >>>
> >>> Thanks for all the contributions and looking forward to more!
> >>>
> >>> Jason, on behalf of the Apache Kafka PMC
> >>>
> -BEGIN PGP SIGNATURE-
>
> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> =68IH
> -END PGP SIGNATURE-
>

Re: Minor version upgrade instructions from 0.11.0.0 to 0.11.0.3

[Manikumar]

Hi,

Your approach is correct.  For minor version upgrade (0.11.0.0 to
0.11.0.3), we can just update the brokers to new version.

Thanks,

On Mon, Jan 20, 2020 at 1:27 AM Sarath Babu
 wrote:

> Hi all,
> Appreciate any help/ pointers on how to upgrade.One thought is to start
> the cluster brokers with 0.11.0.3 and using same old server.properties.
> However I am not completely sure if this is the correct approach, as major
> version upgrade documentation talk a lot about juggling the protocol
> version and Kafka version.
> Thanks in advance.
> Regards,Sarath
>
> On Friday, January 17, 2020, 18:05, Sarath Babu <
> maddalasarathb...@yahoo.com.INVALID> wrote:
>
>  Hi,
>
> We are using kafka 0.11.0.0 and I believe hitting this defect
> - [KAFKA-6003] Replication Fetcher thread for a partition with no data
> fails to start - ASF JIRA
>
> |
> |
> |  |
> [KAFKA-6003] Replication Fetcher thread for a partition with no data fai...
>
>
>  |
>
>  |
>
>  |
>
>
> Solution was patched to 0.11.0.2 so I would like to patch highest minor
> available which is 0.11.0.3
> I couldn't find the minor version upgrade instructions after searching for
> some time.Could anyone of you share the wiki?
> Ours is a production cluster and don't want to create any issues with this
> patch application.
> Thanks in advance.
> Regards.
>
>
>

Re: [ANNOUNCE] Apache Kafka 2.4.0

[Manikumar]

Thanks to all who contributed to the Apache blog post about 2.4.  It's live
now at https://blogs.apache.org/kafka/date/20191216

On Mon, Dec 16, 2019 at 10:30 PM Manikumar  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.4.0
>
> This release includes many new features, including:
> - Allow consumers to fetch from closest replica
> - Support for incremental cooperative rebalancing to the consumer
> rebalance protocol
> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
> engine
> - New Java authorizer Interface
> - Support for non-key joining in KTable
> - Administrative API for replica reassignment
> - Securing Internal connect REST endpoints
> - API to delete consumer offsets and expose it via the AdminClient.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.4.0/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.11, 2.12 and 2.13)
> from:
> https://kafka.apache.org/downloads#2.4.0
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 121 contributors to this release!
>
> A. Sophie Blee-Goldman, Adam Bellemare, Alex Diachenko, Alex Leung, Alex
> Mironov, Alice, Almog Gavra, Anastasia Vela, anatasiavela, Andy Coates,
> Antony Stubbs, Arjun Satish, Arlo Louis O'Keeffe, Arvind Thirunarayanan,
> asutosh936, Bill Bejeck, Bob Barrett, Boyang Chen, Brian Bushree, Bruno
> Cadonna, cadonna, Carlos Manuel Duclos Vergara, Cheng Pan, Chia-Ping Tsai,
> Chris Egerton, Chris Pettitt, Chris Stromberger, Colin Hicks, Colin P.
> Mccabe, Colin Patrick McCabe, cpettitt-confluent, cwildman, Cyrus Vafadari,
> David Arthur, David Jacot, Dejan Stojadinović, Dhruvil Shah, Florian
> Hussonnois, Gardner Vickers, Gemma Singleton, Grant Henke, Greg Harris,
> Gunnar Morling, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, highluck, huxi,
> Igor Soarez, Ismael Juma, James Cheng, Jason Gustafson, Jeff Huang, Joel
> Hamill, John Roesler, jolshan, José Armando García Sancio, Jukka Karvanen,
> Justine Olshan, Kamal Chandraprakash, Karan Kumar, Kengo Seki, Kevin Lu,
> khairy, Konstantine Karantasis, Lee Dongjin, Lifei Chen, Lucas Bradstreet,
> LuyingLiu, Magesh Nandakumar, Manikumar Reddy, Matthias J. Sax, Michał
> Borowiecki, Michał Siatkowski, Mickael Maison, mjarvie, mmanna-sapfgl,
> Nacho Muñoz Gómez, Nathan Murthy, Nigel Liang, NIkhil Bhatia, Nikolay, Omar
> Al-Safi, Omkar Mestry, Paul, pkleindl, Rajan Chauhan, Rajini Sivaram,
> Randall Hauch, Richard Yu, Robert Yokota, Rohan, Ron Dagostino, Ryanne
> Dolan, saisandeep, Scott Hendricks, sdreynolds, Sean Glover, Sergey
> Prokofiev, slim, soondenana, Stanislav Kozlovski, Stanislav Vodetskyi,
> SuryaTeja Duggi, tadsul, teebee, Tirtha Chatterjee, Tu Tran, Tu V. Tran,
> Vahid Hashemian, Victoria Bialas, Vikas Singh, Viktor Somogyi, Viktor
> Somogyi-Vass, vinoth chandar, wenhoujx, Wennn, Will James, wineandcheeze,
> Yaroslav Tkachenko, 康智冬
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Manikumar
>

[ANNOUNCE] Apache Kafka 2.4.0

[Manikumar]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.4.0

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for non-key joining in KTable
- Administrative API for replica reassignment
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.4.0/RELEASE_NOTES.html

You can download the source and binary release (Scala 2.11, 2.12 and 2.13)
from:
https://kafka.apache.org/downloads#2.4.0

---

Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.

With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.

Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 121 contributors to this release!

A. Sophie Blee-Goldman, Adam Bellemare, Alex Diachenko, Alex Leung, Alex
Mironov, Alice, Almog Gavra, Anastasia Vela, anatasiavela, Andy Coates,
Antony Stubbs, Arjun Satish, Arlo Louis O'Keeffe, Arvind Thirunarayanan,
asutosh936, Bill Bejeck, Bob Barrett, Boyang Chen, Brian Bushree, Bruno
Cadonna, cadonna, Carlos Manuel Duclos Vergara, Cheng Pan, Chia-Ping Tsai,
Chris Egerton, Chris Pettitt, Chris Stromberger, Colin Hicks, Colin P.
Mccabe, Colin Patrick McCabe, cpettitt-confluent, cwildman, Cyrus Vafadari,
David Arthur, David Jacot, Dejan Stojadinović, Dhruvil Shah, Florian
Hussonnois, Gardner Vickers, Gemma Singleton, Grant Henke, Greg Harris,
Gunnar Morling, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, highluck, huxi,
Igor Soarez, Ismael Juma, James Cheng, Jason Gustafson, Jeff Huang, Joel
Hamill, John Roesler, jolshan, José Armando García Sancio, Jukka Karvanen,
Justine Olshan, Kamal Chandraprakash, Karan Kumar, Kengo Seki, Kevin Lu,
khairy, Konstantine Karantasis, Lee Dongjin, Lifei Chen, Lucas Bradstreet,
LuyingLiu, Magesh Nandakumar, Manikumar Reddy, Matthias J. Sax, Michał
Borowiecki, Michał Siatkowski, Mickael Maison, mjarvie, mmanna-sapfgl,
Nacho Muñoz Gómez, Nathan Murthy, Nigel Liang, NIkhil Bhatia, Nikolay, Omar
Al-Safi, Omkar Mestry, Paul, pkleindl, Rajan Chauhan, Rajini Sivaram,
Randall Hauch, Richard Yu, Robert Yokota, Rohan, Ron Dagostino, Ryanne
Dolan, saisandeep, Scott Hendricks, sdreynolds, Sean Glover, Sergey
Prokofiev, slim, soondenana, Stanislav Kozlovski, Stanislav Vodetskyi,
SuryaTeja Duggi, tadsul, teebee, Tirtha Chatterjee, Tu Tran, Tu V. Tran,
Vahid Hashemian, Victoria Bialas, Vikas Singh, Viktor Somogyi, Viktor
Somogyi-Vass, vinoth chandar, wenhoujx, Wennn, Will James, wineandcheeze,
Yaroslav Tkachenko, 康智冬

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Manikumar

[RESULTS] [VOTE] Release Kafka version 2.4.0

[Manikumar]

This vote passes with 6 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Gwen Shapira
* Jun Rao
* Guozhang Wang

Committers:
* Mickael Maison

Community:
* Adam Bellemare
* Israel Ekpo

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://markmail.org/message/qlira627sqbmmzz4

I'll continue with the release process and the release announcement will
follow in the next few days.

Manikumar

Re: [VOTE] 2.4.0 RC4

[Manikumar]

Hi all,

Successful Jenkins builds for the 2.4 branch:
Unit/integration tests:
https://builds.apache.org/blue/organizations/jenkins/kafka-2.4-jdk8/detail/kafka-2.4-jdk8/106/
System tests:
http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-12-12--001.1576143096--confluentinc--2.4--9a7f26d/report.html



On Mon, Dec 9, 2019 at 11:02 PM Manikumar  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fifth candidate for release of Apache Kafka 2.4.0.
>
> This release includes many new features, including:
> - Allow consumers to fetch from closest replica
> - Support for incremental cooperative rebalancing to the consumer
> rebalance protocol
> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
> engine
> - New Java authorizer Interface
> - Support for non-key joining in KTable
> - Administrative API for replica reassignment
> - Sticky partitioner
> - Return topic metadata and configs in CreateTopics response
> - Securing Internal connect REST endpoints
> - API to delete consumer offsets and expose it via the AdminClient.
>
> Release notes for the 2.4.0 release:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, December 12, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc4/javadoc/
>
> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
> https://github.com/apache/kafka/releases/tag/2.4.0-rc4
>
> * Documentation:
> https://kafka.apache.org/24/documentation.html
>
> * Protocol:
> https://kafka.apache.org/24/protocol.html
>
> Thanks,
> Manikumar
>

Re: [VOTE] 2.4.0 RC4

[Manikumar]

Hi Israel,

Thanks for testing the release candidate.
We can use  https://issues.apache.org/jira/browse/KAFKA-9283 for fixing the
flaky test.

Thanks,

On Thu, Dec 12, 2019 at 8:18 PM Israel Ekpo  wrote:

>
> @Manikumar , I have rerun the latest release
> candidate (RC4) and all the tests have passed with no errors.
>
> Should we go ahead and close this issue regarding the Flaky test? -
> https://issues.apache.org/jira/browse/KAFKA-9283
>
>
> On Tue, Dec 10, 2019 at 11:11 AM Adam Bellemare 
> wrote:
>
>> - All PGP signatures are good
>> - All md5, sha1sums and sha512sums pass
>>
>> Initial test results:
>> 1310 tests completed, 2 failed, 17 skipped
>>
>> > Task :core:integrationTest FAILED
>>
>> The failed tests:
>> SaslSslAdminClientIntegrationTest. testElectPreferredLeaders
>> SslAdminClientIntegrationTest.
>> testSynchronousAuthorizerAclUpdatesBlockRequestThreads
>>
>> Both failed due to timeout:
>>
>> java.util.concurrent.ExecutionException:
>> org.apache.kafka.common.errors.TimeoutException: Aborted due to
>> timeout.
>>
>> Reran the tests and both passed.
>>
>> +1 from me.
>>
>>
>>
>>
>>
>> On Mon, Dec 9, 2019 at 12:32 PM Manikumar 
>> wrote:
>>
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the fifth candidate for release of Apache Kafka 2.4.0.
>> >
>> > This release includes many new features, including:
>> > - Allow consumers to fetch from closest replica
>> > - Support for incremental cooperative rebalancing to the consumer
>> rebalance
>> > protocol
>> > - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter
>> replication
>> > engine
>> > - New Java authorizer Interface
>> > - Support for non-key joining in KTable
>> > - Administrative API for replica reassignment
>> > - Sticky partitioner
>> > - Return topic metadata and configs in CreateTopics response
>> > - Securing Internal connect REST endpoints
>> > - API to delete consumer offsets and expose it via the AdminClient.
>> >
>> > Release notes for the 2.4.0 release:
>> > https://home.apache.org/~manikumar/kafka-2.4.0-rc4/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by Thursday, December 12, 9am PT
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > https://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > https://home.apache.org/~manikumar/kafka-2.4.0-rc4/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> >
>> > * Javadoc:
>> > https://home.apache.org/~manikumar/kafka-2.4.0-rc4/javadoc/
>> >
>> > * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
>> > https://github.com/apache/kafka/releases/tag/2.4.0-rc4
>> >
>> > * Documentation:
>> > https://kafka.apache.org/24/documentation.html
>> >
>> > * Protocol:
>> > https://kafka.apache.org/24/protocol.html
>> >
>> > Thanks,
>> > Manikumar
>> >
>>
>

Re: [kafka-clients] [VOTE] 2.4.0 RC3

[Manikumar]

Hi all,

We have merged the PR for KAFKA-9212. Thanks to Jason for the fixing the
issue.
Thanks to Yannick for reporting the issue and Michael Jaschob for providing
extra details.

I am canceling this vote and will create new RC shortly.

Thanks,
Manikumar

On Sat, Dec 7, 2019 at 3:19 AM Israel Ekpo  wrote:

>
> Ran the tests in the following environments using Eric's script available
> here:
>
> https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh
>
> OS: Ubuntu 18.04.3 LTS
> Java Version: OpenJDK 11.0.4
> Scala Versions: 2.12.10, 12.13.0, 12.13.1
> Gradle Version: 5.6.2
>
> I have made one observation in the release artifacts here:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc3/kafka-2.4.0-src.tgz
>
> The latest available release for Scala is 12.13.1
>
> It looks like the artifacts were built in an environment where the scala
> version in the CLI is set to 2.12.10 so this is the value in
> gradle.properties for the source artifact here:
>
> scalaVersion=2.12.10
>
> Not sure if that needs to change but it seems in the previous releases
> this is usually set to the highest available scala version at the time of
> release.
>
> Also, the test
> kafka.admin.ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker
> failed the first time I ran all the tests but it passed the second and
> third time I ran the group kafka.admin.ReassignPartitionsClusterTest and
> just the single method
> (kafka.admin.ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker).
>
>
> I will re-run the tests again in Scala versions 12.13.0, 12.13.1 and share
> my observations later.
>
> So far, it looks good.
>
> isekpo@ossvalidator:~$ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:Ubuntu 18.04.3 LTS
> Release:18.04
> Codename:   bionic
>
> isekpo@ossvalidator:~$ java -version
> openjdk version "11.0.4" 2019-07-16
> OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3,
> mixed mode, sharing)
>
> isekpo@ossvalidator:~$ scala -version
> Scala code runner version 2.12.10 -- Copyright 2002-2019, LAMP/EPFL and
> Lightbend, Inc.
> isekpo@ossvalidator:~$
>
> isekpo@ossvalidator:~$ gradle -version
>
> 
> Gradle 5.6.2
> 
>
> Build time:   2019-09-05 16:13:54 UTC
> Revision: 55a5e53d855db8fc7b0e494412fc624051a8e781
>
> Kotlin:   1.3.41
> Groovy:   2.5.4
> Ant:  Apache Ant(TM) version 1.9.14 compiled on March 12 2019
> JVM:  11.0.4 (Ubuntu 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> OS:   Linux 5.0.0-1027-azure amd64
>
> 1309 tests completed, 1 failed, 17 skipped
>
> > Task :core:integrationTest FAILED
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':core:integrationTest'.
> > There were failing tests. See the report at:
> file:///home/isekpo/scratchpad/14891.out/kafka-2.4.0-src/core/build/reports/tests/integrationTest/index.html
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output. Run with --scan to get full insights.
>
> * Get more help at https://help.gradle.org
>
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 6.0.
> Use '--warning-mode all' to show the individual deprecation warnings.
> See
> https://docs.gradle.org/5.6.2/userguide/command_line_interface.html#sec:command_line_warnings
>
> BUILD FAILED in 53m 52s
> 14 actionable tasks: 3 executed, 11 up-to-date
>
>
> Details for failed test:
>
> shouldMoveSinglePartitionWithinBroker
> org.scalatest.exceptions.TestFailedException: Partition should have been
> moved to the expected log directory
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
> at
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
> at org.scalatest.Assertions.fail(Assertions.scala:1091)
> at org.scalatest.Assertions.fail$(Assertions.scala:1087)
> at org.scalatest.Assertions$.fail(Assertions.scala:1389)
> at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:842)
> at
> kafka.admin.ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker(ReassignPartitionsClusterTest.scala:177)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)

Re: [VOTE] 2.4.0 RC2

[Manikumar]

Hi all,

We merged all the blockers and released a new release candidate RC3.
Hopefully this will be a final RC for 2.4 release.


Thanks,


On Thu, Dec 5, 2019 at 6:56 AM Israel Ekpo  wrote:

> Thanks @Randall for the update. I will run the tests on RC3 when it becomes
> available.
>
> On Wed, Dec 4, 2019 at 12:42 PM Randall Hauch  wrote:
>
> > Ah, I merged https://issues.apache.org/jira/browse/KAFKA-9258 last night
> > and forgot to send an email to this thread.
> >
> > Thanks!
> >
> > Randall
> >
> > On Wed, Dec 4, 2019 at 10:42 AM Eric Lalonde  wrote:
> >
> > > > I have also shared a pull request for the changes I have made to the
> > test
> > > > script
> > > >
> > > > https://github.com/elalonde/kafka/pull/2
> > >
> > >
> > > Thanks, this has been merged.
> > >
> >
>

[VOTE] 2.4.0 RC3

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the fourth candidate for release of Apache Kafka 2.4.0.

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for  non-key joining in KTable
- Administrative API for replica reassignment
- Sticky partitioner
- Return topic metadata and configs in CreateTopics response
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

Release notes for the 2.4.0 release:
https://home.apache.org/~manikumar/kafka-2.4.0-rc3/RELEASE_NOTES.html

*** Please download, test and vote by Monday, December 9th, 9 pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-2.4.0-rc3/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-2.4.0-rc3/javadoc/

* Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
https://github.com/apache/kafka/releases/tag/2.4.0-rc3

* Documentation:
https://kafka.apache.org/24/documentation.html

* Protocol:
https://kafka.apache.org/24/protocol.html

Thanks,
Manikumar

Re: [VOTE] 2.4.0 RC2

[Manikumar]

Hi,

Thank you all for reporting the issues.

We'll consider the below JIRAs as blockers for 2.4.0 release.
https://issues.apache.org/jira/browse/KAFKA-9231
https://issues.apache.org/jira/browse/KAFKA-9258
https://issues.apache.org/jira/browse/KAFKA-9156

I will create new release candidate after merging the PRs for above JIRAs.

@Eric
Thanks for actively verifying the releases. I'll will run the failed test
locally.

Thanks,

On Tue, Dec 3, 2019 at 3:01 AM Eric Lalonde  wrote:

> Hi all,
>
> I ran the following:
>
> $ git clone https://github.com/elalonde/kafka <
> https://github.com/elalonde/kafka>
> $ ./kafka/bin/verifykafka-rc.sh 2.4.0
> https://home.apache.org/~manikumar/kafka-2.4.0-rc2 <
> https://home.apache.org/~manikumar/kafka-2.4.0-rc2>
>
> - All sha1, sha256 and md5 sums look good
> - pgp signatures look good
> - unit tests pass
> - Integration tests pass with the following issue:
>
> <=> 12% EXECUTING [30m 21s]
> <=> 12% EXECUTING [30m 22s]mpleted, 16 skipped
> > IDLE
> <=> 12% EXECUTING [30m 27s]
> > IDLE
> > IDLE
> > IDLE
> > :core:integrationTest > Executing test
> kafka.api.AuthorizerIntegrationTest
> > IDLE
> > IDLE
>
>
> After 30 minutes the above test had not finished, seemed stalled, so I
> cancelled it. LMK if I am overly impatient.
>
>

[VOTE] 2.4.0 RC2

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 2.4.0.

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for  non-key joining in KTable
- Administrative API for replica reassignment
- Sticky partitioner
- Return topic metadata and configs in CreateTopics response
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

Release notes for the 2.4.0 release:
https://home.apache.org/~manikumar/kafka-2.4.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, December 5th, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-2.4.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-2.4.0-rc2/javadoc/

* Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
https://github.com/apache/kafka/releases/tag/2.4.0-rc2

* Documentation:
https://kafka.apache.org/24/documentation.html

* Protocol:
https://kafka.apache.org/24/protocol.html

Thanks,
Manikumar

Re: [VOTE] 2.4.0 RC1

[Manikumar]

Hi All,

We will consider KAFKA-9244
<https://issues.apache.org/jira/browse/KAFKA-9244> as blocker and include
the fix in 2.4 release.

I am canceling this VOTE and will create third release candidate.

Thank you all for testing.

On Fri, Nov 29, 2019 at 10:52 AM Matthias J. Sax 
wrote:

> I did not find the bug -- it was reported by Kin Sui
> (https://issues.apache.org/jira/browse/KAFKA-9244)
>
> If the bug is a blocker is a judgment call thought, because it's
> technically not a regression. However, if we don't include the fix in
> 2.4.0, as Adam pointed out, the new foreign-key join would compute
> incorrect results, and thus, it's at least a critical issue.
>
>
> -Matthias
>
>
>
> On 11/28/19 11:48 AM, Adam Bellemare wrote:
> > mjsax found an important issue for the foreign-key joiner, which I think
> > should be a blocker (if it isn't already) since it is functionally
> > incorrect without the fix:
> >
> > https://github.com/apache/kafka/pull/7758
> >
> >
> >
> > On Tue, Nov 26, 2019 at 6:26 PM Sean Glover 
> > wrote:
> >
> >> Hi,
> >>
> >> I also used Eric's test script.  I had a few issues running it that I
> >> address below[0][1], otherwise looks good.
> >>
> >> - Signing keys all good
> >> - All md5, sha1sums and sha512sums are good
> >> - A couple transient test failures that passed on a second run
> >> (ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker,
> >> SaslScramSslEndToEndAuthorizationTest.
> >> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl)
> >> - Passes our own test suite for Alpakka Kafka (
> >> https://travis-ci.org/akka/alpakka-kafka/builds/616861540,
> >> https://github.com/akka/alpakka-kafka/pull/971)
> >>
> >> +1 (non-binding)
> >>
> >> ..
> >>
> >> Issues while running test script:
> >>
> >> [0] Error with Eric test script.  I had an issue running the script
> with my
> >> version of bash (TMPDIR was unassigned), which I provided a PR for (
> >> https://github.com/elalonde/kafka/pull/1)
> >> [1] Gradle incompatibility. I ran into difficulty running the gradle
> build
> >> with the latest version of gradle (6.0.1).  I had to revert to the last
> >> patch of version 5 (5.6.4):
> >>
> >>  ✘ seglo@slice 
> /tmp/verify-kafka-SP06GE1GpP/10169.out/kafka-2.4.0-src 
> >> gradle wrapper --warning-mode all
> >>
> >>> Configure project :
> >> The maven plugin has been deprecated. This is scheduled to be removed in
> >> Gradle 7.0. Please use the maven-publish plugin instead.
> >> at
> >>
> >>
> build_c0129pbfzzxjolwxmds3lsevz$_run_closure5.doCall(/tmp/verify-kafka-SP06GE1GpP/10169.out/kafka-2.4.0-src/build.gradle:160)
> >> (Run with --stacktrace to get the full stack trace of this
> >> deprecation warning.)
> >>
> >> FAILURE: Build failed with an exception.
> >>
> >> * Where:
> >> Build file
> >> '/tmp/verify-kafka-SP06GE1GpP/10169.out/kafka-2.4.0-src/build.gradle'
> line:
> >> 472
> >>
> >> * What went wrong:
> >> A problem occurred evaluating root project 'kafka-2.4.0-src'.
> >>> Could not create task ':clients:spotbugsMain'.
> >>> Could not create task of type 'SpotBugsTask'.
> >>   > Could not create an instance of type
> >> com.github.spotbugs.internal.SpotBugsReportsImpl.
> >>  >
> >>
> >>
> org.gradle.api.reporting.internal.TaskReportContainer.(Ljava/lang/Class;Lorg/gradle/api/Task;)V
> >>
> >> * Try:
> >> Run with --stacktrace option to get the stack trace. Run with --info or
> >> --debug option to get more log output. Run with --scan to get full
> >> insights.
> >>
> >> * Get more help at https://help.gradle.org
> >>
> >> BUILD FAILED in 699ms
> >>
> >> On Tue, Nov 26, 2019 at 1:31 PM Manikumar 
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Please download, test and vote the RC1 in order to provide quality
> >>> assurance for the forthcoming 2.4 release.
> >>>
> >>> Thanks.
> >>>
> >>> On Tue, Nov 26, 2019 at 8:11 PM Adam Bellemare <
> adam.bellem...@gmail.com
> >>>
> >>> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> Ran Eric's test script:
> >>>> $ git clone https://github.com

Re: [VOTE] 2.4.0 RC1

[Manikumar]

Hi All,

Please download, test and vote the RC1 in order to provide quality
assurance for the forthcoming 2.4 release.

Thanks.

On Tue, Nov 26, 2019 at 8:11 PM Adam Bellemare 
wrote:

> Hello,
>
> Ran Eric's test script:
> $ git clone https://github.com/elalonde/kafka
> $ ./kafka/bin/verify-kafka-rc.sh 2.4.0
> https://home.apache.org/~manikumar/kafka-2.4.0-rc1
> <https://home.apache.org/~manikumar/kafka-2.4.0-rc0>
>
> - All PGP signatures are good
> - All md5, sha1sums and sha512sums pass
> - Had a few intermittent failures in tests that passed upon rerunning.
>
> +1 (non-binding) from me.
>
> Adam
>
> On Wed, Nov 20, 2019 at 10:37 AM Manikumar 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 2.4.0.
> >
> > This release includes many new features, including:
> > - Allow consumers to fetch from closest replica
> > - Support for incremental cooperative rebalancing to the consumer
> rebalance
> > protocol
> > - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter
> replication
> > engine
> > - New Java authorizer Interface
> > - Support for  non-key joining in KTable
> > - Administrative API for replica reassignment
> > - Sticky partitioner
> > - Return topic metadata and configs in CreateTopics response
> > - Securing Internal connect REST endpoints
> > - API to delete consumer offsets and expose it via the AdminClient.
> >
> > Release notes for the 2.4.0 release:
> > https://home.apache.org/~manikumar/kafka-2.4.0-rc1/RELEASE_NOTES.html
> >
> > ** Please download, test and vote by Tuesday, November 26, 9am PT **
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~manikumar/kafka-2.4.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~manikumar/kafka-2.4.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.4.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/24/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/24/protocol.html
> >
> > Thanks,
> > Manikumar
> >
>

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

[Manikumar]

Sure, I will mention the details about ZK upgrade to 3.5

Thanks.

On Tue, Nov 26, 2019 at 1:57 PM Sönke Liebau
 wrote:

> Hi Manikumar,
>
> looks great, thanks for the effort!
>
> Like Ismael, I think it would make sense to mention the ZK upgrade to 3.5 I
> know a few people that are waiting for this because of the SSL support so
> there is probably more of them out there :)
>
> Best regards,
> Sönke
>
>
> On Mon, 25 Nov 2019 at 20:11, Ismael Juma  wrote:
>
> > Manikumar,
> >
> > One thing I had previously missed, should we mention the upgrade to ZK
> 3.5?
> >
> > Ismael
> >
> > On Thu, Nov 14, 2019 at 10:41 AM Manikumar 
> > wrote:
> >
> > > Hi all,
> > >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.4.0
> > > release.
> > > Please take a look and let me know if you want to add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>

[VOTE] 2.4.0 RC1

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 2.4.0.

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for  non-key joining in KTable
- Administrative API for replica reassignment
- Sticky partitioner
- Return topic metadata and configs in CreateTopics response
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

Release notes for the 2.4.0 release:
https://home.apache.org/~manikumar/kafka-2.4.0-rc1/RELEASE_NOTES.html

** Please download, test and vote by Tuesday, November 26, 9am PT **

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-2.4.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-2.4.0-rc1/javadoc/

* Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
https://github.com/apache/kafka/releases/tag/2.4.0-rc1

* Documentation:
https://kafka.apache.org/24/documentation.html

* Protocol:
https://kafka.apache.org/24/protocol.html

Thanks,
Manikumar

Re: [VOTE] 2.4.0 RC0

[Manikumar]

Hi All,

Thanks you for testing RC0. The following blocker issues were fixed after
RC0.

https://issues.apache.org/jira/browse/KAFKA-9196
https://github.com/apache/kafka/pull/7691

I am canceling RC0 VOTE and will create new RC soon.

Thanks,
Manikumar

On Thu, Nov 14, 2019 at 11:51 PM Manikumar 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 2.4.0.
> There is work in progress for couple blockers PRs. I am publishing RC0 to
> avoid further delays in testing the release.
>
> This release includes many new features, including:
> - Allow consumers to fetch from closest replica
> - Support for incremental cooperative rebalancing to the consumer
> rebalance protocol
> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
> engine
> - New Java authorizer Interface
> - Support for  non-key joining in KTable
> - Administrative API for replica reassignment
> - Sticky partitioner
> - Return topic metadata and configs in CreateTopics response
> - Securing Internal connect REST endpoints
> - API to delete consumer offsets and expose it via the AdminClient.
>
> Release notes for the 2.4.0 release:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test  by  Thursday, November 20, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
>
> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
> https://github.com/apache/kafka/releases/tag/2.4.0-rc0
>
> * Documentation:
> https://kafka.apache.org/24/documentation.html
>
> * Protocol:
> https://kafka.apache.org/24/protocol.html
>
> Thanks,
> Manikumar
>

Preliminary blog post about the Apache Kafka 2.4.0 release

[Manikumar]

Hi all,

I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0
release.
Please take a look and let me know if you want to add/modify details.
Thanks to all who contributed to this blog post.

https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1

Thanks,
Manikumar

[VOTE] 2.4.0 RC0

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.4.0.
There is work in progress for couple blockers PRs. I am publishing RC0 to
avoid further delays in testing the release.

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for  non-key joining in KTable
- Administrative API for replica reassignment
- Sticky partitioner
- Return topic metadata and configs in CreateTopics response
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

Release notes for the 2.4.0 release:
https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html

*** Please download, test  by  Thursday, November 20, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-2.4.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/

* Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
https://github.com/apache/kafka/releases/tag/2.4.0-rc0

* Documentation:
https://kafka.apache.org/24/documentation.html

* Protocol:
https://kafka.apache.org/24/protocol.html

Thanks,
Manikumar

Re: [ANNOUNCE] New committer: Mickael Maison

[Manikumar]

Congrats Mickeal!

On Fri, Nov 8, 2019 at 9:05 AM Dong Lin  wrote:

> Congratulations Mickael!
>
> On Thu, Nov 7, 2019 at 1:38 PM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > Mickael
> > Maison.
> >
> > Mickael has been contributing to Kafka since 2016. He proposed and
> > implemented multiple KIPs. He has also been propomating Kafka through
> blogs
> > and public talks.
> >
> > Congratulations, Mickael!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
> >
>

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi all,

I put together a draft release plan with Oct 2019 as the release month and
a list of KIPs that have already been voted:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901

Here are the dates:

1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this date in order
to be considered for this release)

2) Feature Freeze:  Oct 2, 2019 (Major features merged & working on
stabilization, minor features have PR,
 release branch cut; anything not in this state will be automatically moved
to the next release in JIRA.

3) Code Freeze:  Oct 16, 2019

4) Release Date: Oct 30, 2019 (tentative)

Please plan accordingly for the features you want push into Apache Kafka
2.4.0 release.

Regards,
Manikumar

On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma  wrote:

> Thanks for volunteering Manikumar. +1
>
> Ismael
>
> On Mon, Aug 12, 2019 at 7:54 AM Manikumar 
> wrote:
>
> > Hi all,
> >
> > I would like to volunteer to be the release manager for our next
> time-based
> > feature release (v2.4.0).
> >
> > If that sounds good, I'll post the release plan over the next few days.
> >
> > Thanks,
> > Manikumar
> >
>

[DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi all,

I would like to volunteer to be the release manager for our next time-based
feature release (v2.4.0).

If that sounds good, I'll post the release plan over the next few days.

Thanks,
Manikumar

Re: Kafka - JMX Metrics for Fetch and Produce

[Manikumar]

Hi,

RequestsPerSec metric was changed to include the API version in Kafka 2.0
release.
You may need to update jmx query.

http://kafka.apache.org/documentation/#upgrade_200_notable
https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric

On Mon, Apr 22, 2019 at 12:23 AM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> Hi there,
>
>
>
> Recently I performed several * rolling upgrades following official steps*
> for our Kafka brokers *from 0.11.0.1 to newer versions in different
> environments*, and apparently working fine in all cases from functional
> point of view: *producers and consumers working as expected*.
>
>
>
> Specifically, I upgraded:
>
>
>
>1. From *0.11.0.1 to 1.0.0*, and then from *1.0.0 to 2.0.0*, and then*
>to* *2.1.1*
>2. *From 0.11.0.1 directly to 2.1.1*
>
>
>
> However, in all cases *JMX metrics for Fetch and Produce* which used to
> show *all producers and consumers working with brokers* are gone, just
> showing queue-size, in our *JMX monitoring clients* -specifically, *Introscope
> Wily*- keeping same configuration*.*
>
>
>
> In fact, I removed Wily filter configuration for JMX in *order to show
> all possible metrics, and keeping both Fetch and Produce still gone*.
>
>
>
> Note I checked if having proper version after rolling upgrade, for
> example, for *2.1.1*, and being as expected:
>
>
>
> *ll /opt/kafka/libs/*
>
> *total 54032*
>
> *-rw-r--r-- 1 kafka kafka69409 Jan  4 08:42 activation-1.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka14768 Jan  4 08:42
> aopalliance-repackaged-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka90347 Jan  4 08:42 argparse4j-0.7.0.jar*
>
> *-rw-r--r-- 1 kafka kafka20437 Jan  4 08:40
> audience-annotations-0.5.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   501879 Jan  4 08:43 commons-lang3-3.8.1.jar*
>
> *-rw-r--r-- 1 kafka kafka96801 Feb  8 18:32 connect-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka18265 Feb  8 18:32
> connect-basic-auth-extension-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka20509 Feb  8 18:32 connect-file-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka45489 Feb  8 18:32 connect-json-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   466588 Feb  8 18:32 connect-runtime-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka90358 Feb  8 18:32
> connect-transforms-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka  2442625 Jan  4 08:43 guava-20.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   186763 Jan  4 08:42 hk2-api-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   189454 Jan  4 08:42 hk2-locator-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   135317 Jan  4 08:42 hk2-utils-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka66894 Jan 11 21:28
> jackson-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   325619 Jan 11 21:27 jackson-core-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka  1347236 Jan 11 21:27 jackson-databind-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32373 Jan 11 21:28
> jackson-jaxrs-base-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka15861 Jan 11 21:28
> jackson-jaxrs-json-provider-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32627 Jan 11 21:28
> jackson-module-jaxb-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   737884 Jan  4 08:43 javassist-3.22.0-CR2.jar*
>
> *-rw-r--r-- 1 kafka kafka26366 Jan  4 08:42
> javax.annotation-api-1.2.jar*
>
> *-rw-r--r-- 1 kafka kafka 2497 Jan  4 08:42 javax.inject-1.jar*
>
> *-rw-r--r-- 1 kafka kafka 5951 Jan  4 08:42 javax.inject-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka95806 Jan  4 08:42
> javax.servlet-api-3.1.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   126898 Jan  4 08:42 javax.ws.rs-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   127509 Jan  4 08:42 javax.ws.rs-api-2.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   125632 Jan  4 08:42 jaxb-api-2.3.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   181563 Jan  4 08:42 jersey-client-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka  1140395 Jan  4 08:43 jersey-common-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka18085 Jan  4 08:42
> jersey-container-servlet-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka59332 Jan  4 08:42
> jersey-container-servlet-core-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka62547 Jan  4 08:42 jersey-hk2-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka71936 Jan  4 08:42 jersey-media-jaxb-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka   933619 Jan  4 08:43 jersey-server-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka   280573 Jan  4 08:43
> jetty-client-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka16680 Jan  4 08:42
> jetty-continuation-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   195664 Jan  4 08:42
> jetty-http-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   139436 Jan  4 08:42
> jetty-io-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka93077 Jan  4 08:42
> jetty-security-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   609290 Jan  4 08:43
> jetty-server-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   110983 Jan  4 08:42
> jetty-servlet-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 

Re: [ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

[Manikumar]

Congrats Harsha!.

On Fri, Apr 19, 2019 at 7:43 AM Dong Lin  wrote:

> Congratulations Sriharsh!
>
> On Thu, Apr 18, 2019 at 11:46 AM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Sriharsh Chintalapan has been active in the Kafka community since he
> became
> > a Kafka committer in 2015. I am glad to announce that Harsh is now a
> member
> > of Kafka PMC.
> >
> > Congratulations, Harsh!
> >
> > Jun
> >
>

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

[Manikumar]

Congrats Matthias!. well deserved.

On Fri, Apr 19, 2019 at 7:44 AM Dong Lin  wrote:

> Congratulations Matthias!
>
> Very well deserved!
>
> On Thu, Apr 18, 2019 at 2:35 PM Guozhang Wang  wrote:
>
> > Hello Everyone,
> >
> > I'm glad to announce that Matthias J. Sax is now a member of Kafka PMC.
> >
> > Matthias has been a committer since Jan. 2018, and since then he
> continued
> > to be active in the community and made significant contributions the
> > project.
> >
> >
> > Congratulations to Matthias!
> >
> > -- Guozhang
> >
>

Re: Kafka console consumer group

[Manikumar]

Hi,

auto offset commit is enabled by default  in consumer.  When auto offset
commit is enabled,
consumer commits the offset (based on auto.offset.reset config, default :
latest), when it first joins the group.

Thanks,
Manikumar

On Fri, Mar 29, 2019 at 8:33 PM Sharmadha Sainath 
wrote:

> Hi all,
>
>
> I have a basic question regarding kafka consumer groups. I will detail it
> with an example :
>
>
> 1.I run a console-producer on a topic and produce messages :
>
>
> kafka-console-producer.sh --broker-list localhost:9092 --topic first_topic
>
> >message1
>
> >message2
>
>
>
> 2.I run console-consumer with —group first_group without —from-beginning
>
>
> kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> first_topic --group  first_group
>
>
> (no messages come)
>
>
>
> 3. when  I described first_group using kafka-consumer-group , I see the
> current offset is 2
>
>
>
> Now I run console-consumer with —group second_group with —from-beginning I
> see the messages  :
>
>
> kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> first_topic —group  second_group —from-beginning
>
>
> message1
>
> message2
>
>
> My question here is , how with the first_group , based on --from-beginning
> flag , current offset is set to 2 without actually reading any messages?
>
>
> Please let me know If I am missing something basic here.
>
>
> Thanks in advance.
>
>
>
> Regards,
>
> Sharmadha S.
>

Re: [VOTE] 2.2.0 RC2

[Manikumar]

+1 (non-binding)

- Verified the artifacts, build from src, ran tests
- Verified the quickstart, ran producer/consumer performance tests.

Thanks for running release!.

Thanks,
Manikumar

On Wed, Mar 20, 2019 at 12:19 AM David Arthur 
wrote:

> +1
>
> Validated signatures, and ran through quick-start.
>
> Thanks!
>
> On Mon, Mar 18, 2019 at 4:00 AM Jakub Scholz  wrote:
>
> > +1 (non-binding). I used the staged binaries and run some of my tests
> > against them. All seems to look good to me.
> >
> > On Sat, Mar 9, 2019 at 11:56 PM Matthias J. Sax 
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka 2.2.0.
> > >
> > >  - Added SSL support for custom principal name
> > >  - Allow SASL connections to periodically re-authenticate
> > >  - Command line tool bin/kafka-topics.sh adds AdminClient support
> > >  - Improved consumer group management
> > >- default group.id is `null` instead of empty string
> > >  - API improvement
> > >- Producer: introduce close(Duration)
> > >- AdminClient: introduce close(Duration)
> > >- Kafka Streams: new flatTransform() operator in Streams DSL
> > >- KafkaStreams (and other classed) now implement AutoClosable to
> > > support try-with-resource
> > >- New Serdes and default method implementations
> > >  - Kafka Streams exposed internal client.id via ThreadMetadata
> > >  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
> > > output `NaN` as default value
> > > Release notes for the 2.2.0 release:
> > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test, and vote by Thursday, March 14, 9am PST.
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag:
> > > https://github.com/apache/kafka/releases/tag/2.2.0-rc2
> > >
> > > * Documentation:
> > > https://kafka.apache.org/22/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/22/protocol.html
> > >
> > > * Jenkins builds for the 2.2 branch:
> > > Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/
> > > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/2.2/
> > >
> > > /**
> > >
> > > Thanks,
> > >
> > > -Matthias
> > >
> > >
> >
>

Re: Kafka 2.1.0 JMX mbean broken ?

[Manikumar]

Hi Raghav,

As you know, KIP-372 added "version" tag to RequestsPerSec metric to
monitor requests for each version.
As mentioned in the KIP, to get total count per request (across all
versions), we need to aggregate over all versions.
You may need to automate this by fetching all metrics using wildcard and
aggregating over all versions.
We may need to use JMX readers which support wildcard queries.

All possible protocol request versions are available here:
https://kafka.apache.org/protocol.html#The_Messages_Produce


On Sat, Mar 9, 2019 at 5:59 AM Raghav  wrote:

> Hello Allen, Ted, Mani, Gwen, James
>
> https://www.mail-archive.com/dev@kafka.apache.org/msg86226.html
>
> I read this email thread and I see that you guys were split in going ahead
> with this change. This has broken our dashboard in 2.1 Kafka, which is ok
> as long as we know how to fix it.
>
> Can you please help us figure out the answer for the email below ? It will
> be greatly appreciated. We just want to know how to find the version number
> ?
>
> Many thanks.
>
> R
>
> On Fri, Dec 14, 2018 at 5:16 PM Raghav  wrote:
>
>> I got it to work. I fired up a console and then saw what beans are
>> registered there and then queries using the code. It works then.
>>
>> But the apiVersion is different and how do we know what apiVersion a
>> producer is using ? Our dashboards cannot have a-priori knowledge of the
>> version numbers, and wildcards don't work. See the screenshot below,
>> apiVersion is 7. Where did this come from ? Can someone please help to
>> understand.
>>
>> [image: jmx.png]
>>
>>
>>
>> On Fri, Dec 14, 2018 at 4:29 PM Raghav  wrote:
>>
>>> Is this a test case for this commit:
>>> https://github.com/apache/kafka/pull/4506 ? I have tried using all
>>> possible cases but cannot get it to work.
>>>
>>> I cannot get it to work using mbean reader via JMX. Can any one please
>>> help? Or atleast confirm that it is broken. Thanks
>>>
>>> R
>>>
>>> On Fri, Dec 14, 2018 at 6:34 AM Raghav  wrote:
>>>
 Thanks Ismael. How to query it in 2.1 ? I tried all possible ways
 including using version, but I am still getting the same exception message.

 Thanks for your help.

 On Thu, Dec 13, 2018 at 7:19 PM Ismael Juma  wrote:

> The metric was changed to include the API version. I believe this was
> in
> the upgrade notes for 2.0.0.
>
> Ismael
>
> On Thu, Dec 13, 2018, 3:35 PM Raghav 
> > Hi
> >
> > We are trying to move from Kafka 1.1.0 to Kafka 2.1.0. We used to
> monitor
> > our 3 node Kafka using JMX. Upon moving to 2.1.0, we have observed
> that the
> > *below* mentioned metric can't be retrie
> > and we get the below exception:
> >
> >
> *"kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce"*
> >
> > javax.management.InstanceNotFoundException:
> > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce
> > at
> >
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
> > at
> >
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1375)
> > at
> >
> >
> com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1462)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:905)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
> > at sun.rmi.transport.Transport$1.run(Transport.java:200)
> > at sun.rmi.transport.Transport$1.run(Transport.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> > at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> > at
> >
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> > at
> >
> >
> 

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Manikumar]

Congrats, Vahid!

On Wed 16 Jan, 2019, 6:53 AM Ismael Juma  Congratulations Vahid!
>
> On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson 
> > Hi All,
> >
> > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > we are
> > pleased to announce that he has accepted!
> >
> > Vahid has made numerous contributions to the Kafka community over the
> past
> > few years. He has authored 13 KIPs with core improvements to the consumer
> > and the tooling around it. He has also contributed nearly 100 patches
> > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > effort into community engagement, helping others on the mail lists and
> > sharing his experience at conferences and meetups.
> >
> > We appreciate the contributions and we are looking forward to more.
> > Congrats Vahid!
> >
> > Jason, on behalf of the Apache Kafka PMC
> >
>

Re: Setup GC log rotate policy for kafka broker servers

[Manikumar]

Hi Parth,

We need to pass JVM system properties to rotate gc logs. These props added
in Kafka 0.11.release.
For releases before 0.11, we need to add these JVM props. Broker restart is
required for changes to take effect.

https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L290



On Fri, Nov 30, 2018 at 6:17 PM Parth Gandhi <
parth.gan...@excellenceinfonet.com> wrote:

> Team,
>
> I took over kafka from a colleague and found that the GC log has been setup
> without setting up the log rotate policy. Now the GC log size has reached
> to 4 GB and we need to move this file and also set up a log rotate policy.
> Can this be achieved without restarting the kafak service? We have a kafka
> cluster with 3 broker and will have to set this up on all three servers. I
> am new to Java and any help on this is highly appreciated.
>
> Thanks,
> Parth Gandhi
> DevOps
>
> Disclaimer
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast Ltd, an innovator in Software as a
> Service (SaaS) for business. Providing a safer and more useful place for
> your human generated data. Specializing in; Security, archiving and
> compliance. To find out more visit the Mimecast website.
>

[ANNOUNCE] Apache Kafka 2.0.1

[Manikumar]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.0.1

This is a bug fix release and it includes fixes and improvements from 51
JIRAs.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.0.1/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.11 and Scala 2.12)
from:
https://kafka.apache.org/downloads#2.0.1

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 40 contributors to this release!

Amit Sela, Anna Povzner, Arjun Satish, Bibin Sebastian, Bill Bejeck, Bob
Barrett, Bridger Howell,
Colin Hicks, Dhruvil Shah, Dong Lin, Flavien Raynaud, Guozhang Wang, huxi,
huxihx, Ismael Juma,
Jason Gustafson, Joan Goyeau, John Roesler, Jon Lee, Kamal Chandraprakash,
Kevin Lafferty,
Konstantine Karantasis, lambdaliu, Lincong Li, Lucas Wang, Maciej Bryński,
Manikumar Reddy,
Matthias J. Sax, Max Zheng, Michal Dziemianko, Michał Borowiecki,
radai-rosenblatt, Rajini Sivaram,
Randall Hauch, Robert Yokota, Simon Clark, Stanislav Kozlovski, Sébastien
Launay, tedyu,
Zhanxiang (Patrick) Huang

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Manikumar

[RESULTS] [VOTE] Release Kafka version 2.0.1

[Manikumar]

This vote passes with 12 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Jun Rao
* Gwen Shapira
* Rajini Sivaram

Committers:
* Harsha
* Ewen Cheslack-Postava
* Manikumar

Community:
* Magnus Edenhill
* Eno Thereska
* Vahid Hashemian
* Jakub Scholz
* Satish Duggana
* Mickael Maison

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://markmail.org/thread/3gy4a6fmjv2xuoou

Thank you all for the votes.
I'll continue with the release process and the release announcement will
follow in the next few days.

Manikumar

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

Yes, I was about to send closing mail.

On Thu, Nov 8, 2018 at 9:39 PM Ismael Juma  wrote:

> Manikumar, looks like you have the necessary votes. :)
>
> Ismael
>
> On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  wrote:
>
>> +1 (binding)
>>
>> Checked source build and unit tests. Ran quickstart with source and
>> binary.
>>
>> Thank you for managing the release, Manikumar!
>>
>> Regards,
>>
>> Rajini
>>
>> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>>
>> > +1 (binding)
>> >
>> > Checked signatures, build and quickstart.
>> >
>> > Thank you for managing the release, Mani!
>> >
>> >
>> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> > wrote:
>> > >
>> > > Hello Kafka users, developers and client-developers,
>> > >
>> > > This is the first candidate for release of Apache Kafka 2.0.1.
>> > >
>> > > This is a bug fix release closing 49 tickets:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> > >
>> > > Release notes for the 2.0.1 release:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> > >
>> > > *** Please download, test and vote by  Tuesday, October 30, end of day
>> > >
>> > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > http://kafka.apache.org/KEYS
>> > >
>> > > * Release artifacts to be voted upon (source and binary):
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> > >
>> > > * Maven artifacts to be voted upon:
>> > > https://repository.apache.org/content/groups/staging/
>> > >
>> > > * Javadoc:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> > >
>> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> > >
>> > > * Documentation:
>> > > http://kafka.apache.org/20/documentation.html
>> > >
>> > > * Protocol:
>> > > http://kafka.apache.org/20/protocol.html
>> > >
>> > > * Successful Jenkins builds for the 2.0 branch:
>> > > Unit/integration tests:
>> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> > >
>> > > /**
>> > >
>> > > Thanks,
>> > > Manikumar
>> >
>> >
>> >
>> > --
>> > Gwen Shapira
>> > Product Manager | Confluent
>> > 650.450.2760 | @gwenshap
>> > Follow us: Twitter | blog
>> >
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZYww8XbFFwx5Os4pOjiANGz-Lq81xcSGiTXs-G%2BwpG%3Duw%40mail.gmail.com
> <https://groups.google.com/d/msgid/kafka-clients/CAD5tkZYww8XbFFwx5Os4pOjiANGz-Lq81xcSGiTXs-G%2BwpG%3Duw%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

+1 (binding)
Built from src and ran tests, verified the artifacts, signatures, Ran core
quick start.

Thanks,
Manikumar

On Thu, Nov 8, 2018 at 12:15 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
>> +1 (binding)
>>
>> Checked signatures, build and quickstart.
>>
>> Thank you for managing the release, Mani!
>>
>>
>> On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> wrote:
>> >
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the first candidate for release of Apache Kafka 2.0.1.
>> >
>> > This is a bug fix release closing 49 tickets:
>> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> >
>> > Release notes for the 2.0.1 release:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by  Tuesday, October 30, end of day
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > http://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/
>> >
>> > * Javadoc:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> >
>> > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> >
>> > * Documentation:
>> > http://kafka.apache.org/20/documentation.html
>> >
>> > * Protocol:
>> > http://kafka.apache.org/20/protocol.html
>> >
>> > * Successful Jenkins builds for the 2.0 branch:
>> > Unit/integration tests:
>> https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> >
>> > /**
>> >
>> > Thanks,
>> > Manikumar
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAOJcB38LDf1Fgkr8HMG2b2%2B1tA5usQ%3DeeRv0Vjcj63Tq2zoYZQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/kafka-clients/CAOJcB38LDf1Fgkr8HMG2b2%2B1tA5usQ%3DeeRv0Vjcj63Tq2zoYZQ%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: [VOTE] 2.0.1 RC0

[Manikumar]

We were waiting for the system test results. There were few failures:
KAFKA-7579,  KAFKA-7559, KAFKA-7561
they are not blockers for 2.0.1 release. We need more votes from
PMC/committers :)

Thanks Stanislav! for the system test results.

Thanks,
Manikumar

On Thu, Nov 1, 2018 at 10:20 PM Eno Thereska  wrote:

> Anything else holding this up?
>
> Thanks
> Eno
>
> On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz  wrote:
>
> > +1 (non-binding) ... I used the staged binaries and run tests with
> > different clients.
> >
> > On Fri, Oct 26, 2018 at 4:29 AM Manikumar 
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 2.0.1.
> > >
> > > This is a bug fix release closing 49 tickets:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > >
> > > Release notes for the 2.0.1 release:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by  Tuesday, October 30, end of day
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > >
> > > * Documentation:
> > > http://kafka.apache.org/20/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/20/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > >
> > > /**
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Re: How to turn-off Time-Based index files in Kafka?

[Manikumar]

We can't disable time-based indexing. What is the issue you are facing?

On Thu, Oct 25, 2018 at 6:04 PM Ashwin Sinha 
wrote:

> Hi Users,
>
> How to turn off time-based index files in Kafka (.timeindex files). Tried
> searching for it in topic and broker configs documentation but could not
> find it.
>
> --
> *Ashwin Sinha *| Data Engineer
> ashwin.si...@go-mmt.com  | 9452075361
>  
> 
>
> --
>
>
> ::DISCLAIMER::
>
>
>
> 
>
>
>
>
>
> This message is intended only for the use of the addressee and may
> contain information that is privileged, confidential and exempt from
> disclosure under applicable law. If the reader of this message is not the
> intended recipient, or the employee or agent responsible for delivering
> the
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify us
> immediately by return e-mail and delete this e-mail and all attachments
> from your system.
>

[VOTE] 2.0.1 RC0

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.0.1.

This is a bug fix release closing 49 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1

Release notes for the 2.0.1 release:
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by  Tuesday, October 30, end of day

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/

* Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
https://github.com/apache/kafka/releases/tag/2.0.1-rc0

* Documentation:
http://kafka.apache.org/20/documentation.html

* Protocol:
http://kafka.apache.org/20/protocol.html

* Successful Jenkins builds for the 2.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/177/

/**

Thanks,
Manikumar

[DISCUSS] 2.0.1 bug fix release

[Manikumar]

Hi all,

I would like to volunteer to be the release manager for 2.0.1 bug fix
release.
2.0 was released July 30, 2018 and 44 issues are fixed so far.

Please find all the resolved tickets here:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.0.1

Please find the Release plan:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1

If you have any JIRA in progress and would like to include it in this
release, please discuss with your reviewer.
There is currently only one blocking issue (
https://issues.apache.org/jira/browse/KAFKA-7464).

Next week, Once the blocking issue gets addressed,  I plan to create the
first RC for 2.0.1 release.

Thanks,
Manikumar

Re: [ANNOUNCE] New Committer: Manikumar Reddy

[Manikumar]

Thanks everyone!

It has been a great learning experience working Apache Kafka community.
I am thankful to the committers/community members for all the support and
help.

Thanks
Manikumar

On Fri, Oct 12, 2018 at 1:55 PM Edoardo Comar  wrote:

> Well done Manikumar !
> --
>
> Edoardo Comar
>
> IBM Event Streams
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
>
> From:   "Matthias J. Sax" 
> To: dev 
> Cc: users 
> Date:   11/10/2018 23:41
> Subject:Re: [ANNOUNCE] New Committer: Manikumar Reddy
>
>
>
> Congrats!
>
>
> On 10/11/18 2:31 PM, Yishun Guan wrote:
> > Congrats Manikumar!
> > On Thu, Oct 11, 2018 at 1:20 PM Sönke Liebau
> >  wrote:
> >>
> >> Great news, congratulations Manikumar!!
> >>
> >> On Thu, Oct 11, 2018 at 9:08 PM Vahid Hashemian
> 
> >> wrote:
> >>
> >>> Congrats Manikumar!
> >>>
> >>> On Thu, Oct 11, 2018 at 11:49 AM Ryanne Dolan 
> >>> wrote:
> >>>
> >>>> Bravo!
> >>>>
> >>>> On Thu, Oct 11, 2018 at 1:48 PM Ismael Juma 
> wrote:
> >>>>
> >>>>> Congratulations Manikumar! Thanks for your continued contributions.
> >>>>>
> >>>>> Ismael
> >>>>>
> >>>>> On Thu, Oct 11, 2018 at 10:39 AM Jason Gustafson
> 
> >>>>> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> The PMC for Apache Kafka has invited Manikumar Reddy as a committer
> >>> and
> >>>>> we
> >>>>>> are
> >>>>>> pleased to announce that he has accepted!
> >>>>>>
> >>>>>> Manikumar has contributed 134 commits including significant work to
> >>> add
> >>>>>> support for delegation tokens in Kafka:
> >>>>>>
> >>>>>> KIP-48:
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>
> >>>>>> KIP-249
> >>>>>> <
> >>>>>
> >>>>
> >>>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+KafkaKIP-249
>
> >>>>>>
> >>>>>> :
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
>
> >>>>>>
> >>>>>> He has broad experience working with many of the core components in
> >>>> Kafka
> >>>>>> and he has reviewed over 80 PRs. He has also made huge progress
> >>>>> addressing
> >>>>>> some of our technical debt.
> >>>>>>
> >>>>>> We appreciate the contributions and we are looking forward to more.
> >>>>>> Congrats Manikumar!
> >>>>>>
> >>>>>> Jason, on behalf of the Apache Kafka PMC
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Sönke Liebau
> >> Partner
> >> Tel. +49 179 7940878
> >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
> [attachment "signature.asc" deleted by Edoardo Comar/UK/IBM]
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: kafka-consumers-group.sh version 2.0.0 does not recognize --zookeeper option

[Manikumar]

Deprecated scala consumers/producers are removed from 2.0 release. Old
consumer related options are removed from /kafka-consumer-groups.sh tool.
http://kafka.apache.org/documentation/#upgrade_200_notable

We missed updating some of the docs. Will be fixed in next release docs.

On Thu, Oct 4, 2018 at 10:01 PM Spadoni Marco 
wrote:

> Hi all,
> I installed kafka_2.11-2.0.0 tarball on a CentOS server :
> [kafka@oxelkk-bo-1:~] $ uname -a
> Linux oxelkk-bo-1 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04
> UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> [kafka@oxelkk-bo-1:~] $
>
> equipped with this JVM:
> [kafka@oxelkk-bo-1:~] $ java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [kafka@oxelkk-bo-1:~] $
>
> I have a strange behavior of the utility bin/kafka-consumer-groups.sh when
> it is called with --zookeeper option, like:
>
> [kafka@oxelkk-bo-1:~] $
> /store1/elk/run/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh --zookeeper
> oxelkk-bo-1.iol.local:2181 --list
> Exception in thread "main" joptsimple.UnrecognizedOptionException:
> zookeeper is not a recognized option
> at
> joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
> at
> joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
> at
> joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
> at joptsimple.OptionParser.parse(OptionParser.java:396)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupCommandOptions.(ConsumerGroupCommand.scala:725)
> at
> kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:42)
> at
> kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> [kafka@oxelkk-bo-1:~] $
>
> Whilst if I use (just for a test) the same utility coming with
> kafka_2.11-0.11.0.0, I have the expected response (the groups are related
> to two Druid Tranquility tasks):
>
> [kafka@oxelkk-bo-1:~] $
> /store1/elk/run/kafka_2.11-0.11.0.0/bin/kafka-consumer-groups.sh
> --zookeeper oxelkk-bo-1.iol.local:2181 --list
> Note: This will only show information about consumers that use ZooKeeper
> (not those using the Java consumer API).
>
> tranquility-kafka-proxy
> tranquility-kafka-ssoeven
> [kafka@oxelkk-bo-1:~] $
>
> I noticed that the documentation for Kafka 2.0.0 at:
> https://kafka.apache.org/documentation/#basic_ops_consumer_lag states:
>
> This tool also works with ZooKeeper-based consumers:
> bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group
> my-group
>
> Do you have any hint on this behavior?
> Thank you very much,
>
> Marco Spadoni
> Italiaonline S.p.A.
>
>
>
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify
> postmas...@italiaonline.it .
> www.italiaonline.it
>
>

Re: Kafka ACL issue - Operation denied despite having full access to the topic

[Manikumar]

PrincipalType  string "User" is case sensitive. Try creating acls for
"User:storm-mytestcluster" principal.

On Sat, Sep 29, 2018 at 9:04 PM Bala  wrote:

>  Yes, I followed all the docs and thing is it's connecting as expected and
> sending the right user to the server. The problem is that when I list ACL's
> I am seeing that the user has all the access. But when the producer trying
> to connect to the server, the server is logging message saying that the
> user is denied. Something is not right and I want to know how to debug this
> further to find out why the server thinks the user has no access.
>
> I even tried adding the user to the super users and still not able to
> access the topic.
> I am using Kafka 1.0.0.
> On Friday, September 28, 2018, 8:14:30 PM EDT, Vahid Hashemian <
> vahid.hashem...@gmail.com> wrote:
>
>  Your produce needs to have Write access to the topic. But as you mentioned
> All should cover Write. Which version of Kafka are you using?
> FYI, more authn/authz information can be found here for some of the common
> client operations:
> https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
>
> --Vahid
>
> On Fri, Sep 28, 2018 at 9:13 AM Bala  wrote:
>
> > Producer using the Java API. I did configure the jaas config as per docs.
> > It looks like is working and the authentication is succeeded but the
> > authorization is not honoring the ACL
> >On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <
> > vahid.hashem...@gmail.com> wrote:
> >
> >  Hi Bala,
> >
> > What operation/command are you trying that gives you this error?
> >
> > --Vahid
> >
> > On Fri, Sep 28, 2018 at 7:12 AM Bala  wrote:
> >
> > > I have a kafka with kerberos security and trying to use the ACL and am
> > not
> > > able to make it work.
> > >
> > > Here is the error I am seeing in the server log.[2018-09-28
> 14:06:54,152]
> > > INFO Principal = User:storm-mytestcluster is Denied Operation =
> Describe
> > > from host =  on resource = Topic:icd_alpha
> > > (kafka.authorizer.logger)
> > > [2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
> > > Denied Operation = Describe from host =  on
> resource
> > =
> > > Topic:icd_alpha (kafka.authorizer.logger)
> > > [2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
> > > Denied Operation = Describe from host =  on
> resource
> > =
> > > Topic:icd_alpha (kafka.authorizer.logger)
> > > [2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
> > > Denied Operation = Describe from host =  on
> resource
> > =
> > > Topic:icd_alpha (kafka.authorizer.logger)
> > > [2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
> > > Denied Operation = Describe from host =  on
> resource
> > =
> > > Topic:icd_alpha (kafka.authorizer.logger)
> > > [2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
> > > Denied Operation = Describe from host =  on
> resource
> > =
> > > Topic:icd_alpha (kafka.authorizer.logger)
> > >
> > >
> > >
> > > But the user has full access to the topic: Here is the output of `list
> `
> > > command
> > >
> > > Current ACLs for resource `Topic:icd_alpha`:
> > >  user:storm-mytestcluster has Allow permission for operations: All
> > > from hosts: *
> > >
> > > Please help me, as I am kind of blocked and don't know how to proceed
> > > further.
> > > ThanksBala
> > >
> >
>

Re: unable to start kafka when zookeeper cluster is in working but unhealthy state

[Manikumar]

Yes, In case of  UnknownHostException, zookeeper client will try to connect
remaining hostnames given in the zk connect string.

On Wed, Sep 26, 2018 at 7:45 PM James Yu  wrote:

> @Liam, the hostname is removed from dns server due to the node is no longer
> alive, so kafka is unable reolve IP for zookeeper-2 thus an
> NullPointerException is thrown.
> @Manikumar, ZOOKEEPER-2184 is to re-resolve IP for new instance of
> zookeeper-2, however, zookeeper-2 stays down hence now IP to be resolved.
> Or does it also work for this case?
>
> Thanks for your responses.
>
> This is a UTF-8 formatted mail
> ---
> James C.-C.Yu
> +886988713275
> +8615618429976
>
> 2018-09-26 19:36 GMT+08:00 Manikumar :
>
> > You can try using Kafka 2.0 release. Original issue is handled in
> > ZOOKEEPER-2184 and corresponding
> > zookeeper version is used in Kafka 2.0.
> >
> >
> > On Wed, Sep 26, 2018 at 3:58 PM Liam Clarke 
> > wrote:
> >
> > > Hi James,
> > >
> > > That's not an unresponsive node that's killing Kafka, that's a failure
> to
> > > resolve the address that's killing it - my personal expectation would
> be
> > > that even though zookeeper-2.zookeeper.etc may be down, its name should
> > > still resolve.
> > >
> > > Regards,
> > >
> > > Liam Clarke
> > >
> > >
> > >
> > > On Wed, Sep 26, 2018 at 10:02 PM James Yu  wrote:
> > >
> > > > Hi,
> > > >
> > > > I fail to start kafka broker when the corresponding zookeeper cluster
> > is
> > > in
> > > > working but unhealthy state.
> > > >
> > > > The zookeeper cluster is made of 3 nodes: zookeeper-0, zookeeper-1,
> > > > zookeeper-2.
> > > >
> > > > I put all 3 zookeeper nodes into kafka's server.properties,
> > specifically
> > > > for zookeeper.connect attribute as as:
> > > >
> > > >
> > > zookeeper.connect=zookeeper-0.zookeeper.default.svc.cluster.
> > local:2181,zookeeper-1.zookeeper.default.svc.cluster.
> > local:2181,zookeeper-2.zookeeper.default.svc.cluster.local:2181
> > > >
> > > > I intentionally terminate zookeeper-2 and let zookeeper cluster work
> > with
> > > > only two nodes (zookeeper-0, zookeeper-1). Then I start start kafka
> and
> > > get
> > > > the following error message:
> > > > [2018-09-26 09:57:23,841] INFO Initiating client connection,
> > > >
> > > >
> > > connectString=zookeeper-0.zookeeper.default.svc.cluster.
> > local:2181,zookeeper-1.zookeeper.default.svc.cluster.
> > local:2181,zookeeper-2.zookeeper.default.svc.cluster.local:2181
> > > > sessionTimeout=6000
> > > > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@
> > 49ec71f8
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2018-09-26 09:57:24,100] ERROR Fatal error during KafkaServer
> startup.
> > > > Prepare to shutdown (kafka.server.KafkaServer)
> > > > java.net.UnknownHostException:
> > > > zookeeper-2.zookeeper.default.svc.cluster.local: Name or service not
> > > known
> > > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > > > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.
> > java:928)
> > > > at
> > > > java.net.InetAddress.getAddressesFromNameService(
> > InetAddress.java:1323)
> > > > at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> > > > at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> > > > at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> > > > at
> > > >
> > > >
> > > org.apache.zookeeper.client.StaticHostProvider.(
> > StaticHostProvider.java:61)
> > > > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> > > > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> > > > at
> > > kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:86)
> > > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1538)
> > > > at
> > > >
> > > >
> > > kafka.server.KafkaServer.kafka$server$KafkaServer$$
> > createZkClient$1(KafkaServer.scala:345)
> > > > at kafka.server.KafkaServer.initZkClient(KafkaServer.
> > scala:369)
> > > >

Re: unable to start kafka when zookeeper cluster is in working but unhealthy state

[Manikumar]

You can try using Kafka 2.0 release. Original issue is handled in
ZOOKEEPER-2184 and corresponding
zookeeper version is used in Kafka 2.0.


On Wed, Sep 26, 2018 at 3:58 PM Liam Clarke 
wrote:

> Hi James,
>
> That's not an unresponsive node that's killing Kafka, that's a failure to
> resolve the address that's killing it - my personal expectation would be
> that even though zookeeper-2.zookeeper.etc may be down, its name should
> still resolve.
>
> Regards,
>
> Liam Clarke
>
>
>
> On Wed, Sep 26, 2018 at 10:02 PM James Yu  wrote:
>
> > Hi,
> >
> > I fail to start kafka broker when the corresponding zookeeper cluster is
> in
> > working but unhealthy state.
> >
> > The zookeeper cluster is made of 3 nodes: zookeeper-0, zookeeper-1,
> > zookeeper-2.
> >
> > I put all 3 zookeeper nodes into kafka's server.properties, specifically
> > for zookeeper.connect attribute as as:
> >
> >
> zookeeper.connect=zookeeper-0.zookeeper.default.svc.cluster.local:2181,zookeeper-1.zookeeper.default.svc.cluster.local:2181,zookeeper-2.zookeeper.default.svc.cluster.local:2181
> >
> > I intentionally terminate zookeeper-2 and let zookeeper cluster work with
> > only two nodes (zookeeper-0, zookeeper-1). Then I start start kafka and
> get
> > the following error message:
> > [2018-09-26 09:57:23,841] INFO Initiating client connection,
> >
> >
> connectString=zookeeper-0.zookeeper.default.svc.cluster.local:2181,zookeeper-1.zookeeper.default.svc.cluster.local:2181,zookeeper-2.zookeeper.default.svc.cluster.local:2181
> > sessionTimeout=6000
> > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@49ec71f8
> > (org.apache.zookeeper.ZooKeeper)
> > [2018-09-26 09:57:24,100] ERROR Fatal error during KafkaServer startup.
> > Prepare to shutdown (kafka.server.KafkaServer)
> > java.net.UnknownHostException:
> > zookeeper-2.zookeeper.default.svc.cluster.local: Name or service not
> known
> > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> > at
> > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> > at
> >
> >
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> > at
> kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:86)
> > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1538)
> > at
> >
> >
> kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:345)
> > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:369)
> > at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
> > at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> > at kafka.Kafka$.main(Kafka.scala:92)
> > at kafka.Kafka.main(Kafka.scala)
> > [2018-09-26 09:57:24,104] INFO shutting down (kafka.server.KafkaServer)
> > [2018-09-26 09:57:24,107] WARN  (kafka.utils.CoreUtils$)
> > java.lang.NullPointerException
> > at
> >
> >
> kafka.server.KafkaServer$$anonfun$shutdown$5.apply$mcV$sp(KafkaServer.scala:569)
> > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:85)
> > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:569)
> > at kafka.server.KafkaServer.startup(KafkaServer.scala:326)
> > at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> > at kafka.Kafka$.main(Kafka.scala:92)
> > at kafka.Kafka.main(Kafka.scala)
> > [2018-09-26 09:57:24,112] INFO shut down completed
> > (kafka.server.KafkaServer)
> > [2018-09-26 09:57:24,112] ERROR Exiting Kafka.
> > (kafka.server.KafkaServerStartable)
> > [2018-09-26 09:57:24,115] INFO shutting down (kafka.server.KafkaServer)
> >
> > I thought we can start a kafka as long as the zookeeper cluster is
> > working.regardless there is one zookeeper node (3 in total) fails to run?
> >
> >
> > This is a UTF-8 formatted mail
> > ---
> > James C.-C.Yu
> > +886988713275
> > +8615618429976
> >
>

Re: Kafka producer huge memory usage (leak?)

[Manikumar]

Hi,

In that case, it is not related to KAFKA-4493. I see StackOverFlowException
in the uploaded heapdump.
I am pretty sure this is related to StackOverFlowException. Some objects
are dangling after this exception.
You need to check/find out when this exception is occuring. check your
application server logs for any errors.
try increasing stack size by adding -Xss2048k.

thanks,

On Fri, Sep 21, 2018 at 2:44 PM Shantanu Deshmukh 
wrote:

> Hi Manikumar,
> I am using correct protocol. SSL and truststore is also correctly
> configured and there is a single port which is for SSL. Otherwise it simply
> wouldn't work. Data is getting produced all fine, it's just that producer
> object is consuming massive amount of memory.
>
> On Fri, Sep 21, 2018 at 2:36 PM Manikumar 
> wrote:
>
> > Hi,
> > Instead trying the PR,  make sure you are setting valid security protocol
> > and connecting to valid broker port.
> > also looks for any errors in producer logs.
> >
> > Thanks,
> >
> >
> >
> >
> >
> > On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh <
> shantanu...@gmail.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > I checked this issue. There is this patch available.
> > > https://github.com/apache/kafka/pull/2408.patch
> > >
> > > I pulled Kafka 0.10.1.0 from github. Then tried applying this patch.
> But
> > > several places I am getting error that patch doesn't apply.
> > > I am new to git and patching process. Can you guide me here?
> > >
> > > On Wed, Sep 19, 2018 at 1:02 PM Manikumar 
> > > wrote:
> > >
> > > > Similar issue reported here:KAFKA-7304, but on broker side.  maybe
> you
> > > can
> > > > create a JIRA and upload the heap dump for analysis.
> > > >
> > > > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> > > shantanu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Any thoughts on this matter? Someone, please help.
> > > > >
> > > > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> > > shantanu...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Additionally, here's the producer config
> > > > > >
> > > > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > > > kafka.acks=0
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > > kafka.max.block.ms=1000
> > > > > > kafka.request.timeout.ms=1000
> > > > > > kafka.max.in.flight.requests.per.connection=1
> > > > > > kafka.retries=0
> > > > > > kafka.compression.type=gzip
> > > > > > kafka.security.protocol=SSL
> > > > > >
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > > > kafka.ssl.truststore.password=XX
> > > > > > kafka.linger.ms=300
> > > > > > logger.level=INFO
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > > > shantanu...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello,
> > > > > >>
> > > > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production.
> There
> > > are
> > > > > >> some applications which have Kafka Producers embedded in them
> > which
> > > > send
> > > > > >> application logs to a topic. This topic has 10 partitions with
> > > > > replication
> > > > > >> factor of 3.
> > > > > >>
> > > > > >> We are observing that memory usage on some of these application
> > > > servers
> > > > > >> keep shooting through the roof intermittently. After taking
> > heapdump
> > > > we
> > > > > >> found out that top suspects were:
> > > > > >> *-*
> > > > > >>
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.Selector -*
> > > > > >>
> > > > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *""*.
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > > > >>
> > > > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *""*
> > > > > >>
> > > > > >> * - *
> > > > > >>
> > > > > >> Both of these were holding about 352MB of space. 3 such
> instances,
> > > so
> > > > > >> they were consuming about 1.2GB of memory.
> > > > > >>
> > > > > >> Now regarding usage of producers. Not a huge amount of logs are
> > > being
> > > > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one
> producer
> > > > > object
> > > > > >> is being used throughout application. Async send function is
> used.
> > > > > >>
> > > > > >> What could be the cause of such huge memory usage? Is this some
> > sort
> > > > of
> > > > > >> memory leak in this specific Kafka version?
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Kafka producer huge memory usage (leak?)

[Manikumar]

Hi,
Instead trying the PR,  make sure you are setting valid security protocol
and connecting to valid broker port.
also looks for any errors in producer logs.

Thanks,





On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh 
wrote:

> Hi Manikumar,
>
> I checked this issue. There is this patch available.
> https://github.com/apache/kafka/pull/2408.patch
>
> I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
> several places I am getting error that patch doesn't apply.
> I am new to git and patching process. Can you guide me here?
>
> On Wed, Sep 19, 2018 at 1:02 PM Manikumar 
> wrote:
>
> > Similar issue reported here:KAFKA-7304, but on broker side.  maybe you
> can
> > create a JIRA and upload the heap dump for analysis.
> >
> > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> shantanu...@gmail.com>
> > wrote:
> >
> > > Any thoughts on this matter? Someone, please help.
> > >
> > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> shantanu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Additionally, here's the producer config
> > > >
> > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > kafka.acks=0
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > kafka.max.block.ms=1000
> > > > kafka.request.timeout.ms=1000
> > > > kafka.max.in.flight.requests.per.connection=1
> > > > kafka.retries=0
> > > > kafka.compression.type=gzip
> > > > kafka.security.protocol=SSL
> > > > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > kafka.ssl.truststore.password=XX
> > > > kafka.linger.ms=300
> > > > logger.level=INFO
> > > >
> > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > shantanu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There
> are
> > > >> some applications which have Kafka Producers embedded in them which
> > send
> > > >> application logs to a topic. This topic has 10 partitions with
> > > replication
> > > >> factor of 3.
> > > >>
> > > >> We are observing that memory usage on some of these application
> > servers
> > > >> keep shooting through the roof intermittently. After taking heapdump
> > we
> > > >> found out that top suspects were:
> > > >> *-*
> > > >>
> > > >>
> > > >> *org.apache.kafka.common.network.Selector -*
> > > >>
> > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *""*.
> > > >>
> > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > >>
> > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *""*
> > > >>
> > > >> * - *
> > > >>
> > > >> Both of these were holding about 352MB of space. 3 such instances,
> so
> > > >> they were consuming about 1.2GB of memory.
> > > >>
> > > >> Now regarding usage of producers. Not a huge amount of logs are
> being
> > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > > object
> > > >> is being used throughout application. Async send function is used.
> > > >>
> > > >> What could be the cause of such huge memory usage? Is this some sort
> > of
> > > >> memory leak in this specific Kafka version?
> > > >>
> > > >>
> > >
> >
>

Re: How to enable Kafka Authentication as on-demand feature and later make mandatory feature?

[Manikumar]

One option is to enable multiple listeners (PLAINTEXT, SASL_PLAINTEXT)  on
brokers and slowly migrate the clients/services.

On Wed, Sep 19, 2018 at 10:38 PM Ashwin Sinha 
wrote:

> Hi,
>
> We have tried setting up Kafka Authentication by SASL/PLAIN
>  and
> SASL/SCRAM
>  (both
> w/o SSL). We found that SASL/SCRAM is more convenient as we do not need to
> restart Kafka/Zookeeper again and again on adding new credentials.
>
> Problem: if we enable cluster-wide authentication by adding broker
> parameters
> <
> https://kafka.apache.org/10/documentation.html#security_sasl_scram_brokerconfig
> >
> then existing consumers will stop getting the data until they use
> authentication
> properties file
> <
> https://kafka.apache.org/10/documentation.html#security_sasl_scram_clientconfig
> >
> at their end. Is there any way/mode/config where we can start it as an
> on-demand feature and slowly change to required(mandatory) feature? Because
> this sudden change can affect many crucial services and jobs.
>
> --
> *Ashwin Sinha *| Data Engineer
> ashwin.si...@go-mmt.com  | 9452075361
>  
> 
>
> --
>
>
> ::DISCLAIMER::
>
>
>
> 
>
>
>
>
>
> This message is intended only for the use of the addressee and may
> contain information that is privileged, confidential and exempt from
> disclosure under applicable law. If the reader of this message is not the
> intended recipient, or the employee or agent responsible for delivering
> the
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify us
> immediately by return e-mail and delete this e-mail and all attachments
> from your system.
>

Re: Kafka producer huge memory usage (leak?)

[Manikumar]

Similar issue reported here:KAFKA-7304, but on broker side.  maybe you can
create a JIRA and upload the heap dump for analysis.

On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh 
wrote:

> Any thoughts on this matter? Someone, please help.
>
> On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh 
> wrote:
>
> > Additionally, here's the producer config
> >
> > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > kafka.acks=0
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > kafka.max.block.ms=1000
> > kafka.request.timeout.ms=1000
> > kafka.max.in.flight.requests.per.connection=1
> > kafka.retries=0
> > kafka.compression.type=gzip
> > kafka.security.protocol=SSL
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > kafka.ssl.truststore.password=XX
> > kafka.linger.ms=300
> > logger.level=INFO
> >
> > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh  >
> > wrote:
> >
> >> Hello,
> >>
> >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
> >> some applications which have Kafka Producers embedded in them which send
> >> application logs to a topic. This topic has 10 partitions with
> replication
> >> factor of 3.
> >>
> >> We are observing that memory usage on some of these application servers
> >> keep shooting through the roof intermittently. After taking heapdump we
> >> found out that top suspects were:
> >> *-*
> >>
> >>
> >> *org.apache.kafka.common.network.Selector -*
> >>
> >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *""*.
> >>
> >> *org.apache.kafka.common.network.KafkaChannel -*
> >>
> >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
> >> instance of *"byte[]"* loaded by *""*
> >>
> >> * - *
> >>
> >> Both of these were holding about 352MB of space. 3 such instances, so
> >> they were consuming about 1.2GB of memory.
> >>
> >> Now regarding usage of producers. Not a huge amount of logs are being
> >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> object
> >> is being used throughout application. Async send function is used.
> >>
> >> What could be the cause of such huge memory usage? Is this some sort of
> >> memory leak in this specific Kafka version?
> >>
> >>
>

Re: Determining broker max buffer size from producer.

[Manikumar]

You can use AdminClient describeConfigs API to fetch topic/broker configs.

https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html

On Tue, Sep 4, 2018 at 1:47 PM claude.war...@wipro.com <
claude.war...@wipro.com> wrote:

> Thank you for the reference but I don't see how it answers my question.
>
>
> Basically, I am looking for a Java call that I can make from the producer
> to determine the maximum message size set on the broker.
>
>
> Assume a process is starting and it has the connection information for the
> broker so it can create a producer.  While the producer can assume that the
> max buffersize is the same as the default this may not be the case.  The
> buffer may have been increased or decreased.  My process would like to know
> what the limit is so I can determine how to split very data streams being
> serialized through kafka.  Is there a way to do this without reading the
> broker configuration (which is probably not available to my process anyway)
>
>
> Claude
>
> 
> From: Satish Duggana 
> Sent: Monday, September 3, 2018 11:03:32 AM
> To: users@kafka.apache.org
> Subject: Re: Determining broker max buffer size from producer.
>
> ** This mail has been sent from an external source. Treat hyperlinks and
> attachments in this email with caution**
>
> Are you asking about the maximum message size that can be sent from a
> producer?
> max.message.bytes is used to control that at broker/topic level. You can
> find more details
> at the below links.
>
> *"The largest record batch size allowed by Kafka. If this is increased and
> there are consumers older than 0.10.2, the consumers' fetch size must also
> be increased so that the they can fetch record batches this large. In the
> latest message format version, records are always grouped into batches for
> efficiency. In previous message format versions, uncompressed records are
> not grouped into batches and this limit only applies to a single record in
> that case."*
>
>
> https://kafka.apache.org/documentation/#topicconfigs
> https://kafka.apache.org/documentation/#brokerconfigs
>
> Thanks,
> Satish.
>
> On Mon, Sep 3, 2018 at 2:12 PM, claude.war...@wipro.com <
> claude.war...@wipro.com> wrote:
>
> > Greetings,
> >
> >
> > I have spent several hours looking through documentation and historical
> > email chains and have been unable to find a solution to my problem.
> >
> >
> > I have a case where I need to construct a producer and I need to know the
> > broker max buffer size so that I don't get any nasty buffer too big
> errors
> > from the server when serializing large objects.  I have been unable to
> find
> > any way to do this short of reading the broker configuration and I can't
> do
> > that from a remote call (as far as I can tell).  Does anybody have a
> > solution?
> >
> >
> > Claude
> >
> > The information contained in this electronic message and any attachments
> > to this message are intended for the exclusive use of the addressee(s)
> and
> > may contain proprietary, confidential or privileged information. If you
> are
> > not the intended recipient, you should not disseminate, distribute or
> copy
> > this e-mail. Please notify the sender immediately and destroy all copies
> of
> > this message and any attachments. WARNING: Computer viruses can be
> > transmitted via email. The recipient should check this email and any
> > attachments for the presence of viruses. The company accepts no liability
> > for any damage caused by any virus transmitted by this email.
> > www.wipro.com
> >
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>

Re: Offset for consumer group set to Null in __consumer_offsets, leads to data loss

[Manikumar]

What is the value set for offsets.retention.minutes?  Default value is
`1day` in Kafka 0.11 release.
Based on your requirement, you may need to increase this value.

On Mon, Aug 27, 2018 at 1:33 PM Biplob Biswas 
wrote:

> Hi,
>
> Can anyone provide any insights to the issue we faced?
>
> Thanks & Regards
> Biplob Biswas
>
>
> On Fri, Aug 24, 2018 at 4:50 PM Biplob Biswas 
> wrote:
>
> > Hi everyone,
> >
> > *Short Version:*
> > We had an unusual situation where after restart of our spark job, rather
> > than reading from the last known offset of the consumer group, it started
> > consuming from the latest offset, thus leading to data loss.
> >
> > *Long Version:*
> > We have a spark job which crashed last Friday due to an NPE introduced by
> > developer code. Over the weekend the job was not restarted, although on
> > monday the bug was fixed and the job was restarted. The consumer-group
> was
> > *NOT* changed, but when we restarted the job, the spark job started
> > reading form the latest offset.
> >
> > We started investigating the reasons for the same and we verified our
> > spark job configuration and it looks good, we tried to reproduce the
> issue
> > by killing the job and restarting while new events are coming in and it
> > reads the buffered data properly. We also verified that the consumer
> group
> > had a valid lag which was reducing over time.
> >
> > Next we checked __consumer_offsets topic and we found the something which
> > could explain why spark job started reading fro latest. We found the
> > following lines which were not compressed by the topic itself
> >
> > [spark-job-consumer-group,spark-job-reading-topic,1]::NULL
> >> [spark-job-consumer-group,spark-job-reading-topic,3]::NULL
> >> [spark-job-consumer-group,spark-job-reading-topic,0]::NULL
> >> [spark-job-consumer-group,spark-job-reading-topic,2]::NULL
> >
> > --
> > These following offset commits came after we started ingesting again on
> > monday after bug fix:
> >
> >
> [spark-job-consumer-group,spark-job-reading-topic,3]::[OffsetMetadata[606979,NO_METADATA],CommitTime
> >> 1534778505013,ExpirationTime 1534864905013]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,1]::[OffsetMetadata[607195,NO_METADATA],CommitTime
> >> 1534778505013,ExpirationTime 1534864905013]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,2]::[OffsetMetadata[607100,NO_METADATA],CommitTime
> >> 1534778505013,ExpirationTime 1534864905013]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,0]::[OffsetMetadata[607330,NO_METADATA],CommitTime
> >> 1534778505013,ExpirationTime 1534864905013]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,3]::[OffsetMetadata[606979,NO_METADATA],CommitTime
> >> 1534778508010,ExpirationTime 1534864908010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,1]::[OffsetMetadata[607195,NO_METADATA],CommitTime
> >> 1534778508010,ExpirationTime 1534864908010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,2]::[OffsetMetadata[607100,NO_METADATA],CommitTime
> >> 1534778508010,ExpirationTime 1534864908010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,0]::[OffsetMetadata[607330,NO_METADATA],CommitTime
> >> 1534778508010,ExpirationTime 1534864908010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,3]::[OffsetMetadata[606979,NO_METADATA],CommitTime
> >> 1534778511010,ExpirationTime 1534864911010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,1]::[OffsetMetadata[607195,NO_METADATA],CommitTime
> >> 1534778511010,ExpirationTime 1534864911010]
> >>
> [spark-job-consumer-group,spark-job-reading-topic,2]::[OffsetMetadata[607100,NO_METADATA],CommitTime
> >> 1534778511010,ExpirationTime 1534864911010]
> >
> >
> >
> > But this only showed the reason of the behaviour not the cause, so we
> > looked into kafka logs -  during the time from when our kafka job died
> and
> > when we restarted, we found a few exception although these doesn't look
> > like that it could be the root cause but I still wanted to make sure:
> >
> >>
> >> 2018-08-20 09:01:20,016 INFO kafka.controller.KafkaController:
> >> [Controller 180]: Starting preferred replica leader election for
> partitions
> >> [__consumer_offsets,2]
> >> 2018-08-20 09:01:20,016 INFO kafka.controller.PartitionStateMachine:
> >> [Partition state machine on Controller 180]: Invoking state change to
> >> OnlinePartition for partitions [__consumer_offsets,2]
> >> 2018-08-20 09:01:20,019 INFO
> >> kafka.controller.PreferredReplicaPartitionLeaderSelector:
> >> [PreferredReplicaPartitionLeaderSelector]: Current leader 180 for
> partition
> >> [__consumer_offsets,2] is not the preferred replica. Triggering
> preferred
> >> replica leader election
> >> 2018-08-20 09:01:20,019 ERROR state.change.logger: Controller 180 epoch
> >> 37 encountered error while electing leader for partition
> >> [__consumer_offsets,2] due to: Preferred replica 182 for partition
> >> [__consumer_offsets,2] is either not alive or not in the isr. Current
> >> 

Re: Cannot create topics when zookeeper.set.acl=true

[Manikumar]

*zk does  NOT support PlainLoginModule.*

On Thu, Aug 23, 2018 at 5:54 PM Manikumar  wrote:

> No, zk does support PlainLoginModule. while using kafka-acls.sh script
> with kerberized zk,
> we need to pass required kerberos credentials.
>
> AdminClient API is an api to perform administrative actions (create/delete
> topics, create/delete acls etc..).
> This avoids direct communication with zk. Check below links for more
> details:
>
> *https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html
> <https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html>*
> Examples: https://github.com/apache/kafka/pull/5200/files
> http://kafka.apache.org/documentation/#adminclientconfigs
> To configure SASL/PLAIN on clients:
> http://kafka.apache.org/documentation/#security_sasl_plain_clientconfig
>
> On Thu, Aug 23, 2018 at 5:20 PM HG  wrote:
>
>> Hi,
>>
>> I am not using kerberos only
>>
>> Client {
>>org.apache.kafka.common.security.plain.PlainLoginModule required
>>username="user"
>>password="user-secret";
>> };
>>
>> Does that make a difference?
>>
>> What do you mean with AdminClient API?
>>
>> Regards Hans
>>
>> Op do 23 aug. 2018 om 13:34 schreef Manikumar > >:
>>
>> > We can pass jaas conf by exporting below variable before starting the
>> > kafka-acls.sh script. another option is to use AdminClient API.
>> >
>> > export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf
>> > -Djava.security.auth.login.config=zk_client_jaas.conf"
>> >
>> > zk_client_jaas.conf:
>> > // Zookeeper client authentication
>> > Client {
>> > com.sun.security.auth.module.Krb5LoginModule required
>> > useKeyTab=true
>> > storeKey=true
>> > keyTab="/etc/security/keytabs/kafka_server.keytab"
>> > principal="kafka/kafka1.hostname@example.com";
>> > };
>> >
>> > On Thu, Aug 23, 2018 at 4:44 PM HG  wrote:
>> >
>> > > Hi,
>> > >
>> > > I searched for an option with which I can provide credentials but I
>> did
>> > not
>> > > find them.
>> > > Is there another way to reach the same goal?
>> > > Regards Hans
>> > >
>> > > Op do 23 aug. 2018 om 13:00 schreef Manikumar <
>> manikumar.re...@gmail.com
>> > >:
>> > >
>> > > > "kafka-acls.sh" script  communicates directly with zookeeper.
>> > > > We should run kafka-acls.sh as kafka user (super user) to get write
>> > > > permission on zk.
>> > > > We should pass required jaas conf to the script.
>> > > >
>> > > > On Thu, Aug 23, 2018 at 3:02 PM HG 
>> wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I have an environment with SSL, SASL and ACL's enabled.
>> > > > > When I set zookeeper.set_acl=true in the server.properties file of
>> > the
>> > > > > brokers I cannot create topics , ACL's  etc.
>> > > > >
>> > > > > [root@host201 kafka]# bin/kafka-acls.sh --authorizer-properties
>> > > > > zookeeper.connect=localhost:2181 --add  --allow-principal
>> User:admin
>> > > > > --operation All --topic '*' --cluster
>> > > > > Error while executing ACL command: KeeperErrorCode = NoAuth for
>> > > > > /kafka-acl/Topic
>> > > > > org.apache.zookeeper.KeeperException$NoAuthException:
>> > KeeperErrorCode =
>> > > > > NoAuth for /kafka-acl/Topic
>> > > > > at
>> > > > >
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
>> > > > > ..
>> > > > > I have specified super users too in the server.properties.
>> > > > >
>> > > > > Any idea's what I am doing wrong?
>> > > > >
>> > > > > Regards Hans
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Cannot create topics when zookeeper.set.acl=true

[Manikumar]

No, zk does support PlainLoginModule. while using kafka-acls.sh script with
kerberized zk,
we need to pass required kerberos credentials.

AdminClient API is an api to perform administrative actions (create/delete
topics, create/delete acls etc..).
This avoids direct communication with zk. Check below links for more
details:

*https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html
<https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html>*
Examples: https://github.com/apache/kafka/pull/5200/files
http://kafka.apache.org/documentation/#adminclientconfigs
To configure SASL/PLAIN on clients:
http://kafka.apache.org/documentation/#security_sasl_plain_clientconfig

On Thu, Aug 23, 2018 at 5:20 PM HG  wrote:

> Hi,
>
> I am not using kerberos only
>
> Client {
>org.apache.kafka.common.security.plain.PlainLoginModule required
>username="user"
>password="user-secret";
> };
>
> Does that make a difference?
>
> What do you mean with AdminClient API?
>
> Regards Hans
>
> Op do 23 aug. 2018 om 13:34 schreef Manikumar :
>
> > We can pass jaas conf by exporting below variable before starting the
> > kafka-acls.sh script. another option is to use AdminClient API.
> >
> > export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf
> > -Djava.security.auth.login.config=zk_client_jaas.conf"
> >
> > zk_client_jaas.conf:
> > // Zookeeper client authentication
> > Client {
> > com.sun.security.auth.module.Krb5LoginModule required
> > useKeyTab=true
> > storeKey=true
> > keyTab="/etc/security/keytabs/kafka_server.keytab"
> > principal="kafka/kafka1.hostname@example.com";
> > };
> >
> > On Thu, Aug 23, 2018 at 4:44 PM HG  wrote:
> >
> > > Hi,
> > >
> > > I searched for an option with which I can provide credentials but I did
> > not
> > > find them.
> > > Is there another way to reach the same goal?
> > > Regards Hans
> > >
> > > Op do 23 aug. 2018 om 13:00 schreef Manikumar <
> manikumar.re...@gmail.com
> > >:
> > >
> > > > "kafka-acls.sh" script  communicates directly with zookeeper.
> > > > We should run kafka-acls.sh as kafka user (super user) to get write
> > > > permission on zk.
> > > > We should pass required jaas conf to the script.
> > > >
> > > > On Thu, Aug 23, 2018 at 3:02 PM HG 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have an environment with SSL, SASL and ACL's enabled.
> > > > > When I set zookeeper.set_acl=true in the server.properties file of
> > the
> > > > > brokers I cannot create topics , ACL's  etc.
> > > > >
> > > > > [root@host201 kafka]# bin/kafka-acls.sh --authorizer-properties
> > > > > zookeeper.connect=localhost:2181 --add  --allow-principal
> User:admin
> > > > > --operation All --topic '*' --cluster
> > > > > Error while executing ACL command: KeeperErrorCode = NoAuth for
> > > > > /kafka-acl/Topic
> > > > > org.apache.zookeeper.KeeperException$NoAuthException:
> > KeeperErrorCode =
> > > > > NoAuth for /kafka-acl/Topic
> > > > > at
> > > > >
> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> > > > > ..
> > > > > I have specified super users too in the server.properties.
> > > > >
> > > > > Any idea's what I am doing wrong?
> > > > >
> > > > > Regards Hans
> > > > >
> > > >
> > >
> >
>

Re: Cannot create topics when zookeeper.set.acl=true

[Manikumar]

We can pass jaas conf by exporting below variable before starting the
kafka-acls.sh script. another option is to use AdminClient API.

export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf
-Djava.security.auth.login.config=zk_client_jaas.conf"

zk_client_jaas.conf:
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab"
principal="kafka/kafka1.hostname@example.com";
};

On Thu, Aug 23, 2018 at 4:44 PM HG  wrote:

> Hi,
>
> I searched for an option with which I can provide credentials but I did not
> find them.
> Is there another way to reach the same goal?
> Regards Hans
>
> Op do 23 aug. 2018 om 13:00 schreef Manikumar :
>
> > "kafka-acls.sh" script  communicates directly with zookeeper.
> > We should run kafka-acls.sh as kafka user (super user) to get write
> > permission on zk.
> > We should pass required jaas conf to the script.
> >
> > On Thu, Aug 23, 2018 at 3:02 PM HG  wrote:
> >
> > > Hi,
> > >
> > > I have an environment with SSL, SASL and ACL's enabled.
> > > When I set zookeeper.set_acl=true in the server.properties file of the
> > > brokers I cannot create topics , ACL's  etc.
> > >
> > > [root@host201 kafka]# bin/kafka-acls.sh --authorizer-properties
> > > zookeeper.connect=localhost:2181 --add  --allow-principal User:admin
> > > --operation All --topic '*' --cluster
> > > Error while executing ACL command: KeeperErrorCode = NoAuth for
> > > /kafka-acl/Topic
> > > org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
> > > NoAuth for /kafka-acl/Topic
> > > at
> > > org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> > > ..
> > > I have specified super users too in the server.properties.
> > >
> > > Any idea's what I am doing wrong?
> > >
> > > Regards Hans
> > >
> >
>

Re: Cannot create topics when zookeeper.set.acl=true

[Manikumar]

"kafka-acls.sh" script  communicates directly with zookeeper.
We should run kafka-acls.sh as kafka user (super user) to get write
permission on zk.
We should pass required jaas conf to the script.

On Thu, Aug 23, 2018 at 3:02 PM HG  wrote:

> Hi,
>
> I have an environment with SSL, SASL and ACL's enabled.
> When I set zookeeper.set_acl=true in the server.properties file of the
> brokers I cannot create topics , ACL's  etc.
>
> [root@host201 kafka]# bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --add  --allow-principal User:admin
> --operation All --topic '*' --cluster
> Error while executing ACL command: KeeperErrorCode = NoAuth for
> /kafka-acl/Topic
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
> NoAuth for /kafka-acl/Topic
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> ..
> I have specified super users too in the server.properties.
>
> Any idea's what I am doing wrong?
>
> Regards Hans
>

Re: Kafka SASL (Kerberos) - UNKNOWN_TOPIC_OR_PARTITION

[Manikumar]

"allow.everyone.if.no.acl.found" config is applicable for Kafka's inbuilt
authorizer implementation (SimpleAclAuthorizer).
 What is the Kafka version? You can check authorizer debug logs starting
with "No acl found for resource..."
make sure no acls for ANONYMOUS user.  Are you using any custom authorizer?


On Tue, Aug 21, 2018 at 12:21 AM Matt L  wrote:

> Thanks for the pointer Manikumar!
>
> It looks like it was my interbroker communication, this was set to SSL so
> inter broker users were coming in as ANONYMOUS. Once i changed this to
> SASL_SSL I was able to publish/consume.
>
> One remaining question i have is around
> allow.everyone.if.no.acl.found=true. Despite setting this, I still see
> "User:ANONYMOUS is Denied Operation= Describe" in authorizer log. Is there
> something else that needs to be set to enable this? Or is "Describe" not
> part of what this flag sets.
>
> Thanks,
> Matt
>
> On Mon, Aug 20, 2018 at 5:03 AM, Manikumar 
> wrote:
>
> > is auto topic creation enabled on server? Any deny logs in
> > kafka-authorizer.log?
> > What is the inter-broker protocol configured? If it is SSL, SSL user
> should
> > have ClusterAction permission.
> >
> > On Mon, Aug 20, 2018 at 3:33 PM Matt L  wrote:
> >
> > > Hello,
> > >
> > > Having trouble when publishing and consuming from a topic with
> > > SASL_PLAINTEXT.
> > >
> > > Both ZK and Kafka start successfully, in logs I see SASL_PLAINTEXT on
> > 9093
> > > as being available.
> > >
> > > kafka.log:[2018-08-20 03:31:08,202] INFO Registered broker 1 at path
> > > /brokers/ids/1 with addresses:
> > >
> > > EndPoint(kafkabroker1,9092,ListenerName(SSL),SSL),
> > EndPoint(kafkabroker1,9093,ListenerName(SASL_PLAINTEXT),SASL_PLAINTEXT)
> > > (kafka.utils.ZkUtils:70)
> > >
> > >
> > > When i try to publish, e.g.
> > >   bin/kafka-console-producer --broker-list kafkabroker1:9093 \
> > >   --topic testtopic1 --producer.config /tmp/sasl-producer.properties
> > >
> > > I get:
> > >
> > > [2018-08-20 08:37:35,075] WARN Error while fetching metadata with
> > > correlation id 3 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-08-20 08:37:35,176] WARN Error while fetching metadata with
> > > correlation id 4 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-08-20 08:37:35,277] WARN Error while fetching metadata with
> > > correlation id 5 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)
> > >
> > >
> > > What I've verified:
> > > 1) Client can resolve advertisted.listeners on all brokers. (prior to
> > > enabling SASL, PLAINTEXT and SSL work with my set
> advertisted.listerners)
> > > 2) In my sasl-producer.properties, im authenticating with user Kafka.
> > Kafka
> > > has been set as super user and in kafka-authorizer.log, I see "
> > >
> > > [2018-08-20 08:27:19,971] DEBUG principal = User:kafka is a super user,
> > > allowing operation without checking acls. (kafka.authorizer.logger)
> > > [2018-08-20 08:27:19,971] DEBUG Principal = User:kafka is Allowed
> > Operation
> > > = Describe from host = 10.10.52.1 on resource = Topic:testtopic1
> > > (kafka.authorizer.logger)
> > > [2018-08-20 08:27:20,072] DEBUG operation = Read on resource = Topic:
> > > testtopic1 from host = 10.10.52.1 is Allow based on acl = User:kafka
> has
> > > Allow permission for operations: All from hosts: *
> > > (kafka.authorizer.logger)
> > >
> > > and from the kafka.log's in DEBUG:
> > > [2018-08-20 09:35:48,364] DEBUG principal = User:kafka is a super user,
> > > allowing operation without checking acls. (kafka.authorizer.logger:159)
> > > [2018-08-20 09:35:48,364] DEBUG Principal = User:kafka is Allowed
> > Operation
> > > = Describe from host = 10.89.64.7 on resource = Topic:kerbtest1
> > > (kafka.authorizer.logger:251)
> > > [2018-08-20 09:35:48,364] DEBUG Completed
> > >
> > > request:{api_key=3,api_version=4,correlation_id=186,
> > client_id=console-producer}
> > > -- {topics=[kerbtest1],allow_auto_topic_creation=true} from connection
> > > 10.10.52.1:9093-10.10.52.1
> > > :42752;totalTime:0.461000,requestQueueTime:0.033000,
> > localTime:0.346000,remoteTime:0.00,throttleTime:0.
> >
> 033000,responseQueueTime:0.03,sendTime:0.066000,securityProtocol:SASL_
> > PLAINTEXT,principal:User:kafka,listener:SASL_PLAINTEXT
> > > (kafka.request.logger:193)
> > >
> > >
> > > I'm assuming everything is okay from an ACL standpoint but when the
> > client
> > > cannot get the topic metadata from the returned advertisted listeners?
> > > Any ideas on what I could be missing? Could this be something with ZK
> > > setup/any authentication I am missing there?  I had even tried "
> > > skipACL=yes"
> > > but that did not change anything.
> > >
> > > Thanks!
> > >
> >
>

Re: Kafka SASL (Kerberos) - UNKNOWN_TOPIC_OR_PARTITION

[Manikumar]

is auto topic creation enabled on server? Any deny logs in
kafka-authorizer.log?
What is the inter-broker protocol configured? If it is SSL, SSL user should
have ClusterAction permission.

On Mon, Aug 20, 2018 at 3:33 PM Matt L  wrote:

> Hello,
>
> Having trouble when publishing and consuming from a topic with
> SASL_PLAINTEXT.
>
> Both ZK and Kafka start successfully, in logs I see SASL_PLAINTEXT on 9093
> as being available.
>
> kafka.log:[2018-08-20 03:31:08,202] INFO Registered broker 1 at path
> /brokers/ids/1 with addresses:
>
> EndPoint(kafkabroker1,9092,ListenerName(SSL),SSL),EndPoint(kafkabroker1,9093,ListenerName(SASL_PLAINTEXT),SASL_PLAINTEXT)
> (kafka.utils.ZkUtils:70)
>
>
> When i try to publish, e.g.
>   bin/kafka-console-producer --broker-list kafkabroker1:9093 \
>   --topic testtopic1 --producer.config /tmp/sasl-producer.properties
>
> I get:
>
> [2018-08-20 08:37:35,075] WARN Error while fetching metadata with
> correlation id 3 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)
> [2018-08-20 08:37:35,176] WARN Error while fetching metadata with
> correlation id 4 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)
> [2018-08-20 08:37:35,277] WARN Error while fetching metadata with
> correlation id 5 : {testtopic1=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)
>
>
> What I've verified:
> 1) Client can resolve advertisted.listeners on all brokers. (prior to
> enabling SASL, PLAINTEXT and SSL work with my set advertisted.listerners)
> 2) In my sasl-producer.properties, im authenticating with user Kafka. Kafka
> has been set as super user and in kafka-authorizer.log, I see "
>
> [2018-08-20 08:27:19,971] DEBUG principal = User:kafka is a super user,
> allowing operation without checking acls. (kafka.authorizer.logger)
> [2018-08-20 08:27:19,971] DEBUG Principal = User:kafka is Allowed Operation
> = Describe from host = 10.10.52.1 on resource = Topic:testtopic1
> (kafka.authorizer.logger)
> [2018-08-20 08:27:20,072] DEBUG operation = Read on resource = Topic:
> testtopic1 from host = 10.10.52.1 is Allow based on acl = User:kafka has
> Allow permission for operations: All from hosts: *
> (kafka.authorizer.logger)
>
> and from the kafka.log's in DEBUG:
> [2018-08-20 09:35:48,364] DEBUG principal = User:kafka is a super user,
> allowing operation without checking acls. (kafka.authorizer.logger:159)
> [2018-08-20 09:35:48,364] DEBUG Principal = User:kafka is Allowed Operation
> = Describe from host = 10.89.64.7 on resource = Topic:kerbtest1
> (kafka.authorizer.logger:251)
> [2018-08-20 09:35:48,364] DEBUG Completed
>
> request:{api_key=3,api_version=4,correlation_id=186,client_id=console-producer}
> -- {topics=[kerbtest1],allow_auto_topic_creation=true} from connection
> 10.10.52.1:9093-10.10.52.1
> :42752;totalTime:0.461000,requestQueueTime:0.033000,localTime:0.346000,remoteTime:0.00,throttleTime:0.033000,responseQueueTime:0.03,sendTime:0.066000,securityProtocol:SASL_PLAINTEXT,principal:User:kafka,listener:SASL_PLAINTEXT
> (kafka.request.logger:193)
>
>
> I'm assuming everything is okay from an ACL standpoint but when the client
> cannot get the topic metadata from the returned advertisted listeners?
> Any ideas on what I could be missing? Could this be something with ZK
> setup/any authentication I am missing there?  I had even tried "
> skipACL=yes"
> but that did not change anything.
>
> Thanks!
>

Re: unable to reconfigure ssl truststore dynamically on broker

[Manikumar]

Yes, we have open JIRA for this:
https://issues.apache.org/jira/browse/KAFKA-4493

On Wed, Aug 15, 2018 at 10:12 PM John Calcote 
wrote:

> Thanks Manikumar - that's very helpful. I never thought to treat the
> AdminClient like the broker or clients and look for a configuration options
> set on that page.
>
> I should point out to those monitoring that have some control over the code
> (perhaps yourself even) - it seems wrong to have a program that crashes
> when running on the default configuration. Just a thought.
>
> Kind regards,
> John
>
> On Tue, Aug 14, 2018 at 11:18 AM Manikumar 
> wrote:
>
> > Hi,
> >
> > There is no specific doc. "--command-config" option takes admin client
> > configs (.
> > AdminClient configs are listed here:
> > http://kafka.apache.org/documentation/#adminclientconfigs
> >
> >
> > On Tue, Aug 14, 2018 at 10:35 PM John Calcote 
> > wrote:
> >
> > > Manikumar,
> > >
> > > Thank you. You are right - the security.protocol is NOT being passed to
> > > adminclient.properties file. Where exactly is the doc for that file? I
> > > searched for hours and finally had to guess at which options should be
> in
> > > there.
> > >
> > > J
> > >
> > > On Tue, Aug 14, 2018, 8:54 AM Manikumar 
> > wrote:
> > >
> > > > looks like port is wrong or security.protocol config is not passed to
> > > > adminclient.properties file
> > > >
> > > > On Tue, Aug 14, 2018 at 7:23 PM John Calcote  >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm using the latest kafka broker - 2.0.0 with scala 2.12. I have a
> > > > > complete SSL configuration working, but I add clients occasionally
> > and
> > > > want
> > > > > to be able to tell the broker to reload it's ssl truststore (as new
> > > certs
> > > > > have been added to it).
> > > > >
> > > > > I've followed the doc here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
> > > > >
> > > > > and this is what I get:
> > > > >
> > > > > $ kafka_2.12-2.0.0/bin/kafka-configs.sh --bootstrap-server
> > > > > data-cluster:9092 --command-config
> > > > > ./kafka_2.12-2.0.0/adminclient.properties --alter --add-config
> > > > > ssl.truststore.password=password --entity-name 0 --entity-type
> > brokers
> > > > > [2018-08-14 07:48:57,223] ERROR Uncaught exception in thread
> > > > > 'kafka-admin-client-thread | adminclient-1':
> > > > > (org.apache.kafka.common.utils.KafkaThread)
> > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> > > > > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
> > > > > at
> > > > >
> > >
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
> > > > > at
> > > > >
> > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:560)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:496)
> > > > > at
> > > > org.apache.kafka.common.network.Selector.poll(Selector.java:425)
> > > > > at
> > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
> > > > > at java.lang.Thread.

Re: unable to reconfigure ssl truststore dynamically on broker

[Manikumar]

Hi,

There is no specific doc. "--command-config" option takes admin client
configs (.
AdminClient configs are listed here:
http://kafka.apache.org/documentation/#adminclientconfigs


On Tue, Aug 14, 2018 at 10:35 PM John Calcote 
wrote:

> Manikumar,
>
> Thank you. You are right - the security.protocol is NOT being passed to
> adminclient.properties file. Where exactly is the doc for that file? I
> searched for hours and finally had to guess at which options should be in
> there.
>
> J
>
> On Tue, Aug 14, 2018, 8:54 AM Manikumar  wrote:
>
> > looks like port is wrong or security.protocol config is not passed to
> > adminclient.properties file
> >
> > On Tue, Aug 14, 2018 at 7:23 PM John Calcote 
> > wrote:
> >
> > > Hi,
> > >
> > > I'm using the latest kafka broker - 2.0.0 with scala 2.12. I have a
> > > complete SSL configuration working, but I add clients occasionally and
> > want
> > > to be able to tell the broker to reload it's ssl truststore (as new
> certs
> > > have been added to it).
> > >
> > > I've followed the doc here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
> > >
> > > and this is what I get:
> > >
> > > $ kafka_2.12-2.0.0/bin/kafka-configs.sh --bootstrap-server
> > > data-cluster:9092 --command-config
> > > ./kafka_2.12-2.0.0/adminclient.properties --alter --add-config
> > > ssl.truststore.password=password --entity-name 0 --entity-type brokers
> > > [2018-08-14 07:48:57,223] ERROR Uncaught exception in thread
> > > 'kafka-admin-client-thread | adminclient-1':
> > > (org.apache.kafka.common.utils.KafkaThread)
> > > java.lang.OutOfMemoryError: Java heap space
> > > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> > > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> > > at
> > >
> >
> org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
> > > at
> > >
> > >
> >
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
> > > at
> > >
> >
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
> > > at
> > >
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
> > > at
> > > org.apache.kafka.common.network.Selector.attemptRead(Selector.java:560)
> > > at
> > >
> > >
> >
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:496)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:425)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Error while executing config command with args '--bootstrap-server
> > > data-cluster:9092 --command-config
> > > ./kafka_2.12-2.0.0/adminclient.properties --alter --add-config
> > > ssl.truststore.password=password --entity-name 0 --entity-type brokers'
> > > java.util.concurrent.TimeoutException
> > > at
> > >
> > >
> >
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
> > > at
> > >
> > >
> >
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
> > > at
> > kafka.admin.ConfigCommand$.brokerConfig(ConfigCommand.scala:346)
> > > at
> > > kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:304)
> > > at
> > > kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:290)
> > > at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:83)
> > > at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> > > $
> > >
> > > There's nothing on the net about this, so I can only assume the issue
> is
> > > either too new, or I'm just doing something obvious (to someone else)
> the
> > > wrong way. :)
> > >
> > > Thanks in advance,
> > > John
> > >
> >
>

Re: unable to reconfigure ssl truststore dynamically on broker

[Manikumar]

looks like port is wrong or security.protocol config is not passed to
adminclient.properties file

On Tue, Aug 14, 2018 at 7:23 PM John Calcote  wrote:

> Hi,
>
> I'm using the latest kafka broker - 2.0.0 with scala 2.12. I have a
> complete SSL configuration working, but I add clients occasionally and want
> to be able to tell the broker to reload it's ssl truststore (as new certs
> have been added to it).
>
> I've followed the doc here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
>
> and this is what I get:
>
> $ kafka_2.12-2.0.0/bin/kafka-configs.sh --bootstrap-server
> data-cluster:9092 --command-config
> ./kafka_2.12-2.0.0/adminclient.properties --alter --add-config
> ssl.truststore.password=password --entity-name 0 --entity-type brokers
> [2018-08-14 07:48:57,223] ERROR Uncaught exception in thread
> 'kafka-admin-client-thread | adminclient-1':
> (org.apache.kafka.common.utils.KafkaThread)
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at
> org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
> at
>
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
> at
> org.apache.kafka.common.network.Selector.attemptRead(Selector.java:560)
> at
>
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:496)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
> at
>
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
> at java.lang.Thread.run(Thread.java:748)
> Error while executing config command with args '--bootstrap-server
> data-cluster:9092 --command-config
> ./kafka_2.12-2.0.0/adminclient.properties --alter --add-config
> ssl.truststore.password=password --entity-name 0 --entity-type brokers'
> java.util.concurrent.TimeoutException
> at
>
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
> at
>
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
> at kafka.admin.ConfigCommand$.brokerConfig(ConfigCommand.scala:346)
> at
> kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:304)
> at
> kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:290)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:83)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> $
>
> There's nothing on the net about this, so I can only assume the issue is
> either too new, or I'm just doing something obvious (to someone else) the
> wrong way. :)
>
> Thanks in advance,
> John
>

Re: Recovering from Leader : -1 and No ISRs after upgrading to Kafka 1.0

[Manikumar]

Related issue:
https://lists.apache.org/thread.html/55df8a7310669a568c3b1e575e9f034ecc7cb242132c17e6df533134@%3Cusers.kafka.apache.org%3E

On Thu, Aug 2, 2018 at 4:29 PM Darpan Patel  wrote:

> Hi guys,
>
> Recently I upgraded from Kafka 0.10 version to kafka 1.0 on one of the
> clusters.
> After the upgrade, we notice that for most of the topics's partitions ( if
> topic has 8 partitions, at least for 5 partitions) we have Leader : -1 and
> ISR : blank.
>
> For example here is the example of describe output:
>
> Topic :  Partition: 0 Leader -1 Replicas: 101, 102, 103, 104 ISR
> :
> Topic :  Partition: 1 Leader -1 Replicas: 108, 105, 103, 103 ISR
> :
>
> For the Kafka upgrade,
> I followed this steps:
> - Stop kafka
> - Added inter.broker.protocol.version = 0.10.1 and
> log.message.format.version=0.10.1
>
> Restarted all Kafka brokers
> Performed HDP upgrade.
> After express upgrade is complete, change the value for
> inter.broker.protocol.version to 1.0 and restarted Kafka brokers ( rolling)
> After rolling restart, changed value for log.message.format.version to 1.0
> and save and restarted all kafka brokers ( rolling)
>
> - After this upgrade found that most of the topic partitions have no
> leaders and no ISRs.
>
> If someone has faced this or knows the solution, can you please share your
> ideas/steps to recover?
>
> Regards,
> Darpan
>

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

[Manikumar]

ka is a distributed streaming platform with four core APIs:
>
>
>
> ** The Producer API allows an application to publish a stream records to
>
> one or more Kafka topics.
>
>
>
> ** The Consumer API allows an application to subscribe to one or more
>
> topics and process the stream of records produced to them.
>
>
>
> ** The Streams API allows an application to act as a stream processor,
>
> consuming an input stream from one or more topics and producing an
>
> output stream to one or more output topics, effectively transforming the
>
> input streams to output streams.
>
>
>
> ** The Connector API allows building and running reusable producers or
>
> consumers that connect Kafka topics to existing applications or data
>
> systems. For example, a connector to a relational database might
>
> capture every change to a table.
>
>
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
>
> ** Building real-time streaming data pipelines that reliably get data
>
> between systems or applications.
>
>
>
> ** Building real-time streaming applications that transform or react
>
> to the streams of data.
>
>
>
>
>
>
>
> Apache Kafka is in use at large and small companies worldwide, including
>
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
>
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
>
>
>
>
> A big thank you for the following 131 contributors to this release!
>
>
>
> Adem Efe Gencer, Alex D, Alex Dunayevsky, Allen Wang, Andras Beni,
>
> Andy Bryant, Andy Coates, Anna Povzner, Arjun Satish, asutosh936,
>
> Attila Sasvari, bartdevylder, Benedict Jin, Bill Bejeck, Blake Miller,
>
> Boyang Chen, cburroughs, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe,
>
> Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, dan norwood,
>
> Daniel Shuy, Daniel Wojda, Dark, David Glasser, Debasish Ghosh, Detharon,
>
> Dhruvil Shah, Dmitry Minkovsky, Dong Lin, Edoardo Comar, emmanuel Harel,
>
> Eugene Sevastyanov, Ewen Cheslack-Postava, Fedor Bobin, fedosov-alexander,
>
> Filipe Agapito, Florian Hussonnois, fredfp, Gilles Degols, gitlw, Gitomain,
>
> Guangxian, Gunju Ko, Gunnar Morling, Guozhang Wang, hmcl, huxi, huxihx,
>
> Igor Kostiakov, Ismael Juma, Jacek Laskowski, Jagadesh Adireddi,
>
> Jarek Rudzinski, Jason Gustafson, Jeff Klukas, Jeremy Custenborder,
>
> Jiangjie (Becket) Qin, Jiangjie Qin, JieFang.He, Jimin Hsieh, Joan Goyeau,
>
> Joel Hamill, John Roesler, Jon Lee, Jorge Quilcate Otoya, Jun Rao,
>
> Kamal C, khairy, Koen De Groote, Konstantine Karantasis, Lee Dongjin,
>
> Liju John, Liquan Pei, lisa2lisa, Lucas Wang, Magesh Nandakumar,
>
> Magnus Edenhill, Magnus Reftel, Manikumar Reddy, Manikumar Reddy O,
>
> manjuapu, Mats Julian Olsen, Matthias J. Sax, Max Zheng, maytals,
>
> Michael Arndt, Michael G. Noll, Mickael Maison, nafshartous, Nick Travers,
>
> nixsticks, Paolo Patierno, parafiend, Patrik Erdes, Radai Rosenblatt,
>
> Rajini Sivaram, Randall Hauch, ro7m, Robert Yokota, Roman Khlebnov,
>
> Ron Dagostino, Sandor Murakozi, Sasaki Toru, Sean Glover,
>
> Sebastian Bauersfeld, Siva Santhalingam, Stanislav Kozlovski, Stephane
> Maarek,
>
> Stuart Perks, Surabhi Dixit, Sönke Liebau, taekyung, tedyu, Thomas Leplus,
>
> UVN, Vahid Hashemian, Valentino Proietti, Viktor Somogyi, Vitaly Pushkar,
>
> Wladimir Schmidt, wushujames, Xavier Léauté, xin, yaphet,
>
> Yaswanth Kumar, ying-zheng, Yu
>
>
>
>
>
>
>
> We welcome your help and feedback. For more information on how to
>
> report problems, and to get involved, visit the project website at
>
> https://kafka.apache.org/
>
>
>
>
>
> Thank you!
>
>
>
>
>
> Regards,
>
>
>
> Rajini
>

Re: cleanup.policy - doesn't accept compact,delete

[Manikumar]

> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
> > at
> > org.apache.kafka.streams.processor.internals.InternalTopicManager.make
> > Ready(InternalTopicManager.java:121)
> >
> >
> >
> > Thanks & Regards,
> >
> > Ashok
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: Thursday, June 07, 2018 12:11 AM
> > To: Users
> > Subject: Re: cleanup.policy - doesn't accept compact,delete
> >
> > As described in usage description, to group the values which contain
> commas,  we need to use square brackets.
> >
> > ex: --add-config cleanup.policy=[compact,delete]
> >
> > On Thu, Jun 7, 2018 at 8:49 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON)
> < ashokkumar.jayara...@cox.com> wrote:
> >
> >> Hi,
> >>
> >> We are on Kafka version 1.0.0.  Per the below new feature, a topic
> >> can allow both compact and delete.  I tried all the combinations, but
> they all
> >> fail to accept values that are not either compact OR delete.   Is this
> >> feature valid in updated releases, since 0.10.2?If this is not a
> >> feature available, how to cleanup the growing compacted topic scenario?
> >>
> >> https://issues.apache.org/jira/browse/KAFKA-4015
> >>
> >> $ ./kafka-configs.sh --zookeeper <>:2181--alter --entity-type
> >> topics --entity-name stream_output --add-config
> >> cleanup.policy=compact,delete Error while executing config command
> >> requirement failed: Invalid entity
> >> config: all configs to be added must be in the format "key=val".
> >> java.lang.IllegalArgumentException: requirement failed: Invalid
> >> entity
> >> config: all configs to be added must be in the format "key=val".
> >> at scala.Predef$.require(Predef.scala:233)
> >> at kafka.admin.ConfigCommand$.parseConfigsToBeAdded(
> >> ConfigCommand.scala:128)
> >> at
> kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:78)
> >> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> >> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> >>
> >>
> >> $ ./kafka-configs.sh --zookeeper <>:2181 --alter --entity-type
> >> topics --entity-name ash_stream_output --add-config
> >> cleanup.policy=compact_delete Error while executing config command
> >> Invalid value compact_delete for configuration cleanup.policy: String
> >> must be one of: compact, delete
> >> org.apache.kafka.common.config.ConfigException: Invalid value
> >> compact_delete for configuration cleanup.policy: String must be one of:
> >> compact, delete
> >> at org.apache.kafka.common.config.ConfigDef$ValidString.
> >> ensureValid(ConfigDef.java:851)
> >> at org.apache.kafka.common.config.ConfigDef$ValidList.
> >> ensureValid(ConfigDef.java:827)
> >> at org.apache.kafka.common.config.ConfigDef.parse(
> >> ConfigDef.java:427)
> >> at kafka.log.LogConfig$.validate(LogConfig.scala:331)
> >> at
> kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:524)
> >> at
> kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:90)
> >> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> >> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> >>
> >>
> >> $ ./kafka-configs.sh --zookeeper <>:2181 --alter --entity-type
> >> topics --entity-name ash_stream_output --add-config
> >> cleanup.policy=compact_and_delete Error while executing config
> >> command Invalid value compact_delete for configuration
> >> cleanup.policy: String must be one of: compact, delete
> >> org.apache.kafka.common.config.ConfigException: Invalid value
> >> compact_delete for configuration cleanup.policy: String must be one of:
> >> compact, delete
> >> at org.apache.kafka.common.config.ConfigDef$ValidString.
> >> ensureValid(ConfigDef.java:851)
> >> at org.apache.kafka.common.config.ConfigDef$ValidList.
> >> ensureValid(ConfigDef.java:827)
> >> at org.apache.kafka.common.config.ConfigDef.parse(
> >> ConfigDef.java:427)
> >> at kafka.log.LogConfig$.validate(LogConfig.scala:331)
> >> at
> kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:524)
> >> at
> kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:90)
> >> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> >> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> >>
> >>
> >> Thanks & Regards,
> >> Ashok
> >>
> >>
>
>

Re: [kafka-clients] Re: [VOTE] 1.0.2 RC1

[Manikumar]

+1 (non-binding) Verified tests, quick start, producer/consumer perf tests.



On Sun, Jul 1, 2018 at 11:27 PM Ismael Juma  wrote:

> +1 (binding)
>
> Verified signature of source artifact, ran tests and verified quickstart
> on source artifact with Java 7, verified quickstart on binary artifact (Scala
> 2.12) with Java 8, sanity checked release notes and Maven staging
> repository.
>
> Ismael
>
> On Fri, Jun 29, 2018 at 10:02 PM Matthias J. Sax 
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for release of Apache Kafka 1.0.2.
>>
>> This is a bug fix release addressing 27 tickets:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
>>
>> Release notes for the 1.0.2 release:
>> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by end of next week (7/6/18).
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/javadoc/
>>
>> * Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
>> https://github.com/apache/kafka/releases/tag/1.0.2-rc1
>>
>> * Documentation:
>> http://kafka.apache.org/10/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/10/protocol.html
>>
>> * Successful Jenkins builds for the 1.0 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/214/
>> System tests:
>> https://jenkins.confluent.io/job/system-test-kafka/job/1.0/225/
>>
>> /**
>>
>> Thanks,
>>   -Matthias
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZZUnorMq4mgKnY5v6t0sAi4TNtMcQPjyYd18R_FC%3DGzzg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: [kafka-clients] Re: [VOTE] 0.11.0.3 RC0

[Manikumar]

+1 (non-binding)   Verified quick start, ran producer/consumer perf tests,

Thanks


On Sun, Jul 1, 2018 at 8:29 PM Ismael Juma  wrote:

> +1 (binding)
>
> Verified signature of source artifact, ran tests and verified quickstart
> on source artifact with Java 7, verified quickstart on binary artifact (Scala
> 2.11) with Java 7, sanity checked release notes and Maven staging
> repository.
>
> Thanks for running the release!
>
> Ismael
>
> On Fri, Jun 22, 2018 at 3:14 PM Matthias J. Sax 
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for release of Apache Kafka 0.11.0.3.
>>
>> This is a bug fix release closing 27 tickets:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3
>>
>> Release notes for the 0.11.0.3 release:
>> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
>> can close the vote on Wednesday.
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/javadoc/
>>
>> * Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.3 tag:
>> https://github.com/apache/kafka/releases/tag/0.11.0.3-rc0
>>
>> * Documentation:
>> http://kafka.apache.org/0110/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0110/protocol.html
>>
>> * Successful Jenkins builds for the 0.11.0 branch:
>> Unit/integration tests:
>> https://builds.apache.org/job/kafka-0.11.0-jdk7/385/
>> System tests:
>> https://jenkins.confluent.io/job/system-test-kafka/job/0.11.0/217/
>>
>> /**
>>
>> Thanks,
>>   -Matthias
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZb38rnx99Ab%2BTbK8O3j%3D3xZLuqoPH2%2BdWxOGetzSpCmpQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: Partitions reassignment is failing in Kafka 1.1.0

[Manikumar]

It will be taken as "any" directory for each replica, which means replica
will placed on any one of the
configured directory on that broker.

Since it is "log_dirs" optional, you can remove from the json.

On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna 
wrote:

> It is problem on my side. The code was changing the replicas count but not
> the log_dirs. Since I am migrating from 0.10 this part of the code was not
> changed.
>
> I have a follow up question what is the default value of log_dirs if I
> don't specify it in reassignment.json ?
>
> On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna 
> wrote:
>
> > I am generating the reassignent.json like below
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --generate --topics-to-move-json-file
> /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 >
> /home/ubuntu/deploy/kafka/reassignment.json"
> >
> > Then I am doing the reassignment using the generated file
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --execute --reassignment-json-file
> /home/ubuntu/deploy/kafka/reassignment.json
> >
> > kafka-reassign-partitions.sh helps states
> >
> > The JSON file with the partition reassignment configurationThe format to
> >> use is -
> >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs"
> is
> >> optional. When it is specified, its length must equal the length of the
> >> replicas list. The value in this list can be either "any" or the
> absolution
> >> path of the log directory on the broker. If absolute log directory path
> is
> >> specified, it is currently required that the replica has not already
> been
> >> created on that broker. The replica will then be created in the
> specified
> >> log directory on the broker later.
> >
> >
> > So it appears reassignment json that is generated by
> > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > some issue in kafka-reassign-partitions.sh or some misconfiguration from
> my
> > side. ?
> >
> > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna  >
> > wrote:
> >
> >> Please find the server.properties from one of the broker.
> >>
> >> broker.id=0
> >> port=9092
> >> num.network.threads=3
> >> num.io.threads=8
> >> socket.send.buffer.bytes=102400
> >> socket.receive.buffer.bytes=102400
> >> socket.request.max.bytes=104857600
> >> log.dirs=/var/lib/kafka/kafka-logs
> >> num.recovery.threads.per.data.dir=1
> >> log.retention.hours=36
> >> log.retention.bytes=1073741824
> >> log.segment.bytes=536870912
> >> log.retention.check.interval.ms=30
> >> log.cleaner.enable=false
> >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> >> message.max.bytes=1500
> >> replica.fetch.max.bytes=1500
> >> auto.create.topics.enable=true
> >> zookeeper.connection.timeout.ms=6000
> >> unclean.leader.election.enable=false
> >> delete.topic.enable=false
> >> offsets.topic.replication.factor=1
> >> transaction.state.log.replication.factor=1
> >> transaction.state.log.min.isr=1
> >>
> >> I have placed server.log from a broker at https://gist.github.com/deb
> >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> >>
> >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu  wrote:
> >>
> >>> Seems to be related to KIP-113.
> >>>
> >>> server.properties didn't go thru. Do you mind pastebin'ing its content
> ?
> >>>
> >>> If you can pastebin logs from broker, that should help.
> >>>
> >>> Thanks
> >>>
> >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> subharaj.ma...@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi
> >>> >
> >>> > I altered a topic like below in kafka 1.1.0
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> >>> 127.0.0.1:2181
> >>> > --alter --topic Topic3 --config min.insync.replicas=2
> >>> >
> >>> > But whenever I am trying to verify the reassignment it is showing the
> >>> > below exception
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> >>> >
> >>> > Partitions reassignment failed due to Size of replicas list Vector(3,
> >>> 0, 2) is different from size of log dirs list Vector(any) for partition
> >>> Topic3-7
> >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
> >>> partition Topic3-7
> >>> >   at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:262)
> >>> >   at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:251)
> >>> >   at scala.collection.Iterator$class.foreach(Iterator.scala:891)

Re: [kafka-clients] [VOTE] 2.0.0 RC1

[Manikumar]

looks like maven artifacts are not updated in the staging repo. They are
still at old timestamp.
https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.11/2.0.0/

On Sat, Jun 30, 2018 at 12:06 AM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the second candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new  KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
> A few notable highlights:
>
>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
>(KIP-277)
>- SASL/OAUTHBEARER implementation (KIP-255)
>- Improved quota communication and customization of quotas (KIP-219,
>KIP-257)
>- Efficient memory usage for down conversion (KIP-283)
>- Fix log divergence between leader and follower during fast leader
>failover (KIP-279)
>- Drop support for Java 7 and remove deprecated code including old
>scala clients
>- Connect REST extension plugin, support for externalizing secrets and
>improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>- Scala API for Kafka Streams and other Streams API improvements
>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
> Release notes for the 2.0.0 release:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/RELEASE_NOTES.html
>
>
>
> *** Please download, test and vote by Tuesday, July 3rd, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/javadoc/
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> https://github.com/apache/kafka/tree/2.0.0-rc1
>
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/66/
>
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.0/15/
>
>
>
> Please test and verify the release artifacts and submit a vote for this RC
> or report any issues so that we can fix them and roll out a new RC ASAP!
>
> Although this release vote requires PMC votes to pass, testing, votes,
> and bug
> reports are valuable and appreciated from everyone.
>
>
> Thanks,
>
>
> Rajini
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAOJcB39GdTWOaK4qysvyPyGU8Ldm82t_TA364x1MP8a8OAod6A%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: Unclean Leader Election Expected Behavior

[Manikumar]

In your case, you need to restart B2 with unclean.leader.election=true.
This will enable B2 to become leader with 90 messages.

On Thu, Jun 28, 2018 at 11:51 PM Jordan Pilat  wrote:

> If I restart the broker, won't that cause all 100 messages to be lost?
>
> On 2018/06/28 02:59:15, Manikumar  wrote:
> > You can enable unclean.leader.election temporarily for specific topic by
> > using kafka-topics.sh command.
> > This requires broker restart to take effect.
> >
> >  http://kafka.apache.org/documentation/#topicconfigs
> >
> > On Thu, Jun 28, 2018 at 2:27 AM Jordan Pilat  wrote:
> >
> > > Heya,
> > >
> > > I had a question about what behavior to expect from a particular
> > > scenario.  Given:
> > > A. Unclean leader elections are disabled
> > > B. A partition is led by Broker1 and followed by Broker2
> > > C. Broker1 is on offset 100
> > > D. Broker2 is on offset 90
> > > E. Broker2 has fallen out of the ISR, leaving only Broker1 in the ISR
> > > F. Broker1 has a hard drive failure and goes down.  All messages for
> the
> > > partition in question are permanently lost.
> > >
> > > _As I understand it_, the only way for the partition to come back
> online
> > > is to bring Broker1 back online, and suffer the loss of 100 messages
> (as
> > > Broker2's log will be truncated to Broker1's offset, which will start
> from
> > > scratch due to the hard drive loss)
> > >
> > > Is there a procedure in such a case, to force-elect Broker2 the leader,
> > > and thus only lose 10 messages?
> > >
> > > Thanks!
> > > - Jordan Pilat
> > >
> >
>

Re: [VOTE] 1.1.1 RC1

[Manikumar]

Yes, looks like maven artifacts are missing on staging repo
https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.11/

On Thu, Jun 28, 2018 at 4:18 PM Odin  wrote:

> There are no 1.1.1-rc1 artifacts in the staging repo listed. Where can
> they be found?
>
> Sincerely
> Odin Standal
>
>
> ​​
>
> ‐‐‐ Original Message ‐‐‐
>
> On June 22, 2018 7:09 PM, Dong Lin  wrote:
>
> > ​​
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 1.1.1.
> >
> > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> >
> > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> since
> >
> > that release. A few of the more significant fixes include:
> >
> > KAFKA-6925 https://issues.apache.org/jira/browse/KAFKA-6925 - Fix memory
> >
> > leak in StreamsMetricsThreadImpl
> >
> > KAFKA-6937 https://issues.apache.org/jira/browse/KAFKA-6937 - In-sync
> >
> > replica delayed during fetch if replica throttle is exceeded
> >
> > KAFKA-6917 https://issues.apache.org/jira/browse/KAFKA-6917 - Process
> txn
> >
> > completion asynchronously to avoid deadlock
> >
> > KAFKA-6893 https://issues.apache.org/jira/browse/KAFKA-6893 - Create
> >
> > processors before starting acceptor to avoid ArithmeticException
> >
> > KAFKA-6870 https://issues.apache.org/jira/browse/KAFKA-6870 -
> >
> > Fix ConcurrentModificationException in SampledStat
> >
> > KAFKA-6878 https://issues.apache.org/jira/browse/KAFKA-6878 - Fix
> >
> > NullPointerException when querying global state store
> >
> > KAFKA-6879 https://issues.apache.org/jira/browse/KAFKA-6879 - Invoke
> >
> > session init callbacks outside lock to avoid Controller deadlock
> >
> > KAFKA-6857 https://issues.apache.org/jira/browse/KAFKA-6857 - Prevent
> >
> > follower from truncating to the wrong offset if undefined leader epoch is
> >
> > requested
> >
> > KAFKA-6854 https://issues.apache.org/jira/browse/KAFKA-6854 - Log
> cleaner
> >
> > fails with transaction markers that are deleted during clean
> >
> > KAFKA-6747 https://issues.apache.org/jira/browse/KAFKA-6747 - Check
> >
> > whether there is in-flight transaction before aborting transaction
> >
> > KAFKA-6748 https://issues.apache.org/jira/browse/KAFKA-6748 - Double
> >
> > check before scheduling a new task after the punctuate call
> >
> > KAFKA-6739 https://issues.apache.org/jira/browse/KAFKA-6739 -
> >
> > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> >
> > KAFKA-6728 https://issues.apache.org/jira/browse/KAFKA-6728 -
> >
> > Fix NullPointerException when instantiating the HeaderConverter
> >
> > Kafka 1.1.1 release plan:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> >
> > Release notes for the 1.1.1 release:
> >
> > http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> >
> > http://kafka.apache.org/KEYS
> >
> > -   Release artifacts to be voted upon (source and binary):
> >
> > http://home.apache.org/~lindong/kafka-1.1.1-rc1/
> >
> > -   Maven artifacts to be voted upon:
> >
> > https://repository.apache.org/content/groups/staging/
> >
> > -   Javadoc:
> >
> > http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
> >
> > -   Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
> >
> > https://github.com/apache/kafka/tree/1.1.1-rc1
> >
> > -   Documentation:
> >
> > http://kafka.apache.org/11/documentation.html
> >
> > -   Protocol:
> >
> > http://kafka.apache.org/11/protocol.html
> >
> > -   Successful Jenkins builds for the 1.1 branch:
> >
> > Unit/integration tests:
> https://builds.apache.org/job/kafka-1.1-jdk7/152/
> >
> > https://builds.apache.org/job/kafka-1.1-jdk7/152/
> >
> >
> > System tests: https://jenkins.confluent.io/job/system-test-
> >
> > kafka-branch-builder/1817
> >
> > Please test and verify the release artifacts and submit a vote for this
> RC,
> >
> > or report any issues so we can fix them and get a new RC out ASAP.
> Although
> >
> > this release vote requires PMC votes to pass, testing, votes, and bug
> >
> > reports are valuable and appreciated from everyone.
> >
> > Cheers,
> >
> > Dong
>
>
>

Re: Unclean Leader Election Expected Behavior

[Manikumar]

You can enable unclean.leader.election temporarily for specific topic by
using kafka-topics.sh command.
This requires broker restart to take effect.

 http://kafka.apache.org/documentation/#topicconfigs

On Thu, Jun 28, 2018 at 2:27 AM Jordan Pilat  wrote:

> Heya,
>
> I had a question about what behavior to expect from a particular
> scenario.  Given:
> A. Unclean leader elections are disabled
> B. A partition is led by Broker1 and followed by Broker2
> C. Broker1 is on offset 100
> D. Broker2 is on offset 90
> E. Broker2 has fallen out of the ISR, leaving only Broker1 in the ISR
> F. Broker1 has a hard drive failure and goes down.  All messages for the
> partition in question are permanently lost.
>
> _As I understand it_, the only way for the partition to come back online
> is to bring Broker1 back online, and suffer the loss of 100 messages (as
> Broker2's log will be truncated to Broker1's offset, which will start from
> scratch due to the hard drive loss)
>
> Is there a procedure in such a case, to force-elect Broker2 the leader,
> and thus only lose 10 messages?
>
> Thanks!
> - Jordan Pilat
>

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

[Manikumar]

+1 (non-binding)  Ran tests,  Verified quick start,  producer/consumer perf
tests


On Sat, Jun 23, 2018 at 8:11 AM Dong Lin  wrote:

> Thank you for testing and voting the release!
>
> I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and
> vote by Tuesday, June 26, 12 pm PT.
>
> Thanks,
> Dong
>
> On Fri, Jun 22, 2018 at 10:09 AM, Dong Lin  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for release of Apache Kafka 1.1.1.
>>
>> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
>> first released with 1.1.0 about 3 months ago. We have fixed about 25 issues
>> since that release. A few of the more significant fixes include:
>>
>> KAFKA-6925  - Fix
>> memory leak in StreamsMetricsThreadImpl
>> KAFKA-6937  - In-sync
>> replica delayed during fetch if replica throttle is exceeded
>> KAFKA-6917  - Process
>> txn completion asynchronously to avoid deadlock
>> KAFKA-6893  - Create
>> processors before starting acceptor to avoid ArithmeticException
>> KAFKA-6870  -
>> Fix ConcurrentModificationException in SampledStat
>> KAFKA-6878  - Fix
>> NullPointerException when querying global state store
>> KAFKA-6879  - Invoke
>> session init callbacks outside lock to avoid Controller deadlock
>> KAFKA-6857  - Prevent
>> follower from truncating to the wrong offset if undefined leader epoch is
>> requested
>> KAFKA-6854  - Log
>> cleaner fails with transaction markers that are deleted during clean
>> KAFKA-6747  - Check
>> whether there is in-flight transaction before aborting transaction
>> KAFKA-6748  - Double
>> check before scheduling a new task after the punctuate call
>> KAFKA-6739  -
>> Fix IllegalArgumentException when down-converting from V2 to V0/V1
>> KAFKA-6728  -
>> Fix NullPointerException when instantiating the HeaderConverter
>>
>> Kafka 1.1.1 release plan:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>>
>> Release notes for the 1.1.1 release:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
>>
>> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
>> https://github.com/apache/kafka/tree/1.1.1-rc1
>>
>> * Documentation:
>> http://kafka.apache.org/11/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/11/protocol.html
>>
>> * Successful Jenkins builds for the 1.1 branch:
>> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/152/
>> *
>> System tests:
>> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1817
>>
>>
>> Please test and verify the release artifacts and submit a vote for this
>> RC,
>> or report any issues so we can fix them and get a new RC out ASAP.
>> Although
>> this release vote requires PMC votes to pass, testing, votes, and bug
>> reports are valuable and appreciated from everyone.
>>
>> Cheers,
>> Dong
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAAaarBZCqdUPK8asaZS0ws0yr_vjFw0o8RxFcdRv07%3Df_7g%3DkQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: ISR shrink rate & IsrExpandsPerSec issue

[Manikumar]

These metrics are meter type metrics, which tracks  count, mean rate,  1-,
5-, and 15-minute moving averages.
You maybe observing count measure, which gives number of events which have
been marked.
You can try monitoring 1/5/15 miniture averages.

On Wed, Jun 20, 2018 at 12:33 AM Arunkumar 
wrote:

> Hi All,
> I am seeing  IsrShrinksPerSec & IsrExpandsPerSec increases (>0) when one
> or more brokers goes down and comes back up into the cluster. But the
> values should get back to 0 once the servers catches up, is my
> understanding after reading most of the documents. But in our production
> environment (3 node cluster) these values does not reduce and keeps
> increasing. This keeps throwing alerts every few seconds. Any insight is
> highly appreciated.
>
> Thanks in advanceArunkumar Pichaimuthu, PMP

Re: How set log compaction policies at cluster level

[Manikumar]

Those configs are topic-level config names. To configure  in
server.properties,
we need to use server config property names (log.cleanup.policy,
log.cleaner.delete.retention.ms etc..).

check the "SERVER DEFAULT PROPERTY"  column in the table given in the
below link
http://kafka.apache.org/documentation/#topicconfigs



On Thu, Jun 14, 2018 at 3:51 PM David Espinosa  wrote:

> Hi all,
>
> I would like to apply log compaction configuration for any topic in my
> kafka cluster, as default properties. These configuration properties are:
>
>- cleanup.policy
>- delete.retention.ms
>- segment.ms
>- min.cleanable.dirty.ratio
>
> I have tried to place them in the server.properties file, but they are not
> applied. I could only apply them when using kafka-topics create topic
> command.
>
> Somebody knows how to apply those properties as default for any topic
> created?
>
> Thanks in advance,
> David.
>

Re: Problem consuming from broker 1.1.0

[Manikumar]

Can you post consumer debug logs?
You can enable console consumer debug logs here:
kafka/config/tools-log4j.properties

On Wed, Jun 13, 2018 at 9:55 AM Craig Ching  wrote:

> Hi!
>
> We’re having a problem with a new kafka cluster at 1.1.0.  The problem is,
> in general, that consumers can’t consume from the different broker (old
> broker was 0.11 I think).  The easiest recipe I have for reproducing the
> problem is that downloading kafka 1.0.1 and running console consumer can’t
> consume from the 1.1.0 cluster while a 1.1.0 console consumer can.  We’re
> invoking console consumer like this:
>
> bin/kafka-console-consumer.sh \
> —bootstrap-server [kafka server] \
> —topic [our topic] \
> —max-messages 3
>
> That works for a 1.1.0 console consumer, but not for 1.0.1.  However, if
> we change that to:
>
> bin/kafka-console-consumer.sh \
> —zookeeper [zookeeper server] \
> —topic [our topic] \
> —max-messages 3
>
> Then it works for 1.0.1.
>
> I was wondering, is the zookeeper schema published for 1.1.0?  I have a
> feeling that maybe something is wrong in zookeeper and I know earlier
> versions of kafka used to publish the zk schema, could there be a problem
> in zk?  if so, what might I look for?
>
> Any help is greatly appreciated!
>
> Cheers,
> Craig

Re: cleanup.policy - doesn't accept compact,delete

[Manikumar]

As described in usage description, to group the values which contain
commas,  we need to use square brackets.

ex: --add-config cleanup.policy=[compact,delete]

On Thu, Jun 7, 2018 at 8:49 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON) <
ashokkumar.jayara...@cox.com> wrote:

> Hi,
>
> We are on Kafka version 1.0.0.  Per the below new feature, a topic can
> allow both compact and delete.  I tried all the combinations, but they all
> fail to accept values that are not either compact OR delete.   Is this
> feature valid in updated releases, since 0.10.2?If this is not a
> feature available, how to cleanup the growing compacted topic scenario?
>
> https://issues.apache.org/jira/browse/KAFKA-4015
>
> $ ./kafka-configs.sh --zookeeper <>:2181--alter --entity-type
> topics --entity-name stream_output --add-config
> cleanup.policy=compact,delete
> Error while executing config command requirement failed: Invalid entity
> config: all configs to be added must be in the format "key=val".
> java.lang.IllegalArgumentException: requirement failed: Invalid entity
> config: all configs to be added must be in the format "key=val".
> at scala.Predef$.require(Predef.scala:233)
> at kafka.admin.ConfigCommand$.parseConfigsToBeAdded(
> ConfigCommand.scala:128)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:78)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
>
>
> $ ./kafka-configs.sh --zookeeper <>:2181 --alter --entity-type
> topics --entity-name ash_stream_output --add-config
> cleanup.policy=compact_delete
> Error while executing config command Invalid value compact_delete for
> configuration cleanup.policy: String must be one of: compact, delete
> org.apache.kafka.common.config.ConfigException: Invalid value
> compact_delete for configuration cleanup.policy: String must be one of:
> compact, delete
> at org.apache.kafka.common.config.ConfigDef$ValidString.
> ensureValid(ConfigDef.java:851)
> at org.apache.kafka.common.config.ConfigDef$ValidList.
> ensureValid(ConfigDef.java:827)
> at org.apache.kafka.common.config.ConfigDef.parse(
> ConfigDef.java:427)
> at kafka.log.LogConfig$.validate(LogConfig.scala:331)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:524)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:90)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
>
>
> $ ./kafka-configs.sh --zookeeper <>:2181 --alter --entity-type
> topics --entity-name ash_stream_output --add-config
> cleanup.policy=compact_and_delete
> Error while executing config command Invalid value compact_delete for
> configuration cleanup.policy: String must be one of: compact, delete
> org.apache.kafka.common.config.ConfigException: Invalid value
> compact_delete for configuration cleanup.policy: String must be one of:
> compact, delete
> at org.apache.kafka.common.config.ConfigDef$ValidString.
> ensureValid(ConfigDef.java:851)
> at org.apache.kafka.common.config.ConfigDef$ValidList.
> ensureValid(ConfigDef.java:827)
> at org.apache.kafka.common.config.ConfigDef.parse(
> ConfigDef.java:427)
> at kafka.log.LogConfig$.validate(LogConfig.scala:331)
> at kafka.admin.AdminUtils$.changeTopicConfig(AdminUtils.scala:524)
> at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:90)
> at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:65)
> at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
>
>
> Thanks & Regards,
> Ashok
>
>

Re: Custom AuthenticateCallbackHandler

[Manikumar]

This feature will be part upcoming Kafka 2.0.0 release.

Doc PR is here : https://github.com/apache/kafka/pull/4890

configs here:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L57

On Fri, Jun 1, 2018 at 10:51 PM, Roy van der Valk <
roy.van.der.v...@gmail.com> wrote:

> Dear Kafka community,
>
> Can somebody help me setting up a custom AuthenticateCallbackHandler as
> described in KIP-86 recently added by Rajini Sivaram or point me to good
> documentation?
>
> I described my question in more detail on Stackoverflow:
> https://stackoverflow.com/questions/50412589/kafka-custom-
> authenticatecallbackhandler
>
> All help is greatly appreciated!
>
> Roy
>

Re: Logging authentication/authorization failures

[Manikumar]

Currently authentication logs are not available. In recent Kafka versions,
authorization failures
will be logged in logs/kafka-authorizer.log

On Thu, May 31, 2018 at 5:34 PM, Gérald Quintana 
wrote:

> Hello,
>
> I am using SASL Plaintext authentication and ACLs.
> I'd like to be able to detect potential security attacks on Kafka broker
>
> Is it possible to log, on broker side, authentication failures (wrong
> password) and authorization failures (not granted)?
>
> I read this blog post
> https://www.confluent.io/blog/apache-kafka-security-
> authorization-authentication-encryption/
> and tried setting to DEBUG log4j.logger.kafka.authorizer.logger=DEBUG,
> authorizerAppender
> But it doesn't help.
>
> Thanks for your help,
> Gérald
>

Re: Long start time for consumer

[Manikumar]

Pls check "group.initial.rebalance.delay.ms" broker config property.  This
will be the delay for the initial consumer rebalance.

from docs

"The rebalance will be further delayed by the value of
group.initial.rebalance.delay.ms as new members join the group,
 up to a maximum of max.poll.interval.ms"


http://kafka.apache.org/documentation/#upgrade_1100_notable

On Tue, May 29, 2018 at 6:51 PM, Shantanu Deshmukh 
wrote:

> No, no dynamic topic creation.
>
> On Tue, May 29, 2018 at 6:38 PM Jaikiran Pai 
> wrote:
>
> > Are your topics dynamically created? If so, see this
> > threadhttps://www.mail-archive.com/dev@kafka.apache.org/msg67224.html
> >
> > -Jaikiran
> >
> >
> > On 29/05/18 5:21 PM, Shantanu Deshmukh wrote:
> > > Hello,
> > >
> > > We have 3 broker Kafka 0.10.0.1 cluster. We have 5 topics, each with 10
> > > partitions. I have an application which consumes from all these topics
> by
> > > creating multiple consumer processes. All of these consumers are under
> a
> > > same consumer group. I am noticing that every time we restart this
> > > application. It takes almost 5 minutes for consumers to start
> consuming.
> > > What might be going wrong?
> > >
> >
> >
>

Re: Frequent consumer rebalance, auto commit failures

[Manikumar]

heartbeat.interval.ms should be lower than session.timeout.ms.

Check here:
http://kafka.apache.org/0101/documentation.html#newconsumerconfigs


On Thu, May 24, 2018 at 2:39 PM, Shantanu Deshmukh 
wrote:

> Someone please help me. I am suffering due to this issue since a long time
> and not finding any solution.
>
> On Wed, May 23, 2018 at 3:48 PM Shantanu Deshmukh 
> wrote:
>
> > We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10
> > partitions each. We have an application which spawns threads as
> consumers.
> > We spawn 5 consumers for each topic. I am observing that consider group
> > randomly keeps rebalancing. Then many times we see logs saying "Revoking
> > partitions for". This happens almost every 10 minutes. Consumption during
> > this time completely stops.
> >
> > I have applied this configuration
> > max.poll.records 20
> > heartbeat.interval.ms 1
> > Session.timeout.ms 6000
> >
> > Still this did not help. Strange thing is I observed consumer writing
> logs
> > saying "auto commit failed because poll() loop spent too much time
> > processing records" even when there was no data in partition to process.
> We
> > have polling interval of 500 ms, specified as argument in poll().
> Initially
> > I had set same consumer group for all three topics' consumers. Then I
> > specified different CGs for different topics' consumers. Even this is not
> > helping.
> >
> > I am trying to search over the web, checked my code, tried many
> > combinations of configuration but still no luck. Please help me.
> >
> > *Thanks & Regards,*
> >
> > *Shantanu Deshmukh*
> >
>

Re: Transfer data between kerberized kafka clusters (different principals)

[Manikumar]

Yes. Sample example/format here:
https://kafka.apache.org/documentation/#security_kerberos_sasl_clientconfig

On Tue, Apr 24, 2018 at 11:30 PM, Zieger, Antoine <
antoine.zie...@morganstanley.com> wrote:

> Hi,
>
> Thank you very much for this quick answer. I am not sure I understand
> where/how to use this property. I see an example here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+
> configuration+for+Kafka+clients but I don't understand how to use it
> while creating a Producer/Consumer instance. Could you help me with this
> please?
>
> A quick example would be the following?
>
> //Producer with specific config: principal 'abc'
> Properties producerConfig = new Properties();
> producerConfig.put("sasl.jaas.config" ,  )
> KafkaProducer<String, String> producer = new KafkaProducer<>(
> producerConfig)
>
> //Consumer with specific config: principal 'xyz'
> Properties consumerConfig = new Properties();
> consumerConfig.put("sasl.jaas.config" ,  )
> KafkaConsumerr<String, String> producer = new KafkaConsumer<>(
> consumerConfig)
>
> Thanks in advance.
> Antoine
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: Tuesday, April 24, 2018 1:39 PM
> To: Users
> Subject: Re: Transfer data between kerberized kafka clusters (different
> principals)
>
> Hi,
>
> From Kafka 0.10.2.0, we can configure  producer/consumer jaas configuration
> using "sasl.jaas.config"  config property. Using this we can configure
> different principals.
>
> On Tue, Apr 24, 2018 at 10:58 PM, Zieger, Antoine <
> antoine.zie...@morganstanley.com> wrote:
>
> > Hi,
> >
> > I am trying to transfer data between two kerberized kafka clusters. The
> > brokers are running under different users on both clusters and so, I am
> > providing a different Kerberos.service.name to consumer and producer.
> For
> > security reasons and access management rules in places, having the same
> > principal on both sides is not possible. Whenever I start my java
> program I
> > get the error "principals don't match"
> >
> > My understanding is that kafka mirror-maker is doing the same thing and
> > does not support different principals. (https://community.
> > hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html )
> > "In kafka 0.9.x and 0.10.0.1, 0.10.1.0, consumers and producers in
> > mirror-maker cannot run with different principals/keytabs as they both
> run
> > inside a single JVM"
> >
> > As I understand it this is a Kafka API limitation right? I looked at the
> > release notes of versions > 0.10.1.0 and didn't see any ticket that
> > resolved this, unless I missed it?
> >
> > By any chance are you aware of a workaround? Or the only solution in case
> > of two different principals is to have two different JVMs? (one for the
> > consumer and one for the producer, which means we need some kind of
> buffer
> > in between).
> >
> > Thanks in advance for your help.
> >
> > Antoine
> >
> >
> > 
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall
> > Street Reform and Consumer Protection Act. If you have received this
> > communication in error, please destroy all electronic and paper copies
> and
> > notify the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to the
> > extent required and/or permitted under applicable law, to monitor
> > electronic communications, including telephone calls with Morgan Stanley
> > personnel. This message is subject to the Morgan Stanley General
> > Disclaimers available at the following link:
> http://www.morganstanley.com/
> > disclaimers.  If you cannot access the links, please notify us by reply
> > message and we will send the contents to you. By communicating with
> Morgan
> > Stanley you acknowledge that you have read, understand and consent,
> (where
> > applicable), to the foregoing and the Morgan Stanley General Disclaimers.
> >
>
> 
> 
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dod