Re: [ANNOUNCE] New Committer: Manikumar Reddy

[Satish Duggana]

Congratulations Mani!


On Fri, Oct 12, 2018 at 9:41 PM Colin McCabe  wrote:
>
> Congratulations, Manikumar!  Well done.
>
> best,
> Colin
>
>
> On Fri, Oct 12, 2018, at 01:25, Edoardo Comar wrote:
> > Well done Manikumar !
> > --
> >
> > Edoardo Comar
> >
> > IBM Event Streams
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> >
> >
> >
> > From:   "Matthias J. Sax" 
> > To: dev 
> > Cc: users 
> > Date:   11/10/2018 23:41
> > Subject:Re: [ANNOUNCE] New Committer: Manikumar Reddy
> >
> >
> >
> > Congrats!
> >
> >
> > On 10/11/18 2:31 PM, Yishun Guan wrote:
> > > Congrats Manikumar!
> > > On Thu, Oct 11, 2018 at 1:20 PM Sönke Liebau
> > >  wrote:
> > >>
> > >> Great news, congratulations Manikumar!!
> > >>
> > >> On Thu, Oct 11, 2018 at 9:08 PM Vahid Hashemian
> > 
> > >> wrote:
> > >>
> > >>> Congrats Manikumar!
> > >>>
> > >>> On Thu, Oct 11, 2018 at 11:49 AM Ryanne Dolan 
> > >>> wrote:
> > >>>
> >  Bravo!
> > 
> >  On Thu, Oct 11, 2018 at 1:48 PM Ismael Juma 
> > wrote:
> > 
> > > Congratulations Manikumar! Thanks for your continued contributions.
> > >
> > > Ismael
> > >
> > > On Thu, Oct 11, 2018 at 10:39 AM Jason Gustafson
> > 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> The PMC for Apache Kafka has invited Manikumar Reddy as a committer
> > >>> and
> > > we
> > >> are
> > >> pleased to announce that he has accepted!
> > >>
> > >> Manikumar has contributed 134 commits including significant work to
> > >>> add
> > >> support for delegation tokens in Kafka:
> > >>
> > >> KIP-48:
> > >>
> > >>
> > >
> > 
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >
> > >> KIP-249
> > >> <
> > >
> > 
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+KafkaKIP-249
> >
> > >>
> > >> :
> > >>
> > >>
> > >
> > 
> > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
> >
> > >>
> > >> He has broad experience working with many of the core components in
> >  Kafka
> > >> and he has reviewed over 80 PRs. He has also made huge progress
> > > addressing
> > >> some of our technical debt.
> > >>
> > >> We appreciate the contributions and we are looking forward to more.
> > >> Congrats Manikumar!
> > >>
> > >> Jason, on behalf of the Apache Kafka PMC
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >> Sönke Liebau
> > >> Partner
> > >> Tel. +49 179 7940878
> > >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
> > [attachment "signature.asc" deleted by Edoardo Comar/UK/IBM]
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Re: [VOTE] 2.0.1 RC0

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 2.0.1
- Ran through quickstart of core/streams on builds generated from tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.
On Sat, Nov 3, 2018 at 2:32 AM Ewen Cheslack-Postava  wrote:
>
> +1
>
> -Ewen
>
> On Thu, Nov 1, 2018 at 10:10 AM Manikumar  wrote:
>
> > We were waiting for the system test results. There were few failures:
> > KAFKA-7579,  KAFKA-7559, KAFKA-7561
> > they are not blockers for 2.0.1 release. We need more votes from
> > PMC/committers :)
> >
> > Thanks Stanislav! for the system test results.
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Nov 1, 2018 at 10:20 PM Eno Thereska 
> > wrote:
> >
> > > Anything else holding this up?
> > >
> > > Thanks
> > > Eno
> > >
> > > On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding) ... I used the staged binaries and run tests with
> > > > different clients.
> > > >
> > > > On Fri, Oct 26, 2018 at 4:29 AM Manikumar 
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 2.0.1.
> > > > >
> > > > > This is a bug fix release closing 49 tickets:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > > > >
> > > > > Release notes for the 2.0.1 release:
> > > > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by  Tuesday, October 30, end of
> > day
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > > > >
> > > > > * Documentation:
> > > > > http://kafka.apache.org/20/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > http://kafka.apache.org/20/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 2.0 branch:
> > > > > Unit/integration tests:
> > > > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >

Re: [VOTE] 2.1.0 RC0

[Satish Duggana]

Hi Dong,
Is there a RC1 planned with configs documentation fixes and
https://github.com/apache/kafka/pull/5857 ?

Thanks,
Satish.
On Thu, Nov 1, 2018 at 4:05 PM Jakub Scholz  wrote:
>
> +1 (non-binding) ... I used the staged binaries and checked it with
> different clients.
>
> On Wed, Oct 24, 2018 at 10:17 AM Dong Lin  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for feature release of Apache Kafka 2.1.0.
> >
> > This is a major version release of Apache Kafka. It includes 28 new KIPs
> > and
> >
> > critical bug fixes. Please see the Kafka 2.1.0 release plan for more
> > details:
> >
> > *
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=91554044*
> >  > >
> >
> > Here are a few notable highlights:
> >
> > - Java 11 support
> > - Support for Zstandard, which achieves compression comparable to gzip with
> > higher compression and especially decompression speeds(KIP-110)
> > - Avoid expiring committed offsets for active consumer group (KIP-211)
> > - Provide Intuitive User Timeouts in The Producer (KIP-91)
> > - Kafka's replication protocol now supports improved fencing of zombies.
> > Previously, under certain rare conditions, if a broker became partitioned
> > from Zookeeper but not the rest of the cluster, then the logs of replicated
> > partitions could diverge and cause data loss in the worst case (KIP-320)
> > - Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356)
> > - Admin script and admin client API improvements to simplify admin
> > operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340)
> > - DNS handling improvements (KIP-235, KIP-302)
> >
> > Release notes for the 2.1.0 release:
> > http://home.apache.org/~lindong/kafka-2.1.0-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote ***
> >
> > * Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~lindong/kafka-2.1.0-rc0/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/
> >
> > * Javadoc:
> > http://home.apache.org/~lindong/kafka-2.1.0-rc0/javadoc/
> >
> > * Tag to be voted upon (off 2.1 branch) is the 2.1.0-rc0 tag:
> > https://github.com/apache/kafka/tree/2.1.0-rc0
> >
> > * Documentation:
> > *http://kafka.apache.org/21/documentation.html*
> > 
> >
> > * Protocol:
> > http://kafka.apache.org/21/protocol.html
> >
> > * Successful Jenkins builds for the 2.1 branch:
> > Unit/integration tests: *https://builds.apache.org/job/kafka-2.1-jdk8/38/
> > *
> >
> > Please test and verify the release artifacts and submit a vote for this RC,
> > or report any issues so we can fix them and get a new RC out ASAP. Although
> > this release vote requires PMC votes to pass, testing, votes, and bug
> > reports are valuable and appreciated from everyone.
> >
> > Cheers,
> > Dong
> >

Re: [VOTE] 2.1.0 RC1

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 2.1.0 successfully without any failures.
- Ran through quickstart of core/streams on builds generated from tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.
On Sun, Nov 18, 2018 at 4:58 AM Jakub Scholz  wrote:
>
> +1 (non-binding) ... I used the staged artifacts and binaries and run my
> tests
>
> On Sat, Nov 10, 2018 at 12:33 AM Dong Lin  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for feature release of Apache Kafka 2.1.0.
> >
> > This is a major version release of Apache Kafka. It includes 28 new KIPs
> > and
> >
> > critical bug fixes. Please see the Kafka 2.1.0 release plan for more
> > details:
> >
> > *
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=91554044*
> >  > >
> >
> > Here are a few notable highlights:
> >
> > - Java 11 support
> > - Support for Zstandard, which achieves compression comparable to gzip with
> > higher compression and especially decompression speeds(KIP-110)
> > - Avoid expiring committed offsets for active consumer group (KIP-211)
> > - Provide Intuitive User Timeouts in The Producer (KIP-91)
> > - Kafka's replication protocol now supports improved fencing of zombies.
> > Previously, under certain rare conditions, if a broker became partitioned
> > from Zookeeper but not the rest of the cluster, then the logs of replicated
> > partitions could diverge and cause data loss in the worst case (KIP-320)
> > - Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356)
> > - Admin script and admin client API improvements to simplify admin
> > operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340)
> > - DNS handling improvements (KIP-235, KIP-302)
> >
> > Release notes for the 2.1.0 release:
> > http://home.apache.org/~lindong/kafka-2.1.0-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, Nov 15, 12 pm PT ***
> >
> > * Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~lindong/kafka-2.1.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/
> >
> > * Javadoc:
> > http://home.apache.org/~lindong/kafka-2.1.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 2.1 branch) is the 2.1.0-rc1 tag:
> > https://github.com/apache/kafka/tree/2.1.0-rc1
> >
> > * Documentation:
> > *http://kafka.apache.org/21/documentation.html*
> > 
> >
> > * Protocol:
> > http://kafka.apache.org/21/protocol.html
> >
> > * Successful Jenkins builds for the 2.1 branch:
> > Unit/integration tests: *https://builds.apache.org/job/kafka-2.1-jdk8/50/
> > *
> >
> > Please test and verify the release artifacts and submit a vote for this RC,
> > or report any issues so we can fix them and get a new RC out ASAP. Although
> > this release vote requires PMC votes to pass, testing, votes, and bug
> > reports are valuable and appreciated from everyone.
> >
> > Cheers,
> > Dong
> >

Re: [DISCUSS] KIP-395: Encypt-then-MAC Delegation token metadata

[Satish Duggana]

Agree with Manikumar on having pluggable mechanism for entities
required/created for delegation token mechanism. I will cover that as
part of KAFKA-7694.

Thanks,
Satish.
On Tue, Dec 11, 2018 at 12:35 PM Manikumar  wrote:
>
> Hi,
>
> Thanks for the KIP.
>
> Currently, master/secret key is stored as plain text in server.properties
> config file.
> Using master secret key as shared secret is again a security risk. We have
> raised KAFKA-7694
> to implement a ZooKeeper based master/secret key management to automate
> secret key rotation.
>
> As you mentioned in the alternatives sections, it is good to have pluggable
> mechanism for
> token storage and master key generation. We can implement pluggable
> interfaces for token storage
> and master key generation as part of KAFKA-7694. This will provide us out
> of the box implementation
> using ZooKeeper and pluggable interfaces for custom implementations.
>
> What do you think?
>
> Thanks,
> Manikumar
>
> On Sat, Dec 1, 2018 at 9:37 PM Attila Sasvári  wrote:
>
> > Hi All,
> >
> > I have a proposal to allow Kafka brokers to encrypt sensitive metadata
> > information about delegation tokens.
> >
> > As of now, delegation token metadata is stored in an unencrypted format in
> > Zookeeper. Having the possibility to encrypt-then-MAC token information
> > would be beneficial in Kafka installations where Zookeeper is not on a
> > private network.
> >
> > Please take a look at
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-395%3A+Encypt-then-MAC+Delegation+token+metadata
> > and let me know what you think.
> >
> > - Attila
> >

Re: Fail-fast builds?

[Satish Duggana]

>>This would allow for fast failure on
compilation and checkstyle problems, but let the whole test suite run in
spite of test failures.

+1 for that as it will be very useful.

Thanks,
Satish.

On Fri, Dec 21, 2018 at 8:10 PM David Arthur  wrote:
>
> Since this is a relatively simple change, I went ahead and opened up a PR
> here https://github.com/apache/kafka/pull/6059
>
> On Fri, Dec 21, 2018 at 2:15 AM Manikumar  wrote:
>
> > +1 fo the suggestion.
> >
> > On Fri, Dec 21, 2018 at 2:38 AM David Arthur  wrote:
> >
> > > In the jenkins.sh file, we have the following comment:
> > >
> > > "In order to provide faster feedback, the tasks are ordered so that
> > faster
> > > tasks are executed in every module before slower tasks (if possible)"
> > >
> > >
> > > but then we proceed to use the Gradle --continue flag. This means PRs
> > won't
> > > get notified of problems until the whole build finishes.
> > >
> > >
> > > What do folks think about splitting the build invocation into a
> > validation
> > > step and a test step? The validation step would omit the continue flag,
> > but
> > > the test step would include it. This would allow for fast failure on
> > > compilation and checkstyle problems, but let the whole test suite run in
> > > spite of test failures.
> > >
> > >
> > > Cheers,
> > > David
> > >
> >
>
>
> --
> David Arthur

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Satish Duggana]

Congratulations Vahid!

On Tue, Jan 15, 2019 at 5:47 PM Yishun Guan  wrote:
>
> Congratulations!
>
> On Tue, Jan 15, 2019, 16:46 James Cheng  wrote:
>
> > Congrats, Vahid!!
> >
> > -James
> >
> > > On Jan 15, 2019, at 2:44 PM, Jason Gustafson  wrote:
> > >
> > > Hi All,
> > >
> > > The PMC for Apache Kafka has invited Vahid Hashemian as a project
> > committer and
> > > we are
> > > pleased to announce that he has accepted!
> > >
> > > Vahid has made numerous contributions to the Kafka community over the
> > past
> > > few years. He has authored 13 KIPs with core improvements to the consumer
> > > and the tooling around it. He has also contributed nearly 100 patches
> > > affecting all parts of the codebase. Additionally, Vahid puts a lot of
> > > effort into community engagement, helping others on the mail lists and
> > > sharing his experience at conferences and meetups.
> > >
> > > We appreciate the contributions and we are looking forward to more.
> > > Congrats Vahid!
> > >
> > > Jason, on behalf of the Apache Kafka PMC
> >
> >

Re: [VOTE] 1.1.1 RC0

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag
- Ran through quickstart of core/streams on builds.

Thanks,
Satish.


On Thu, Jun 21, 2018 at 11:51 AM, zhenya Sun  wrote:

> +1 non-binding
>
> > 在 2018年6月21日，下午2:18，Andras Beni  写道：
> >
> > +1 (non-binding)
> >
> > Built .tar.gz, created a cluster from it and ran a basic end-to-end test:
> > performed a rolling restart while console-producer and console-consumer
> ran
> > at around 20K messages/sec. No errors or data loss.
> >
> > Ran unit and integration tests successfully 3 out of 5 times. Encountered
> > some flakies:
> > - DescribeConsumerGroupTest.testDescribeGroupWithShortInit
> ializationTimeout
> > - LogDirFailureTest.testIOExceptionDuringCheckpoint
> > - SimpleAclAuthorizerTest.testHighConcurrencyModificationOfResourceAcls
> >
> >
> > Andras
> >
> >
> > On Wed, Jun 20, 2018 at 4:59 AM Ted Yu  wrote:
> >
> >> +1
> >>
> >> Ran unit test suite which passed.
> >>
> >> Checked signatures.
> >>
> >> On Tue, Jun 19, 2018 at 4:47 PM, Dong Lin  wrote:
> >>
> >>> Re-send to kafka-clie...@googlegroups.com
> >>>
> >>> On Tue, Jun 19, 2018 at 4:29 PM, Dong Lin  wrote:
> >>>
>  Hello Kafka users, developers and client-developers,
> 
>  This is the first candidate for release of Apache Kafka 1.1.1.
> 
>  Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> >> first
>  released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> >>> since
>  that release. A few of the more significant fixes include:
> 
>  KAFKA-6925  - Fix
>  memory leak in StreamsMetricsThreadImpl
>  KAFKA-6937  -
> >> In-sync
>  replica delayed during fetch if replica throttle is exceeded
>  KAFKA-6917  -
> >> Process
>  txn completion asynchronously to avoid deadlock
>  KAFKA-6893  -
> Create
>  processors before starting acceptor to avoid ArithmeticException
>  KAFKA-6870  -
>  Fix ConcurrentModificationException in SampledStat
>  KAFKA-6878  - Fix
>  NullPointerException when querying global state store
>  KAFKA-6879  -
> Invoke
>  session init callbacks outside lock to avoid Controller deadlock
>  KAFKA-6857  -
> >> Prevent
>  follower from truncating to the wrong offset if undefined leader epoch
> >> is
>  requested
>  KAFKA-6854  - Log
>  cleaner fails with transaction markers that are deleted during clean
>  KAFKA-6747  - Check
>  whether there is in-flight transaction before aborting transaction
>  KAFKA-6748  -
> Double
>  check before scheduling a new task after the punctuate call
>  KAFKA-6739  -
>  Fix IllegalArgumentException when down-converting from V2 to V0/V1
>  KAFKA-6728  -
>  Fix NullPointerException when instantiating the HeaderConverter
> 
>  Kafka 1.1.1 release plan:
>  https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> 
>  Release notes for the 1.1.1 release:
>  http://home.apache.org/~lindong/kafka-1.1.1-rc0/RELEASE_NOTES.html
> 
>  *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
> 
>  Kafka's KEYS file containing PGP keys we use to sign the release:
>  http://kafka.apache.org/KEYS
> 
>  * Release artifacts to be voted upon (source and binary):
>  http://home.apache.org/~lindong/kafka-1.1.1-rc0/
> 
>  * Maven artifacts to be voted upon:
>  https://repository.apache.org/content/groups/staging/
> 
>  * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc0 tag:
>  https://github.com/apache/kafka/tree/1.1.1-rc0
> 
>  * Documentation:
>  http://kafka.apache.org/11/documentation.html
> 
>  * Protocol:
>  http://kafka.apache.org/11/protocol.html
> 
>  * Successful Jenkins builds for the 1.1 branch:
>  Unit/integration tests: https://builds.apache.org/job/
> >>> kafka-1.1-jdk7/150/
> 
>  Please test and verify the release artifacts and submit a vote for
> this
> >>> RC,
>  or report any issues so we can fix them and get a new RC out ASAP.
> >>> Although
>  this release vote requires PMC votes to pass, testing, votes, and bug
>  reports are valuable and appreciated from everyone.
> 
>  Cheers,
>  Dong
> 
> 
> 
> >>>
> >>
>
>

Re: [kafka-clients] [VOTE] 2.0.0 RC1

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag
- Ran through quickstart of core/streams on builds.

Thanks,
Satish.



On Sun, Jul 1, 2018 at 6:00 AM, Ted Yu  wrote:

> +1
>
> Checked signatures
> Ran test suite
>
> On Sat, Jun 30, 2018 at 2:37 AM, Rajini Sivaram 
> wrote:
>
> > Hi Manikumar,
> >
> > Thank you for pointing that out, I had forgotten to drop the old
> artifacts.
> > New artifacts should be there now.
> >
> > Regards,
> >
> > Rajini
> >
> > On Sat, Jun 30, 2018 at 7:44 AM, Manikumar 
> > wrote:
> >
> > > looks like maven artifacts are not updated in the staging repo. They
> are
> > > still at old timestamp.
> > > https://repository.apache.org/content/groups/staging/org/
> > > apache/kafka/kafka_2.11/2.0.0/
> > >
> > > On Sat, Jun 30, 2018 at 12:06 AM Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >>
> > >> This is the second candidate for release of Apache Kafka 2.0.0.
> > >>
> > >>
> > >> This is a major version release of Apache Kafka. It includes 40 new
> > KIPs
> > >> and
> > >>
> > >> several critical bug fixes. Please see the 2.0.0 release plan for more
> > >> details:
> > >>
> > >> https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=80448820
> > >>
> > >>
> > >> A few notable highlights:
> > >>
> > >>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> > >>CreateTopics (KIP-277)
> > >>- SASL/OAUTHBEARER implementation (KIP-255)
> > >>- Improved quota communication and customization of quotas
> (KIP-219,
> > >>KIP-257)
> > >>- Efficient memory usage for down conversion (KIP-283)
> > >>- Fix log divergence between leader and follower during fast leader
> > >>failover (KIP-279)
> > >>- Drop support for Java 7 and remove deprecated code including old
> > >>scala clients
> > >>- Connect REST extension plugin, support for externalizing secrets
> > >>and improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > >>- Scala API for Kafka Streams and other Streams API improvements
> > >>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> > >>
> > >> Release notes for the 2.0.0 release:
> > >>
> > >> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/RELEASE_NOTES.html
> > >>
> > >>
> > >>
> > >> *** Please download, test and vote by Tuesday, July 3rd, 4pm PT
> > >>
> > >>
> > >> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>
> > >> http://kafka.apache.org/KEYS
> > >>
> > >>
> > >> * Release artifacts to be voted upon (source and binary):
> > >>
> > >> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/
> > >>
> > >>
> > >> * Maven artifacts to be voted upon:
> > >>
> > >> https://repository.apache.org/content/groups/staging/
> > >>
> > >>
> > >> * Javadoc:
> > >>
> > >> http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/javadoc/
> > >>
> > >>
> > >> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> > >>
> > >> https://github.com/apache/kafka/tree/2.0.0-rc1
> > >>
> > >>
> > >> * Documentation:
> > >>
> > >> http://kafka.apache.org/20/documentation.html
> > >>
> > >>
> > >> * Protocol:
> > >>
> > >> http://kafka.apache.org/20/protocol.html
> > >>
> > >>
> > >> * Successful Jenkins builds for the 2.0 branch:
> > >>
> > >> Unit/integration tests: https://builds.apache.org/job/
> > kafka-2.0-jdk8/66/
> > >>
> > >> System tests: https://jenkins.confluent.io/job/system-test-
> > >> kafka/job/2.0/15/
> > >>
> > >>
> > >>
> > >> Please test and verify the release artifacts and submit a vote for
> this
> > RC
> > >> or report any issues so that we can fix them and roll out a new RC
> ASAP!
> > >>
> > >> Although this release vote requires PMC votes to pass, testing, votes,
> > >> and bug
> > >> reports are valuable and appreciated from everyone.
> > >>
> > >>
> > >> Thanks,
> > >>
> > >>
> > >> Rajini
> > >>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "kafka-clients" group.
> > >> To unsubscribe from this group and stop receiving emails from it, send
> > an
> > >> email to kafka-clients+unsubscr...@googlegroups.com.
> > >> To post to this group, send email to kafka-clie...@googlegroups.com.
> > >> Visit this group at https://groups.google.com/group/kafka-clients.
> > >> To view this discussion on the web visit https://groups.google.com/d/
> > >> msgid/kafka-clients/CAOJcB39GdTWOaK4qysvyPyGU8Ldm8
> > >> 2t_TA364x1MP8a8OAod6A%40mail.gmail.com
> > >>  > CAOJcB39GdTWOaK4qysvyPyGU8Ldm82t_TA364x1MP8a8OAod6A%40mail.
> > gmail.com?utm_medium=email&utm_source=footer>
> > >> .
> > >> For more options, visit https://groups.google.com/d/optout.
> > >>
> > >
> >
>

Re: [VOTE] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

[Satish Duggana]

+1

Thanks,
Satish.

On Wed, Jul 4, 2018 at 4:11 PM, Daniele Ascione  wrote:

> +1
>
> Thanks,
> Daniele
>
> Il giorno mar 3 lug 2018 alle ore 23:55 Harsha  ha
> scritto:
>
> > +1.
> >
> > Thanks,
> > Harsha
> >
> > On Tue, Jul 3rd, 2018 at 9:22 AM, Ted Yu  wrote:
> >
> > >
> > >
> > >
> > > +1
> > >
> > > On Tue, Jul 3, 2018 at 9:05 AM, Mickael Maison <
> > mickael.mai...@gmail.com >
> > >
> > > wrote:
> > >
> > > > +1 (non binding)
> > > > Thanks for the KIP
> > > >
> > > > On Tue, Jul 3, 2018 at 4:59 PM, Vahid S Hashemian
> > > > < vahidhashem...@us.ibm.com > wrote:
> > > > > +1 (non-binding)
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > > From: Gwen Shapira < g...@confluent.io >
> > > > > To: dev < dev@kafka.apache.org >
> > > > > Date: 07/03/2018 08:49 AM
> > > > > Subject: Re: [VOTE] KIP-322: Return new error code for
> > > > DeleteTopics
> > > > > API when topic deletion disabled.
> > > > >
> > > > >
> > > > >
> > > > > +1
> > > > >
> > > > > On Tue, Jul 3, 2018 at 8:24 AM, Manikumar <
> > manikumar.re...@gmail.com >
> > >
> > > > > wrote:
> > > > >
> > > > >> Manikumar < manikumar.re...@gmail.com >
> > > > >> Fri, Jun 29, 7:59 PM (4 days ago)
> > > > >> to dev
> > > > >> Hi All,
> > > > >>
> > > > >> I would like to start voting on KIP-322 which would return new
> error
> > > > > code
> > > > >> for DeleteTopics API when topic deletion disabled.
> > > > >>
> > > > >>
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=87295558
> > > > >
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter <
> > > > > https://twitter.com/ConfluentInc
> > > > >> | blog
> > > > > <
> > > > > http://www.confluent.io/blog
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
>

Re: [VOTE] 1.1.1 RC3

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag with jdk-8.
- Ran through quickstart of core/streams on builds.
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.


On Mon, Jul 9, 2018 at 11:39 PM, Harsha  wrote:

> +1.
>
> * Ran unit tests
> * Installed in a cluster and ran simple tests
>
> Thanks,
> Harsha
>
> On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu  wrote:
>
> >
> >
> >
> > +1
> >
> > Ran test suite.
> >
> > Checked signatures.
> >
> >
> >
> > On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon...@gmail.com > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the fourth candidate for release of Apache Kafka 1.1.1.
> > >
> > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> > first
> > > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> > since
> > > that release. A few of the more significant fixes include:
> > >
> > > KAFKA-6925 < https://issues.apache.org/jira/browse/KAFKA-6925> - Fix
> > memory
> > > leak in StreamsMetricsThreadImpl
> > > KAFKA-6937 < https://issues.apache.org/jira/browse/KAFKA-6937> -
> In-sync
> >
> > > replica delayed during fetch if replica throttle is exceeded
> > > KAFKA-6917 < https://issues.apache.org/jira/browse/KAFKA-6917> -
> Process
> >
> > > txn
> > > completion asynchronously to avoid deadlock
> > > KAFKA-6893 < https://issues.apache.org/jira/browse/KAFKA-6893> -
> Create
> > > processors before starting acceptor to avoid ArithmeticException
> > > KAFKA-6870 < https://issues.apache.org/jira/browse/KAFKA-6870> -
> > > Fix ConcurrentModificationException in SampledStat
> > > KAFKA-6878 < https://issues.apache.org/jira/browse/KAFKA-6878> - Fix
> > > NullPointerException when querying global state store
> > > KAFKA-6879 < https://issues.apache.org/jira/browse/KAFKA-6879> -
> Invoke
> > > session init callbacks outside lock to avoid Controller deadlock
> > > KAFKA-6857 < https://issues.apache.org/jira/browse/KAFKA-6857> -
> Prevent
> >
> > > follower from truncating to the wrong offset if undefined leader epoch
> > is
> > > requested
> > > KAFKA-6854 < https://issues.apache.org/jira/browse/KAFKA-6854> - Log
> > > cleaner
> > > fails with transaction markers that are deleted during clean
> > > KAFKA-6747 < https://issues.apache.org/jira/browse/KAFKA-6747> - Check
> > > whether there is in-flight transaction before aborting transaction
> > > KAFKA-6748 < https://issues.apache.org/jira/browse/KAFKA-6748> -
> Double
> > > check before scheduling a new task after the punctuate call
> > > KAFKA-6739 < https://issues.apache.org/jira/browse/KAFKA-6739> -
> > > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> > > KAFKA-6728 < https://issues.apache.org/jira/browse/KAFKA-6728> -
> > > Fix NullPointerException when instantiating the HeaderConverter
> > >
> > > Kafka 1.1.1 release plan:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> > >
> > > Release notes for the 1.1.1 release:
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Thursday, July 12, 12pm PT ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
> > >
> > > * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> > > https://github.com/apache/kafka/tree/1.1.1-rc3
> > >
> > > * Documentation:
> > > http://kafka.apache.org/11/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/11/protocol.html
> > >
> > > * Successful Jenkins builds for the 1.1 branch:
> > > Unit/integration tests: * https://builds.apache.org/job/
> kafka-1.1-jdk7/162
> >
> > > < https://builds.apache.org/job/kafka-1.1-jdk7/162>*
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
> > >
> > > Please test and verify the release artifacts and submit a vote for this
> > RC,
> > > or report any issues so we can fix them and get a new RC out ASAP.
> > Although
> > > this release vote requires PMC votes to pass, testing, votes, and bug
> > > reports are valuable and appreciated from everyone.
> > >
> > >
> > > Regards,
> > > Dong
> > >
> >
> >
> >
> >
> >
> >
>

Re: [VOTE] 2.0.0 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 2.0.0-rc2 tag
- Ran through quickstart of core/streams on builds generated from tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.

On Sun 15 Jul, 2018, 9:55 PM Rajini Sivaram, 
wrote:

> Hi Ismael,
>
> Thank you for pointing that out. I have re-uploaded the RC2 artifacts to
> maven including streams-scala_2.12. Also submitted a PR to update build &
> release scripts to include this.
>
> Thank you,
>
> Rajini
>
>
>
> On Fri, Jul 13, 2018 at 7:19 AM, Ismael Juma  wrote:
>
> > Hi Rajini,
> >
> > Thanks for generating the RC. It seems like the kafka-streams-scala 2.12
> > artifact is missing from the Maven repository:
> >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > Since this is the first time we are publishing this artifact, it is
> > possible that this never worked properly.
> >
> > Ismael
> >
> > On Tue, Jul 10, 2018 at 10:17 AM Rajini Sivaram  >
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > >
> > > This is the third candidate for release of Apache Kafka 2.0.0.
> > >
> > >
> > > This is a major version release of Apache Kafka. It includes 40 new
> KIPs
> > > and
> > >
> > > several critical bug fixes. Please see the 2.0.0 release plan for more
> > > details:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=80448820
> > >
> > >
> > > A few notable highlights:
> > >
> > >- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> CreateTopics
> > >(KIP-277)
> > >- SASL/OAUTHBEARER implementation (KIP-255)
> > >- Improved quota communication and customization of quotas (KIP-219,
> > >KIP-257)
> > >- Efficient memory usage for down conversion (KIP-283)
> > >- Fix log divergence between leader and follower during fast leader
> > >failover (KIP-279)
> > >- Drop support for Java 7 and remove deprecated code including old
> > scala
> > >clients
> > >- Connect REST extension plugin, support for externalizing secrets
> and
> > >improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > >- Scala API for Kafka Streams and other Streams API improvements
> > >(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> > >
> > >
> > > Release notes for the 2.0.0 release:
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
> > >
> > >
> > > *** Please download, test and vote by Friday, July 13, 4pm PT
> > >
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >
> > > http://kafka.apache.org/KEYS
> > >
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
> > >
> > >
> > > * Maven artifacts to be voted upon:
> > >
> > > https://repository.apache.org/content/groups/staging/
> > >
> > >
> > > * Javadoc:
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
> > >
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> > >
> > > https://github.com/apache/kafka/tree/2.0.0-rc2
> > >
> > >
> > >
> > > * Documentation:
> > >
> > > http://kafka.apache.org/20/documentation.html
> > >
> > >
> > > * Protocol:
> > >
> > > http://kafka.apache.org/20/protocol.html
> > >
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > >
> > > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.0-jdk8/72/
> > >
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
> > >
> > >
> > > /**
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Rajini
> > >
> >
>

Re: [ANNOUNCE] New committer: Matthias J. Sax

[Satish Duggana]

Congratulations Mathias!

On Tue, Jan 16, 2018 at 11:52 AM, Becket Qin  wrote:

> Congrats, Matthias!
>
> On Mon, Jan 15, 2018 at 9:54 PM, Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Matthias! Congratulations!
> >
> > Konstantine
> >
> > On Mon, Jan 15, 2018 at 4:18 AM, Edoardo Comar 
> wrote:
> >
> > > Congratulations Matthias !
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Message Hub
> > >
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > >
> > >
> > > From:   UMESH CHAUDHARY 
> > > To: dev@kafka.apache.org
> > > Date:   15/01/2018 11:47
> > > Subject:Re: [ANNOUNCE] New committer: Matthias J. Sax
> > >
> > >
> > >
> > > Congratulations Matthias :)
> > >
> > > On Mon, 15 Jan 2018 at 17:06 Michael Noll 
> wrote:
> > >
> > > > Herzlichen Glückwunsch, Matthias. ;-)
> > > >
> > > >
> > > >
> > > > On Mon, 15 Jan 2018 at 11:38 Viktor Somogyi  >
> > > > wrote:
> > > >
> > > > > Congrats Matthias!
> > > > >
> > > > > On Mon, Jan 15, 2018 at 9:17 AM, Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jo...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Matthias!!
> > > > > >
> > > > > > El lun., 15 ene. 2018 a las 9:08, Boyang Chen
> > > ()
> > > > > > escribió:
> > > > > >
> > > > > > > Great news Matthias!
> > > > > > >
> > > > > > >
> > > > > > > 
> > > > > > > From: Kaufman Ng 
> > > > > > > Sent: Monday, January 15, 2018 11:32 AM
> > > > > > > To: us...@kafka.apache.org
> > > > > > > Cc: dev@kafka.apache.org
> > > > > > > Subject: Re: [ANNOUNCE] New committer: Matthias J. Sax
> > > > > > >
> > > > > > > Congrats Matthias!
> > > > > > >
> > > > > > > On Sun, Jan 14, 2018 at 4:52 AM, Rajini Sivaram <
> > > > > rajinisiva...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations Matthias!
> > > > > > > >
> > > > > > > > On Sat, Jan 13, 2018 at 11:34 AM, Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Matthias !
> > > > > > > > >
> > > > > > > > > On Sat, Jan 13, 2018 at 7:01 AM, Paolo Patierno <
> > > > > ppatie...@live.com>
> > > > > > > > > wrote:
> > > > > > > > > > Congratulations Matthias ! Very well deserved !
> > > > > > > > > > 
> > > > > > > > > > From: Guozhang Wang 
> > > > > > > > > > Sent: Friday, January 12, 2018 11:59:21 PM
> > > > > > > > > > To: dev@kafka.apache.org; us...@kafka.apache.org
> > > > > > > > > > Subject: [ANNOUNCE] New committer: Matthias J. Sax
> > > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > The PMC of Apache Kafka is pleased to announce Matthias
> J.
> > > Sax
> > > > as
> > > > > > our
> > > > > > > > > > newest Kafka committer.
> > > > > > > > > >
> > > > > > > > > > Matthias has made tremendous contributions to Kafka
> Streams
> > > API
> > > > > > since
> > > > > > > > > early
> > > > > > > > > > 2016. His footprint has been all over the places in
> > Streams:
> > > in
> > > > > the
> > > > > > > > past
> > > > > > > > > > two years he has been the main driver on improving the
> join
> > > > > > semantics
> > > > > > > > > > inside Streams DSL, summarizing all their shortcomings
> and
> > > > > bridging
> > > > > > > the
> > > > > > > > > > gaps; he has also been largely working on the
> exactly-once
> > > > > > semantics
> > > > > > > of
> > > > > > > > > > Streams by leveraging on the transaction messaging
> feature
> > > in
> > > > > > 0.11.0.
> > > > > > > > In
> > > > > > > > > > addition, Matthias have been very active in community
> > > activity
> > > > > that
> > > > > > > > goes
> > > > > > > > > > beyond mailing list: he's getting the close to 1000 up
> > votes
> > > > and
> > > > > > 100
> > > > > > > > > > helpful flags on SO for answering almost all questions
> > about
> > > > > Kafka
> > > > > > > > > Streams.
> > > > > > > > > >
> > > > > > > > > > Thank you for your contribution and welcome to Apache
> > Kafka,
> > > > > > > Matthias!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Guozhang, on behalf of the Apache Kafka PMC
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Kaufman Ng
> > > > > > > +1 646 961 8063 <+1%20646-961-8063> <(646)%20961-8063>
> > > > > > > Solutions Architect | Confluent | www.confluent.io<
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www&d=
> > > DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=EzRhmSah4IHsUZVekRUIINhltZK7U0
> > > OaeRo7hgW4_tQ&m=1kMl7t3vDn8RaG7aDYpjxKcgtAA0RVnWDeMBhN7yjzE&s=
> > > qPzEPox7QPw6NE5uYyUmMHo1Z4LPhLoC9hZBexOiBT8&e=
> > > .
> > > > > > confluent.io
> > > > > > > >
> > > > > > > [
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
> > > confluent.io_wp-2Dcontent_uploads_Untitled-2Ddesign-
> > > 2D12.png&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> > > EzRhmSah4IHsUZVekRUIINh

Re: [VOTE] KIP-249: Add Delegation Token Operations to Kafka Admin Client

[Satish Duggana]

 +1, thanks for the KIP.

~Satish.

On Wed, Jan 24, 2018 at 5:09 AM, Jun Rao  wrote:

> Hi, Mani,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Sun, Jan 21, 2018 at 7:44 AM, Manikumar 
> wrote:
>
> > Hi All,
> >
> > I would like to start a vote on KIP-249 which would add delegation token
> > operations
> > to Java Admin Client.
> >
> > We have merged DelegationToken API PR recently. We want to include admin
> > client changes in the upcoming release. This will make the feature
> > complete.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
> >
> > Thanks,
> >
>

Re: [VOTE] 1.1.0 RC2

[Satish Duggana]

Hi Damian,
Given release plan link in earlier mail is about 1.0 release. You may want
to replace that with 1.1.0 release plan link[1].

1 -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

Thanks,
Satish.

On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 1.1.0.
>
> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> Please see the release plan for more details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
> Release notes for the 1.1.0 release:
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, March 16, 1pm PDT>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> https://github.com/apache/kafka/tree/1.1.0-rc
> 2
>
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
> 
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
> 
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/
> 38/
>
> /**
>
> Thanks,
> Damian
>
>
> *
>

Re: [VOTE] 1.1.0 RC2

[Satish Duggana]

Hi Damian,
Thanks for starting vote thread for 1.1.0 release.

There may be a typo on the tag to be voted upon for this release candidate.
I guess it should be https://github.com/apache/kafka/tree/1.1.0-rc2 instead
of https://github.com/apache/kafka/tree/1.1.0-rc.

On Wed, Mar 14, 2018 at 8:27 AM, Satish Duggana 
wrote:

> Hi Damian,
> Given release plan link in earlier mail is about 1.0 release. You may want
> to replace that with 1.1.0 release plan link[1].
>
> 1 - https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75957546
>
> Thanks,
> Satish.
>
> On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the third candidate for release of Apache Kafka 1.1.0.
>>
>> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
>> Please see the release plan for more details:
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
>>
>> A few highlights:
>>
>> * Significant Controller improvements (much faster and session expiration
>> edge cases fixed)
>> * Data balancing across log directories (JBOD)
>> * More efficient replication when the number of partitions is large
>> * Dynamic Broker Configs
>> * Delegation tokens (KIP-48)
>> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>>
>> Release notes for the 1.1.0 release:
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, March 16, 1pm PDT>
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
>>
>> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
>> https://github.com/apache/kafka/tree/1.1.0-rc
>> <https://github.com/apache/kafka/tree/1.1.0-rc1>2
>>
>>
>> * Documentation:
>> http://kafka.apache.org/11/documentation.html
>> <http://kafka.apache.org/1/documentation.html>
>>
>> * Protocol:
>> http://kafka.apache.org/11/protocol.html
>> <http://kafka.apache.org/1/protocol.html>
>>
>> * Successful Jenkins builds for the 1.1 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
>> System tests: https://jenkins.confluent.io/j
>> ob/system-test-kafka/job/1.1/38/
>>
>> /**
>>
>> Thanks,
>> Damian
>>
>>
>> *
>>
>
>

Re: [VOTE] 1.1.0 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 1.1.0-RC2 tag
- Ran through quickstart of core/streams
- Ran a cluster from binaries built form sources
  - with multiple operations.

Thanks,
Satish.


On Thu, Mar 15, 2018 at 3:25 AM, Harsha  wrote:

> +1
> Ran tests
> Ran a 3 node cluster to test basic operations.
>
> Thanks,
> Harsha
>
> On Wed, Mar 14, 2018, at 9:04 AM, Ted Yu wrote:
> > +1
> >
> > Ran test suite - passed (apart from testMetricsLeak which is flaky).
> >
> > On Wed, Mar 14, 2018 at 3:30 AM, Damian Guy 
> wrote:
> >
> > > Thanks for pointing out Satish. Links updated:
> > >
> > > 
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka 1.1.0.
> > >
> > > This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> > > Please see the release plan for more details:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75957546
> > >
> > > A few highlights:
> > >
> > > * Significant Controller improvements (much faster and session
> expiration
> > > edge cases fixed)
> > > * Data balancing across log directories (JBOD)
> > > * More efficient replication when the number of partitions is large
> > > * Dynamic Broker Configs
> > > * Delegation tokens (KIP-48)
> > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > >
> > > Release notes for the 1.1.0 release:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday, March 16, 1pm PDT>
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > https://github.com/apache/kafka/tree/1.1.0-rc2
> > >
> > >
> > > * Documentation:
> > > http://kafka.apache.org/11/documentation.html
> > > <http://kafka.apache.org/1/documentation.html>
> > >
> > > * Protocol:
> > > http://kafka.apache.org/11/protocol.html
> > > <http://kafka.apache.org/1/protocol.html>
> > >
> > > * Successful Jenkins builds for the 1.1 branch:
> > > Unit/integration tests: https://builds.apache.org/job/
> kafka-1.1-jdk7/78
> > > System tests: https://jenkins.confluent.io/
> job/system-test-kafka/job/1.1/
> > > 38/
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75957546
> > >
> > > On Wed, 14 Mar 2018 at 04:41 Satish Duggana 
> > > wrote:
> > >
> > > > Hi Damian,
> > > > Thanks for starting vote thread for 1.1.0 release.
> > > >
> > > > There may be a typo on the tag to be voted upon for this release
> > > candidate.
> > > > I guess it should be https://github.com/apache/kafka/tree/1.1.0-rc2
> > > > instead
> > > > of https://github.com/apache/kafka/tree/1.1.0-rc.
> > > >
> > > > On Wed, Mar 14, 2018 at 8:27 AM, Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Damian,
> > > > > Given release plan link in earlier mail is about 1.0 release. You
> may
> > > > want
> > > > > to replace that with 1.1.0 release plan link[1].
> > > > >
> > > > > 1 - https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=75957546
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy  >
> > > > wrote:
> > > > >
> > > > >> Hello Kafka users, developers and client-developers,
> > > > >>
> > > > >> This is the third candidate for release of Apache Kafka 1.1.0.
> > > > >>
> > > > >>

Re: [VOTE] KIP-289: Improve the default group id behavior in KafkaConsumer

[Satish Duggana]

+1 (non binding)
Thanks Vahid,

On Tue, Aug 14, 2018 at 2:22 AM, Gwen Shapira  wrote:

> +1 (binding)
>
> On Tue, Aug 7, 2018 at 11:14 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-289 to modify the default group id of
> > KafkaConsumer.
> > The KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer
> > The discussion thread:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg87379.html
> >
> > Thanks!
> > --Vahid
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [VOTE] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

[Satish Duggana]

+1

Thanks,
Satish.

On Fri, Aug 17, 2018 at 1:45 AM, Bill Bejeck  wrote:

> +1
>
> Thanks,
> Bill
>
> On Thu, Aug 16, 2018 at 3:13 PM Ted Yu  wrote:
>
> > +1
> >
> > On Thu, Aug 16, 2018 at 12:05 PM Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > I would like to start a vote on KIP-325 which aims at adding a
> beginning
> > > offset column to consumer group command describe output.
> > >
> > > The KIP:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 325%3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> > > Discussion thread:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg89203.html
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

[Satish Duggana]

Congrats Dong Lin!

On Wed, Aug 22, 2018 at 10:08 AM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:

> Congratulations, Dong!
>
> On Wed, Aug 22, 2018 at 6:20 AM Dhruvil Shah  wrote:
>
> > Congratulations, Dong!
> >
> > On Tue, Aug 21, 2018 at 4:38 PM Jason Gustafson 
> > wrote:
> >
> > > Congrats!
> > >
> > > On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang 
> wrote:
> > >
> > > > Congrats Dong!
> > > >
> > > > -Ray
> > > >
> > > >
> > > > On 8/21/18 9:33 AM, Becket Qin wrote:
> > > >
> > > >> Congrats, Dong!
> > > >>
> > > >> On Aug 21, 2018, at 11:03 PM, Eno Thereska 
> > > >>> wrote:
> > > >>>
> > > >>> Congrats Dong!
> > > >>>
> > > >>> Eno
> > > >>>
> > > >>> On Tue, Aug 21, 2018 at 7:05 AM, Ted Yu 
> wrote:
> > > >>>
> > > >>> Congratulation Dong!
> > > 
> > >  On Tue, Aug 21, 2018 at 1:59 AM Viktor Somogyi-Vass <
> > >  viktorsomo...@gmail.com>
> > >  wrote:
> > > 
> > >  Congrats Dong! :)
> > > >
> > > > On Tue, Aug 21, 2018 at 10:09 AM James Cheng <
> wushuja...@gmail.com
> > >
> > > >
> > >  wrote:
> > > 
> > > > Congrats Dong!
> > > >>
> > > >> -James
> > > >>
> > > >> On Aug 20, 2018, at 3:54 AM, Ismael Juma 
> > wrote:
> > > >>>
> > > >>> Hi everyone,
> > > >>>
> > > >>> Dong Lin became a committer in March 2018. Since then, he has
> > > >>>
> > > >> remained
> > > 
> > > > active in the community and contributed a number of patches,
> > reviewed
> > > >>> several pull requests and participated in numerous KIP
> > > discussions. I
> > > >>>
> > > >> am
> > > >
> > > >> happy to announce that Dong is now a member of the
> > > >>> Apache Kafka PMC.
> > > >>>
> > > >>> Congratulation Dong! Looking forward to your future
> > contributions.
> > > >>>
> > > >>> Ismael, on behalf of the Apache Kafka PMC
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Satish Duggana]

+1

On Wed, Aug 22, 2018 at 4:45 PM, Ted Yu  wrote:

> +1
>  Original message From: Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> Date: 8/22/18  3:19 AM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-336: Consolidate
> ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
> +1
>
> Thanks for the KIP!
>
> On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on this KIP (
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=87298242)
> > which aims to refactor ExtendedSerializer/Serializer and
> > ExtendedDeserializer/Deserializer.
> >
> > To summarize what's the motivation:
> >
> > When headers were introduced by KIP-82 the ExtendedSerializer and
> > ExtendedDeserializer classes were created in order to keep interface
> > compatibility but still add `T deserialize(String topic, Headers headers,
> > byte[] data);` and `byte[] serialize(String topic, Headers headers, T
> > data);` methods that consume the headers for
> serialization/deserialization.
> > The reason for doing so was that Kafka at that time needed be compatbile
> > with Java 7. Since we're not compiling on Java 7 anymore (KAFKA-4423)
> we'll
> > try consolidate the way we're using these in a backward compatible
> fashion:
> > deprecating the Extended* classes and moving the aforementioned methods
> up
> > in the class hierarchy.
> >
> > I'd be happy to get votes or additional feedback on this.
> >
> > Viktor
> >
>

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

[Satish Duggana]

Hi Mani,
Just a minor comment on the output of the command as given in KIP-357, you
may want to remove "User:User1 has " as it is redundant  for each ACL.
It may be good to accept multiple principals option to avoid running this
script multiple times with each principal to achieve the same.


>> sh kafka-acls.sh

--authorizer-properties zookeeper.connect=localhost:

2181 --list --principal User:User1
ACLs for principal `User:User1`
Current ACLs for resource `Group:PREFIXED:TEST_GROUP`:
User:User1 has Allow permission for operations: Read from hosts: *

Current ACLs for resource `Topic:PREFIXED:TEST_TOPIC`:
User:User1 has Allow permission for operations: Read from hosts: *
User:User1 has Allow permission for operations: Create from hosts: *
User:User1 has Allow permission for operations: Write from hosts: *
User:User1 has Allow permission for operations: Describe from hosts: *

Thanks,
Satish.


On Fri, Aug 24, 2018 at 1:13 AM, Harsha  wrote:

> +1 (binding)
>
> Thanks,
> Harsha
>
> On Wed, Aug 22, 2018, at 9:15 AM, Manikumar wrote:
> > Hi Viktor,
> > We already have a method in Authorizer interface to get acls for a given
> > principal.
> > We will use this method to fetch acls and filter the results for
> requested
> > Resources.
> > Authorizer {
> >def getAcls(principal: KafkaPrincipal): Map[Resource, Set[Acl]]
> > }
> > Currently AdminClient API doesn't have a API to fetch acls for a given
> > principal.
> > So while using AclCommand with AdminClient API (KIP-332), we just filter
> > the results returned
> > from describeAcls API. We can add new AdminClient API/new
> > DescribeAclsRequest if required in future.
> >
> > Updated the KIP. Thanks for the review.
> >
> > Thanks,
> >
> > On Wed, Aug 22, 2018 at 5:53 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > Implementation-wise is it just a filter over the returned ACL listing
> or do
> > > you plan to add new methods to the Authorizer as well?
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Fri, Aug 17, 2018 at 9:18 PM Priyank Shah 
> > > wrote:
> > >
> > > > +1(non-binding)
> > > >
> > > > Thanks.
> > > > Priyank
> > > >
> > > > On 8/16/18, 6:01 AM, "Manikumar"  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I have created a minor KIP to add support to list ACLs per
> principal
> > > > using
> > > > AclCommand (kafka-acls.sh)
> > > >
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 357%3A++Add+support+to+list+ACLs+per+principal
> > > >
> > > > Please take a look.
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > >
>

Re: [VOTE] KIP-357: Add support to list ACLs per principal

[Satish Duggana]

+1 (non-binding)

On Tue, Aug 28, 2018 at 2:59 AM, Harsha  wrote:

> +1 (binding)
>
> -Harsha
>
> On Mon, Aug 27, 2018, at 12:46 PM, Jakub Scholz wrote:
> > +1 (non-binding)
> >
> > On Mon, Aug 27, 2018 at 6:24 PM Manikumar 
> wrote:
> >
> > > Hi All,
> > >
> > > I would like to start voting on KIP-357 which allows to list ACLs per
> > > principal using AclCommand (kafka-acls.sh)
> > >
> > > KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 357%3A++Add+support+to+list+ACLs+per+principal
> > >
> > > Discussion Thread:
> > >
> > > https://lists.apache.org/thread.html/dc7f6005845a372a0a48a40872a32d
> 9ece03807a4fb1bb89d3645afb@%3Cdev.kafka.apache.org%3E
> > >
> > > Thanks,
> > > Manikumar
> > >
>

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[Satish Duggana]

+including both dev and user mailing lists.

Hi,
Thanks for the KIP.

"* For us, the message keys represent some metadata which we use to either
ignore messages (if a loop-back to the sender), or log some information.*"

Above statement was mentioned in the KIP about how key value is used. I
guess the topic is not configured to be compacted and you do not want to
have partitioning based on that key. IMHO, it qualifies more as a header
than a key. What do you think about building records with a specific header
and consumers to execute the logic whether to process or ignore the
messages based on that header value.

Thanks,
Satish.


On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana 
wrote:

> Hi,
> Thanks for the KIP.
>
> "* For us, the message keys represent some metadata which we use to
> either ignore messages (if a loop-back to the sender), or log some
> information.*"
>
> Above statement was mentioned in the KIP about how key value is used. I
> guess the topic is not configured to be compacted and you do not want to
> have partitioning based on that key. IMHO, it qualifies more as a header
> than a key. What do you think about building records with a specific header
> and consumers to execute the logic whether to process or ignore the
> messages based on that header value.
>
> Thanks,
> Satish.
>
>
> On Fri, Aug 31, 2018 at 12:02 AM, M. Manna  wrote:
>
>> Hi Harsha,
>>
>> thanks for reading the KIP.
>>
>> The intent is to use the DefaultPartitioner logic for round-robin
>> selection
>> of partition regardless of key type.
>>
>> Implementing Partitioner interface isn’t the issue here, you would have to
>> do that anyway if  you are implementing your own. But we also want this to
>> be part of formal codebase.
>>
>> Regards,
>>
>> On Thu, 30 Aug 2018 at 16:58, Harsha  wrote:
>>
>> > Hi,
>> >   Thanks for the KIP. I am trying to understand the intent of the
>> > KIP.  Is the use case you specified can't be achieved by implementing
>> the
>> > Partitioner interface here?
>> > https://github.com/apache/kafka/blob/trunk/clients/src/main/
>> java/org/apache/kafka/clients/producer/Partitioner.java#L28
>> > .
>> > Use your custom partitioner to be configured in your producer clients.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
>> > > Hello,
>> > >
>> > > I opened a very simple KIP and there exists a JIRA for it.
>> > >
>> > > I would be grateful if any comments are available for action.
>> > >
>> > > Regards,
>> >
>>
>
>

Re: [VOTE] KIP-359: Verify leader epoch in produce requests

[Satish Duggana]

Thanks for the KIP,
+1 (non-binding)

On Sun, Sep 2, 2018 at 7:59 PM, Matthias J. Sax 
wrote:

> +1 (binding)
>
> On 8/30/18 11:30 PM, Dongjin Lee wrote:
> > Thanks for the KIP. I'm +1 (non-binding).
> >
> > Best,
> > Dongjin
> >
> > On Fri, Aug 31, 2018 at 9:25 AM Dong Lin  wrote:
> >
> >> Thanks for the KIP!
> >>
> >> +1 (binding)
> >>
> >> On Thu, Aug 30, 2018 at 3:56 PM, Jason Gustafson 
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I'd like to start the vote on KIP-359: https://cwiki.apache.org/
> >>> confluence/display/KAFKA/KIP-359%3A+Verify+leader+epoch+in+
> >>> produce+requests.
> >>> Thanks in advance for reviewing.
> >>>
> >>> -Jason
> >>>
> >>
> >
> >
>
>

Re: [VOTE] KIP-371: Add a configuration to build custom SSL principal name

[Satish Duggana]

+1 (non binding)

Thanks,
Satish.

On Fri, Sep 21, 2018 at 3:26 PM, Rajini Sivaram  wrote:
> Hi Manikumar,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> On Thu, Sep 20, 2018 at 8:53 PM, Priyank Shah  wrote:
>
>> +1(non-binding)
>>
>> On 9/20/18, 9:18 AM, "Harsha Chintalapani"  wrote:
>>
>> +1 (binding).
>>
>> Thanks,
>> Harsha
>>
>>
>> On September 19, 2018 at 5:19:51 AM, Manikumar (
>> manikumar.re...@gmail.com) wrote:
>>
>> Hi All,
>>
>> I would like to start voting on KIP-371, which adds a configuration
>> option
>> for building custom SSL principal names.
>>
>> KIP:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 371%3A+Add+a+configuration+to+build+custom+SSL+principal+name
>>
>> Discussion Thread:
>> https://lists.apache.org/thread.html/e346f5e3e3dd1feb863594e40eac1e
>> d54138613a667f319b99344710@%3Cdev.kafka.apache.org%3E
>>
>> Thanks,
>> Manikumar
>>
>>
>>

Re: [ANNOUNCE] New committer: Colin McCabe

[Satish Duggana]

Congratulations Colin!



On Wed, Sep 26, 2018 at 5:52 AM, Vahid Hashemian
 wrote:
> Congratulations Colin!
>
> Regards.
> --Vahid
>
> On Tue, Sep 25, 2018 at 3:43 PM Colin McCabe  wrote:
>
>> Thanks, everyone!
>>
>> best,
>> Colin
>>
>>
>> On Tue, Sep 25, 2018, at 15:26, Robert Barrett wrote:
>> > Congratulations Colin!
>> >
>> > On Tue, Sep 25, 2018 at 1:51 PM Matthias J. Sax 
>> > wrote:
>> >
>> > > Congrats Colin! The was over due for some time :)
>> > >
>> > > -Matthias
>> > >
>> > > On 9/25/18 1:51 AM, Edoardo Comar wrote:
>> > > > Congratulations Colin !
>> > > > --
>> > > >
>> > > > Edoardo Comar
>> > > >
>> > > > IBM Event Streams
>> > > > IBM UK Ltd, Hursley Park, SO21 2JN
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > From:   Ismael Juma 
>> > > > To: Kafka Users , dev <
>> dev@kafka.apache.org>
>> > > > Date:   25/09/2018 09:40
>> > > > Subject:[ANNOUNCE] New committer: Colin McCabe
>> > > >
>> > > >
>> > > >
>> > > > Hi all,
>> > > >
>> > > > The PMC for Apache Kafka has invited Colin McCabe as a committer and
>> we
>> > > > are
>> > > > pleased to announce that he has accepted!
>> > > >
>> > > > Colin has contributed 101 commits and 8 KIPs including significant
>> > > > improvements to replication, clients, code quality and testing. A few
>> > > > highlights were KIP-97 (Improved Clients Compatibility Policy),
>> KIP-117
>> > > > (AdminClient), KIP-227 (Incremental FetchRequests to Increase
>> Partition
>> > > > Scalability), the introduction of findBugs and adding Trogdor (fault
>> > > > injection and benchmarking tool).
>> > > >
>> > > > In addition, Colin has reviewed 38 pull requests and participated in
>> more
>> > > > than 50 KIP discussions.
>> > > >
>> > > > Thank you for your contributions Colin! Looking forward to many
>> more. :)
>> > > >
>> > > > Ismael, for the Apache Kafka PMC
>> > > >
>> > > >
>> > > >
>> > > > Unless stated otherwise above:
>> > > > IBM United Kingdom Limited - Registered in England and Wales with
>> number
>> > > > 741598.
>> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
>> PO6
>> > > 3AU
>> > > >
>> > >
>> > >
>>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Satish Duggana]

Thanks for the KIP! +1 (non-binding)

On Tue, Sep 25, 2018 at 1:38 PM, Ismael Juma  wrote:
> Thanks, +1 (binding).
>
> Ismael
>
> On Thu, Sep 20, 2018 at 8:12 PM Chia-Ping Tsai  wrote:
>
>> KIP-336[1] has been merged so it is time to activate this thread
>> (KIP-331[2]). Last discussion is about "Should we add FunctionalInterface
>> annotation to Serializer and Deserializer". In discussion of KIP-336 we
>> mentioned that we probably add the default implementation for headless
>> method later. Hence, adding FunctionalInterface annotation is not suitable
>> now.
>>
>> KIP-331 has removed the change of adding FunctionalInterface annotation.
>> Please take a look again.
>>
>> [1]
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
>> [2]
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>>
>> Cheers,
>> Chia-Ping
>>
>>
>> On 2018/07/20 10:56:59, Ismael Juma  wrote:
>> > Part of the motivation for this KIP is to make these interfaces
>> functional
>> > interfaces. But I think that may not be desirable due to the method that
>> > passes headers. So, it doesn't make sense to discuss two separate changes
>> > to the same interfaces in isolation, we should figure out how we want
>> them
>> > to work holistically.
>> >
>> > Ismael
>> >
>> > On Fri, Jul 20, 2018 at 3:50 AM Chia-Ping Tsai 
>> wrote:
>> >
>> > > > The KIP needs 3 binding votes to pass.
>> > >
>> > > Thanks for the reminder. I will reopen the ballot box until we get 3
>> > > tickets.
>> > >
>> > > > I still think we should include the details of how things will look
>> like
>> > > > with the headers being passed to serializers/deserializers to ensure
>> > > > things actually make sense as a whole.
>> > >
>> > > This KIP is unrelated to the both methods - serialize() and
>> deserialize().
>> > > We won't add the default implementation to them in this kip. Please
>> correct
>> > > me if I didn't catch what you said.
>> > >
>> > > Cheers,
>> > > Chia-Ping
>> > >
>> > > On 2018/07/09 01:55:41, Ismael Juma  wrote:
>> > > > The KIP needs 3 binding votes to pass. I still think we should
>> include
>> > > the
>> > > > details of how things will look like with the headers being passed to
>> > > > serializers/deserializers to ensure things actually make sense as a
>> > > whole.
>> > > >
>> > > > Ismael
>> > > >
>> > > >
>> > > > On Sun, 8 Jul 2018, 18:31 Chia-Ping Tsai, 
>> wrote:
>> > > >
>> > > > > All,
>> > > > >
>> > > > > The 72 hours has passed. The vote result of KIP-313 is shown below.
>> > > > >
>> > > > > 1 binding vote (Matthias J. Sax)
>> > > > > 4 non-binding votes (John Roesler, Richard Yu, vito jeng and
>> Chia-Ping)
>> > > > >
>> > > > > Cheers,
>> > > > > Chia-Ping
>> > > > >
>> > > > > On 2018/07/05 14:45:01, Chia-Ping Tsai 
>> wrote:
>> > > > > > hi all,
>> > > > > >
>> > > > > > I would like to start voting on "KIP-331 Add default
>> implementation
>> > > to
>> > > > > close() and configure() for Serializer, Deserializer and Serde"
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Chia-Ping
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [DISCUSS] KIP-209 Connection String Support

[Satish Duggana]

 You may need to update KIP with the details discussed in this thread in
proposed changes section.

>>My proposed format for the connection string would be:
>>IP1:host1,IP2:host2,...IPN:hostn;parameterName=value1;parameterName2=value2;...
parameterNameN=valueN
Format should be
host1:port1,host2:port2,…host:portn;param-name1=param-val1,..

>>Invalid conversions would throw InvalidArgumentException (with a
description of the invalid conversion)
>>Invalid parameters would throw InvalidArgumentException (with the name of
the invalid parameter).

Should throw IllegalArgumentException with respective message.

Thanks,
Satish.

On Tue, Oct 17, 2017 at 4:46 AM, Clebert Suconic 
wrote:

> That works.
>
> On Mon, Oct 16, 2017 at 6:59 PM Ted Yu  wrote:
>
> > Can't you use IllegalArgumentException ?
> >
> > Some example in current code base:
> >
> > clients/src/main/java/org/apache/kafka/clients/Metadata.java:
> >  throw new IllegalArgumentException("Max time to wait for metadata
> updates
> > should not be < 0 milliseconds");
> >
> > On Mon, Oct 16, 2017 at 3:06 PM, Clebert Suconic <
> > clebert.suco...@gmail.com>
> > wrote:
> >
> > > I updated the wiki with the list on the proposed arguments.
> > >
> > > I also changed it to include a new Exception class that would be named
> > > InvalidParameterException (since I couldn't find an existing Exception
> > > class that I could reuse into this). (I could review the name or the
> > > exception of course.. just my current proposal)
> > >
> > > On Mon, Oct 16, 2017 at 5:55 PM, Jakub Scholz  wrote:
> > > > Hi Clebert,
> > > >
> > > > I think it would be good if this could cover not only KafkaConsumer
> and
> > > > KafkaProducer but also the AdminClient. So that all three can be
> > > configured
> > > > the same way.
> > > >
> > > > The bootstrap servers are a list - you can provide multiple bootstrap
> > > > servers. Maybe you add an example of how that will be configured. I
> > > assume
> > > > it will be
> > > > "host:port,host2:port2;parameterName=value1;parameterName2=value2"
> but
> > > it
> > > > would be great to have it documented.
> > > >
> > > > Thanks & Regards
> > > > Jakub
> > > >
> > > > On Mon, Oct 16, 2017 at 11:30 PM, Clebert Suconic <
> > > clebert.suco...@gmail.com
> > > >> wrote:
> > > >
> > > >> I would like to start a discussion about KIP-209
> > > >> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> 209+-+Connection+String+Support)
> > > >>
> > > >> This is an extension of my previous thread:
> > > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
> > > >> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
> > > >> oyv...@mail.gmail.com%3e
> > > >>
> > > >> this could make the bootstrap of a consumer or producer similar to
> > > >> what users are already used when connecting into other systems,
> being
> > > >> a simple addition to Producer and Consumer, without breaking any
> > > >> previous client usage.
> > > >>
> > > >>
> > > >> --
> > > >> Clebert Suconic
> > > >>
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> >
> --
> Clebert Suconic
>

Re: [VOTE] KIP-207:The Offsets which ListOffsetsResponse returns should monotonically increase even during a partition leader change

[Satish Duggana]

+1 (non binding)

Thanks,
Satish.

On Wed, Oct 18, 2017 at 1:56 AM, Jun Rao  wrote:

> Hi, Colin,
>
> Thanks for the KIP. +1. Just a minor comment. For the old client requests,
> would it be better to return a LEADER_NOT_AVAILABLE error instead?
>
> Jun
>
> On Tue, Oct 17, 2017 at 11:11 AM, Colin McCabe  wrote:
>
> > Hi all,
> >
> > I'd like to start the voting process for KIP-207:The  Offsets which
> > ListOffsetsResponse returns should monotonically increase even during a
> > partition leader change.
> >
> > See
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+
> > monotonically+increasing+even+during+a+partition+leader+change
> > for details.
> >
> > The voting process will run for at least 72 hours.
> >
> > regards,
> > Colin
> >
>

Re: [VOTE] 1.0.0 RC4

[Satish Duggana]

+1 (non-binding)
Verified signatures, ran tests on src dist.

Thanks,
Satish.


On Wed, Nov 1, 2017 at 12:37 AM, Jeff Chao  wrote:

> +1 (non-binding). We ran our usual performance and regression suite and
> found no noticeable negative impacts.
>
> - Jeff
> Heroku
>
> On Tue, Oct 31, 2017 at 8:54 AM, Ted Yu  wrote:
>
> > +1 (non-binding)
> >
> > Verified signatures.
> > Ran test suite.
> >
> > On Tue, Oct 31, 2017 at 8:53 AM, Manikumar 
> > wrote:
> >
> > > +1 (non-binding). Verified quickstart, ran producer/consumer perf
> > scripts,
> > > streams quickstart
> > > ran tests on src distribution.
> > >
> > > On Tue, Oct 31, 2017 at 8:42 PM, Ismael Juma 
> wrote:
> > >
> > > > +1 (binding) from me. Tested the quickstart with the source and
> binary
> > > > (Scala 2.12) artifacts, ran the tests on the source artifact and
> > verified
> > > > some signatures and hashes on source and binary (Scala 2.11)
> artifacts.
> > > >
> > > > Thanks for running the release, Guozhang!
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Oct 27, 2017 at 6:28 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the fifth candidate for release of Apache Kafka 1.0.0. The
> > main
> > > > PRs
> > > > > that gets merged in after RC3 are the following:
> > > > >
> > > > > *https://github.com/apache/kafka/commit/def1a768a6301c14ad66
> > 11358716ab
> > > > > 03de04e76b
> > > > >  > 11358716ab
> > > > > 03de04e76b>*
> > > > >
> > > > > *https://github.com/apache/kafka/commit/b9fc0f2e6892062efa1f
> > ff0c6f7bfc
> > > > > 683c8ba7ab
> > > > >  > ff0c6f7bfc
> > > > > 683c8ba7ab>*
> > > > >
> > > > > *https://github.com/apache/kafka/commit/a51fdcd2ee7efbd14857
> > 448a2fb7ec
> > > > > b71531e1f9
> > > > >  > 448a2fb7ec
> > > > > b71531e1f9>*
> > > > >
> > > > > *https://github.com/apache/kafka/commit/109a60c77a56d4afed48
> > 8c3ba35dc8
> > > > > 459fde15ce
> > > > >  > 8c3ba35dc8
> > > > > 459fde15ce>*
> > > > >
> > > > > It's worth noting that starting in this version we are using a
> > > different
> > > > > version protocol with three digits: *major.minor.bug-fix*
> > > > >
> > > > > Any and all testing is welcome, but the following areas are worth
> > > > > highlighting:
> > > > >
> > > > > 1. Client developers should verify that their clients can
> > > produce/consume
> > > > > to/from 1.0.0 brokers (ideally with compressed and uncompressed
> > data).
> > > > > 2. Performance and stress testing. Heroku and LinkedIn have helped
> > with
> > > > > this in the past (and issues have been found and fixed).
> > > > > 3. End users can verify that their apps work correctly with the new
> > > > > release.
> > > > >
> > > > > This is a major version release of Apache Kafka. It includes 29 new
> > > KIPs.
> > > > > See the release notes and release plan
> > > > > (*https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=71764913
> > > > >  > > > pageId=71764913
> > > > > >*)
> > > > > for more details. A few feature highlights:
> > > > >
> > > > > * Java 9 support with significantly faster TLS and CRC32C
> > > implementations
> > > > > * JBOD improvements: disk failure only disables failed disk but not
> > the
> > > > > broker (KIP-112/KIP-113 part I)
> > > > > * Controller improvements: reduced logging change to greatly
> > accelerate
> > > > > admin request handling.
> > > > > * Newly added metrics across all the modules (KIP-164, KIP-168,
> > > KIP-187,
> > > > > KIP-188, KIP-196)
> > > > > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 /
> > > 161),
> > > > > and drop compatibility "Evolving" annotations
> > > > >
> > > > > Release notes for the 1.0.0 release:
> > > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/
> RELEASE_NOTES.html
> > > > >  RELEASE_NOTES.html
> > >*
> > > > >
> > > > >
> > > > >
> > > > > *** Please download, test and vote by Tuesday, October 31, 8pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/
> > > > > *
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/org/apa
> > che/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > *http://home.apache.org/~guozhang/kafka-1.0.0-rc4/javadoc/
> > > > > *
> > > > >
> > > > > * Tag to be voted upon (off 1.0 branch) is the 1

Re: [VOTE] 0.11.0.2 RC0

[Satish Duggana]

+1 (non-binding)

- Ran testAll on source
- Verified signatures on binaries built with Scala-2.11
- Ran through quickstart of core/streams



On Wed, Nov 15, 2017 at 12:46 AM, Gwen Shapira  wrote:

> +1 (binding)
>
> Validated signatures, compiled sources, ran through quickstart on my
> machine.
>
> On Mon, Nov 13, 2017 at 7:53 PM Ted Yu  wrote:
>
> > Understood.
> >
> > The test passed when run standalone.
> >
> > +1 from me.
> >
> > On Mon, Nov 13, 2017 at 1:19 AM, Rajini Sivaram  >
> > wrote:
> >
> > > Hi Ted,
> > >
> > > Thank you for running the tests. As you have mentioned in KAFKA-4262,
> > this
> > > has been a known intermittent test failure. It has been difficult to
> make
> > > integration tests for quotas reliable since they verify the time taken
> to
> > > complete after running for a short duration. We should make the tests
> > > reliable, but this shouldn't be a blocker for the release.
> > >
> > >
> > > On Sat, Nov 11, 2017 at 5:29 PM, Ted Yu  wrote:
> > >
> > > > Running test suite I saw:
> > > >
> > > > kafka.admin.ReassignPartitionsClusterTest >
> > > > shouldExecuteThrottledReassignment FAILED
> > > > java.lang.AssertionError: Expected replication to be < 1 but
> > was
> > > > 10138
> > > > at org.junit.Assert.fail(Assert.java:88)
> > > > at org.junit.Assert.assertTrue(Assert.java:41)
> > > > at
> > > > kafka.admin.ReassignPartitionsClusterTest.shouldExecuteThrot
> > > > tledReassignment(ReassignPartitionsClusterTest.scala:183)
> > > >
> > > > Has anyone else seen the above ?
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Nov 10, 2017 at 4:37 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 0.11.0.2.
> > > > >
> > > > >
> > > > > This is a bug fix release and it includes fixes and improvements
> from
> > > 16
> > > > > JIRAs,
> > > > > including a few critical bugs.
> > > > >
> > > > >
> > > > > Release notes for the 0.11.0.2 release:
> > > > >
> > > > >
> > http://home.apache.org/~rsivaram/kafka-0.11.0.2-rc0/RELEASE_NOTES.html
> > > > >
> > > > >
> > > > > *** Please download, test and vote by Wednesday the 15th of
> November,
> > > 8PM
> > > > > PT
> > > > >
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > >
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > >
> > > > > http://home.apache.org/~rsivaram/kafka-0.11.0.2-rc0/
> > > > >
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > >
> > > > > * Javadoc:
> > > > >
> > > > > http://home.apache.org/~rsivaram/kafka-0.11.0.2-rc0/javadoc/
> > > > >
> > > > >
> > > > > * Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.2 tag:
> > > > >
> > > > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > > > > 25639822d6e23803c599cba35ad3dc1a2817b404
> > > > >
> > > > >
> > > > >
> > > > > * Documentation:
> > > > >
> > > > > Note the documentation can't be pushed live due to changes that
> will
> > > > > not go live
> > > > > until the release. You can manually verify by downloading
> > > > >
> > > > > http://home.apache.org/~rsivaram/kafka-0.11.0.2-rc0/
> > > > > kafka_2.11-0.11.0.2-site-docs.tgz
> > > > >
> > > > >
> > > > >
> > > > > * Protocol:
> > > > >
> > > > > http://kafka.apache.org/0110/protocol.html
> > > > >
> > > > >
> > > > > * Successful Jenkins builds for the 0.11.0 branch:
> > > > >
> > > > > Unit/integration tests: https://builds.apache.org/job/
> > > > > kafka-0.11.0-jdk7/333/
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> >
>

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

[Satish Duggana]

Thanks for the KIP, +1 from me.

On Wed, Dec 27, 2017 at 7:42 AM, Bill Bejeck  wrote:

> Thanks for the KIP.  +1 for me.
>
> On Tue, Dec 26, 2017 at 6:22 PM Ted Yu  wrote:
>
> > +1 from me as well.
> >
> > On Tue, Dec 26, 2017 at 10:41 AM, Matthias J. Sax  >
> > wrote:
> >
> > > Thanks for the KIP Boyang!
> > >
> > > I don't have any further comments.
> > >
> > > +1 from me.
> > >
> > > @Ted: This is a rather simple KIP, thus, skipping the DISCUSS thread
> > > seems ok to me.
> > >
> > >
> > >
> > > -Matthias
> > >
> > >
> > > @Boyang: it's recommended to use this format for the subject
> > >
> > > "[VOTE] KIP-245: ..."
> > >
> > > Same for DISCUSS threads. People are used to those headlines and they
> > > pay more attention than. For this KIP, just leave it as it though. For
> > > future reference only
> > > .
> > >
> > >
> > > On 12/26/17 4:55 AM, Ted Yu wrote:
> > > > Normally a DISCUSS thread precedes VOTE thread so that people have
> > ample
> > > time examining the proposal.
> > > >  Original message From: Boyang Chen <
> > bche...@outlook.com>
> > > Date: 12/26/17  1:22 AM  (GMT-07:00) To: dev@kafka.apache.org Subject:
> > > Vote for KIP-245: Use Properties instead of StreamsConfig in
> KafkaStreams
> > > constructor
> > > > Hi there,
> > > >
> > > > I'm Boyang who is a newbie contributor to Kafka. I would like to
> start
> > a
> > > vote for the KIP-245:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > 245%3A+Use+Properties+instead+of+StreamsConfig+in+
> KafkaStreams+constructor
> > > >
> > > >
> > > > This is linked with JIRA: https://issues.apache.org/
> > > jira/browse/KAFKA-6386
> > > >
> > > > [KAFKA-6386] Deprecate KafkaStreams constructor taking ...<
> > > https://issues.apache.org/jira/browse/KAFKA-6386>
> > > > issues.apache.org
> > > > Currently, KafkaStreams constructor has overloads that take either
> > > Properties or StreamsConfig a parameters. Because StreamsConfig is
> > > immutable and is created from a ...
> > > >
> > > > And my pull request is here:
> > > >
> > > >
> > > > https://github.com/apache/kafka/pull/4354
> > > >
> > > >
> > > > Since this is my first time doing this, feel free to let me know if
> > this
> > > is the correct format!
> > > >
> > > >
> > > > Best,
> > > >
> > > > Boyang
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Thanks, Harsha for the KIP. It is a good start for tiered storage in
Kafka. I have a few comments/questions.

It may be good to have a configuration to keep the number of local
segments instead of keeping only the active segment. This config can
be exposed at cluster and topic levels with default value as 1. In
some use cases, few consumers may lag over one segment, it will be
better to serve from local storage instead of remote storage.

It may be better to keep “remote.log.storage.enable” and respective
configuration at topic level along with cluster level. It will be
helpful in environments where few topics are configured with
local-storage and other topics are configured with remote storage.

Each topic-partition leader pushes its log segments with respective
index files to remote whenever active log rolls over, it updates the
remote log index file for the respective remote log segment. The
second option is to add offset index files also for each segment. It
can serve consumer fetch requests for old segments from local log
segment instead of serving directly from the remote log which may
cause high latencies. There can be different strategies in when the
remote segment is copied to a local segment.

What is “remote.log.manager.scheduler.interval.ms” config about?

How do followers sync RemoteLogSegmentIndex files? Do they request
from leader replica? This looks to be important as the failed over
leader should have RemoteLogSegmentIndex updated and ready to avoid
high latencies in serving old data stored in remote logs.

Thanks,
Satish.

On Tue, Feb 5, 2019 at 10:42 PM Ryanne Dolan  wrote:
>
> Thanks Harsha, makes sense.
>
> Ryanne
>
> On Mon, Feb 4, 2019 at 5:53 PM Harsha Chintalapani  wrote:
>
> > "I think you are saying that this enables additional (potentially cheaper)
> > storage options without *requiring* an existing ETL pipeline. “
> > Yes.
> >
> > " But it's not really a replacement for the sort of pipelines people build
> > with Connect, Gobblin etc.”
> >
> > It is not. But also making an assumption that everyone runs these
> > pipelines for storing raw Kafka data into HDFS or S3 is also wrong
> >  assumption.
> > The aim of this KIP is to provide tiered storage as whole package not
> > asking users to ship the data on their own using existing ETL, which means
> > running a consumer and maintaining those pipelines.
> >
> > " My point was that, if you are already offloading records in an ETL
> > pipeline, why do you need a new pipeline built into the broker to ship the
> > same data to the same place?”
> >
> > As you said its ETL pipeline, which means users of these pipelines are
> > reading the data from broker and transforming its state and storing it
> > somewhere.
> > The point of this KIP is store log segments as it is without changing
> > their structure so that we can use the existing offset mechanisms to look
> > it up when the consumer needs to read old data. When you do load it via
> > your existing pipelines you are reading the topic as a whole , which
> > doesn’t guarantee that you’ll produce this data back into HDFS in S3 in the
> > same order and who is going to generate the Index files again.
> >
> >
> > "So you'd end up with one of 1)cold segments are only useful to Kafka; 2)
> > you have the same data written to HDFS/etc twice, once for Kafka and once
> > for everything else, in two separate formats”
> >
> > You are talking two different use cases. If someone is storing raw data
> > out of Kafka for long term access.
> > By storing the data as it is in HDFS though Kafka will solve this issue.
> > They do not need to run another pipe-line to ship these logs.
> >
> > If they are running pipelines to store in HDFS in a different format,
> > thats a different use case. May be they are transforming Kafka logs to ORC
> > so that they can query through Hive.  Once you transform the log segment it
> > does loose its ability to use the existing offset index.
> > Main objective here not to change the existing protocol and still be able
> > to write and read logs from remote storage.
> >
> >
> > -Harsha
> >
> > On Feb 4, 2019, 2:53 PM -0800, Ryanne Dolan ,
> > wrote:
> > > Thanks Harsha, makes sense for the most part.
> > >
> > > > tiered storage is to get away from this and make this transparent to
> > the
> > > user
> > >
> > > I think you are saying that this enables additional (potentially cheaper)
> > > storage options without *requiring* an existing ETL pipeline. But it's
> > not
> > > really a replacement for the sort of pipelines people build with Connect,
> > > Gobblin etc. My point was that, if you are already offloading records in
> > an
> > > ETL pipeline, why do you need a new pipeline built into the broker to
> > ship
> > > the same data to the same place? I think in most cases this will be an
> > > additional pipeline, not a replacement, because the segments written to
> > > cold storage won't be useful outside Kafka. So you'd end up with one of
> > 1)
> > > cold segments a

Re: [VOTE] 2.1.1 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll  successfully with NO failures.
- Ran through quickstart of core/streams on builds generated from tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks for running the release Colin!

Satish.

On Mon, Feb 11, 2019 at 2:29 PM Mickael Maison  wrote:
>
> +1 (non binding)
> I built from source and ran Quickstart. All good. Thanks Colin
>
>
> On Mon, Feb 11, 2019 at 12:43 AM Vahid Hashemian
>  wrote:
> >
> > +1
> >
> > I built from the source, and ran the Quickstart on Ubuntu 18.04 with Java 8.
> > Some unit tests fail for me but none consistently across several runs.
> >
> > Thanks for running the release Colin.
> >
> > --Vahid
> >
> > On Sat, Feb 9, 2019 at 2:24 PM Jakub Scholz  wrote:
> >
> > > +1 (non-binding). I built it from source and run my tests. Everything 
> > > seems
> > > to be fine.
> > >
> > > On Sat, Feb 9, 2019 at 12:10 AM Magnus Edenhill 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Passes librdkafka test suite.
> > > >
> > > > Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe :
> > > >
> > > > > Hi all,
> > > > >
> > > > > This is the third candidate for release of Apache Kafka 2.1.1.  This
> > > > > release includes many bug fixes for Apache Kafka 2.1.
> > > > >
> > > > > Compared to rc1, this release includes the following changes:
> > > > > * MINOR: release.py: fix some compatibility problems.
> > > > > * KAFKA-7897; Disable leader epoch cache when older message formats 
> > > > > are
> > > > > used
> > > > > * KAFKA-7902: Replace original loginContext if SASL/OAUTHBEARER 
> > > > > refresh
> > > > > login fails
> > > > > * MINOR: Fix more places where the version should be bumped from 2.1.0
> > > ->
> > > > > 2.1.1
> > > > > * KAFKA-7890: Invalidate ClusterConnectionState cache for a broker if
> > > the
> > > > > hostname of the broker changes.
> > > > > * KAFKA-7873; Always seek to beginning in KafkaBasedLog
> > > > > * MINOR: Correctly set dev version in version.py
> > > > >
> > > > > Check out the release notes here:
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/RELEASE_NOTES.html
> > > > >
> > > > > The vote will go until Wednesday, February 13st.
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~cmccabe/kafka-2.1.1-rc2/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 2.1 branch) is the 2.1.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/2.1.1-rc2
> > > > >
> > > > > * Jenkins builds for the 2.1 branch:
> > > > > Unit/integration tests: https://builds.apache.org/job/kafka-2.1-jdk8/
> > > > >
> > > > > Thanks to everyone who tested the earlier RCs.
> > > > >
> > > > > cheers,
> > > > > Colin
> > > > >
> > > >
> > >

Re: [DISCUSS] KIP-430 - Return Authorized Operations in Describe Responses

[Satish Duggana]

Hi Rajini,
Thanks for the KIP.
KIP proposes to add a new field called `authorized_operations` which
is an array of Byte values. I guess these are APIKeys#id for
respective operations. Do you plan to have an array of ids or an array
of respective ApiKeys enum values in
MetadataResponse/DescribGroupsResponse classes?

Thanks,
Satish.

On Wed, Feb 13, 2019 at 12:33 AM Rajini Sivaram  wrote:
>
> Hi all,
>
> I have created a KIP to optionally request authorised operations on
> resources when describing resources:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-430+-+Return+Authorized+Operations+in+Describe+Responses
>
> This includes only information that users with Describe access can obtain
> using other means and hence is consistent with our security model. It is
> intended to made it easier for clients to obtain this information.
>
> Feedback and suggestions welcome.
>
> Thank you,
>
> Rajini

Re: [DISCUSS] KIP-430 - Return Authorized Operations in Describe Responses

[Satish Duggana]

Hi Rajini,
That makes sense, thanks for the clarification.

Satish.

On Wed, Feb 13, 2019 at 7:30 PM Rajini Sivaram  wrote:
>
> Thanks for the reviews!
>
> Hi Satish,
>
> The authorised operations returned will use the same values as the
> operations returned by the existing DescribeAclsResponse. AdminClient will
> return these using the existing enum AclOperation.
>
> Hi Magnus,
>
> The MetadataResponse contains these two lines:
>
>- Metadata Response => throttle_time_ms [brokers] cluster_id
>controller_id [topic_metadata] [authorized_operations] <== ADDED
>authorized_operations
>- topic_metadata => error_code topic is_internal [partition_metadata]
>[authorized_operations]  <== ADDED authorized_operations
>
> The first is for the cluster's authorized operations and the second for
> each topic. Did I misunderstand your question? The full set of operations
> for each resource type is included in the subsection `AdminClient API
> Changes`.
>
> Under `Rejected Alternatives` I have included addition of a separate
> request to get this information rather than extend an existing one. The
> rationale for including all the information in one request is to enable
> clients to get all relevant metadata using a single API rather than have to
> send multiple requests, get responses and combine the two while resource or
> ACLs may have changed in between. It seems neater to use a single API since
> a user getting authorized operations is almost definitely going to do a
> Describe first and access control for both of these is controlled using
> Describe access. If we add new resource types with a corresponding
> Describe, we would simply need to add `authorized_operations` for their
> Describe.
>
> Hi Manikumar,
>
> Added IdempotentWrite for Cluster, thanks for pointing that out! I was
> thinking that if authorizer is not configured, we could return all
> supported operations since the user can perform all operations. Added a
> note to the KIP.
>
> Regards,
>
> Rajini
>
>
>
> On Wed, Feb 13, 2019 at 11:07 AM Manikumar 
> wrote:
>
> > Hi,
> >
> > Thanks for the KIP.
> >
> > 1. Can't we include IdempotentWrite/ClusterResource Operations for Cluster
> > resource.
> > 2. What will be the API behaviour when the authorizer is not configured?. I
> > assume we return empty list.
> >
> > Thanks,
> > Manikumar
> >
> > On Wed, Feb 13, 2019 at 12:33 AM Rajini Sivaram 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created a KIP to optionally request authorised operations on
> > > resources when describing resources:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-430+-+Return+Authorized+Operations+in+Describe+Responses
> > >
> > > This includes only information that users with Describe access can obtain
> > > using other means and hence is consistent with our security model. It is
> > > intended to made it easier for clients to obtain this information.
> > >
> > > Feedback and suggestions welcome.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> >

Re: [ANNOUNCE] New Committer: Bill Bejeck

[Satish Duggana]

Congratulations Bill!

On Thu, Feb 14, 2019 at 6:41 AM Marcelo Barbosa
 wrote:
>
> Wow! Congrats Bill!
> Cheers,
> Barbosa
> Em quarta-feira, 13 de fevereiro de 2019 23:03:54 BRST, Guozhang Wang 
>  escreveu:
>
>  Hello all,
>
> The PMC of Apache Kafka is happy to announce that we've added Bill Bejeck
> as our newest project committer.
>
> Bill has been active in the Kafka community since 2015. He has made
> significant contributions to the Kafka Streams project with more than 100
> PRs and 4 authored KIPs, including the streams topology optimization
> framework. Bill's also very keen on tightening Kafka's unit test / system
> tests coverage, which is a great value to our project codebase.
>
> In addition, Bill has been very active in evangelizing Kafka for stream
> processing in the community. He has given several Kafka meetup talks in the
> past year, including a presentation at Kafka Summit SF. He's also authored
> a book about Kafka Streams (
> https://www.manning.com/books/kafka-streams-in-action), as well as various
> of posts in public venues like DZone as well as his personal blog (
> http://codingjunkie.net/).
>
> We really appreciate the contributions and are looking forward to see more
> from him. Congratulations, Bill !
>
>
> Guozhang, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Randall Hauch

[Satish Duggana]

Congratulations Randall!

On Fri, Feb 15, 2019 at 1:51 PM Mickael Maison  wrote:
>
> Congrats Randall!
>
> On Fri, Feb 15, 2019 at 6:37 AM James Cheng  wrote:
> >
> > Congrats, Randall! Well deserved!
> >
> > -James
> >
> > Sent from my iPhone
> >
> > > On Feb 14, 2019, at 6:16 PM, Guozhang Wang  wrote:
> > >
> > > Hello all,
> > >
> > > The PMC of Apache Kafka is happy to announce another new committer joining
> > > the project today: we have invited Randall Hauch as a project committer 
> > > and
> > > he has accepted.
> > >
> > > Randall has been participating in the Kafka community for the past 3 
> > > years,
> > > and is well known as the founder of the Debezium project, a popular 
> > > project
> > > for database change-capture streams using Kafka (https://debezium.io). 
> > > More
> > > recently he has become the main person keeping Kafka Connect moving
> > > forward, participated in nearly all KIP discussions and QAs on the mailing
> > > list. He's authored 6 KIPs and authored 50 pull requests and conducted 
> > > over
> > > a hundred reviews around Kafka Connect, and has also been evangelizing
> > > Kafka Connect at several Kafka Summit venues.
> > >
> > >
> > > Thank you very much for your contributions to the Connect community 
> > > Randall
> > > ! And looking forward to many more :)
> > >
> > >
> > > Guozhang, on behalf of the Apache Kafka PMC

Re: [VOTE] KIP-412: Extend Admin API to support dynamic application log levels

[Satish Duggana]

Thanks for the KIP
+1 (non-binding)

On Thu, Feb 21, 2019 at 9:38 AM Harsha  wrote:
>
> +1 (binding).
>
> Thanks,
> Harsha
>
> On Tue, Feb 19, 2019, at 7:53 AM, Andrew Schofield wrote:
> > Thanks for the KIP.
> >
> > +1 (non-binding)
> >
> > On 18/02/2019, 12:48, "Stanislav Kozlovski"  wrote:
> >
> > Hey everybody, I'm starting a VOTE thread for KIP-412. This feature 
> > should
> > significantly improve the flexibility and ease in debugging Kafka in run
> > time
> >
> > KIP-412 -
> >
> > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-412%253A%2BExtend%2BAdmin%2BAPI%2Bto%2Bsupport%2Bdynamic%2Bapplication%2Blog%2Blevels&data=02%7C01%7C%7C69bc63a9d7864e25ec3c08d69596eec4%7C84df9e7fe9f640afb435%7C1%7C0%7C636860872825557120&sdata=XAnMhy6EPC7JkB77NBBhLR%2FvE7XrTutuS5Rlt%2FDpwfU%3D&reserved=0
> >
> >
> > --
> > Best,
> > Stanislav
> >
> >
> >

Re: [VOTE] KIP-430 - Return Authorized Operations in Describe Responses

[Satish Duggana]

Thanks for the KIP, +1 (non-binding)

~Satish.

On Thu, Feb 21, 2019 at 3:58 PM Rajini Sivaram  wrote:
>
> I would like to start vote on KIP-430 to optionally obtain authorized
> operations when describing resources:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-430+-+Return+Authorized+Operations+in+Describe+Responses
>
> Thank you,
>
> Rajini

Re: Release Date : 2.2.0

[Satish Duggana]

KAFKA-7895 is addressed in 2.2.0-rc0. You can look at
https://github.com/apache/kafka/commits/2.2.0-rc0 which has the respective
commit

.

Thanks,
Satish.


On Wed, Feb 27, 2019 at 9:36 AM Vaibhav Shukla 
wrote:

> I am not seeing the below defect
>
> https://issues.apache.org/jira/browse/KAFKA-7895
>
> is this not being a part of 2.2.0?
>
> -Vaibhav-
>
> On Tue, Feb 26, 2019 at 9:19 PM Matthias J. Sax 
> wrote:
>
> > Target release date is 2/28 (cf.
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827512
> > )
> >
> > The first RC was cut on Saturday and the VOTE is running.
> >
> >
> > -Matthias
> >
> > On 2/26/19 7:06 PM, Vaibhav Shukla wrote:
> > > Guys,
> > >
> > > Any idea when we have the release 2.2.0.
> > >
> > > Vaibhav
> > >
> >
> >
>

Re: [VOTE] 2.2.0 RC0

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with NO failures.
- Ran through quickstart of core/streams on builds generated from 2.2.0-rc0
tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks for running the release Matthias!

On Tue, Feb 26, 2019 at 8:17 PM Adam Bellemare 
wrote:

> Downloaded, compiled and passed all tests successfully.
>
> Ran quickstart (https://kafka.apache.org/quickstart) up to step 6 without
> issue.
>
> (+1 non-binding).
>
> Adam
>
>
>
> On Mon, Feb 25, 2019 at 9:19 PM Matthias J. Sax 
> wrote:
>
> > @Stephane
> >
> > Thanks! You are right (I copied the list from an older draft without
> > double checking).
> >
> > On the release Wiki page, it's correctly listed as postponed:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827512
> >
> >
> > @Viktor
> >
> > Thanks. This will not block the release, but I'll make sure to include
> > it in the webpage update.
> >
> >
> >
> > -Matthias
> >
> > On 2/25/19 5:16 AM, Viktor Somogyi-Vass wrote:
> > > Hi Matthias,
> > >
> > > I've noticed a minor line break issue in the upgrade docs. I've
> created a
> > > small PR for that: https://github.com/apache/kafka/pull/6320
> > >
> > > Best,
> > > Viktor
> > >
> > > On Sun, Feb 24, 2019 at 10:16 PM Stephane Maarek <
> > kafka.tutori...@gmail.com>
> > > wrote:
> > >
> > >> Hi Matthias
> > >>
> > >> Thanks for this
> > >> Running through the list of KIPs. I think this is not included in 2.2:
> > >>
> > >> - Allow clients to suppress auto-topic-creation
> > >>
> > >> Regards
> > >> Stephane
> > >>
> > >> On Sun, Feb 24, 2019 at 1:03 AM Matthias J. Sax <
> matth...@confluent.io>
> > >> wrote:
> > >>
> > >>> Hello Kafka users, developers and client-developers,
> > >>>
> > >>> This is the first candidate for the release of Apache Kafka 2.2.0.
> > >>>
> > >>> This is a minor release with the follow highlight:
> > >>>
> > >>>  - Added SSL support for custom principle name
> > >>>  - Allow SASL connections to periodically re-authenticate
> > >>>  - Improved consumer group management
> > >>>- default group.id is `null` instead of empty string
> > >>>  - Add --under-min-isr option to describe topics command
> > >>>  - Allow clients to suppress auto-topic-creation
> > >>>  - API improvement
> > >>>- Producer: introduce close(Duration)
> > >>>- AdminClient: introduce close(Duration)
> > >>>- Kafka Streams: new flatTransform() operator in Streams DSL
> > >>>- KafkaStreams (and other classed) now implement AutoClosable to
> > >>> support try-with-resource
> > >>>- New Serdes and default method implementations
> > >>>  - Kafka Streams exposed internal client.id via ThreadMetadata
> > >>>  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will
> now
> > >>> output `NaN` as default value
> > >>>
> > >>>
> > >>> Release notes for the 2.2.0 release:
> > >>> http://home.apache.org/~mjsax/kafka-2.2.0-rc0/RELEASE_NOTES.html
> > >>>
> > >>> *** Please download, test and vote by Friday, March 1, 9am PST.
> > >>>
> > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>> http://kafka.apache.org/KEYS
> > >>>
> > >>> * Release artifacts to be voted upon (source and binary):
> > >>> http://home.apache.org/~mjsax/kafka-2.2.0-rc0/
> > >>>
> > >>> * Maven artifacts to be voted upon:
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>
> > >>> * Javadoc:
> > >>> http://home.apache.org/~mjsax/kafka-2.2.0-rc0/javadoc/
> > >>>
> > >>> * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag:
> > >>> https://github.com/apache/kafka/releases/tag/2.2.0-rc0
> > >>>
> > >>> * Documentation:
> > >>> https://kafka.apache.org/22/documentation.html
> > >>>
> > >>> * Protocol:
> > >>> https://kafka.apache.org/22/protocol.html
> > >>>
> > >>> * Successful Jenkins builds for the 2.2 branch:
> > >>> Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.2-jdk8/31/
> > >>>
> > >>> * System tests:
> > >>> https://jenkins.confluent.io/job/system-test-kafka/job/2.2/
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> -Matthias
> > >>>
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] KIP-436 Add a metric indicating start time

[Satish Duggana]

Thanks for the KIP,
+1 (non-binding)

~Satish.

On Thu, Mar 7, 2019 at 11:58 PM Manikumar  wrote:

> +1 (binding).
>
> Thanks for the KIP.
>
> Thanks,
> Manikumar
>
>
> On Thu, Mar 7, 2019 at 11:52 PM Colin McCabe  wrote:
>
> > +1 (binding).
> >
> > Thanks, Stanislav.
> >
> > best,
> > Colin
> >
> > On Tue, Mar 5, 2019, at 05:23, Stanislav Kozlovski wrote:
> > > Hey everybody,
> > >
> > > I'd like to start a vote thread about the lightweight KIP-436
> > > KIP: KIP-436
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-436%3A+Add+a+metric+indicating+start+time
> > >
> > > JIRA: KAFKA-7992 
> > > Pull Request: 6318 
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
>

Re: [VOTE] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

[Satish Duggana]

Nice work Konstantine!
+1 (non-binding)

On Fri, Mar 15, 2019 at 7:48 AM Ewen Cheslack-Postava 
wrote:

> +1 (binding)
>
> -Ewen
>
> On Wed, Mar 13, 2019 at 2:04 PM Randall Hauch  wrote:
>
> > Excellent work, Konstantine!
> >
> > +1 (binding)
> >
> > On Mon, Mar 11, 2019 at 8:05 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Thanks Jason!
> > > That makes perfect sense. The change is reflected in the KIP now.
> > > "compatible" will be the default mode for "connect.protocol"
> > >
> > > Cheers,
> > > Konstantine
> > >
> > >
> > > On Mon, Mar 11, 2019 at 4:31 PM Jason Gustafson 
> > > wrote:
> > >
> > > > +1 Thanks for all the work on this. My only minor comment is that
> > > > `connect.protocol` probably should be `compatible` by default. The
> cost
> > > is
> > > > low and it will save upgrade confusion.
> > > >
> > > > Best,
> > > > Jason
> > > >
> > > > On Fri, Mar 8, 2019 at 10:37 AM Robert Yokota 
> > > wrote:
> > > >
> > > > > Thanks for the great KIP Konstantine!
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Robert
> > > > >
> > > > > On Thu, Mar 7, 2019 at 2:56 PM Guozhang Wang 
> > > wrote:
> > > > >
> > > > > > Thanks Konstantine, I've read the updated section on
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect
> > > > > > and it lgtm.
> > > > > >
> > > > > > I'm +1 on the KIP.
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 7, 2019 at 2:35 PM Konstantine Karantasis <
> > > > > > konstant...@confluent.io> wrote:
> > > > > >
> > > > > > > Thanks Guozhang. This is a valid observation regarding the
> > current
> > > > > status
> > > > > > > of the PR.
> > > > > > >
> > > > > > > I updated the KIP to explicitly call out how the downgrade
> > process
> > > > > should
> > > > > > > work in the section Compatibility, Deprecation, and Migration.
> > > > > > >
> > > > > > > Additionally, I reduced the configuration modes for the
> > > > > connect.protocol
> > > > > > to
> > > > > > > only two: eager and compatible.
> > > > > > > That's because there's no way at the moment to select a
> protocol
> > > > based
> > > > > on
> > > > > > > simple majority and not unanimity across at least one option
> for
> > > the
> > > > > > > sub-protocol.
> > > > > > > Therefore there's no way to lock a group of workers in a
> > > > > cooperative-only
> > > > > > > mode at the moment, if we account for accidental joins of
> workers
> > > > > running
> > > > > > > at an older version.
> > > > > > >
> > > > > > > The changes have been reflected in the KIP doc and will be
> > > reflected
> > > > in
> > > > > > the
> > > > > > > PR in a subsequent commit.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Konstantine
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 7, 2019 at 1:17 PM Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Konstantine,
> > > > > > > >
> > > > > > > > Thanks for the updated KIP and the PR as well (which is huge
> > :) I
> > > > > > briefly
> > > > > > > > looked through it as well as the KIP, and I have one minor
> > > comment
> > > > to
> > > > > > add
> > > > > > > > (otherwise I'm binding +1 on it as well) about the backward
> > > > > > > compatibility.
> > > > > > > > I'll use one example to illustrate the issue:
> > > > > > > >
> > > > > > > > 1) Suppose you have workerA and B on newer version and
> > configured
> > > > the
> > > > > > > > connect.protocol as "compatible", they will send both V0/V1
> to
> > > the
> > > > > > leader
> > > > > > > > (say it's workerA) who will choose V1 as the current
> protocol,
> > > this
> > > > > > will
> > > > > > > be
> > > > > > > > sent back to A and B who would remember the current protocol
> > > > version
> > > > > is
> > > > > > > > already V1. So after this rebalance everyone remembers that
> V1
> > > can
> > > > be
> > > > > > > used,
> > > > > > > > which means that upon prepareJoin they will not revoke all
> the
> > > > > assigned
> > > > > > > > tasks.
> > > > > > > >
> > > > > > > > 2) Now let's say a new worker joins but with old version V0
> > > > > > (practically
> > > > > > > > this is rare, but for illustration purposes some common
> > scenarios
> > > > may
> > > > > > > falls
> > > > > > > > into this, e.g. an existing worker being downgraded, which is
> > > > > > essentially
> > > > > > > > as being kicked out of the group, and then rejoined as a new
> > > member
> > > > > on
> > > > > > > the
> > > > > > > > older version), the leader realized that at least one of the
> > > member
> > > > > > does
> > > > > > > > not know V1 and hence would fall back to use version V0 to
> > > perform
> > > > > > > > assignment. V0 algorithm would do eager rebalance which may
> > move
> > > > some
> > > > > > > tasks
> > > > > > > > to the new comer immediately from the existing members, as it
> > > > assumes
> 

Re: [VOTE] 2.2.0 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 2.2.0-rc2
tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks for running the release Matthias!

On Wed, Mar 20, 2019 at 12:43 AM Manikumar 
wrote:

> +1 (non-binding)
>
> - Verified the artifacts, build from src, ran tests
> - Verified the quickstart, ran producer/consumer performance tests.
>
> Thanks for running release!.
>
> Thanks,
> Manikumar
>
> On Wed, Mar 20, 2019 at 12:19 AM David Arthur 
> wrote:
>
> > +1
> >
> > Validated signatures, and ran through quick-start.
> >
> > Thanks!
> >
> > On Mon, Mar 18, 2019 at 4:00 AM Jakub Scholz  wrote:
> >
> > > +1 (non-binding). I used the staged binaries and run some of my tests
> > > against them. All seems to look good to me.
> > >
> > > On Sat, Mar 9, 2019 at 11:56 PM Matthias J. Sax  >
> > > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the third candidate for release of Apache Kafka 2.2.0.
> > > >
> > > >  - Added SSL support for custom principal name
> > > >  - Allow SASL connections to periodically re-authenticate
> > > >  - Command line tool bin/kafka-topics.sh adds AdminClient support
> > > >  - Improved consumer group management
> > > >- default group.id is `null` instead of empty string
> > > >  - API improvement
> > > >- Producer: introduce close(Duration)
> > > >- AdminClient: introduce close(Duration)
> > > >- Kafka Streams: new flatTransform() operator in Streams DSL
> > > >- KafkaStreams (and other classed) now implement AutoClosable to
> > > > support try-with-resource
> > > >- New Serdes and default method implementations
> > > >  - Kafka Streams exposed internal client.id via ThreadMetadata
> > > >  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will
> now
> > > > output `NaN` as default value
> > > > Release notes for the 2.2.0 release:
> > > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test, and vote by Thursday, March 14, 9am PST.
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~mjsax/kafka-2.2.0-rc2/javadoc/
> > > >
> > > > * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/2.2.0-rc2
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/22/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/22/protocol.html
> > > >
> > > > * Jenkins builds for the 2.2 branch:
> > > > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.2-jdk8/
> > > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/2.2/
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > >
> > > > -Matthias
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

[Satish Duggana]

Congrats Harsha!

On Fri, Apr 19, 2019 at 2:58 PM Mickael Maison 
wrote:

> Congratulations Harsha!
>
>
> On Fri, Apr 19, 2019 at 5:49 AM Manikumar 
> wrote:
> >
> > Congrats Harsha!.
> >
> > On Fri, Apr 19, 2019 at 7:43 AM Dong Lin  wrote:
> >
> > > Congratulations Sriharsh!
> > >
> > > On Thu, Apr 18, 2019 at 11:46 AM Jun Rao  wrote:
> > >
> > > > Hi, Everyone,
> > > >
> > > > Sriharsh Chintalapan has been active in the Kafka community since he
> > > became
> > > > a Kafka committer in 2015. I am glad to announce that Harsh is now a
> > > member
> > > > of Kafka PMC.
> > > >
> > > > Congratulations, Harsh!
> > > >
> > > > Jun
> > > >
> > >
>

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

[Satish Duggana]

Congratulations Matthias!

On Fri, Apr 19, 2019 at 2:59 PM Jorge Quilcate 
wrote:

> Congrats Matthias!!
>
> On 4/19/19 11:28 AM, Mickael Maison wrote:
> > Congrats Matthias!
> >
> > On Fri, Apr 19, 2019 at 6:07 AM Vahid Hashemian
> >  wrote:
> >> Congratulations Matthias!
> >>
> >> --Vahid
> >>
> >> On Thu, Apr 18, 2019 at 9:39 PM Manikumar 
> wrote:
> >>
> >>> Congrats Matthias!. well deserved.
> >>>
> >>> On Fri, Apr 19, 2019 at 7:44 AM Dong Lin  wrote:
> >>>
>  Congratulations Matthias!
> 
>  Very well deserved!
> 
>  On Thu, Apr 18, 2019 at 2:35 PM Guozhang Wang 
> >>> wrote:
> > Hello Everyone,
> >
> > I'm glad to announce that Matthias J. Sax is now a member of Kafka
> PMC.
> >
> > Matthias has been a committer since Jan. 2018, and since then he
>  continued
> > to be active in the community and made significant contributions the
> > project.
> >
> >
> > Congratulations to Matthias!
> >
> > -- Guozhang
> >
> >>
> >> --
> >>
> >> Thanks!
> >> --Vahid
>

Re: [VOTE] 2.2.1 RC1

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 2.2.1-rc1
tag
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.

On Sat, Jun 1, 2019 at 5:12 AM Matthias J. Sax 
wrote:

> Thanks for running the release Harsha!
>
> - downloaded source code, built, and run tests locally;
>   required 3 runs to get tests passing
>
> Failing test tracked as:
>  - https://issues.apache.org/jira/browse/KAFKA-8260
>  - https://issues.apache.org/jira/browse/KAFKA-8458
>
>
> - run quickstart with Scala 2.11 binaries
>
> - verified signatures for source and both binaries
>
>
>
> +1 (binding)
>
>
>
> -Matthias
>
>
> On 5/31/19 8:30 AM, Mickael Maison wrote:
> > +1 non-binding
> >
> > We've been running it for a few days in a few clusters, so far no
> > issues. I also ran unit tests and checked signatures.
> > Thanks Vahid for running this release
> >
> > On Fri, May 31, 2019 at 9:01 AM Andrew Schofield
> >  wrote:
> >>
> >> +1 (non-binding)
> >>
> >> Built and ran source and sink connectors.
> >>
> >> Andrew Schofield - IBM
> >>
> >> On 31/05/2019, 08:55, "Viktor Somogyi-Vass" 
> wrote:
> >>
> >> +1 (non-binding)
> >>
> >> 1. Ran unit tests
> >> 2. Ran some basic automatic end-to-end tests over plaintext and SSL
> too
> >> 3. Ran systests sanity checks
> >>
> >> Viktor
> >>
> >> On Thu, May 23, 2019 at 6:04 PM Harsha  wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > 1. Ran unit tests
> >> > 2. System tests
> >> > 3. 3 node cluster with few manual tests.
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Wed, May 22, 2019, at 8:09 PM, Vahid Hashemian wrote:
> >> > > Bumping this thread to get some more votes, especially from
> committers,
> >> > so
> >> > > we can hopefully make a decision on this RC by the end of the
> week.
> >> > >
> >> > > Thanks,
> >> > > --Vahid
> >> > >
> >> > > On Mon, May 13, 2019 at 8:15 PM Vahid Hashemian <
> >> > vahid.hashem...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hello Kafka users, developers and client-developers,
> >> > > >
> >> > > > This is the second candidate for release of Apache Kafka
> 2.2.1.
> >> > > >
> >> > > > Compared to RC0, this release candidate also fixes the
> following
> >> > issues:
> >> > > >
> >> > > >- [KAFKA-6789] - Add retry logic in AdminClient requests
> >> > > >- [KAFKA-8348] - Document of kafkaStreams improvement
> >> > > >- [KAFKA-7633] - Kafka Connect requires permission to
> create
> >> > internal
> >> > > >topics even if they exist
> >> > > >- [KAFKA-8240] - Source.equals() can fail with NPE
> >> > > >- [KAFKA-8335] - Log cleaner skips Transactional mark and
> batch
> >> > > >record, causing unlimited growth of __consumer_offsets
> >> > > >- [KAFKA-8352] - Connect System Tests are failing with 404
> >> > > >
> >> > > > Release notes for the 2.2.1 release:
> >> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~vahid%2Fkafka-2.2.1-rc1%2FRELEASE_NOTES.html&data=02%7C01%7C%7C139a19a5204441d9a17908d6e59d49bf%7C84df9e7fe9f640afb435%7C1%7C0%7C636948861037978009&sdata=PFuHh6XL6d20YKU%2B6V2qZa5qDVL0ET%2BR0QOFjVjqmho%3D&reserved=0
> >> > > >
> >> > > > *** Please download, test and vote by Thursday, May 16, 9:00
> pm PT.
> >> > > >
> >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> >> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2FKEYS&data=02%7C01%7C%7C139a19a5204441d9a17908d6e59d49bf%7C84df9e7fe9f640afb435%7C1%7C0%7C636948861037978009&sdata=nB8hxA70E7kbIUd0IHPOXZ1i%2F4TgqatRrakwtqsCDOE%3D&reserved=0
> >> > > >
> >> > > > * Release artifacts to be voted upon (source and binary):
> >> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~vahid%2Fkafka-2.2.1-rc1%2F&data=02%7C01%7C%7C139a19a5204441d9a17908d6e59d49bf%7C84df9e7fe9f640afb435%7C1%7C0%7C636948861037978009&sdata=6t0AGO3glzUF8ip9lcsMep9Y4Am2k6RuMthH9FuU4s4%3D&reserved=0
> >> > > >
> >> > > > * Maven artifacts to be voted upon:
> >> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fkafka%2F&data=02%7C01%7C%7C139a19a5204441d9a17908d6e59d49bf%7C84df9e7fe9f640afb435%7C1%7C0%7C636948861037978009&sdata=usesbkDDEUiZLHBjkkQVBFecT1vB437tUnFx%2B4glIlo%3D&reserved=0
> >> > > >
> >> > > > * Javadoc:
> >> > > >
> https://nam04.safelinks.protection.outlook.com/?url=https:%2F%2Fhome.apache.org%2F~vahid%2Fkafka-2.2.1-rc1%2Fjavadoc%2F&data=02%7C01%7C%7C139a19a5204441d9a17908d6e59d49bf%7C84df9e7fe9f640afb435%7C1%7C0%7C636948861037988014&sd

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

[Satish Duggana]

Thanks Viktor, proposed metrics are really useful to monitor replication
status on brokers.

+1 (non-binding)

On Thu, Jun 6, 2019 at 2:05 AM Colin McCabe  wrote:

> +1 (binding)
>
> best,
> Colin
>
>
> On Wed, Jun 5, 2019, at 03:38, Viktor Somogyi-Vass wrote:
> > Hi Folks,
> >
> > This vote sunk a bit, I'd like to draw some attention to this again in
> the
> > hope I get some feedback or votes.
> >
> > Thanks,
> > Viktor
> >
> > On Tue, May 7, 2019 at 4:28 PM Harsha  wrote:
> >
> > > Thanks for the kip. LGTM +1.
> > >
> > > -Harsha
> > >
> > > On Mon, Apr 29, 2019, at 8:14 AM, Viktor Somogyi-Vass wrote:
> > > > Hi Jason,
> > > >
> > > > I too agree this is more of a problem in older versions and
> therefore we
> > > > could backport it. Were you thinking of any specific versions? I
> guess
> > > the
> > > > 2.x and 1.x versions are definitely targets here but I was thinking
> that
> > > we
> > > > might not want to further.
> > > >
> > > > Viktor
> > > >
> > > > On Mon, Apr 29, 2019 at 12:55 AM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > > > Thanks for the work done, Viktor! +1 (non-binding)
> > > > >
> > > > > I strongly agree with Jason that this monitoring-focused KIP is
> worth
> > > > > porting back to older versions. I am sure users will find it very
> > > useful
> > > > >
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > > On Fri, Apr 26, 2019 at 9:38 PM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Thanks, that works for me. +1
> > > > > >
> > > > > > By the way, we don't normally port KIPs to older releases, but I
> > > wonder
> > > > > if
> > > > > > it's worth making an exception here. From recent experience, it
> > > tends to
> > > > > be
> > > > > > the older versions that are more prone to fetcher failures.
> Thoughts?
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Apr 26, 2019 at 5:18 AM Viktor Somogyi-Vass <
> > > > > > viktorsomo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Let me have a second thought, I'll just add the clientId
> instead to
> > > > > > follow
> > > > > > > the convention, so it'll change DeadFetcherThreadCount but
> with the
> > > > > > > clientId tag.
> > > > > > >
> > > > > > > On Fri, Apr 26, 2019 at 11:29 AM Viktor Somogyi-Vass <
> > > > > > > viktorsomo...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Yea I think it could make sense. In this case I would rename
> the
> > > > > > > > DeadFetcherThreadCount to DeadReplicaFetcherThreadCount and
> > > introduce
> > > > > > the
> > > > > > > > metric you're referring to as DeadLogDirFetcherThreadCount.
> > > > > > > > I'll update the KIP to reflect this.
> > > > > > > >
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Thu, Apr 25, 2019 at 8:07 PM Jason Gustafson <
> > > ja...@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Viktor,
> > > > > > > >>
> > > > > > > >> This looks good. Just one question I had is whether we may
> as
> > > well
> > > > > > cover
> > > > > > > >> the log dir fetchers as well.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Jason
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Apr 25, 2019 at 7:46 AM Viktor Somogyi-Vass <
> > > > > > > >> viktorsomo...@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Folks,
> > > > > > > >> >
> > > > > > > >> > This thread sunk a bit but I'd like to bump it hoping to
> get
> > > some
> > > > > > > >> feedback
> > > > > > > >> > and/or votes.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Viktor
> > > > > > > >> >
> > > > > > > >> > On Thu, Mar 28, 2019 at 8:47 PM Viktor Somogyi-Vass <
> > > > > > > >> > viktorsomo...@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Sorry, the end of the message cut off.
> > > > > > > >> > >
> > > > > > > >> > > So I tried to be consistent with the convention in
> > > LogManager,
> > > > > > hence
> > > > > > > >> the
> > > > > > > >> > > hyphens and in AbstractFetcherManager, hence the camel
> > > case. It
> > > > > > > would
> > > > > > > >> be
> > > > > > > >> > > nice though to decide with one convention across the
> whole
> > > > > > project,
> > > > > > > >> > however
> > > > > > > >> > > it requires a major refactor (especially for the
> components
> > > that
> > > > > > > >> leverage
> > > > > > > >> > > metrics for monitoring).
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Viktor
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Mar 28, 2019 at 8:44 PM Viktor Somogyi-Vass <
> > > > > > > >> > > viktorsomo...@gmail.com> wrote:
> > > > > > > >> > >
> > > > > > > >> > >> Hi Dhruvil,
> > > > > > > >> > >>
> > > > > > > >> > >> Thanks for the feedback and the vote. I fixed the typo
> in
> > > the
> > > > > > KIP.
> > > > > > > >> > >> The naming is interesting though. Unfortunately kafka
> > > overall
> > > > > is
> > > > > > > not

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Satish Duggana]

Hi Andy,
Thanks for the KIP. This is a good change and it gives the user a better
handle on Admin client usage. I agree with the proposal except the new
`Admin` interface having all the methods from `AdminClient` abstract class.
It should be kept clean having only the admin operations as methods from
KafkaClient abstract class but not the factory methods as mentioned in the
earlier mail.

I know about dynamic proxies(which were widely used in RMI/EJB world). I am
curious about the usecase using dynamic proxies with Admin client
interface. Dynamic proxy can have performance penalty if it is used in
critical path. Is that the primary motivation for creating the KIP?

Thanks,
Satish.

On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:

> I'm not married to that part.  That was only done to keep it more or less
> inline with what's already there, (an abstract class that has a factory
> method that returns a subclass sounds like the same anti-pattern ;))
>
> An alternative would to have an `AdminClients` utility class to create the
> admin client.
>
> On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
> wrote:
>
> > Hmmm...
> >
> > So the new interface, returns an instance of a class that implements the
> > interface. This sounds a little bit like an anti-pattern? Shouldn't
> > interfaces actually not know anything about classes that implement the
> > interface?
> >
> >
> > -Matthias
> >
> > On 6/10/19 11:22 AM, Andy Coates wrote:
> > > `AdminClient` would be deprecated purely because it would no longer
> serve
> > > any purpose and would be virtually empty, getting all of its
> > implementation
> > > from the new interfar. It would be nice to remove this from the API at
> > the
> > > next major version bump, hence the need to deprecate.
> > >
> > > `AdminClient.create()` would return what it does today, (so not a
> > breaking
> > > change).
> > >
> > > On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan 
> wrote:
> > >
> > >>> The existing `AdminClient` will be marked as deprecated.
> > >>
> > >> What's the reasoning behind this? I'm fine with the other changes, but
> > >> would prefer to keep the existing public API intact if it's not
> hurting
> > >> anything.
> > >>
> > >> Also, what will AdminClient.create() return? Would it be a breaking
> > change?
> > >>
> > >> Ryanne
> > >>
> > >> On Tue, Jun 4, 2019, 11:17 AM Andy Coates  wrote:
> > >>
> > >>> Hi folks
> > >>>
> > >>> As there's been no chatter on this KIP I'm assuming it's
> > non-contentious,
> > >>> (or just boring), hence I'd like to call a vote for KIP-476:
> > >>>
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-476%3A+Add+Java+AdminClient+Interface
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Andy
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] 2.3.0 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 2.3.0-rc2 tag.
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.

On Mon, Jun 17, 2019 at 7:25 PM David Arthur  wrote:
>
> +1 binding
>
> Verified signatures, pulled down kafka_2.12-2.3.0 and ran producer/consumer
> perf test scripts.
>
> -David
>
> On Mon, Jun 17, 2019 at 1:48 AM Vahid Hashemian 
> wrote:
>
> > +1 (non-binding)
> >
> > I also verifies signatures, build from source and tested the Quickstart
> > successfully on the built binary.
> >
> > BTW, I don't see a link to documentation for 2.3. Is there a reason?
> >
> > Thanks,
> > --Vahid
> >
> > On Sat, Jun 15, 2019 at 6:38 PM Gwen Shapira  wrote:
> >
> > > +1 (binding)
> > >
> > > Verified signatures, built from sources, ran quickstart on binary and
> > > checked out the passing jenkins build on the branch.
> > >
> > > Gwen
> > >
> > >
> > > On Thu, Jun 13, 2019 at 11:58 AM Colin McCabe 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Good news: I have run a junit test build for RC2, and it passed.  Check
> > > out https://builds.apache.org/job/kafka-2.3-jdk8/51/
> > > >
> > > > Also, the vote will go until Saturday, June 15th (sorry for the typo
> > > earlier in the vote end time).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Wed, Jun 12, 2019, at 15:55, Colin McCabe wrote:
> > > > > Hi all,
> > > > >
> > > > > We discovered some problems with the first release candidate (RC1) of
> > > > > 2.3.0.  Specifically, KAFKA-8484 and KAFKA-8500.  I have created a
> > new
> > > > > release candidate that includes fixes for these issues.
> > > > >
> > > > > Check out the release notes for the 2.3.0 release here:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > The vote will go until Friday, June 7th, or until we create another R
> > > > >
> > > > > * Kafka's KEYS file containing PGP keys we use to sign the release
> > can
> > > > > be found here:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * The release artifacts to be voted upon (source and binary) are
> > here:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~cmccabe/kafka-2.3.0-rc2/javadoc/
> > > > >
> > > > > * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/2.3.0-rc2
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Satish Duggana]

+1 Matthias/Andy.
IMHO, interface is about the contract, it should not have/expose any
implementation. I am fine with either way as it is more of taste or
preference.

Agree with Ismael/Colin/Ryanne on not deprecating for good reasons.


On Mon, Jun 24, 2019 at 8:33 PM Andy Coates  wrote:
>
> I agree Matthias.
>
> (In Scala, such factory methods are on a companion object. As Java doesn't
> have the concept of a companion object, an equivalent would be a utility
> class with a similar name...)
>
> However, I'll update the KIP to include the factory method on the interface.
>
> On Fri, 21 Jun 2019 at 23:40, Matthias J. Sax  wrote:
>
> > I still think, that an interface does not need to know anything about
> > its implementation. But I am also fine if we add a factory method to the
> > new interface if that is preferred by most people.
> >
> >
> > -Matthias
> >
> > On 6/21/19 7:10 AM, Ismael Juma wrote:
> > > This is even more reason not to deprecate immediately, there is very
> > little
> > > maintenance cost for us. We should be mindful that many of our users (eg
> > > Spark, Flink, etc.) typically allow users to specify the kafka clients
> > > version and hence avoid using new classes/interfaces for some time. They
> > > would get a bunch of warnings they cannot do anything about apart from
> > > suppressing.
> > >
> > > Ismael
> > >
> > > On Fri, Jun 21, 2019 at 4:00 AM Andy Coates  wrote:
> > >
> > >> Hi Ismael,
> > >>
> > >> I’m happy enough to not deprecate the existing `AdminClient` class as
> > part
> > >> of this change.
> > >>
> > >> However, note that, the class will likely be empty, i.e. all methods and
> > >> implementations will be inherited from the interface:
> > >>
> > >> public abstract class AdminClient implements Admin {
> > >> }
> > >>
> > >> Not marking it as deprecated has the benefit that users won’t see any
> > >> deprecation warnings on the next release. Conversely, deprecating it
> > will
> > >> mean we can choose to remove this, now pointless class, in the future
> > if we
> > >> choose.
> > >>
> > >> That’s my thinking for deprecation, but as I’ve said I’m happy either
> > way.
> > >>
> > >> Andy
> > >>
> > >>> On 18 Jun 2019, at 16:09, Ismael Juma  wrote:
> > >>>
> > >>> I agree with Ryanne, I think we should avoid deprecating AdminClient
> > and
> > >>> causing so much churn for users who don't actually care about this
> > niche
> > >>> use case.
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Tue, Jun 18, 2019 at 6:43 AM Andy Coates  wrote:
> > >>>
> > >>>> Hi Ryanne,
> > >>>>
> > >>>> If we don't change the client code, then everywhere will still expect
> > >>>> subclasses of `AdminClient`, so the interface will be of no use, i.e.
> > I
> > >>>> can't write a class that implements the new interface and pass it to
> > the
> > >>>> client code.
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Andy
> > >>>>
> > >>>> On Mon, 17 Jun 2019 at 19:01, Ryanne Dolan 
> > >> wrote:
> > >>>>
> > >>>>> Andy, while I agree that the new interface is useful, I'm not
> > convinced
> > >>>>> adding an interface requires deprecating AdminClient and changing so
> > >> much
> > >>>>> client code. Why not just add the Admin interface, have AdminClient
> > >>>>> implement it, and have done?
> > >>>>>
> > >>>>> Ryanne
> > >>>>>
> > >>>>> On Mon, Jun 17, 2019 at 12:09 PM Andy Coates 
> > >> wrote:
> > >>>>>
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> I think I've addressed all concerns. Let me know if I've not.  Can I
> > >>>> call
> > >>>>>> another round of votes please?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Andy
> > >>>>>>
> > >>>>>> 

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Satish Duggana]

+1 (non-binding)

On Wed, Jun 26, 2019 at 8:37 AM Satish Duggana  wrote:
>
> +1 Matthias/Andy.
> IMHO, interface is about the contract, it should not have/expose any
> implementation. I am fine with either way as it is more of taste or
> preference.
>
> Agree with Ismael/Colin/Ryanne on not deprecating for good reasons.
>
>
> On Mon, Jun 24, 2019 at 8:33 PM Andy Coates  wrote:
> >
> > I agree Matthias.
> >
> > (In Scala, such factory methods are on a companion object. As Java doesn't
> > have the concept of a companion object, an equivalent would be a utility
> > class with a similar name...)
> >
> > However, I'll update the KIP to include the factory method on the interface.
> >
> > On Fri, 21 Jun 2019 at 23:40, Matthias J. Sax  wrote:
> >
> > > I still think, that an interface does not need to know anything about
> > > its implementation. But I am also fine if we add a factory method to the
> > > new interface if that is preferred by most people.
> > >
> > >
> > > -Matthias
> > >
> > > On 6/21/19 7:10 AM, Ismael Juma wrote:
> > > > This is even more reason not to deprecate immediately, there is very
> > > little
> > > > maintenance cost for us. We should be mindful that many of our users (eg
> > > > Spark, Flink, etc.) typically allow users to specify the kafka clients
> > > > version and hence avoid using new classes/interfaces for some time. They
> > > > would get a bunch of warnings they cannot do anything about apart from
> > > > suppressing.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jun 21, 2019 at 4:00 AM Andy Coates  wrote:
> > > >
> > > >> Hi Ismael,
> > > >>
> > > >> I’m happy enough to not deprecate the existing `AdminClient` class as
> > > part
> > > >> of this change.
> > > >>
> > > >> However, note that, the class will likely be empty, i.e. all methods 
> > > >> and
> > > >> implementations will be inherited from the interface:
> > > >>
> > > >> public abstract class AdminClient implements Admin {
> > > >> }
> > > >>
> > > >> Not marking it as deprecated has the benefit that users won’t see any
> > > >> deprecation warnings on the next release. Conversely, deprecating it
> > > will
> > > >> mean we can choose to remove this, now pointless class, in the future
> > > if we
> > > >> choose.
> > > >>
> > > >> That’s my thinking for deprecation, but as I’ve said I’m happy either
> > > way.
> > > >>
> > > >> Andy
> > > >>
> > > >>> On 18 Jun 2019, at 16:09, Ismael Juma  wrote:
> > > >>>
> > > >>> I agree with Ryanne, I think we should avoid deprecating AdminClient
> > > and
> > > >>> causing so much churn for users who don't actually care about this
> > > niche
> > > >>> use case.
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>> On Tue, Jun 18, 2019 at 6:43 AM Andy Coates  wrote:
> > > >>>
> > > >>>> Hi Ryanne,
> > > >>>>
> > > >>>> If we don't change the client code, then everywhere will still expect
> > > >>>> subclasses of `AdminClient`, so the interface will be of no use, i.e.
> > > I
> > > >>>> can't write a class that implements the new interface and pass it to
> > > the
> > > >>>> client code.
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Andy
> > > >>>>
> > > >>>> On Mon, 17 Jun 2019 at 19:01, Ryanne Dolan 
> > > >> wrote:
> > > >>>>
> > > >>>>> Andy, while I agree that the new interface is useful, I'm not
> > > convinced
> > > >>>>> adding an interface requires deprecating AdminClient and changing so
> > > >> much
> > > >>>>> client code. Why not just add the Admin interface, have AdminClient
> > > >>>>> implement it, and have done?
> > > >>>>>
> > > >>>>> Ryanne
> > > >>>>>
> > > >>>>>

Re: [DISCUSS] KIP-492 Add java security providers in Kafka Security config

[Satish Duggana]

Hi Sandeep,
Thanks for the KIP, I have few comments below.

>>“To take advantage of these custom algorithms, we want to support java 
>>security provider parameter in security config. This param can be used by 
>>kafka brokers or kafka clients(when connecting to the kafka brokers). The 
>>security providers can also be used for configuring security algorithms in 
>>SASL based communication.”

You may want to mention use case like
spiffe.provider.SpiffeProvider[1] in streaming applications like
Flink, Spark or Storm etc.

>>"We add new config parameter in KafkaConfig named “security.provider.class”. 
>>The value of “security.provider” is expected to be a string representing the 
>>provider’s full classname. This provider class will be added to the JVM 
>>properties through Security.addProvider api. Security class can be used to 
>>programmatically add the provider classes to the JVM."

It is good to have this property as a list of providers instead of a
single property. This will allow configuring multiple providers if it
is needed in the future without introducing hacky solutions like
security.provider.class.name.x, where x is a sequence number. You can
change the property name to “security.provider.class.names” and its
value is a list of fully qualified provider class names separated by
‘,'.
For example:
security.provider.class.names=spiffe.provider.SpiffeProvider,com.foo.MyProvider

Typo in existing properties section:
“ssl.provider” instead of “ssl.providers”.

Thanks,
Satish.

1. https://github.com/spiffe/java-spiffe


On Mon, Jul 15, 2019 at 11:41 AM Sandeep Mopuri  wrote:
>
> Hello all,
>
> I'd like to start a discussion thread for KIP-492.
> This KIP plans on introducing a new security config parameter for a custom
> security providers. Please take a look and let me know what do you think.
>
> More information can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> --
> Thanks,
> Sai Sandeep

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

[Satish Duggana]

Thanks for the KIP. I have put my comments below.

This is a nice improvement to avoid cumbersome maintenance.

>> The following is the requirements this KIP is trying to accomplish:
   The ability to add and remove the preferred leader deprioritized
list/blacklist. e.g. new ZK path/node or new dynamic config.

This can be moved to the"Proposed changes" section.

>>The logic to determine the priority/order of which broker should be
preferred leader should be modified.  The broker in the preferred leader
blacklist should be moved to the end (lowest priority) when
determining leadership.

I believe there is no change required in the ordering of the preferred
replica list. Brokers in the preferred leader blacklist are skipped
until other brokers int he list are unavailable.

>>The blacklist can be at the broker level. However, there might be use cases
where a specific topic should blacklist particular brokers, which
would be at the
Topic level Config. For this use cases of this KIP, it seems that broker level
blacklist would suffice.  Topic level preferred leader blacklist might
be future enhancement work.

I agree that the broker level preferred leader blacklist would be
sufficient. Do you have any use cases which require topic level
preferred blacklist?

You can add the below workaround as an item in the rejected alternatives section
"Reassigning all the topic/partitions which the intended broker is a
replica for."

Thanks,
Satish.

On Fri, Jul 19, 2019 at 7:33 AM Stanislav Kozlovski
 wrote:
>
> Hey George,
>
> Thanks for the KIP, it's an interesting idea.
>
> I was wondering whether we could achieve the same thing via the
> kafka-reassign-partitions tool. As you had also said in the JIRA,  it is
> true that this is currently very tedious with the tool. My thoughts are
> that we could improve the tool and give it the notion of a "blacklisted
> preferred leader".
> This would have some benefits like:
> - more fine-grained control over the blacklist. we may not want to
> blacklist all the preferred leaders, as that would make the blacklisted
> broker a follower of last resort which is not very useful. In the cases of
> an underpowered AWS machine or a controller, you might overshoot and make
> the broker very underutilized if you completely make it leaderless.
> - is not permanent. If we are to have a blacklist leaders config,
> rebalancing tools would also need to know about it and manipulate/respect
> it to achieve a fair balance.
> It seems like both problems are tied to balancing partitions, it's just
> that KIP-491's use case wants to balance them against other factors in a
> more nuanced way. It makes sense to have both be done from the same place
>
> To make note of the motivation section:
> > Avoid bouncing broker in order to lose its leadership
> The recommended way to make a broker lose its leadership is to run a
> reassignment on its partitions
> > The cross-data center cluster has AWS cloud instances which have less
> computing power
> We recommend running Kafka on homogeneous machines. It would be cool if the
> system supported more flexibility in that regard but that is more nuanced
> and a preferred leader blacklist may not be the best first approach to the
> issue
>
> Adding a new config which can fundamentally change the way replication is
> done is complex, both for the system (the replication code is complex
> enough) and the user. Users would have another potential config that could
> backfire on them - e.g if left forgotten.
>
> Could you think of any downsides to implementing this functionality (or a
> variation of it) in the kafka-reassign-partitions.sh tool?
> One downside I can see is that we would not have it handle new partitions
> created after the "blacklist operation". As a first iteration I think that
> may be acceptable
>
> Thanks,
> Stanislav
>
> On Fri, Jul 19, 2019 at 3:20 AM George Li 
> wrote:
>
> >  Hi,
> >
> > Pinging the list for the feedbacks of this KIP-491  (
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982
> > )
> >
> >
> > Thanks,
> > George
> >
> > On Saturday, July 13, 2019, 08:43:25 PM PDT, George Li <
> > sql_consult...@yahoo.com.INVALID> wrote:
> >
> >   Hi,
> >
> > I have created KIP-491 (
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982)
> > for putting a broker to the preferred leader blacklist or deprioritized
> > list so when determining leadership,  it's moved to the lowest priority for
> > some of the listed use-cases.
> >
> > Please provide your comments/feedbacks.
> >
> > Thanks,
> > George
> >
> >
> >
> >   - Forwarded Message - From: Jose Armando Garcia Sancio (JIRA) <
> > j...@apache.org>To: "sql_consult...@yahoo.com" 
> > Sent:
> > Tuesday, July 9, 2019, 01:06:05 PM PDTSubject: [jira] [Commented]
> > (KAFKA-8638) Preferred Leader Blacklist (deprioritized list)
> >
> > [
> > https://issues.apache.org/jira/browse/KAFKA-8638?page=com.atlassian.jira.plugin.system.issuetabpanel

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>>Under the proposed definition of RemoteTier, would it be possible to
have an implementation that transfers older log segments to a slower storage
tier, but one that is still local?
Examples of slower local(ie mounted locally) tiers being HDDs vs SSDs,
or NFS volumes.

No, it does not allow moving logs to different mount points in the
same broker. In the current design, follower replicas may not have the
log segments locally available but it will fetch the remote log
indexes for older segments. Follower replicas will not replicate the
data beyond local log retention. So, the data that is out of local log
retention period/size will be fetched from the remote tier.

Thanks,
Satish.

On Thu, Jul 25, 2019 at 6:01 AM Habib Nahas  wrote:
>
> Hi,
>
> Under the proposed definition of RemoteTier, would it be possible to have an 
> implementation that transfers older log segments to a slower storage tier, 
> but one that is still local?
> Examples of slower local(ie mounted locally) tiers being HDDs vs SSDs, or NFS 
> volumes.
>
> Let me know if I"m missing an existing solution for this usecase.
> Thanks,
>
> Habib
>
>
> On 2019/04/09 05:04:17, Harsha  wrote:
> > Thanks, Ron. Updating the KIP. will add answers here as well>
> >
> > 1) If the cold storage technology can be cross-region, is there a>
> > possibility for a disaster recovery Kafka cluster to share the messages in>
> > cold storage? My guess is the answer is no, and messages replicated to the>
> > D/R cluster have to be migrated to cold storage from there independently.>
> > (The same cross-region cold storage medium could be used, but every message>
> > would appear there twice).>
> >
> > If I understand the question correctly, what you are saying is Kafka A 
> > cluster (active) shipping logs to remote storage which cross-region 
> > replication and another Kafka Cluster B (Passive) will it be able to use 
> > the remote storage copied logs directly.>
>
>
>
>
> > For the initial version my answer is No. We can handle this in subsequent 
> > changes after this one.>
> >
> > 2) Can/should external (non-Kafka) tools have direct access to the messages>
> > in cold storage. I think this might have been addressed when someone asked>
> > about ACLs, and I believe the answer is "no" -- if some external tool needs>
> > to operate on that data then that external tool should read that data by>
> > acting as a Kafka consumer. Again, just asking to get the answer clearly>
> > documented in case it is unclear.>
> >
> > The answer is No. All tools/clients must go through broker APIs to access 
> > any data (local or remote). >
> > Only Kafka broker user will have access to remote storage logs and 
> > Security/ACLs will work the way it does today.>
> > Tools/Clients going directly to the remote storage might help in terms of 
> > efficiency but this requires Protocol changes and some way of syncing ACLs 
> > in Kafka to the Remote storage. >
>
>
>
>
> >
> >
> > Thanks,>
> > Harsha>
> >
> > On Mon, Apr 8, 2019, at 8:48 AM, Ron Dagostino wrote:>
> > > Hi Harsha. A couple of questions. I think I know the answers, but it>
> > > would be good to see them explicitly documented.>
> > > >
> > > 1) If the cold storage technology can be cross-region, is there a>
> > > possibility for a disaster recovery Kafka cluster to share the messages 
> > > in>
> > > cold storage? My guess is the answer is no, and messages replicated to 
> > > the>
> > > D/R cluster have to be migrated to cold storage from there independently.>
> > > (The same cross-region cold storage medium could be used, but every 
> > > message>
> > > would appear there twice).>
> > > >
> > > 2) Can/should external (non-Kafka) tools have direct access to the 
> > > messages>
> > > in cold storage. I think this might have been addressed when someone 
> > > asked>
> > > about ACLs, and I believe the answer is "no" -- if some external tool 
> > > needs>
> > > to operate on that data then that external tool should read that data by>
> > > acting as a Kafka consumer. Again, just asking to get the answer clearly>
> > > documented in case it is unclear.>
> > > >
> > > Ron>
> > > >
> > > >
> > > On Thu, Apr 4, 2019 at 12:53 AM Harsha  wrote:>
> > > >
> > > > Hi Viktor,>
> > > >>
> > > >>
> > > > "Now, will the consumer be able to consume a remote segment if:>
> > > > - the remote segment is stored in the remote storage, BUT>
> > > > - the leader broker failed right after this AND>
> > > > - the follower which is to become a leader didn't scan yet for a new>
> > > > segment?">
> > > >>
> > > > If I understand correctly, after a local log segment copied to remote 
> > > > and>
> > > > leader is failed to write the index files and leadership changed to a>
> > > > follower. In this case we consider the log segment copy failed and 
> > > > newly>
> > > > elected leader will start copying the data from last the known offset 
> > > > in>
> > > > the remote to copy. Consumers who are looking for the offset which 
> > > > might>
> > >

Re: [VOTE] KIP-492 Add java security providers in Kafka Security config

[Satish Duggana]

+1 (non-binding)

Thanks,
Satish.

On Tue, Jul 30, 2019 at 5:18 AM Harsha Chintalapani  wrote:
>
> Thanks for the KIP Sandeep.
>
> +1 (binding)
>
> Thanks,
> Harsha
> On Jul 29, 2019, 12:22 PM -0700, Sandeep Mopuri , wrote:
> > Hi all, after some good discussion
> >  about the
> > KIP
> > ,
> > I'm starting the voting.
> >
> > This KIP proposes adding new security configuration to accept custom
> > security providers that can provide algorithms for SSL or SASL.
> >
> > --
> > Thanks,
> > M.Sai Sandeep

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Satish Duggana]

Hi Justine,
Thanks for the KIP. This is a nice addition to the producer client
without running admin-client’s create topic APIs. Does producer wait
for the topic to be created successfully before it tries to publish
messages to that topic? I assume that this will not throw an error
that the topic does not exist.

As mentioned by others, overriding broker behavior by producer looks
to be a concern. IMHO, broker should have a way to use either default
constraints or configure custom constraints before these can be
overridden by clients but not vice versa. There should be an option on
brokers whether those constraints can be overridden by producers or
not.

Thanks,
Satish.

On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan  wrote:
>
> Hi Harsha,
>
> After taking this all into consideration, I've updated the KIP to no longer
> allow client-side configuration of replication factor and partitions.
> Instead, the broker defaults will be used as long as the broker supports
> KIP 464.
> If the broker does not support this KIP, then the client can not create
> topics on its own. (Behavior that exists now)
>
> I think this will help with your concerns. Please let me know if you
> further feedback.
>
> Thank you,
> Justine
>
> On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani  wrote:
>
> > Hi,
> > Even with policies one needs to implement that, so for every user who
> > doesn't want a producer to create topics or have limits around partitions &
> > replication factor they have to implement a policy.
> >   The KIP is changing the behavior , it might not be introducing the
> > new functionality but it will enable producers to override the create topic
> > config settings on the broker. What I am asking for to provide a config
> > that will disable auto creation of topics and if its enabled set some sane
> > defaults so that clients can create a topic with in those limits. I don't
> > see how this not related to this KIP.
> >  If the server config options as I suggested doesn't interest you at
> > least have a default CreateTopicPolices in place.
> >To give an example, In our environment we disable the
> > auto.create.topic.enable and force users to go through a centralized
> > service as we want capture more details about the topic creation and
> > requirements. With this KIP, a producer can create a topic with no bounds.
> >  Another example max.message.size we define that at cluster level and one
> > can override max.messsage.size at topic level, any misconfiguration at this
> > will cause service degradation.  Its not always about the rogue clients,
> > users can easily misconfigure and can cause an outage.
> > Again we can talk about CreateTopicPolicy but without having a default
> > implementation and asking everyone to implement their own while changing
> > the behavior in producer  doesn't make sense to me.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma  wrote:
> >
> > > Hi Harsha,
> > >
> > > I mentioned policies and the authorizer. For example, with
> > > CreateTopicPolicy, you can implement the limits you describe. If you have
> > > ideas of how that should be improved, please submit a KIP. My point is
> > that
> > > this KIP is not introducing any new functionality with regards to what
> > > rogue clients can do. It's using the existing protocol that is already
> > > exposed via the AdminClient. So, I don't think we need to address it in
> > > this KIP. Does that make sense?
> > >
> > > Ismael
> > >
> > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani 
> > > wrote:
> > >
> > > Ismael,
> > > Sure AdminClient can do that and we should've shipped a config or use the
> > > existing one to block that. Not all users are yet to upgrade to
> > AdminClient
> > > and start using that to cause issues yet. In shared environment we should
> > > allow server to set sane defaults and not allow every client to go ahead
> > > create random no.of topic/partitions and replication factor. Even if the
> > > users want to allow topic creation proposed in the KIP , it makes sense
> > to
> > > have some guards against the no.of partitions and replication factor.
> > > Authorizer is not always an answer to block requests and having users set
> > > server side configs to protect a multi-tenant environment is required.
> > In a
> > > non-secure environment Authorizer is a blunt instrument either you end up
> > > blocking everyone or allowing everyone.
> > > I am asking to have server side that allow clients to create topics or
> > not
> > > , if they are allowed set a ceiling on max no.of partitions and
> > > replication-factor.
> > >
> > > -Harsha
> > >
> > > On Mon, Aug 5 2019 at 8:58 PM,  wrote:
> > >
> > > Harsha,
> > >
> > > Rogue clients can use the admin client to create topics and partitions.
> > > ACLs and policies can help in that case as well as this one.
> > >
> > > Ismael
> > >
> > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani 
> > >
> > > wrote:
> > >
> > > 

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Satish Duggana]

Hi Justine,
Thanks for the clarifications.

I understand that auto-creation of topics will happen through
CreateTopic request instead of metadata request. What I meant in
earlier mail is producer client should not override broker config
about auto-creation of topics. I agree with Harsha on other mail about
this behavior. If auto-creation is disabled on broker, producer
clients will never be allowed to create topics even if
'allow.auto.create.topics' is true in producer client.

On Wed, Aug 7, 2019 at 1:28 AM Justine Olshan  wrote:
>
> Hi Satish,
>
> Thanks for looking at the KIP.
>
> Yes, the producer will wait for the topic to be created before it can send
> any messages to it.
>
> I would like to clarify "overriding" broker behavior. If the client enables
> client-side autocreation, the only difference will be that the topic
> auto-creation will no longer occur in the metadata request and will instead
> come from a CreateTopic request on the producer.
> Partitions and replication factor will be determined by the broker configs.
>
> Is this similar to what you were thinking? Please let me know if there is
> something you think I missed.
>
> Thank you,
> Justine
>
> On Tue, Aug 6, 2019 at 12:01 PM Satish Duggana 
> wrote:
>
> > Hi Justine,
> > Thanks for the KIP. This is a nice addition to the producer client
> > without running admin-client’s create topic APIs. Does producer wait
> > for the topic to be created successfully before it tries to publish
> > messages to that topic? I assume that this will not throw an error
> > that the topic does not exist.
> >
> > As mentioned by others, overriding broker behavior by producer looks
> > to be a concern. IMHO, broker should have a way to use either default
> > constraints or configure custom constraints before these can be
> > overridden by clients but not vice versa. There should be an option on
> > brokers whether those constraints can be overridden by producers or
> > not.
> >
> > Thanks,
> > Satish.
> >
> > On Tue, Aug 6, 2019 at 11:39 PM Justine Olshan 
> > wrote:
> > >
> > > Hi Harsha,
> > >
> > > After taking this all into consideration, I've updated the KIP to no
> > longer
> > > allow client-side configuration of replication factor and partitions.
> > > Instead, the broker defaults will be used as long as the broker supports
> > > KIP 464.
> > > If the broker does not support this KIP, then the client can not create
> > > topics on its own. (Behavior that exists now)
> > >
> > > I think this will help with your concerns. Please let me know if you
> > > further feedback.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Tue, Aug 6, 2019 at 10:49 AM Harsha Chintalapani 
> > wrote:
> > >
> > > > Hi,
> > > > Even with policies one needs to implement that, so for every user
> > who
> > > > doesn't want a producer to create topics or have limits around
> > partitions &
> > > > replication factor they have to implement a policy.
> > > >   The KIP is changing the behavior , it might not be introducing
> > the
> > > > new functionality but it will enable producers to override the create
> > topic
> > > > config settings on the broker. What I am asking for to provide a config
> > > > that will disable auto creation of topics and if its enabled set some
> > sane
> > > > defaults so that clients can create a topic with in those limits. I
> > don't
> > > > see how this not related to this KIP.
> > > >  If the server config options as I suggested doesn't interest you
> > at
> > > > least have a default CreateTopicPolices in place.
> > > >To give an example, In our environment we disable the
> > > > auto.create.topic.enable and force users to go through a centralized
> > > > service as we want capture more details about the topic creation and
> > > > requirements. With this KIP, a producer can create a topic with no
> > bounds.
> > > >  Another example max.message.size we define that at cluster level and
> > one
> > > > can override max.messsage.size at topic level, any misconfiguration at
> > this
> > > > will cause service degradation.  Its not always about the rogue
> > clients,
> > > > users can easily misconfigure and can cause an outage.
> > > > Again we can talk about CreateTopicPolicy but without having a default
> >

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

[Satish Duggana]

en a broker has high CPU
> > usage, trying to identify the big topics (High MsgIn, High BytesIn,
> > etc), then try to move the leaders away from this broker,  before doing
> > an actual reassignment to change its preferred leader,  try to put this
> > preferred_leader_blacklist in the Topic Level config, and run preferred
> > leader election, and see whether CPU decreases for this broker,  if
> > yes, then do the reassignments to change the preferred leaders to be
> > "permanent" (the topic may have many partitions like 256 that has quite
> > a few of them having this broker as preferred leader).  So this Topic
> > Level config is an easy way of doing trial and check the result.
> >
> >
> > > You can add the below workaround as an item in the rejected alternatives 
> > > section
> > > "Reassigning all the topic/partitions which the intended broker is a
> > > replica for."
> >
> > Updated the KIP-491.
> >
> >
> >
> > Thanks,
> > George
> >
> >On Friday, July 19, 2019, 08:20:22 AM PDT, Satish Duggana
> >  wrote:
> >
> >  Thanks for the KIP. I have put my comments below.
> >
> > This is a nice improvement to avoid cumbersome maintenance.
> >
> > >> The following is the requirements this KIP is trying to accomplish:
> >   The ability to add and remove the preferred leader deprioritized
> > list/blacklist. e.g. new ZK path/node or new dynamic config.
> >
> > This can be moved to the"Proposed changes" section.
> >
> > >>The logic to determine the priority/order of which broker should be
> > preferred leader should be modified.  The broker in the preferred leader
> > blacklist should be moved to the end (lowest priority) when
> > determining leadership.
> >
> > I believe there is no change required in the ordering of the preferred
> > replica list. Brokers in the preferred leader blacklist are skipped
> > until other brokers int he list are unavailable.
> >
> > >>The blacklist can be at the broker level. However, there might be use 
> > >>cases
> > where a specific topic should blacklist particular brokers, which
> > would be at the
> > Topic level Config. For this use cases of this KIP, it seems that broker 
> > level
> > blacklist would suffice.  Topic level preferred leader blacklist might
> > be future enhancement work.
> >
> > I agree that the broker level preferred leader blacklist would be
> > sufficient. Do you have any use cases which require topic level
> > preferred blacklist?
> >
> > You can add the below workaround as an item in the rejected alternatives 
> > section
> > "Reassigning all the topic/partitions which the intended broker is a
> > replica for."
> >
> > Thanks,
> > Satish.
> >
> > On Fri, Jul 19, 2019 at 7:33 AM Stanislav Kozlovski
> >  wrote:
> > >
> > > Hey George,
> > >
> > > Thanks for the KIP, it's an interesting idea.
> > >
> > > I was wondering whether we could achieve the same thing via the
> > > kafka-reassign-partitions tool. As you had also said in the JIRA,  it is
> > > true that this is currently very tedious with the tool. My thoughts are
> > > that we could improve the tool and give it the notion of a "blacklisted
> > > preferred leader".
> > > This would have some benefits like:
> > > - more fine-grained control over the blacklist. we may not want to
> > > blacklist all the preferred leaders, as that would make the blacklisted
> > > broker a follower of last resort which is not very useful. In the cases of
> > > an underpowered AWS machine or a controller, you might overshoot and make
> > > the broker very underutilized if you completely make it leaderless.
> > > - is not permanent. If we are to have a blacklist leaders config,
> > > rebalancing tools would also need to know about it and manipulate/respect
> > > it to achieve a fair balance.
> > > It seems like both problems are tied to balancing partitions, it's just
> > > that KIP-491's use case wants to balance them against other factors in a
> > > more nuanced way. It makes sense to have both be done from the same place
> > >
> > > To make note of the motivation section:
> > > > Avoid bouncing broker in order to lose its leadership
> > > The recommended way to make a broker lose its leadership is to run a
> > > reassignment on its partitions
> > > > The cross-data

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Satish Duggana]

I felt the same need when we want to add a pluggable API for core
server functionality. This does not need to be part of this KIP, it
can be a separate KIP. I can contribute those refactoring changes if
others are OK with that.

It is better to have a structure like below.

kafka-common:
common classes which can be used in any of the other modules in Kafka
like client, Kafka-server-common and server etc.

kafka-client-common:
common classes which can be used in the client module. This can be
part of client module itself.

kafka-server-common:
classes required only for kafka-server.

Thanks.
Satish.

On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani  wrote:
>
> Thanks for the KIP Rajini.
> Quick thought, it would be good to have a common module outside of clients
> that only applies to server side interfaces & changes. It looks like we are
> increasingly in favor of using Java interface for pluggable modules  on the
> broker side.
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > I have created a KIP to replace the Scala Authorizer API with a new Java
> > API:
> >
> > -
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-504+-+Add+new+Java+Authorizer+Interface
> >
> > This is replacement for KIP-50 which was accepted but never merged. Apart
> > from moving to a Java API consistent with other pluggable interfaces in the
> > broker, KIP-504 also attempts to address known limitations in the
> > authorizer. If you have come across other limitations that you would like
> > to see addressed in the new API, please raise these on the discussion
> > thread so that we can consider those too. All suggestions and feedback are
> > welcome.
> >
> > Thank you,
> >
> > Rajini
> >

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Satish Duggana]

Hi Rajini,
Sure, I will start a discussion thread soon on dev mailing list.

Thanks,
Satish.

On Thu, Aug 8, 2019 at 1:29 AM Rajini Sivaram  wrote:
>
> Hi Ron/Harsha/Satish,
>
> Thanks for reviewing the KIP!
>
> We should perhaps have a wider discussion outside this KIP for refactoring
> clients so that others who are not following this KIP also notice the
> discussion. Satish, would you like to start a discussion thread on dev?
>
> Regards,
>
> Rajini
>
>
> On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana 
> wrote:
>
> > I felt the same need when we want to add a pluggable API for core
> > server functionality. This does not need to be part of this KIP, it
> > can be a separate KIP. I can contribute those refactoring changes if
> > others are OK with that.
> >
> > It is better to have a structure like below.
> >
> > kafka-common:
> > common classes which can be used in any of the other modules in Kafka
> > like client, Kafka-server-common and server etc.
> >
> > kafka-client-common:
> > common classes which can be used in the client module. This can be
> > part of client module itself.
> >
> > kafka-server-common:
> > classes required only for kafka-server.
> >
> > Thanks.
> > Satish.
> >
> > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani 
> > wrote:
> > >
> > > Thanks for the KIP Rajini.
> > > Quick thought, it would be good to have a common module outside of
> > clients
> > > that only applies to server side interfaces & changes. It looks like we
> > are
> > > increasingly in favor of using Java interface for pluggable modules  on
> > the
> > > broker side.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a KIP to replace the Scala Authorizer API with a new
> > Java
> > > > API:
> > > >
> > > > -
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >
> > > > This is replacement for KIP-50 which was accepted but never merged.
> > Apart
> > > > from moving to a Java API consistent with other pluggable interfaces
> > in the
> > > > broker, KIP-504 also attempts to address known limitations in the
> > > > authorizer. If you have come across other limitations that you would
> > like
> > > > to see addressed in the new API, please raise these on the discussion
> > > > thread so that we can consider those too. All suggestions and feedback
> > are
> > > > welcome.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> >

[DISCUSS] Modularization of kafka client separating server related classes/interfaces

[Satish Duggana]

Hi,

There are many classes in the client module that are not really related to
the client. It is good to have common modules structure with respective
classes/interfaces. Implementors/providers need to have dependency only on
those modules instead of having a dependency on the client module.

Below common module structure may be a good starting point for the
discussion.

kafka-common:

Common classes which can be used by any of the other modules

like client, kafka-server-common, and server(core), streams etc.

Kafka-server-common(or kafka-core-common):

Classes required only for server(core) and for the implementors/providers.



Below are some of the common server interfaces/classes which are required
only on the server(or core). These are not related to the client module.


org.apache.kafka.server.policy

AlterConfigPolicy

CreateTopicPolicy

org.apache.kafka.server.quota

ClientQuotaCallback

ClientQuotaEntity

ClientQuotaType

org.apache.kafka.common.replica

ClientMetadata

PartitionView

RackAwareReplicaSelector

ReplicaSelector

ReplicaView

One more example, java Authorizer interface is introduced with KIP-504. Its
implementation will be applicable only for server(core) module. This can be
moved to kafka-server-common module instead of pushing into kafka client
module.

Please let me know your thoughts/opinions.

Thanks,

Satish.

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

[Satish Duggana]

+1 (non-binding) Thanks for the KIP, so useful.

On Fri, Aug 9, 2019 at 4:42 PM Mickael Maison 
wrote:

> +1 (non binding)
> Thanks for the KIP!
>
> On Fri, Aug 9, 2019 at 9:36 AM Andrew Schofield
>  wrote:
> >
> > +1 (non-binding)
> >
> > On 09/08/2019, 08:39, "Sönke Liebau" 
> wrote:
> >
> > +1 (non-binding)
> >
> >
> >
> > On Fri, 9 Aug 2019 at 04:45, Harsha Chintalapani 
> wrote:
> >
> > > +1  (binding). much needed!!
> > >
> > >
> > > On Thu, Aug 08, 2019 at 6:43 PM, Gwen Shapira 
> wrote:
> > >
> > > > +1 (binding) THANK YOU. It would be +100 if I could.
> > > >
> > > > On Thu, Aug 8, 2019 at 6:37 PM Mitchell 
> wrote:
> > > >
> > > > Hello Dev,
> > > > After the discussion I would like to start the vote for KIP-499
> > > >
> > > > The following command line tools will have the
> `--bootstrap-server`
> > > > command line argument added: kafka-console-producer.sh,
> > > > kafka-consumer-groups.sh, kafka-consumer-perf-test.sh,
> > > > kafka-verifiable-consumer.sh, kafka-verifiable-producer.sh
> > > >
> > > > Thanks,
> > > > -Mitch
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
> >
>

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

[Satish Duggana]

Hi Jungtaek,
Thanks for the KIP. I have a couple of questions here.
Is not Spark using Kafka's consumer group management across multiple
consumers?

Is Spark using KafkaConsumer#subscribe(Pattern pattern,
ConsumerRebalanceListener listener) only to get all the topics for a
pattern based subscription and Spark manually assigns those
topic-partitions across consumers on workers?

Thanks,
Satish.

On Mon, Aug 12, 2019 at 4:17 AM Matthias J. Sax 
wrote:

> If am not sure if I fully understand yet.
>
> The fact, that Spark does not stores offsets in Kafka but as part of its
> own checkpoint mechanism seems to be orthogonal. Maybe I am missing
> something here.
>
> As you are using subscribe(), you use Kafka consumer group mechanism,
> that takes care of the assignment of partitions to clients within the
> group. Therefore, I am not sure what you mean by:
>
> > which Spark needs to
> >> know to coordinate multiple consumers to pull correctly.
>
> Multiple thoughts that may help:
>
>  - if Spark needs more control about the partition assignment, you can
> provide a custom `ConsumerPartitionAssignor` (via the consumer
> configuration)
>
>  - you may also want to register `ConsumerRebalanceListener` via
> `subscribe()` to get informed when the group rebalances
>
> As you pointed out, using pattern subscription metadata can change if
> topic are added/deleted. However, each metadata change will triggering a
> rebalance and thus you would get corresponding calls to you rebalance
> listener to learn about it and react accordingly.
>
> Maybe you can explain why neither of both approaches works and what gap
> the new API would close?
>
>
> -Matthias
>
> On 8/11/19 5:11 AM, Jungtaek Lim wrote:
> > Let me elaborate my explanation a bit more. Here we say about Apache
> Spark,
> > but this will apply for everything which want to control offset of Kafka
> > consumers.
> >
> > Spark is managing the committed offsets and the offsets which should be
> > polled now. Topics and partitions as well. This is required as Spark
> itself
> > has its own general checkpoint mechanism and Kafka is just a one of
> > source/sink (though it's considered as very important).
> >
> > To pull records from Kafka, Spark provides to Kafka which topics and
> > partitions it wants to subscribe(, and do seek and poll), but as Spark
> can
> > also provide "patterns" of topics, as well as subscription can be changed
> > in Kafka side (topic added/dropped, partitions added) which Spark needs
> to
> > know to coordinate multiple consumers to pull correctly.
> >
> > Looks like assignment() doesn't update the assignment information in
> > consumer. It just returns known one. There's only one known approach
> doing
> > this, calling `poll`, but Spark is not interested on returned records, so
> > there's a need for a hack `poll(0)`, and Kafka deprecated the API. This
> KIP
> > proposes to support this as official approach.
> >
> >
> > On Sun, Aug 11, 2019 at 8:18 PM Jungtaek Lim  wrote:
> >
> >> Sorry I didn't recognize you're also asking it here as well. I'm in
> favor
> >> of describing it in this discussion thread so the discussion itself can
> go
> >> forward. So copying my answer here:
> >>
> >> We have some use case which we don't just rely on everything what Kafka
> >> consumer provides. We want to know current assignment on this consumer,
> and
> >> to get the latest assignment, we called the hack `poll(0)`.
> >>
> >> That said, we don't want to pull any records here, and if I'm not
> missing
> >> here, there's no way to accomplish this. Please guide me if I'm missing
> >> something.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >>
> >>
> >> On Sat, Aug 10, 2019 at 2:11 AM Matthias J. Sax 
> >> wrote:
> >>
> >>> Thanks for the KIP.
> >>>
> >>> Can you elaborate a little bit more on the use case for this feature?
> >>> Why would a consumer need to update it's metadata explicitly?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 8/8/19 8:46 PM, Jungtaek Lim wrote:
>  Hi devs,
> 
>  I'd like to initiate discussion around KIP-505, exposing new public
> >>> method
>  to only update assignment metadata in consumer.
> 
>  `poll(0)` has been misused as according to Kafka doc it doesn't
> >>> guarantee
>  that it doesn't pull any records, and new method `poll(Duration)`
> >>> doesn't
>  have same semantic, so would like to propose new public API which only
> >>> does
>  the desired behavior.
> 
>  KIP page: https://cwiki.apache.org/confluence/x/z5NiBw
> 
>  Please feel free to suggest any improvements on proposal, as I'm new
> to
>  Kafka community and may not catch preferences (like TimeoutException
> vs
>  boolean, etc.) on Kafka project.
> 
>  Thanks in advance!
>  Jungtaek Lim (HeartSaVioR)
> 
> >>>
> >>>
> >>
> >> --
> >> Name : Jungtaek Lim
> >> Blog : http://medium.com/@heartsavior
> >> Twitter : http://twitter.com/heartsavior
> >> LinkedIn : http://www.linkedin.com/in/hear

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[Satish Duggana]

Hi Colin,
Thanks for the KIP. Optional fields and var length encoding support is a great
improvement for the protocol.

>>Optional fields can have any type, except that they cannot be arrays.
Note that the restriction against having tagged arrays is just to simplify
serialization.  We can relax this restriction in the future without changing
the protocol on the wire.

Can an Optional field have a struct type which internally contains an array
field at any level?

Thanks,
Satish.



On Tue, Aug 13, 2019 at 11:49 PM David Jacot  wrote:
>
> Hi Colin,
>
> Thank you for the KIP! Things are well explained!. It is huge improvement
> for the Kafka protocol. I have few comments on the proposal:
>
> 1. The interleaved tag/length header sounds like a great optimisation as it
> would be shorter on average. The downside, as
> you already pointed out, is that it makes the decoding and the specs more
> complex. Personally, I would also favour using two
> vaints in this particular case to keep things simple.
>
> 2. As discussed, I wonder if it would make sense to extend to KIP to also
> support optional fields in the Record Header. I think
> that it could be interesting to have such capability for common fields
> across all the requests or responses (e.g. tracing id).
>
> Regards,
> David
>
>
>
> On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson  wrote:
>
> > > Right, I was planning on doing exactly that for all the auto-generated
> > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > better use of time to convert the manual ones to auto gen first (with the
> > possible exception of Fetch/Produce, where the ROI may be higher for the
> > manual work)
> >
> > Yeah, that makes sense. Maybe we can include the version bump for all RPCs
> > in this KIP, but we can implement it lazily as the protocols are converted.
> >
> > -Jason
> >
> > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:
> >
> > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > > > Hi Colin,
> > > >
> > > > Thanks for the KIP! This is a significant improvement. One of my
> > personal
> > > > interests in this proposal is solving the compatibility problems we
> > have
> > > > with the internal schemas used to define consumer offsets and
> > transaction
> > > > metadata. Currently we have to guard schema bumps with the inter-broker
> > > > protocol format. Once the format is bumped, there is no way to
> > downgrade.
> > > > By fixing this, we can potentially begin using the new schemas before
> > the
> > > > IBP is bumped while still allowing downgrade.
> > > >
> > > > There are a surprising number of other situations we have encountered
> > > this
> > > > sort of problem. We have hacked around it in special cases by allowing
> > > > nullable fields to the end of the schema, but this is not really an
> > > > extensible approach. I'm looking forward to having a better option.
> > >
> > > Yeah, this problem keeps coming up.
> > >
> > > >
> > > > With that said, I have a couple questions on the proposal:
> > > >
> > > > 1. For each request API, we need one version bump to begin support for
> > > > "flexible versions." Until then, we won't have the option of using
> > tagged
> > > > fields even if the broker knows how to handle them. Does it make sense
> > to
> > > > go ahead and do a universal bump of each request API now so that we'll
> > > have
> > > > this option going forward?
> > >
> > > Right, I was planning on doing exactly that for all the auto-generated
> > > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > > better use of time to convert the manual ones to auto gen first (with the
> > > possible exception of Fetch/Produce, where the ROI may be higher for the
> > > manual work)
> > >
> > > > 2. The alternating length/tag header encoding lets us save a byte in
> > the
> > > > common case. The downside is that it's a bit more complex to specify.
> > It
> > > > also has some extra cost if the length exceeds the tag substantially.
> > > > Basically we'd have to pad the tag, right? I think I'm wondering if we
> > > > should just bite the bullet and use two varints instead.
> > >
> > > That’s a fair point. It would be shorter on average, but worse for some
> > > exceptional cases. Also, the decoding would be more complex, which might
> > be
> > > a good reason to go for just having two varints. Yeah, let’s simplify.
> > >
> > > Regards,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Fri, Aug 9, 2019 at 4:31 PM Colin McCabe 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've made some updates to this KIP. Specifically, I wanted to avoid
> > > > > including escape bytes in the serialization format, since it was too
> > > > > complex. Also, I think this is a good opportunity to slim down our
> > > > > variable length fields.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Thu, Jul 11, 2019, at 20:52, Colin McCabe wrote:
> > > > > > On

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

[Satish Duggana]

+1 (non-binding), Thanks Jason for nice improvements on ISR
propagation protocol!

On Wed, Aug 14, 2019 at 8:29 PM David Arthur  wrote:
>
> +1 binding, this looks great!
>
> -David
>
> On Tue, Aug 13, 2019 at 4:55 PM Guozhang Wang  wrote:
>
> > +1 (binding). This is a great KIP, thanks Jason!
> >
> > Regarding the naming of the zkVersion, I'm actually fine to name it more
> > generally and leave a note that at the moment its value is defined as the
> > zk version.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Aug 12, 2019 at 2:22 PM Jason Gustafson 
> > wrote:
> >
> > > Hi Viktor,
> > >
> > > I originally named the field `CurrentVersion`. I didn't have 'Zk' in the
> > > name in anticipation of KIP-500. I thought about it and decided it makes
> > > sense to keep naming consistent with other APIs. Even if KIP-500 passes,
> > > there will be some time during which it only refers to the zk version.
> > > Eventually we'll have to decide whether it makes sense to change the name
> > > or just introduce a new field.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Aug 9, 2019 at 9:19 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > +1 from me too.
> > > > One note though: since it's a new protocol we could perhaps rename
> > > > CurrentZkVersion to something like "IsrEpoch" or "IsrVersion". I think
> > > > that'd reflect its purpose better.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on KIP-497:
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR
> > > > > .
> > > > > +1
> > > > > from me.
> > > > >
> > > > > -Jason
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> David Arthur

Not able to download tests jar of kafka and kafka-streams from maven repo.

[Satish Duggana]

Hi,

Below dependency is added in one of our repos to use EmbeddedKafkaCluster
but dependency installation fails with an error mentioned later.


org.apache.kafka
kafka-streams
0.10.0.0
test-jar
test


This fails with an error below as
https://repository.apache.org/content/repositories/snapshots/org/apache/kafka/kafka-streams/0.10.0.0/kafka-streams-0.10.0.0-tests.jar
not available. But
https://repository.apache.org/content/repositories/snapshots/org/apache/kafka/kafka-streams/0.10.0.0/kafka-streams-0.10.0.0-test.jar
is available. You may need to fix POM to install right name which is
kafka-streams-0.10.0.0-test.jar instead of kafka-streams-0.10.0.0-tests.jar


[ERROR] Failed to execute goal on project schema-registry-avro: Could not
resolve dependencies for project
com.hortonworks.registries:schema-registry-avro:jar:0.1.0-SNAPSHOT: The
following artifacts could not be resolved:
org.apache.kafka:kafka-clients:jar:tests:0.10.0.0,
org.apache.kafka:kafka-streams:jar:tests:0.10.0.0: Could not find artifact
org.apache.kafka:kafka-clients:jar:tests:0.10.0.0 in central (
http://repo1.maven.org/maven2/) -> [Help 1]


JIRA is raised at https://issues.apache.org/jira/browse/KAFKA-4156.

Thanks,
Satish.

Re: [VOTE] KIP-503: deleted topics metric

[Satish Duggana]

+1(non-binding), very useful metrics for admin/ops.

Thanks,
Satish.


On Sat, Aug 17, 2019 at 4:08 PM Manikumar  wrote:

> +1 (binding).
>
> Thanks for the KIP. LGTM.
>
> Thanks,
> Manikumar
>
> On Wed, Aug 14, 2019 at 11:54 PM David Jacot  wrote:
>
> > +1 (non-binding)
> >
> > Thanks for the KIP! Simple yet very useful.
> >
> > Best,
> > David
> >
> > On Wed, Aug 14, 2019 at 9:24 AM Robert Barrett  >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > This will be good to have, thanks David!
> > >
> > > Bob
> > >
> > > On Wed, Aug 14, 2019 at 6:08 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 non binding
> > > > Thank you!
> > > >
> > > > On Tue, Aug 13, 2019 at 9:07 PM Stanislav Kozlovski
> > > >  wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks for the simple but very useful KIP!
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > > On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani <
> ka...@harsha.io
> > >
> > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 13, 2019 at 12:08 PM, David Arthur <
> > > davidart...@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I'd like to start the vote on KIP-503
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> > > > > > >
> > > > > > > Thanks!
> > > > > > > David
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Stanislav
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[Satish Duggana]

Sorry! Colin, I may not have been clear in my earlier query about
optional field type restriction. It is mentioned in one of your
replies "optional fields are serialized starting with their total
length". This brings the question of whether optional fields support
struct types (with or without array values). It seems struct types are
currently not serialized with total length. I may be missing something
here.

Thanks,
Satish.


On Wed, Aug 14, 2019 at 8:03 AM Satish Duggana  wrote:
>
> Hi Colin,
> Thanks for the KIP. Optional fields and var length encoding support is a great
> improvement for the protocol.
>
> >>Optional fields can have any type, except that they cannot be arrays.
> Note that the restriction against having tagged arrays is just to simplify
> serialization.  We can relax this restriction in the future without changing
> the protocol on the wire.
>
> Can an Optional field have a struct type which internally contains an array
> field at any level?
>
> Thanks,
> Satish.
>
>
>
> On Tue, Aug 13, 2019 at 11:49 PM David Jacot  wrote:
> >
> > Hi Colin,
> >
> > Thank you for the KIP! Things are well explained!. It is huge improvement
> > for the Kafka protocol. I have few comments on the proposal:
> >
> > 1. The interleaved tag/length header sounds like a great optimisation as it
> > would be shorter on average. The downside, as
> > you already pointed out, is that it makes the decoding and the specs more
> > complex. Personally, I would also favour using two
> > vaints in this particular case to keep things simple.
> >
> > 2. As discussed, I wonder if it would make sense to extend to KIP to also
> > support optional fields in the Record Header. I think
> > that it could be interesting to have such capability for common fields
> > across all the requests or responses (e.g. tracing id).
> >
> > Regards,
> > David
> >
> >
> >
> > On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson  wrote:
> >
> > > > Right, I was planning on doing exactly that for all the auto-generated
> > > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > > better use of time to convert the manual ones to auto gen first (with the
> > > possible exception of Fetch/Produce, where the ROI may be higher for the
> > > manual work)
> > >
> > > Yeah, that makes sense. Maybe we can include the version bump for all RPCs
> > > in this KIP, but we can implement it lazily as the protocols are 
> > > converted.
> > >
> > > -Jason
> > >
> > > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:
> > >
> > > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for the KIP! This is a significant improvement. One of my
> > > personal
> > > > > interests in this proposal is solving the compatibility problems we
> > > have
> > > > > with the internal schemas used to define consumer offsets and
> > > transaction
> > > > > metadata. Currently we have to guard schema bumps with the 
> > > > > inter-broker
> > > > > protocol format. Once the format is bumped, there is no way to
> > > downgrade.
> > > > > By fixing this, we can potentially begin using the new schemas before
> > > the
> > > > > IBP is bumped while still allowing downgrade.
> > > > >
> > > > > There are a surprising number of other situations we have encountered
> > > > this
> > > > > sort of problem. We have hacked around it in special cases by allowing
> > > > > nullable fields to the end of the schema, but this is not really an
> > > > > extensible approach. I'm looking forward to having a better option.
> > > >
> > > > Yeah, this problem keeps coming up.
> > > >
> > > > >
> > > > > With that said, I have a couple questions on the proposal:
> > > > >
> > > > > 1. For each request API, we need one version bump to begin support for
> > > > > "flexible versions." Until then, we won't have the option of using
> > > tagged
> > > > > fields even if the broker knows how to handle them. Does it make sense
> > > to
> > > > > go ahead and do a universal bump of each request API now so that we'll
> > > > have
> > > > > this option going forward?
> > > >
> > > > Right

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[Satish Duggana]

Please read struct type as a complex record type in my earlier mail.
The complex type seems to be defined as Schema[1] in the protocol
types.

1. 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java#L27


On Mon, Aug 19, 2019 at 9:46 AM Satish Duggana  wrote:
>
> Sorry! Colin, I may not have been clear in my earlier query about
> optional field type restriction. It is mentioned in one of your
> replies "optional fields are serialized starting with their total
> length". This brings the question of whether optional fields support
> struct types (with or without array values). It seems struct types are
> currently not serialized with total length. I may be missing something
> here.
>
> Thanks,
> Satish.
>
>
> On Wed, Aug 14, 2019 at 8:03 AM Satish Duggana  
> wrote:
> >
> > Hi Colin,
> > Thanks for the KIP. Optional fields and var length encoding support is a 
> > great
> > improvement for the protocol.
> >
> > >>Optional fields can have any type, except that they cannot be arrays.
> > Note that the restriction against having tagged arrays is just to simplify
> > serialization.  We can relax this restriction in the future without changing
> > the protocol on the wire.
> >
> > Can an Optional field have a struct type which internally contains an array
> > field at any level?
> >
> > Thanks,
> > Satish.
> >
> >
> >
> > On Tue, Aug 13, 2019 at 11:49 PM David Jacot  wrote:
> > >
> > > Hi Colin,
> > >
> > > Thank you for the KIP! Things are well explained!. It is huge improvement
> > > for the Kafka protocol. I have few comments on the proposal:
> > >
> > > 1. The interleaved tag/length header sounds like a great optimisation as 
> > > it
> > > would be shorter on average. The downside, as
> > > you already pointed out, is that it makes the decoding and the specs more
> > > complex. Personally, I would also favour using two
> > > vaints in this particular case to keep things simple.
> > >
> > > 2. As discussed, I wonder if it would make sense to extend to KIP to also
> > > support optional fields in the Record Header. I think
> > > that it could be interesting to have such capability for common fields
> > > across all the requests or responses (e.g. tracing id).
> > >
> > > Regards,
> > > David
> > >
> > >
> > >
> > > On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson  
> > > wrote:
> > >
> > > > > Right, I was planning on doing exactly that for all the auto-generated
> > > > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > > > better use of time to convert the manual ones to auto gen first (with 
> > > > the
> > > > possible exception of Fetch/Produce, where the ROI may be higher for the
> > > > manual work)
> > > >
> > > > Yeah, that makes sense. Maybe we can include the version bump for all 
> > > > RPCs
> > > > in this KIP, but we can implement it lazily as the protocols are 
> > > > converted.
> > > >
> > > > -Jason
> > > >
> > > > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:
> > > >
> > > > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > > > > > Hi Colin,
> > > > > >
> > > > > > Thanks for the KIP! This is a significant improvement. One of my
> > > > personal
> > > > > > interests in this proposal is solving the compatibility problems we
> > > > have
> > > > > > with the internal schemas used to define consumer offsets and
> > > > transaction
> > > > > > metadata. Currently we have to guard schema bumps with the 
> > > > > > inter-broker
> > > > > > protocol format. Once the format is bumped, there is no way to
> > > > downgrade.
> > > > > > By fixing this, we can potentially begin using the new schemas 
> > > > > > before
> > > > the
> > > > > > IBP is bumped while still allowing downgrade.
> > > > > >
> > > > > > There are a surprising number of other situations we have 
> > > > > > encountered
> > > > > this
> > > > > > sort of problem. We have hacked around it in special cases by 
> > > > > > allowing
> > > > >

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[Satish Duggana]

Hi Jason,
+1 (non binding) Thanks for the KIP!

Do we need to have a separate JIRA to update the docs as it introduces new
metrics and a change in behavior for the existing metric?



On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison 
wrote:

> +1 (non binding)
> Thanks Jason
>
> On Wed, Aug 21, 2019 at 8:15 AM David Jacot  wrote:
> >
> > +1 (non-binding)
> >
> > Thanks for the KIP!
> >
> > Best,
> > David
> >
> > On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson 
> wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455
> to fix
> > > a long-known shortcoming of URP reporting and to improve reassignment
> > > monitoring:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> > > .
> > >
> > > Note that I have added one new metric following the discussion. It
> seemed
> > > useful to have a lag metric for reassigning partitions.
> > >
> > > Thanks,
> > > Jason
> > >
>

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[Satish Duggana]

Hi David,
Thanks for the KIP. I have a couple of questions.

>> For the Java client, the idea is to define two constants in the code to 
>> store its name and its version. If possible, the version will be set 
>> automatically based on metadata coming from gradle (or the repo itself) to 
>> avoid having to do manual changes.

Did you consider taking version property by loading
“kafka/kafka-version.properties” as a resource while java client is
initialized?  “kafka/kafka-version.properties” is shipped with
kafka-clients jar.

>> kafka.server:type=ClientMetrics,name=ConnectedClients
I assume this metric value will be the total no of clients connected
to a broker irrespective of whether name and version follow the
expected pattern ([-.\w]+) or not.

>> kafka.server:type=ClientMetrics,name=ConnectedClients,clientname=([-.\w]+),clientversion=([-.\w]+)
It seems client name and client version are treated as tags for
`ConnectedClients` metric. If so, you may implement this metric
similar to `BrokerTopicMetrics` with topic tag as mentioned here[1].
When is the metric removed for a specific client-name and
client-version?

1. 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaRequestHandler.scala#L231

Thanks,
Satish.




On Wed, Aug 21, 2019 at 5:33 PM David Jacot  wrote:
>
> Hi all,
>
> I would like to start a discussion for KIP-511:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers
>
> Let me know what you think.
>
> Best,
> David

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

[Satish Duggana]

+1 (non-binding) Thanks for the nice KIP.

You may want to update the KIP saying that optional tagged fields do
not support complex types(or structs).

On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
 wrote:
>
> +1 (non-binding)
>
> Looking forward to this improvement.
>
> On Tue, Sep 3, 2019 at 12:49 PM David Jacot  wrote:
>
> > +1 (non-binding)
> >
> > Thank for the KIP. Great addition to the Kafka protocol!
> >
> > Best,
> > David
> >
> > Le mar. 3 sept. 2019 à 19:17, Colin McCabe  a écrit :
> >
> > > Hi all,
> > >
> > > I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> > > Optional Tagged Fields.
> > >
> > > KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > >
> > > Discussion thread here:
> > >
> > https://lists.apache.org/thread.html/cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.apache.org%3E
> > >
> > > best,
> > > Colin
> > >
> >
>
>
> --
> -Jose

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

[Satish Duggana]

; > > could be the preferred leader.
> >
> > I agree that it would be nice if we could treat some brokers
> > differently for the purposes of placing replicas, selecting leaders,
> > etc.  Right now, we don't have any way of implementing that without
> > forking the broker.  I would support a new PlacementPolicy class that
> > would close this gap.  But I don't think this KIP is flexible enough to
> > fill this role.  For example, it can't prevent users from creating new
> > single-replica topics that get put on the "bad" replica.  Perhaps we
> > should reopen the discussion about
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-201%3A+Rationalising+Policy+interfaces
> >
> > regards,
> > Colin
> >
> > >
> > > Please let me know there are more question.
> > >
> > >
> > > Thanks,
> > > George
> > >
> > >On Thursday, July 25, 2019, 08:38:28 AM PDT, Colin McCabe
> > >  wrote:
> > >
> > >  We still want to give the "blacklisted" broker the leadership if
> > > nobody else is available.  Therefore, isn't putting a broker on the
> > > blacklist pretty much the same as moving it to the last entry in the
> > > replicas list and then triggering a preferred leader election?
> > >
> > > If we want this to be undone after a certain amount of time, or under
> > > certain conditions, that seems like something that would be more
> > > effectively done by an external system, rather than putting all these
> > > policies into Kafka.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Fri, Jul 19, 2019, at 18:23, George Li wrote:
> > > >  Hi Satish,
> > > > Thanks for the reviews and feedbacks.
> > > >
> > > > > > The following is the requirements this KIP is trying to accomplish:
> > > > > This can be moved to the"Proposed changes" section.
> > > >
> > > > Updated the KIP-491.
> > > >
> > > > > >>The logic to determine the priority/order of which broker should be
> > > > > preferred leader should be modified.  The broker in the preferred 
> > > > > leader
> > > > > blacklist should be moved to the end (lowest priority) when
> > > > > determining leadership.
> > > > >
> > > > > I believe there is no change required in the ordering of the preferred
> > > > > replica list. Brokers in the preferred leader blacklist are skipped
> > > > > until other brokers int he list are unavailable.
> > > >
> > > > Yes. partition assignment remained the same, replica & ordering. The
> > > > blacklist logic can be optimized during implementation.
> > > >
> > > > > >>The blacklist can be at the broker level. However, there might be 
> > > > > >>use cases
> > > > > where a specific topic should blacklist particular brokers, which
> > > > > would be at the
> > > > > Topic level Config. For this use cases of this KIP, it seems that 
> > > > > broker level
> > > > > blacklist would suffice.  Topic level preferred leader blacklist might
> > > > > be future enhancement work.
> > > > >
> > > > > I agree that the broker level preferred leader blacklist would be
> > > > > sufficient. Do you have any use cases which require topic level
> > > > > preferred blacklist?
> > > >
> > > >
> > > >
> > > > I don't have any concrete use cases for Topic level preferred leader
> > > > blacklist.  One scenarios I can think of is when a broker has high CPU
> > > > usage, trying to identify the big topics (High MsgIn, High BytesIn,
> > > > etc), then try to move the leaders away from this broker,  before doing
> > > > an actual reassignment to change its preferred leader,  try to put this
> > > > preferred_leader_blacklist in the Topic Level config, and run preferred
> > > > leader election, and see whether CPU decreases for this broker,  if
> > > > yes, then do the reassignments to change the preferred leaders to be
> > > > "permanent" (the topic may have many partitions like 256 that has quite
> > > > a few of them having this broker as preferred leader).  So this Topic
> > > > Level config is an easy way of doing trial and check the result.

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Satish Duggana]

Hi Kevin,
Thanks for adding useful metrics with the KIP.

On Wed, 18 Sep, 2019, 1:49 AM Kevin Lu,  wrote:

> Hi Manikumar,
>
> Thanks for the support.
>
> Since we have added a couple additional metrics, I have renamed the KIP
> title to reflect the content better:  KIP-517: Add consumer metrics to
> observe user poll behavior
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior
> >
>
> Regards,
> Kevin
>
> On Tue, Sep 17, 2019 at 11:07 AM Manikumar 
> wrote:
>
> > Hi Kevin,
> >
> > Thanks for the KIP.  LGTM. This will be useful.
> >
> > Thanks,
> >
> > On Mon, Sep 16, 2019 at 10:17 PM Harsha Chintalapani 
> > wrote:
> >
> > > Thanks. +1 LGTM.
> > >
> > >
> > > On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu 
> wrote:
> > >
> > > > Hi Harsha,
> > > >
> > > > Thanks for the feedback. I have added *last-poll-seconds-ago* to the
> > KIP
> > > > (being consistent with *last-heartbeat-seconds-ago*).
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani  >
> > > > wrote:
> > > >
> > > > Thanks Kevin for the KIP. Overall LGTM.
> > > > On you second point, I think the metric will be really useful to
> > indicate
> > > > the perf bottlenecks on user code vs kakfa consumer/broker.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu 
> > wrote:
> > > >
> > > > Hi Radai & Jason,
> > > >
> > > > Thanks for the support and suggestion.
> > > >
> > > > 1. I think ratio is a good additional metric since the current
> proposed
> > > > metrics are only absolute times which may not be useful in all
> > scenarios.
> > > >
> > > > I have added this to the KIP:
> > > > * - poll-idle-ratio*: The fraction of time the consumer spent waiting
> > for
> > > > the user to process records from poll.
> > > >
> > > > Thoughts on the metric name/description?
> > > >
> > > > 2. Would it be useful to include a metric measuring the time since
> poll
> > > > was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> > > > *poll-last-ms-ago.
> > > > *This could be useful if (1) the user has a very high
> > *max.poll.interval.
> > > > ms
> > > > * configured and typically spends a
> long
> > > > time processing, or (2) comparing this metric with others such as
> > > > *heartbeat-last-seconds-ago* or something else for gathering data in
> > root
> > > > cause analyses (or identifying potential consumer bugs related to
> > poll).
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson  >
> > > > wrote:
> > > >
> > > > Hi Kevin,
> > > >
> > > > This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> > > > willing. Something like an idle ratio for the consumer would be
> > helpful.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Fri, Sep 13, 2019 at 10:08 AM radai 
> > > > wrote:
> > > >
> > > > while youre at it another metric that we have found to be useful is %
> > > >
> > > > time
> > > >
> > > > spent in user code vs time spent in poll() (so time between poll
> calls
> > /
> > > > time inside poll calls) - the higher the % value the more indicative
> of
> > > > user code being the cause of performance bottlenecks.
> > > >
> > > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu 
> > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Happy Friday! Bumping this. Any thoughts?
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu 
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I'd like to propose a new consumer metric that measures the time
> > > >
> > > > between
> > > >
> > > > calls to poll() for use in issues related to hitting
> > > >
> > > > max.poll.interval.ms
> > > >
> > > > due to long processing time.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
> > > >
> > > > Please give it a read and let me know what you think.
> > > >
> > > > Thanks!
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-525 - Return topic metadata and configs in CreateTopics response

[Satish Duggana]

Thanks for the KIP, +1 (non binding)

This looks to be a useful API for admin tools. When we worked on an
admin tool for Kafka, we had to explicitly call the config API for
showing the configs for the newly created topic.
I have a minor-nit comment on `CreateResult` which seems to be a
slightly confusing name as it is not really about the CreateResult of
a topic but it is about topic metadata and config(maybe
TopicMetadataWithConfig). This does not need to be addressed in the
KIP now as it is not exposed to clients. But you may want to address
this in the PR later.


On Sun, Sep 22, 2019 at 9:02 PM Harsha Chintalapani  wrote:
>
> +1 (binding).
> -Harsha
>
>
> On Sat, Sep 21, 2019 at 12:13 AM, Manikumar 
> wrote:
>
> > +1 (binding), Thanks for the KIP.
> >
> > Thanks,
> >
> > On Sat, Sep 21, 2019 at 2:49 AM Colin McCabe  wrote:
> >
> > +1 (binding). Thanks, Rajini.
> >
> > best,
> > Colin
> >
> > On Fri, Sep 20, 2019, at 00:43, Rajini Sivaram wrote:
> >
> > Hi all,
> >
> > I would like to start vote on KIP-525 to return configs in CreateTopics
> > response. This is a minor KIP that returns additional data in the
> >
> > response
> >
> > without breaking compatibility.
> >
> > -
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-525+-+Return+topic+metadata+and+configs+in+CreateTopics+response
> >
> > Thank you,
> >
> > Rajini
> >
> >

Re: [VOTE] KIP-524: Allow users to choose config source when describing configs

[Satish Duggana]

Thanks Jason for the KIP. It looks to be a minor improvement but very useful.

+1 (non-binding)

On Wed, Sep 25, 2019 at 9:29 PM Rajini Sivaram  wrote:
>
> Thanks for the KIP, Jason!
>
> +1 (binding)
>
> Regards,
>
> Rajini
>
> On Wed, Sep 25, 2019 at 4:41 PM Colin McCabe  wrote:
>
> > Looks good.  +1 (binding)
> >
> > best,
> > Colin
> >
> > On Tue, Sep 24, 2019, at 09:42, Jason Gustafson wrote:
> > > Hi All,
> > >
> > > I'm starting a vote for KIP-524, which is a small change to the config
> > > tool:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs
> > > .
> > >
> > > +1 from me.
> > >
> > > Thanks,
> > > Jason
> > >
> >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

for the comments.
> > > > > > >
> > > > > > > "I have a rather technical question to this. How do you plan to
> > > > package
> > > > > > > this
> > > > > > > extension? Does this mean that Kafka will depend on HDFS?
> > > > > > > I think it'd be nice to somehow separate this off to a
> different
> > > > package
> > > > > > in
> > > > > > > the project so that it could be built and released separately
> from
> > > > the
> > > > > > main
> > > > > > > Kafka packages."
> > > > > > >
> > > > > > > We would like all of this code to be part of Apache Kafka . In
> early
> > > > days
> > > > > > > of Kafka, there is external module which used to contain kafka
> to
> > > > hdfs
> > > > > > copy
> > > > > > > tools and dependencies.  We would like to have RLM (class
> > > > implementation)
> > > > > > > and RSM(interface) to be in core and as you suggested,
> > > > implementation of
> > > > > > > RSM could be in another package so that the dependencies of
> RSM won't
> > > > > > come
> > > > > > > into Kafka's classpath unless someone explicity configures
> them.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 1, 2019, at 1:02 AM, Viktor Somogyi-Vass wrote:
> > > > > > > > Hey Harsha,
> > > > > > > >
> > > > > > > > I have a rather technical question to this. How do you plan
> to
> > > > package
> > > > > > > this
> > > > > > > > extension? Does this mean that Kafka will depend on HDFS?
> > > > > > > > I think it'd be nice to somehow separate this off to a
> different
> > > > > > package
> > > > > > > in
> > > > > > > > the project so that it could be built and released
> separately from
> > > > the
> > > > > > > main
> > > > > > > > Kafka packages.
> > > > > > > > This decoupling would be useful when direct dependency on
> HDFS (or
> > > > > > other
> > > > > > > > implementations) is not needed and would also encourage
> decoupling
> > > > for
> > > > > > > > other storage implementations.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Mon, Apr 1, 2019 at 3:44 AM Ambud Sharma <
> > > > asharma52...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Harsha,
> > > > > > > > >
> > > > > > > > > Thank you for proposing this KIP. We are looking forward
> to this
> > > > > > > feature as
> > > > > > > > > well.
> > > > > > > > >
> > > > > > > > > A few questions around the design & implementation:
> > > > > > > > >
> > > > > > > > > 1. Wouldn't implicit checking for old offsets in remote
> location
> > > > if
> > > > > > not
> > > > > > > > > found locally on the leader i.e. do we really need remote
> index
> > > > > > files?
> > > > > > > > > Since the storage path for a given topic would presumably
> be
> > > > constant
> > > > > > > > > across all the brokers, the remote topic-partition path
> could
> > > > simply
> > > > > > be
> > > > > > > > > checked to see if there are any segment file names that
> would
> > > > meet
> > > > > > the
> > > > > > > > > offset requ

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

[Satish Duggana]

Thanks Jason for the KIP.
+1 (non-binding).

I assume this change will be added to the upgrade notes as the new
values are effective for clusters that were using earlier defaults.
Cluster may get into the race condition of having lower
replica.lag.time.max.ms  on a leader than zookeeper.session.timeout on
a follower replica till all the brokers are upgraded but that would
not break any semantics.

On Tue, Oct 22, 2019 at 2:20 PM M. Manna  wrote:
>
> +1 binding. This kip will possibly fix a lot of extra line of configuration.
>
> Thanks,
>
> On Tue, 22 Oct 2019 at 09:06, Stanislav Kozlovski 
> wrote:
>
> > Solid KIP. This also has the side-effect of bumping
> > zookeeper.connection.timeout.ms to a more reasonable default (the 18s
> > session timeout). Perhaps worth mentioning
> >
> > +1 (non-binding)
> >
> > On Tue, Oct 22, 2019 at 5:47 AM Ismael Juma  wrote:
> >
> > > +1 (binding)
> > >
> > > On Mon, Oct 21, 2019, 5:28 PM Jason Gustafson 
> > wrote:
> > >
> > > > I'd like to start a vote for KIP-537:
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout
> > > > .
> > > >
> > > > +1 from me
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

; > > > > optionally cache this information."
> > > > > > > > >
> > > > > > > > > By storing the remote index files locally , it will be faster
> > > for us
> > > > > > to
> > > > > > > > > determine for a requested offset which file might contain the
> > > data.
> > > > > > This
> > > > > > > > > will help us resolve the remote file quickly and return the
> > > response.
> > > > > > > > > Instead of making a call to remote tier for index look up.
> > > Given
> > > > > > index
> > > > > > > > > files are smaller , it won't be much hit to the storage
> > space.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > "2. Would it make sense to create an internal compacted Kafka
> > > topic
> > > > > > to
> > > > > > > > > publish & record remote segment information? This would
> > enable
> > > the
> > > > > > > > > followers to get updates about new segments rather than
> > running
> > > > > > list()
> > > > > > > > > operations on remote storage to detect new segments which may
> > > be
> > > > > > > > > expensive."
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think Ron also alluding to this. We thought shipping remote
> > > index
> > > > > > files
> > > > > > > > > to remote storage files and let the follower's RLM picking
> > > that up
> > > > > > makes
> > > > > > > > it
> > > > > > > > > easy to have the current replication protocol without any
> > > changes.
> > > > > > So we
> > > > > > > > > don't determine if a follower is in ISR or not based on
> > another
> > > > > > topic's
> > > > > > > > > replication.  We will run small tests and determine if use of
> > > topic
> > > > > > is
> > > > > > > > > better for this. Thanks for the suggestion.
> > > > > > > > >
> > > > > > > > > 3. For RLM to scan local segment rotations are you thinking
> > of
> > > > > > leveraging
> > > > > > > > > java.nio.file.WatchService or simply running listFiles() on a
> > > > > > periodic
> > > > > > > > > basis? Since WatchService implementation is heavily OS
> > > dependent it
> > > > > > might
> > > > > > > > > create some complications around missing FS Events.
> > > > > > > > >
> > > > > > > > > Ideally we want to introduce file events like you suggested.
> > > For POC
> > > > > > work
> > > > > > > > > we are using just listFiles(). Also copying these files to
> > > remote
> > > > > > can be
> > > > > > > > > slower and we will not delete the files from local disk until
> > > the
> > > > > > segment
> > > > > > > > > is copied and any requests to the data in these files will be
> > > served
> > > > > > from
> > > > > > > > > local disk. So I don't think we need to be aggressive and
> > > optimize
> > > > > > the
> > > > > > > > this
> > > > > > > > > copy segment to remote path.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Viktor,
> > > > > > > > >  Thanks for the comments.
> > > > > > > > >
> > > > > > > > > "I have a rather technical question to this. How do you plan
> > to
> > > > > > package
> > > > > > > > > this
> > > > > > > > > extension? Does this mean that Kafka will depend on HDFS?
> > > > > > > > > I think it'd be nice to somehow separate this off to a
> > > different
> > > > > > package
> > &

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,
Thanks for the clarifications.

>Regarding feature branch, if the goal is faster collaboration, it seems
that doing the development on your own fork is better since non-committers
can push changes there.

Sure, we will have the branch in our repo and collaborate with others
in that branch like we are doing now. We can revisit this later if
needed.

>Regarding the dependencies, this is an important thing to clarify. My
understanding for this KIP is that in Apache Kafka, we won't provide any
specific implementation for a particular block storage. There are many
block storage systems out there (HDFS, S3, Google storage, Azure storage,
Ceph, etc). We don't want to drag in all those dependencies in Apache
Kafka, even if they are in a separate module. Doing that will make the
Kafka repo much harder to manage. We have used the same approach for
connect. The connect framework is in Apache Kafka, but all specific
connectors are hosted externally.

I understand the rationale of not having external dependencies here.
Can we have HDFS(or S3) implementation in PR till it is ready to be
merged so that it will be easier to review and test the proposed APIs
end to end? We can remove this module before the PR is merged in and
it will be hosted in an external repo.

Thanks,
Satish.

On Thu, Oct 24, 2019 at 12:19 AM Jun Rao  wrote:
>
> Hi, Harsha,
>
> Regarding feature branch, if the goal is faster collaboration, it seems
> that doing the development on your own fork is better since non-committers
> can push changes there.
>
> Regarding the dependencies, this is an important thing to clarify. My
> understanding for this KIP is that in Apache Kafka, we won't provide any
> specific implementation for a particular block storage. There are many
> block storage systems out there (HDFS, S3, Google storage, Azure storage,
> Ceph, etc). We don't want to drag in all those dependencies in Apache
> Kafka, even if they are in a separate module. Doing that will make the
> Kafka repo much harder to manage. We have used the same approach for
> connect. The connect framework is in Apache Kafka, but all specific
> connectors are hosted externally.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Oct 23, 2019 at 8:41 AM Eno Thereska  wrote:
>
> > Thanks Satish, Harsha,
> >
> > It's probably worth it making it clearer in the KIP what exact
> > libraries will be added to libs, if any. The KIP specifies the remote
> > storage interface but it isn't clear if particular implementations
> > will be added to Kafka's repository or whether they will reside in
> > other repositories. If I understand the intention correctly, you are
> > proposing to have an HDFS and S3 implementation as part of the Kafka
> > repository working out of the box. Is that correct?
> >
> > Thanks
> > Eno
> >
> > On Wed, Oct 23, 2019 at 5:01 AM Satish Duggana 
> > wrote:
> > >
> > > >Regarding the HDFS dependency its not a direct dependency rather
> > > its implementing the RemoteStorageManager interface.
> > > We packaged it along with core to make it more convenient to test it. We
> > > can move this to external module and keep it there.
> > > Let me know what you think.
> > >
> > > Let me elaborate more on this point. With the new changes in the PR,
> > > kafka core or any other existing module is not dependent on HDFS. We
> > > created a new module called `remote-storage-managers/hdfs`. Libraries
> > > generated by this module are added to libs while packaging the
> > > distribution. This makes easy for users to try HDFS tiered storage
> > > instead of users building hdfs module and add it to libs on their own.
> > > We have plans to push these libs into external/libs/ directory and
> > > they will not be added to the classpath by default. We can add them to
> > > the classpath in scripts based on a system property.
> > >
> > > On Wed, Oct 23, 2019 at 6:26 AM Harsha Chintalapani 
> > wrote:
> > > >
> > > > Hi Jun,
> > > >Thanks for the feedback. Given the no.of engineers involved
> > in
> > > > cross-team effort
> > > > it would be great to have this as feature branch. Irrespective of if
> > its in
> > > > my fork
> > > > or in Apache Kafka's branch it needs to be constantly rebased from
> > trunk to
> > > > keep it current.
> > > > Our proposal is to merge it in feature branch and open a PR so its no
> > > > different than current PR except that
> > > > its in central repo rather my fork. Having it in Kafka's branch
> > > > m

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Eno and Viktor,

IMHO, it is helpful to have HDFS(os S3) implementation to be part of
this PR until it is ready to be merged. We will remove this module
before merging the PR. As suggested by Viktor, we can also have a
simple RI of `RemoteStorageManager` for users to understand the APIs
better.  I updated the KIP that Apache Kafka repo/distribution will
not have HDFS or S3 implementations, these will be available at
external repos.

Thanks,
Satish.

On Thu, Oct 24, 2019 at 2:08 PM Viktor Somogyi-Vass
 wrote:
>
> Hi Jun & Harsha,
>
> I think it would be beneficial to at least provide one simple reference
> implementation (file system based?) as we do with connect too.
> That would as a simple example and would help plugin developers to better
> understand the concept and the interfaces.
>
> Best,
> Viktor
>
> On Wed, Oct 23, 2019 at 8:49 PM Jun Rao  wrote:
>
> > Hi, Harsha,
> >
> > Regarding feature branch, if the goal is faster collaboration, it seems
> > that doing the development on your own fork is better since non-committers
> > can push changes there.
> >
> > Regarding the dependencies, this is an important thing to clarify. My
> > understanding for this KIP is that in Apache Kafka, we won't provide any
> > specific implementation for a particular block storage. There are many
> > block storage systems out there (HDFS, S3, Google storage, Azure storage,
> > Ceph, etc). We don't want to drag in all those dependencies in Apache
> > Kafka, even if they are in a separate module. Doing that will make the
> > Kafka repo much harder to manage. We have used the same approach for
> > connect. The connect framework is in Apache Kafka, but all specific
> > connectors are hosted externally.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Oct 23, 2019 at 8:41 AM Eno Thereska 
> > wrote:
> >
> > > Thanks Satish, Harsha,
> > >
> > > It's probably worth it making it clearer in the KIP what exact
> > > libraries will be added to libs, if any. The KIP specifies the remote
> > > storage interface but it isn't clear if particular implementations
> > > will be added to Kafka's repository or whether they will reside in
> > > other repositories. If I understand the intention correctly, you are
> > > proposing to have an HDFS and S3 implementation as part of the Kafka
> > > repository working out of the box. Is that correct?
> > >
> > > Thanks
> > > Eno
> > >
> > > On Wed, Oct 23, 2019 at 5:01 AM Satish Duggana  > >
> > > wrote:
> > > >
> > > > >Regarding the HDFS dependency its not a direct dependency rather
> > > > its implementing the RemoteStorageManager interface.
> > > > We packaged it along with core to make it more convenient to test it.
> > We
> > > > can move this to external module and keep it there.
> > > > Let me know what you think.
> > > >
> > > > Let me elaborate more on this point. With the new changes in the PR,
> > > > kafka core or any other existing module is not dependent on HDFS. We
> > > > created a new module called `remote-storage-managers/hdfs`. Libraries
> > > > generated by this module are added to libs while packaging the
> > > > distribution. This makes easy for users to try HDFS tiered storage
> > > > instead of users building hdfs module and add it to libs on their own.
> > > > We have plans to push these libs into external/libs/ directory and
> > > > they will not be added to the classpath by default. We can add them to
> > > > the classpath in scripts based on a system property.
> > > >
> > > > On Wed, Oct 23, 2019 at 6:26 AM Harsha Chintalapani 
> > > wrote:
> > > > >
> > > > > Hi Jun,
> > > > >Thanks for the feedback. Given the no.of engineers
> > involved
> > > in
> > > > > cross-team effort
> > > > > it would be great to have this as feature branch. Irrespective of if
> > > its in
> > > > > my fork
> > > > > or in Apache Kafka's branch it needs to be constantly rebased from
> > > trunk to
> > > > > keep it current.
> > > > > Our proposal is to merge it in feature branch and open a PR so its no
> > > > > different than current PR except that
> > > > > its in central repo rather my fork. Having it in Kafka's branch
> > > > > makes it easier for everyone to collaborate 

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

ka will depend on HDFS?
> > > > > > > > I think it'd be nice to somehow separate this off to a
> different
> > > > > package
> > > > > > > in
> > > > > > > > the project so that it could be built and released
> separately from
> > > > > the
> > > > > > > main
> > > > > > > > Kafka packages."
> > > > > > > >
> > > > > > > > We would like all of this code to be part of Apache Kafka .
> In early
> > > > > days
> > > > > > > > of Kafka, there is external module which used to contain
> kafka to
> > > > > hdfs
> > > > > > > copy
> > > > > > > > tools and dependencies.  We would like to have RLM (class
> > > > > implementation)
> > > > > > > > and RSM(interface) to be in core and as you suggested,
> > > > > implementation of
> > > > > > > > RSM could be in another package so that the dependencies of
> RSM won't
> > > > > > > come
> > > > > > > > into Kafka's classpath unless someone explicity configures
> them.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Harsha
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Apr 1, 2019, at 1:02 AM, Viktor Somogyi-Vass wrote:
> > > > > > > > > Hey Harsha,
> > > > > > > > >
> > > > > > > > > I have a rather technical question to this. How do you
> plan to
> > > > > package
> > > > > > > > this
> > > > > > > > > extension? Does this mean that Kafka will depend on HDFS?
> > > > > > > > > I think it'd be nice to somehow separate this off to a
> different
> > > > > > > package
> > > > > > > > in
> > > > > > > > > the project so that it could be built and released
> separately from
> > > > > the
> > > > > > > > main
> > > > > > > > > Kafka packages.
> > > > > > > > > This decoupling would be useful when direct dependency on
> HDFS (or
> > > > > > > other
> > > > > > > > > implementations) is not needed and would also encourage
> decoupling
> > > > > for
> > > > > > > > > other storage implementations.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Mon, Apr 1, 2019 at 3:44 AM Ambud Sharma <
> > > > > asharma52...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Harsha,
> > > > > > > > > >
> > > > > > > > > > Thank you for proposing this KIP. We are looking forward
> to this
> > > > > > > > feature as
> > > > > > > > > > well.
> > > > > > > > > >
> > > > > > > > > > A few questions around the design & implementation:
> > > > > > > > > >
> > > > > > > > > > 1. Wouldn't implicit checking for old offsets in remote
> location
> > > > > if
> > > > > > > not
> > > > > > > > > > found locally on the leader i.e. do we really need
> remote index
> > > > > > > files?
> > > > > > > > > > Since the storage path for a given topic would
> presumably be
> > > > > constant
> > > > > > > > > > across all the brokers, the remote topic-partition path
> could
> > > > > simply
> > > > > > > be
> > > > > > > > > > checked to see if there are any segment file names that
> would
> > > &g

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Viktor,
>1. Can we allow RLM Followers to serve read requests? After all segments on
the cold storage are closed ones, no modification is allowed. Besides
KIP-392 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica)
would introduce follower fetching too, so I think it would be nice to
prepare RLM for this as well.

That is a good point. We plan to support fetching remote storage from
followers too. Current code in the PR work fine for this scenario
though there may be some edge cases to be handled. We have not yet
tested this scenario.

>2. I think the remote.log.storage.enable config is redundant. By specifying
remote.log.storage.manager.class.name one already declares that they want
to use remote storage. Would it make sense to remove
the remote.log.storage.enable config?

I do not think it is really needed. `remote.log.storage.enable`
property can be removed.

Thanks,
Satish.


On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
 wrote:
>
> Hi Harsha,
>
> A couple more questions:
> 1. Can we allow RLM Followers to serve read requests? After all segments on
> the cold storage are closed ones, no modification is allowed. Besides
> KIP-392 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica)
> would introduce follower fetching too, so I think it would be nice to
> prepare RLM for this as well.
> 2. I think the remote.log.storage.enable config is redundant. By specifying
> remote.log.storage.manager.class.name one already declares that they want
> to use remote storage. Would it make sense to remove
> the remote.log.storage.enable config?
>
> Thanks,
> Viktor
>
>
> On Thu, Oct 24, 2019 at 10:37 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
> > Hi Jun & Harsha,
> >
> > I think it would be beneficial to at least provide one simple reference
> > implementation (file system based?) as we do with connect too.
> > That would as a simple example and would help plugin developers to better
> > understand the concept and the interfaces.
> >
> > Best,
> > Viktor
> >
> > On Wed, Oct 23, 2019 at 8:49 PM Jun Rao  wrote:
> >
> >> Hi, Harsha,
> >>
> >> Regarding feature branch, if the goal is faster collaboration, it seems
> >> that doing the development on your own fork is better since non-committers
> >> can push changes there.
> >>
> >> Regarding the dependencies, this is an important thing to clarify. My
> >> understanding for this KIP is that in Apache Kafka, we won't provide any
> >> specific implementation for a particular block storage. There are many
> >> block storage systems out there (HDFS, S3, Google storage, Azure storage,
> >> Ceph, etc). We don't want to drag in all those dependencies in Apache
> >> Kafka, even if they are in a separate module. Doing that will make the
> >> Kafka repo much harder to manage. We have used the same approach for
> >> connect. The connect framework is in Apache Kafka, but all specific
> >> connectors are hosted externally.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >>
> >> On Wed, Oct 23, 2019 at 8:41 AM Eno Thereska 
> >> wrote:
> >>
> >> > Thanks Satish, Harsha,
> >> >
> >> > It's probably worth it making it clearer in the KIP what exact
> >> > libraries will be added to libs, if any. The KIP specifies the remote
> >> > storage interface but it isn't clear if particular implementations
> >> > will be added to Kafka's repository or whether they will reside in
> >> > other repositories. If I understand the intention correctly, you are
> >> > proposing to have an HDFS and S3 implementation as part of the Kafka
> >> > repository working out of the box. Is that correct?
> >> >
> >> > Thanks
> >> > Eno
> >> >
> >> > On Wed, Oct 23, 2019 at 5:01 AM Satish Duggana <
> >> satish.dugg...@gmail.com>
> >> > wrote:
> >> > >
> >> > > >Regarding the HDFS dependency its not a direct dependency rather
> >> > > its implementing the RemoteStorageManager interface.
> >> > > We packaged it along with core to make it more convenient to test it.
> >> We
> >> > > can move this to external module and keep it there.
> >> > > Let me know what you think.
> >> > >
> >> > > Let me elaborate more on this point. With the new chan

[DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Satish Duggana]

Hi All,
I wrote a short KIP about avoiding out-of-sync or offline partitions
when follower fetch requests are not processed in time by the leader
replica.
KIP-501 is located at https://s.apache.org/jhbpn

Please take a look, I would like to hear your feedback and suggestions.

JIRA: https://issues.apache.org/jira/browse/KAFKA-8733

Thanks,
Satish.

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,
Thanks for looking into the updated KIP and clarifying our earlier queries.

>20. It's fine to keep the HDFS binding temporarily in the PR. We just need
to remove it before it's merged to trunk. As Victor mentioned, we can
provide a reference implementation based on a mocked version of remote
storage.

Sure, sounds good.

>21. I am not sure that I understood the need for RemoteLogIndexEntry and
its relationship with RemoteLogSegmentInfo. It seems
that RemoteLogIndexEntry are offset index entries pointing to record
batches inside a segment. That seems to be the same as the .index file?

That is a good point. `RemoteLogManager` does not put a restriction on
`RemoteStorageManager(RSM)` for maintaining positions in the remote
segment same as the local segments or keeping a correlation between
local segment's positions to the remote segment positions. RSM gives
back the respective entries for a given log segment, call RSM to fetch
the data by giving the respective entry. This allows RSM to have
better control in managing the given log segments.

Thanks,
Satish.

On Fri, Nov 1, 2019 at 2:28 AM Jun Rao  wrote:
>
> Hi, Harsha,
>
> I am still looking at the KIP and the PR. A couple of quick
> comments/questions.
>
> 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> to remove it before it's merged to trunk. As Victor mentioned, we can
> provide a reference implementation based on a mocked version of remote
> storage.
>
> 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> its relationship with RemoteLogSegmentInfo. It seems
> that RemoteLogIndexEntry are offset index entries pointing to record
> batches inside a segment. That seems to be the same as the .index file?
>
> Thanks,
>
> Jun
>
> On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> wrote:
>
> > Hi Viktor,
> > >1. Can we allow RLM Followers to serve read requests? After all segments
> > on
> > the cold storage are closed ones, no modification is allowed. Besides
> > KIP-392 (
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > )
> > would introduce follower fetching too, so I think it would be nice to
> > prepare RLM for this as well.
> >
> > That is a good point. We plan to support fetching remote storage from
> > followers too. Current code in the PR work fine for this scenario
> > though there may be some edge cases to be handled. We have not yet
> > tested this scenario.
> >
> > >2. I think the remote.log.storage.enable config is redundant. By
> > specifying
> > remote.log.storage.manager.class.name one already declares that they want
> > to use remote storage. Would it make sense to remove
> > the remote.log.storage.enable config?
> >
> > I do not think it is really needed. `remote.log.storage.enable`
> > property can be removed.
> >
> > Thanks,
> > Satish.
> >
> >
> > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> >  wrote:
> > >
> > > Hi Harsha,
> > >
> > > A couple more questions:
> > > 1. Can we allow RLM Followers to serve read requests? After all segments
> > on
> > > the cold storage are closed ones, no modification is allowed. Besides
> > > KIP-392 (
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > )
> > > would introduce follower fetching too, so I think it would be nice to
> > > prepare RLM for this as well.
> > > 2. I think the remote.log.storage.enable config is redundant. By
> > specifying
> > > remote.log.storage.manager.class.name one already declares that they
> > want
> > > to use remote storage. Would it make sense to remove
> > > the remote.log.storage.enable config?
> > >
> > > Thanks,
> > > Viktor
> > >
> > >
> > > On Thu, Oct 24, 2019 at 10:37 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com> wrote:
> > >
> > > > Hi Jun & Harsha,
> > > >
> > > > I think it would be beneficial to at least provide one simple reference
> > > > implementation (file system based?) as we do with connect too.
> > > > That would as a simple example and would help plugin developers to
> > better
> > > > understand the concept and the interfaces.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Wed, Oct 23, 2019 at 8:49 PM Jun Rao  wrote:
> > > >
> > > >> Hi, H

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,

>21. Could you elaborate a bit why the positions in remote segment is
different from the local one? I thought that they are identical copies.

They may not always be the same. Let me take an example here. If
remote storage is enabled with encryption then those local positions
may not be the same as the positions copied to remote storage.

Thanks,
Satish.


On Tue, Nov 5, 2019 at 3:46 AM Jun Rao  wrote:
>
> Hi, Satish,
>
> Thanks for the response.
>
> 21. Could you elaborate a bit why the positions in remote segment is
> different from the local one? I thought that they are identical copies.
>
> Jun
>
>
> On Fri, Nov 1, 2019 at 4:26 AM Satish Duggana 
> wrote:
>
> > Hi Jun,
> > Thanks for looking into the updated KIP and clarifying our earlier queries.
> >
> > >20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > to remove it before it's merged to trunk. As Victor mentioned, we can
> > provide a reference implementation based on a mocked version of remote
> > storage.
> >
> > Sure, sounds good.
> >
> > >21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > That is a good point. `RemoteLogManager` does not put a restriction on
> > `RemoteStorageManager(RSM)` for maintaining positions in the remote
> > segment same as the local segments or keeping a correlation between
> > local segment's positions to the remote segment positions. RSM gives
> > back the respective entries for a given log segment, call RSM to fetch
> > the data by giving the respective entry. This allows RSM to have
> > better control in managing the given log segments.
> >
> > Thanks,
> > Satish.
> >
> > On Fri, Nov 1, 2019 at 2:28 AM Jun Rao  wrote:
> > >
> > > Hi, Harsha,
> > >
> > > I am still looking at the KIP and the PR. A couple of quick
> > > comments/questions.
> > >
> > > 20. It's fine to keep the HDFS binding temporarily in the PR. We just
> > need
> > > to remove it before it's merged to trunk. As Victor mentioned, we can
> > > provide a reference implementation based on a mocked version of remote
> > > storage.
> > >
> > > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > > its relationship with RemoteLogSegmentInfo. It seems
> > > that RemoteLogIndexEntry are offset index entries pointing to record
> > > batches inside a segment. That seems to be the same as the .index file?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana  > >
> > > wrote:
> > >
> > > > Hi Viktor,
> > > > >1. Can we allow RLM Followers to serve read requests? After all
> > segments
> > > > on
> > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > KIP-392 (
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > )
> > > > would introduce follower fetching too, so I think it would be nice to
> > > > prepare RLM for this as well.
> > > >
> > > > That is a good point. We plan to support fetching remote storage from
> > > > followers too. Current code in the PR work fine for this scenario
> > > > though there may be some edge cases to be handled. We have not yet
> > > > tested this scenario.
> > > >
> > > > >2. I think the remote.log.storage.enable config is redundant. By
> > > > specifying
> > > > remote.log.storage.manager.class.name one already declares that they
> > want
> > > > to use remote storage. Would it make sense to remove
> > > > the remote.log.storage.enable config?
> > > >
> > > > I do not think it is really needed. `remote.log.storage.enable`
> > > > property can be removed.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > > >
> > > > > Hi Harsha,
> > > > >
> > > > > A couple more questions:
> > &

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Satish Duggana]

Hi Dhruvil,
Thanks for looking into the KIP.

10. I have an initial sketch of the KIP-500 in commit[a] which
discusses tracking the pending fetch requests. Tracking is not done in
Partition#readRecords because if it takes longer in reading any of the
partitions then we do not want any of the replicas of this fetch
request to go out of sync.

11. I think `Replica` class should be thread-safe to handle the remote
scenario of concurrent requests running for a follower replica. Or I
may be missing something here. This is a separate issue from KIP-500.
I will file a separate JIRA to discuss that issue.

a - 
https://github.com/satishd/kafka/commit/c69b525abe8f6aad5059236076a003cdec4c4eb7

Thanks,
Satish.

On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah  wrote:
>
> Hi Satish,
>
> Thanks for the KIP, those seems very useful. Could you elaborate on how
> pending fetch requests are tracked?
>
> Thanks,
> Dhruvil
>
> On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana 
> wrote:
>
> > Hi All,
> > I wrote a short KIP about avoiding out-of-sync or offline partitions
> > when follower fetch requests are not processed in time by the leader
> > replica.
> > KIP-501 is located at https://s.apache.org/jhbpn
> >
> > Please take a look, I would like to hear your feedback and suggestions.
> >
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-8733
> >
> > Thanks,
> > Satish.
> >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>Depends on the implementation, the data of one segment may not necessary be
stored in a single file.
There could be a maximum object / chunk / file size restriction on the
remote storage. So, one Kafka
segment could be saved in multiple chunks in remote storage.

Having one local segment can be stored in multiple files and each file
can have a base position as part of the metadata(like name) of file or
object etc.
File/object name can be --. So
any read request for a position with in that segment can be found by
computing relative position viz `fetchPosition-basePosition`.



On Thu, Nov 7, 2019 at 6:04 AM Ying Zheng  wrote:
>
> 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> its relationship with RemoteLogSegmentInfo. It seems
> that RemoteLogIndexEntry are offset index entries pointing to record
> batches inside a segment. That seems to be the same as the .index file?
>
> We do not assume the how the data is stored in the remote storage.
> Depends on the implementation, the data of one segment may not necessary be
> stored in a single file.
> There could be a maximum object / chunk / file size restriction on the
> remote storage. So, one Kafka
> segment could be saved in multiple chunks in remote storage.
>
> The remote log index also have a larger index interval. The default
> interval of the local .index file
> (log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
> the default remote
> index interval (hdfs.remote.index.interval.bytes) is 256KB. The
> coarse-grained remote index saves
> some local disk space. The smaller size also makes it more likely to be
> cached in physical memory.
>
>
>
>
> On Thu, Oct 31, 2019 at 1:58 PM Jun Rao  wrote:
>
> > Hi, Harsha,
> >
> > I am still looking at the KIP and the PR. A couple of quick
> > comments/questions.
> >
> > 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > to remove it before it's merged to trunk. As Victor mentioned, we can
> > provide a reference implementation based on a mocked version of remote
> > storage.
> >
> > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> > wrote:
> >
> > > Hi Viktor,
> > > >1. Can we allow RLM Followers to serve read requests? After all segments
> > > on
> > > the cold storage are closed ones, no modification is allowed. Besides
> > > KIP-392 (
> > >
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D392-253A-2BAllow-2Bconsumers-2Bto-2Bfetch-2Bfrom-2Bclosest-2Breplica&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=HTPACirRO-wVmOHmGEMlTIAov4szGHn38xrbFbMZK_I&e=
> > > )
> > > would introduce follower fetching too, so I think it would be nice to
> > > prepare RLM for this as well.
> > >
> > > That is a good point. We plan to support fetching remote storage from
> > > followers too. Current code in the PR work fine for this scenario
> > > though there may be some edge cases to be handled. We have not yet
> > > tested this scenario.
> > >
> > > >2. I think the remote.log.storage.enable config is redundant. By
> > > specifying
> > >
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__remote.log.storage.manager.class.name&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=g7ujYPRBvNrON18SBeCt4g&m=CKNMp77DfMghjYo1JqbWr5jl-DRDBGF2owao5zUXDeE&s=QsUunkBFX3dne_4caCiEAbp9xKUPrFx1srwznOR_Sfc&e=
> > one already declares that they want
> > > to use remote storage. Would it make sense to remove
> > > the remote.log.storage.enable config?
> > >
> > > I do not think it is really needed. `remote.log.storage.enable`
> > > property can be removed.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Thu, Oct 24, 2019 at 2:46 PM Viktor Somogyi-Vass
> > >  wrote:
> > > >
> > > > Hi Harsha,
> > > >
> > > > A couple more questions:
> > > > 1. Can we allow RLM Followers to serve read requests? After all
> > segments
> > > on
> > > > the cold storage are closed ones, no mo

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>So that means a consumer which gets behind by half an hour will find its
reads being served from remote storage. And, if I understand the proposed
algorithm, each such consumer fetch request could result in a separate
fetch request from the remote storage. I.e. there's no mechanism to
amortize the cost of the fetching between multiple consumers fetching
similar ranges?

local log segments are deleted according to the local
log.retention.time/.size settings though they may have been already
copied to remote storage. Consumers would still be able to fetch the
messages from local storage if they are not yet deleted based on the
retention. They will be served from remote storage only when they are
not locally available.

Thanks,
Satish.

On Thu, Nov 7, 2019 at 7:58 AM Tom Bentley  wrote:
>
> Hi Ying,
>
> Because only inactive segments can be shipped to remote storage, to be able
> > to ship log data as soon
> > as possible, we will roll log segment very fast (e.g. every half hour).
> >
>
> So that means a consumer which gets behind by half an hour will find its
> reads being served from remote storage. And, if I understand the proposed
> algorithm, each such consumer fetch request could result in a separate
> fetch request from the remote storage. I.e. there's no mechanism to
> amortize the cost of the fetching between multiple consumers fetching
> similar ranges?
>
> (Actually the doc for RemoteStorageManager.read() says "It will read at
> least one batch, if the 1st batch size is larger than maxBytes.". Does that
> mean the broker might have to retry with increased maxBytes if the first
> request fails to read a batch? If so, how does it know how much to increase
> maxBytes by?)
>
> Thanks,
>
> Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Tom,'
Sorry, I missed the other question.

>(Actually the doc for RemoteStorageManager.read() says "It will read at
least one batch, if the 1st batch size is larger than maxBytes.". Does that
mean the broker might have to retry with increased maxBytes if the first
request fails to read a batch? If so, how does it know how much to increase
maxBytes by?)

broker or RemoteLogManager does not need to retry here.
RemoteStorageManager can return `Records` which can be more than
maxBytes if the first batch containing startOffset has more than
maxBytes.

Thanks,
Satish.

On Thu, Nov 7, 2019 at 8:33 AM Satish Duggana  wrote:
>
> >So that means a consumer which gets behind by half an hour will find its
> reads being served from remote storage. And, if I understand the proposed
> algorithm, each such consumer fetch request could result in a separate
> fetch request from the remote storage. I.e. there's no mechanism to
> amortize the cost of the fetching between multiple consumers fetching
> similar ranges?
>
> local log segments are deleted according to the local
> log.retention.time/.size settings though they may have been already
> copied to remote storage. Consumers would still be able to fetch the
> messages from local storage if they are not yet deleted based on the
> retention. They will be served from remote storage only when they are
> not locally available.
>
> Thanks,
> Satish.
>
> On Thu, Nov 7, 2019 at 7:58 AM Tom Bentley  wrote:
> >
> > Hi Ying,
> >
> > Because only inactive segments can be shipped to remote storage, to be able
> > > to ship log data as soon
> > > as possible, we will roll log segment very fast (e.g. every half hour).
> > >
> >
> > So that means a consumer which gets behind by half an hour will find its
> > reads being served from remote storage. And, if I understand the proposed
> > algorithm, each such consumer fetch request could result in a separate
> > fetch request from the remote storage. I.e. there's no mechanism to
> > amortize the cost of the fetching between multiple consumers fetching
> > similar ranges?
> >
> > (Actually the doc for RemoteStorageManager.read() says "It will read at
> > least one batch, if the 1st batch size is larger than maxBytes.". Does that
> > mean the broker might have to retry with increased maxBytes if the first
> > request fails to read a batch? If so, how does it know how much to increase
> > maxBytes by?)
> >
> > Thanks,
> >
> > Tom

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

>>Depends on the implementation, the data of one segment may not necessary be
stored in a single file.
There could be a maximum object / chunk / file size restriction on the
remote storage. So, one Kafka
segment could be saved in multiple chunks in remote storage.

>Having one local segment can be stored in multiple files and each file
can have a base position as part of the metadata(like name) of file or
object etc.
File/object name can be --. So
any read request for a position with in that segment can be found by
computing relative position viz `fetchPosition-basePosition`.

Let me elaborate further on how to address a single local segment file
being copied to multiple files/blocks in remote storage without the
need to map local segment positions to remote segment positions.
Let us say a local segment file has offsets from 1000-95000. This may
be copied to remote storage in multiple files/blocks. Each file or
block can be created with name or any other metadata containing
--. This does not require
recomputing positions for the remote segments.

local segment file has offsets: 1000 - 95000

remote segment file suffix format can be :
--
remote-segment-file-1: 1000-20200-0
remote-segment-file-2: 20201-45003-942346
remote-segment-file-3: 45004-78008-6001235
remote-segment-file-4: 78009-95000-20024761

If a read comes for 52340 offset and position as 7321236, relative
position in remote segment-3 is: 7321236-6001235 = 1320001

Thanks,
Satish.

On Thu, Nov 7, 2019 at 7:55 AM Satish Duggana  wrote:
>
> >Depends on the implementation, the data of one segment may not necessary be
> stored in a single file.
> There could be a maximum object / chunk / file size restriction on the
> remote storage. So, one Kafka
> segment could be saved in multiple chunks in remote storage.
>
> Having one local segment can be stored in multiple files and each file
> can have a base position as part of the metadata(like name) of file or
> object etc.
> File/object name can be --. So
> any read request for a position with in that segment can be found by
> computing relative position viz `fetchPosition-basePosition`.
>
>
>
> On Thu, Nov 7, 2019 at 6:04 AM Ying Zheng  wrote:
> >
> > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > its relationship with RemoteLogSegmentInfo. It seems
> > that RemoteLogIndexEntry are offset index entries pointing to record
> > batches inside a segment. That seems to be the same as the .index file?
> >
> > We do not assume the how the data is stored in the remote storage.
> > Depends on the implementation, the data of one segment may not necessary be
> > stored in a single file.
> > There could be a maximum object / chunk / file size restriction on the
> > remote storage. So, one Kafka
> > segment could be saved in multiple chunks in remote storage.
> >
> > The remote log index also have a larger index interval. The default
> > interval of the local .index file
> > (log.index.interval.bytes) is 4KB. In the current HDFS RSM implementation,
> > the default remote
> > index interval (hdfs.remote.index.interval.bytes) is 256KB. The
> > coarse-grained remote index saves
> > some local disk space. The smaller size also makes it more likely to be
> > cached in physical memory.
> >
> >
> >
> >
> > On Thu, Oct 31, 2019 at 1:58 PM Jun Rao  wrote:
> >
> > > Hi, Harsha,
> > >
> > > I am still looking at the KIP and the PR. A couple of quick
> > > comments/questions.
> > >
> > > 20. It's fine to keep the HDFS binding temporarily in the PR. We just need
> > > to remove it before it's merged to trunk. As Victor mentioned, we can
> > > provide a reference implementation based on a mocked version of remote
> > > storage.
> > >
> > > 21. I am not sure that I understood the need for RemoteLogIndexEntry and
> > > its relationship with RemoteLogSegmentInfo. It seems
> > > that RemoteLogIndexEntry are offset index entries pointing to record
> > > batches inside a segment. That seems to be the same as the .index file?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 28, 2019 at 9:11 PM Satish Duggana 
> > > wrote:
> > >
> > > > Hi Viktor,
> > > > >1. Can we allow RLM Followers to serve read requests? After all 
> > > > >segments
> > > > on
> > > > the cold storage are closed ones, no modification is allowed. Besides
> > > > KIP-392 (
> > > >
> > > >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

[Satish Duggana]

+1 (non-binding)

On Thu, Nov 7, 2019 at 8:58 PM Ismael Juma  wrote:
>
> +1 (binding)
>
> On Thu, Oct 24, 2019 at 9:33 PM radai  wrote:
>
> > Hello,
> >
> > I'd like to initiate a vote on KIP-514.
> >
> > links:
> > the kip -
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > the PR - https://github.com/apache/kafka/pull/7569
> >
> > Thank you
> >

Re: [ANNOUNCE] New committer: Mickael Maison

[Satish Duggana]

Congratulations Mickael!!

On Fri, Nov 8, 2019 at 2:50 PM Rajini Sivaram  wrote:
>
> Congratulations, Mickael, well deserved!!
>
> Regards,
>
> Rajini
>
> On Fri, Nov 8, 2019 at 9:08 AM David Jacot  wrote:
>
> > Congrats Mickeal, well deserved!
> >
> > On Fri, Nov 8, 2019 at 8:56 AM Tom Bentley  wrote:
> >
> > > Congratulations Mickael!
> > >
> > > On Fri, Nov 8, 2019 at 6:41 AM Vahid Hashemian <
> > vahid.hashem...@gmail.com>
> > > wrote:
> > >
> > > > Congrats Mickael,
> > > >
> > > > Well deserved!
> > > >
> > > > --Vahid
> > > >
> > > > On Thu, Nov 7, 2019 at 9:10 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > > > Congratulations Mickael!
> > > > >
> > > > > On Thu, Nov 7, 2019 at 8:27 PM Manikumar 
> > > > > wrote:
> > > > >
> > > > > > Congrats Mickeal!
> > > > > >
> > > > > > On Fri, Nov 8, 2019 at 9:05 AM Dong Lin 
> > wrote:
> > > > > >
> > > > > > > Congratulations Mickael!
> > > > > > >
> > > > > > > On Thu, Nov 7, 2019 at 1:38 PM Jun Rao  wrote:
> > > > > > >
> > > > > > > > Hi, Everyone,
> > > > > > > >
> > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > committer
> > > > > > > > Mickael
> > > > > > > > Maison.
> > > > > > > >
> > > > > > > > Mickael has been contributing to Kafka since 2016. He proposed
> > > and
> > > > > > > > implemented multiple KIPs. He has also been propomating Kafka
> > > > through
> > > > > > > blogs
> > > > > > > > and public talks.
> > > > > > > >
> > > > > > > > Congratulations, Mickael!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > >
> >

Re: Subject: [VOTE] 2.2.2 RC2

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 2.3.0-rc2
tag.
- Ran a few internal apps with ~2500 topic-partitions on 5 a node cluster.

Thanks,
Satish.

On Sat, Nov 9, 2019 at 4:41 AM Bill Bejeck  wrote:

>- verified signatures
>- built it from source
>- ran unit tests
>
> +1 (non-binding)
>
> Thanks for running the release, Randall!
>
> -Bill
>
> On Fri, Nov 8, 2019 at 4:40 PM David Arthur  wrote:
>
> > * Glanced through docs, release notes
> > * Downloaded RC2 binaries, verified signatures
> > * Ran through quickstart
> >
> > +1 binding
> >
> > Thanks for managing this release, Randall!
> >
> > -David
> >
> > On Wed, Nov 6, 2019 at 7:39 PM Eric Lalonde  wrote:
> >
> > > Hello,
> > >
> > > In an effort to assist in the verification of release candidates, I
> have
> > > authored the following quick-and-dirty utility to help people verify
> > > release candidate artifacts:
> > > https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh <
> > > https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh>
> .
> > I
> > > have executed this script for 2.2.2 rc2 and everything looks good:
> > > - all checksums verify
> > > - all executed gradle commands succeed
> > > - all unit and integration tests pass.
> > >
> > > Hope this helps in the release of 2.2.2.
> > >
> > > - Eric
> > >
> > > > On Nov 5, 2019, at 7:55 AM, Randall Hauch  wrote:
> > > >
> > > > Thanks, Mickael!
> > > >
> > > > Anyone else get a chance to validate the 2.2.2 RC2 build? It'd be
> great
> > > to
> > > > get this out the door.
> > > >
> > > > Randall
> > > >
> > > > On Tue, Nov 5, 2019 at 6:34 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> +1 (non binding)
> > > >> I verified signatures, built it from source, ran unit tests and
> > > quickstart
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Oct 25, 2019 at 3:10 PM Randall Hauch 
> > wrote:
> > > >>>
> > > >>> Hello all, we identified around three dozen bug fixes, including an
> > > >> update
> > > >>> of a third party dependency, and wanted to release a patch release
> > for
> > > >> the
> > > >>> Apache Kafka 2.2.0 release.
> > > >>>
> > > >>> This is the *second* candidate for release of Apache Kafka 2.2.2.
> > (RC1
> > > >> did
> > > >>> not include a fix for
> > https://issues.apache.org/jira/browse/KAFKA-9053
> > > ,
> > > >> but
> > > >>> the fix appeared before RC1 was announced so it was easier to just
> > > create
> > > >>> RC2.)
> > > >>>
> > > >>> Check out the release notes for a complete list of the changes in
> > this
> > > >>> release candidate:
> > > >>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/RELEASE_NOTES.html
> > > >>>
> > > >>> *** Please download, test and vote by Wednesday, October 30, 9am
> PT>
> > > >>>
> > > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >>> https://kafka.apache.org/KEYS
> > > >>>
> > > >>> * Release artifacts to be voted upon (source and binary):
> > > >>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/
> > > >>>
> > > >>> * Maven artifacts to be voted upon:
> > > >>>
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>>
> > > >>> * Javadoc:
> > > >>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/
> > > >>>
> > > >>> * Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag:
> > > >>> https://github.com/apache/kafka/releases/tag/2.2.2-rc2
> > > >>>
> > > >>> * Documentation:
> > > >>> https://kafka.apache.org/22/documentation.html
> > > >>>
> > > >>> * Protocol:
> > > >>> https://kafka.apache.org/22/protocol.html
> > > >>>
> > > >>> * Successful Jenkins builds for the 2.2 branch:
> > > >>> Unit/integration tests:
> > > https://builds.apache.org/job/kafka-2.2-jdk8/1/
> > > >>> System tests:
> > > >>> https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/
> > > >>>
> > > >>> /**
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Randall Hauch
> > > >>
> > >
> > >
> >
> > --
> > David Arthur
> >
>

Re: [ANNOUNCE] New committer: Boyang Chen

[Satish Duggana]

Congratulations Boyang!

On Fri, Jun 26, 2020 at 8:36 PM David Jacot  wrote:
>
> Congrats, Boyang!
>
> On Thu, Jun 25, 2020 at 1:57 PM Viktor Somogyi-Vass 
> wrote:
>
> > Congrats :)
> >
> > On Thu, Jun 25, 2020 at 12:28 AM Liquan Pei  wrote:
> >
> > > Congrats!
> > >
> > > On Wed, Jun 24, 2020 at 9:42 AM Raymond Ng  wrote:
> > >
> > > > Congrats Boyang! Look forward to more awesome contributions from you in
> > > the
> > > > future.
> > > >
> > > > Regards,
> > > > Ray
> > > >
> > > > On Wed, Jun 24, 2020 at 6:07 AM Ismael Juma  wrote:
> > > >
> > > > > Congratulations Boyang!
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jun 22, 2020 at 4:26 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > The PMC for Apache Kafka has invited Boyang Chen as a committer and
> > > we
> > > > > are
> > > > > > pleased to announce that he has accepted!
> > > > > >
> > > > > > Boyang has been active in the Kafka community more than two years
> > > ago.
> > > > > > Since then he has presented his experience operating with Kafka
> > > Streams
> > > > > at
> > > > > > Pinterest as well as several feature development including
> > rebalance
> > > > > > improvements (KIP-345) and exactly-once scalability improvements
> > > > > (KIP-447)
> > > > > > in various Kafka Summit and Kafka Meetups. More recently he's also
> > > been
> > > > > > participating in Kafka broker development including post-Zookeeper
> > > > > > controller design (KIP-500). Besides all the code contributions,
> > > Boyang
> > > > > has
> > > > > > also helped reviewing even more PRs and KIPs than his own.
> > > > > >
> > > > > > Thanks for all the contributions Boyang! And look forward to more
> > > > > > collaborations with you on Apache Kafka.
> > > > > >
> > > > > >
> > > > > > -- Guozhang, on behalf of the Apache Kafka PMC
> > > > > >
> > > > >
> > > >
> > > --
> > > Liquan Pei
> > > Software Engineer, Confluent Inc
> > >
> >

Re: [ANNOUNCE] New committer: Xi Hu

[Satish Duggana]

Congratulations Xi!

On Mon, Jun 29, 2020 at 10:41 AM Konstantine Karantasis
 wrote:
>
> Congrats Xi!
>
> -Konstantine
>
> On Sat, Jun 27, 2020 at 7:13 PM Matt Wang  wrote:
>
> > Congratulations ~
> >
> >
> > --
> >
> > Best,
> > Matt Wang
> >
> >
> > On 06/26/2020 23:06，David Jacot wrote：
> > Congrats!
> >
> > On Thu, Jun 25, 2020 at 4:08 PM Hu Xi  wrote:
> >
> > Thank you, everyone. It is my great honor to be a part of the community.
> > Will make a greater contribution in the coming days.
> >
> > 
> > 发件人: Roc Marshal 
> > 发送时间: 2020年6月25日 10:20
> > 收件人: us...@kafka.apache.org 
> > 主题: Re:Re: [ANNOUNCE] New committer: Xi Hu
> >
> > Congratulations ! Xi Hu.
> >
> >
> > Best,
> > Roc Marshal.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2020-06-25 01:30:33, "Boyang Chen"  wrote:
> > Congratulations Xi! Well deserved.
> >
> > On Wed, Jun 24, 2020 at 10:10 AM AJ Chen  wrote:
> >
> > Congratulations, Xi.
> > -aj
> >
> >
> >
> > On Wed, Jun 24, 2020 at 9:27 AM Guozhang Wang 
> > wrote:
> >
> > The PMC for Apache Kafka has invited Xi Hu as a committer and we are
> > pleased to announce that he has accepted!
> >
> > Xi Hu has been actively contributing to Kafka since 2016, and is well
> > recognized especially for his non-code contributions: he maintains a
> > tech
> > blog post evangelizing Kafka in the Chinese speaking community (
> > https://www.cnblogs.com/huxi2b/), and is one of the most active
> > answering
> > member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
> > presented in Kafka meetup events in the past and authored a
> > book deep-diving on Kafka architecture design and operations as well (
> > https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed
> > 75
> > patches so far.
> >
> >
> > Thanks for all the contributions Xi. Congratulations!
> >
> > -- Guozhang, on behalf of the Apache Kafka PMC
> >
> >
> >
> >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

sign details in the KIP, including how to
> > >> keep consistency between replicas (especially when there is leadership
> > >> changes / log truncations) and new metrics. We also made some other minor
> > >> changes in the doc. We will finish the KIP changes in the next couple of
> > >> days. We will let you know when we are done. Most of the changes are
> > >> already updated to the wiki KIP. You can take a look. But it's not the
> > >> final version yet.
> > >>
> > >> As for the implementation, the code is mostly done and we already had 
> > >> some
> > >> feature tests / system tests. I have added the performance test results 
> > >> in
> > >> the KIP. However the recent design changes (e.g. leader epoch info
> > >> management / log truncation / some of the new metrics) have not been
> > >> implemented yet. It will take about 2 weeks for us to implement after you
> > >> review and agree with those design changes.
> > >>
> > >>
> > >>
> > >> On Tue, Jul 7, 2020 at 9:23 AM Jun Rao  wrote:
> > >>
> > >> > Hi, Satish, Harsha,
> > >> >
> > >> > Any new updates on the KIP? This feature is one of the most important
> > >> and
> > >> > most requested features in Apache Kafka right now. It would be helpful
> > >> if
> > >> > we can make sustained progress on this. Could you share how far along 
> > >> > is
> > >> > the design/implementation right now? Is there anything that other 
> > >> > people
> > >> > can help to get it across the line?
> > >> >
> > >> > As for "transactional support" and "follower requests/replication", no
> > >> > further comments from me as long as the producer state and leader epoch
> > >> can
> > >> > be restored properly from the object store when needed.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Tue, Jun 9, 2020 at 3:39 AM Satish Duggana  > >> >
> > >> > wrote:
> > >> >
> > >> > > We did not want to add many implementation details in the KIP. But we
> > >> > > decided to add them in the KIP as appendix or sub-sections(including
> > >> > > follower fetch protocol) to describe the flow with the main cases.
> > >> > > That will answer most of the queries. I will update on this mail
> > >> > > thread when the respective sections are updated.
> > >> > >
> > >> > > Thanks,
> > >> > > Satish.
> > >> > >
> > >> > > On Sat, Jun 6, 2020 at 7:49 PM Alexandre Dupriez
> > >> > >  wrote:
> > >> > > >
> > >> > > > Hi Satish,
> > >> > > >
> > >> > > > A couple of questions specific to the section "Follower
> > >> > > > Requests/Replication", pages 16:17 in the design document [1].
> > >> > > >
> > >> > > > 900. It is mentioned that followers fetch auxiliary states from the
> > >> > > > remote storage.
> > >> > > >
> > >> > > > 900.a Does the consistency model of the external storage impacts
> > >> reads
> > >> > > > of leader epochs and other auxiliary data?
> > >> > > >
> > >> > > > 900.b What are the benefits of using a mechanism to store and 
> > >> > > > access
> > >> > > > the leader epochs which is different from other metadata associated
> > >> to
> > >> > > > tiered segments? What are the benefits of retrieving this
> > >> information
> > >> > > > on-demand from the follower rather than relying on propagation via
> > >> the
> > >> > > > topic __remote_log_metadata? What are the advantages over using a
> > >> > > > dedicated control structure (e.g. a new record type) propagated via
> > >> > > > this topic? Since in the document, different control paths are
> > >> > > > operating in the system, how are the metadata stored in
> > >> > > > __remote_log_metadata [which also include the epoch of the leader
> > >> > > > which offloaded