Transitioning from Java 10 to Java 11 in Jenkins

[Ismael Juma]

Hi all,

Java 11 was released recently and Java 10 is no longer supported. Java 11
is the first LTS release since the new Java release cadence was announced.
As of today, all of our tests pass with Java 11 so it's time to transition
Jenkins builds to use Java 11 instead of Java 10. I have updated the trunk
job[1] and will update the PR job in a couple of days to give time for PRs
to be rebased to include the required commits.

Let me know if you have any questions.

Ismael

[1] https://builds.apache.org/job/kafka-trunk-jdk11/

Re: Transitioning from Java 10 to Java 11 in Jenkins

[Ismael Juma]

Updated PR jobs, they are now configured in the following way:

https://builds.apache.org/job/kafka-pr-jdk7-scala2.10: Only enabled for
0.10.0, 0.10.1, 0.10.2 branches as Scala 2.10 support was dropped in 0.11.0.
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11: Only enabled for
0.11.0, 1.0, 1.1 branches as Java 7 support was dropped in 2.0 and we have
kafka-pr-jdk7-scala2.10 for 0.10.x branches.
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11: Enabled for all
branches.
https://builds.apache.org/job/kafka-pr-jdk10-scala2.12: Only enabled for
1.0, 1.1 and 2.0 branches. Java 9/10 support was added in 1.0 and
kafka-pr-jdk11-scala2.12 replaced this in 2.1 and newer.
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12: Enabled for trunk,
2.1 and newer release branches. Java 11 support was added in 2.1.

Ismael

On Wed, Oct 10, 2018 at 1:47 PM Ismael Juma  wrote:

> Hi all,
>
> Java 11 was released recently and Java 10 is no longer supported. Java 11
> is the first LTS release since the new Java release cadence was announced.
> As of today, all of our tests pass with Java 11 so it's time to transition
> Jenkins builds to use Java 11 instead of Java 10. I have updated the trunk
> job[1] and will update the PR job in a couple of days to give time for PRs
> to be rebased to include the required commits.
>
> Let me know if you have any questions.
>
> Ismael
>
> [1] https://builds.apache.org/job/kafka-trunk-jdk11/
>

Re: [ANNOUNCE] New Committer: Manikumar Reddy

[Ismael Juma]

Congratulations Manikumar! Thanks for your continued contributions.

Ismael

On Thu, Oct 11, 2018 at 10:39 AM Jason Gustafson  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Manikumar Reddy as a committer and we
> are
> pleased to announce that he has accepted!
>
> Manikumar has contributed 134 commits including significant work to add
> support for delegation tokens in Kafka:
>
> KIP-48:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> KIP-249
> 
> :
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
>
> He has broad experience working with many of the core components in Kafka
> and he has reviewed over 80 PRs. He has also made huge progress addressing
> some of our technical debt.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Manikumar!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: Build failed in Jenkins: kafka-trunk-jdk8 #3137

[Ismael Juma]

I configured Jenkins to delete the workspace once as it looks like the Git
checkout was corrupted.

Ismael

On Sun, Oct 14, 2018 at 7:13 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
> --
> Started by an SCM change
> [EnvInject] - Loading node environment variables.
> Building remotely on ubuntu-eu2 (ubuntu trusty) in workspace <
> https://builds.apache.org/job/kafka-trunk-jdk8/ws/>
>  > git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>  > git config remote.origin.url https://github.com/apache/kafka.git #
> timeout=10
> Fetching upstream changes from https://github.com/apache/kafka.git
>  > git --version # timeout=10
>  > git fetch --tags --progress https://github.com/apache/kafka.git
> +refs/heads/*:refs/remotes/origin/*
> ERROR: Error fetching remote repo 'origin'
> hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/kafka.git
> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
> at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
> at hudson.scm.SCM.checkout(SCM.java:504)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> at hudson.model.Run.execute(Run.java:1794)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:429)
> Caused by: hudson.plugins.git.GitException: Command "git fetch --tags
> --progress https://github.com/apache/kafka.git
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> stdout:
> stderr: error: Could not read 6e79e5da0308783ba646378efc44f018cb4f39ac
> error: Could not read bb745c0f9142717ddf68dc83bbd940dfe0c59b9a
> remote: Enumerating objects: 3878, done.
> remote: Counting objects:   0% (1/3878)   remote: Counting
> objects:   1% (39/3878)   remote: Counting objects:   2% (78/3878)
>  remote: Counting objects:   3% (117/3878)   remote:
> Counting objects:   4% (156/3878)   remote: Counting objects:   5%
> (194/3878)   remote: Counting objects:   6% (233/3878)
>  remote: Counting objects:   7% (272/3878)   remote: Counting
> objects:   8% (311/3878)   remote: Counting objects:   9%
> (350/3878)   remote: Counting objects:  10% (388/3878)
>  remote: Counting objects:  11% (427/3878)   remote: Counting
> objects:  12% (466/3878)   remote: Counting objects:  13%
> (505/3878)   remote: Counting objects:  14% (543/3878)
>  remote: Counting objects:  15% (582/3878)   remote: Counting
> objects:  16% (621/3878)   remote: Counting objects:  17%
> (660/3878)   remote: Counting objects:  18% (699/3878)
>  remote: Counting objects:  19% (737/3878)   remote: Counting
> objects:  20% (776/3878)   remote: Counting objects:  21%
> (815/3878)   remote: Counting objects:  22% (854/3878)
>  remote: Counting objects:  23% (892/3878)   remote: Counting
> objects:  24% (931/3878)   remote: Counting objects:  25%
> (970/3878)   remote: Counting objects:  26% (1009/3878)
>  remote: Counting objects:  27% (1048/3878)   remote: Counting
> objects:  28% (1086/3878)   remote: Counting objects:  29%
> (1125/3878)   remote: Counting objects:  30% (1164/3878)
>  remote: Counting objects:  31% (1203/3878)   remote: Counting
> objects:  32% (1241/3878)   remote: Counting objects:  33%
> (1280/3878)   remote: Counting objects:  34% (1319/3878)
>  remote: Counting objects:  35% (1358/3878)   remote: Counting
> objects:  36% (1397/3878)   remote: Counting objects:  37%
> (1435/3878)   remote: Counting objects:  38% (1474/3878)
>  remote: Counting objects:  39% (1513/3878)   remote: Counting
> objects:  40% (1552/3878)   remote: Counting objects:  41%
> (1590/3878)   remote: Counting objects:  42% (1629/3878)
>  remote: Counting objects:  43% (1668/3878)   remote: Counting
> objects:  44% (1707/3878)   remote: Counting objects:  45%
> (1746/3878)   remote: Counting objects:  46% (1784/3878)
>  remote: Counting objects:  47% (1823/3878)   remote: Counting
> objects:  48% (1862/3878)   remote: Counting objects:  49%
> (1901/3878)   remote: Counting objects:  50% (1939/3878)
>  remote: Counting objects:  51% (1978/3878) 

Re: [DISCUSS] 2.0.1 bug fix release

[Ismael Juma]

Thanks for managing the release Manikumar!

Ismael

On Tue, 16 Oct 2018, 12:13 Manikumar,  wrote:

> Hi all,
>
> I would like to volunteer to be the release manager for 2.0.1 bug fix
> release.
> 2.0 was released July 30, 2018 and 44 issues are fixed so far.
>
> Please find all the resolved tickets here:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.0.1
>
> Please find the Release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>
> If you have any JIRA in progress and would like to include it in this
> release, please discuss with your reviewer.
> There is currently only one blocking issue (
> https://issues.apache.org/jira/browse/KAFKA-7464).
>
> Next week, Once the blocking issue gets addressed,  I plan to create the
> first RC for 2.0.1 release.
>
> Thanks,
> Manikumar
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[Ismael Juma]

Thanks for the KIP. Can you please elaborate on the need for the key in
this case? The KIP simply states that the key is needed for metadata, but
doesn't give any more details.

Ismael

On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:

> Hello,
>
> I have made necessary changes as per the original discussion thread, and
> would like to put it for votes.
>
> Thank you very much for your suggestion and guidance so far.
>
> Regards,
>

Re: Throwing away prefetched records optimisation.

[Ismael Juma]

Hi,

I think a KIP to discuss a concrete proposal makes sense. One suggestion is
to explore the possibility of fixing the issue without a new config. Would
that break existing users? Generally, we should strive for avoiding configs
if at all possible.

Ismael

On 16 Oct 2018 12:30 am, "Zahari Dichev"  wrote:

Hi there Kafka developers,

I am currently trying to find a solution to an issue that has been
manifesting itself in the Akka streams implementation of the Kafka
connector. When it comes to consuming messages, the implementation relies
heavily on the fact that we can pause and resume partitions. In some
situations when a single consumer instance is shared among several streams,
we might end up with frequently pausing and unpausing a set of topic
partitions, which is the main facility that allows us to implement back
pressure. This however has certain disadvantages, especially when there are
two consumers that differ in terms of processing speed.

To articulate the issue more clearly, imagine that a consumer maintains
assignments for two topic partitions *TP1* and *TP2*. This consumer is
shared by two streams - S1 and S2. So effectively when we have demand from
only one of the streams - *S1*, we will pause one of the topic partitions
*TP2* and call *poll()* on the consumer to only retrieve the records for
the demanded topic partition - *TP1*. The result of that is all the records
that have been prefetched for *TP2* are now thrown away by the fetcher
("*Not
returning fetched records for assigned partition TP2 since it is no longer
fetchable"*). If we extrapolate that to multiple streams sharing the same
consumer, we might quickly end up in a situation where we throw prefetched
data quite often. This does not seem like the most efficient approach and
in fact produces quite a lot of overlapping fetch requests as illustrated
in the following issue:

https://github.com/akka/alpakka-kafka/issues/549

I am writing this email to get some initial opinion on a KIP I was thinking
about. What if we give the clients of the Consumer API a bit more control
of what to do with this prefetched data. Two options I am wondering about:

1. Introduce a configuration setting, such as*
"return-prefetched-data-for-paused-topic-partitions = false"* (have to
think of a better name), which when set to true will return what is
prefetched instead of throwing it away on calling *poll()*. Since this is
amount of data that is bounded by the maximum size of the prefetch, we can
control what is the most amount of records returned. The client of the
consumer API can then be responsible for keeping that data around and use
it when appropriate (i.e. when demand is present)

2. Introduce a facility to pass in a buffer into which the prefetched
records are drained when poll is called and paused partitions have some
prefetched records.

Any opinions on the matter are welcome. Thanks a lot !


Zahari Dichev

Re: [VOTE] 2.0.1 RC0

[Ismael Juma]

Since that was just a system tests fix, it doesn't seem to require another
RC. We just need a couple more votes. I'll ping some PMC members.

Ismael

On Wed, Nov 7, 2018 at 5:01 AM Manikumar  wrote:

> KAFKA-7581, KAFKA-7579 are not blockers for 2.0.1 release. KAFKA-7579 got
> fixed on 2.0 branch.
> This can be part of 2.0.1, if we are going with another RC.
>
> We need couple of more PMC votes to pass this vote thread.
>
> On Wed, Nov 7, 2018 at 4:43 PM Eno Thereska 
> wrote:
>
> > Two JIRAs are still marked as blockers, although it's not clear to me if
> > they really are. Any update?
> > Thanks
> > Eno
> >
> > On Thu, Nov 1, 2018 at 5:10 PM Manikumar 
> > wrote:
> >
> > > We were waiting for the system test results. There were few failures:
> > > KAFKA-7579,  KAFKA-7559, KAFKA-7561
> > > they are not blockers for 2.0.1 release. We need more votes from
> > > PMC/committers :)
> > >
> > > Thanks Stanislav! for the system test results.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Thu, Nov 1, 2018 at 10:20 PM Eno Thereska 
> > > wrote:
> > >
> > > > Anything else holding this up?
> > > >
> > > > Thanks
> > > > Eno
> > > >
> > > > On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz 
> wrote:
> > > >
> > > > > +1 (non-binding) ... I used the staged binaries and run tests with
> > > > > different clients.
> > > > >
> > > > > On Fri, Oct 26, 2018 at 4:29 AM Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the first candidate for release of Apache Kafka 2.0.1.
> > > > > >
> > > > > > This is a bug fix release closing 49 tickets:
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > > > > >
> > > > > > Release notes for the 2.0.1 release:
> > > > > >
> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by  Tuesday, October 30, end
> of
> > > day
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > http://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > > https://repository.apache.org/content/groups/staging/
> > > > > >
> > > > > > * Javadoc:
> > > > > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > > > > >
> > > > > > * Documentation:
> > > > > > http://kafka.apache.org/20/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > http://kafka.apache.org/20/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 2.0 branch:
> > > > > > Unit/integration tests:
> > > > > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > > > > >
> > > > > > /**
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] 2.0.1 RC0

[Ismael Juma]

Manikumar, looks like you have the necessary votes. :)

Ismael

On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > Checked signatures, build and quickstart.
> >
> > Thank you for managing the release, Mani!
> >
> >
> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
> > wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 2.0.1.
> > >
> > > This is a bug fix release closing 49 tickets:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > >
> > > Release notes for the 2.0.1 release:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by  Tuesday, October 30, end of day
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > >
> > > * Documentation:
> > > http://kafka.apache.org/20/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/20/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > >
> > > /**
> > >
> > > Thanks,
> > > Manikumar
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[Ismael Juma]

I agree, the current KIP doesn't discuss the public API that we would be
exposing and it's extensive if the normal usage would allow for casting
AbstractRequest into the various subclasses and potentially even accessing
Records and related for produce request.

There are many use cases where this could be useful, but it requires quite
a bit of thinking around the APIs that we expose and the expected usage.

Ismael

On Thu, Nov 8, 2018, 6:09 PM Colin McCabe  Hi Lincong Li,
>
> I agree that server-side instrumentation is helpful.  However, I don't
> think this is the right approach.
>
> The problem is that RequestChannel.Request and AbstractResponse are
> internal classes that should not be exposed.  These are implementation
> details that we may change in the future.  Freezing these into a public API
> would really hold back the project.  For example, for really large
> responses, we might eventually want to avoid materializing the whole
> response all at once.  It would make more sense to return it in a streaming
> fashion.  But if we need to support this API forever, we can't do that.
>
> I think it's fair to say that this is, at best, half a solution to the
> problem of tracing requests.  Users still need to write the plugin code and
> arrange for it to be on their classpath to make this work.  I think the
> alternative here is not client-side instrumentation, but simply making the
> change to the broker without using a plugin interface.
>
> If a public interface is absolutely necessary here we should expose only
> things like the API key, client ID, time, etc. that don't constrain the
> implementation a lot in the future.  I think we should also use java here
> to avoid the compatibility issues we have had with Scala APIs in the past.
>
> best,
> Colin
>
>
> On Thu, Nov 8, 2018, at 11:34, radai wrote:
> > another downside to client instrumentation (beyond the number of
> > client codebases one would need to cover) is that in a large
> > environments you'll have a very long tail of applications using older
> > clients to upgrade - it would be a long and disruptive process (as
> > opposed to updating broker-side instrumentation)
> > On Thu, Nov 8, 2018 at 11:04 AM Peter M. Elias 
> wrote:
> > >
> > > I know we have a lot of use cases for this type of functionality at my
> > > enterprise deployment. I think it's helpful for maintaining
> reliability of
> > > the cluster especially and identifying clients that are not properly
> tuned
> > > and therefore applying excessive load to the brokers. Additionally,
> there
> > > is a bit of a dark spot without something like as currently. For
> example,
> > > if a client is not using a consumer group, there is no direct way to
> query
> > > the state of the consumer without looking at raw network connections to
> > > determine the extent of the traffic generated by that particular
> consumer.
> > >
> > > While client instrumentation can certainly help with this currently,
> given
> > > that Kafka is intended to be a shared service across a potentially very
> > > large surface area of clients, central observation of client activity
> is in
> > > my opinion an essential feature.
> > >
> > > Peter
> > >
> > > On Thu, Nov 8, 2018 at 12:13 PM radai 
> wrote:
> > >
> > > > bump.
> > > >
> > > > I think the proposed API (Observer) is useful for any sort of
> > > > multi-tenant environment for chargeback and reporting purposes.
> > > >
> > > > if no one wants to comment, can we initiate a vote?
> > > > On Mon, Nov 5, 2018 at 6:31 PM Lincong Li 
> wrote:
> > > > >
> > > > > Hi everyone. Here
> > > > > <
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response
> > > > >
> > > > > is
> > > > > my KIP. Any feedback is appreciated.
> > > > >
> > > > > Thanks,
> > > > > Lincong Li
> > > >
>

Heads up: javac warnings are now treated as errors

[Ismael Juma]

Hi all,

As part of KAFKA-7612, all javac warnings were fixed or suppressed. To
prevent them from reappearing, javac warnings are now treated as errors. We
still have some scalac warnings (see KAFKA-7614 for details on what's
needed to eliminate them) and 3 xlint warnings are not yet enabled
(KAFKA-7613).

Before merging PRs that were submitted before KAFKA-7612 was merged, it's a
good idea to rerun the PR tests.

Ismael

Re: [VOTE] 2.1.0 RC1

[Ismael Juma]

Hi Dong,

I was looking through the Maven repository for 2.1.0 and there are some
weird files there that didn't exist in 2.0.1:

kafka_2.12-2.1.0.mapping Fri Nov 09 23:08:52 UTC 2018 198
kafka_2.12-2.1.0.mapping.asc Fri Nov 09 23:09:03 UTC 2018 821
kafka_2.12-2.1.0.mapping.asc.md5 Fri Nov 09 23:09:04 UTC 2018 32
kafka_2.12-2.1.0.mapping.asc.sha1 Fri Nov 09 23:09:03 UTC 2018 40
kafka_2.12-2.1.0.mapping.md5 Fri Nov 09 23:08:52 UTC 2018 32
kafka_2.12-2.1.0.mapping.sha1 Fri Nov 09 23:08:52 UTC 2018 40

https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/2.1.0/

Ismael

On Fri, Nov 9, 2018 at 3:33 PM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for feature release of Apache Kafka 2.1.0.
>
> This is a major version release of Apache Kafka. It includes 28 new KIPs
> and
>
> critical bug fixes. Please see the Kafka 2.1.0 release plan for more
> details:
>
> *
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=91554044*
>  >
>
> Here are a few notable highlights:
>
> - Java 11 support
> - Support for Zstandard, which achieves compression comparable to gzip with
> higher compression and especially decompression speeds(KIP-110)
> - Avoid expiring committed offsets for active consumer group (KIP-211)
> - Provide Intuitive User Timeouts in The Producer (KIP-91)
> - Kafka's replication protocol now supports improved fencing of zombies.
> Previously, under certain rare conditions, if a broker became partitioned
> from Zookeeper but not the rest of the cluster, then the logs of replicated
> partitions could diverge and cause data loss in the worst case (KIP-320)
> - Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356)
> - Admin script and admin client API improvements to simplify admin
> operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340)
> - DNS handling improvements (KIP-235, KIP-302)
>
> Release notes for the 2.1.0 release:
> http://home.apache.org/~lindong/kafka-2.1.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, Nov 15, 12 pm PT ***
>
> * Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-2.1.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-2.1.0-rc1/javadoc/
>
> * Tag to be voted upon (off 2.1 branch) is the 2.1.0-rc1 tag:
> https://github.com/apache/kafka/tree/2.1.0-rc1
>
> * Documentation:
> *http://kafka.apache.org/21/documentation.html*
> 
>
> * Protocol:
> http://kafka.apache.org/21/protocol.html
>
> * Successful Jenkins builds for the 2.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-2.1-jdk8/50/
> *
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
> Cheers,
> Dong
>

Re: [ANNOUNCE] Apache Kafka 2.1.0

[Ismael Juma]

Thanks for running the release Dong and thanks to all who contributed to
the release!

Ismael

On Wed, Nov 21, 2018, 10:09 AM Dong Lin  The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.1.0
>
>
> This is a major release and includes significant features from 28 KIPs. It
> contains fixes and improvements from 179 JIRSs, including a few critical
> bug fixes. Here is a summary of some notable changes
>
> ** Java 11 support
> ** Support for Zstandard, which achieves compression comparable to gzip
> with higher compression and especially decompression speeds(KIP-110)
> ** Avoid expiring committed offsets for active consumer group (KIP-211)
> ** Provide Intuitive User Timeouts in The Producer (KIP-91)
> ** Kafka's replication protocol now supports improved fencing of zombies.
> Previously, under certain rare conditions, if a broker became partitioned
> from Zookeeper but not the rest of the cluster, then the logs of replicated
> partitions could diverge and cause data loss in the worst case (KIP-320)
> ** Streams API improvements (KIP-319, KIP-321, KIP-330, KIP-353, KIP-356)
> ** Admin script and admin client API improvements to simplify admin
> operation (KIP-231, KIP-308, KIP-322, KIP-324, KIP-338, KIP-340)
> ** DNS handling improvements (KIP-235, KIP-302)
>
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.1.0/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala ) from:
> https://kafka.apache.org/downloads#2.1.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 100 contributors to this release!
>
> Ahmed Al Mehdi, Aleksei Izmalkin, Alex Dunayevsky, Amit Sela, Andras
> Katona, Andy Coates, Anna Povzner, Arjun Satish, Attila Sasvari, Aviem Zur,
> Bibin Sebastian, Bill Bejeck, Bob Barrett, Brandon Kirchner, Bridger
> Howell, Chia-Ping Tsai, Colin Hicks, Colin Patrick McCabe, Dhruvil Shah,
> Dong Lin, Edoardo Comar, Eugen Feller, Ewen Cheslack-Postava, Filipe
> Agapito, Flavien Raynaud, Gantigmaa Selenge, Gardner Vickers, Gitomain,
> Gunnar Morling, Guozhang Wang, hashangayasri, huxi, huxihx, Ismael Juma,
> Jagadesh Adireddi, Jason Gustafson, Jim Galasyn, Jimin Hsieh, Jimmy Casey,
> Joan Goyeau, John Roesler, Jon Lee, jonathanskrzypek, Jun Rao, Kamal
> Chandraprakash, Kevin Lafferty, Kevin Lu, Koen De Groote, Konstantine
> Karantasis, lambdaliu, Lee Dongjin, Lincong Li, Liquan Pei, lucapette,
> Lucas Wang, Maciej Bryński, Magesh Nandakumar, Manikumar Reddy, Manikumar
> Reddy O, Mario Molina, Marko Stanković, Matthias J. Sax, Matthias
> Wessendorf, Max Zheng, Mayank Tankhiwale, mgharat, Michal Dziemianko,
> Michał Borowiecki, Mickael Maison, Mutasem Aldmour, Nikolay, nixsticks,
> nprad, okumin, Radai Rosenblatt, radai-rosenblatt, Rajini Sivaram, Randall
> Hauch, Robert Yokota, Rohan, Ron Dagostino, Sam Lendle, Sandor Murakozi,
> Simon Clark, Stanislav Kozlovski, Stephane Maarek, Sébastien Launay, Sönke
> Liebau, Ted Yu, uncleGen, Vahid Hashemian, Viktor Somogyi, wangshao,
> xinzhg, Xiongqi Wesley Wu, Xiongqi Wu, ying-zheng, Yishun Guan, Yu Yang,
> Zhanxiang (Patrick) Huang
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Dong
>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

 Thanks for the KIP, this is helpful. A few questions:

1. Have we considered whether we want to allow a similar topic config?
2. Can we rely on a method from the library to pick the default compression
level if compression.level is not set? We do it for gzip and it would seem
reasonable to do something similar for the other compression libraries.
3. Do we want to allow the buffer/block size to be configurable? This has
an impact on memory usage and people may want to trade compression for
less/more memory in some cases. For example, the default for LZ4 is 64KB
which is a bit high.

Ismael

On Sun, Nov 18, 2018, 2:07 PM Dongjin Lee  Hello dev,
>
> I hope to initiate the discussion of KIP-390: Add producer option to adjust
> compression level
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Add+producer+option+to+adjust+compression+level
> >.
> All feedbacks will be highly appreciated.
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> slideshare:
> www.slideshare.net/dongjinleekr
> *
>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

Hi Dongjin,

To clarify, I mean a broker topic config with regards to point 1. As you
know, compression can be done by the producer and/or by the broker. The
default is for the broker to just use whatever compression was used by the
producer, but this can be changed by the user on a per topic basis. It
seems like it would make sense for the configs to be . consistent between
producer and broker.

For point 2, I haven't looked at the implementation, but we could do it in
the `CompressionType` enum by invoking the right constructor or retrieving
the default value via a constant (if defined). That's an implementation
detail and can be discussed in the PR. The more general point is to rely on
the library defaults instead of choosing one ourselves.

For point 3, I'm in favour of doing that in this KIP.

Ismael

On Wed, Nov 28, 2018 at 7:01 AM Dongjin Lee  wrote:

> Thank you Ismael, here are the answers:
>
> *1. About topic config*
>
> After some consideration, I concluded that topic config doesn't need to
> support compression.level. Here is why: since the compression is conducted
> by the client, the one who can select the best compression level is the
> client itself. Let us assume that the compression level is set at the topic
> config level. In that case, there is a possibility that the compression
> level is not optimal for some producers. Actually, Kafka's go client also
> supports compression level functionality for the producer config only.
> <https://github.com/Shopify/sarama/blob/master/config.go> (wait, do we
> need
> to add this reasoning in the KIP, rejected alternatives section?)
>
> *2. About default level*
>
> As of current draft implementation, the default compression is set on the
> CompressionType enum. Of course, changing this strategy into relying on a
> method from the library to pick the default compression level seems
> possible, like `GZIPBlockOutputStream` does. In this case, we need to add
> similar wrapper class for zstd and modify lz4 the wrapper also. Add to
> this, it seems like we need to explicitly state that we follow the default
> compression level of the codec in the documentation. Is this what you
> intended?
>
> *3. Whether to allow the buffer/block size to be configurable*
>
> Well, As of current draft implementation, the lz4 level is implemented as
> block size; this is caused by my misunderstanding on lz4. After reviewing
> lz4 today, I found that it also supports compression level of 1~16
> (default: 1), not block size. I will fix it in this weekend by updating the
> wrapper class.
>
> For the problem of the buffer/block size, I have no strong opinion. If the
> community needs it, I will do it all together. How do you think?
>
> In short, it seems like I need to update the KIP document for issue #1 and
> update the compression wrapper for issue #2, #3. Is this okay?
>
> Thanks,
> Dongjin
>
> On Wed, Nov 28, 2018 at 12:34 AM Ismael Juma  wrote:
>
> >  Thanks for the KIP, this is helpful. A few questions:
> >
> > 1. Have we considered whether we want to allow a similar topic config?
> > 2. Can we rely on a method from the library to pick the default
> compression
> > level if compression.level is not set? We do it for gzip and it would
> seem
> > reasonable to do something similar for the other compression libraries.
> > 3. Do we want to allow the buffer/block size to be configurable? This has
> > an impact on memory usage and people may want to trade compression for
> > less/more memory in some cases. For example, the default for LZ4 is 64KB
> > which is a bit high.
> >
> > Ismael
> >
> > On Sun, Nov 18, 2018, 2:07 PM Dongjin Lee  >
> > > Hello dev,
> > >
> > > I hope to initiate the discussion of KIP-390: Add producer option to
> > adjust
> > > compression level
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Add+producer+option+to+adjust+compression+level
> > > >.
> > > All feedbacks will be highly appreciated.
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github:  <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > www.slideshare.net/dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*
>

Heads up: spotBugs enabled for Java 11 builds

[Ismael Juma]

Hi,

As per https://issues.apache.org/jira/browse/KAFKA-7389, spotBugs is now
enabled for Java 11 builds and the Java 11 PR build will fail for old PRs
that don't include the commit from KAFKA-7389. The fix is to rebase/merge.

Ismael

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

The updated title sounds fine to me.

Ismael

On Sun, Dec 2, 2018, 5:25 AM Dongjin Lee  Hi Ismael,
>
> Got it. Your direction is perfectly reasonable. I am now updating the KIP
> document and the implementation.
>
> By allowing the buffer/block size to be configurable, it would be better to
> update the title of the KIP like 'Allow fine-grained configuration for
> compression'. Is that right?
>
> @Other committers:
>
> Is there any other opinion on allowing the buffer/block size to be
> configurable?
>
> Thanks,
> Dongjin
>
> On Thu, Nov 29, 2018 at 1:45 AM Ismael Juma  wrote:
>
> > Hi Dongjin,
> >
> > To clarify, I mean a broker topic config with regards to point 1. As you
> > know, compression can be done by the producer and/or by the broker. The
> > default is for the broker to just use whatever compression was used by
> the
> > producer, but this can be changed by the user on a per topic basis. It
> > seems like it would make sense for the configs to be . consistent between
> > producer and broker.
> >
> > For point 2, I haven't looked at the implementation, but we could do it
> in
> > the `CompressionType` enum by invoking the right constructor or
> retrieving
> > the default value via a constant (if defined). That's an implementation
> > detail and can be discussed in the PR. The more general point is to rely
> on
> > the library defaults instead of choosing one ourselves.
> >
> > For point 3, I'm in favour of doing that in this KIP.
> >
> > Ismael
> >
> > On Wed, Nov 28, 2018 at 7:01 AM Dongjin Lee  wrote:
> >
> > > Thank you Ismael, here are the answers:
> > >
> > > *1. About topic config*
> > >
> > > After some consideration, I concluded that topic config doesn't need to
> > > support compression.level. Here is why: since the compression is
> > conducted
> > > by the client, the one who can select the best compression level is the
> > > client itself. Let us assume that the compression level is set at the
> > topic
> > > config level. In that case, there is a possibility that the compression
> > > level is not optimal for some producers. Actually, Kafka's go client
> also
> > > supports compression level functionality for the producer config only.
> > > <https://github.com/Shopify/sarama/blob/master/config.go> (wait, do we
> > > need
> > > to add this reasoning in the KIP, rejected alternatives section?)
> > >
> > > *2. About default level*
> > >
> > > As of current draft implementation, the default compression is set on
> the
> > > CompressionType enum. Of course, changing this strategy into relying
> on a
> > > method from the library to pick the default compression level seems
> > > possible, like `GZIPBlockOutputStream` does. In this case, we need to
> add
> > > similar wrapper class for zstd and modify lz4 the wrapper also. Add to
> > > this, it seems like we need to explicitly state that we follow the
> > default
> > > compression level of the codec in the documentation. Is this what you
> > > intended?
> > >
> > > *3. Whether to allow the buffer/block size to be configurable*
> > >
> > > Well, As of current draft implementation, the lz4 level is implemented
> as
> > > block size; this is caused by my misunderstanding on lz4. After
> reviewing
> > > lz4 today, I found that it also supports compression level of 1~16
> > > (default: 1), not block size. I will fix it in this weekend by updating
> > the
> > > wrapper class.
> > >
> > > For the problem of the buffer/block size, I have no strong opinion. If
> > the
> > > community needs it, I will do it all together. How do you think?
> > >
> > > In short, it seems like I need to update the KIP document for issue #1
> > and
> > > update the compression wrapper for issue #2, #3. Is this okay?
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Wed, Nov 28, 2018 at 12:34 AM Ismael Juma 
> wrote:
> > >
> > > >  Thanks for the KIP, this is helpful. A few questions:
> > > >
> > > > 1. Have we considered whether we want to allow a similar topic
> config?
> > > > 2. Can we rely on a method from the library to pick the default
> > > compression
> > > > level if compression.level is not set? We do it for gzip and it would
> > > seem
> > > > reasonable to do somethi

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Ismael Juma]

Hi Daniel,

We moved to Gitbox a while back for kafka and kafka-site. Which repository
is still using git-wip-us.apache.org?

Ismael

On Fri, Dec 7, 2018 at 8:53 AM Daniel Gruno  wrote:

> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
>   DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
>
> Hello Apache projects,
>
> I am writing to you because you may have git repositories on the
> git-wip-us server, which is slated to be decommissioned in the coming
> months. All repositories will be moved to the new gitbox service which
> includes direct write access on github as well as the standard ASF
> commit access via gitbox.apache.org.
>
> ## Why this move? ##
> The move comes as a result of retiring the git-wip service, as the
> hardware it runs on is longing for retirement. In lieu of this, we
> have decided to consolidate the two services (git-wip and gitbox), to
> ease the management of our repository systems and future-proof the
> underlying hardware. The move is fully automated, and ideally, nothing
> will change in your workflow other than added features and access to
> GitHub.
>
> ## Timeframe for relocation ##
> Initially, we are asking that projects voluntarily request to move
> their repositories to gitbox, hence this email. The voluntary
> timeframe is between now and January 9th 2019, during which projects
> are free to either move over to gitbox or stay put on git-wip. After
> this phase, we will be requiring the remaining projects to move within
> one month, after which we will move the remaining projects over.
>
> To have your project moved in this initial phase, you will need:
>
> - Consensus in the project (documented via the mailing list)
> - File a JIRA ticket with INFRA to voluntarily move your project repos
>over to gitbox (as stated, this is highly automated and will take
>between a minute and an hour, depending on the size and number of
>your repositories)
>
> To sum up the preliminary timeline;
>
> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
>relocation
> - January 9th -> February 6th: Mandated (coordinated) relocation
> - February 7th: All remaining repositories are mass migrated.
>
> This timeline may change to accommodate various scenarios.
>
> ## Using GitHub with ASF repositories ##
> When your project has moved, you are free to use either the ASF
> repository system (gitbox.apache.org) OR GitHub for your development
> and code pushes. To be able to use GitHub, please follow the primer
> at: https://reference.apache.org/committer/github
>
>
> We appreciate your understanding of this issue, and hope that your
> project can coordinate voluntarily moving your repositories in a
> timely manner.
>
> All settings, such as commit mail targets, issue linking, PR
> notification schemes etc will automatically be migrated to gitbox as
> well.
>
> With regards, Daniel on behalf of ASF Infra.
>
> PS:For inquiries, please reply to us...@infra.apache.org, not your
> project's dev list :-).
>
>
>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Ismael Juma]

Thanks Daniel and Christopher! My bad for missing the header. :)

Ismael

On Fri, Dec 7, 2018, 9:31 AM Daniel Gruno  On 12/7/18 6:28 PM, Ismael Juma wrote:
> > Hi Daniel,
> >
> > We moved to Gitbox a while back for kafka and kafka-site. Which
> > repository is still using git-wip-us.apache.org
> > <http://git-wip-us.apache.org>?
> >
> > Ismael
>
> Hi Ismael,
> You do not have any repositories that need to be moved, Kafka can safely
> disregard the email, as stated in the header :)
>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

Thanks Dongjin. One minor suggestion: we should mention that the broker
side configs are also topic configs (i.e. can be set for a given topic).

Ismael

On Sun, Jan 6, 2019, 10:37 AM Dongjin Lee  Happy new year.
>
> I just updated the title and contents of KIP and Jira issue, with updated
> draft implementation. Now both of compression level and buffer size options
> are available to producer and broker configuration. You can check the
> updated KIP from modified URL:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Allow+fine-grained+configuration+for+compression
>
> Please have a look when you are free.
>
> Thanks,
> Dongjin
>
> On Mon, Dec 3, 2018 at 12:50 AM Ismael Juma  wrote:
>
> > The updated title sounds fine to me.
> >
> > Ismael
> >
> > On Sun, Dec 2, 2018, 5:25 AM Dongjin Lee  >
> > > Hi Ismael,
> > >
> > > Got it. Your direction is perfectly reasonable. I am now updating the
> KIP
> > > document and the implementation.
> > >
> > > By allowing the buffer/block size to be configurable, it would be
> better
> > to
> > > update the title of the KIP like 'Allow fine-grained configuration for
> > > compression'. Is that right?
> > >
> > > @Other committers:
> > >
> > > Is there any other opinion on allowing the buffer/block size to be
> > > configurable?
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Thu, Nov 29, 2018 at 1:45 AM Ismael Juma  wrote:
> > >
> > > > Hi Dongjin,
> > > >
> > > > To clarify, I mean a broker topic config with regards to point 1. As
> > you
> > > > know, compression can be done by the producer and/or by the broker.
> The
> > > > default is for the broker to just use whatever compression was used
> by
> > > the
> > > > producer, but this can be changed by the user on a per topic basis.
> It
> > > > seems like it would make sense for the configs to be . consistent
> > between
> > > > producer and broker.
> > > >
> > > > For point 2, I haven't looked at the implementation, but we could do
> it
> > > in
> > > > the `CompressionType` enum by invoking the right constructor or
> > > retrieving
> > > > the default value via a constant (if defined). That's an
> implementation
> > > > detail and can be discussed in the PR. The more general point is to
> > rely
> > > on
> > > > the library defaults instead of choosing one ourselves.
> > > >
> > > > For point 3, I'm in favour of doing that in this KIP.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Nov 28, 2018 at 7:01 AM Dongjin Lee 
> > wrote:
> > > >
> > > > > Thank you Ismael, here are the answers:
> > > > >
> > > > > *1. About topic config*
> > > > >
> > > > > After some consideration, I concluded that topic config doesn't
> need
> > to
> > > > > support compression.level. Here is why: since the compression is
> > > > conducted
> > > > > by the client, the one who can select the best compression level is
> > the
> > > > > client itself. Let us assume that the compression level is set at
> the
> > > > topic
> > > > > config level. In that case, there is a possibility that the
> > compression
> > > > > level is not optimal for some producers. Actually, Kafka's go
> client
> > > also
> > > > > supports compression level functionality for the producer config
> > only.
> > > > > <https://github.com/Shopify/sarama/blob/master/config.go> (wait,
> do
> > we
> > > > > need
> > > > > to add this reasoning in the KIP, rejected alternatives section?)
> > > > >
> > > > > *2. About default level*
> > > > >
> > > > > As of current draft implementation, the default compression is set
> on
> > > the
> > > > > CompressionType enum. Of course, changing this strategy into
> relying
> > > on a
> > > > > method from the library to pick the default compression level seems
> > > > > possible, like `GZIPBlockOutputStream` does. In this case, we need
> to
> > > add
> > > > > similar wrapper class for zstd and modify lz4 the wrapper also. Add
> > to
> > > > > th

Re: [DISCUSS] Kafka 2.2.0 in February 2018

[Ismael Juma]

Thanks for volunteering Matthias! The plan sounds good to me.

Ismael

On Tue, Jan 8, 2019, 1:07 PM Matthias J. Sax  Hi all,
>
> I would like to propose a release plan (with me being release manager)
> for the next time-based feature release 2.2.0 in February.
>
> The recent Kafka release history can be found at
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.
> The release plan (with open issues and planned KIPs) for 2.2.0 can be
> found at
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827512
> .
>
>
> Here are the suggested dates for Apache Kafka 2.2.0 release:
>
> 1) KIP Freeze: Jan 24, 2019.
>
> A KIP must be accepted by this date in order to be considered for this
> release)
>
> 2) Feature Freeze: Jan 31, 2019
>
> Major features merged & working on stabilization, minor features have
> PR, release branch cut; anything not in this state will be automatically
> moved to the next release in JIRA.
>
> 3) Code Freeze: Feb 14, 2019
>
> The KIP and feature freeze date is about 2-3 weeks from now. Please plan
> accordingly for the features you want push into Apache Kafka 2.2.0 release.
>
> 4) Release Date: Feb 28, 2019 (tentative)
>
>
> -Matthias
>
>

Re: [VOTE] KIP-389: Introduce a configurable consumer group size limit

[Ismael Juma]

+1 (binding), thanks.

On Fri, Jan 11, 2019, 1:09 AM Stanislav Kozlovski  Hey folks,
>
> I'd like to initiate a vote thread about KIP-389
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Introduce+a+configurable+consumer+group+size+limit
> >
> .
>
> --
> Best,
> Stanislav
>

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

[Ismael Juma]

I think the point is that we distribute the time more fairly between
connection handling and other operations where before we could block on the
TLS handshake for a long time given a large number of connections.

Ismael

On Tue, Jan 15, 2019 at 11:39 AM Colin McCabe  wrote:

> Hi Rajini,
>
> Thanks for this.  The KIP looks really useful.
>
> >
>  > A new metric will be added to track the amount of time Acceptor is
> blocked
>  > from accepting connections due to backpressure. This will be a yammer
>  > Meter, consistent with other SocketServer metrics.
> >
> >
> kafka.network:type=Acceptor,name=AcceptorIdlePercent,listener={listenerName}
> >
>
> Hmm.  I was a bit confused by this.  When the acceptor is not accepting
> connections because there are none coming in, does that count as idle?
> When the acceptor is not accepting connections because the connect rate is
> being backpressured, does that count as idle?  Would it would be more
> intuitive to have a metric titled AcceptorBackpressuredPercent?
>
> Also, I sort of wonder if titling this "Limit incoming connection
> connection rate" or similar would be clearer than "improving fairness."  I
> guess it is unfair that a lot of incoming connections can swamp the network
> threads right now.  But limiting the rate of new connections is unfair to
> people connecting.  Overall the goal seems to be usability, not fairness.
>
> best,
> Colin
>
>
>
> On Tue, Jan 15, 2019, at 04:27, Rajini Sivaram wrote:
> > Hi Jan,
> >
> > If the queue of one Processor is full, we move to the next Processor
> > immediately without blocking. So as long as the queue of any Processor is
> > not full, we accept the connection immediately. If the queue of all
> > Processors are full, we assign a Processor and block until the connection
> > can be added. There is currently no timeout for this. The PR is here:
> > https://github.com/apache/kafka/pull/6022
> >
> > Thanks,
> >
> > Rajini
> >
> > On Tue, Jan 15, 2019 at 12:02 PM Jan Filipiak 
> > wrote:
> >
> > >
> > >
> > > > The connection queue for Processors will be changed to
> > > ArrayBlockingQueue with a fixed size of 20. Acceptor will use
> round-robin
> > > allocation to allocate each new connection to the next available
> Processor
> > > to which the connection can be added without blocking. If a Processor's
> > > queue is full, the next Processor will be chosen. If the connection
> queue
> > > on all Processors are full, Acceptor blocks until the connection can be
> > > added to the selected Processor. No new connections will be accepted
> during
> > > this period. The amount of time Acceptor is blocked can be monitored
> using
> > > the new AcceptorIdlePercent metric.
> > >
> > > So if the queue of one Processor is full, what is the strategy to move
> > > to the next queue? Are we using offer with a timeout here? How else can
> > > we make sure that a single slow processor will not block the entire
> > > processing? I assume we do not allow us to get stuck during put as you
> > > mention that all queues full is a scenario. I think there is quite some
> > > uncertainty here. Is there any code one could check out?
> > >
> > > Best Jan
> > >
> >
>

Re: [DISCUSS] 2.1.1 bug-fix release

[Ismael Juma]

Thanks for volunteering Colin. There are some important fixes since 2.1.0
so I think this would be great.

Ismael

On Tue, Jan 15, 2019 at 11:50 AM Colin McCabe  wrote:

> Hi all,
>
> I'd like to volunteer to be the release manager for the 2.1.1 bug fix
> release.
>
> 2.1 was released November 20, 2018.  There are 34 fixes scheduled for
> inclusion in 2.1.1 so far.
>
> Please find all the resolved tickets here:
>
> https://issues.apache.org/jira/browse/KAFKA-7818?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.1.1%20%20
>
> Please find the Release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.1.1
>
> regards,
> Colin
>

Re: [ANNOUNCE] New Committer: Vahid Hashemian

[Ismael Juma]

Congratulations Vahid!

On Tue, Jan 15, 2019, 2:45 PM Jason Gustafson  Hi All,
>
> The PMC for Apache Kafka has invited Vahid Hashemian as a project
> committer and
> we are
> pleased to announce that he has accepted!
>
> Vahid has made numerous contributions to the Kafka community over the past
> few years. He has authored 13 KIPs with core improvements to the consumer
> and the tooling around it. He has also contributed nearly 100 patches
> affecting all parts of the codebase. Additionally, Vahid puts a lot of
> effort into community engagement, helping others on the mail lists and
> sharing his experience at conferences and meetups.
>
> We appreciate the contributions and we are looking forward to more.
> Congrats Vahid!
>
> Jason, on behalf of the Apache Kafka PMC
>

Re: [VOTE] KIP-402: Improve fairness in SocketServer processors

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Tue, Jan 15, 2019, 3:38 PM Rajini Sivaram  Hi all,
>
> I would like to start vote on KIP-402 to improve fairness in channel
> processing in SocketServer to protect brokers from connection storms and
> limit the total number of connections in brokers to avoid OOM. The KIP is
> here:
>
>-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketServer+processors
>
>
> Thanks,
>
> Rajini
>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

Hi Dongjin,

For topic level, you can only have a single compression type so the way it
was before was fine, right? The point you raise is how to set broker
defaults that vary depending on the compression type, correct?

Ismael

On Mon, Jan 14, 2019 at 10:18 AM Dongjin Lee  wrote:

> I just realized that there was a missing hole in the KIP, so I fixed it.
> The draft implementation will be updated soon.
>
> In short, the proposed change did not regard the case of the topic or
> broker's 'compression.type' is 'producer'; in this case, the broker has to
> handle all kinds of the supported codec. So I added additional options
> (compression.[gzip,snappy,lz4, zstd].level, compression.[gzip,snappy,lz4,
> zstd].buffer.size) with handling routines.
>
> Please have a look when you are free.
>
> Thanks,
> Dongjin
>
> On Mon, Jan 7, 2019 at 6:23 AM Dongjin Lee  wrote:
>
> > Thanks for pointing out Ismael. It's now updated.
> >
> > Best,
> > Dongjin
> >
> > On Mon, Jan 7, 2019 at 4:36 AM Ismael Juma  wrote:
> >
> >> Thanks Dongjin. One minor suggestion: we should mention that the broker
> >> side configs are also topic configs (i.e. can be set for a given topic).
> >>
> >> Ismael
> >>
> >> On Sun, Jan 6, 2019, 10:37 AM Dongjin Lee  >>
> >> > Happy new year.
> >> >
> >> > I just updated the title and contents of KIP and Jira issue, with
> >> updated
> >> > draft implementation. Now both of compression level and buffer size
> >> options
> >> > are available to producer and broker configuration. You can check the
> >> > updated KIP from modified URL:
> >> >
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Allow+fine-grained+configuration+for+compression
> >> >
> >> > Please have a look when you are free.
> >> >
> >> > Thanks,
> >> > Dongjin
> >> >
> >> > On Mon, Dec 3, 2018 at 12:50 AM Ismael Juma 
> wrote:
> >> >
> >> > > The updated title sounds fine to me.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Sun, Dec 2, 2018, 5:25 AM Dongjin Lee  >> > >
> >> > > > Hi Ismael,
> >> > > >
> >> > > > Got it. Your direction is perfectly reasonable. I am now updating
> >> the
> >> > KIP
> >> > > > document and the implementation.
> >> > > >
> >> > > > By allowing the buffer/block size to be configurable, it would be
> >> > better
> >> > > to
> >> > > > update the title of the KIP like 'Allow fine-grained configuration
> >> for
> >> > > > compression'. Is that right?
> >> > > >
> >> > > > @Other committers:
> >> > > >
> >> > > > Is there any other opinion on allowing the buffer/block size to be
> >> > > > configurable?
> >> > > >
> >> > > > Thanks,
> >> > > > Dongjin
> >> > > >
> >> > > > On Thu, Nov 29, 2018 at 1:45 AM Ismael Juma 
> >> wrote:
> >> > > >
> >> > > > > Hi Dongjin,
> >> > > > >
> >> > > > > To clarify, I mean a broker topic config with regards to point
> 1.
> >> As
> >> > > you
> >> > > > > know, compression can be done by the producer and/or by the
> >> broker.
> >> > The
> >> > > > > default is for the broker to just use whatever compression was
> >> used
> >> > by
> >> > > > the
> >> > > > > producer, but this can be changed by the user on a per topic
> >> basis.
> >> > It
> >> > > > > seems like it would make sense for the configs to be .
> consistent
> >> > > between
> >> > > > > producer and broker.
> >> > > > >
> >> > > > > For point 2, I haven't looked at the implementation, but we
> could
> >> do
> >> > it
> >> > > > in
> >> > > > > the `CompressionType` enum by invoking the right constructor or
> >> > > > retrieving
> >> > > > > the default value via a constant (if defined). That's an
> >> > implementation
> >&g

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Ismael Juma]

Hi Dongjin,

When the compression type is "producer", then the broker doesn't recompress
though. Thinking about it some more, there are some uncommon cases where
recompression does happen (the old (and hopefully hardly used by now)
message format == 0 and some edge cases), so it is a good point you raised.

It's a bit unfortunate to add so many topic configs for cases that probably
don't matter. That is, if you are using "producer" compression, you
probably don't need to configure these settings and can live with the
defaults. Perhaps we should only support the topic config for the cases
where you are actually recompressing in the broker.

What do you think? I'd be interested in other people's thoughts too.

Ismael

On Sun, Jan 20, 2019 at 2:14 AM Dongjin Lee  wrote:

> Hi Ismael,
>
> It seems like it needs more explanation. Here is the detailed reasoning.
>
> You know, topic and broker's 'compression.type' allows 'uncompressed',
> 'producer' with standard codecs (i.e., gzip, snappy, lz4, zstd.) And this
> configuration is used by the broker in the re-compressing process after
> offset assignment. After this feature, the new configs, 'compression.level'
> and 'compression.buffer.size', also will be used in this process.
>
> The problem arises when given topic's compression type (whether it was
> inherited from broker's configuration or explicitly set) is 'producer.'
> With this setting, the compression codec to be used is decided by the
> producer client. Since there is no way to restore the compression level and
> buffer size from the message, we can take the following strategies:
>
> 1. Just use given 'compression.level' and 'compression.buffer.size'
> settings.
>
> It will cause so many errors. Let's imagine the case of topic's
> configuration is { compression.type=producer, compression.level=10,
> compression.buffer.size=8192 }. In this case, all producers with gzip or
> lz4 compressed messages will result in an error. (gzip doesn't allow
> compression level 10, and lz4 also for a buffer size of 8192.)
>
> 2. Extend the message format to include compression configurations.
>
> With this strategy, we need to change the message format - it's a too big
> change.
>
> 3. If topic's compression.type is 'producer', use the default configuration
> for the given codec.
>
> With this strategy, allowing fine-grained compression configuration is
> meaningless.
>
> For the above reasons, I think the only alternative is providing options
> that can be used when the topic's 'compression.type' is 'producer.' In
> other words, adding compression.[gzip, lz4, zstd].level and
> compression.[gzip.snappy.lz4].buffer.size options - and it is what I did in
> the last modification.
>
> (wait, the reasoning above should be included in the KIP in the rejected
> alternatives section, isn't it?)
>
> Thanks,
> Dongjin
>
> On Sun, Jan 20, 2019 at 2:33 AM Ismael Juma  wrote:
>
> > Hi Dongjin,
> >
> > For topic level, you can only have a single compression type so the way
> it
> > was before was fine, right? The point you raise is how to set broker
> > defaults that vary depending on the compression type, correct?
> >
> > Ismael
> >
> > On Mon, Jan 14, 2019 at 10:18 AM Dongjin Lee  wrote:
> >
> > > I just realized that there was a missing hole in the KIP, so I fixed
> it.
> > > The draft implementation will be updated soon.
> > >
> > > In short, the proposed change did not regard the case of the topic or
> > > broker's 'compression.type' is 'producer'; in this case, the broker has
> > to
> > > handle all kinds of the supported codec. So I added additional options
> > > (compression.[gzip,snappy,lz4, zstd].level,
> compression.[gzip,snappy,lz4,
> > > zstd].buffer.size) with handling routines.
> > >
> > > Please have a look when you are free.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Mon, Jan 7, 2019 at 6:23 AM Dongjin Lee  wrote:
> > >
> > > > Thanks for pointing out Ismael. It's now updated.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Mon, Jan 7, 2019 at 4:36 AM Ismael Juma 
> wrote:
> > > >
> > > >> Thanks Dongjin. One minor suggestion: we should mention that the
> > broker
> > > >> side configs are also topic configs (i.e. can be set for a given
> > topic).
> > > >>
> 

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

[Ismael Juma]

Hi,

I'm late to the discussion, so apologies. One question, did we consider
having the client generate a member id in the first join group? This could
be random or known to be unique and would avoid a second join group request
in the common case. If we considered and rejected this option, it would be
good to include why in the "Rejected Alternatives" section.

Ismael

On Mon, Nov 26, 2018, 5:48 PM Boyang Chen  Hey friends,
>
>
> I would like to start a discussion thread for KIP-394 which is trying to
> mitigate broker cache bursting issue due to anonymous join group requests:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request
>
>
> Thanks!
>
> Boyang
>

Re: Permissions to create KIP

[Ismael Juma]

Hi Tejal,

Thanks for your interest. Access has been granted.

Ismael

On Sun, Jan 20, 2019 at 3:08 PM Tejal Adsul  wrote:

> Hi,
>
> I work at Confluent. Please could you grant me permission to create a KIP
> for apache kafka, I wanted to propose a change to AbstractConfig in Kafka.
> Following are my details
> Full NameTEJAL ADSUL
> emailte...@confluent.io
> ID: tejal
>
> Thanks,
> Tejal
>

Re: Java 10 replacing Java 9 in Jenkins for trunk

[Ismael Juma]

Hi Ted,

We are in the process of updating the system tests infrastructure so that
it works with Java 8. Once that happens, we will switch the build to use
Java 8.

Ismael

On Sat, 5 May 2018, 19:57 Ted Yu,  wrote:

> In PR build, I noticed the following (
> https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/622/console) :
>
> *02:32:11*
> :clients:compileJava/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk10-scala2.12/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:1262:
> error: lambda expressions are not supported in -source 7*02:32:12*
> client.poll(pollTimeout, nowMs, () -> {*02:32:12*
>   ^*02:32:12*   (use -source 8 or higher to
> enable lambda expressions)*02:32:12* 1 error
>
>
> Could the above be due to the following in build.gradle :
>
> if (JavaVersion.current().isJava9Compatible())
>   options.compilerArgs << "--release" << "7"
>
> If so, we need to adjust it.
>
> Cheers
>
> On Mon, Apr 9, 2018 at 9:46 AM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > Java 10 was recently released and support for Java 9 has ended since it's
> > not a LTS release. I've added a kafka-trunk Jenkins job for Java 10 and
> > disabled the Java 9 job. I also added a PR Jenkins job for Java 10 and
> will
> > soon disable the Java 9 PR job.
> >
> > The general idea is to have a separate Jenkins job for the latest non LTS
> > release (Java 10) and all supported LTS releases (Java 8 and Java 7
> > currently, soon to become Java 8 only).
> >
> > Let me know if you have any questions or concerns.
> >
> > Ismael
> >
>

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

[Ismael Juma]

Maybe it should message instead of record to be consistent with
message.format.version.

On Wed, 9 May 2018, 09:04 Jason Gustafson,  wrote:

> Hi Dhruvil,
>
> Thanks for the KIP. +1 from me. Just a minor nitpick on the name of the new
> config. I would suggest "record.downconversion.enable". The "record" prefix
> emphasizes what is being down-converted and similar existing configs use
> "enable" rather than "enabled."
>
> -Jason
>
> On Wed, May 2, 2018 at 9:35 AM, Ted Yu  wrote:
>
> > +1
> >
> > On Wed, May 2, 2018 at 9:27 AM, Dhruvil Shah 
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start the vote on KIP-238: Efficient Memory Usage for
> > > Down-Conversion.
> > >
> > > For reference, the link to the KIP is here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 283%3A+Efficient+Memory+Usage+for+Down-Conversion
> > >
> > > and the discussion thread is here:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86799.html
> > >
> > > Thanks,
> > > Dhruvil
> > >
> >
>

Re: Use of a formatter like Scalafmt

[Ismael Juma]

Sounds good about doing this for Kafka streams scala first. Core is a bit
more complicated so may require more discussion.

Ismael

On Wed, 9 May 2018, 16:59 Matthias J. Sax,  wrote:

> Joan,
>
> thanks for starting this initiative. I am overall +1
>
> However, I am worried about applying it to `core` module as the change
> is massive. For the Kafka Streams Scala module, that is new and was not
> released yet, I am +1.
>
> A personal thing about the style: the 2-space indention for JavaDocs is
> a little weird.
>
> /**
>  *
>  */
>
> is changed to
>
> /**
>   *
>   */
>
> Not sure if this can be fixed easily in the style file? If not, I am
> also fine with the change.
>
> This change also affect the license headers of many files and exposing
> that those use the wrong comment format anyway. They should use regular
> comments
>
> /*
>  *
>  */
>
> but not JavaDoc comments
>
> /**
>  *
>  */
>
> (We fixed this for Java source code in the past already -- maybe it's
> time to fix it for Scala code base, too.
>
>
>
> -Matthias
>
> On 5/9/18 4:45 PM, Joan Goyeau wrote:
> > Hi Ted,
> >
> > As far as I understand this is an issue for PRs and back porting the
> > changes to other branches.
> >
> > Applying the tool to the other branches should also resolve the conflicts
> > as the formattings will match, leaving only the actual changes in the
> diffs.
> > That's what we did sometime ago at my work and it went quiet smoothly.
> >
> > If we don't want to do a big bang commit then I'm thinking we might want
> to
> > make it gradually by applying it module by module?
> > This is one idea do you have any other?
> >
> > I know formatting sounds like the useless thing that doesn't matter and I
> > totally agree with this, that's why I don't want to care about it while
> > coding.
> >
> > Thanks
> >
> > On Thu, 10 May 2018 at 00:15 Ted Yu  wrote:
> >
> >> Applying the tool across code base would result in massive changes.
> >> How would this be handled ?
> >>  Original message From: Joan Goyeau 
> >> Date: 5/9/18  3:31 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Use
> >> of a formatter like Scalafmt
> >> Hi,
> >>
> >> Contributing to Kafka Streams' Scala API, I've been kinda lost on how
> >> should I format my code.
> >> I know formatting is the start of religion wars but personally I have no
> >> preference at all. I just want consistency across the codebase, no
> >> unnecessary formatting diffs in PRs and offload the formatting to a tool
> >> that will do it for me and concentrate on what matters (not formatting).
> >>
> >> So I opened the following PR where I put arbitrary rules in
> .scalafmt.conf
> >> <
> >>
> https://github.com/apache/kafka/pull/4965/files#diff-8af3e1355c23c331ee2b848e12c5219f
> >>>
> >> :
> >> https://github.com/apache/kafka/pull/4965
> >>
> >> Please let me know what do you think and if we can move this forward and
> >> settle something.
> >>
> >> Thanks
> >>
> >
>
>

Re: [VOTE] KIP-294 - Enable TLS hostname verification by default

[Ismael Juma]

Thanks for the KIP, +1 (binding) from me.

Ismael

On Wed, May 9, 2018 at 8:29 AM Rajini Sivaram 
wrote:

> Hi all,
>
> Since there have been no objections on this straightforward KIP, I would
> like to initiate the voting process. KIP-294 proposes to use a secure
> default value for endpoint identification when using SSL as the security
> protocol. The KIP Is here:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default
>
> If there are any concerns, please add them to this thread or the discussion
> thread (https://www.mail-archive.com/dev@kafka.apache.org/msg87549.html)
>
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Ismael Juma]

Thanks for the KIP, +1 (binding). A few suggestions:

1. We normally include the time unit in configs. Not sure if we do it for
command line parameters though, so can we please verify and make it
consistent?
2. The KIP mentions --polling-loop-timeout and --timeout. Which is it?
3. Can we include the description of the new parameter in the KIP? In the
PR it says "Consumer polling loop timeout", but I think this is a bit
unclear. What are we actually measuring here?

Ismael

On Mon, Apr 16, 2018 at 2:25 PM Alex Dunayevsky 
wrote:

> Hello friends,
>
> Let's start the vote for KIP-281: ConsumerPerformance: Increase Polling
> Loop Timeout and Make It Reachable by the End User:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+and+Make+It+Reachable+by+the+End+User
>
> Thank you,
> Alexander Dunayevsky
>

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

[Ismael Juma]

Thanks for the KIP, Jonathan. It would be helpful to have more detail on
how SSL authentication could be broken if the new behaviour is the default.
I know this was discussed in the mailing list thread, but it's important to
include it in the KIP since it's the main reason why a new config is needed
(and configs should be avoided whenever we can just do the right thing).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

[Ismael Juma]

Thanks for the KIP Dhruvil, this is a welcome improvement! My understanding
is that you have done some work to validate that the change has the desired
effect, it would be good to include that information in the "Testing
Strategy" section.

+1 (binding)

Ismael

On Wed, May 2, 2018 at 9:27 AM Dhruvil Shah  wrote:

> Hi all,
>
> I would like to start the vote on KIP-238: Efficient Memory Usage for
> Down-Conversion.
>
> For reference, the link to the KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion
>
> and the discussion thread is here:
> https://www.mail-archive.com/dev@kafka.apache.org/msg86799.html
>
> Thanks,
> Dhruvil
>

Re: [VOTE] KIP-282: Add the listener name to the authentication context

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Wed, Apr 25, 2018 at 1:52 AM Mickael Maison 
wrote:

> Hi,
>
> There has been no objections in the DISCUSS thread so I'd like to
> start a vote on KIP-282:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+authentication+context
>
> Thanks
>

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Ismael Juma]

Thanks for the KIP, +1 (binding). Can you also please describe the
compatibility impact of changing the error code from
CLUSTER_AUTHORIZATION_FAILED to TOPIC_AUTHORIZATION_FAILED?

Ismael

On Wed, Apr 25, 2018 at 2:45 AM Edoardo Comar  wrote:

> Hi,
>
> The discuss thread on KIP-277 (
> https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html )
> seems to have been fruitful and concerns have been addressed, please allow
> me start a vote on it:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> I will update the small PR to the latest KIP semantics if the vote passes
> (as I hope :-).
>
> cheers
> Edo
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah  wrote:

> Hi,
>
> I would like to start a vote on KIP-306 which proposes to add a
> configuration to delay responses to failed authentication.
>
> Link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
>
> Thanks,
> Dhruvil
>

Re: KAFKA-6733 feedback

[Ismael Juma]

Hi Mateusz,

This is a good change, but it requires a KIP as mentioned in the PR. You
gave an example of a similar PR without a KIP, but there was a KIP for it:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

Ismael

On Mon, May 21, 2018 at 4:00 PM Mateusz Zakarczemny 
wrote:

> Hi,
> Could I ask for some feedback regarding
> https://github.com/apache/kafka/pull/4807 ?
> It's waiting 1,5 month. I had to resolve conflicts with trunk couple of
> times.
> I would be grateful if someone could take a look it.
>
> Regards,
> Mateusz Zakarczemny
>

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Ismael Juma]

It got 3 binding votes already and the deadline is only tomorrow. :)

Ismael

On Mon, 21 May 2018, 21:50 Colin McCabe,  wrote:

> On Mon, May 21, 2018, at 04:53, Andy Coates wrote:
> > Hey Piyush,
> >
> > Thanks for the updated KIP! Couple of minor points from me:
> >
> > When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end
> for
> > the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
> > *kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions,
> i.e.
> > this is a place for storing wildcard-suffixed Acls, so it implicitly ends
> > in an asterisk. If you include the asterisk in the path then you need to
> > validate each entry, when reading, ends with an asterisk, and do
> something
> > if they don't. If you don't include the training asterisk then there is
> > nothing to validate and you can treat the prefix as a literal, (i.e. no
> > escaping needed).  TBH I'd probably drop the asterisk from the in-memory
> > representation as well, for the same reason.
>
> Hi Andy,
>
> I agree.  If everything in ZK under /kafka-wildcard-acl/ is a prefix ACL,
> there is no need to include the star at the end.  And really, it should be
> called something like /kafka-prefix-acl/, since it's only vaguely related
> to the idea of wildcards.
>
> >
> > Rather than creating an enum to indicate the type of a resource, you
> could
> > instead use polymorphism, e.g. make Resource an interface and have two
> > implementations: LiteralResource and WildcardSuffixedResource.  This is
> > also extensible, but may also allow for a cleaner implementation.
>
> Since Resource is a concrete class now, we can't make it an interface
> without breaking API compatibility.
>
> Even if it were possible to do compatibly, I would argue it's a bad idea.
> If we need to add another bit of state like case insensitivity, we don't
> want to have LiteralCaseInsensistiveResource,
> WildcardSuffixedCaseInsensitiveResource, etc. etc.  You need 2^n subclasses
> classes to represent N bits of state.
>
> I would argue that there should be a field in Resource like NameType which
> can be LITERAL or PREFIX.  That leaves us in a good position when someone
> inevitably comes up with a new NameType.
>
> Does this still have a chance to get in, or has the KIP window closed?  I
> would argue with one or two minor changes it's ready to go.  Pretty much
> all of the compatibility problems are solved with the separate ZK hierarchy.
>
> best,
> Colin
>
> >
> > Andy
> >
> > On 21 May 2018 at 01:58, Rajini Sivaram  wrote:
> >
> > > Hi Piyush, Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Sun, May 20, 2018 at 2:53 PM, Andy Coates 
> wrote:
> > >
> > > > Awesome last minute effort Piyush.
> > > >
> > > > Really appreciate your time and input,
> > > >
> > > > Andy
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 19 May 2018, at 03:43, Piyush Vijay 
> wrote:
> > > > >
> > > > > Updated the KIP.
> > > > >
> > > > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > > > classes.
> > > > > 2. modify getAcls() and rely on ResourceNameType' field in
> Resource to
> > > > > return either exact matches or all matches based on
> wildcard-suffix.
> > > > > 3. CLI changes to identify if resource name is literal or
> > > wildcard-suffix
> > > > > 4. Escaping doesn't work and isn't required if we're keeping a
> separate
> > > > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > > > 5. New API keys for Create / Delete / Describe Acls request with a
> new
> > > > > field in schemas for 'ResourceNameType'.
> > > > >
> > > > > Looks ready to me for the vote, will start voting thread now.
> Thanks
> > > > > everyone for the valuable feedback.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> > > wrote:
> > > > >>
> > > > >> Hi Piyush,
> > > > >>
> > > > >> We're fast approaching the KIP deadline. Are you actively working
> on
> > > > this?
> > > > >> If you're not I can take over.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Andy
> > > > >>
> > > > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > > > >>>
> > > > >>> OK I've read it now.
> > > > >>>
> > > > >>> 1. I see you have an example:
> > > >  For example: If I want to fetch all ACLs that match ’topicA*’,
> it’s
> > > > not
> > > > >>> possible without introducing new API AND maintaining backwards
> > > > >>> compatibility.
> > > > >>> getAcls takes a Resource, right, which would be either a full
> > > resource
> > > > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get
> all
> > > > ACLs
> > > > >>> relating to a specific resource, not a partial resource like
> > > 'topicA*'.
> > > > >>> Currently, I'm guessing / half-remembering that if you ask it for
> > > ACLs
> > > > >> for
> > > > >>> topic 'foo' it doesn't include global 'ALL' AC

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Ismael Juma]

Breaking API without a deprecation cycle doesn't seem good. Are we sure
about this?

Ismael

On Mon, 21 May 2018, 15:04 Guozhang Wang,  wrote:

> Hello Matthias,
>
> I've tried it out on the PR, the implementation should be fine but one
> concern I had is that, as you may also realize users of
> DynamicStreamPartitioner needs to implement two interface functions, with
> and without the topic name if it is extending from StreamPartitioner; we
> could also let it to not extend from StreamPartition so it has one function
> only but then we'd need Produced to have two functions allowing
> StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros and
> cons I'm think it may be better to just change the interface of
> StreamPartitioner itself, since even without dynamic routing, allowing the
> topic name could let users to give one partitioner implementation that
> branch on the topic names other than having one partitioner per topic.
>
>
> Guozhang
>
>
> On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax 
> wrote:
>
> > I think that the risk of the change is moderate as I expect most people
> > to use the DefaultStreamPartitioner.
> >
> > However, there would still be possibility to define a new interface
> > instead of changing the old:
> >
> > > public interface DynamicStreamPartitioner {
> > > Integer partition(String topic, K key, V value, int numPartitions);
> > > }
> >
> > The newly added methods `Topology#addSink` and `KStream#to` would take
> > this new interface instead of the old.
> >
> > Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
> > runtime code work though...
> >
> > WDYT?
> >
> > -Matthias
> >
> > On 5/21/18 11:47 AM, Guozhang Wang wrote:
> > > Hello everyone,
> > >
> > > While implementing the PR for this KIP I realized there is once place
> > which
> > > we should consider modifying on public APIs as well:
> > > StreamPartitioner#partition, to add the topic name string. Note it will
> > be
> > > a incompatible change that requires users who have customized
> > > StreamPartitioner implementations.
> > >
> > > I've updated the wiki page of KIP-303, please recast your vote on this
> > > thread. Thanks!!!
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, May 17, 2018 at 3:15 PM, John Roesler 
> wrote:
> > >
> > >> +1 non-binding
> > >>
> > >> On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > >> wrote:
> > >>
> > >>> +1 (binding)
> > >>>
> > >>>
> > >>> On 5/17/18 12:18 PM, Ted Yu wrote:
> >  +1
> >   Original message From: Gwen Shapira <
> > >> g...@confluent.io>
> > >>> Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> > >>> Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> > >> Streams'
> > >>> Topology Sink
> >  Yay, its about time :)
> > 
> >  +1
> > 
> >  On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang  >
> > >>> wrote:
> > 
> > > Hello folks,
> > >
> > > I'd like to start a voting thread on adding dynamic routing
> > >>> functionality
> > > in Streams sink node. Please find a KIP here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> > >
> > >
> > > And the PR itself ready for review as well under KAFKA-4936:
> > >
> > > https://github.com/apache/kafka/pull/5018
> > >
> > >
> > >
> > > Thanks!
> > > -- Guozhang
> > >
> > 
> > 
> > 
> > >>>
> > >>>
> > >>
> > >
> > >
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Java 8 switch in trunk

[Ismael Juma]

Hi all,

We have switched to Java 8 in trunk. We can now use Java 8 for new code,
but I suggest we don't refactor existing code to use Java 8 features like
lambdas just yet. Let's focus on getting the 2.0.0 release out and after
the first bug fix, we can consider refactors that will make backports
harder.

Ismael

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

[Ismael Juma]

Thanks for the KIP. I think this is a good and low risk change. It would be
good to ensure that it works well with KIP-302 if we think that makes sense
too. In any case, +1 (binding).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>

Re: [VOTE] KIP-176: Remove deprecated new-consumer option for tools

[Ismael Juma]

+1 (binding).

Ismael

On Wed, 23 May 2018, 09:04 Paolo Patierno,  wrote:

> Sorry ... I hope it's not too late but I created the KIP-176 on September
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools
>
> but due to be a breaking change, I needed to wait for a major release ...
> and the right time is now.
> Can you vote for that adding to the release plan, please ?
>
> Thanks,
>
> Paolo Patierno
> Principal Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

Option 1 sounds good to me provided that we can come up with a good
default. What would you suggest?

Ismael

On Wed, May 30, 2018 at 9:41 AM Jason Gustafson  wrote:

> Hi Everyone,
>
> There remains some inconsistency in the timeout behavior of the consumer
> APIs which do not accept a timeout. Some of them block forever (e.g.
> position()) and some of them use request.timeout.ms (e.g.
> parititonsFor()).
> I think we'd probably all agree that blocking forever is not useful
> behavior and using request.timeout.ms has always been a hack since it
> controls a separate concern. I think there are basically two options to
> address this:
>
> 1. We can add max.block.ms to match the producer and use it as the default
> timeout when a timeout is not explicitly provided. This will fix the
> indefinite blocking behavior and avoid conflating request.timeout.ms.
> 2. We can deprecate the methods which don't accept a timeout.
>
> I'm leaning toward the first solution because I think we want to push users
> to specifying timeouts through configuration rather than in code (Jay's
> original argument). I think the overloads are still useful for advanced
> usage (e.g. in kafka streams), but we should give users an easy option with
> reasonable default behavior.
>
> If that sounds ok, I'd propose we add it to this KIP and fix it now. This
> gives users an easy way to get the benefit of the improvements from this
> KIP without changing any code.
>
> Thanks,
> Jason
>
>
>
>
> On Sun, May 13, 2018 at 7:58 PM, Richard Yu 
> wrote:
>
> > Hi,
> >
> > With 3 binding votes and 6 non-binding, this KIP would be accepted.
> >
> > Thanks for participating.
> >
> > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > >
> > > > +1 non-binding
> > > >
> > > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > > >
> > > > > +1 (non-binding).
> > > > > Thanks.
> > > > >
> > > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> +1 (non binding)
> > > > >> Thanks
> > > > >>
> > > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > >> wrote:
> > > > >>> Hi Richard, Thanks for the KIP.
> > > > >>>
> > > > >>> +1 (binding)
> > > > >>>
> > > > >>> Regards,
> > > > >>>
> > > > >>> Rajini
> > > > >>>
> > > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > >> wrote:
> > > > >>>
> > > >  +1 from me, thanks!
> > > > 
> > > > 
> > > >  Guozhang
> > > > 
> > > >  On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > > ja...@confluent.io>
> > > >  wrote:
> > > > 
> > > > > Thanks for the KIP, +1 (binding).
> > > > >
> > > > > One small correction: the KIP mentions that close() will be
> > > > >> deprecated,
> > > >  but
> > > > > we do not want to do this because it is needed by the Closeable
> > > >  interface.
> > > > > We only want to deprecate close(long, TimeUnit) in favor of
> > > > > close(Duration).
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Tue, May 8, 2018 at 12:43 AM, khaireddine Rezgui <
> > > > > khaireddine...@gmail.com> wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> 2018-05-07 20:35 GMT+01:00 Bill Bejeck :
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Bill
> > > > >>>
> > > > >>> On Fri, May 4, 2018 at 7:21 PM, Richard Yu <
> > > >  yohan.richard...@gmail.com
> > > > >>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi all, I would like to bump this thread since discussion in
> > the
> > > >  KIP
> > > >  appears to be reaching its conclusion.
> > > > 
> > > > 
> > > > 
> > > >  On Thu, Mar 15, 2018 at 3:30 PM, Richard Yu <
> > > > >> yohan.richard...@gmail.com>
> > > >  wrote:
> > > > 
> > > > > Hi all,
> > > > >
> > > > > Since there does not seem to be too much discussion in
> > > > >> KIP-266, I
> > > > >> will
> > > > >>> be
> > > > > starting a voting thread.
> > > > > Here is the link to KIP-266 for reference:
> > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > >  action?pageId=75974886
> > > > >
> > > > > Recently, I have made some updates to the KIP. To
> reiterate,
> > I
> > > >  have
> > > > > included KafkaConsumer's commitSync,
> > > > > poll, and committed in the KIP. (we will be adding to a
> > > > >>> TimeoutException
> > > > > to them as well, in a similar manner
> > > > > to what we will be doing for position())
> > > > >
> > > > > Thanks,
> > > > > Richard Yu
> > > > >
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >>
> > > > >>>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

Sounds good to me,

On Wed, May 30, 2018 at 12:40 PM Jason Gustafson  wrote:

> Perhaps one minute? That is the default used by the producer.
>
> -Jason
>
> On Wed, May 30, 2018 at 9:50 AM, Ismael Juma  wrote:
>
> > Option 1 sounds good to me provided that we can come up with a good
> > default. What would you suggest?
> >
> > Ismael
> >
> > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson 
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > There remains some inconsistency in the timeout behavior of the
> consumer
> > > APIs which do not accept a timeout. Some of them block forever (e.g.
> > > position()) and some of them use request.timeout.ms (e.g.
> > > parititonsFor()).
> > > I think we'd probably all agree that blocking forever is not useful
> > > behavior and using request.timeout.ms has always been a hack since it
> > > controls a separate concern. I think there are basically two options to
> > > address this:
> > >
> > > 1. We can add max.block.ms to match the producer and use it as the
> > default
> > > timeout when a timeout is not explicitly provided. This will fix the
> > > indefinite blocking behavior and avoid conflating request.timeout.ms.
> > > 2. We can deprecate the methods which don't accept a timeout.
> > >
> > > I'm leaning toward the first solution because I think we want to push
> > users
> > > to specifying timeouts through configuration rather than in code (Jay's
> > > original argument). I think the overloads are still useful for advanced
> > > usage (e.g. in kafka streams), but we should give users an easy option
> > with
> > > reasonable default behavior.
> > >
> > > If that sounds ok, I'd propose we add it to this KIP and fix it now.
> This
> > > gives users an easy way to get the benefit of the improvements from
> this
> > > KIP without changing any code.
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > >
> > > On Sun, May 13, 2018 at 7:58 PM, Richard Yu <
> yohan.richard...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > With 3 binding votes and 6 non-binding, this KIP would be accepted.
> > > >
> > > > Thanks for participating.
> > > >
> > > > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar 
> > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > > > >
> > > > > > +1 non-binding
> > > > > >
> > > > > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > > > > >
> > > > > > > +1 (non-binding).
> > > > > > > Thanks.
> > > > > > >
> > > > > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> +1 (non binding)
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > > > > rajinisiva...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>> Hi Richard, Thanks for the KIP.
> > > > > > >>>
> > > > > > >>> +1 (binding)
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>>
> > > > > > >>> Rajini
> > > > > > >>>
> > > > > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang <
> > > wangg...@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>>> +1 from me, thanks!
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Guozhang
> > > > > > >>>>
> > > > > > >>>> On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > > > > ja...@confluent.io>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> Thanks for 

Re: referencing OffsetCheckpoint in downstream project

[Ismael Juma]

Hi Ted,

There are two such classes. The example you have is for the broker class,
not the Streams one.

Ismael

On Thu, 31 May 2018, 09:03 Ted Yu,  wrote:

> Hi,
> OffsetCheckpoint has been relocated
> to org.apache.kafka.streams.state.internals package.
>
> Does this mean that downstream project should no longer reference this
> class ?
>
> This is how the class is used (against Kafka 0.10.0.1 release) :
>
> // ensure that topic is removed from all cleaner offsets
> assert(servers.forall(server => topicAndPartitions.forall { tp =>
>   val checkpoints = server.getLogManager().logDirs.map { logDir =>
> new OffsetCheckpoint(new File(logDir,
> "cleaner-offset-checkpoint")).read()
>   }
>   checkpoints.forall(checkpointsPerLogDir =>
> !checkpointsPerLogDir.contains(tp))
> }), s"checkpoint for topic $topic still exists")
>
> Cheers
>

Re: [DISCUSS] 0.10.2.2 bug fix release

[Ismael Juma]

Thanks for doing this Matthias, +1.

Ismael

On Thu, Jun 7, 2018 at 1:50 PM Matthias J. Sax 
wrote:

> Dear all,
>
> I want to propose a 0.10.2.2 bug fix release. 0.10.2.1 is over a year
> old and a couple of critical fixes are available for 0.10.2.2.
>
> Please find a list of all 24 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6566?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.2%20AND%20resolution%20!%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20
>
> There are no open tickets with target version 0.10.2.2 at the moment. If
> there are any tickets you want to get included in 0.10.2.2 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.10.2.2 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2
>
>
> Thanks a lot,
>   Matthias
>
>
>
>

Re: [DISCUSS] 1.0.2 bug fix release

[Ismael Juma]

+1, thanks!

On Thu, 7 Jun 2018, 11:16 Matthias J. Sax,  wrote:

> Dear all,
>
> I want to propose a 1.0.2 bug fix release. 1.0.1 is 3 months old and a
> couple of critical fixes are available for 1.0.2.
>
> Please find a list of all 14 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6937?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.0.2
>
> There are 7 open tickets with target version 1.0.2.
>
>
> https://issues.apache.org/jira/browse/KAFKA-6083?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%201.0.2
>
> If you own one of these tickets, please let us know if you plan to
> resolve them soon. Otherwise, please change the target version to a
> future release.
>
> If there are any other tickets you want to get included in 1.0.2 please
> let us know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 1.0.2 next
> Thursday. Please find a summary in the wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Copy+of+Release+Plan+1.0.2
>
>
> Thanks a lot,
>   Matthias
>
>
>

Re: [DISCUSS] 0.11.0.3 bug fix release

[Ismael Juma]

+1, thanks!

On Thu, 7 Jun 2018, 11:16 Matthias J. Sax,  wrote:

> Dear all,
>
> I want to propose a 0.11.0.3 bug fix release. 0.11.0.2 is 6 months old
> and a couple of critical fixes are available for 0.11.0.3.
>
> Please find a list of all 16 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6925?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.11.0.3
>
> There are no open tickets with target version 0.11.0.3 at the moment. If
> there are any tickets you want to get included in 0.11.0.3 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.11.0.3 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3
>
>
> Thanks a lot,
>   Matthias
>
>
>

Re: [DISCUSS] 0.10.2.2 bug fix release

[Ismael Juma]

One more thing, I suggest we go with what we have  for all 3 releases you
are doing. We should aim to make the big fix release process as smooth as
possible and we should strive to avoid last minute additions to older
release branches. We can be a bit more flexible for 1.0.2 since it's more
recent.

Ismael

On 7 Jun 2018 4:34 pm, "Ismael Juma"  wrote:

Thanks for doing this Matthias, +1.

Ismael

On Thu, Jun 7, 2018 at 1:50 PM Matthias J. Sax 
wrote:

> Dear all,
>
> I want to propose a 0.10.2.2 bug fix release. 0.10.2.1 is over a year
> old and a couple of critical fixes are available for 0.10.2.2.
>
> Please find a list of all 24 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6566?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.2%20AND%20resolution%20!%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20
>
> There are no open tickets with target version 0.10.2.2 at the moment. If
> there are any tickets you want to get included in 0.10.2.2 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.10.2.2 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2
>
>
> Thanks a lot,
>   Matthias
>
>
>
>

Re: kafka ack=all and min-isr

[Ismael Juma]

The key point is:

if (leaderReplica.highWatermark.messageOffset >= requiredOffset) {

The high watermark only moves when all the replicas in ISR have that
particular offset. Does that clarify it?

Ismael

On Thu, Jun 7, 2018 at 10:31 PM Carl Samuelson 
wrote:

> Hi
>
> Hopefully this is the correct email address and forum for this.
> I asked this question on stack overflow, but did not get an answer:
> https://stackoverflow.com/questions/50689177/kafka-ack-all-and-min-isr
>
> *Summary*
>
> The docs and code comments for Kafka suggest that when the producer setting
> acks is set to allthen an ack will only be sent to the producer when *all
> in-sync replicas have caught up*, but the code (Partition.Scala,
> checkEnoughReplicasReachOffset) seems to suggest that the ack is sent as
> soon as *min in-sync replicas have caught up*.
>
> *Details*
>
> The kafka docs have this:
>
> acks=all This means the leader will wait for the full set of in-sync
> replicas to acknowledge the record. source
> 
>
> Also, looking at the Kafka source code - partition.scala
> checkEnoughReplicasReachOffset() has the following comment (emphasis mine):
>
> Note that this method will only be called if requiredAcks = -1 and we are
> waiting for *all replicas*in ISR to be fully caught up to the (local)
> leader's offset corresponding to this produce request before we acknowledge
> the produce request.
>
> Finally, this answer  on
> Stack
> Overflow (emphasis mine again)
>
> Also the min in-sync replica setting specifies the minimum number of
> replicas that need to be in-sync for the partition to remain available for
> writes. When a producer specifies ack (-1 / all config) it will still wait
> for acks from *all in sync replicas* at that moment (independent of the
> setting for min in-sync replicas).
>
> But when I look at the code in Partition.Scala (note minIsr <
> curInSyncReplicas.size):
>
> def checkEnoughReplicasReachOffset(requiredOffset: Long): (Boolean,
> Errors) = {
>   ...
>   val minIsr = leaderReplica.log.get.config.minInSyncReplicas
>   if (leaderReplica.highWatermark.messageOffset >= requiredOffset) {
> if (minIsr <= curInSyncReplicas.size)
>   (true, Errors.NONE)
>
> The code that calls this returns the ack:
>
> if (error != Errors.NONE || hasEnough) {
>   status.acksPending = false
>   status.responseStatus.error = error
> }
>
> So, the code looks like it returns an ack as soon as the in-sync replica
> set are greater than min in-sync replicas. However, the documentation and
> comments suggest that the ack is only sent once all in-sync replicas have
> caught up. What am I missing? At the very least, the comment above
> checkEnoughReplicasReachOffset looks like it should be changed.
> Regards,
>
> Carl
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

; > > >
> > > > > > > > > > > > +1 for using `default.block.ms`.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dhruvil
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck <
> > > > > bbej...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > > >
> > > > > > > > > > > > > At first, I thought the same name between the
> > producer
> > > > and
> > > > > the
> > > > > > > > > consumer
> > > > > > > > > > > > was
> > > > > > > > > > > > > ideal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But your comment makes me realize consistent names
> > with
> > > > > > > different
> > > > > > > > > > > > semantics
> > > > > > > > > > > > > is even more confusing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm +1 for not using `max.block.ms`.  I like
> > Guozhang's
> > > > > > > > > suggestion of
> > > > > > > > > > > `
> > > > > > > > > > > > > default.block.ms` for the name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Bill
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang <
> > > > > > > wangg...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah I agree that "max.block.ms" makes people
> > thinking
> > > > > of
> > > > > > > the
> > > > > > > > > > > > producer's
> > > > > > > > > > > > > > config with the same name, but their semantics
> are
> > > > > different.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On the other hand, I'm a bit concerned with the
> > reusing
> > > > > of
> > > > > > > the
> > > > > > > > > term
> > > > > > > > > > > > > > `timeout` as we already have `session.timeout.ms
> `
> > and
> > > > `
> > > > > > > > > > > > > request.timeout.ms`
> > > > > > > > > > > > > > in ConsumerConfig.. How about using the name `
> > > > > > > > > default.api.block.ms`
> > > > > > > > > > > or
> > > > > > > > > > > > > > simply `default.block.ms`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Guozhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson <
> > > > > > > > > ja...@confluent.io>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey All,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > &g

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Ismael Juma]

Sorry for the delay Dongjin. Everyone is busy finalising 2.0.0. This KIP
seems like a great candidate for 2.1.0 and hopefully there will be more of
a discussion next week. :)

Ismael

On Wed, 13 Jun 2018, 05:17 Dongjin Lee,  wrote:

> Hello. I just updated my draft implementation:
>
> 1. Rebased to latest trunk (commit 5145d6b)
> 2. Apply ZStd 1.3.4
>
> You can check out the implementation from here
> . If you experience any problem
> running it, don't hesitate to give me a mention.
>
> Best,
> Dongjin
>
> On Tue, Jun 12, 2018 at 6:50 PM Dongjin Lee  wrote:
>
> > Here is the short conclusion about the license problem: *We can use zstd
> > and zstd-jni without any problem, but we need to include their license,
> > e.g., BSD license.*
> >
> > Both of BSD 2 Clause License & 3 Clause License requires to include the
> > license used, and BSD 3 Clause License requires that the name of the
> > contributor can't be used to endorse or promote the product. That's it
> > <
> http://www.mikestratton.net/2011/12/is-bsd-license-compatible-with-apache-2-0-license/
> >
> > - They are not listed in the list of prohibited licenses
> >  also.
> >
> > Here is how Spark did for it
> > :
> >
> > - They made a directory dedicated to the dependency license files
> >  and added
> licenses
> > for Zstd
> > 
> &
> > Zstd-jni
> > <
> https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd-jni.txt>
> > .
> > - Added a link to the original license files in LICENSE.
> > 
> >
> > If needed, I can make a similar update.
> >
> > Thanks for pointing out this problem, Viktor! Nice catch!
> >
> > Best,
> > Dongjin
> >
> >
> >
> > On Mon, Jun 11, 2018 at 11:50 PM Dongjin Lee  wrote:
> >
> >> I greatly appreciate your comprehensive reasoning. so: +1 for b until
> now.
> >>
> >> For the license issues, I will have a check on how the over projects are
> >> doing and share the results.
> >>
> >> Best,
> >> Dongjin
> >>
> >> On Mon, Jun 11, 2018 at 10:08 PM Viktor Somogyi <
> viktorsomo...@gmail.com>
> >> wrote:
> >>
> >>> Hi Dongjin,
> >>>
> >>> A couple of comments:
> >>> I would vote for option b. in the "backward compatibility" section. My
> >>> reasoning for this is that users upgrading to a zstd compatible version
> >>> won't start to use it automatically, so manual reconfiguration is
> >>> required.
> >>> Therefore an upgrade won't mess up the cluster. If not all the clients
> >>> are
> >>> upgraded but just some of them and they'd start to use zstd then it
> would
> >>> cause errors in the cluster. I'd like to presume though that this is a
> >>> very
> >>> obvious failure case and nobody should be surprised if it didn't work.
> >>> I wouldn't choose a. as I think we should bump the fetch and produce
> >>> requests if it's a change in the message format. Moreover if some of
> the
> >>> producers and the brokers are upgraded but some of the consumers are
> not,
> >>> then we wouldn't prevent the error when the old consumer tries to
> consume
> >>> the zstd compressed messages.
> >>> I wouldn't choose c. either as I think binding the compression type to
> an
> >>> API is not so obvious from the developer's perspective.
> >>>
> >>> I would also prefer to use the existing binding, however we must
> respect
> >>> the licenses:
> >>> "The code for these JNI bindings is licenced under 2-clause BSD
> license.
> >>> The native Zstd library is licensed under 3-clause BSD license and
> GPL2"
> >>> Based on the FAQ page
> >>> https://www.apache.org/legal/resolved.html#category-a
> >>> we may use 2- and 3-clause BSD licenses but the Apache license is not
> >>> compatible with GPL2. I'm hoping that the "3-clause BSD license and
> GPL2"
> >>> is really not an AND but an OR in this case, but I'm no lawyer, just
> >>> wanted
> >>> to make the point that we should watch out for licenses. :)
> >>>
> >>> Regards,
> >>> Viktor
> >>>
> >>>
> >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou  wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > This is Ivan and I still very much support the fact that zstd
> >>> compression
> >>> > should be included out of the box.
> >>> >
> >>> > Please think about the environment, you can save quite a lot of
> >>> hardware
> >>> > with it.
> >>> >
> >>> > Thank you.
> >>> >
> >>> > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee  wrote:
> >>> >
> >>> > > Since there are no responses for a week, I decided to reinitiate
> the
> >>> > > discussion thread.
> >>> > >
> >>> > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression
> >>> > >
> >>> > > This KIP is about to introduce ZStandard Compression into Apache
> >>> Kafka.
> >>> > > The reason why it is

Re: [VOTE] 1.1.1 RC0

[Ismael Juma]

Thanks Dong. One issue that came up is that Gradle dependency resolution no
longer works with Java 7 due to a minimum TLS version change in Maven
Central. Gradle 4.8.1 solves the issue (
https://github.com/apache/kafka/pull/5263). This means that if someone
tries to build from source with Java 7, it won't work unless they have the
artifacts in the local cache already. Do you think it makes sense to do RC1
with the Gradle bump to avoid this issue?

Ismael

On Tue, Jun 19, 2018 at 4:29 PM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc0 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc0
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/150/
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
> Cheers,
> Dong
>

Re: [VOTE] 0.11.0.3 RC0

[Ismael Juma]

+1 (binding)

Verified signature of source artifact, ran tests and verified quickstart on
source artifact with Java 7, verified quickstart on binary artifact (Scala
2.11) with Java 7, sanity checked release notes and Maven staging
repository.

Thanks for running the release!

Ismael

On Fri, Jun 22, 2018 at 3:14 PM Matthias J. Sax 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 0.11.0.3.
>
> This is a bug fix release closing 27 tickets:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3
>
> Release notes for the 0.11.0.3 release:
> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
> can close the vote on Wednesday.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/javadoc/
>
> * Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.3 tag:
> https://github.com/apache/kafka/releases/tag/0.11.0.3-rc0
>
> * Documentation:
> http://kafka.apache.org/0110/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0110/protocol.html
>
> * Successful Jenkins builds for the 0.11.0 branch:
> Unit/integration tests:
> https://builds.apache.org/job/kafka-0.11.0-jdk7/385/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/0.11.0/217/
>
> /**
>
> Thanks,
>   -Matthias
>
>

Re: [VOTE] 1.0.2 RC1

[Ismael Juma]

+1 (binding)

Verified signature of source artifact, ran tests and verified quickstart on
source artifact with Java 7, verified quickstart on binary artifact (Scala
2.12) with Java 8, sanity checked release notes and Maven staging
repository.

Ismael

On Fri, Jun 29, 2018 at 10:02 PM Matthias J. Sax 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 1.0.2.
>
> This is a bug fix release addressing 27 tickets:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
>
> Release notes for the 1.0.2 release:
> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by end of next week (7/6/18).
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~mjsax/kafka-1.0.2-rc1/javadoc/
>
> * Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
> https://github.com/apache/kafka/releases/tag/1.0.2-rc1
>
> * Documentation:
> http://kafka.apache.org/10/documentation.html
>
> * Protocol:
> http://kafka.apache.org/10/protocol.html
>
> * Successful Jenkins builds for the 1.0 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/214/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.0/225/
>
> /**
>
> Thanks,
>   -Matthias
>
>
>

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

[Ismael Juma]

Thanks Matthias!

On Tue, 3 Jul 2018, 11:31 Matthias J. Sax,  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 0.11.0.3.
>
>
> This is a bug fix release and it includes fixes and improvements from
> 27 JIRAs, including a few critical bugs.
>
>
> All of the changes in this release can be found in the release notes:
>
>
> https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.
> html
> <https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/RELEASE_NOTES.html>
>
>
>
> You can download the source release from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.
> 3-src.tgz
> <https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka-0.11.0.3-src.tgz>
>
>
> and binary releases from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.11-0.
> 11.0.3.tgz
> (Scala 2.11)
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.3/kafka_2.12-0.
> 11.0.3.tgz
> (Scala 2.12)
>
>
> - 
> - ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records
> to one or more Kafka topics.
>
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.three key capabilities:
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
>
>
> A big thank you for the following 26 contributors to this release!
>
>
> Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
>
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
>
> Thank you!
>
>
> Regards,
>  -Matthias
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> =4xXe
> -END PGP SIGNATURE-
>

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Ismael Juma]

Thanks for the KIP, Stanislav. The following PR looks related:

https://github.com/apache/kafka/pull/4093/files

Ismael

On Thu, Jul 5, 2018 at 8:44 AM Stanislav Kozlovski 
wrote:

> Hey everybody,
>
> I just created a new KIP about exposing more information in exceptions
> caused by consumer record deserialization/validation. Please have a look at
> it, it is a very short page.
>
> I am working under the assumption that all invalid record or
> deserialization exceptions in the consumer pass through the `Fetcher`
> class. Please confirm if that is true, otherwise I might miss some places
> where the exceptions are raised in my implementation
>
> One concern I have is the name of the second exception -
> `InoperativeRecordException`. I would have named it
> `InvalidRecordException` but that is taken. The `Fetcher` class catches
> `InvalidRecordException` (here
> <
> https://github.com/apache/kafka/blob/c5b00d20d3703b7fc4358b7258d5d6adb890136f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1081
> >
> and here
> <
> https://github.com/apache/kafka/blob/c5b00d20d3703b7fc4358b7258d5d6adb890136f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1092
> >)
> and re-raises it as `KafkaException`, which exposes it as a non-retriable
> exception to the user (`InvalidRecordException` extends
> `RetriableExecption`, but `KafkaException` doesn't).
> A suggestion I got for an alternative name was
> `InvalidFetchRecordException`. Please chime in if you have ideas
>
> Confluence page: KIP-334
>  >
> JIRA Issue: KAFKA-5682 
> --
> Best,
> Stanislav
>

Re: [kafka-clients] [VOTE] 1.1.1 RC2

[Ismael Juma]

Thanks Dong. There are a couple of quickstart links below:

https://repository.apache.org/content/groups/staging/org/apache/kafka/

Ismael

On Thu, Jul 5, 2018 at 10:32 AM Dong Lin  wrote:

> Hey everyone,
>
> Thanks for the vote! Sorry for late reply. I have been trying to understand
> whether streams-quickstart is required for release and how to fix the
> issue. I have limited availability to fix this issue in the past few days.
>
> It seems that all contents to be verified (e.g. source file, binary file,
> quick start) and verification steps (e.g. signature,
> https://kafka.apache.org/11/documentation/streams/quickstart) as shown in
> the first email are not affected directly by the kafka-streams-quickstart
> artifacts. So I think this RC can go ahead after kafka-streams-quickstart
> artifacts is uploaded. Hopefully community does not have to re-test and
> re-vote for this release again :) Please correct me if this is wrong.
>
> Hey Matthias,
>
> Thanks much for helping me with the steps to upload streams-quickstart! I
> have successfully run the "mvn deploy" command for the streams-quickstart.
> Do you know where to check if this has been successfully uploaded?
>
> Thanks,
> Dong
>
>
> On Thu, Jul 5, 2018 at 2:06 AM, Skrzypek, Jonathan <
> jonathan.skrzy...@gs.com
> > wrote:
>
> > Hi,
> >
> > Will this RC go ahead or should a RC3 be put together ?
> >
> >
> > -Original Message-
> > From: Matthias J. Sax [mailto:matth...@confluent.io]
> > Sent: 30 June 2018 06:13
> > To: Rajini Sivaram; Users
> > Cc: Dong Lin; dev; kafka-clients
> > Subject: Re: [kafka-clients] [VOTE] 1.1.1 RC2
> >
> > Hi Dong,
> >
> > it seems that the kafka-streams-quickstart artifacts are missing. Is it
> > just me or is the RC incomplete?
> >
> >
> > -Matthias
> >
> >
> > On 6/29/18 4:07 PM, Rajini Sivaram wrote:
> > > Hi Dong,
> > >
> > > +1 (binding)
> > >
> > > Verified binary using quick start, ran tests from source, checked
> > > release notes.
> > >
> > > Thanks for running the release!
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Fri, Jun 29, 2018 at 11:11 PM, Jun Rao  > > > wrote:
> > >
> > > Hi, Dong,
> > >
> > > Thanks for running the release. Verified quickstart on scala 2.12
> > > binary. +1
> > >
> > > Jun
> > >
> > > On Thu, Jun 28, 2018 at 6:12 PM, Dong Lin  > > > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for release of Apache Kafka 1.1.1.
> > > >
> > > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that
> > > was first
> > > > released with 1.1.0 about 3 months ago. We have fixed about 25
> > > issues since
> > > > that release. A few of the more significant fixes include:
> > > >
> > > > KAFKA-6925  > > > - Fix
> > > > memory leak in StreamsMetricsThreadImpl
> > > > KAFKA-6937  > > > - In-sync
> > > > replica delayed during fetch if replica throttle is exceeded
> > > > KAFKA-6917  > > > - Process
> > > > txn completion asynchronously to avoid deadlock
> > > > KAFKA-6893  > > > - Create
> > > > processors before starting acceptor to avoid ArithmeticException
> > > > KAFKA-6870  > > > -
> > > > Fix ConcurrentModificationException in SampledStat
> > > > KAFKA-6878  > > > - Fix
> > > > NullPointerException when querying global state store
> > > > KAFKA-6879  > > > - Invoke
> > > > session init callbacks outside lock to avoid Controller deadlock
> > > > KAFKA-6857  > > > - Prevent
> > > > follower from truncating to the wrong offset if undefined leader
> > > epoch is
> > > > requested
> > > > KAFKA-6854  > > > - Log
> > > > cleaner fails with transaction markers that are deleted during
> > clean
> > > > KAFKA-6747  > > > - Check
> > > > whether there is in-flight transaction before 

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Ismael Juma]

Yes, the Scala consumers have been removed in 2.0.0, which simplifies some
of this. The following commit was an initial step in unifying the exception
handling:

https://github.com/apache/kafka/commit/96bcfdfc7c9aac075635b2034e65e412a725672e

But more can be done as you mentioned.

Ismael

On 5 Jul 2018 9:36 am, "Stanislav Kozlovski"  wrote:

Hey Ismael,

It is only slightly related - my PR would attach two new attributes and
also touch upon deserialization exceptions.

But this PR did provide me with some insight:
Maybe the best approach would be to make `InvalidRecordException` a public
exception instead of introducing a new one - I did not realize it was not
publicly exposed.
Does the following:

 InvalidMessageException extends CorruptRecordException for temporary
compatibility with the old Scala clients.
 * We want to update the server side code to use and catch the new
CorruptRecordException.
 * Because ByteBufferMessageSet.scala and Message.scala are used in
both server and client code having
 * InvalidMessageException extend CorruptRecordException allows us to
change server code without affecting the client.

still apply? I can see that the `ByteBufferMessageSet` and `Message` scala
classes are not present in the codebase anymore. AFAIA the old scala
clients were removed with 2.0.0 and we can thus update the server side code
to use the `CorruptRecordException` while changing and exposing
`InvalidRecordException` to the public. WDYT?

I will also make sure to not expose the cause of the exception when not
needed, maybe I'll outright remove the `cause` attribute


On Thu, Jul 5, 2018 at 4:55 PM Ismael Juma  wrote:

> Thanks for the KIP, Stanislav. The following PR looks related:
>
> https://github.com/apache/kafka/pull/4093/files
>
> Ismael
>
> On Thu, Jul 5, 2018 at 8:44 AM Stanislav Kozlovski  >
> wrote:
>
> > Hey everybody,
> >
> > I just created a new KIP about exposing more information in exceptions
> > caused by consumer record deserialization/validation. Please have a look
> at
> > it, it is a very short page.
> >
> > I am working under the assumption that all invalid record or
> > deserialization exceptions in the consumer pass through the `Fetcher`
> > class. Please confirm if that is true, otherwise I might miss some
places
> > where the exceptions are raised in my implementation
> >
> > One concern I have is the name of the second exception -
> > `InoperativeRecordException`. I would have named it
> > `InvalidRecordException` but that is taken. The `Fetcher` class catches
> > `InvalidRecordException` (here
> > <
> >
>
https://github.com/apache/kafka/blob/c5b00d20d3703b7fc4358b7258d5d6adb890136f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1081
> > >
> > and here
> > <
> >
>
https://github.com/apache/kafka/blob/c5b00d20d3703b7fc4358b7258d5d6adb890136f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1092
> > >)
> > and re-raises it as `KafkaException`, which exposes it as a
non-retriable
> > exception to the user (`InvalidRecordException` extends
> > `RetriableExecption`, but `KafkaException` doesn't).
> > A suggestion I got for an alternative name was
> > `InvalidFetchRecordException`. Please chime in if you have ideas
> >
> > Confluence page: KIP-334
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87297793
> > >
> > JIRA Issue: KAFKA-5682 <https://issues.apache.org/jira/browse/KAFKA-5682
> >
> > --
> > Best,
> > Stanislav
> >
>


-- 
Best,
Stanislav

Re: [kafka-clients] [VOTE] 1.1.1 RC2

[Ismael Juma]

We need to decide if KAFKA-7136 is a blocker or not. At the moment, it's
not clear to me.

Ismael

On Thu, 5 Jul 2018, 14:41 Dong Lin,  wrote:

> Thanks much for the information Ismael. It seems that the the
> streams-quickstart files (link
> <https://repository.apache.org/content/groups/staging/org/apache/kafka/streams-quickstart/1.1.1/>)
> are uploaded on July 5 whereas the binary files (link
> <https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/1.1.1/>)
> are uploaded on Jun 28.
>
> Maybe this RC is good to go? If so, can we get another +1 from PMC?
>
> Thanks!
> Dong
>
> On Thu, Jul 5, 2018 at 10:38 AM, Ismael Juma  wrote:
>
Thanks Dong. There are a couple of quickstart links below:
>>
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> Ismael
>>
>> On Thu, Jul 5, 2018 at 10:32 AM Dong Lin  wrote:
>>
>>> Hey everyone,
>>>
>>> Thanks for the vote! Sorry for late reply. I have been trying to
>>> understand
>>> whether streams-quickstart is required for release and how to fix the
>>> issue. I have limited availability to fix this issue in the past few
>>> days.
>>>
>>> It seems that all contents to be verified (e.g. source file, binary file,
>>> quick start) and verification steps (e.g. signature,
>>> https://kafka.apache.org/11/documentation/streams/quickstart) as shown
>>> in
>>> the first email are not affected directly by the kafka-streams-quickstart
>>> artifacts. So I think this RC can go ahead after kafka-streams-quickstart
>>> artifacts is uploaded. Hopefully community does not have to re-test and
>>> re-vote for this release again :) Please correct me if this is wrong.
>>>
>>> Hey Matthias,
>>>
>>> Thanks much for helping me with the steps to upload streams-quickstart! I
>>> have successfully run the "mvn deploy" command for the
>>> streams-quickstart.
>>> Do you know where to check if this has been successfully uploaded?
>>>
>>> Thanks,
>>> Dong
>>>
>>>
>>> On Thu, Jul 5, 2018 at 2:06 AM, Skrzypek, Jonathan <
>>> jonathan.skrzy...@gs.com
>>> > wrote:
>>>
>>> > Hi,
>>> >
>>> > Will this RC go ahead or should a RC3 be put together ?
>>> >
>>> >
>>> > -Original Message-
>>> > From: Matthias J. Sax [mailto:matth...@confluent.io]
>>> > Sent: 30 June 2018 06:13
>>> > To: Rajini Sivaram; Users
>>> > Cc: Dong Lin; dev; kafka-clients
>>> > Subject: Re: [kafka-clients] [VOTE] 1.1.1 RC2
>>> >
>>> > Hi Dong,
>>> >
>>> > it seems that the kafka-streams-quickstart artifacts are missing. Is it
>>> > just me or is the RC incomplete?
>>> >
>>> >
>>> > -Matthias
>>> >
>>> >
>>> > On 6/29/18 4:07 PM, Rajini Sivaram wrote:
>>> > > Hi Dong,
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > Verified binary using quick start, ran tests from source, checked
>>> > > release notes.
>>> > >
>>> > > Thanks for running the release!
>>> > >
>>> > > Regards,
>>> > >
>>> > > Rajini
>>> > >
>>> > > On Fri, Jun 29, 2018 at 11:11 PM, Jun Rao >> > > <mailto:j...@confluent.io>> wrote:
>>> > >
>>> > > Hi, Dong,
>>> > >
>>> > > Thanks for running the release. Verified quickstart on scala 2.12
>>> > > binary. +1
>>> > >
>>> > > Jun
>>> > >
>>> > > On Thu, Jun 28, 2018 at 6:12 PM, Dong Lin >> > > <mailto:lindon...@gmail.com>> wrote:
>>> > >
>>> > > > Hello Kafka users, developers and client-developers,
>>> > > >
>>> > > > This is the second candidate for release of Apache Kafka 1.1.1.
>>> > > >
>>> > > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that
>>> > > was first
>>> > > > released with 1.1.0 about 3 months ago. We have fixed about 25
>>> > > issues since
>>> > > > that release. A few of the more significant fixes include:
>>> > > >
>>> > > > 

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Ismael Juma]

Thanks for the KIP. I think we should be thinking about
https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.

Ismael

On Thu, 5 Jul 2018, 07:45 Chia-Ping Tsai,  wrote:

> hi all,
>
> I would like to start voting on "KIP-331 Add default implementation to
> close() and configure() for Serializer, Deserializer and Serde"
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>
> Cheers,
> Chia-Ping
>

Re: [VOTE] KIP-280: Enhanced log compaction

[Ismael Juma]

Thanks for the KIP, Luis. A brief comment below.

On Wed, Jul 4, 2018 at 11:11 AM Luís Cabral 
wrote:

> As a reader, I tend to prefer brief documentation on new features (they
> tend to be too many for me to find the willpower to read a 200-page essay
> about each one), so that influences me to avoid writing about every
> micro-impact that may exist, and simply leave it inferred (as you have just
> done).
> But I also don’t feel strongly enough about it to argue either way. So,
> after reading my argument, if you still insist, I’ll happily add this there.
>

KIPs are not your typical user level documentation. We strive to document
details like the one Jun pointed out as they're beneficial during review,
but also from understanding the operations impact.

Ismael

Re: Old deprecated producer

[Ismael Juma]

The old Scala producers were removed in 2.0.0. The Java producer supports
batching.

Ismael


On Sat, 7 Jul 2018, 05:38 jna,  wrote:

> Hello,
>
> I'm using the old producer API, and i saw since a long time (few
> versions) that this API is deprecated. When will you decide to remove
> this old API? If you won't remove it, perhaps you could remove the
> deprecated. Will you provide a new way to produce a user batch of
> records or the transactions replace already this batch producer?
>
> Thanks.
>
>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Ismael Juma]

The KIP needs 3 binding votes to pass. I still think we should include the
details of how things will look like with the headers being passed to
serializers/deserializers to ensure things actually make sense as a whole.

Ismael


On Sun, 8 Jul 2018, 18:31 Chia-Ping Tsai,  wrote:

> All,
>
> The 72 hours has passed. The vote result of KIP-313 is shown below.
>
> 1 binding vote (Matthias J. Sax)
> 4 non-binding votes (John Roesler, Richard Yu, vito jeng and Chia-Ping)
>
> Cheers,
> Chia-Ping
>
> On 2018/07/05 14:45:01, Chia-Ping Tsai  wrote:
> > hi all,
> >
> > I would like to start voting on "KIP-331 Add default implementation to
> close() and configure() for Serializer, Deserializer and Serde"
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >
> > Cheers,
> > Chia-Ping
> >
>

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

Thanks for the KIP. It would be helpful to understand the user experience
for the case where the implementor uses the headers. It seems like it would
require overriding two methods?

Ismael

On Mon, Jul 9, 2018 at 1:50 AM Viktor Somogyi 
wrote:

> Hi folks,
>
> I've published KIP-336 which is about consolidating the
> Serializer/Deserializer interfaces.
>
> Basically the story here is when ExtendedSerializer and
> ExtendedDeserializer were added we still supported Java 7 and therefore had
> to use compatible constructs which now seem unnecessary since we dropped
> support for Java 7. Now in this KIP I propose a way to deprecate those
> patterns:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
>
> I'd be happy to receive some feedback about the KIP I published.
>
> Cheers,
> Viktor
>

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

[Ismael Juma]

Late +1 (binding) from me too.

Ismael

On Mon, Jul 9, 2018 at 3:18 PM Vahid S Hashemian 
wrote:

> KIP-231 is approved with 3 "+1" binding votes by Ewen, Jason, and Harsha.
>
> Thanks to everyone who provided feedback and/or voted.
>
> Regards.
> --Vahid
>
>
>
>
> From:   Vahid S Hashemian/Silicon Valley/IBM
> To: dev 
> Date:   12/19/2017 11:30 AM
> Subject:[VOTE] KIP-231: Improve the Required ACL of ListGroups API
>
>
> I believe the concerns on this KIP have been addressed so far.
> Therefore, I'd like to start a vote.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API
>
> Thanks.
> --Vahid
>
>
>
>

Re: [VOTE] 2.0.0 RC2

[Ismael Juma]

Hi Rajini,

Thanks for generating the RC. It seems like the kafka-streams-scala 2.12
artifact is missing from the Maven repository:

https://repository.apache.org/content/groups/staging/org/apache/kafka/

Since this is the first time we are publishing this artifact, it is
possible that this never worked properly.

Ismael

On Tue, Jul 10, 2018 at 10:17 AM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new  KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
> A few notable highlights:
>
>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
>(KIP-277)
>- SASL/OAUTHBEARER implementation (KIP-255)
>- Improved quota communication and customization of quotas (KIP-219,
>KIP-257)
>- Efficient memory usage for down conversion (KIP-283)
>- Fix log divergence between leader and follower during fast leader
>failover (KIP-279)
>- Drop support for Java 7 and remove deprecated code including old scala
>clients
>- Connect REST extension plugin, support for externalizing secrets and
>improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>- Scala API for Kafka Streams and other Streams API improvements
>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

Hi Viktor,

The approach where all the methods have a default implementation and the
user chooses to override one of them seems the most appealing to me given
the current state. It doesn't seem like we give up much in that case, what
do you think?

Ismael

On Tue, Jul 10, 2018 at 7:15 AM Viktor Somogyi 
wrote:

> Hi Ismael,
>
> Well, yes. If we care about headers only then you'd need to add a dummy
> implementation for the 2 parameter method as well. Although it is not
> ideal, we're using the Serializer interface everywhere and convert it to
> extended with ensureExtended(serializer) and delegate to the 2 parameter
> method inside the wrapper which is returned in ensureExtended. Because of
> backward compatibility we have to keep delegating but I could perhaps add a
> dummy implementation for the 2 parameter too if you and others think that
> would be better. In this case though we'd have an interface where all
> methods are default (given the improvements of KIP-331
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >)
> and would have to rethink if this interface should be a
> @FunctionalInterface.
> I don't really have a context currently on how the 3 parameter method is
> used, most the code samples I found on github were using the 2 parameter
> method. I think I found one instance where the 3 parameter one was used but
> that delegated to the 2 param one :). Have to say though that this research
> is not representative.
> All in all I think it wouldn't hurt to provide a default implementation for
> the 2 param method too but then we have to give up the @FunctionalInterface
> annotation and we'll end up with an interface with no abstract methods but
> only defaults.
> What do you think?
>
> Cheers,
> Viktor
>
>
> On Mon, Jul 9, 2018 at 11:02 AM Ismael Juma  wrote:
>
> > Thanks for the KIP. It would be helpful to understand the user experience
> > for the case where the implementor uses the headers. It seems like it
> would
> > require overriding two methods?
> >
> > Ismael
> >
> > On Mon, Jul 9, 2018 at 1:50 AM Viktor Somogyi 
> > wrote:
> >
> > > Hi folks,
> > >
> > > I've published KIP-336 which is about consolidating the
> > > Serializer/Deserializer interfaces.
> > >
> > > Basically the story here is when ExtendedSerializer and
> > > ExtendedDeserializer were added we still supported Java 7 and therefore
> > had
> > > to use compatible constructs which now seem unnecessary since we
> dropped
> > > support for Java 7. Now in this KIP I propose a way to deprecate those
> > > patterns:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
> > >
> > > I'd be happy to receive some feedback about the KIP I published.
> > >
> > > Cheers,
> > > Viktor
> > >
> >
>

Re: [VOTE] 1.1.1 RC3

[Ismael Juma]

+1 (binding)

Verified signature of source artifact, ran tests and verified quickstart on
source artifact with Java 8, verified quickstart on binary artifact (Scala
2.12) with Java 8, sanity checked release notes and Maven staging
repository.

Thanks for managing the release Dong!

Ismael

On Sun, Jul 8, 2018 at 3:36 PM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> *
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
>
> Regards,
> Dong
>

Re: [VOTE] 2.0.0 RC2

[Ismael Juma]

Thanks Rajini! A documentation issue that we must fix before the release
(but does not require another RC), 1.2 (which became 2.0) is mentioned in
the upgrade notes:

http://kafka.apache.org/20/documentation.html#upgrade

Ismael

On Sun, Jul 15, 2018 at 9:25 AM Rajini Sivaram 
wrote:

> Hi Ismael,
>
> Thank you for pointing that out. I have re-uploaded the RC2 artifacts to
> maven including streams-scala_2.12. Also submitted a PR to update build &
> release scripts to include this.
>
> Thank you,
>
> Rajini
>
>
>
> On Fri, Jul 13, 2018 at 7:19 AM, Ismael Juma  wrote:
>
> > Hi Rajini,
> >
> > Thanks for generating the RC. It seems like the kafka-streams-scala 2.12
> > artifact is missing from the Maven repository:
> >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > Since this is the first time we are publishing this artifact, it is
> > possible that this never worked properly.
> >
> > Ismael
> >
> > On Tue, Jul 10, 2018 at 10:17 AM Rajini Sivaram  >
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > >
> > > This is the third candidate for release of Apache Kafka 2.0.0.
> > >
> > >
> > > This is a major version release of Apache Kafka. It includes 40 new
> KIPs
> > > and
> > >
> > > several critical bug fixes. Please see the 2.0.0 release plan for more
> > > details:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=80448820
> > >
> > >
> > > A few notable highlights:
> > >
> > >- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> CreateTopics
> > >(KIP-277)
> > >- SASL/OAUTHBEARER implementation (KIP-255)
> > >- Improved quota communication and customization of quotas (KIP-219,
> > >KIP-257)
> > >- Efficient memory usage for down conversion (KIP-283)
> > >- Fix log divergence between leader and follower during fast leader
> > >failover (KIP-279)
> > >- Drop support for Java 7 and remove deprecated code including old
> > scala
> > >clients
> > >- Connect REST extension plugin, support for externalizing secrets
> and
> > >improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > >- Scala API for Kafka Streams and other Streams API improvements
> > >(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> > >
> > >
> > > Release notes for the 2.0.0 release:
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
> > >
> > >
> > > *** Please download, test and vote by Friday, July 13, 4pm PT
> > >
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >
> > > http://kafka.apache.org/KEYS
> > >
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
> > >
> > >
> > > * Maven artifacts to be voted upon:
> > >
> > > https://repository.apache.org/content/groups/staging/
> > >
> > >
> > > * Javadoc:
> > >
> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
> > >
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> > >
> > > https://github.com/apache/kafka/tree/2.0.0-rc2
> > >
> > >
> > >
> > > * Documentation:
> > >
> > > http://kafka.apache.org/20/documentation.html
> > >
> > >
> > > * Protocol:
> > >
> > > http://kafka.apache.org/20/protocol.html
> > >
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > >
> > > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.0-jdk8/72/
> > >
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
> > >
> > >
> > > /**
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Rajini
> > >
> >
>

Re: [ANNOUNCE] Apache Kafka 1.1.1 Released

[Ismael Juma]

Thank you for managing the release Dong!

Ismael

On Thu, 19 Jul 2018, 16:54 Dong Lin,  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 1.1.1.
>
> This is a bug fix release and it includes fixes and improvements from 43
> JIRAs, including a few critical bugs.
>
> All of the changes in this release can be found in the release notes:
>
> https://dist.apache.org/repos/dist/release/kafka/1.1.1/RELEASE_NOTES.html
>
> You can download the source release from:
>
> *
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka-1.1.1-src.tgz
> <
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka-1.1.1-src.tgz
> >*
>
> and binary releases from:
>
> *
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz
> <
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz
> >*
> (Scala 2.11)
>
> *
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.12-1.1.1.tgz
> <
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.12-1.1.1.tgz
> >*
> (Scala 2.12)
>
>
> 
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one
> or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics
> and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming
> an input stream from one or more topics and producing an output stream to
> one or more output topics, effectively transforming the input streams to
> output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers
> that connect Kafka topics to existing applications or data systems. For
> example, a connector to a relational database might capture every change to
> a table.three key capabilities:
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between
> systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams
> of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 29 contributors to this release!
>
> Ismael Juma, Rajini Sivaram, Matthias J. Sax, Guozhang Wang, Anna Povzner,
> tedyu, Jagadesh Adireddi, John Roesler, Manikumar Reddy O, Randall Hauch,
> Attila Sasvari, Chia-Ping Tsai, Colin Patrick McCabe, Dhruvil Shah, Fedor
> Bobin, Gitomain, Gunnar Morling, Jarek Rudzinski, Jason Gustafson, Jun Rao,
> Mickael Maison, Robert Yokota, Vahid Hashemian, Valentino Proietti, fredfp,
> huxi, maytals, ro7m, yaphet
>
> We welcome your help and feedback. For more information on how to report
> problems,and to get involved, visit the project website at
> http://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> Dong
>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Ismael Juma]

Part of the motivation for this KIP is to make these interfaces functional
interfaces. But I think that may not be desirable due to the method that
passes headers. So, it doesn't make sense to discuss two separate changes
to the same interfaces in isolation, we should figure out how we want them
to work holistically.

Ismael

On Fri, Jul 20, 2018 at 3:50 AM Chia-Ping Tsai  wrote:

> > The KIP needs 3 binding votes to pass.
>
> Thanks for the reminder. I will reopen the ballot box until we get 3
> tickets.
>
> > I still think we should include the details of how things will look like
> > with the headers being passed to serializers/deserializers to ensure
> > things actually make sense as a whole.
>
> This KIP is unrelated to the both methods - serialize() and deserialize().
> We won't add the default implementation to them in this kip. Please correct
> me if I didn't catch what you said.
>
> Cheers,
> Chia-Ping
>
> On 2018/07/09 01:55:41, Ismael Juma  wrote:
> > The KIP needs 3 binding votes to pass. I still think we should include
> the
> > details of how things will look like with the headers being passed to
> > serializers/deserializers to ensure things actually make sense as a
> whole.
> >
> > Ismael
> >
> >
> > On Sun, 8 Jul 2018, 18:31 Chia-Ping Tsai,  wrote:
> >
> > > All,
> > >
> > > The 72 hours has passed. The vote result of KIP-313 is shown below.
> > >
> > > 1 binding vote (Matthias J. Sax)
> > > 4 non-binding votes (John Roesler, Richard Yu, vito jeng and Chia-Ping)
> > >
> > > Cheers,
> > > Chia-Ping
> > >
> > > On 2018/07/05 14:45:01, Chia-Ping Tsai  wrote:
> > > > hi all,
> > > >
> > > > I would like to start voting on "KIP-331 Add default implementation
> to
> > > close() and configure() for Serializer, Deserializer and Serde"
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> > > >
> > > > Cheers,
> > > > Chia-Ping
> > > >
> > >
> >
>

Re: Question about issues of Kafka release version 1.1.1

[Ismael Juma]

Seems like you're right Lambdaliu. Rajini/Jason, can you please check and
update the JIRAs?

Ismael

On Mon, Jul 23, 2018 at 7:09 AM lambdaliu(刘少波) 
wrote:

> Hi team,
>
> I Have downloaded the source release of kafka version 1.1.1 and found the
> JIRA
> issues KAFKA-6911 and KAFKA-6809 listed in the release notes but it's PR
> looks
> like doesn't contain in the source release. Is this a valid situation?
> Should we
> create a JIRA issue to trace it?
>
> Regards,
> Lambdaliu(Shaobo Liu)
> 
>

Re: Build failed in Jenkins: kafka-trunk-jdk10 #342

[Ismael Juma]

Seems like the git repository was corrupt. I forced a fresh clone and the
build is proceeding.

Ismael

On Sat, Jul 28, 2018 at 8:37 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
> --
> Started by user ijuma
> [EnvInject] - Loading node environment variables.
> Building remotely on H23 (ubuntu xenial) in workspace <
> https://builds.apache.org/job/kafka-trunk-jdk10/ws/>
>  > git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>  > git config remote.origin.url https://github.com/apache/kafka.git #
> timeout=10
> Cleaning workspace
>  > git rev-parse --verify HEAD # timeout=10
> Resetting working tree
>  > git reset --hard # timeout=10
> ERROR: Error fetching remote repo 'origin'
> hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/kafka.git
> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
> at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
> at hudson.scm.SCM.checkout(SCM.java:504)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> at hudson.model.Run.execute(Run.java:1794)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:429)
> Caused by: hudson.plugins.git.GitException: Command "git reset --hard"
> returned status code 128:
> stdout:
> stderr: fatal: unable to read tree 032712170749ccd777fa5a97c2a6cddf6de2e692
>
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:463)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:786)
> at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)
> at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> at hudson.remoting.Request$2.run(Request.java:369)
> at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to H23
> at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> at hudson.remoting.Channel.call(Channel.java:955)
> at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> at com.sun.proxy.$Proxy117.clean(Unknown Source)
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:450)
> at
> hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:30)
> at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:884)
> at
> hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
> at hudson.scm.SCM.checkout(SCM.java:504)
> at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> at

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

[Ismael Juma]

>
>
>
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
>
> ** The Producer API allows an application to publish a stream records to
>
> one or more Kafka topics.
>
>
>
> ** The Consumer API allows an application to subscribe to one or more
>
> topics and process the stream of records produced to them.
>
>
>
> ** The Streams API allows an application to act as a stream processor,
>
> consuming an input stream from one or more topics and producing an
>
> output stream to one or more output topics, effectively transforming the
>
> input streams to output streams.
>
>
>
> ** The Connector API allows building and running reusable producers or
>
> consumers that connect Kafka topics to existing applications or data
>
> systems. For example, a connector to a relational database might
>
> capture every change to a table.
>
>
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
>
> ** Building real-time streaming data pipelines that reliably get data
>
> between systems or applications.
>
>
>
> ** Building real-time streaming applications that transform or react
>
> to the streams of data.
>
>
>
>
>
>
>
> Apache Kafka is in use at large and small companies worldwide, including
>
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
>
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
>
>
>
>
> A big thank you for the following 131 contributors to this release!
>
>
>
> Adem Efe Gencer, Alex D, Alex Dunayevsky, Allen Wang, Andras Beni,
>
> Andy Bryant, Andy Coates, Anna Povzner, Arjun Satish, asutosh936,
>
> Attila Sasvari, bartdevylder, Benedict Jin, Bill Bejeck, Blake Miller,
>
> Boyang Chen, cburroughs, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe,
>
> Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, dan norwood,
>
> Daniel Shuy, Daniel Wojda, Dark, David Glasser, Debasish Ghosh, Detharon,
>
> Dhruvil Shah, Dmitry Minkovsky, Dong Lin, Edoardo Comar, emmanuel Harel,
>
> Eugene Sevastyanov, Ewen Cheslack-Postava, Fedor Bobin, fedosov-alexander,
>
> Filipe Agapito, Florian Hussonnois, fredfp, Gilles Degols, gitlw, Gitomain,
>
> Guangxian, Gunju Ko, Gunnar Morling, Guozhang Wang, hmcl, huxi, huxihx,
>
> Igor Kostiakov, Ismael Juma, Jacek Laskowski, Jagadesh Adireddi,
>
> Jarek Rudzinski, Jason Gustafson, Jeff Klukas, Jeremy Custenborder,
>
> Jiangjie (Becket) Qin, Jiangjie Qin, JieFang.He, Jimin Hsieh, Joan Goyeau,
>
> Joel Hamill, John Roesler, Jon Lee, Jorge Quilcate Otoya, Jun Rao,
>
> Kamal C, khairy, Koen De Groote, Konstantine Karantasis, Lee Dongjin,
>
> Liju John, Liquan Pei, lisa2lisa, Lucas Wang, Magesh Nandakumar,
>
> Magnus Edenhill, Magnus Reftel, Manikumar Reddy, Manikumar Reddy O,
>
> manjuapu, Mats Julian Olsen, Matthias J. Sax, Max Zheng, maytals,
>
> Michael Arndt, Michael G. Noll, Mickael Maison, nafshartous, Nick Travers,
>
> nixsticks, Paolo Patierno, parafiend, Patrik Erdes, Radai Rosenblatt,
>
> Rajini Sivaram, Randall Hauch, ro7m, Robert Yokota, Roman Khlebnov,
>
> Ron Dagostino, Sandor Murakozi, Sasaki Toru, Sean Glover,
>
> Sebastian Bauersfeld, Siva Santhalingam, Stanislav Kozlovski, Stephane
> Maarek,
>
> Stuart Perks, Surabhi Dixit, Sönke Liebau, taekyung, tedyu, Thomas Leplus,
>
> UVN, Vahid Hashemian, Valentino Proietti, Viktor Somogyi, Vitaly Pushkar,
>
> Wladimir Schmidt, wushujames, Xavier Léauté, xin, yaphet,
>
> Yaswanth Kumar, ying-zheng, Yu
>
>
>
>
>
>
>
> We welcome your help and feedback. For more information on how to
>
> report problems, and to get involved, visit the project website at
>
> https://kafka.apache.org/
>
>
>
>
>
> Thank you!
>
>
>
>
>
> Regards,
>
>
>
> Rajini
>

Re: KIP-352: Distinguish URPs caused by reassignment

[Ismael Juma]

Thanks Jason. This is definitely a pain point. I actually prefer the option
to redefine what under-replicated means (currently under rejected
alternatives). Also, do we need to make changes to what we store in ZK? If
so, that should be in the KIP too.

Ismael

On Thu, Aug 2, 2018 at 11:45 AM Jason Gustafson  wrote:

> Hey All,
>
> Another day, another KIP. This one is hopefully straightforward:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> .
> Have a look and let me know what you think!
>
> Thanks,
> Jason
>

Re: KIP-352: Distinguish URPs caused by reassignment

[Ismael Juma]

RIght, that was my thinking too.

Ismael

On Fri, Aug 3, 2018 at 12:04 PM Gwen Shapira  wrote:

> On Fri, Aug 3, 2018 at 11:23 AM, Jason Gustafson 
> wrote:
>
> > Hey Ismael,
> >
> > Yeah, my initial inclination was to redefine URP as well. My only doubt
> was
> > how it would affect existing tools which might depend on URPs to track
> the
> > progress of a reassignment. I decided to be conservative in the end, but
> > I'd reconsider if we think it is not a major concern. It is annoying to
> > need a new category.
> >
>
> There are existing tools that use URP to track reassignment, but there are
> many more tools that use URP for monitoring and alerting. If I understand
> Ismael's suggestion correctly, a re-definition will improve the reliability
> of the monitoring tools (since there won't be false alerts in case of
> re-assignment) without having to switch to a new metric.
>
> I think we should choose the proposal that improves the more common usage
> of the metric, in this case, failure monitoring rather than reassignment.
>
>
> >
> > About your question about storage in ZK, I can't think of anything
> > additional that we need. Probably the main difficulty is getting access
> to
> > the replication factor in the topic utility. My basic thought was just to
> > collect the URPs (as we know them today) and use the config API to
> > partition them based on the replication factor. Do you see any problems
> > with this?
> >
> > -Jason
> >
> >
> > On Thu, Aug 2, 2018 at 12:14 PM, Ismael Juma  wrote:
> >
> > > Thanks Jason. This is definitely a pain point. I actually prefer the
> > option
> > > to redefine what under-replicated means (currently under rejected
> > > alternatives). Also, do we need to make changes to what we store in ZK?
> > If
> > > so, that should be in the KIP too.
> > >
> > > Ismael
> > >
> > > On Thu, Aug 2, 2018 at 11:45 AM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey All,
> > > >
> > > > Another day, another KIP. This one is hopefully straightforward:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%
> > > 3A+Distinguish+URPs+caused+by+reassignment
> > > > .
> > > > Have a look and let me know what you think!
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

[Ismael Juma]

On Fri, Jan 5, 2018 at 7:15 PM, Jason Gustafson  wrote:
>
> class OffsetAndMetadata {
>   long offset;
>   byte[] offsetMetadata;
>   String metadata;
> }


> Admittedly, the naming is a bit annoying, but we can probably come up with
> something better. Internally the byte array would have a version. If in the
> future we have anything else we need to add, we can update the version and
> we wouldn't need any new APIs.
>

We can also add fields to a class in a compatible way. So, it seems to me
that the main advantage of the byte array is that it's opaque to the user.
Is that correct? If so, we could also add any opaque metadata in a subclass
so that users don't even see it (unless they cast it, but then they're on
their own).

Ismael

The corresponding seek() and position() APIs might look something like this:
>
> void seek(TopicPartition partition, long offset, byte[] offsetMetadata);
> byte[] positionMetadata(TopicPartition partition);
>
> What do you think?
>
> Thanks,
> Jason
>
> On Thu, Jan 4, 2018 at 7:04 PM, Dong Lin  wrote:
>
> > Hey Jun, Jason,
> >
> > Thanks much for all the feedback. I have updated the KIP based on the
> > latest discussion. Can you help check whether it looks good?
> >
> > Thanks,
> > Dong
> >
> > On Thu, Jan 4, 2018 at 5:36 PM, Dong Lin  wrote:
> >
> > > Hey Jun,
> > >
> > > Hmm... thinking about this more, I am not sure that the proposed API is
> > > sufficient. For users that store offset externally, we probably need
> > extra
> > > API to return the leader_epoch and partition_epoch for all partitions
> > that
> > > consumers are consuming. I suppose these users currently use position()
> > to
> > > get the offset. Thus we probably need a new method
> positionWithEpoch(..)
> > to
> > > return . Does this sound
> > reasonable?
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > > On Thu, Jan 4, 2018 at 5:26 PM, Jun Rao  wrote:
> > >
> > >> Hi, Dong,
> > >>
> > >> Yes, that's what I am thinking. OffsetEpoch will be composed of
> > >> (partition_epoch,
> > >> leader_epoch).
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Thu, Jan 4, 2018 at 4:22 PM, Dong Lin  wrote:
> > >>
> > >> > Hey Jun,
> > >> >
> > >> > Thanks much. I like the the new API that you proposed. I am not sure
> > >> what
> > >> > you exactly mean by offset_epoch. I suppose that we can use the pair
> > of
> > >> > (partition_epoch, leader_epoch) as the offset_epoch, right?
> > >> >
> > >> > Thanks,
> > >> > Dong
> > >> >
> > >> > On Thu, Jan 4, 2018 at 4:02 PM, Jun Rao  wrote:
> > >> >
> > >> > > Hi, Dong,
> > >> > >
> > >> > > Got it. The api that you proposed works. The question is whether
> > >> that's
> > >> > the
> > >> > > api that we want to have in the long term. My concern is that
> while
> > >> the
> > >> > api
> > >> > > change is simple, the new api seems harder to explain and use. For
> > >> > example,
> > >> > > a consumer storing offsets externally now needs to call
> > >> > > waitForMetadataUpdate() after calling seek().
> > >> > >
> > >> > > An alternative approach is to make the following compatible api
> > >> changes
> > >> > in
> > >> > > Consumer.
> > >> > > * Add an additional OffsetEpoch field in OffsetAndMetadata. (no
> need
> > >> to
> > >> > > change the CommitSync() api)
> > >> > > * Add a new api seek(TopicPartition partition, long offset,
> > >> OffsetEpoch
> > >> > > offsetEpoch). We can potentially deprecate the old api
> > >> > seek(TopicPartition
> > >> > > partition, long offset) in the future.
> > >> > >
> > >> > > The alternative approach has similar amount of api changes as
> yours
> > >> but
> > >> > has
> > >> > > the following benefits.
> > >> > > 1. The api works in a similar way as how offset management works
> now
> > >> and
> > >> > is
> > >> > > probably what we want in the long term.
> > >> > > 2. It can reset offsets better when there is data loss due to
> > unclean
> > >> > > leader election or correlated replica failure.
> > >> > > 3. It can reset offsets better when topic is recreated.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > >
> > >> > > On Thu, Jan 4, 2018 at 2:05 PM, Dong Lin 
> > wrote:
> > >> > >
> > >> > > > Hey Jun,
> > >> > > >
> > >> > > > Yeah I agree that ideally we don't want an ever growing global
> > >> metadata
> > >> > > > version. I just think it may be more desirable to keep the
> > consumer
> > >> API
> > >> > > > simple.
> > >> > > >
> > >> > > > In my current proposal, metadata version returned in the fetch
> > >> response
> > >> > > > will be stored with the offset together. More specifically, the
> > >> > > > metadata_epoch in the new offset topic schema will be the
> largest
> > >> > > > metadata_epoch from all the MetadataResponse and FetchResponse
> > ever
> > >> > > > received by this consumer.
> > >> > > >
> > >> > > > We probably don't have to change the consumer API for
> > >> > > > commitSync(Map). If user
> calls
> > >> > > > commitSync(...) to commit offset 10 for a given partition, fo

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ismael Juma]

Hi Rajini,

Quick question (sorry if this was already discussed). How were the
following chosen?

Name: password.encoder.keyfactory.algorithm  Type: String Default:
PBKDF2WithHmacSHA512 if available, otherwise PBKDF2WithHmacSHA1 (e.g. Java7)
Name: password.encoder.cipher.algorithm  Type: String  Default:
AES/CBC/PKCS5Padding
Name: password.encoder.key.length Type: Integer  Default: 128
Name: password.encoder.iterations  Type: Integer Default: 2048

Also, was a AES/GCM variant considered as the default cipher algorithm?

Ismael

On Mon, Nov 20, 2017 at 1:57 PM, Rajini Sivaram 
wrote:

> Hi all,
>
> I have submitted KIP-226 to enable dynamic reconfiguration of brokers
> without restart:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 226+-+Dynamic+Broker+Configuration
>
> The KIP proposes to extend the current dynamic replication quota
> configuration for brokers to support dynamic reconfiguration of a limited
> set of configuration options that are typically updated during the lifetime
> of a broker.
>
> Feedback and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ismael Juma]

Thanks Rajini. Sounds good.

Ismael

On Wed, Jan 10, 2018 at 11:41 AM, Rajini Sivaram 
wrote:

> Hi Ismael,
>
> I have updated the KIP to use AES-256 if available and AES-128 otherwise
> for password encryption. Looking at GCM, it looks like GCM is typically
> used with a variable initialization vector, while we are using a random,
> but constant IV per-password. Also, AES/GCM is not supported by Java7.
> Since the authentication and performance benefits of GCM are not required
> for this scenario, I am thinking I will leave the default as CBC, but make
> sure we test GCM as well so that users have the choice.
>
> On Wed, Jan 10, 2018 at 1:01 AM, Colin McCabe  wrote:
>
> > Thanks, Rajini.  That makes sense.
> >
> > regards,
> > Colin
> >
> > On Tue, Jan 9, 2018, at 14:38, Rajini Sivaram wrote:
> > > Hi Colin,
> > >
> > > Thank you for reviewing.
> > >
> > > Yes, validation is done on the broker, not the client.
> > >
> > > All configs from ZooKeeper are processed and any config that could not
> be
> > > applied are logged as warnings. This includes any configs that are not
> > > dynamic in the broker version or any configs that are not supported in
> > the
> > > broker version. If you downgrade to a version that is older than this
> KIP
> > > (1.0 for example), then you don't get any warnings however.
> > >
> > >
> > > On Tue, Jan 9, 2018 at 9:38 PM, Colin McCabe 
> wrote:
> > >
> > > > On Mon, Dec 18, 2017, at 13:40, Jason Gustafson wrote:
> > > > > Hi Rajini,
> > > > >
> > > > > Looking good. Just a few questions.
> > > > >
> > > > > 1. (Related to Jay's comment) Is the validate() method on
> > Reconfigurable
> > > > > necessary? I would have thought we'd validate using the ConfigDef.
> > Do you
> > > > > have a use case in mind in which the reconfigurable component only
> > > > permits
> > > > > certain reconfigurations?
> > > >
> > > > Hi,
> > > >
> > > > Sorry if this is a dumb question, but when we talk about validating
> on
> > the
> > > > ConfigDef, we're talking about validating on the server side, right?
> > The
> > > > software on the client side might be older or newer than the software
> > on
> > > > the broker side, so it seems inadvisable to do the validation there.
> > > >
> > > > Also, after a software downgrade, when the broker is restarted, it
> > might
> > > > find that there is a configuration key that is stored in ZK that is
> not
> > > > dynamic in its (older) software version.  It seems like, with the
> > current
> > > > proposal, the broker will use the value found in the local
> > configuration
> > > > (config file) rather than the new ZK version.  Should the broker
> print
> > out
> > > > a WARN message in that scenario?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > 2. Should Reconfigurable extend Configurable or is the initial
> > > > > configuration also done through reconfigure()? I ask because not
> all
> > > > > plugins interfaces currently extend Configurable (e.g.
> > > > > KafkaPrincipalBuilder).
> > > > > 3. You mentioned a couple changes to DescribeConfigsOptions and
> > > > > DescribeConfigsResult. Perhaps we should list the changes
> > explicitly? One
> > > > > not totally obvious case is what the synonyms() getter would return
> > if
> > > > the
> > > > > option is not specified (i.e. should it raise an exception or
> return
> > an
> > > > > empty list?).
> > > > > 4. Config entries in the DescribeConfigs response have an
> is_default
> > > > flag.
> > > > > Could that be replaced with the more general config_source?
> > > > > 5. Bit of an internal question, but how do you handle config
> > > > dependencies?
> > > > > For example, suppose I want to add a listener and configure its
> > principal
> > > > > builder at once. You'd have to validate the principal builder
> config
> > in
> > > > the
> > > > > context of the listener config, so I guess the order of the entries
> > in
> > > > > AlterConfigs is significant?
> > > > > 6. KIP-48 (delegation tokens) gives us a master secret which is
> > shared by
> > > > > all brokers. Do you think we would make this dynamically
> > configurable?
> > > > > Alternatively, it might be possible to use it to encrypt the other
> > > > > passwords we store in zookeeper.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Dec 18, 2017 at 10:16 AM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jay,
> > > > > >
> > > > > > Thank you for reviewing the KIP.
> > > > > >
> > > > > > 1) Yes, makes sense. I will update the PR. There are some config
> > > > updates
> > > > > > that may be allowed depending on the context (e.g. some security
> > > > configs
> > > > > > can be updated for new listeners, but not existing listeners).
> > Perhaps
> > > > it
> > > > > > is ok to mark them dynamic in the documentation. AdminClient
> would
> > give
> > > > > > appropriate error messages if the update is not allowed.
> > > > > > 2) Internally, in the implementat

Re: [DISCUSS] February Release Plan

[Ismael Juma]

Thanks for volunteering Damian!

On Fri, Jan 12, 2018 at 11:04 AM, Damian Guy  wrote:

> Hi all,
>
> I would like to volunteer to be the release manager
> for our next time-based feature release (v1.1.0). See
> https://cwiki.apache.org/
> confluence/display/KAFKA/Time+Based+Release+Plan if you missed
> previous communication on time-based releases or need a reminder.
>
> I put together a draft release plan
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
> with
> February 2018 as the release month
> (as previously agreed) and a list of KIPs that have already been voted:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
>
> Some important (and fast approaching) dates:
> KIP Freeze: January 23, 2018
> Feature Freeze: January 30, 2018
>
> Thanks,
> Damian
>

Re: [ANNOUNCE] New committer: Matthias J. Sax

[Ismael Juma]

Congratulations Matthias!

On 12 Jan 2018 10:59 pm, "Guozhang Wang"  wrote:

> Hello everyone,
>
> The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our
> newest Kafka committer.
>
> Matthias has made tremendous contributions to Kafka Streams API since early
> 2016. His footprint has been all over the places in Streams: in the past
> two years he has been the main driver on improving the join semantics
> inside Streams DSL, summarizing all their shortcomings and bridging the
> gaps; he has also been largely working on the exactly-once semantics of
> Streams by leveraging on the transaction messaging feature in 0.11.0. In
> addition, Matthias have been very active in community activity that goes
> beyond mailing list: he's getting the close to 1000 up votes and 100
> helpful flags on SO for answering almost all questions about Kafka Streams.
>
> Thank you for your contribution and welcome to Apache Kafka, Matthias!
>
>
>
> Guozhang, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Ismael Juma]

Congratulations Rajini!

On 17 Jan 2018 10:49 am, "Gwen Shapira"  wrote:

Dear Kafka Developers, Users and Fans,

Rajini Sivaram became a committer in April 2017.  Since then, she remained
active in the community and contributed major patches, reviews and KIP
discussions. I am glad to announce that Rajini is now a member of the
Apache Kafka PMC.

Congratulations, Rajini and looking forward to your future contributions.

Gwen, on behalf of Apache Kafka PMC

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

[Ismael Juma]

Thanks for the KIP, +1 (binding).

On Tue, Dec 19, 2017 at 7:28 PM, Colin McCabe  wrote:

> Hi all,
>
> I'd like to start the vote on KIP-227: Incremental Fetch Requests.
>
> The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 227%3A+Introduce+Incremental+FetchRequests+to+Increase+
> Partition+Scalability
>
> and discussion thread earlier: https://www.mail-archive.com/
> dev@kafka.apache.org/msg83115.html
>
> thanks,
> Colin
>

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

[Ismael Juma]

Colin,

You get a cumulative count for rates since we added
https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics

Ismael

On Tue, Jan 23, 2018 at 4:21 PM, Colin McCabe  wrote:

> On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote:
> > Hi, Collin,
> >
> > Thanks for the updated KIP. +1. Just a minor comment. It seems that it's
> > better for TotalIncrementalFetchSessionsEvicted to be a rate, instead of
> > just an ever-growing count.
>
> Thanks.  Perhaps we can add the rate in addition to the total eviction
> count?
>
> best,
> Colin
>
> >
> > Jun
> >
> > On Mon, Jan 22, 2018 at 4:35 PM, Jason Gustafson 
> wrote:
> >
> > > >
> > > > What if we want to have fetch sessions for non-incremental fetches
> in the
> > > > future, though?  Also, we don't expect this configuration to be
> changed
> > > > often, so it doesn't really need to be short.
> > >
> > >
> > > Hmm.. But in that case, I'm not sure we'd need to distinguish the two
> > > cases. If the non-incremental sessions are occupying space
> proportional to
> > > the fetched partitions, using the same config for both would be
> reasonable.
> > > If they are not (which is more likely), we probably wouldn't need a
> config
> > > at all. Given that, I'd probably still opt for the more concise name.
> It's
> > > not a blocker for me though.
> > >
> > > +1 on the KIP.
> > >
> > > -Jason
> > >
> > > On Mon, Jan 22, 2018 at 3:56 PM, Colin McCabe 
> wrote:
> > >
> > > > On Mon, Jan 22, 2018, at 15:42, Jason Gustafson wrote:
> > > > > Hi Colin,
> > > > >
> > > > > This is looking good to me. A few comments:
> > > > >
> > > > > 1. The fetch type seems unnecessary in the request and response
> schemas
> > > > > since it can be inferred by the sessionId/epoch.
> > > >
> > > > Hi Jason,
> > > >
> > > > Fair enough... if we need it later, we can always bump the RPC
> version.
> > > >
> > > > > 2. I agree with Jun that a separate array for partitions to remove
> > > would
> > > > be
> > > > > more intuitive.
> > > >
> > > > OK.  I'll switch it to using a separate array.
> > > >
> > > > > 3. I'm not super thrilled with the cache configuration since it
> seems
> > > to
> > > > > tie us a bit too closely to the implementation. You've mostly
> convinced
> > > > me
> > > > > on the need for the slots config, but I wonder if we can at least
> do
> > > > > without "min.incremental.fetch.session.eviction.ms"? For one, I
> think
> > > > the
> > > > > broker should reserve the right to evict sessions at will. We
> shouldn't
> > > > be
> > > > > stuck maintaining a small session at the expense of a much larger
> one
> > > > just
> > > > > to enforce this timeout. Internally, I think having some cache
> > > stickiness
> > > > > to avoid thrashing makes sense, but I think static values are
> likely to
> > > > be
> > > > > good enough and that lets us retain some flexibility to change the
> > > > behavior
> > > > > in the future.
> > > >
> > > > OK.
> > > >
> > > > > 4. I think the word "incremental" is redundant in the config names.
> > > Maybe
> > > > > it could just be "max.fetch.session.cache.slots" for example?
> > > >
> > > > What if we want to have fetch sessions for non-incremental fetches
> in the
> > > > future, though?  Also, we don't expect this configuration to be
> changed
> > > > often, so it doesn't really need to be short.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Jan 20, 2018 at 12:54 PM, Colin McCabe  >
> > > > wrote:
> > > > >
> > > > > > On Fri, Jan 19, 2018, at 15:02, Jun Rao wrote:
> > > > > > > Hi, Colin,
> > > > > > >
> > > > > > > Thanks for the KIP. Looks good to me overall. Just a couple of
> more
> > > > > > > comments.
> > > > > > >
> > > > > > > 1. As I mentioned earlier, it might be useful to add some
> metrics
> > > for
> > > > > > > monitoring the usage of the session cache. For example, it
> would be
> > > > > > useful
> > > > > > > to know how many slots are being used (or unused), # of total
> > > > partitions
> > > > > > in
> > > > > > > the cached slots (to understand space), the eviction rate (to
> see
> > > if
> > > > > > there
> > > > > > > is any churn), etc.
> > > > > >
> > > > > > Thanks, Jun.  Sorry-- I meant to address this earlier, but I
> forgot
> > > > about
> > > > > > it.  I just added some proposed metrics to the KIP wiki.
> > > > > >
> > > > > > >
> > > > > > > 2. Using max_bytes to 0 represent the removal of a partition
> seems
> > > > > > > unintuitive. Perhaps it's better to either add a flag per
> partition
> > > > or
> > > > > > add
> > > > > > > a removed partition list.
> > > > > >
> > > > > > Perhaps if we use max_bytes -1 to represent removal, it will be
> more
> > > > > > intuitive?  After all, -1 bytes is clearly not a valid amount of
> > > bytes
> > > > to
> > > > > > fetch.  Or should be have a separate array of removed
> > > TopicPartitions?
> > > > > >
> > > > > > On a

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

[Ismael Juma]

Hi Rajini,

I think the proposal makes sense. One suggestion: can we just allow the
config to be passed? That is, leave out the properties config for now.

On Tue, Jan 23, 2018 at 3:01 PM, Rajini Sivaram 
wrote:

> Since we are running out of time to get the whole ConfigCommand converted
> to using the new AdminClient for 1.1.0 (KIP-248), we need a way to enable
> ConfigCommand to handle broker config updates (implemented by KIP-226). As
> a simple first step, it would make sense to use the existing ConfigCommand
> tool to perform broker config updates enabled by this KIP. Since config
> validation and password encryption are performed by the broker, this will
> be easier to do with the new AdminClient. To do this, we need to add
> command line options for new admin client to kafka-configs.sh. Dynamic
> broker config updates alone will be done under KIP-226 using the new admin
> client to make this feature usable.. The new command line options
> (consistent with KIP-248) that will be added to ConfigCommand will be:
>
>- --bootstrap-server *host:port*
>- --adminclient.config *config-file*
>- --adminclient.properties *k1=v1,k2=v2*
>
> If anyone has any concerns about these options being added to
> kafka-configs.sh, please let me know. Otherwise, I will update KIP-226 and
> add the options to one of the KIP-226 PRs.
>
> Thank you,
>
> Rajini
>
> On Wed, Jan 10, 2018 at 5:14 AM, Ismael Juma  wrote:
>
> > Thanks Rajini. Sounds good.
> >
> > Ismael
> >
> > On Wed, Jan 10, 2018 at 11:41 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > I have updated the KIP to use AES-256 if available and AES-128
> otherwise
> > > for password encryption. Looking at GCM, it looks like GCM is typically
> > > used with a variable initialization vector, while we are using a
> random,
> > > but constant IV per-password. Also, AES/GCM is not supported by Java7.
> > > Since the authentication and performance benefits of GCM are not
> required
> > > for this scenario, I am thinking I will leave the default as CBC, but
> > make
> > > sure we test GCM as well so that users have the choice.
> > >
> > > On Wed, Jan 10, 2018 at 1:01 AM, Colin McCabe 
> > wrote:
> > >
> > > > Thanks, Rajini.  That makes sense.
> > > >
> > > > regards,
> > > > Colin
> > > >
> > > > On Tue, Jan 9, 2018, at 14:38, Rajini Sivaram wrote:
> > > > > Hi Colin,
> > > > >
> > > > > Thank you for reviewing.
> > > > >
> > > > > Yes, validation is done on the broker, not the client.
> > > > >
> > > > > All configs from ZooKeeper are processed and any config that could
> > not
> > > be
> > > > > applied are logged as warnings. This includes any configs that are
> > not
> > > > > dynamic in the broker version or any configs that are not supported
> > in
> > > > the
> > > > > broker version. If you downgrade to a version that is older than
> this
> > > KIP
> > > > > (1.0 for example), then you don't get any warnings however.
> > > > >
> > > > >
> > > > > On Tue, Jan 9, 2018 at 9:38 PM, Colin McCabe 
> > > wrote:
> > > > >
> > > > > > On Mon, Dec 18, 2017, at 13:40, Jason Gustafson wrote:
> > > > > > > Hi Rajini,
> > > > > > >
> > > > > > > Looking good. Just a few questions.
> > > > > > >
> > > > > > > 1. (Related to Jay's comment) Is the validate() method on
> > > > Reconfigurable
> > > > > > > necessary? I would have thought we'd validate using the
> > ConfigDef.
> > > > Do you
> > > > > > > have a use case in mind in which the reconfigurable component
> > only
> > > > > > permits
> > > > > > > certain reconfigurations?
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Sorry if this is a dumb question, but when we talk about
> validating
> > > on
> > > > the
> > > > > > ConfigDef, we're talking about validating on the server side,
> > right?
> > > > The
> > > > > > software on the client side might be older or newer than the
> > software
> > > > on
> > > > > > the broker 

Re: [VOTE] KIP-86: Configurable SASL callback handlers

[Ismael Juma]

Thanks for the KIP, Rajini. This is a useful improvement, so +1 (binding)
from me.

I really don't like how the Java Security classes work, so I would have
preferred to avoid emulating them, but the KIP is consistent with previous
related KIPs and that's the direction we chose previously. Also, I think I
might have tried to reduce the number of configs from 2 to 1 (in broker and
client) by relying more on Java, but I don't have a concrete proposal and
it would result in a larger API surface area.

Ismael

On Thu, Apr 6, 2017 at 2:53 AM, Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start the voting process for KIP-86:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 86%3A+Configurable+SASL+callback+handlers
>
> The KIP makes callback handlers for SASL configurable to make it simpler to
> integrate with custom authentication database or custom authentication
> servers. This is particularly useful for SASL/PLAIN where the
> implementation in Kafka based on credentials stored in jaas.conf is not
> suitable for production use. It is also useful for SCRAM in environments
> where ZooKeeper is not secure.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

[Ismael Juma]

Agreed, Jun.

Ismael

On Wed, Jan 24, 2018 at 4:08 PM, Jun Rao  wrote:

> Since this is a server side metric, it's probably better to use Yammer Rate
> (which has count) for consistency.
>
> Thanks,
>
> Jun
>
> On Tue, Jan 23, 2018 at 10:17 PM, Colin McCabe  wrote:
>
> > On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote:
> > > Colin,
> > >
> > > You get a cumulative count for rates since we added
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics>
> >
> > Oh, good point.
> >
> > C.
> >
> >
> > > Ismael
> > >
> > > On Tue, Jan 23, 2018 at 4:21 PM, Colin McCabe
> > >  wrote:>
> > > > On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote:
> > > > > Hi, Collin,
> > > > >
> > > > > Thanks for the updated KIP. +1. Just a minor comment. It seems
> > > > > that it's> > > better for TotalIncrementalFetchSessionsEvicted to
> > be a rate,
> > > > > instead of> > > just an ever-growing count.
> > > >
> > > > Thanks.  Perhaps we can add the rate in addition to the total
> > > > eviction> > count?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 22, 2018 at 4:35 PM, Jason Gustafson
> > > > > > > wrote:
> > > > >
> > > > > > >
> > > > > > > What if we want to have fetch sessions for non-incremental
> > > > > > > fetches> > in the
> > > > > > > future, though?  Also, we don't expect this configuration to
> > > > > > > be> > changed
> > > > > > > often, so it doesn't really need to be short.
> > > > > >
> > > > > >
> > > > > > Hmm.. But in that case, I'm not sure we'd need to distinguish
> > > > > > the two> > > > cases. If the non-incremental sessions are
> > occupying space
> > > > proportional to
> > > > > > the fetched partitions, using the same config for both would be>
> >
> > reasonable.
> > > > > > If they are not (which is more likely), we probably wouldn't
> > > > > > need a> > config
> > > > > > at all. Given that, I'd probably still opt for the more concise
> > > > > > name.> > It's
> > > > > > not a blocker for me though.
> > > > > >
> > > > > > +1 on the KIP.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Mon, Jan 22, 2018 at 3:56 PM, Colin McCabe
> > > > > > > > wrote:
> > > > > >
> > > > > > > On Mon, Jan 22, 2018, at 15:42, Jason Gustafson wrote:
> > > > > > > > Hi Colin,
> > > > > > > >
> > > > > > > > This is looking good to me. A few comments:
> > > > > > > >
> > > > > > > > 1. The fetch type seems unnecessary in the request and
> > > > > > > >response> > schemas
> > > > > > > > since it can be inferred by the sessionId/epoch.
> > > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Fair enough... if we need it later, we can always bump the RPC>
> > > version.
> > > > > > >
> > > > > > > > 2. I agree with Jun that a separate array for partitions to
> > > > > > > >remove> > > > would
> > > > > > > be
> > > > > > > > more intuitive.
> > > > > > >
> > > > > > > OK.  I'll switch it to using a separate array.
> > > > > > >
> > > > > > > > 3. I'm not super thrilled with the cache configuration since
> > > > > > > >it> > seems
> > > > > > to
> > > > > > > > tie us a bit too closely to the implementation. You've
> > > > > > > > mostly> > convinced
> > > > > > > me
> > > > > > > > on the need for the slots co

Re: Excessive Memory Usage with Compression enabled and possible resolutions

[Ismael Juma]

Thanks for the report. I haven't looked at the code, but it seems like we
would want to do both 1 and 2. Can you please file a JIRA with 1.1.0 as the
target version?

Ismael

On 30 Jan 2018 5:46 pm, "Kyle Tinker"  wrote:

> I'm using Kafka 1.0.0 and the Java producer.
>
> I've noticed high memory usage in the producer when enabling compression
> (gzip or lz4).  I don't observe the behavior with compression off, but with
> it on I'll run out of heap (2GB).  Using a Java profiler, I see the data is
> in the KafkaLZ4BlockOutputStream (or related class for gzip).   I see that
> MemoryRecordsBuilder:closeForRecordAppends() is trying to deal with this,
> but is not successful.  I'm most likely network bottlenecked, so I expect
> the producer buffers to be full while the job is running and potentially a
> lot of unacknowledged records.
>
> I've tried using the default buffer.memory with 20 producers (across 20
> threads) and sending data as quickly as I can.  I've also tried 1MB of
> buffer.memory, which seemed to reduce memory consumption but I could still
> run OOM in certain cases.  I have max.in.flight.requests.per.connection
> set to 1.  In short, I should only have ~20 MB (20* 1MB) of data in
> buffers, but I can easily exhaust 2000 MB used by Kafka.
>
> In looking at the code more, it looks like the KafkaLZ4BlockOutputStream
> doesn't clear the compressedBuffer or buffer when close() is called.  In my
> heap dump, both of those are ~65k size each, meaning that each batch is
> taking up ~148k of space, of which 131k is buffers.
> (buffer.memory=1,000,000 and messages are 1k each until the batch fills).
>
> Kafka tries to manage memory usage by calling 
> MemoryRecordsBuilder:closeForRecordAppends(),
> which as documented as "Release resources required for record appends (e.g.
> compression buffers)".  However, this method doesn't actually clear those
> buffers because KafkaLZ4BlockOutputStream.close() only writes the block
> and end mark and closes the output stream.  It doesn't actually clear the
> buffer and compressedBuffer in KafkaLZ4BlockOutputStream.  Those stay
> allocated in RAM until the block is acknowledged by the broker, processed
> in Sender:handleProduceResponse(), and the batch is deallocated.  This
> memory usage therefore increases, possibly without bound.  In my test
> program, the program died with approximately 345 unprocessed batches per
> producer (20 producers), despite having max.in.flight.requests.per.
> connection=1.
>
> There are a few possible optimizations I can think of:
> 1) We could declare KafkaLZ4BlockOutputStream.buffer and compressedBuffer
> as non-final and null them in the close() method
> 2) We could declare the MemoryRecordsBuilder.appendStream non-final and
> null it in the closeForRecordAppends() method
> 3) We could have the ProducerBatch discard the recordsBuilder in
> closeForRecordAppends(), however, this is likely a bad idea because the
> recordsBuilder contains significant metadata that is likely needed after
> the stream is closed.  It is also final.
> 4) We could try to limit the number of non-acknowledged batches in
> flight.  This would bound the maximum memory usage but may negatively
> impact performance.
>
> Fix #1 would only improve the LZ4 algorithm, and not any other algorithms.
> Fix #2 would improve all algorithms, compression and otherwise.  Of the 3
> proposed here, it seems the best.  This would also involve having to check
> appendStreamIsClosed in every usage of appendStream within
> MemoryRecordsBuilder to avoid NPE's.
>
> Are there any thoughts or suggestions on how to proceed?
>
> If requested I can provide standalone testcase code demonstrating this
> problem.
>
> Thanks,
> -Kyle
>
>
>
>
>
>
> This message is intended exclusively for the individual or entity to which
> it is addressed. This communication may contain information that is
> proprietary, privileged, confidential or otherwise legally exempt from
> disclosure. If you are not the named addressee, or have been inadvertently
> and erroneously referenced in the address line, you are not authorized to
> read, print, retain, copy or disseminate this message or any part of it. If
> you have received this message in error, please notify the sender
> immediately by e-mail and delete all copies of the message. (ID m031214)
>

Re: Excessive Memory Usage with Compression enabled and possible resolutions

[Ismael Juma]

Hi Kyle,

Are you interested in submitting a pull request?

Ismael

On Wed, Jan 31, 2018 at 3:00 PM, Kyle Tinker 
wrote:

> Ismael,
>
> I have filed https://issues.apache.org/jira/browse/KAFKA-6512 for this
> issue.  I could not find a target version field.  Let me know if you need
> any additional information.  I'm new to this project so hopefully the
> format is what you were looking for.
>
> - Kyle
>
> -Original Message-
> From: Ismael Juma [mailto:isma...@gmail.com]
> Sent: Tuesday, January 30, 2018 9:01 PM
> To: dev 
> Subject: Re: Excessive Memory Usage with Compression enabled and possible
> resolutions
>
> Thanks for the report. I haven't looked at the code, but it seems like we
> would want to do both 1 and 2. Can you please file a JIRA with 1.1.0 as the
> target version?
>
> Ismael
>
> On 30 Jan 2018 5:46 pm, "Kyle Tinker" 
> wrote:
>
> > I'm using Kafka 1.0.0 and the Java producer.
> >
> > I've noticed high memory usage in the producer when enabling
> > compression (gzip or lz4).  I don't observe the behavior with
> > compression off, but with it on I'll run out of heap (2GB).  Using a
> Java profiler, I see the data is
> > in the KafkaLZ4BlockOutputStream (or related class for gzip).   I see
> that
> > MemoryRecordsBuilder:closeForRecordAppends() is trying to deal with
> > this, but is not successful.  I'm most likely network bottlenecked, so
> > I expect the producer buffers to be full while the job is running and
> > potentially a lot of unacknowledged records.
> >
> > I've tried using the default buffer.memory with 20 producers (across
> > 20
> > threads) and sending data as quickly as I can.  I've also tried 1MB of
> > buffer.memory, which seemed to reduce memory consumption but I could
> > still run OOM in certain cases.  I have
> > max.in.flight.requests.per.connection
> > set to 1.  In short, I should only have ~20 MB (20* 1MB) of data in
> > buffers, but I can easily exhaust 2000 MB used by Kafka.
> >
> > In looking at the code more, it looks like the
> > KafkaLZ4BlockOutputStream doesn't clear the compressedBuffer or buffer
> > when close() is called.  In my heap dump, both of those are ~65k size
> > each, meaning that each batch is taking up ~148k of space, of which 131k
> is buffers.
> > (buffer.memory=1,000,000 and messages are 1k each until the batch fills).
> >
> > Kafka tries to manage memory usage by calling
> > MemoryRecordsBuilder:closeForRecordAppends(),
> > which as documented as "Release resources required for record appends
> (e.g.
> > compression buffers)".  However, this method doesn't actually clear
> > those buffers because KafkaLZ4BlockOutputStream.close() only writes
> > the block and end mark and closes the output stream.  It doesn't
> > actually clear the buffer and compressedBuffer in
> > KafkaLZ4BlockOutputStream.  Those stay allocated in RAM until the
> > block is acknowledged by the broker, processed in
> > Sender:handleProduceResponse(), and the batch is deallocated.  This
> > memory usage therefore increases, possibly without bound.  In my test
> > program, the program died with approximately 345 unprocessed batches per
> producer (20 producers), despite having max.in.flight.requests.per.
> > connection=1.
> >
> > There are a few possible optimizations I can think of:
> > 1) We could declare KafkaLZ4BlockOutputStream.buffer and
> > compressedBuffer as non-final and null them in the close() method
> > 2) We could declare the MemoryRecordsBuilder.appendStream non-final
> > and null it in the closeForRecordAppends() method
> > 3) We could have the ProducerBatch discard the recordsBuilder in
> > closeForRecordAppends(), however, this is likely a bad idea because
> > the recordsBuilder contains significant metadata that is likely needed
> > after the stream is closed.  It is also final.
> > 4) We could try to limit the number of non-acknowledged batches in
> > flight.  This would bound the maximum memory usage but may negatively
> > impact performance.
> >
> > Fix #1 would only improve the LZ4 algorithm, and not any other
> algorithms.
> > Fix #2 would improve all algorithms, compression and otherwise.  Of
> > the 3 proposed here, it seems the best.  This would also involve
> > having to check appendStreamIsClosed in every usage of appendStream
> > within MemoryRecordsBuilder to avoid NPE's.
> >
> > Are there any thoughts or suggestions on how to proceed?
> >
> > If requested I can provide standalone testcase cod

Re: 1.1 release progress

[Ismael Juma]

Hi Becket,

Thanks for filing that. Are you working on a fix?

Ismael

On Thu, Feb 15, 2018 at 2:51 PM, Becket Qin  wrote:

> Hi Damian,
>
> I just created another ticket KAFKA-6568, which I believe should also be a
> blocker unless people disagree.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Feb 14, 2018 at 8:52 AM, Damian Guy  wrote:
>
> > Hi All,
> >
> > The first 1.1 RC is due to be cut, however we currently have 2 blockers
> > outstanding:
> > https://issues.apache.org/jira/browse/KAFKA-6517 (which i suspect will
> be
> > merged shortly)
> > and
> > https://issues.apache.org/jira/browse/KAFKA-6549
> >
> > Once we have finished these issues I can create the first RC. Hopefully
> > before the end of the week.
> >
> > Regards,
> > Damian
> >
> >
> > On Mon, 5 Feb 2018 at 18:04 Damian Guy  wrote:
> >
> > > Hi all,
> > >
> > > We are just over a week away from code freeze. We currently have 44
> > issues
> > > in progress and 3 open blockers:
> > > https://issues.apache.org/jira/projects/KAFKA/versions/12339769
> > >
> > > If you have something that is in progress that is not going to make it
> > > into 1.1 can you please move it to a future release.
> > >
> > > Thanks,
> > > Damian
> > >
> >
>

Re: 1.1 release progress

[Ismael Juma]

Sounds good. Thanks Becket.

Ismael

On 15 Feb 2018 5:30 pm, "Becket Qin"  wrote:

> Hi Ismael,
>
> Yes, I am working on the fix. Will submit patch today.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Feb 15, 2018 at 2:53 PM, Ismael Juma  wrote:
>
> > Hi Becket,
> >
> > Thanks for filing that. Are you working on a fix?
> >
> > Ismael
> >
> > On Thu, Feb 15, 2018 at 2:51 PM, Becket Qin 
> wrote:
> >
> > > Hi Damian,
> > >
> > > I just created another ticket KAFKA-6568, which I believe should also
> be
> > a
> > > blocker unless people disagree.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Feb 14, 2018 at 8:52 AM, Damian Guy 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > The first 1.1 RC is due to be cut, however we currently have 2
> blockers
> > > > outstanding:
> > > > https://issues.apache.org/jira/browse/KAFKA-6517 (which i suspect
> will
> > > be
> > > > merged shortly)
> > > > and
> > > > https://issues.apache.org/jira/browse/KAFKA-6549
> > > >
> > > > Once we have finished these issues I can create the first RC.
> Hopefully
> > > > before the end of the week.
> > > >
> > > > Regards,
> > > > Damian
> > > >
> > > >
> > > > On Mon, 5 Feb 2018 at 18:04 Damian Guy  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are just over a week away from code freeze. We currently have 44
> > > > issues
> > > > > in progress and 3 open blockers:
> > > > > https://issues.apache.org/jira/projects/KAFKA/versions/12339769
> > > > >
> > > > > If you have something that is in progress that is not going to make
> > it
> > > > > into 1.1 can you please move it to a future release.
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > >
> > >
> >
>

Re: Do the Jackson security vulnerabilities affect Kafka at all?

[Ismael Juma]

Hi Jeff,

Have you checked trunk and 1.1? They should be using the latest version.

Ismael

On Tue, Feb 20, 2018 at 10:38 PM, Jeff Widman  wrote:

> The Jackson JSON parser library had a couple of CVE's announced:
> 1. CVE-2017-7525
> 2. CVE 2017-15095
>
> Here's a skimmable summary:
> https://adamcaudill.com/2017/10/04/exploiting-jackson-rce-cve-2017-7525/
>
> Looking at the source, it appears Kafka uses an older version of Jackson
> which has the vulnerabilities.
>
> However, these vulnerabilities only happen when Jackson is used in specific
> ways. I'm not familiar enough with all the places that Kafka uses Jackson
> to understand whether Kafka is susceptible, and I come from a non-Java
> background so it's difficult for me to parse the Java source with 100%
> confidence that I understand what's happening.
>
> I know primarily Kafka uses JSON for inter-cluster communication through
> Zookeeper, so if an attacker could access Zookeeper could they update the
> znode payloads to exploit this? Additionally, I think there are some util
> scripts that (de)serialize JSON files, for example the
> partition-reassignment scripts...
>
> So do these CVE's apply to Kafka?
>
> If so, it seem the patch is fairly trivial of just upgrading to a newer
> version of Jackson...
> should this also be backported to the 1.0.1 release?
>
>
>
> --
>
> *Jeff Widman*
> jeffwidman.com  | 740-WIDMAN-J (943-6265)
> <><
>

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Ismael Juma]

Yes, let's revert the incompatible changes. There was no mention of
compatibility impact on the KIP and we should ensure that is the case for
1.1.0.

Ismael

On Thu, Feb 22, 2018 at 9:55 AM, Jason Gustafson  wrote:

> I know it's a been a while since this vote passed, but I think we need to
> reconsider the incompatible changes to the consumer reset tool.
> Specifically, we have removed the --execute option without deprecating it
> first, and we have changed the default behavior to execute rather than do a
> dry run. The latter in particular seems dangerous since users who were
> previously using the default behavior to view offsets will now suddenly
> find the offsets already committed. As far as I can tell, this change was
> done mostly for cosmetic reasons. Without a compelling reason, I think we
> should err on the side of maintaining compatibility. At a minimum, if we
> really want to break compatibility, we should wait for the next major
> release.
>
> Note that I have submitted a patch to revert this change here:
> https://github.com/apache/kafka/pull/4611.
>
> Thoughts?
>
> Thanks,
> Jason
>
>
>
> On Tue, Nov 14, 2017 at 3:26 AM, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Thanks to everyone for your feedback.
> >
> > KIP has been accepted and discussion is moved to PR.
> >
> > Cheers,
> > Jorge.
> >
> > El lun., 6 nov. 2017 a las 17:31, Rajini Sivaram (<
> rajinisiva...@gmail.com
> > >)
> > escribió:
> >
> > > +1 (binding)
> > > Thanks for the KIP,  Jorge.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Tue, Oct 31, 2017 at 9:58 AM, Damian Guy 
> > wrote:
> > >
> > > > Thanks for the KIP - +1 (binding)
> > > >
> > > > On Mon, 23 Oct 2017 at 18:39 Guozhang Wang 
> wrote:
> > > >
> > > > > Thanks Jorge for driving this KIP! +1 (binding).
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 16, 2017 at 2:11 PM, Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Fri, Oct 13, 2017 at 6:36 PM, Ted Yu 
> > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Fri, Oct 13, 2017 at 3:32 PM, Matthias J. Sax <
> > > > > matth...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 9/11/17 3:04 PM, Jorge Esteban Quilcate Otoya wrote:
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > It seems that there is no further concern with the KIP-171.
> > > > > > > > > At this point we would like to start the voting process.
> > > > > > > > >
> > > > > > > > > The KIP can be found here:
> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > 171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+
> > Application
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Ismael Juma]

Hi Guozhang,

To clarify my comment: any change with a backwards compatibility impact
should be mentioned in the "Compatibility, Deprecation, and Migration Plan"
section (in addition to the deprecation period and only happening in a
major release as you said).

Ismael

On Thu, Feb 22, 2018 at 11:10 AM, Guozhang Wang  wrote:

> Just to clarify, the KIP itself has mentioned about the change so the PR
> was not un-intentional:
>
> "
>
> 3. Keep execution parameters uniform between both tools: It will execute by
> default, and have a `dry-run` parameter just show the results. This will
> involve change current `ConsumerGroupCommand` to change execution options.
>
> "
>
> We were agreed that the proposed change is better than the current status,
> since may people not using "--execute" on consumer reset tool were actually
> surprised that nothing gets executed. What we were concerning as a
> hind-sight is that instead of doing such change in a minor release like
> 1.1, we should consider only doing that in the next major release as it
> breaks compatibility. In the past when we are going to remove / replace
> certain option we would first add a going-to-be-deprecated warning in the
> previous releases until it was finally removed. So Jason's suggestion is to
> do the same: we are not reverting this change forever, but trying to delay
> it after 1.1.
>
>
> Guozhang
>
>
> On Thu, Feb 22, 2018 at 10:56 AM, Colin McCabe  wrote:
>
> > Perhaps, if the user doesn't pass the --execute flag, the tool should
> > print a prompt like "would you like to perform this reset?" and wait for
> a
> > Y / N (or yes or no) input from the command-line.  Then, if the --execute
> > flag is passed, we skip this.  That seems 99% compatible, and also
> > accomplishes the goal of making the tool less confusing.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Feb 22, 2018, at 10:23, Ismael Juma wrote:
> > > Yes, let's revert the incompatible changes. There was no mention of
> > > compatibility impact on the KIP and we should ensure that is the case
> for
> > > 1.1.0.
> > >
> > > Ismael
> > >
> > > On Thu, Feb 22, 2018 at 9:55 AM, Jason Gustafson 
> > wrote:
> > >
> > > > I know it's a been a while since this vote passed, but I think we
> need
> > to
> > > > reconsider the incompatible changes to the consumer reset tool.
> > > > Specifically, we have removed the --execute option without
> deprecating
> > it
> > > > first, and we have changed the default behavior to execute rather
> than
> > do a
> > > > dry run. The latter in particular seems dangerous since users who
> were
> > > > previously using the default behavior to view offsets will now
> suddenly
> > > > find the offsets already committed. As far as I can tell, this change
> > was
> > > > done mostly for cosmetic reasons. Without a compelling reason, I
> think
> > we
> > > > should err on the side of maintaining compatibility. At a minimum, if
> > we
> > > > really want to break compatibility, we should wait for the next major
> > > > release.
> > > >
> > > > Note that I have submitted a patch to revert this change here:
> > > > https://github.com/apache/kafka/pull/4611.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Tue, Nov 14, 2017 at 3:26 AM, Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com> wrote:
> > > >
> > > > > Thanks to everyone for your feedback.
> > > > >
> > > > > KIP has been accepted and discussion is moved to PR.
> > > > >
> > > > > Cheers,
> > > > > Jorge.
> > > > >
> > > > > El lun., 6 nov. 2017 a las 17:31, Rajini Sivaram (<
> > > > rajinisiva...@gmail.com
> > > > > >)
> > > > > escribió:
> > > > >
> > > > > > +1 (binding)
> > > > > > Thanks for the KIP,  Jorge.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Tue, Oct 31, 2017 at 9:58 AM, Damian Guy <
> damian@gmail.com>
> > > > > wrote:
> > > > > >
> > > 

Re: [VOTE] 1.0.1 RC2

[Ismael Juma]

Thanks for running the release Ewen!

Ismael

On Fri, Mar 2, 2018 at 10:10 AM, Ewen Cheslack-Postava 
wrote:

> Thanks everyone for voting. This passes with 3 binding +1, 5 non-binding
> +1, and no dissenting votes.
>
> I'll work on getting the release finalized and send out an announcement
> when it is ready.
>
> -Ewen
>
> On Tue, Feb 27, 2018 at 11:18 PM, Jason Gustafson 
> wrote:
>
> > +1. Verified artifacts and ran the basic quickstart.
> >
> > -Jason
> >
> > On Mon, Feb 26, 2018 at 1:08 AM, Manikumar 
> > wrote:
> >
> > > +1 (non-binding)
> > > Built src and ran tests
> > > Ran core quick start
> > >
> > > On Sat, Feb 24, 2018 at 8:44 PM, Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding) ... I used the Scala 2.12 binaries and run my tests
> > with
> > > > producers / consumers.
> > > >
> > > > On Thu, Feb 22, 2018 at 1:06 AM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the third candidate for release of Apache Kafka 1.0.1.
> > > > >
> > > > > This is a bugfix release for the 1.0 branch that was first released
> > > with
> > > > > 1.0.0 about 3 months ago. We've fixed 49 issues since that release.
> > > Most
> > > > of
> > > > > these are non-critical, but in aggregate these fixes will have
> > > > significant
> > > > > impact. A few of the more significant fixes include:
> > > > >
> > > > > * KAFKA-6277: Make loadClass thread-safe for class loaders of
> Connect
> > > > > plugins
> > > > > * KAFKA-6185: Selector memory leak with high likelihood of OOM in
> > case
> > > of
> > > > > down conversion
> > > > > * KAFKA-6269: KTable state restore fails after rebalance
> > > > > * KAFKA-6190: GlobalKTable never finishes restoring when consuming
> > > > > transactional messages
> > > > > * KAFKA-6529: Stop file descriptor leak when client disconnects
> with
> > > > staged
> > > > > receives
> > > > > * KAFKA-6238: Issues with protocol version when applying a rolling
> > > > upgrade
> > > > > to 1.0.0
> > > > >
> > > > > Release notes for the 1.0.1 release:
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Saturday Feb 24, 9pm PT ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 1.0 branch) is the 1.0.1 tag:
> > > > > https://github.com/apache/kafka/tree/1.0.1-rc2
> > > > >
> > > > > * Documentation:
> > > > > http://kafka.apache.org/10/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > http://kafka.apache.org/10/protocol.html
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Ewen Cheslack-Postava
> > > > >
> > > >
> > >
> >
>