Kafka Performance Producer - Extension

[Srinivas, Kaushik (Nokia - IN/Bangalore)]

Hi All,

This is in regard to extend few features support for java kafka performance 
producer.

We have a kafka producer application written in java with maven build.
Attaching the documentation which gives detailed description of features 
supported by this application as of now.

Would like to know if this can be integrated into kafka performance producer 
java implementation which is present with limited features.
Kindly review the attached document, and help with feedback on how well this 
could be integrated.

These features would help any team load testing kafka brokers/schema registry 
components efficiently.

Best Regards,
-kaushik


KAFKA_PRODUCER_FEATURES_DOC.docx
Description: KAFKA_PRODUCER_FEATURES_DOC.docx

[jira] [Created] (KAFKA-7607) NetworkClientUtils.sendAndReceive can take a long time to return during shutdown

[Bob Barrett (JIRA)]

Bob Barrett created KAFKA-7607:
--

 Summary: NetworkClientUtils.sendAndReceive can take a long time to 
return during shutdown
 Key: KAFKA-7607
 URL: https://issues.apache.org/jira/browse/KAFKA-7607
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Bob Barrett


If a RequestSendThread is shut down while waiting on the underlying Selector to 
return from a select() call, the Selector will swallow the interrupt, wake up 
and return immediately. NetworkClientUtils.sendAndReceive will then potentially 
re-poll the client, forcing the thread shutdown to wait for the request to 
complete. We should check the thread interrupt status before re-polling the 
client to prevent this delay.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-380: Detect outdated control requests and bounced brokers using broker generation

[Patrick Huang]

Hi,

In this KIP, we are also going to add a new exception and a new error code 
"STALE_BROKER_EPOCH" in order to allow the broker to respond back the right 
error when it sees outdated broker epoch in the control requests. Since adding 
a new exception and error code is also considered as public interface change, I 
have updated the original KIP accordingly to include this change. Feel free to 
comment if there is any concern.

Thanks,
Zhanxiang (Patrick) Huang


From: Patrick Huang 
Sent: Tuesday, October 23, 2018 6:20
To: Jun Rao; dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-380: Detect outdated control requests and bounced 
brokers using broker generation

Agreed. I have updated the PR to add czxid in ControlledShutdownRequest 
(https://github.com/apache/kafka/pull/5821). Appreciated if you can take a look.

Btw, I also have the vote thread for this KIP: 
https://lists.apache.org/thread.html/3689d83db537d5aa86d10967dee7ee29578897fc123daae4f77a8605@%3Cdev.kafka.apache.org%3E

Best,
Zhanxiang (Patrick) Huang


From: Jun Rao 
Sent: Monday, October 22, 2018 21:31
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-380: Detect outdated control requests and bounced 
brokers using broker generation

Hi, Patrick,

Yes, that's the general sequence. After step 2, the shutting down broker
can give up the controlled shutdown process and proceed to shut down. When
it's restarted, it could still receive StopReplica requests from the
controller in reaction to the previous controlled shutdown requests. This
could lead the restarted broker to a bad state.

Thanks,

Jun


On Mon, Oct 22, 2018 at 4:32 PM, Patrick Huang  wrote:

> Hi Jun,
>
> That is a good point. I want to make it clear about the scenario you
> mentioned. Correct me if I am wrong. Here is the sequence of the event:
>
>1. Broker sends ControlledShutdown request 1 to controller
>2. Broker sends ControlledShutdown request 2 to controller due to
>reties
>3. Controller processes ControlledShutdown request 1
>4. Controller sends control requests to the broker
>5. Broker receives ControlledShutdown response 1 from controller
>6. Broker shuts down and restarts quickly
>7. Controller processes ControllerShutdown request 2
>8. Controller sends control requests to the broker
>
> It is possible that controller processes the broker change event between
> 6) and 7). In this case, controller already updates the cached czxid to the
> up-to-date ones so the bounced broker will not reject control requests in
> 8), which cause a correctness problem.
>
>
> Best,
> Zhanxiang (Patrick) Huang
>
> --
> *From:* Jun Rao 
> *Sent:* Monday, October 22, 2018 14:45
> *To:* dev
> *Subject:* Re: [DISCUSS] KIP-380: Detect outdated control requests and
> bounced brokers using broker generation
>
> Hi, Patrick,
>
> There is another thing that may be worth considering.
>
> 10. It will be useful to include the czxid also in the ControlledShutdown
> request. This way, if the broker has been restarted, the controller can
> ignore an old ControlledShutdown request(e.g., due to retries). This will
> prevent the restarted broker from incorrectly stopping replicas.
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 11, 2018 at 5:56 PM, Patrick Huang 
> wrote:
>
> > Hi Jun,
> >
> > Thanks a lot for the comments.
> >
> > 1. czxid is globally unique and monotonically increasing based on the
> > zookeeper doc.
> > References (from
> > https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html):
> > "Every change to the ZooKeeper state receives a stamp in the form of a
> > *zxid* (ZooKeeper Transaction Id). This exposes the total ordering of all
> > changes to ZooKeeper. Each change will have a unique zxid and if zxid1 is
> > smaller than zxid2 then zxid1 happened before zxid2."
> > "czxid: The zxid of the change that caused this znode to be created."
> >
> > 2. You are right. There will be only on broker change event fired in the
> > case I mentioned because we will not register the watcher before the
> read.
> >
> > 3. Let's say we want to initialize the states of broker set A and we want
> > the cluster to be aware of the absence of broker set B. The currently
> live
> > broker set in the cluster is C.
> >
> > From the design point of view, here are the rules for broker state
> > transition:
> > - Pass in broker ids of A for onBrokerStartup() and pass in broker
> ids
> > of B for onBrokerFailure().
> > - When processing onBrokerStartup(), we use the broker generation
> > controller read from zk to send requests to broker set A and use the
> > previously cached broker generation to send requests to (C-A).
> > - When processing onBrokerFailure(), we use the previously cached
> > broker generation to send requests to C.
> >
> > From the implementation point of view, here are the steps we need to
> > follow when processing BrokerChangeEvent:
> > -  Reads 

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[Peter M. Elias]

I know we have a lot of use cases for this type of functionality at my
enterprise deployment. I think it's helpful for maintaining reliability of
the cluster especially and identifying clients that are not properly tuned
and therefore applying excessive load to the brokers. Additionally, there
is a bit of a dark spot without something like as currently. For example,
if a client is not using a consumer group, there is no direct way to query
the state of the consumer without looking at raw network connections to
determine the extent of the traffic generated by that particular consumer.

While client instrumentation can certainly help with this currently, given
that Kafka is intended to be a shared service across a potentially very
large surface area of clients, central observation of client activity is in
my opinion an essential feature.

Peter

On Thu, Nov 8, 2018 at 12:13 PM radai  wrote:

> bump.
>
> I think the proposed API (Observer) is useful for any sort of
> multi-tenant environment for chargeback and reporting purposes.
>
> if no one wants to comment, can we initiate a vote?
> On Mon, Nov 5, 2018 at 6:31 PM Lincong Li  wrote:
> >
> > Hi everyone. Here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response
> >
> > is
> > my KIP. Any feedback is appreciated.
> >
> > Thanks,
> > Lincong Li
>

Jenkins build is back to normal : kafka-2.0-jdk8 #185

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7609) Add Protocol Generator for Kafka

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-7609:
--

 Summary: Add Protocol Generator for Kafka
 Key: KAFKA-7609
 URL: https://issues.apache.org/jira/browse/KAFKA-7609
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe


Rather than hand-writing the code for sending and receiving all the different 
versions of the Kafka RPC protocol, we should have a protocol generator which 
can generate this code from an interface definition language (IDL).

This will make it less labor-intensive and error-prone to add new message types 
and versions.  It will also make it easier to support these new RPC changes in 
clients such as librdkafka.

Eventually, we should be able to get rid of the Struct classes and serialize 
messages directly to byte buffers.  This will greatly reduce the garbage 
collection overhead of the network stack, especially when handling large 
messages.

Furthermore, having a formal definition for the Kafka protocol may eventually 
allow us to expose other transports such as REST without writing lots of glue 
code to translate back and forth.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[radai]

another downside to client instrumentation (beyond the number of
client codebases one would need to cover) is that in a large
environments you'll have a very long tail of applications using older
clients to upgrade - it would be a long and disruptive process (as
opposed to updating broker-side instrumentation)
On Thu, Nov 8, 2018 at 11:04 AM Peter M. Elias  wrote:
>
> I know we have a lot of use cases for this type of functionality at my
> enterprise deployment. I think it's helpful for maintaining reliability of
> the cluster especially and identifying clients that are not properly tuned
> and therefore applying excessive load to the brokers. Additionally, there
> is a bit of a dark spot without something like as currently. For example,
> if a client is not using a consumer group, there is no direct way to query
> the state of the consumer without looking at raw network connections to
> determine the extent of the traffic generated by that particular consumer.
>
> While client instrumentation can certainly help with this currently, given
> that Kafka is intended to be a shared service across a potentially very
> large surface area of clients, central observation of client activity is in
> my opinion an essential feature.
>
> Peter
>
> On Thu, Nov 8, 2018 at 12:13 PM radai  wrote:
>
> > bump.
> >
> > I think the proposed API (Observer) is useful for any sort of
> > multi-tenant environment for chargeback and reporting purposes.
> >
> > if no one wants to comment, can we initiate a vote?
> > On Mon, Nov 5, 2018 at 6:31 PM Lincong Li  wrote:
> > >
> > > Hi everyone. Here
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response
> > >
> > > is
> > > my KIP. Any feedback is appreciated.
> > >
> > > Thanks,
> > > Lincong Li
> >

[jira] [Created] (KAFKA-7606) Remove deprecated --zookeeper option from StreamsResetter

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-7606:
--

 Summary: Remove deprecated --zookeeper option from StreamsResetter
 Key: KAFKA-7606
 URL: https://issues.apache.org/jira/browse/KAFKA-7606
 Project: Kafka
  Issue Type: Task
  Components: streams, tools
Reporter: Matthias J. Sax
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[radai]

bump.

I think the proposed API (Observer) is useful for any sort of
multi-tenant environment for chargeback and reporting purposes.

if no one wants to comment, can we initiate a vote?
On Mon, Nov 5, 2018 at 6:31 PM Lincong Li  wrote:
>
> Hi everyone. Here
> 
> is
> my KIP. Any feedback is appreciated.
>
> Thanks,
> Lincong Li

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

+1 (binding)
Built from src and ran tests, verified the artifacts, signatures, Ran core
quick start.

Thanks,
Manikumar

On Thu, Nov 8, 2018 at 12:15 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
>> +1 (binding)
>>
>> Checked signatures, build and quickstart.
>>
>> Thank you for managing the release, Mani!
>>
>>
>> On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> wrote:
>> >
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the first candidate for release of Apache Kafka 2.0.1.
>> >
>> > This is a bug fix release closing 49 tickets:
>> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> >
>> > Release notes for the 2.0.1 release:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by  Tuesday, October 30, end of day
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > http://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/
>> >
>> > * Javadoc:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> >
>> > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> >
>> > * Documentation:
>> > http://kafka.apache.org/20/documentation.html
>> >
>> > * Protocol:
>> > http://kafka.apache.org/20/protocol.html
>> >
>> > * Successful Jenkins builds for the 2.0 branch:
>> > Unit/integration tests:
>> https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> >
>> > /**
>> >
>> > Thanks,
>> > Manikumar
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAOJcB38LDf1Fgkr8HMG2b2%2B1tA5usQ%3DeeRv0Vjcj63Tq2zoYZQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

[RESULTS] [VOTE] Release Kafka version 2.0.1

[Manikumar]

This vote passes with 12 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Jun Rao
* Gwen Shapira
* Rajini Sivaram

Committers:
* Harsha
* Ewen Cheslack-Postava
* Manikumar

Community:
* Magnus Edenhill
* Eno Thereska
* Vahid Hashemian
* Jakub Scholz
* Satish Duggana
* Mickael Maison

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://markmail.org/thread/3gy4a6fmjv2xuoou

Thank you all for the votes.
I'll continue with the release process and the release announcement will
follow in the next few days.

Manikumar

[VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Hey all,


thanks so much for all the inputs on KIP-345 so far. The original proposal has 
enhanced a lot with your help. To make sure the implementation go smoothly 
without back and forth, I would like to start a vote on the final design 
agreement now:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-

345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...


Let me know if you have any questions.


Best,

Boyang

Re: [VOTE] - KIP-213 Support non-key joining in KTable

[John Roesler]

Hey Adam,

I'm sorry, I haven't been plugged in to your KIP lately. I'll have to read
it over again. Would it be discouraging if I wound up responding to your
discussion thread?

Thanks,
-John

On Wed, Nov 7, 2018 at 3:25 PM Adam Bellemare 
wrote:

> Bumping this thread, as per convention - 1
>
> On Fri, Nov 2, 2018 at 8:22 AM Adam Bellemare 
> wrote:
>
> > As expected :) But still, thanks none-the-less!
> >
> > On Fri, Nov 2, 2018 at 3:36 AM Jan Filipiak 
> > wrote:
> >
> >> reminder
> >>
> >> On 30.10.2018 15:47, Adam Bellemare wrote:
> >> > Hi All
> >> >
> >> > I would like to call a vote on
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable
> >> .
> >> > This allows a Kafka Streams DSL user to perform KTable to KTable
> >> > foreign-key joins on their data. I have been using this in production
> >> for
> >> > some time and I have composed a PR that enables this. It is a fairly
> >> > extensive PR, but I believe it will add considerable value to the
> Kafka
> >> > Streams DSL.
> >> >
> >> > The PR can be found here:
> >> > https://github.com/apache/kafka/pull/5527
> >> >
> >> > See
> >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201810.mbox/browser
> >> > for previous discussion thread.
> >> >
> >> > I would also like to give a shout-out to Jan Filipiak who helped me
> out
> >> > greatly in this project, and who led the initial work into this
> problem.
> >> > Without Jan's help and insight I do not think this would have been
> >> possible
> >> > to get to this point.
> >> >
> >> > Adam
> >> >
> >>
> >
>

Re: [VOTE] KIP-386: Standardize on Min/Avg/Max metrics' default values

[Jun Rao]

Hi, Stanislav,

Thanks for the KIP. +1. I guess this only covers the Kafka metrics (not the
Yammer metrics). It would be useful to make this clear.

Jun

On Tue, Nov 6, 2018 at 1:00 AM, Stanislav Kozlovski 
wrote:

> Hey everybody,
>
> I'm starting a vote thread on KIP-386: Standardize on Min/Avg/Max metrics'
> default values
>  >
> In short, after the discussion thread
>  4b9014fbb50f663bf14e5aec67@%3Cdev.kafka.apache.org%3E>,
> we decided to have all min/avg/max metrics output `NaN` as a default value.
>
> --
> Best,
> Stanislav
>

Re: [VOTE] 2.0.1 RC0

[Ismael Juma]

Manikumar, looks like you have the necessary votes. :)

Ismael

On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > Checked signatures, build and quickstart.
> >
> > Thank you for managing the release, Mani!
> >
> >
> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
> > wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 2.0.1.
> > >
> > > This is a bug fix release closing 49 tickets:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > >
> > > Release notes for the 2.0.1 release:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by  Tuesday, October 30, end of day
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > >
> > > * Documentation:
> > > http://kafka.apache.org/20/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/20/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > >
> > > /**
> > >
> > > Thanks,
> > > Manikumar
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: [VOTE] - KIP-213 Support non-key joining in KTable

[Adam Bellemare]

Hi John 

By all means, reply how you wish! I am interested in ensuring that everything 
has been addressed.

Thanks

> On Nov 8, 2018, at 4:57 PM, John Roesler  wrote:
> 
> Hey Adam,
> 
> I'm sorry, I haven't been plugged in to your KIP lately. I'll have to read
> it over again. Would it be discouraging if I wound up responding to your
> discussion thread?
> 
> Thanks,
> -John
> 
> On Wed, Nov 7, 2018 at 3:25 PM Adam Bellemare 
> wrote:
> 
>> Bumping this thread, as per convention - 1
>> 
>> On Fri, Nov 2, 2018 at 8:22 AM Adam Bellemare 
>> wrote:
>> 
>>> As expected :) But still, thanks none-the-less!
>>> 
>>> On Fri, Nov 2, 2018 at 3:36 AM Jan Filipiak 
>>> wrote:
>>> 
 reminder
 
> On 30.10.2018 15:47, Adam Bellemare wrote:
> Hi All
> 
> I would like to call a vote on
> 
 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable
 .
> This allows a Kafka Streams DSL user to perform KTable to KTable
> foreign-key joins on their data. I have been using this in production
 for
> some time and I have composed a PR that enables this. It is a fairly
> extensive PR, but I believe it will add considerable value to the
>> Kafka
> Streams DSL.
> 
> The PR can be found here:
> https://github.com/apache/kafka/pull/5527
> 
> See
 http://mail-archives.apache.org/mod_mbox/kafka-dev/201810.mbox/browser
> for previous discussion thread.
> 
> I would also like to give a shout-out to Jan Filipiak who helped me
>> out
> greatly in this project, and who led the initial work into this
>> problem.
> Without Jan's help and insight I do not think this would have been
 possible
> to get to this point.
> 
> Adam
> 
 
>>> 
>> 

[jira] [Resolved] (KAFKA-6641) Consider auto repartitioning for Stream.transform() API

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6641.

Resolution: Duplicate

> Consider auto repartitioning for Stream.transform() API
> ---
>
> Key: KAFKA-6641
> URL: https://issues.apache.org/jira/browse/KAFKA-6641
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: api
>
> Today with map / mapValues of Streams DSL, we will set a flag / not set a 
> flag for the underlying topology builder; but for transform / 
> transformValues, we do not make such marking choices. Maybe the topology 
> builder can still make such flagging for transform() to indicate that since 
> the key maybe changed, we should issue a repartition for the downstream 
> stateful operators when necessary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-386: Standardize on Min/Avg/Max metrics' default values

[Dong Lin]

Thanks for the KIP Stanislav. +1 (binding)

On Thu, Nov 8, 2018 at 2:40 PM Jun Rao  wrote:

> Hi, Stanislav,
>
> Thanks for the KIP. +1. I guess this only covers the Kafka metrics (not the
> Yammer metrics). It would be useful to make this clear.
>
> Jun
>
> On Tue, Nov 6, 2018 at 1:00 AM, Stanislav Kozlovski <
> stanis...@confluent.io>
> wrote:
>
> > Hey everybody,
> >
> > I'm starting a vote thread on KIP-386: Standardize on Min/Avg/Max
> metrics'
> > default values
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652345
> > >
> > In short, after the discussion thread
> >  > 4b9014fbb50f663bf14e5aec67@%3Cdev.kafka.apache.org%3E>,
> > we decided to have all min/avg/max metrics output `NaN` as a default
> value.
> >
> > --
> > Best,
> > Stanislav
> >
>

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Adding discussion thread here:

https://lists.apache.org/thread.html/9868b07692960d7ab2924d2c66ad6ce7e42e492c35af9cbb7c9981ae@%3Cdev.kafka.apache.org%3E



From: Boyang Chen 
Sent: Friday, November 9, 2018 6:35 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances


Hey all,


thanks so much for all the inputs on KIP-345 so far. The original proposal has 
enhanced a lot with your help. To make sure the implementation go smoothly 
without back and forth, I would like to start a vote on the final design 
agreement now:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-

345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...


Let me know if you have any questions.


Best,

Boyang

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

Yes, I was about to send closing mail.

On Thu, Nov 8, 2018 at 9:39 PM Ismael Juma  wrote:

> Manikumar, looks like you have the necessary votes. :)
>
> Ismael
>
> On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  wrote:
>
>> +1 (binding)
>>
>> Checked source build and unit tests. Ran quickstart with source and
>> binary.
>>
>> Thank you for managing the release, Manikumar!
>>
>> Regards,
>>
>> Rajini
>>
>> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>>
>> > +1 (binding)
>> >
>> > Checked signatures, build and quickstart.
>> >
>> > Thank you for managing the release, Mani!
>> >
>> >
>> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> > wrote:
>> > >
>> > > Hello Kafka users, developers and client-developers,
>> > >
>> > > This is the first candidate for release of Apache Kafka 2.0.1.
>> > >
>> > > This is a bug fix release closing 49 tickets:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> > >
>> > > Release notes for the 2.0.1 release:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> > >
>> > > *** Please download, test and vote by  Tuesday, October 30, end of day
>> > >
>> > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > http://kafka.apache.org/KEYS
>> > >
>> > > * Release artifacts to be voted upon (source and binary):
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> > >
>> > > * Maven artifacts to be voted upon:
>> > > https://repository.apache.org/content/groups/staging/
>> > >
>> > > * Javadoc:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> > >
>> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> > >
>> > > * Documentation:
>> > > http://kafka.apache.org/20/documentation.html
>> > >
>> > > * Protocol:
>> > > http://kafka.apache.org/20/protocol.html
>> > >
>> > > * Successful Jenkins builds for the 2.0 branch:
>> > > Unit/integration tests:
>> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> > >
>> > > /**
>> > >
>> > > Thanks,
>> > > Manikumar
>> >
>> >
>> >
>> > --
>> > Gwen Shapira
>> > Product Manager | Confluent
>> > 650.450.2760 | @gwenshap
>> > Follow us: Twitter | blog
>> >
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZYww8XbFFwx5Os4pOjiANGz-Lq81xcSGiTXs-G%2BwpG%3Duw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Build failed in Jenkins: kafka-trunk-jdk8 #3188

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-7165: Retry the BrokerInfo registration into ZooKeeper (#5575)

--
[...truncated 2.73 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[Colin McCabe]

Hi Lincong Li,

I agree that server-side instrumentation is helpful.  However, I don't think 
this is the right approach.

The problem is that RequestChannel.Request and AbstractResponse are internal 
classes that should not be exposed.  These are implementation details that we 
may change in the future.  Freezing these into a public API would really hold 
back the project.  For example, for really large responses, we might eventually 
want to avoid materializing the whole response all at once.  It would make more 
sense to return it in a streaming fashion.  But if we need to support this API 
forever, we can't do that.

I think it's fair to say that this is, at best, half a solution to the problem 
of tracing requests.  Users still need to write the plugin code and arrange for 
it to be on their classpath to make this work.  I think the alternative here is 
not client-side instrumentation, but simply making the change to the broker 
without using a plugin interface.

If a public interface is absolutely necessary here we should expose only things 
like the API key, client ID, time, etc. that don't constrain the implementation 
a lot in the future.  I think we should also use java here to avoid the 
compatibility issues we have had with Scala APIs in the past.

best,
Colin


On Thu, Nov 8, 2018, at 11:34, radai wrote:
> another downside to client instrumentation (beyond the number of
> client codebases one would need to cover) is that in a large
> environments you'll have a very long tail of applications using older
> clients to upgrade - it would be a long and disruptive process (as
> opposed to updating broker-side instrumentation)
> On Thu, Nov 8, 2018 at 11:04 AM Peter M. Elias  wrote:
> >
> > I know we have a lot of use cases for this type of functionality at my
> > enterprise deployment. I think it's helpful for maintaining reliability of
> > the cluster especially and identifying clients that are not properly tuned
> > and therefore applying excessive load to the brokers. Additionally, there
> > is a bit of a dark spot without something like as currently. For example,
> > if a client is not using a consumer group, there is no direct way to query
> > the state of the consumer without looking at raw network connections to
> > determine the extent of the traffic generated by that particular consumer.
> >
> > While client instrumentation can certainly help with this currently, given
> > that Kafka is intended to be a shared service across a potentially very
> > large surface area of clients, central observation of client activity is in
> > my opinion an essential feature.
> >
> > Peter
> >
> > On Thu, Nov 8, 2018 at 12:13 PM radai  wrote:
> >
> > > bump.
> > >
> > > I think the proposed API (Observer) is useful for any sort of
> > > multi-tenant environment for chargeback and reporting purposes.
> > >
> > > if no one wants to comment, can we initiate a vote?
> > > On Mon, Nov 5, 2018 at 6:31 PM Lincong Li  wrote:
> > > >
> > > > Hi everyone. Here
> > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response
> > > >
> > > > is
> > > > my KIP. Any feedback is appreciated.
> > > >
> > > > Thanks,
> > > > Lincong Li
> > >

[jira] [Resolved] (KAFKA-7412) Bug prone response from producer.send(ProducerRecord, Callback) if Kafka broker is not running

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-7412.

   Resolution: Fixed
Fix Version/s: 2.2.0

Clarified the javadoc in producer callback. Metadata is not null with non-null 
exception.

Merged the PR to trunk.

> Bug prone response from producer.send(ProducerRecord, Callback) if Kafka 
> broker is not running
> --
>
> Key: KAFKA-7412
> URL: https://issues.apache.org/jira/browse/KAFKA-7412
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 2.0.0
>Reporter: Michal Turek
>Assignee: huxihx
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: both_metadata_and_exception.png, 
> metadata_when_kafka_is_stopped.png
>
>
> Hi there, I have probably found a bug in Java Kafka producer client.
> Scenario & current behavior:
> - Start Kafka broker, single instance.
> - Start application that produces messages to Kafka.
> - Let the application to load partitions for a topic to warm up the producer, 
> e.g. send a message to Kafka. I'm not sure if this is necessary step, but our 
> code does it.
> - Gracefully stop the Kafka broker.
> - Application logs now contains "org.apache.kafka.clients.NetworkClient: 
> [Producer clientId=...] Connection to node 0 could not be established. Broker 
> may not be available." so the client is aware about the Kafka unavailability.
> - Trigger the producer to send a message using 
> KafkaProducer.send(ProducerRecord, Callback) method.
> - The callback that notifies business code receives non-null RecordMetadata 
> and null Exception after request.timeout.ms. The metadata contains offset -1 
> which is value of ProduceResponse.INVALID_OFFSET.
> Expected behavior:
> - If the Kafka is not running and the message is not appended to the log, the 
> callback should contain null RecordMetadata and non-null Exception. At least 
> I subjectively understand the Javadoc this way, "exception on production 
> error" in simple words.
> - Developer that is not aware of this behavior and that doesn't test for 
> offset -1, may consider the message as successfully send and properly acked 
> by the broker.
> Known workaround
> - Together with checking for non-null exception in the callback, add another 
> condition for ProduceResponse.INVALID_OFFSET.
> {noformat}
> try {
> producer.send(record, (metadata, exception) -> {
> if (metadata != null) {
> if (metadata.offset() != 
> ProduceResponse.INVALID_OFFSET) {
> // Success
> } else {
> // Failure
> }
> } else {
> // Failure
> }
> });
> } catch (Exception e) {
> // Failure
> }
> {noformat}
> Used setup
> - Latest Kafka 2.0.0 for both broker and Java client.
> - Originally found with broker 0.11.0.1 and client 2.0.0.
> - Code is analogy of the one in Javadoc of KafkaProducer.send().
> - Used producer configuration (others use defaults).
> {noformat}
> bootstrap.servers = "localhost:9092"
> client.id = "..."
> acks = "all"
> retries = 1
> linger.ms = "20"
> compression.type = "lz4"
> request.timeout.ms = 5000 # The same behavior is with default, this is to 
> speed up the tests
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #86

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-7412: clarify the doc for producer callback (#5798)

--
[...truncated 2.33 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

Re: Kafka Performance Producer - Extension

[Colin McCabe]

On Thu, Nov 8, 2018, at 01:38, Srinivas, Kaushik (Nokia - IN/Bangalore) wrote:> 
Hi All,


>  


> This is in regard to extend few features support for java kafka
> performance producer.>  



Hi Kaushik,

Have you created a pull request against the Kafka repo at
https://github.com/apache/kafka/ ?
I also suggest checking out Trogdor, which has a suite of many workloads
and stress tests that can be run against Kafka.  See
https://github.com/apache/kafka/blob/trunk/TROGDOR.md
> We have a kafka producer application written in java with maven build.> 
> Attaching the documentation which gives detailed description of
> features supported by this application as of now.>  


> Would like to know if this can be integrated into kafka performance
> producer java implementation which is present with limited features.> Kindly 
> review the attached document, and help with feedback on how
> well this could be integrated.
I'm sorry, but I do not open MS Word attachments because of
security issues.
best,
Colin

> 
>  
> These features would help any team load testing kafka brokers/schema
> registry components efficiently.>  


> Best Regards,


> -kaushik


> Email had 1 attachment:


>  * KAFKA_PRODUCER_FEATURES_DOC.docx 22k (application/vnd.openxmlformats-
>officedocument.wordprocessingml.document)

Re: [VOTE] KIP-374: Add '--help' option to all available Kafka CLI commands

[Colin McCabe]

+1 (binding)



On Wed, Oct 31, 2018, at 05:42, Srinivas Reddy wrote:
> Hi All,
>
> I would like to call for a vote on KIP-374:
> https://cwiki.apache.org/confluence/x/FgSQBQ
>
> Summary:
> Currently, the '--help' option is recognized by some Kafka commands
> but not all. To provide a consistent user experience, it would
> be nice to> add a '--help' option to all Kafka commands.
>
> I'd appreciate any votes or feedback.
>
> --
> Srinivas Reddy
>
> http://mrsrinivas.com/
>
>
> (Sent via gmail web)

Jenkins build is back to normal : kafka-trunk-jdk8 #3189

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk11 #85

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

[Ismael Juma]

I agree, the current KIP doesn't discuss the public API that we would be
exposing and it's extensive if the normal usage would allow for casting
AbstractRequest into the various subclasses and potentially even accessing
Records and related for produce request.

There are many use cases where this could be useful, but it requires quite
a bit of thinking around the APIs that we expose and the expected usage.

Ismael

On Thu, Nov 8, 2018, 6:09 PM Colin McCabe  Hi Lincong Li,
>
> I agree that server-side instrumentation is helpful.  However, I don't
> think this is the right approach.
>
> The problem is that RequestChannel.Request and AbstractResponse are
> internal classes that should not be exposed.  These are implementation
> details that we may change in the future.  Freezing these into a public API
> would really hold back the project.  For example, for really large
> responses, we might eventually want to avoid materializing the whole
> response all at once.  It would make more sense to return it in a streaming
> fashion.  But if we need to support this API forever, we can't do that.
>
> I think it's fair to say that this is, at best, half a solution to the
> problem of tracing requests.  Users still need to write the plugin code and
> arrange for it to be on their classpath to make this work.  I think the
> alternative here is not client-side instrumentation, but simply making the
> change to the broker without using a plugin interface.
>
> If a public interface is absolutely necessary here we should expose only
> things like the API key, client ID, time, etc. that don't constrain the
> implementation a lot in the future.  I think we should also use java here
> to avoid the compatibility issues we have had with Scala APIs in the past.
>
> best,
> Colin
>
>
> On Thu, Nov 8, 2018, at 11:34, radai wrote:
> > another downside to client instrumentation (beyond the number of
> > client codebases one would need to cover) is that in a large
> > environments you'll have a very long tail of applications using older
> > clients to upgrade - it would be a long and disruptive process (as
> > opposed to updating broker-side instrumentation)
> > On Thu, Nov 8, 2018 at 11:04 AM Peter M. Elias 
> wrote:
> > >
> > > I know we have a lot of use cases for this type of functionality at my
> > > enterprise deployment. I think it's helpful for maintaining
> reliability of
> > > the cluster especially and identifying clients that are not properly
> tuned
> > > and therefore applying excessive load to the brokers. Additionally,
> there
> > > is a bit of a dark spot without something like as currently. For
> example,
> > > if a client is not using a consumer group, there is no direct way to
> query
> > > the state of the consumer without looking at raw network connections to
> > > determine the extent of the traffic generated by that particular
> consumer.
> > >
> > > While client instrumentation can certainly help with this currently,
> given
> > > that Kafka is intended to be a shared service across a potentially very
> > > large surface area of clients, central observation of client activity
> is in
> > > my opinion an essential feature.
> > >
> > > Peter
> > >
> > > On Thu, Nov 8, 2018 at 12:13 PM radai 
> wrote:
> > >
> > > > bump.
> > > >
> > > > I think the proposed API (Observer) is useful for any sort of
> > > > multi-tenant environment for chargeback and reporting purposes.
> > > >
> > > > if no one wants to comment, can we initiate a vote?
> > > > On Mon, Nov 5, 2018 at 6:31 PM Lincong Li 
> wrote:
> > > > >
> > > > > Hi everyone. Here
> > > > > <
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response
> > > > >
> > > > > is
> > > > > my KIP. Any feedback is appreciated.
> > > > >
> > > > > Thanks,
> > > > > Lincong Li
> > > >
>

[VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

[Kevin Lu]

Hi All,

I'm starting the vote thread for KIP-351: Add --under-min-isr option to
describe topics command.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command

Discussion thread:
https://lists.apache.org/thread.html/90d1652ebc03a7be4100dd101b92a7dcefe63d144856c5f6c132381b@%3Cdev.kafka.apache.org%3E

Thanks!

Regards,
Kevin

[jira] [Created] (KAFKA-7608) A Kafka Streams DSL transform or processor call should trigger a repartition like a join

[Andy Bryant (JIRA)]

Andy Bryant created KAFKA-7608:
--

 Summary: A Kafka Streams DSL transform or processor call should 
trigger a repartition like a join
 Key: KAFKA-7608
 URL: https://issues.apache.org/jira/browse/KAFKA-7608
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.0
Reporter: Andy Bryant


Currently in Kafka Streams, if any DSL operation occurs that may modify the 
keys of the record stream, the stream is flagged for repartitioning. Currently 
this flag is checked prior to a stream join or an aggregation and if set the 
stream is piped through a transient repartition topic. This ensures messages 
with the same key are always co-located in the same partition and hence same 
stream task and state store.

The same mechanism should be used to trigger repartitioning prior to stream 
{{transform}}, {{transformValues}} and {{process}} calls that specify one or 
more state stores.

Currently without the forced repartitioning, for streams where the key has been 
modified, there is no guarantee the same keys will be processed by the same 
task which would be what you expect when using a state store. Given that 
aggregations and joins already automatically make this guarantee it seems 
inconsistent that {{transform}} and {{process}} do not provide the same 
guarantees.

To achieve the same guarantees currently, developers must manually pipe the 
stream through a topic to force the repartitioning. This works, but is 
sub-optimal since you don't get the handy optimisation where the repartition 
topic contents is purged after use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #84

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-7604; Fix flaky unit test

--
[...truncated 2.33 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

RE: Kafka Performance Producer - Extension

[Srinivas, Kaushik (Nokia - IN/Bangalore)]

Hi Colin,

I haven’t yet created the pull request.

Wanted initial feedback for the same.

-kaushik

-Original Message-
From: Colin McCabe  
Sent: Friday, November 09, 2018 7:27 AM
To: dev@kafka.apache.org
Subject: Re: Kafka Performance Producer - Extension

On Thu, Nov 8, 2018, at 01:38, Srinivas, Kaushik (Nokia - IN/Bangalore) wrote:> 
Hi All,


>  


> This is in regard to extend few features support for java kafka 
> performance producer.>



Hi Kaushik,

Have you created a pull request against the Kafka repo at
https://github.com/apache/kafka/ ?
I also suggest checking out Trogdor, which has a suite of many workloads
and stress tests that can be run against Kafka.  See
https://github.com/apache/kafka/blob/trunk/TROGDOR.md
> We have a kafka producer application written in java with maven build.> 
> Attaching the documentation which gives detailed description of
> features supported by this application as of now.>  


> Would like to know if this can be integrated into kafka performance
> producer java implementation which is present with limited features.> Kindly 
> review the attached document, and help with feedback on how
> well this could be integrated.
I'm sorry, but I do not open MS Word attachments because of
security issues.
best,
Colin

> 
>  
> These features would help any team load testing kafka brokers/schema
> registry components efficiently.>  


> Best Regards,


> -kaushik


> Email had 1 attachment:


>  * KAFKA_PRODUCER_FEATURES_DOC.docx 22k (application/vnd.openxmlformats-
>officedocument.wordprocessingml.document)

RE: Kafka Performance Producer - Extension

[Srinivas, Kaushik (Nokia - IN/Bangalore)]

Attaching the pdf file with summary of features implemented in our kafka 
producer application.

Kindly review and provide the feedback.

-kaushik

-Original Message-
From: Srinivas, Kaushik (Nokia - IN/Bangalore) 
Sent: Friday, November 09, 2018 11:00 AM
To: dev@kafka.apache.org; 'cmcc...@apache.org' 
Subject: RE: Kafka Performance Producer - Extension

Hi Colin,

I haven’t yet created the pull request.

Wanted initial feedback for the same.

-kaushik

-Original Message-
From: Colin McCabe 
Sent: Friday, November 09, 2018 7:27 AM
To: dev@kafka.apache.org
Subject: Re: Kafka Performance Producer - Extension

On Thu, Nov 8, 2018, at 01:38, Srinivas, Kaushik (Nokia - IN/Bangalore) wrote:> 
Hi All,


>  


> This is in regard to extend few features support for java kafka 
> performance producer.>



Hi Kaushik,

Have you created a pull request against the Kafka repo at 
https://github.com/apache/kafka/ ?
I also suggest checking out Trogdor, which has a suite of many workloads and 
stress tests that can be run against Kafka.  See 
https://github.com/apache/kafka/blob/trunk/TROGDOR.md
> We have a kafka producer application written in java with maven 
> build.> Attaching the documentation which gives detailed description 
> of features supported by this application as of now.>


> Would like to know if this can be integrated into kafka performance 
> producer java implementation which is present with limited features.> 
> Kindly review the attached document, and help with feedback on how well this 
> could be integrated.
I'm sorry, but I do not open MS Word attachments because of security issues.
best,
Colin

> 
>  
> These features would help any team load testing kafka brokers/schema 
> registry components efficiently.>


> Best Regards,


> -kaushik


> Email had 1 attachment:


>  * KAFKA_PRODUCER_FEATURES_DOC.docx 22k (application/vnd.openxmlformats-
>officedocument.wordprocessingml.document)

Jenkins build is back to normal : kafka-trunk-jdk8 #3187

[Apache Jenkins Server]

See