Re: [DISCUSS] KIP-73: Replication Quotas

[Joel Koshy]

On Wed, Aug 17, 2016 at 9:13 PM, Ben Stopford  wrote:

>
> Let's us know if you have any further thoughts on KIP-73, else we'll kick
> off a vote.
>

I think the mechanism for throttling replicas looks good. Just had a few
more thoughts on the configuration section. What you have looks reasonable,
but I was wondering if it could be made simpler. You probably thought
through these, so I'm curious to know your take.

My guess is that most of the time, users would want to throttle all effect
replication - due to partition reassignments, adding brokers or a broker
coming back online after an extended period of time. In all these scenarios
it may be possible to distinguish bootstrap (effect) vs normal replication
- based on how far the replica has to catch up. I'm wondering if it is
enough to just set an umbrella "effect" replication quota with perhaps
per-topic overrides (say if some topics are more important than others) as
opposed to designating throttled replicas.

Also, IIRC during client-side quota discussions we had considered the
possibility of allowing clients to go above their quotas when resources are
available. We ended up not doing that, but for replication throttling it
may make sense - i.e., to treat the quota as a soft limit. Another way to
look at it is instead of ensuring "effect replication traffic does not flow
faster than X bytes/sec" it may be useful to instead ensure that "effect
replication traffic only flows as slowly as necessary (so as not to
adversely affect normal replication traffic)."

Thanks,

Joel

> >
> > > On Thu, Aug 11, 2016 at 2:43 PM, Jun Rao  > > wrote:
> > >
> > > > Hi, Joel,
> > > >
> > > > Yes, the response size includes both throttled and unthrottled
> > replicas.
> > > > However, the response is only delayed up to max.wait if the response
> > size
> > > > is less than min.bytes, which matches the current behavior. So, there
> > is
> > > no
> > > > extra delay to due throttling, right? For replica fetchers, the
> default
> > > > min.byte is 1. So, the response is only delayed if there is no byte
> in
> > > the
> > > > response, which is what we want.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Aug 11, 2016 at 11:53 AM, Joel Koshy  > >
> > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > I'm not sure that would work unless we have separate replica
> > fetchers,
> > > > > since this would cause all replicas (including ones that are not
> > > > throttled)
> > > > > to get delayed. Instead, we could just have the leader populate the
> > > > > throttle-time field of the response as a hint to the follower as to
> > how
> > > > > long it should wait before it adds those replicas back to its
> > > subsequent
> > > > > replica fetch requests.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Joel
> > > > >
> > > > > On Thu, Aug 11, 2016 at 9:50 AM, Jun Rao  > > wrote:
> > > > >
> > > > > > Mayuresh,
> > > > > >
> > > > > > That's a good question. I think if the response size (after
> leader
> > > > > > throttling) is smaller than min.bytes, we will just delay the
> > sending
> > > > of
> > > > > > the response up to max.wait as we do now. This should prevent
> > > frequent
> > > > > > empty responses to the follower.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Aug 10, 2016 at 9:17 PM, Mayuresh Gharat <
> > > > > > gharatmayures...@gmail.com 
> > > > > > > wrote:
> > > > > >
> > > > > > > This might have been answered before.
> > > > > > > I was wondering when the leader quota is reached and it sends
> > empty
> > > > > > > response ( If the inclusion of a partition, listed in the
> > leader's
> > > > > > > throttled-replicas list, causes the LeaderQuotaRate to be
> > exceeded,
> > > > > that
> > > > > > > partition is omitted from the response (aka returns 0 bytes).).
> > At
> > > > this
> > > > > > > point the follower quota is NOT reached and the follower is
> still
> > > > going
> > > > > > to
> > > > > > > ask for the that partition in the next fetch request. Would it
> be
> > > > fair
> > > > > to
> > > > > > > add some logic there so that the follower backs off ( for some
> > > > > > configurable
> > > > > > > time) from including those partitions in the next fetch
> request?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mayuresh
> > > > > > >
> > > > > > > On Wed, Aug 10, 2016 at 8:06 AM, Ben Stopford <
> b...@confluent.io
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks again for the responses everyone. I’ve removed the the
> > > extra
> > > > > > > > fetcher threads from the proposal, switching to the
> > > inclusion-based
> > > > > > > > approach. The relevant section is:
> > > > > > > >
> > > > > > > > The follower makes a requests, using the fixed size of
> > > > > > > > replica.fetch.response.max.bytes as per 

Re: [ANNOUNCE] New Kafka PMC member Gwen Shapira

[Joel Koshy]

Congrats!

On Wed, Aug 17, 2016 at 9:28 PM, Sriram Subramanian 
wrote:

> Congratulations Gwen!
>
> On Wed, Aug 17, 2016 at 9:24 PM, Neha Narkhede  wrote:
>
> > Congratulations and welcome, Gwen!
> > On Wed, Aug 17, 2016 at 6:44 PM Jun Rao  wrote:
> >
> > > Hi, Everyone,
> > >
> > > Gwen Shapira has been active in the Kafka community since she became a
> > > Kafka committer
> > > about a year ago. I am glad to announce that Gwen is now a member of
> > Kafka
> > > PMC.
> > >
> > > Congratulations, Gwen!
> > >
> > > Jun
> > >
> > --
> > Thanks,
> > Neha
> >
>

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425906#comment-15425906
 ] 

Joel Koshy commented on KAFKA-4050:
---

[~toddpalino] I had left a comment about this on the PR - one option is to 
default to SHA1PRNG and fall-back to null on NoSuchAlgorithmException

> Allow configuration of the PRNG used for SSL
> 
>
> Key: KAFKA-4050
> URL: https://issues.apache.org/jira/browse/KAFKA-4050
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.0.1
>Reporter: Todd Palino
>Assignee: Todd Palino
>  Labels: security, ssl
>
> This change will make the pseudo-random number generator (PRNG) 
> implementation used by the SSLContext configurable. The configuration is not 
> required, and the default is to use whatever the default PRNG for the JDK/JRE 
> is. Providing a string, such as "SHA1PRNG", will cause that specific 
> SecureRandom implementation to get passed to the SSLContext.
> When enabling inter-broker SSL in our certification cluster, we observed 
> severe performance issues. For reference, this cluster can take up to 600 
> MB/sec of inbound produce traffic over SSL, with RF=2, before it gets close 
> to saturation, and the mirror maker normally produces about 400 MB/sec 
> (unless it is lagging). When we enabled inter-broker SSL, we saw persistent 
> replication problems in the cluster at any inbound rate of more than about 6 
> or 7 MB/sec per-broker. This was narrowed down to all the network threads 
> blocking on a single lock in the SecureRandom code.
> It turns out that the default PRNG implementation on Linux is NativePRNG. 
> This uses randomness from /dev/urandom (which, by itself, is a non-blocking 
> read) and mixes it with randomness from SHA1. The problem is that the entire 
> application shares a single SecureRandom instance, and NativePRNG has a 
> global lock within the implNextBytes method. Switching to another 
> implementation (SHA1PRNG, which has better performance characteristics and is 
> still considered secure) completely eliminated the bottleneck and allowed the 
> cluster to work properly at saturation.
> The SSLContext initialization has an optional argument to provide a 
> SecureRandom instance, which the code currently sets to null. This change 
> creates a new config to specify an implementation, and instantiates that and 
> passes it to SSLContext if provided. This will also let someone select a 
> stronger source of randomness (obviously at a performance cost) if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Kafka PMC member Gwen Shapira

[Sriram Subramanian]

Congratulations Gwen!

On Wed, Aug 17, 2016 at 9:24 PM, Neha Narkhede  wrote:

> Congratulations and welcome, Gwen!
> On Wed, Aug 17, 2016 at 6:44 PM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Gwen Shapira has been active in the Kafka community since she became a
> > Kafka committer
> > about a year ago. I am glad to announce that Gwen is now a member of
> Kafka
> > PMC.
> >
> > Congratulations, Gwen!
> >
> > Jun
> >
> --
> Thanks,
> Neha
>

Re: [ANNOUNCE] New Kafka PMC member Gwen Shapira

[Neha Narkhede]

Congratulations and welcome, Gwen!
On Wed, Aug 17, 2016 at 6:44 PM Jun Rao  wrote:

> Hi, Everyone,
>
> Gwen Shapira has been active in the Kafka community since she became a
> Kafka committer
> about a year ago. I am glad to announce that Gwen is now a member of Kafka
> PMC.
>
> Congratulations, Gwen!
>
> Jun
>
-- 
Thanks,
Neha

Re: [DISCUSS] KIP-73: Replication Quotas

[Ben Stopford]

Hi everyone

Let's us know if you have any further thoughts on KIP-73, else we'll kick
off a vote.

Thanks

B

On Friday, 12 August 2016, Jun Rao  wrote:

> Mayuresh,
>
> I was thinking of the following.
>
> If P1 has data and P2 is throttled, we will return empty data for P2 and
> send the response back immediately. The follower will issue the next fetch
> request immediately, but the leader won't return any data in P2 until the
> quota is not exceeded. We are not delaying the fetch requests here.
> However, there is no additional overhead compared with no throttling since
> P1 always has data.
>
> If P1 has no data and P2 is throttled, the leader will return empty data
> for both P1 and P2 after waiting in the Purgatory up to max.wait. This
> prevents the follower from getting empty responses too frequently.
>
> Thanks,
>
> Jun
>
> On Thu, Aug 11, 2016 at 5:33 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com 
> > wrote:
>
> > Hi Jun,
> >
> > Correct me if I am wrong.
> > If the response size includes throttled and unthrottled replicas, I am
> > thinking if this is possible :
> > The leader broker B1 receives a fetch request partition P1 and P2 for a
> > topic from replica broker B2. In this case lets say that only P2 is
> > throttled on the leader and P1 is not. In that case we will add the data
> > for P1 in the response in which case the min.bytes threshold will be
> > crossed and the response will be returned back right?
> > If we say that with this kip, we will throttle this fetch request
> entirely,
> > then we are essentially delaying response for partition P1 which is not
> the
> > throttled partition.
> >
> > Is it fair to say we can indicate to the follower in the fetch response,
> > how much time it should wait till it adds back a fetch request for
> > partition P2.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Thu, Aug 11, 2016 at 2:43 PM, Jun Rao  > wrote:
> >
> > > Hi, Joel,
> > >
> > > Yes, the response size includes both throttled and unthrottled
> replicas.
> > > However, the response is only delayed up to max.wait if the response
> size
> > > is less than min.bytes, which matches the current behavior. So, there
> is
> > no
> > > extra delay to due throttling, right? For replica fetchers, the default
> > > min.byte is 1. So, the response is only delayed if there is no byte in
> > the
> > > response, which is what we want.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Aug 11, 2016 at 11:53 AM, Joel Koshy  >
> > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > I'm not sure that would work unless we have separate replica
> fetchers,
> > > > since this would cause all replicas (including ones that are not
> > > throttled)
> > > > to get delayed. Instead, we could just have the leader populate the
> > > > throttle-time field of the response as a hint to the follower as to
> how
> > > > long it should wait before it adds those replicas back to its
> > subsequent
> > > > replica fetch requests.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Thu, Aug 11, 2016 at 9:50 AM, Jun Rao  > wrote:
> > > >
> > > > > Mayuresh,
> > > > >
> > > > > That's a good question. I think if the response size (after leader
> > > > > throttling) is smaller than min.bytes, we will just delay the
> sending
> > > of
> > > > > the response up to max.wait as we do now. This should prevent
> > frequent
> > > > > empty responses to the follower.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Aug 10, 2016 at 9:17 PM, Mayuresh Gharat <
> > > > > gharatmayures...@gmail.com 
> > > > > > wrote:
> > > > >
> > > > > > This might have been answered before.
> > > > > > I was wondering when the leader quota is reached and it sends
> empty
> > > > > > response ( If the inclusion of a partition, listed in the
> leader's
> > > > > > throttled-replicas list, causes the LeaderQuotaRate to be
> exceeded,
> > > > that
> > > > > > partition is omitted from the response (aka returns 0 bytes).).
> At
> > > this
> > > > > > point the follower quota is NOT reached and the follower is still
> > > going
> > > > > to
> > > > > > ask for the that partition in the next fetch request. Would it be
> > > fair
> > > > to
> > > > > > add some logic there so that the follower backs off ( for some
> > > > > configurable
> > > > > > time) from including those partitions in the next fetch request?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > > On Wed, Aug 10, 2016 at 8:06 AM, Ben Stopford  >
> > > > wrote:
> > > > > >
> > > > > > > Thanks again for the responses everyone. I’ve removed the the
> > extra
> > > > > > > fetcher threads from the proposal, switching to the
> > inclusion-based
> > > > > > > approach. The relevant section is:
> > > > > > >
> > > > > > > The follower makes a 

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Gwen Shapira]

Jayesh, if you have a github repo with your tool, we can add it to the
ecosystem page in the wiki, so more people will find it.

On Aug 17, 2016 8:00 PM, "Jayesh Thakrar" 
wrote:

> Thank you Grant and Gwen for the valuable feedback and suggestions.Yes, it
> makes sense to wait for the admin client API.In the meantime, I can spin
> this off as a standalone tool and allow things to bake, giving the utility
> more field usage and maturity.
> As for the endpoint info, I included it in the KIP because the information
> was available in the current API.I am sure once the admin API is released,
> the KIP can be revisited.
> Thanks for sharing your insights!
> Jayesh
>
>   From: Grant Henke 
>  To: dev@kafka.apache.org
> Cc: Jayesh Thakrar 
>  Sent: Wednesday, August 17, 2016 9:17 PM
>  Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
>
> Hi Jayesh,
> Like Gwen said KIP-4 is adding fields and requests to the wire protocols
> in order to allow all admin tools to talk directly to Kafka and a client
> api to support those requests. Talking to Kafka as opposed to Zookeeper
> allows us to leverage authorization, lock down zookeeper, and improve
> compatibility. Like Gwen said waiting until some of the KIP-4 work is done
> may avoid rework. I can't make a commitment to how fast the client will be
> available as it depends on many factors but progress is being made
> regularly and I do have some mock client work done locally.
> It looks like the only thing in your proposal that can not be supported
> via the wire protocol today is the endpoints metadata. It's sort of a
> catch-22 because the bootstrap is required to connect to a Kafka cluster
> (as opposed to zookeeper) and at that point the Metadata returned assumes
> an endpoint of the connecting security protocol. Is that an acceptable
> limitation?
> Thanks,Grant
> On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira  wrote:
>
> Thanks Jayesh.
>
> I think this can be a useful addition to Apache Kafka.
> One potential issue is that you are getting all the information for
> ZooKeeper. We already added a protocol that allows adding the
> information to Kafka itself and are in the process of adding an admin
> client (i.e. Java client, not CLI).
> If you add this as planned, we'll need to re-work it to work with
> Kafka directly instead of ZooKeeper once the admin client lands. If
> you choose, you can wait for the admin client to arrive first, and
> avoid the re-work.
>
> Gwen
>
> On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
>  wrote:
> > All,
> > If there is no discussion, feedback or objection, is there any concern
> in going to the next step of voting?
> > Thanks,Jayesh
> >   From: Jayesh Thakrar 
> >  To: "dev@kafka.apache.org" 
> >  Sent: Saturday, August 13, 2016 11:44 PM
> >  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
> >
> > This is to start off a discussion on the above KIP at
> > https://cwiki.apache.org/ confluence/display/KAFKA/KIP-
> 59%3A+Proposal+for+a+kafka+ broker+commandThe proposal is to fill the void
> of a command line tool/utility that can provide information on the brokers
> in a Kafka cluster.
> > The code is available on GitHub at https://github.com/JThakrar/
> kafkaThe KIP page has the help documentation as well as the output from the
> command with various options.Thank you,Jayesh Thakrar
> >
> >
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
>
>
> --
> Grant Henke
> Software Engineer | clouderagr...@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
>
>

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

Thank you Grant and Gwen for the valuable feedback and suggestions.Yes, it 
makes sense to wait for the admin client API.In the meantime, I can spin this 
off as a standalone tool and allow things to bake, giving the utility more 
field usage and maturity.
As for the endpoint info, I included it in the KIP because the information was 
available in the current API.I am sure once the admin API is released, the KIP 
can be revisited.
Thanks for sharing your insights!
Jayesh 

  From: Grant Henke 
 To: dev@kafka.apache.org 
Cc: Jayesh Thakrar 
 Sent: Wednesday, August 17, 2016 9:17 PM
 Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh
   
Hi Jayesh,
Like Gwen said KIP-4 is adding fields and requests to the wire protocols in 
order to allow all admin tools to talk directly to Kafka and a client api to 
support those requests. Talking to Kafka as opposed to Zookeeper allows us to 
leverage authorization, lock down zookeeper, and improve compatibility. Like 
Gwen said waiting until some of the KIP-4 work is done may avoid rework. I 
can't make a commitment to how fast the client will be available as it depends 
on many factors but progress is being made regularly and I do have some mock 
client work done locally. 
It looks like the only thing in your proposal that can not be supported via the 
wire protocol today is the endpoints metadata. It's sort of a catch-22 because 
the bootstrap is required to connect to a Kafka cluster (as opposed to 
zookeeper) and at that point the Metadata returned assumes an endpoint of the 
connecting security protocol. Is that an acceptable limitation?
Thanks,Grant
On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira  wrote:

Thanks Jayesh.

I think this can be a useful addition to Apache Kafka.
One potential issue is that you are getting all the information for
ZooKeeper. We already added a protocol that allows adding the
information to Kafka itself and are in the process of adding an admin
client (i.e. Java client, not CLI).
If you add this as planned, we'll need to re-work it to work with
Kafka directly instead of ZooKeeper once the admin client lands. If
you choose, you can wait for the admin client to arrive first, and
avoid the re-work.

Gwen

On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
 wrote:
> All,
> If there is no discussion, feedback or objection, is there any concern in 
> going to the next step of voting?
> Thanks,Jayesh
>       From: Jayesh Thakrar 
>  To: "dev@kafka.apache.org" 
>  Sent: Saturday, August 13, 2016 11:44 PM
>  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
>kafka-brokers.sh
>
> This is to start off a discussion on the above KIP at
> https://cwiki.apache.org/ confluence/display/KAFKA/KIP- 
> 59%3A+Proposal+for+a+kafka+ broker+commandThe proposal is to fill the void of 
> a command line tool/utility that can provide information on the brokers in a 
> Kafka cluster.
> The code is available on GitHub at https://github.com/JThakrar/ kafkaThe KIP 
> page has the help documentation as well as the output from the command with 
> various options.Thank you,Jayesh Thakrar
>
>
>



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog




-- 
Grant Henke 
Software Engineer | clouderagr...@cloudera.com | twitter.com/gchenke | 
linkedin.com/in/granthenke

  

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Grant Henke]

Hi Jayesh,

Like Gwen said KIP-4 is adding fields and requests to the wire protocols in
order to allow all admin tools to talk directly to Kafka and a client api
to support those requests. Talking to Kafka as opposed to Zookeeper allows
us to leverage authorization, lock down zookeeper, and improve
compatibility. Like Gwen said waiting until some of the KIP-4 work is done
may avoid rework. I can't make a commitment to how fast the client will be
available as it depends on many factors but progress is being made
regularly and I do have some mock client work done locally.

It looks like the only thing in your proposal that can not be supported via
the wire protocol today is the endpoints metadata. It's sort of a catch-22
because the bootstrap is required to connect to a Kafka cluster (as opposed
to zookeeper) and at that point the Metadata returned assumes an endpoint
of the connecting security protocol. Is that an acceptable limitation?

Thanks,
Grant

On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira  wrote:

> Thanks Jayesh.
>
> I think this can be a useful addition to Apache Kafka.
> One potential issue is that you are getting all the information for
> ZooKeeper. We already added a protocol that allows adding the
> information to Kafka itself and are in the process of adding an admin
> client (i.e. Java client, not CLI).
> If you add this as planned, we'll need to re-work it to work with
> Kafka directly instead of ZooKeeper once the admin client lands. If
> you choose, you can wait for the admin client to arrive first, and
> avoid the re-work.
>
> Gwen
>
> On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
>  wrote:
> > All,
> > If there is no discussion, feedback or objection, is there any concern
> in going to the next step of voting?
> > Thanks,Jayesh
> >   From: Jayesh Thakrar 
> >  To: "dev@kafka.apache.org" 
> >  Sent: Saturday, August 13, 2016 11:44 PM
> >  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
> >
> > This is to start off a discussion on the above KIP at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 59%3A+Proposal+for+a+kafka+broker+commandThe proposal is to fill the void
> of a command line tool/utility that can provide information on the brokers
> in a Kafka cluster.
> > The code is available on GitHub at https://github.com/JThakrar/kafkaThe
> KIP page has the help documentation as well as the output from the command
> with various options.Thank you,Jayesh Thakrar
> >
> >
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

[ANNOUNCE] New Kafka PMC member Gwen Shapira

[Jun Rao]

Hi, Everyone,

Gwen Shapira has been active in the Kafka community since she became a
Kafka committer
about a year ago. I am glad to announce that Gwen is now a member of Kafka
PMC.

Congratulations, Gwen!

Jun

[jira] [Commented] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425777#comment-15425777
 ] 

ASF GitHub Bot commented on KAFKA-4053:
---

GitHub user sh-z reopened a pull request:

https://github.com/apache/kafka/pull/1751

KAFKA-4053: remove redundant if/else statements in TopicCommand[WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sh-z/kafka KAFKA-4053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1751


commit 6e7458e03cb0725db6a7079466fd7fce4a2f7603
Author: Shuai Zhang 
Date:   2016-08-17T09:29:54Z

remove redundant if/else statements




> Refactor TopicCommand to remove redundant if/else statements
> 
>
> Key: KAFKA-4053
> URL: https://issues.apache.org/jira/browse/KAFKA-4053
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.0.1
>Reporter: Shuai Zhang
>Priority: Minor
> Fix For: 0.10.0.2
>
>
> In TopicCommand, there are a lot of redundant if/else statements, such as
> ```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
> false```
> We can refactor it as the following statement:
> ```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425774#comment-15425774
 ] 

ASF GitHub Bot commented on KAFKA-4053:
---

Github user sh-z closed the pull request at:

https://github.com/apache/kafka/pull/1751


> Refactor TopicCommand to remove redundant if/else statements
> 
>
> Key: KAFKA-4053
> URL: https://issues.apache.org/jira/browse/KAFKA-4053
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.0.1
>Reporter: Shuai Zhang
>Priority: Minor
> Fix For: 0.10.0.2
>
>
> In TopicCommand, there are a lot of redundant if/else statements, such as
> ```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
> false```
> We can refactor it as the following statement:
> ```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1751: KAFKA-4053: remove redundant if/else statements in...

[sh-z]

GitHub user sh-z reopened a pull request:

https://github.com/apache/kafka/pull/1751

KAFKA-4053: remove redundant if/else statements in TopicCommand[WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sh-z/kafka KAFKA-4053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1751


commit 6e7458e03cb0725db6a7079466fd7fce4a2f7603
Author: Shuai Zhang 
Date:   2016-08-17T09:29:54Z

remove redundant if/else statements




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1751: KAFKA-4053: remove redundant if/else statements in...

[sh-z]

Github user sh-z closed the pull request at:

https://github.com/apache/kafka/pull/1751


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1751: KAFKA-4053: remove redundant if/else statements in...

[sh-z]

Github user sh-z closed the pull request at:

https://github.com/apache/kafka/pull/1751


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425772#comment-15425772
 ] 

ASF GitHub Bot commented on KAFKA-4053:
---

Github user sh-z closed the pull request at:

https://github.com/apache/kafka/pull/1751


> Refactor TopicCommand to remove redundant if/else statements
> 
>
> Key: KAFKA-4053
> URL: https://issues.apache.org/jira/browse/KAFKA-4053
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.0.1
>Reporter: Shuai Zhang
>Priority: Minor
> Fix For: 0.10.0.2
>
>
> In TopicCommand, there are a lot of redundant if/else statements, such as
> ```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
> false```
> We can refactor it as the following statement:
> ```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1751: KAFKA-4053: remove redundant if/else statements in...

[sh-z]

GitHub user sh-z reopened a pull request:

https://github.com/apache/kafka/pull/1751

KAFKA-4053: remove redundant if/else statements in TopicCommand[WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sh-z/kafka KAFKA-4053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1751


commit 6e7458e03cb0725db6a7079466fd7fce4a2f7603
Author: Shuai Zhang 
Date:   2016-08-17T09:29:54Z

remove redundant if/else statements




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425773#comment-15425773
 ] 

ASF GitHub Bot commented on KAFKA-4053:
---

GitHub user sh-z reopened a pull request:

https://github.com/apache/kafka/pull/1751

KAFKA-4053: remove redundant if/else statements in TopicCommand[WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sh-z/kafka KAFKA-4053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1751


commit 6e7458e03cb0725db6a7079466fd7fce4a2f7603
Author: Shuai Zhang 
Date:   2016-08-17T09:29:54Z

remove redundant if/else statements




> Refactor TopicCommand to remove redundant if/else statements
> 
>
> Key: KAFKA-4053
> URL: https://issues.apache.org/jira/browse/KAFKA-4053
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.0.1
>Reporter: Shuai Zhang
>Priority: Minor
> Fix For: 0.10.0.2
>
>
> In TopicCommand, there are a lot of redundant if/else statements, such as
> ```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
> false```
> We can refactor it as the following statement:
> ```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[Todd Palino (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425756#comment-15425756
 ] 

Todd Palino commented on KAFKA-4050:


So first off, yes, the thread dump (which [~jjkoshy] posted) shows that the 
offending line of code is "NativePRNG.java:481". I checked, and that's very 
clearly in the non-blocking NativePRNG variant that explictly uses /dev/urandom.

I had considered changing the default, [~ijuma], and I actually thought about 
adding a note to this ticket about it earlier today. Despite the fact that the 
default clearly has performance issues, I don't think we should change the 
default behavior, which is to let the JRE pick the PRNG implementation. The 
reason is that we can't be sure that on any given system, in any given JRE, 
that the new one we set explicitly will exist, and that would cause the default 
behavior to break. The SHA1PRNG implementation should exist everywhere, but I'd 
rather not take the risk. I think it's better to leave the default as is, and 
call out the issue very clearly in the documentation.

> Allow configuration of the PRNG used for SSL
> 
>
> Key: KAFKA-4050
> URL: https://issues.apache.org/jira/browse/KAFKA-4050
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.0.1
>Reporter: Todd Palino
>Assignee: Todd Palino
>  Labels: security, ssl
>
> This change will make the pseudo-random number generator (PRNG) 
> implementation used by the SSLContext configurable. The configuration is not 
> required, and the default is to use whatever the default PRNG for the JDK/JRE 
> is. Providing a string, such as "SHA1PRNG", will cause that specific 
> SecureRandom implementation to get passed to the SSLContext.
> When enabling inter-broker SSL in our certification cluster, we observed 
> severe performance issues. For reference, this cluster can take up to 600 
> MB/sec of inbound produce traffic over SSL, with RF=2, before it gets close 
> to saturation, and the mirror maker normally produces about 400 MB/sec 
> (unless it is lagging). When we enabled inter-broker SSL, we saw persistent 
> replication problems in the cluster at any inbound rate of more than about 6 
> or 7 MB/sec per-broker. This was narrowed down to all the network threads 
> blocking on a single lock in the SecureRandom code.
> It turns out that the default PRNG implementation on Linux is NativePRNG. 
> This uses randomness from /dev/urandom (which, by itself, is a non-blocking 
> read) and mixes it with randomness from SHA1. The problem is that the entire 
> application shares a single SecureRandom instance, and NativePRNG has a 
> global lock within the implNextBytes method. Switching to another 
> implementation (SHA1PRNG, which has better performance characteristics and is 
> still considered secure) completely eliminated the bottleneck and allowed the 
> cluster to work properly at saturation.
> The SSLContext initialization has an optional argument to provide a 
> SecureRandom instance, which the code currently sets to null. This change 
> creates a new config to specify an implementation, and instantiates that and 
> passes it to SSLContext if provided. This will also let someone select a 
> stronger source of randomness (obviously at a performance cost) if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Gwen Shapira]

Thanks Jayesh.

I think this can be a useful addition to Apache Kafka.
One potential issue is that you are getting all the information for
ZooKeeper. We already added a protocol that allows adding the
information to Kafka itself and are in the process of adding an admin
client (i.e. Java client, not CLI).
If you add this as planned, we'll need to re-work it to work with
Kafka directly instead of ZooKeeper once the admin client lands. If
you choose, you can wait for the admin client to arrive first, and
avoid the re-work.

Gwen

On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
 wrote:
> All,
> If there is no discussion, feedback or objection, is there any concern in 
> going to the next step of voting?
> Thanks,Jayesh
>   From: Jayesh Thakrar 
>  To: "dev@kafka.apache.org" 
>  Sent: Saturday, August 13, 2016 11:44 PM
>  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
> kafka-brokers.sh
>
> This is to start off a discussion on the above KIP at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
>  proposal is to fill the void of a command line tool/utility that can provide 
> information on the brokers in a Kafka cluster.
> The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
> page has the help documentation as well as the output from the command with 
> various options.Thank you,Jayesh Thakrar
>
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425587#comment-15425587
 ] 

Ismael Juma commented on KAFKA-4050:


It's clear that we should make this configurable. One question is whether we 
should also change the default. Are we sure that /dev/urandom was being used 
(and not /dev/random)? SecureRandom can use either of them and the blocking 
behaviour of /dev/random when there isn't enough entropy would have an even 
more severe impact.

> Allow configuration of the PRNG used for SSL
> 
>
> Key: KAFKA-4050
> URL: https://issues.apache.org/jira/browse/KAFKA-4050
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.0.1
>Reporter: Todd Palino
>Assignee: Todd Palino
>  Labels: security, ssl
>
> This change will make the pseudo-random number generator (PRNG) 
> implementation used by the SSLContext configurable. The configuration is not 
> required, and the default is to use whatever the default PRNG for the JDK/JRE 
> is. Providing a string, such as "SHA1PRNG", will cause that specific 
> SecureRandom implementation to get passed to the SSLContext.
> When enabling inter-broker SSL in our certification cluster, we observed 
> severe performance issues. For reference, this cluster can take up to 600 
> MB/sec of inbound produce traffic over SSL, with RF=2, before it gets close 
> to saturation, and the mirror maker normally produces about 400 MB/sec 
> (unless it is lagging). When we enabled inter-broker SSL, we saw persistent 
> replication problems in the cluster at any inbound rate of more than about 6 
> or 7 MB/sec per-broker. This was narrowed down to all the network threads 
> blocking on a single lock in the SecureRandom code.
> It turns out that the default PRNG implementation on Linux is NativePRNG. 
> This uses randomness from /dev/urandom (which, by itself, is a non-blocking 
> read) and mixes it with randomness from SHA1. The problem is that the entire 
> application shares a single SecureRandom instance, and NativePRNG has a 
> global lock within the implNextBytes method. Switching to another 
> implementation (SHA1PRNG, which has better performance characteristics and is 
> still considered secure) completely eliminated the bottleneck and allowed the 
> cluster to work properly at saturation.
> The SSLContext initialization has an optional argument to provide a 
> SecureRandom instance, which the code currently sets to null. This change 
> creates a new config to specify an implementation, and instantiates that and 
> passes it to SSLContext if provided. This will also let someone select a 
> stronger source of randomness (obviously at a performance cost) if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4060) Remove ZkClient dependency in Kafka Streams

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4060:
-
Assignee: Hojjat Jafarpour

> Remove ZkClient dependency in Kafka Streams
> ---
>
> Key: KAFKA-4060
> URL: https://issues.apache.org/jira/browse/KAFKA-4060
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Hojjat Jafarpour
>
> In Kafka Streams we need to dynamically create or update those internal 
> topics (i.e. repartition topics) upon rebalance, inside 
> {{InternalTopicManager}} which is triggered by {{StreamPartitionAssignor}}. 
> Currently we are using {{ZkClient}} to talk to ZK directly for such actions.
> With create and delete topics request merged in by [~granthenke] as part of 
> KIP-4, we should now be able to remove the ZkClient dependency and directly 
> use these requests.
> Related: 
> 1. KIP-4. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> 2. Consumer Reblance Protocol. 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1480

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3888: send consumer heartbeats from a background thread 
(KIP-62)

--
[...truncated 12002 lines...]

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStandbyReplicas PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithNewTasks STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithNewTasks PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldExposeHostStateToTopicPartitionsOnAssignment STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldExposeHostStateToTopicPartitionsOnAssignment PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStates STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithStates PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignBasic STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignBasic PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testOnAssignment STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testOnAssignment PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithInternalTopicThatsSourceIsAnotherInternalTopic STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithInternalTopicThatsSourceIsAnotherInternalTopic PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testSubscription PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithInternalTopics STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
testAssignWithInternalTopics PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullApplicationIdOnBuild STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullApplicationIdOnBuild PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testSourceTopics 
STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > testSourceTopics PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSink STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSink PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testNamedTopicMatchesAlreadyProvidedPattern STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testNamedTopicMatchesAlreadyProvidedPattern PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddSinkWithSameName STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 

[jira] [Commented] (KAFKA-3937) Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425523#comment-15425523
 ] 

ASF GitHub Bot commented on KAFKA-3937:
---

Github user wiyu closed the pull request at:

https://github.com/apache/kafka/pull/1743


> Kafka Clients Leak Native Memory For Longer Than Needed With Compressed 
> Messages
> 
>
> Key: KAFKA-3937
> URL: https://issues.apache.org/jira/browse/KAFKA-3937
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Linux, latest oracle java-8
>Reporter: Tom Crayford
>Priority: Minor
>
> In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that 
> brokers can crash when performing log recovery, as they leak native memory 
> whilst decompressing compressed segments, and that native memory isn't 
> cleaned up rapidly enough by garbage collection and finalizers. The work to 
> fix that in the brokers is taking part in 
> https://github.com/apache/kafka/pull/1598. As part of that PR, Ismael Juma 
> asked me to fix similar issues in the client. Rather than have one large PR 
> that fixes everything, I'd rather break this work up into seperate things, so 
> I'm filing this JIRA to track the followup work. I should get to a PR on this 
> at some point relatively soon, once the other PR has landed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1743: KAFKA-3937: Kafka Clients Leak Native Memory For L...

[wiyu]

Github user wiyu closed the pull request at:

https://github.com/apache/kafka/pull/1743


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4060) Remove ZkClient dependency in Kafka Streams

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4060:
-
Description: 
In Kafka Streams we need to dynamically create or update those internal topics 
(i.e. repartition topics) upon rebalance, inside {{InternalTopicManager}} which 
is triggered by {{StreamPartitionAssignor}}. Currently we are using 
{{ZkClient}} to talk to ZK directly for such actions.

With create and delete topics request merged in by [~granthenke] as part of 
KIP-4, we should now be able to remove the ZkClient dependency and directly use 
these requests.

Related: 
1. KIP-4. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

2. Consumer Reblance Protocol. 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal

  was:
In Kafka Streams we need to dynamically create or update those internal topics 
(i.e. repartition topics) upon rebalance, inside {InternalTopicManager} which 
is triggered by {StreamPartitionAssignor}. Currently we are using {ZkClient} to 
talk to ZK directly for such actions.

With create and delete topics request merged in by [~granthenke] as part of 
KIP-4, we should now be able to remove the ZkClient dependency and directly use 
these requests.

Related: 
1. KIP-4. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

2. Consumer Reblance Protocol. 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal


> Remove ZkClient dependency in Kafka Streams
> ---
>
> Key: KAFKA-4060
> URL: https://issues.apache.org/jira/browse/KAFKA-4060
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>
> In Kafka Streams we need to dynamically create or update those internal 
> topics (i.e. repartition topics) upon rebalance, inside 
> {{InternalTopicManager}} which is triggered by {{StreamPartitionAssignor}}. 
> Currently we are using {{ZkClient}} to talk to ZK directly for such actions.
> With create and delete topics request merged in by [~granthenke] as part of 
> KIP-4, we should now be able to remove the ZkClient dependency and directly 
> use these requests.
> Related: 
> 1. KIP-4. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> 2. Consumer Reblance Protocol. 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4060) Remove ZkClient dependency in Kafka Streams

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-4060:


 Summary: Remove ZkClient dependency in Kafka Streams
 Key: KAFKA-4060
 URL: https://issues.apache.org/jira/browse/KAFKA-4060
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


In Kafka Streams we need to dynamically create or update those internal topics 
(i.e. repartition topics) upon rebalance, inside {InternalTopicManager} which 
is triggered by {StreamPartitionAssignor}. Currently we are using {ZkClient} to 
talk to ZK directly for such actions.

With create and delete topics request merged in by [~granthenke] as part of 
KIP-4, we should now be able to remove the ZkClient dependency and directly use 
these requests.

Related: 
1. KIP-4. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

2. Consumer Reblance Protocol. 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4059) Documentation still refers to AsyncProducer and SyncProducer

[Andrew B (JIRA)]

Andrew B created KAFKA-4059:
---

 Summary: Documentation still refers to AsyncProducer and 
SyncProducer
 Key: KAFKA-4059
 URL: https://issues.apache.org/jira/browse/KAFKA-4059
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.10.0.1
Reporter: Andrew B


The 0.10 docs are still referring to AsyncProducer and SyncProducer.

See: https://github.com/apache/kafka/search?utf8=%E2%9C%93=AsyncProducer



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1530) howto update continuously

[Alexey Ozeritskiy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425454#comment-15425454
 ] 

Alexey Ozeritskiy commented on KAFKA-1530:
--

I think this ticket may be closed
unclean.leader.election.enable=false helps us
Also we've developed tool kafka-restarter that restarts kafka node by node and 
controls isr status.
And we've developed tool fix-isr that can fix isr after cluster power failure.

> howto update continuously
> -
>
> Key: KAFKA-1530
> URL: https://issues.apache.org/jira/browse/KAFKA-1530
> Project: Kafka
>  Issue Type: Wish
>Reporter: Stanislav Gilmulin
>Assignee: Guozhang Wang
>Priority: Minor
>  Labels: operating_manual, performance
>
> Hi,
>  
> Could I ask you a question about the Kafka update procedure?
> Is there a way to update software, which doesn't require service interruption 
> or lead to data losses?
> We can't stop message brokering during the update as we have a strict SLA.
>  
> Best regards
> Stanislav Gilmulin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #820

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3888: send consumer heartbeats from a background thread 
(KIP-62)

--
[...truncated 5136 lines...]

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs STARTED

kafka.api.PlaintextConsumerTest > testMaxPollIntervalMs PASSED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription STARTED

kafka.api.PlaintextConsumerTest > testSubsequentPatternSubscription PASSED

kafka.api.PlaintextConsumerTest > testAsyncCommit STARTED

kafka.api.PlaintextConsumerTest > testAsyncCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance STARTED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > testSeek STARTED

kafka.api.PlaintextConsumerTest > testSeek PASSED

kafka.api.PlaintextConsumerTest > testPositionAndCommit STARTED

kafka.api.PlaintextConsumerTest > testPositionAndCommit PASSED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic STARTED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose PASSED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge STARTED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose PASSED

kafka.api.PlaintextConsumerTest > testListTopics STARTED

kafka.api.PlaintextConsumerTest > testListTopics PASSED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testInterceptors STARTED

kafka.api.PlaintextConsumerTest > testInterceptors PASSED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription PASSED

kafka.api.PlaintextConsumerTest > testGroupConsumption STARTED

kafka.api.PlaintextConsumerTest > testGroupConsumption PASSED

Re: [DISCUSS] Time-based releases for Apache Kafka

[Vahid S Hashemian]

Time-based releases is a good idea and something that has proved to be 
working in a number of open source projects. One successful example is 
Node.js, that goes through two major releases a year. The interesting fact 
about the two releases is that only one (the even-number release) comes 
with a long term support (LTS) plan (30 months). More can be read here: 
https://github.com/nodejs/LTS. The odd-number releases still come with 
major changes and help build the ecosystem, but as far as LTS goes, there 
is only one per year. This LTS plan makes most enterprises want to stick 
to even-number releases, which is okay since frequent upgrades is not 
something they are normally interested in anyway.

There could be several minor releases (non-breaking) in between major 
releases. A major release contains all the features / bug fixes in the 
master branch a month before the release date, with the potential addition 
of (non-breaking) bug fixes until the release day. The deprecation cycle 
is one major release: any functionality that is decided to be removed is 
deprecated in the next major release, and removed in the major release 
after that.

Because of the success of LTS models in this and other open source 
projects, I would suggest implementing a formal LTS plan for Kafka too.

Regards,
--Vahid



From:   Gwen Shapira 
To: dev@kafka.apache.org
Date:   08/09/2016 04:49 PM
Subject:[DISCUSS] Time-based releases for Apache Kafka



Dear Kafka Developers and Users,

In the past, our releases have been quite unpredictable. We'll notice
that a large number of nice features made it in (or are close),
someone would suggest a release and we'd do it. This is fun, but makes
planning really hard - we saw it during the last release which we
decided to delay by a few weeks to allow more features to "land".

Many other communities have adopted time-based releases successfully
(Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
will make sense for the Apache Kafka community to try doing the same.

The benefits of this approach are:

1. A quicker feedback cycle and users can benefit from features
quicker (assuming for reasonably short time between releases - I was
thinking 4 months)

2. Predictability for contributors and users:
* Developers and reviewers can decide in advance what release they are
aiming for with specific features.
* If a feature misses a release we have a good idea of when it will show 
up.
* Users know when to expect their features

3. Transparency - There will be a published cut-off date (AKA feature
freeze) for the release and people will know about it in advance.
Hopefully this will remove the contention around which features make
it.

4. Quality - we've seen issues pop up in release candidates due to
last-minute features that didn't have proper time to bake in. More
time between feature freeze and release will let us test more,
document more and resolve more issues.

Since nothing is ever perfect, there will be some downsides:

1. Most notably, features that miss the feature-freeze date for a
release will have to wait few month for the next release. Features
will reach users faster overall as per benefit #1, but individual
features that just miss the cut will lose out

2. More releases a year mean that being a committer is more work -
release management is still some headache and we'll have more of
those. Hopefully we'll get better at it. Also, the committer list is
growing and hopefully it will be less than once-a-year effort for each
committer.

3. For users, figuring out which release to use and having frequent
new releases to upgrade to may be a bit confusing.

4. Frequent releases mean we need to do bugfix releases for older
branches. Right now we only do bugfix releases to latest release.

I think the benefits outweigh the drawbacks. Or at least suggest that
its worth trying - we can have another discussion in few releases to
see if we want to keep it that way or try something else.

My suggestion for the process:

1. We decide on a reasonable release cadence
2. We decide on release dates (even rough estimate such as "end of
February" or something) and work back feature freeze dates.
3. Committers volunteer to be "release managers" for specific
releases. We can coordinate on the list or on a wiki. If no committer
volunteers, we assume the community doesn't need a release and skip
it.
4. At the "feature freeze" date, the release manager announces the
contents of the release (which KIPs made it in on time), creates the
release branch and starts the release process as usual. From this
point onwards, only bug fixes should be double-committed to the
release branch while trunk can start collecting features for the
subsequent release.

Comments and improvements are appreciated.

Gwen Shapira
Former-release-manager

[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425206#comment-15425206
 ] 

ASF GitHub Bot commented on KAFKA-2170:
---

GitHub user soumyajit-sahu opened a pull request:

https://github.com/apache/kafka/pull/1757

KAFKA-2170: Fixes for Windows

This PR is to revive the older PR https://github.com/apache/kafka/pull/154 
made by @mpoindexter (Mike Poindexter). I have replicated his work over the new 
code base.
This lets 6 LogTest unit tests to pass on Windows.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Microsoft/kafka windowsFix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1757


commit 9db7c0493395d9426d13a0697ffe7146a724f169
Author: Som Sahu 
Date:   2016-08-17T17:08:43Z

Windows Filesystem related fix as per to Java RandomAccessFile API bug
6357433

commit 5da9c9e2e1f3edc956288a032652750a37899659
Author: Som Sahu 
Date:   2016-08-17T19:04:40Z

code cleaning - removing a commented line




> 10 LogTest cases failed for  file.renameTo failed under windows
> ---
>
> Key: KAFKA-2170
> URL: https://issues.apache.org/jira/browse/KAFKA-2170
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.1.0
> Environment: Windows
>Reporter: Honghai Chen
>Assignee: Jay Kreps
>
> get latest code from trunk, then run test 
> gradlew  -i core:test --tests kafka.log.LogTest
> Got 10 cases failed for same reason:
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 0
>   at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259)
>   at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756)
>   at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at kafka.log.Log.deleteOldSegments(Log.scala:514)
>   at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> 

[GitHub] kafka pull request #1757: KAFKA-2170: Fixes for Windows

[soumyajit-sahu]

GitHub user soumyajit-sahu opened a pull request:

https://github.com/apache/kafka/pull/1757

KAFKA-2170: Fixes for Windows

This PR is to revive the older PR https://github.com/apache/kafka/pull/154 
made by @mpoindexter (Mike Poindexter). I have replicated his work over the new 
code base.
This lets 6 LogTest unit tests to pass on Windows.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Microsoft/kafka windowsFix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1757


commit 9db7c0493395d9426d13a0697ffe7146a724f169
Author: Som Sahu 
Date:   2016-08-17T17:08:43Z

Windows Filesystem related fix as per to Java RandomAccessFile API bug
6357433

commit 5da9c9e2e1f3edc956288a032652750a37899659
Author: Som Sahu 
Date:   2016-08-17T19:04:40Z

code cleaning - removing a commented line




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4058) Failure in org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425204#comment-15425204
 ] 

ASF GitHub Bot commented on KAFKA-4058:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1756

KAFKA-4058: Failure in 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset

 - added retry on ResetTool error

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4058-reset-tool-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1756.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1756


commit 154ad8160a6b9c70e5e788a3081b81448e39cfe1
Author: Matthias J. Sax 
Date:   2016-08-17T19:12:03Z

KAFKA-4058: Failure in 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset
 - added retry on ResetTool error




> Failure in 
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset
> --
>
> Key: KAFKA-4058
> URL: https://issues.apache.org/jira/browse/KAFKA-4058
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Matthias J. Sax
>  Labels: test
>
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.kafka.streams.integration.ResetIntegrationTest.cleanGlobal(ResetIntegrationTest.java:225)
>   at 
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset(ResetIntegrationTest.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> 

[GitHub] kafka pull request #1756: KAFKA-4058: Failure in org.apache.kafka.streams.in...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1756

KAFKA-4058: Failure in 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset

 - added retry on ResetTool error

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4058-reset-tool-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1756.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1756


commit 154ad8160a6b9c70e5e788a3081b81448e39cfe1
Author: Matthias J. Sax 
Date:   2016-08-17T19:12:03Z

KAFKA-4058: Failure in 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset
 - added retry on ResetTool error




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1732: MINOR: Clarification in producer config documentat...

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1732

MINOR: Clarification in producer config documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka 
minor/clarify_producer_config_documentation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1732.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1732


commit 32e362be3ce2328cfa8d814848fd295766829ac6
Author: Vahid Hashemian 
Date:   2016-08-13T00:12:52Z

MINOR: Clarification in producer config documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1732: MINOR: Clarification in producer config documentat...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1732


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-74: Add Fetch Response Size Limit in Bytes

[Jason Gustafson]

Hi Andrey,

Thanks for picking this up and apologies for the late comment.

One thing worth mentioning is that the consumer actually sends multiple
parallel fetch requests, one for each broker that is hosting some of the
assigned partitions. Unless you were planning to modify this behavior, this
KIP actually changes the maximum memory used by the consumer from

max.partition.fetch.bytes * num_partitions

to

fetch.response.max.bytes * num_brokers

I guess it's really the minimum of the two values since
max.partition.fetch.bytes is still supported. I think this is still a very
helpful feature, but it's probably worth calling this out in the KIP.

Also, one question on naming: would it make sense to change
"fetch.response.max.bytes" to "max.fetch.bytes"? Seems to fit nicer with
"max.partition.fetch.bytes".


Thanks,
Jason



On Tue, Aug 16, 2016 at 1:22 PM, Ismael Juma  wrote:

> Hi Andrey,
>
> Can you please start a new thread for the vote? Gmail is showing your vote
> message in the discuss thread.
>
> Ismael
>
> On Tue, Aug 16, 2016 at 9:15 PM, Andrey L. Neporada <
> anepor...@yandex-team.ru> wrote:
>
> > Hi!
> >
> > I would like to initiate the voting process for KIP-74:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 74%3A+Add+Fetch+Response+Size+Limit+in+Bytes
> >
> > Thanks,
> > Andrey.
>

[jira] [Commented] (KAFKA-3888) Allow consumer to send heartbeats in background thread (KIP-62)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425167#comment-15425167
 ] 

ASF GitHub Bot commented on KAFKA-3888:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1627


> Allow consumer to send heartbeats in background thread (KIP-62)
> ---
>
> Key: KAFKA-3888
> URL: https://issues.apache.org/jira/browse/KAFKA-3888
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.10.1.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0
>
>
> This ticket covers the implementation of KIP-62 as documented here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3888) Allow consumer to send heartbeats in background thread (KIP-62)

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3888.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1627
[https://github.com/apache/kafka/pull/1627]

> Allow consumer to send heartbeats in background thread (KIP-62)
> ---
>
> Key: KAFKA-3888
> URL: https://issues.apache.org/jira/browse/KAFKA-3888
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.10.1.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.0
>
>
> This ticket covers the implementation of KIP-62 as documented here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1627: KAFKA-3888: send consumer heartbeats from a backgr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1627


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4058) Failure in org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-4058:


 Summary: Failure in 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset
 Key: KAFKA-4058
 URL: https://issues.apache.org/jira/browse/KAFKA-4058
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Guozhang Wang
Assignee: Matthias J. Sax


{code}
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.kafka.streams.integration.ResetIntegrationTest.cleanGlobal(ResetIntegrationTest.java:225)
at 
org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset(ResetIntegrationTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at 
org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}

Example: 

[jira] [Commented] (KAFKA-4039) Exit Strategy: using exceptions instead of inline invocation of exit/halt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425123#comment-15425123
 ] 

ASF GitHub Bot commented on KAFKA-4039:
---

GitHub user maysamyabandeh opened a pull request:

https://github.com/apache/kafka/pull/1755

KAFKA-4039: delay invocation of System.exit via FatalExitException

@resetius would be great if you can confirm that the deadlock no longer 
manifests with the path.
Thanks

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maysamyabandeh/kafka KAFKA-4039

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1755.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1755


commit d902c2bc81c4799ad1eb7bc7ec50e67d04d1c9c2
Author: Maysam Yabandeh 
Date:   2016-08-17T18:21:07Z

KAFKA-4039: delay invocation of System.exit via FatalExitException




> Exit Strategy: using exceptions instead of inline invocation of exit/halt
> -
>
> Key: KAFKA-4039
> URL: https://issues.apache.org/jira/browse/KAFKA-4039
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Maysam Yabandeh
>
> The current practice is to directly invoke halt/exit right after the line 
> that intends to terminate the execution. In the case of System.exit this 
> could cause deadlocks if the thread invoking System.exit is holding  a lock 
> that will be requested by the shutdown hook threads that will be started by 
> System.exit. An example is reported by [~aozeritsky] in KAFKA-3924. This 
> would also makes testing more difficult as it would require mocking static 
> methods of System and Runtime classes, which is not natively supported in 
> Java.
> One alternative suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-3924?focusedCommentId=15420269=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15420269]
>  would be to throw some dedicated exceptions that will eventually invoke 
> exit/halt:
> {quote} it would be great to move away from executing `System.exit` inline in 
> favour of throwing an exception (two examples, but maybe we can find better 
> names: FatalExitException and FatalHaltException) that is caught by some 
> central code that then does the `System.exit` or `Runtime.getRuntime.halt`. 
> This helps in a couple of ways:
> (1) Avoids issues with locks being held as in this issue
> (2) It makes it possible to abstract the action, which is very useful in 
> tests. At the moment, we can't easily test for these conditions as they cause 
> the whole test harness to exit. Worse, these conditions are sometimes 
> triggered in the tests and it's unclear why.
> (3) We can have more consistent logging around these actions and possibly 
> extended logging for tests
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1755: KAFKA-4039: delay invocation of System.exit via Fa...

[maysamyabandeh]

GitHub user maysamyabandeh opened a pull request:

https://github.com/apache/kafka/pull/1755

KAFKA-4039: delay invocation of System.exit via FatalExitException

@resetius would be great if you can confirm that the deadlock no longer 
manifests with the path.
Thanks

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maysamyabandeh/kafka KAFKA-4039

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1755.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1755


commit d902c2bc81c4799ad1eb7bc7ec50e67d04d1c9c2
Author: Maysam Yabandeh 
Date:   2016-08-17T18:21:07Z

KAFKA-4039: delay invocation of System.exit via FatalExitException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1479

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-4044; log actual socket send/receive buffer size after connecting

--
[...truncated 12171 lines...]
org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testRestoreWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testRestore 
STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testRestore 
PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRange STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRange PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRangeWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores PASSED

org.apache.kafka.streams.state.internals.WindowStoreUtilsTest > 
testSerialization STARTED

org.apache.kafka.streams.state.internals.WindowStoreUtilsTest > 
testSerialization PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldNotReturnKVStoreWhenIsWindowStore STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldNotReturnKVStoreWhenIsWindowStore PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnNullIfKVStoreDoesntExist STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnNullIfKVStoreDoesntExist PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnNullIfWindowStoreDoesntExist STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnNullIfWindowStoreDoesntExist PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldNotReturnWindowStoreWhenIsKVStore STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldNotReturnWindowStoreWhenIsKVStore PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnKVStoreWhenItExists STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnKVStoreWhenItExists PASSED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnWindowStoreWhenItExists STARTED

org.apache.kafka.streams.state.internals.QueryableStoreProviderTest > 
shouldReturnWindowStoreWhenItExists PASSED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
STARTED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetch STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetch PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetchBefore STARTED

Build failed in Jenkins: kafka-trunk-jdk8 #819

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-4044; log actual socket send/receive buffer size after connecting

--
[...truncated 12204 lines...]
org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
testTracking PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424982#comment-15424982
 ] 

Mayuresh Gharat commented on KAFKA-4050:


Just a heads up, this has also been seen as an issue in other systems, for 
example :
https://issues.jenkins-ci.org/browse/JENKINS-20108



> Allow configuration of the PRNG used for SSL
> 
>
> Key: KAFKA-4050
> URL: https://issues.apache.org/jira/browse/KAFKA-4050
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.0.1
>Reporter: Todd Palino
>Assignee: Todd Palino
>  Labels: security, ssl
>
> This change will make the pseudo-random number generator (PRNG) 
> implementation used by the SSLContext configurable. The configuration is not 
> required, and the default is to use whatever the default PRNG for the JDK/JRE 
> is. Providing a string, such as "SHA1PRNG", will cause that specific 
> SecureRandom implementation to get passed to the SSLContext.
> When enabling inter-broker SSL in our certification cluster, we observed 
> severe performance issues. For reference, this cluster can take up to 600 
> MB/sec of inbound produce traffic over SSL, with RF=2, before it gets close 
> to saturation, and the mirror maker normally produces about 400 MB/sec 
> (unless it is lagging). When we enabled inter-broker SSL, we saw persistent 
> replication problems in the cluster at any inbound rate of more than about 6 
> or 7 MB/sec per-broker. This was narrowed down to all the network threads 
> blocking on a single lock in the SecureRandom code.
> It turns out that the default PRNG implementation on Linux is NativePRNG. 
> This uses randomness from /dev/urandom (which, by itself, is a non-blocking 
> read) and mixes it with randomness from SHA1. The problem is that the entire 
> application shares a single SecureRandom instance, and NativePRNG has a 
> global lock within the implNextBytes method. Switching to another 
> implementation (SHA1PRNG, which has better performance characteristics and is 
> still considered secure) completely eliminated the bottleneck and allowed the 
> cluster to work properly at saturation.
> The SSLContext initialization has an optional argument to provide a 
> SecureRandom instance, which the code currently sets to null. This change 
> creates a new config to specify an implementation, and instantiates that and 
> passes it to SSLContext if provided. This will also let someone select a 
> stronger source of randomness (obviously at a performance cost) if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kafka broker is dropping the messages after acknowledging librdkafka

[Mazhar Shaikh]

Hi Jun,

I'm using default configuration (ack=1),
changing it t0 all or 2 will not help, as the producer queue will be
exhausted is any kafka broker goes down for long time.


Thanks.

Regards,
Mazhar Shaikh.


On Wed, Aug 17, 2016 at 8:11 PM, Jun Rao  wrote:

> Are you using acks=1 or acks=all in the producer? Only the latter
> guarantees acked messages won't be lost after leader failure.
>
> Thanks,
>
> Jun
>
> On Wed, Aug 10, 2016 at 11:41 PM, Mazhar Shaikh <
> mazhar.shaikh...@gmail.com>
> wrote:
>
> > Hi Kafka Team,
> >
> > I'm using kafka (kafka_2.11-0.9.0.1) with librdkafka (0.8.1) API for
> > producer
> > During a run of 2hrs, I notice the total number of messaged ack'd by
> > librdkafka delivery report is greater than the maxoffset of a partition
> in
> > kafka broker.
> > I'm running kafka broker with replication factor of 2.
> >
> > Here, message has been lost between librdkafka - kafka broker.
> >
> > As librdkafka is providing success delivery report for all the messages.
> >
> > Looks like kafka broker is dropping the messages after acknowledging
> > librdkafka.
> >
> > Requesting you help in solving this issue.
> >
> > Thank you.
> >
> >
> > Regards
> > Mazhar Shaikh
> >
>

[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

[Oleg Gorobets (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424715#comment-15424715
 ] 

Oleg Gorobets commented on KAFKA-3172:
--

I experience the same problem with new consumer API (kafka_2.10 0.9.0.1) when 
number of messages gets higher (around 2000/sec). The thread is just blocked on 
poll(), no error messages whatsoever. I didn't have this issue with old (scala) 
API.

ConsumerRecords records = consumer.poll(200);

"CoreKafkaConsumer" prio=10 tid=0x7fb3e87da000 nid=0x2666 runnable 
[0x7fb3dc56a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000707d943a0> (a sun.nio.ch.Util$2)
- locked <0x000707d94390> (a java.util.Collections$UnmodifiableSet)
- locked <0x000707d93878> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.kafka.common.network.Selector.select(Selector.java:425)
at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256)
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853)

> Consumer threads stay in 'Watiting' status and are blocked at consumer poll 
> method
> --
>
> Key: KAFKA-3172
> URL: https://issues.apache.org/jira/browse/KAFKA-3172
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: linux
>Reporter: Dany Benjamin
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.9.0.0
>
> Attachments: jmx_info.png, jstack.png, lagSample.png
>
>
> When running multiple consumers on same group (400 - for a 400 partitioned 
> topic), the application for all threads blocks at consumer.poll() method. The 
> timeout parameter sent in is 1.
> Stack dump:
> "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 
> waiting on condition [0x7f63df8f7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000605812710> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 
> tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0006058283e8> (a sun.nio.ch.Util$2)
> - locked <0x0006058283d8> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:425)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at 
> 

[jira] [Commented] (KAFKA-4044) log actual socket send/receive buffer size after connecting in Selector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424676#comment-15424676
 ] 

ASF GitHub Bot commented on KAFKA-4044:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1750


> log actual socket send/receive buffer size after connecting in Selector
> ---
>
> Key: KAFKA-4044
> URL: https://issues.apache.org/jira/browse/KAFKA-4044
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Manikumar Reddy
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> In BlockingChannel, we had the following code to log the actual socket buffer 
> size when the socket connection is established. This can be helpful when 
> tuning the socket buffer size for long latency network. It would be useful to 
> add that in Selector.pollSelectionKeys when the socket is connected.
> val msg = "Created socket with SO_TIMEOUT = %d (requested %d), 
> SO_RCVBUF = %d (requested %d), SO_SNDBUF = %d (requested %d), 
> connectTimeoutMs = %d."
> debug(msg.format(channel.socket.getSoTimeout,
>  readTimeoutMs,
>  channel.socket.getReceiveBufferSize, 
>  readBufferSize,
>  channel.socket.getSendBufferSize,
>  writeBufferSize,
>  connectTimeoutMs))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1750: KAFKA-4044: log actual socket send/receive buffer ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1750


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4044) log actual socket send/receive buffer size after connecting in Selector

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-4044.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1750
[https://github.com/apache/kafka/pull/1750]

> log actual socket send/receive buffer size after connecting in Selector
> ---
>
> Key: KAFKA-4044
> URL: https://issues.apache.org/jira/browse/KAFKA-4044
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Manikumar Reddy
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> In BlockingChannel, we had the following code to log the actual socket buffer 
> size when the socket connection is established. This can be helpful when 
> tuning the socket buffer size for long latency network. It would be useful to 
> add that in Selector.pollSelectionKeys when the socket is connected.
> val msg = "Created socket with SO_TIMEOUT = %d (requested %d), 
> SO_RCVBUF = %d (requested %d), SO_SNDBUF = %d (requested %d), 
> connectTimeoutMs = %d."
> debug(msg.format(channel.socket.getSoTimeout,
>  readTimeoutMs,
>  channel.socket.getReceiveBufferSize, 
>  readBufferSize,
>  channel.socket.getSendBufferSize,
>  writeBufferSize,
>  connectTimeoutMs))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4056) Kafka logs values of sensitive configs like passwords

[Mickael Maison (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison reassigned KAFKA-4056:
-

Assignee: Mickael Maison

> Kafka logs values of sensitive configs like passwords
> -
>
> Key: KAFKA-4056
> URL: https://issues.apache.org/jira/browse/KAFKA-4056
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: jaikiran pai
>Assignee: Mickael Maison
>
> From the mail discussion here: 
> https://www.mail-archive.com/dev@kafka.apache.org/msg55012.html
> {quote}
> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers and 
> producers. In one of our consumers, our consumer config had a SSL specific 
> property which ended up being used against a non-SSL Kafka broker port. As a 
> result, the logs ended up seeing messages like:
> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration 
> *ssl.truststore.password = foobar* was supplied but isn't a known config.
> The log message is fine and makes sense, but can Kafka please not log the 
> values of the properties and instead just include the config name which it 
> considers as unknown? That way it won't ended up logging these potentially 
> sensitive values. I understand that only those with access to these log files 
> can end up seeing these values but even then some of our internal processes 
> forbid logging such sensitive information to the logs. This log message will 
> still end up being useful if only the config name is logged without the 
> value. 
> {quote}
> Apparently (as noted in that thread), there's already code in the Kafka 
> library which masks sensitive values like passwords, but it looks like 
> there's a bug where it unintentionally logs these raw values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4057) Allow to specify the request version and replica ID in kafka.javaapi.FetchRequest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424659#comment-15424659
 ] 

ASF GitHub Bot commented on KAFKA-4057:
---

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/1754

KAFKA-4057: Allow to specify the request version and replica ID in ka…

…fka.javaapi.FetchRequest

- Added new arguments for versionId and replicaId in the constructor 
instead of using hardcoded values

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka KAFKA-4057

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1754






> Allow to specify the request version and replica ID in 
> kafka.javaapi.FetchRequest
> -
>
> Key: KAFKA-4057
> URL: https://issues.apache.org/jira/browse/KAFKA-4057
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Minor
>
> kafka.javaapi.FetchRequest is used to create FetchRequests (as they are still 
> using the old type) from Java. However it doesn't allow to specify the 
> request version nor the replica ID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1754: KAFKA-4057: Allow to specify the request version a...

[mimaison]

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/1754

KAFKA-4057: Allow to specify the request version and replica ID in ka…

…fka.javaapi.FetchRequest

- Added new arguments for versionId and replicaId in the constructor 
instead of using hardcoded values

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka KAFKA-4057

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1754






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: kafka broker is dropping the messages after acknowledging librdkafka

[Jun Rao]

Are you using acks=1 or acks=all in the producer? Only the latter
guarantees acked messages won't be lost after leader failure.

Thanks,

Jun

On Wed, Aug 10, 2016 at 11:41 PM, Mazhar Shaikh 
wrote:

> Hi Kafka Team,
>
> I'm using kafka (kafka_2.11-0.9.0.1) with librdkafka (0.8.1) API for
> producer
> During a run of 2hrs, I notice the total number of messaged ack'd by
> librdkafka delivery report is greater than the maxoffset of a partition in
> kafka broker.
> I'm running kafka broker with replication factor of 2.
>
> Here, message has been lost between librdkafka - kafka broker.
>
> As librdkafka is providing success delivery report for all the messages.
>
> Looks like kafka broker is dropping the messages after acknowledging
> librdkafka.
>
> Requesting you help in solving this issue.
>
> Thank you.
>
>
> Regards
> Mazhar Shaikh
>

[jira] [Commented] (KAFKA-2544) Replication tools wiki page needs to be updated

[Lauren Moos (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424617#comment-15424617
 ] 

Lauren Moos commented on KAFKA-2544:


I can take this if someone can add me to the contributors list 

> Replication tools wiki page needs to be updated
> ---
>
> Key: KAFKA-2544
> URL: https://issues.apache.org/jira/browse/KAFKA-2544
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.1
>Reporter: Stevo Slavic
>Priority: Minor
>  Labels: documentation, newbie
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools is 
> outdated, mentions tools which have been heavily refactored or replaced by 
> other tools, e.g. add partition tool, list/create topics tools, etc.
> Please have the replication tools wiki page updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4057) Allow to specify the request version and replica ID in kafka.javaapi.FetchRequest

[Mickael Maison (JIRA)]

Mickael Maison created KAFKA-4057:
-

 Summary: Allow to specify the request version and replica ID in 
kafka.javaapi.FetchRequest
 Key: KAFKA-4057
 URL: https://issues.apache.org/jira/browse/KAFKA-4057
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.10.0.1, 0.10.0.0
Reporter: Mickael Maison
Assignee: Mickael Maison
Priority: Minor


kafka.javaapi.FetchRequest is used to create FetchRequests (as they are still 
using the old type) from Java. However it doesn't allow to specify the request 
version nor the replica ID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3492) support quota based on authenticated user name

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-3492:
--
Fix Version/s: 0.10.1.0
   Status: Patch Available  (was: Open)

> support quota based on authenticated user name
> --
>
> Key: KAFKA-3492
> URL: https://issues.apache.org/jira/browse/KAFKA-3492
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> Currently, quota is based on the client.id set in the client configuration, 
> which can be changed easily. Ideally, quota should be set on the 
> authenticated user name. We will need to have a KIP proposal/discussion on 
> this first.
> Details are in KIP-55: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4019) LogCleaner should grow read/write buffer to max message size for the topic

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram reassigned KAFKA-4019:
-

Assignee: Rajini Sivaram

> LogCleaner should grow read/write buffer to max message size for the topic
> --
>
> Key: KAFKA-4019
> URL: https://issues.apache.org/jira/browse/KAFKA-4019
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
>
> Currently, the LogCleaner.growBuffers() only grows the buffer up to the 
> default max message size. However, since the max message size can be 
> customized at the topic level, LogCleaner should allow the buffer to grow up 
> to the max message allowed by the topic. Otherwise, the cleaner will get 
> stuck on a large message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Do not log value of configs that Kafka doesn't recognize

[Jaikiran Pai]

Created https://issues.apache.org/jira/browse/KAFKA-4056.

-Jaikiran

On Wednesday 17 August 2016 06:28 PM, Ismael Juma wrote:

Yes, please file a JIRA.

Thanks,
Ismael

On Wed, Aug 17, 2016 at 1:46 PM, Jaikiran Pai 
wrote:


Thanks for the inputs.

I think it's fine if Kafka selectively logs values for certain configs and
masks others (the passwords). So from what I understand, this looks more
like a bug where it unintentionally ended up logging these values. Do you
want me to file a JIRA for this?

-Jaikiran



On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:


Note that we already mask password related config values[1]. The issue is
that it only happens after parsing, and the output of `logUnused` is based
on the `originals` map, which has the values before parsing:

public void logUnused() {
  for (String key : unused())
  log.warn("The configuration {} = {} was supplied but isn't a
known config.", key, this.originals.get(key));
}

Ismael

[1]
https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b
4acfd93d9581dabd7c1

On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <
manikumar.re...@gmail.com


wrote:
During server/client startup,  we are logging all the supplied configs.
May
be we can just mask
the password related config values for both valid/invalid configs.

On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai 
wrote:

Any opinion about this proposed change?

-Jaikiran

On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:

We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers

and producers. In one of our consumers, our consumer config had a SSL
specific property which ended up being used against a non-SSL Kafka


broker
port. As a result, the logs ended up seeing messages like:

17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration
*ssl.truststore.password = foobar* was supplied but isn't a known


config.
The log message is fine and makes sense, but can Kafka please not log
the
values of the properties and instead just include the config name which
it
considers as unknown? That way it won't ended up logging these
potentially
sensitive values. I understand that only those with access to these log

files can end up seeing these values but even then some of our internal
processes forbid logging such sensitive information to the logs. This


log
message will still end up being useful if only the config name is logged

without the value.

Can I add this as a JIRA and provide a patch?

-Jaikiran

[jira] [Resolved] (KAFKA-4054) Quota related metrics and sensors are never deleted

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-4054.
---
Resolution: Not A Problem

As pointed out by [~ijuma] in KAFKA-3980, metrics and sensors are expired.

> Quota related metrics and sensors are never deleted
> ---
>
> Key: KAFKA-4054
> URL: https://issues.apache.org/jira/browse/KAFKA-4054
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Rajini Sivaram
>
> Metrics and sensors used for quotas are never deleted. If random client-ids 
> are used by clients, this could result in a lot of unused metrics and sensors 
> in the broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4056) Kafka logs values of sensitive configs like passwords

[jaikiran pai (JIRA)]

jaikiran pai created KAFKA-4056:
---

 Summary: Kafka logs values of sensitive configs like passwords
 Key: KAFKA-4056
 URL: https://issues.apache.org/jira/browse/KAFKA-4056
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.1
Reporter: jaikiran pai


>From the mail discussion here: 
>https://www.mail-archive.com/dev@kafka.apache.org/msg55012.html

{quote}
We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers and 
producers. In one of our consumers, our consumer config had a SSL specific 
property which ended up being used against a non-SSL Kafka broker port. As a 
result, the logs ended up seeing messages like:


17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration 
*ssl.truststore.password = foobar* was supplied but isn't a known config.


The log message is fine and makes sense, but can Kafka please not log the 
values of the properties and instead just include the config name which it 
considers as unknown? That way it won't ended up logging these potentially 
sensitive values. I understand that only those with access to these log files 
can end up seeing these values but even then some of our internal processes 
forbid logging such sensitive information to the logs. This log message will 
still end up being useful if only the config name is logged without the value. 
{quote}

Apparently (as noted in that thread), there's already code in the Kafka library 
which masks sensitive values like passwords, but it looks like there's a bug 
where it unintentionally logs these raw values.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424492#comment-15424492
 ] 

Rajini Sivaram commented on KAFKA-3980:
---

[~ijuma] Thank you, I will close KAFKA-4054 since the expiry addresses the 
issue I was worried about. Perhaps this JIRA needs more investigation.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Time-based releases for Apache Kafka

[Mickael Maison]

+1

Having better predictability when features will land is a huge benefit.

On Tue, Aug 16, 2016 at 5:34 PM, Jim Jagielski  wrote:
> I'm following along on the thread so for sure! :)
>
>> On Aug 16, 2016, at 12:19 PM, Gwen Shapira  wrote:
>>
>> Absolutely!
>>
>> If you have any concrete suggestions for steps we can take to improve
>> the process, this will be most awesome. We'd love to learn from your
>> long experience in Apache :)
>>
>> Gwen
>>
>> On Tue, Aug 16, 2016 at 6:59 AM, Jim Jagielski  wrote:
>>> By being aware of the potential issues, it's easier to address
>>> them at the start, and to create a process which does what
>>> it can to "ensure" the problems don't pop up :)
>>>
 On Aug 16, 2016, at 9:48 AM, Ismael Juma  wrote:

 Hi Jim,

 Thanks for your feedback. We value the community and we definitely want
 Kafka to remain a fun and friendly place to participate. Under this
 proposal, volunteers will still be able to do the work when they can. The
 benefit is that it is likely to reach users faster since the next release
 is never far away.

 Ismael

 On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:

> The idea of time-based releases make sense. The issue is
> when they become the tail wagging the dog.
>
> Recall that all developers and contributors are assumed to
> be doing this because they are personally invested in the
> project. Their is also the assumption that, as such, they
> are volunteers and do the work "when they can". And finally,
> there is the fact that working on Apache projects should be
> FUN. It should be someplace where you aren't beholden to,
> or under, some artificial schedule.
>
> If time-based releases are put in place, and held to under
> unforgiving standards, all the above are put at risk. And
> when that happens it puts the project and the community at
> risk as well.
>
> So having a set schedule is fine... it's how "we" do it that
> is key.
>
>>>
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>

Re: Do not log value of configs that Kafka doesn't recognize

[Ismael Juma]

Yes, please file a JIRA.

Thanks,
Ismael

On Wed, Aug 17, 2016 at 1:46 PM, Jaikiran Pai 
wrote:

> Thanks for the inputs.
>
> I think it's fine if Kafka selectively logs values for certain configs and
> masks others (the passwords). So from what I understand, this looks more
> like a bug where it unintentionally ended up logging these values. Do you
> want me to file a JIRA for this?
>
> -Jaikiran
>
>
>
> On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:
>
>> Note that we already mask password related config values[1]. The issue is
>> that it only happens after parsing, and the output of `logUnused` is based
>> on the `originals` map, which has the values before parsing:
>>
>> public void logUnused() {
>>  for (String key : unused())
>>  log.warn("The configuration {} = {} was supplied but isn't a
>> known config.", key, this.originals.get(key));
>> }
>>
>> Ismael
>>
>> [1]
>> https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b
>> 4acfd93d9581dabd7c1
>>
>> On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <
>> manikumar.re...@gmail.com
>>
>>> wrote:
>>> During server/client startup,  we are logging all the supplied configs.
>>> May
>>> be we can just mask
>>> the password related config values for both valid/invalid configs.
>>>
>>> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai 
>>> wrote:
>>>
>>> Any opinion about this proposed change?

 -Jaikiran

 On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:

 We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
> and producers. In one of our consumers, our consumer config had a SSL
> specific property which ended up being used against a non-SSL Kafka
>
 broker
>>>
 port. As a result, the logs ended up seeing messages like:
>
> 17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration
> *ssl.truststore.password = foobar* was supplied but isn't a known
>
 config.
>>>
 The log message is fine and makes sense, but can Kafka please not log
>
 the
>>>
 values of the properties and instead just include the config name which
>
 it
>>>
 considers as unknown? That way it won't ended up logging these
>
 potentially
>>>
 sensitive values. I understand that only those with access to these log
> files can end up seeing these values but even then some of our internal
> processes forbid logging such sensitive information to the logs. This
>
 log
>>>
 message will still end up being useful if only the config name is logged
> without the value.
>
> Can I add this as a JIRA and provide a patch?
>
> -Jaikiran
>
>

>

Re: Do not log value of configs that Kafka doesn't recognize

[Jaikiran Pai]

Thanks for the inputs.

I think it's fine if Kafka selectively logs values for certain configs 
and masks others (the passwords). So from what I understand, this looks 
more like a bug where it unintentionally ended up logging these values. 
Do you want me to file a JIRA for this?


-Jaikiran


On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:

Note that we already mask password related config values[1]. The issue is
that it only happens after parsing, and the output of `logUnused` is based
on the `originals` map, which has the values before parsing:

public void logUnused() {
 for (String key : unused())
 log.warn("The configuration {} = {} was supplied but isn't a
known config.", key, this.originals.get(key));
}

Ismael

[1]
https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b4acfd93d9581dabd7c1

On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy 
wrote:


Any opinion about this proposed change?

-Jaikiran

On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:


We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
and producers. In one of our consumers, our consumer config had a SSL
specific property which ended up being used against a non-SSL Kafka

broker

port. As a result, the logs ended up seeing messages like:

17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration
*ssl.truststore.password = foobar* was supplied but isn't a known

config.

The log message is fine and makes sense, but can Kafka please not log

the

values of the properties and instead just include the config name which

it

considers as unknown? That way it won't ended up logging these

potentially

sensitive values. I understand that only those with access to these log
files can end up seeing these values but even then some of our internal
processes forbid logging such sensitive information to the logs. This

log

message will still end up being useful if only the config name is logged
without the value.

Can I add this as a JIRA and provide a patch?

-Jaikiran

[jira] [Commented] (KAFKA-3492) support quota based on authenticated user name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424443#comment-15424443
 ] 

ASF GitHub Bot commented on KAFKA-3492:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1753

KAFKA-3492: Secure quotas for authenticated users

Implementation and tests for secure quotas at  and  
levels as described in KIP-55. Also adds dynamic default quotas for 
,  and . For each client connection, the most 
specific quota matching the connection is used, with user quota taking 
precedence over client-id quota.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3492

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1753.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1753






> support quota based on authenticated user name
> --
>
> Key: KAFKA-3492
> URL: https://issues.apache.org/jira/browse/KAFKA-3492
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
>
> Currently, quota is based on the client.id set in the client configuration, 
> which can be changed easily. Ideally, quota should be set on the 
> authenticated user name. We will need to have a KIP proposal/discussion on 
> this first.
> Details are in KIP-55: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1753: KAFKA-3492: Secure quotas for authenticated users

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1753

KAFKA-3492: Secure quotas for authenticated users

Implementation and tests for secure quotas at  and  
levels as described in KIP-55. Also adds dynamic default quotas for 
,  and . For each client connection, the most 
specific quota matching the connection is used, with user quota taking 
precedence over client-id quota.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3492

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1753.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1753






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424436#comment-15424436
 ] 

Ismael Juma commented on KAFKA-3980:


[~rsivaram], did you see my comment above? We have code to expire client quota 
metrics in the broker. See `Metrics.metricsScheduler`. 

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Do not log value of configs that Kafka doesn't recognize

[Ismael Juma]

Note that we already mask password related config values[1]. The issue is
that it only happens after parsing, and the output of `logUnused` is based
on the `originals` map, which has the values before parsing:

public void logUnused() {
for (String key : unused())
log.warn("The configuration {} = {} was supplied but isn't a
known config.", key, this.originals.get(key));
}

Ismael

[1]
https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b4acfd93d9581dabd7c1

On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy  wrote:

> During server/client startup,  we are logging all the supplied configs. May
> be we can just mask
> the password related config values for both valid/invalid configs.
>
> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai 
> wrote:
>
> > Any opinion about this proposed change?
> >
> > -Jaikiran
> >
> > On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
> >
> >> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
> >> and producers. In one of our consumers, our consumer config had a SSL
> >> specific property which ended up being used against a non-SSL Kafka
> broker
> >> port. As a result, the logs ended up seeing messages like:
> >>
> >> 17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration
> >> *ssl.truststore.password = foobar* was supplied but isn't a known
> config.
> >>
> >> The log message is fine and makes sense, but can Kafka please not log
> the
> >> values of the properties and instead just include the config name which
> it
> >> considers as unknown? That way it won't ended up logging these
> potentially
> >> sensitive values. I understand that only those with access to these log
> >> files can end up seeing these values but even then some of our internal
> >> processes forbid logging such sensitive information to the logs. This
> log
> >> message will still end up being useful if only the config name is logged
> >> without the value.
> >>
> >> Can I add this as a JIRA and provide a patch?
> >>
> >> -Jaikiran
> >>
> >
> >
>

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424402#comment-15424402
 ] 

Rajini Sivaram commented on KAFKA-3980:
---

[~omkreddy] Thanks for the link. I was searching for an open defect, but had 
missed this one.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Do not log value of configs that Kafka doesn't recognize

[Manikumar Reddy]

During server/client startup,  we are logging all the supplied configs. May
be we can just mask
the password related config values for both valid/invalid configs.

On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai 
wrote:

> Any opinion about this proposed change?
>
> -Jaikiran
>
> On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
>
>> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
>> and producers. In one of our consumers, our consumer config had a SSL
>> specific property which ended up being used against a non-SSL Kafka broker
>> port. As a result, the logs ended up seeing messages like:
>>
>> 17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration
>> *ssl.truststore.password = foobar* was supplied but isn't a known config.
>>
>> The log message is fine and makes sense, but can Kafka please not log the
>> values of the properties and instead just include the config name which it
>> considers as unknown? That way it won't ended up logging these potentially
>> sensitive values. I understand that only those with access to these log
>> files can end up seeing these values but even then some of our internal
>> processes forbid logging such sensitive information to the logs. This log
>> message will still end up being useful if only the config name is logged
>> without the value.
>>
>> Can I add this as a JIRA and provide a patch?
>>
>> -Jaikiran
>>
>
>

[jira] [Commented] (KAFKA-3492) support quota based on authenticated user name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424351#comment-15424351
 ] 

ASF GitHub Bot commented on KAFKA-3492:
---

Github user rajinisivaram closed the pull request at:

https://github.com/apache/kafka/pull/1256


> support quota based on authenticated user name
> --
>
> Key: KAFKA-3492
> URL: https://issues.apache.org/jira/browse/KAFKA-3492
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
>
> Currently, quota is based on the client.id set in the client configuration, 
> which can be changed easily. Ideally, quota should be set on the 
> authenticated user name. We will need to have a KIP proposal/discussion on 
> this first.
> Details are in KIP-55: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1256: KAFKA-3492: Secure quotas for authenticated users

[rajinisivaram]

Github user rajinisivaram closed the pull request at:

https://github.com/apache/kafka/pull/1256


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Do not log value of configs that Kafka doesn't recognize

[Jaikiran Pai]

Any opinion about this proposed change?

-Jaikiran
On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers 
and producers. In one of our consumers, our consumer config had a SSL 
specific property which ended up being used against a non-SSL Kafka 
broker port. As a result, the logs ended up seeing messages like:


17:53:33,722  WARN [o.a.k.c.c.ConsumerConfig] - The configuration 
*ssl.truststore.password = foobar* was supplied but isn't a known config.


The log message is fine and makes sense, but can Kafka please not log 
the values of the properties and instead just include the config name 
which it considers as unknown? That way it won't ended up logging 
these potentially sensitive values. I understand that only those with 
access to these log files can end up seeing these values but even then 
some of our internal processes forbid logging such sensitive 
information to the logs. This log message will still end up being 
useful if only the config name is logged without the value.


Can I add this as a JIRA and provide a patch?

-Jaikiran

[jira] [Comment Edited] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424315#comment-15424315
 ] 

Manikumar Reddy edited comment on KAFKA-3980 at 8/17/16 11:22 AM:
--

as suspected, this may be related to quota metrics leak : KAFKA-4054

cc [~rsivaram]


was (Author: omkreddy):
as suspected, this may be related to quota leak : KAFKA-4054

cc [~rsivaram]

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424315#comment-15424315
 ] 

Manikumar Reddy commented on KAFKA-3980:


as suspected, this may be related to quota leak : KAFKA-4054

cc [~rsivaram]

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4055) Add system tests for secure quotas

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-4055:
-

 Summary: Add system tests for secure quotas
 Key: KAFKA-4055
 URL: https://issues.apache.org/jira/browse/KAFKA-4055
 Project: Kafka
  Issue Type: Test
  Components: system tests
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 0.10.1.0


Add system tests for quotas for authenticated users and  
(corresponding to KIP-55). Implementation is being done under KAFKA-3492.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4054) Quota related metrics and sensors are never deleted

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-4054:
-

 Summary: Quota related metrics and sensors are never deleted
 Key: KAFKA-4054
 URL: https://issues.apache.org/jira/browse/KAFKA-4054
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.0.1
Reporter: Rajini Sivaram


Metrics and sensors used for quotas are never deleted. If random client-ids are 
used by clients, this could result in a lot of unused metrics and sensors in 
the broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kafka broker is dropping the messages after acknowledging librdkafka

[Mazhar Shaikh]

Hi Tom,


Thank you for responding and sorry for delay.


I'm running with all the default configuration provided by kafka.

I don't have this config elements in my server.properties file.

However the default values specified in kafka documentation are as below (
http://kafka.apache.org/documentation.html)
unclean.leader.election.enable = true
min.insync.replicas = 1


I'm using a 6 node kafka cluster with 3 zookeeper's, one topic containing
96 partitions with replication factor of 2.

  == 
node 1  =   0   = b0.kafka
node 2  =   1   = b1.kafka
node 3  =   2   = b2.kafka
node 4  =   3   = b3.kafka
node 5  =   4   = b4.kafka
node 6  =   5   = b5.kafka


from the logs, I didn't find any important logs other than below.
Looks like there is connection problem during between brokers and kafka
broker is loosing records during this process.

Log from broker node 3:

[2016-08-17 12:54:50,293] TRACE [Controller 2]: checking need to trigger
partition rebalance (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] DEBUG [Controller 2]: preferred replicas by
broker Map(0 -> Map([topic1,45] -> List(0, 1), [topic1,17] -> List(0, 1),
[topic1,19] -> List(0, 1), [topic1,42] -> List(0, 1), [topic1,43] ->
List(0, 1), [topic1,44] -> List(0, 1), [topic1,16] -> List(0, 1),
[topic1,46] -> List(0, 1), [topic1,20] -> List(0, 1), [topic1,41] ->
List(0, 1), [topic1,18] -> List(0, 1), [topic1,22] -> List(0, 1),
[topic1,40] -> List(0, 1), [topic1,47] -> List(0, 1), [topic1,23] ->
List(0, 1), [topic1,21] -> List(0, 1)), 5 -> Map([topic1,78] -> List(5, 3),
[topic1,84] -> List(5, 3), [topic1,87] -> List(5, 3), [topic1,74] ->
List(5, 3), [topic1,81] -> List(5, 3), [topic1,73] -> List(5, 3),
[topic1,80] -> List(5, 3), [topic1,77] -> List(5, 3), [topic1,75] ->
List(5, 3), [topic1,85] -> List(5, 3), [topic1,76] -> List(5, 3),
[topic1,83] -> List(5, 3), [topic1,86] -> List(5, 3), [topic1,72] ->
List(5, 3), [topic1,79] -> List(5, 3), [topic1,82] -> List(5, 3)), 1 ->
Map([topic1,92] -> List(1, 0), [topic1,95] -> List(1, 0), [topic1,69] ->
List(1, 0), [topic1,93] -> List(1, 0), [topic1,70] -> List(1, 0),
[topic1,67] -> List(1, 0), [topic1,65] -> List(1, 0), [topic1,88] ->
List(1, 0), [topic1,90] -> List(1, 0), [topic1,66] -> List(1, 0),
[topic1,94] -> List(1, 0), [topic1,64] -> List(1, 0), [topic1,89] ->
List(1, 0), [topic1,68] -> List(1, 0), [topic1,71] -> List(1, 0),
[topic1,91] -> List(1, 0)), 2 -> Map([topic1,8] -> List(2, 4), [topic1,3]
-> List(2, 4), [topic1,15] -> List(2, 4), [topic1,2] -> List(2, 4),
[topic1,1] -> List(2, 4), [topic1,6] -> List(2, 4), [topic1,9] -> List(2,
4), [topic1,12] -> List(2, 4), [topic1,14] -> List(2, 4), [topic1,11] ->
List(2, 4), [topic1,13] -> List(2, 4), [topic1,0] -> List(2, 4), [topic1,4]
-> List(2, 4), [topic1,5] -> List(2, 4), [topic1,10] -> List(2, 4),
[topic1,7] -> List(2, 4)), 3 -> Map([topic1,33] -> List(3, 5), [topic1,30]
-> List(3, 5), [topic1,24] -> List(3, 5), [topic1,36] -> List(3, 5),
[topic1,38] -> List(3, 5), [topic1,26] -> List(3, 5), [topic1,27] ->
List(3, 5), [topic1,39] -> List(3, 5), [topic1,29] -> List(3, 5),
[topic1,34] -> List(3, 5), [topic1,28] -> List(3, 5), [topic1,32] ->
List(3, 5), [topic1,35] -> List(3, 5), [topic1,25] -> List(3, 5),
[topic1,31] -> List(3, 5), [topic1,37] -> List(3, 5)), 4 -> Map([topic1,53]
-> List(4, 2), [topic1,56] -> List(4, 2), [topic1,49] -> List(4, 2),
[topic1,50] -> List(4, 2), [topic1,51] -> List(4, 2), [topic1,58] ->
List(4, 2), [topic1,63] -> List(4, 2), [topic1,54] -> List(4, 2),
[topic1,48] -> List(4, 2), [topic1,61] -> List(4, 2), [topic1,62] ->
List(4, 2), [topic1,57] -> List(4, 2), [topic1,60] -> List(4, 2),
[topic1,52] -> List(4, 2), [topic1,55] -> List(4, 2), [topic1,59] ->
List(4, 2))) (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] DEBUG [Controller 2]: topics not in preferred
replica Map() (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] TRACE [Controller 2]: leader imbalance ratio for
broker 0 is 0.00 (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] DEBUG [Controller 2]: topics not in preferred
replica Map() (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] TRACE [Controller 2]: leader imbalance ratio for
broker 5 is 0.00 (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] DEBUG [Controller 2]: topics not in preferred
replica Map() (kafka.controller.KafkaController)
[2016-08-17 12:54:50,294] TRACE [Controller 2]: leader imbalance ratio for
broker 1 is 0.00 (kafka.controller.KafkaController)
[2016-08-17 12:54:50,295] DEBUG [Controller 2]: topics not in preferred
replica Map() (kafka.controller.KafkaController)
[2016-08-17 12:54:50,295] TRACE [Controller 2]: leader imbalance ratio for
broker 2 is 0.00 (kafka.controller.KafkaController)
[2016-08-17 12:54:50,295] DEBUG [Controller 2]: topics not in 

[jira] [Commented] (KAFKA-3776) Unify store and downstream caching in streams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424264#comment-15424264
 ] 

ASF GitHub Bot commented on KAFKA-3776:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1752

KAFKA-3776: Unify store and downstream caching in streams [WiP]

Work-in-progress PoC, not to be merged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1752


commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
Author: Eno Thereska 
Date:   2016-08-16T09:45:00Z

Initial commit with stub cache

commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
Author: Eno Thereska 
Date:   2016-08-17T08:22:23Z

Adjustments so tests compile

commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
Author: Eno Thereska 
Date:   2016-08-17T10:21:11Z

Remove old cache from RocksDbStore and add global cache to RocksDb




> Unify store and downstream caching in streams
> -
>
> Key: KAFKA-3776
> URL: https://issues.apache.org/jira/browse/KAFKA-3776
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> This is an umbrella story for capturing changes to processor caching in 
> Streams as first described in KIP-63. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1752: KAFKA-3776: Unify store and downstream caching in ...

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1752

KAFKA-3776: Unify store and downstream caching in streams [WiP]

Work-in-progress PoC, not to be merged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka KAFKA-3776-poc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1752.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1752


commit 9e6d1e0b7b50b8c9bb1848c9e72913701a93cbb0
Author: Eno Thereska 
Date:   2016-08-16T09:45:00Z

Initial commit with stub cache

commit 2a3e46770b2aaddefd843f7cc6cc6727dd1cbc9e
Author: Eno Thereska 
Date:   2016-08-17T08:22:23Z

Adjustments so tests compile

commit 86bb6dcb6beac3d136e01f45cd727e6a6a691b5b
Author: Eno Thereska 
Date:   2016-08-17T10:21:11Z

Remove old cache from RocksDbStore and add global cache to RocksDb




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[Umesh Chaudhary (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424224#comment-15424224
 ] 

Umesh Chaudhary commented on KAFKA-4053:


I will do that. Please assign it to me.

> Refactor TopicCommand to remove redundant if/else statements
> 
>
> Key: KAFKA-4053
> URL: https://issues.apache.org/jira/browse/KAFKA-4053
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.0.1
>Reporter: Shuai Zhang
>Priority: Minor
> Fix For: 0.10.0.2
>
>
> In TopicCommand, there are a lot of redundant if/else statements, such as
> ```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
> false```
> We can refactor it as the following statement:
> ```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4039) Exit Strategy: using exceptions instead of inline invocation of exit/halt

[Alexey Ozeritskiy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424201#comment-15424201
 ] 

Alexey Ozeritskiy commented on KAFKA-4039:
--

Thanks [~maysamyabandeh]
I'll be happy to test your patch.

> Exit Strategy: using exceptions instead of inline invocation of exit/halt
> -
>
> Key: KAFKA-4039
> URL: https://issues.apache.org/jira/browse/KAFKA-4039
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Maysam Yabandeh
>
> The current practice is to directly invoke halt/exit right after the line 
> that intends to terminate the execution. In the case of System.exit this 
> could cause deadlocks if the thread invoking System.exit is holding  a lock 
> that will be requested by the shutdown hook threads that will be started by 
> System.exit. An example is reported by [~aozeritsky] in KAFKA-3924. This 
> would also makes testing more difficult as it would require mocking static 
> methods of System and Runtime classes, which is not natively supported in 
> Java.
> One alternative suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-3924?focusedCommentId=15420269=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15420269]
>  would be to throw some dedicated exceptions that will eventually invoke 
> exit/halt:
> {quote} it would be great to move away from executing `System.exit` inline in 
> favour of throwing an exception (two examples, but maybe we can find better 
> names: FatalExitException and FatalHaltException) that is caught by some 
> central code that then does the `System.exit` or `Runtime.getRuntime.halt`. 
> This helps in a couple of ways:
> (1) Avoids issues with locks being held as in this issue
> (2) It makes it possible to abstract the action, which is very useful in 
> tests. At the moment, we can't easily test for these conditions as they cause 
> the whole test harness to exit. Worse, these conditions are sometimes 
> triggered in the tests and it's unclear why.
> (3) We can have more consistent logging around these actions and possibly 
> extended logging for tests
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1751: remove redundant if/else statements in TopicComman...

[sh-z]

GitHub user sh-z opened a pull request:

https://github.com/apache/kafka/pull/1751

remove redundant if/else statements in TopicCommand



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sh-z/kafka KAFKA-4053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1751.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1751


commit 6e7458e03cb0725db6a7079466fd7fce4a2f7603
Author: Shuai Zhang 
Date:   2016-08-17T09:29:54Z

remove redundant if/else statements




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4053) Refactor TopicCommand to remove redundant if/else statements

[Shuai Zhang (JIRA)]

Shuai Zhang created KAFKA-4053:
--

 Summary: Refactor TopicCommand to remove redundant if/else 
statements
 Key: KAFKA-4053
 URL: https://issues.apache.org/jira/browse/KAFKA-4053
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 0.10.0.1
Reporter: Shuai Zhang
Priority: Minor
 Fix For: 0.10.0.2


In TopicCommand, there are a lot of redundant if/else statements, such as
```val ifNotExists = if (opts.options.has(opts.ifNotExistsOpt)) true else 
false```

We can refactor it as the following statement:
```val ifNotExists = opts.options.has(opts.ifNotExistsOpt)```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4044) log actual socket send/receive buffer size after connecting in Selector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424138#comment-15424138
 ] 

ASF GitHub Bot commented on KAFKA-4044:
---

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1750

KAFKA-4044: log actual socket send/receive buffer size after connecting in 
Selector



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-4044-LOG

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1750


commit b9427653ab35324d4544de6c922d0c0f50ee3bd1
Author: Manikumar Reddy O 
Date:   2016-08-17T06:26:08Z

KAFKA-4044: log actual socket send/receive buffer size after connecting in 
Selector




> log actual socket send/receive buffer size after connecting in Selector
> ---
>
> Key: KAFKA-4044
> URL: https://issues.apache.org/jira/browse/KAFKA-4044
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Manikumar Reddy
>  Labels: newbie
>
> In BlockingChannel, we had the following code to log the actual socket buffer 
> size when the socket connection is established. This can be helpful when 
> tuning the socket buffer size for long latency network. It would be useful to 
> add that in Selector.pollSelectionKeys when the socket is connected.
> val msg = "Created socket with SO_TIMEOUT = %d (requested %d), 
> SO_RCVBUF = %d (requested %d), SO_SNDBUF = %d (requested %d), 
> connectTimeoutMs = %d."
> debug(msg.format(channel.socket.getSoTimeout,
>  readTimeoutMs,
>  channel.socket.getReceiveBufferSize, 
>  readBufferSize,
>  channel.socket.getSendBufferSize,
>  writeBufferSize,
>  connectTimeoutMs))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1750: KAFKA-4044: log actual socket send/receive buffer ...

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1750

KAFKA-4044: log actual socket send/receive buffer size after connecting in 
Selector



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-4044-LOG

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1750


commit b9427653ab35324d4544de6c922d0c0f50ee3bd1
Author: Manikumar Reddy O 
Date:   2016-08-17T06:26:08Z

KAFKA-4044: log actual socket send/receive buffer size after connecting in 
Selector




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4045) Investigate feasibility of hooking into RocksDb's cache

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424099#comment-15424099
 ] 

Damian Guy commented on KAFKA-4045:
---

https://github.com/facebook/rocksdb/issues/1282

> Investigate feasibility of hooking into RocksDb's cache
> ---
>
> Key: KAFKA-4045
> URL: https://issues.apache.org/jira/browse/KAFKA-4045
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> Ideally we could hook a listener into RockDb's cache so that when entries are 
> flushed or evicted from the cache the listener is called (and can 
> subsequently perform Kafka Streams-specific functions, like forward a record 
> downstream). That way we don't build our own cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)