[jira] [Updated] (KAFKA-4026) consumer block

[ https://issues.apache.org/jira/browse/KAFKA-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wxmimperio updated KAFKA-4026: -- Description: when i use high level api make consumer. it is a block consumer,how can i know the time of

[jira] [Updated] (KAFKA-4026) consumer block

[ https://issues.apache.org/jira/browse/KAFKA-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wxmimperio updated KAFKA-4026: -- Description: when i use high level api make consumer. it is a block consumer,how can i know the time of

[jira] [Created] (KAFKA-4026) consumer block

wxmimperio created KAFKA-4026: - Summary: consumer block Key: KAFKA-4026 URL: https://issues.apache.org/jira/browse/KAFKA-4026 Project: Kafka Issue Type: Test Components: consumer

[jira] [Resolved] (KAFKA-3954) Consumer should use internal topics information returned by the broker

[ https://issues.apache.org/jira/browse/KAFKA-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-3954. -- Resolution: Fixed Issue resolved by pull request 1613

[jira] [Commented] (KAFKA-3954) Consumer should use internal topics information returned by the broker

[ https://issues.apache.org/jira/browse/KAFKA-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412868#comment-15412868 ] ASF GitHub Bot commented on KAFKA-3954: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #1613: KAFKA-3954; Consumer should use internal topics in...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1613 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk8 #806

See Changes: [ismael] MINOR: Add `fetchTopicMetadataFromZk` overload that takes [ismael] KAFKA-4013; Include exception cause in SaslServerCallbackHandler -- [...truncated 11383 lines...]

[GitHub] kafka pull request #1623: MINOR: Remove redundant clause in secureAclsEnable...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1623 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3973) Investigate feasibility of caching bytes vs. records

[ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412838#comment-15412838 ] Ben Manes commented on KAFKA-3973: -- It would also be impactful to consider the policy's hit rate. If a

[jira] [Resolved] (KAFKA-4013) SaslServerCallbackHandler should include cause for exception

[ https://issues.apache.org/jira/browse/KAFKA-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4013. Resolution: Fixed Fix Version/s: 0.10.1.0 Issue resolved by pull request 1695

[jira] [Commented] (KAFKA-4013) SaslServerCallbackHandler should include cause for exception

[ https://issues.apache.org/jira/browse/KAFKA-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412779#comment-15412779 ] ASF GitHub Bot commented on KAFKA-4013: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #1695: KAFKA-4013: Included exception cause in SaslServer...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1695 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Thought about this again. If I understand correctly Jun's concern is about the cascading effect. Currently the processor will try to put all the requests received in one poll() call into the RequestChannel. This could potentially be long if the queue is moving really really slowly. If we don't

[GitHub] kafka pull request #1713: MINOR: Add security protocol option for fetch

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1713 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-3975) Error if zookeeper.set.acl is true and ZK security is not enabled

[ https://issues.apache.org/jira/browse/KAFKA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3975. Resolution: Invalid The code is actually correct although it has a redundant clause. Thanks to

[jira] [Updated] (KAFKA-3975) Error if zookeeper.set.acl is true and ZK security is not enabled

[ https://issues.apache.org/jira/browse/KAFKA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3975: --- Fix Version/s: (was: 0.10.1.0) > Error if zookeeper.set.acl is true and ZK security is not

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

I agree that filling up the request queue can cause clients to time out (and presumably retry?). However, for the workloads where we expect this configuration to be useful the alternative is currently an OOM crash. In my opinion an initial implementation of this feature could be constrained to a

[jira] [Commented] (KAFKA-3989) Add JMH module for Benchmarks

[ https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412649#comment-15412649 ] Ismael Juma commented on KAFKA-3989: [~bbejeck], if the shadow plugin is simpler, then it makes sense

[jira] [Commented] (KAFKA-2629) Enable getting SSL password from an executable rather than passing plaintext password

[ https://issues.apache.org/jira/browse/KAFKA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412631#comment-15412631 ] Bharat Viswanadham commented on KAFKA-2629: --- Hi [~singhashish] Any updates on this task can you

[jira] [Commented] (KAFKA-4011) allow sizing RequestQueue in bytes

[ https://issues.apache.org/jira/browse/KAFKA-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412571#comment-15412571 ] ASF GitHub Bot commented on KAFKA-4011: --- GitHub user radai-rosenblatt opened a pull request:

[GitHub] kafka pull request #1714: KAFKA-4011 - fix issues and beef up tests around B...

GitHub user radai-rosenblatt opened a pull request: https://github.com/apache/kafka/pull/1714 KAFKA-4011 - fix issues and beef up tests around ByteBoundedBlockingQueue as discussed under KIP-72 You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-2630) Add Namespaces to Kafka

[ https://issues.apache.org/jira/browse/KAFKA-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412564#comment-15412564 ] Vahid Hashemian commented on KAFKA-2630: [~singhashish] Thanks for the update. I look forward to

[jira] [Commented] (KAFKA-2630) Add Namespaces to Kafka

[ https://issues.apache.org/jira/browse/KAFKA-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412553#comment-15412553 ] Ashish K Singh commented on KAFKA-2630: --- [~vahid] apologies for the delayed response. This is still

[jira] [Commented] (KAFKA-4022) TopicCommand is using default max.message.bytes instead of broker's setting

[ https://issues.apache.org/jira/browse/KAFKA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412542#comment-15412542 ] Vahid Hashemian commented on KAFKA-4022: [~kornicameis...@gmail.com] I believe this is simply a

Re: [DISCUSS] KIP-4 ACL Admin Schema

Thank you for the feedback everyone. Below I respond to the last batch of emails: You mention that "delete" actions > will get processed before "add" actions, which makes sense to me. An > alternative to avoid the confusion in the first place would be to replace > the AlterAcls APIs with separate

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Nice write up Radai. I think what Jun said is a valid concern. If I am not wrong as per the proposal, we are depending on the entire pipeline to flow smoothly from accepting requests to handling it, calling KafkaApis and handing back the responses. Thanks, Mayuresh On Mon, Aug 8, 2016 at 12:22

[jira] [Commented] (KAFKA-3894) LogCleaner thread crashes if not even one segment can fit in the offset map

[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412430#comment-15412430 ] Elias Dorneles commented on KAFKA-3894: --- I've bumped the bumped into this same issue (log cleaner

[jira] [Comment Edited] (KAFKA-3894) LogCleaner thread crashes if not even one segment can fit in the offset map

[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412430#comment-15412430 ] Elias Dorneles edited comment on KAFKA-3894 at 8/8/16 8:45 PM: --- I've bumped

[jira] [Commented] (KAFKA-3556) Improve group coordinator metrics

[ https://issues.apache.org/jira/browse/KAFKA-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412416#comment-15412416 ] Jun Rao commented on KAFKA-3556: It would also be useful to add metrics on the consumer side to track

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412359#comment-15412359 ] Kane Kim commented on KAFKA-2729: - For us the reason was high percentage of lost packets to one of ZK

Re: [DISCUSS] KIP-73: Replication Quotas

Hi Ben, Thanks for the detailed write-up. So the proposal involves self-throttling on the fetcher side and throttling at the leader. Can you elaborate on the reasoning that is given on the wiki: *“The throttle is applied to both leaders and followers. This allows the admin to exert strong

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

> > . >> >> > Hi Becket, > > I don't think progress can be made in the processor's run loop if the > queue fills up. i.e., I think Jun's point is that if the queue is full > (either due to the proposed max.bytes or today due to max.requests hitting > the limit) then processCompletedReceives will

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

> > > > 2. Good point about the consequence when the processor threads are > blocking. I agree it would be important to keep the processor thread > running, but I am not sure if it would be a problem of the current > proposal. In most of the time, the request queue should be close to empty, > so

Re: [VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

Thanks Vahid. +1 from me (non-binding). On Mon, Aug 8, 2016 at 11:53 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > I would like to initiate the voting process for KIP-70 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 70%3A+Revise+Partition+Assignment+Semantics+on+New+

[VOTE] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

I would like to initiate the voting process for KIP-70 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change ). The only issue that was discussed in the discussion thread is compatibility, but because it

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Hi Jun, 1. The requests in the queue are RequestChannel.Request. It contains the raw ByteBuffer from the socket, we can probably just use that. This might not be 100% accurate about the memory taken, but probably is good enough. 2. Good point about the consequence when the processor threads are

[GitHub] kafka pull request #1713: MINOR: Add security protocol option for fetch

GitHub user norwood opened a pull request: https://github.com/apache/kafka/pull/1713 MINOR: Add security protocol option for fetch @ijuma You can merge this pull request into a Git repository by running: $ git pull https://github.com/norwood/kafka

[jira] [Commented] (KAFKA-3989) Add JMH module for Benchmarks

[ https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412231#comment-15412231 ] Bill Bejeck commented on KAFKA-3989: [~ijuma] After your comment, I tried to implement the

[jira] [Updated] (KAFKA-3989) Add JMH module for Benchmarks

[ https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-3989: --- Status: Patch Available (was: In Progress) > Add JMH module for Benchmarks >

[jira] [Commented] (KAFKA-3989) Add JMH module for Benchmarks

[ https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412216#comment-15412216 ] ASF GitHub Bot commented on KAFKA-3989: --- GitHub user bbejeck opened a pull request:

[GitHub] kafka pull request #1712: KAFKA-3989: initial support for adding a JMH bench...

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/1712 KAFKA-3989: initial support for adding a JMH benchmarking module. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka

[jira] [Commented] (KAFKA-4020) Kafka consumer stop taking messages from kafka server

[ https://issues.apache.org/jira/browse/KAFKA-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412204#comment-15412204 ] Shawn He commented on KAFKA-4020: - One thing to mention is that I am still using the old API at both

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412133#comment-15412133 ] Jun Rao commented on KAFKA-1211: [~fpj], for (a), this is because the leader needs to first wait for the

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412126#comment-15412126 ] Michael Sandrof commented on KAFKA-2729: We seem to be having a similar problem running 0.10.0.0.

[jira] [Commented] (KAFKA-2063) Bound fetch response size

[ https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412106#comment-15412106 ] Andrey Neporada commented on KAFKA-2063: Created KIP-74 > Bound fetch response size >

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412111#comment-15412111 ] Jun Rao commented on KAFKA-1211: [~fpj], yes, the idea is to always first write the new LGS before writing

[jira] [Commented] (KAFKA-3971) Consumers drop from coordinator and cannot reconnet

[ https://issues.apache.org/jira/browse/KAFKA-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412093#comment-15412093 ] Lei Wang commented on KAFKA-3971: - thanks, i will try assign the partitions manually instead of the

[DISCUSS] KIP-74: Add Fetch Response Size Limit in Bytes

Hi all! I’ve just created KIP-74: Add Fetch Response Size Limit in Bytes. The idea is to limit client memory consumption when fetching many partitions (especially useful for replication). Full details are here:

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

Radai, Thanks for the proposal. A couple of comments on this. 1. Since we store request objects in the request queue, how do we get an accurate size estimate for those requests? 2. Currently, it's bad if the processor blocks on adding a request to the request queue. Once blocked, the processor

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412077#comment-15412077 ] Jeffers Ji commented on KAFKA-1194: --- Hello Kreps, I am glad that this issue will be resolved in the new

[jira] [Comment Edited] (KAFKA-3973) Investigate feasibility of caching bytes vs. records

[ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411821#comment-15411821 ] Bill Bejeck edited comment on KAFKA-3973 at 8/8/16 4:55 PM: I used JMH to

Jenkins build is back to normal : kafka-0.10.0-jdk7 #186

See

[jira] [Comment Edited] (KAFKA-3973) Investigate feasibility of caching bytes vs. records

[ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411821#comment-15411821 ] Bill Bejeck edited comment on KAFKA-3973 at 8/8/16 4:45 PM: I used JMH to

[jira] [Comment Edited] (KAFKA-3973) Investigate feasibility of caching bytes vs. records

[ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411821#comment-15411821 ] Bill Bejeck edited comment on KAFKA-3973 at 8/8/16 3:34 PM: I used JMH to

[RESULTS] [VOTE] Release Kafka 0.10.0.1

The vote for the Kafka 0.10.0.1 release passed with 14 +1 votes (4 binding) and no 0 or -1 votes. +1 votes PMC Members: * Neha Narkhede * Guozhang Wang * Jun Rao * Joel Koshy Committers: * Gwen Shapira * Harsha Chintalapani * Ismael Juma Community: * Jim Jagielski * Tom Crayford * Dana Powers *

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

+1 (non-binding), verified checksums and signature, ran tests on source package and verified quickstart on source and binary packages. On Mon, Aug 8, 2016 at 2:17 AM, Harsha Chintalapani wrote: > +1 (binding) > 1. Ran 3 node cluser > 2. Ran few tests in creating, producing ,

[jira] [Updated] (KAFKA-3936) Validate user parameters as early as possible

[ https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy updated KAFKA-3936: -- Status: Patch Available (was: Open) > Validate user parameters as early as possible >

[jira] [Commented] (KAFKA-3936) Validate user parameters as early as possible

[ https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411868#comment-15411868 ] ASF GitHub Bot commented on KAFKA-3936: --- GitHub user dguy opened a pull request:

[GitHub] kafka pull request #1711: KAFKA-3936: Validate parameters as early as possib...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1711 KAFKA-3936: Validate parameters as early as possible Added non null checks to parameters supplied via the DSL and `TopologyBuilder` You can merge this pull request into a Git repository by running:

[jira] [Commented] (KAFKA-3973) Investigate feasibility of caching bytes vs. records

[ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411821#comment-15411821 ] Bill Bejeck commented on KAFKA-3973: I used JMH to benchmark the performance of caching bytes vs

RE: Kafka consumer getting duplicate message

Thank you , Ewen for your response. Actually we are using 1.0.0.M2 Spring Kafka release that uses Kafka 0.9 release. Yes, we see a lot of duplicates and here is our producer and consumer settings in application. We don't see any duplicacy at Producer end I mean if we send 1000 messages to a

[DISCUSS] KIP-73: Replication Quotas

We’ve created KIP-73: Replication Quotas The idea is to allow an admin to throttle moving replicas. Full details are here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas Please take a

[jira] [Commented] (KAFKA-2063) Bound fetch response size

[ https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411650#comment-15411650 ] Ismael Juma commented on KAFKA-2063: I've given you the permissions required [~nepal]. > Bound fetch

[jira] [Commented] (KAFKA-2063) Bound fetch response size

[ https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411520#comment-15411520 ] Andrey Neporada commented on KAFKA-2063: Hi! Looks like I don't have necessary permissions to

[DISCUSS] KIP-71 Enable log compaction and deletion to co-exist

Hi, We have created KIP 71: Enable log compaction and deletion to co-exist` https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist Please take a look. Feedback is appreciated. Thank you

[jira] [Resolved] (KAFKA-3972) kafka java consumer poll returns 0 records after seekToBeginning

[ https://issues.apache.org/jira/browse/KAFKA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-3972. -- Resolution: Invalid Assignee: Ewen Cheslack-Postava > kafka java consumer

[jira] [Commented] (KAFKA-3967) Excessive Network IO between Kafka brokers

[ https://issues.apache.org/jira/browse/KAFKA-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411376#comment-15411376 ] Ewen Cheslack-Postava commented on KAFKA-3967: -- [~Krishna82] Have you accounted for all