Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Ewen Cheslack-Postava
+1 on trying to reuse the NetworkClient code. I think Jun's approach could work, but I'm wondering if refactoring a bit could get better separation of concerns without a somewhat awkward nop implementation of Metadata. I'm not sure what combination of delegation or subclassing makes sense yet,

[jira] [Created] (KAFKA-2173) Kafka died after throw more error

2015-05-06 Thread Truyet Nguyen (JIRA)
Truyet Nguyen created KAFKA-2173: Summary: Kafka died after throw more error Key: KAFKA-2173 URL: https://issues.apache.org/jira/browse/KAFKA-2173 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

2015-05-06 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530071#comment-14530071 ] Honghai Chen commented on KAFKA-2170: - [~junrao] this failure not related with

[jira] [Updated] (KAFKA-2174) Wrong TopicMetadata deserialization

2015-05-06 Thread Alexey Ozeritskiy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Ozeritskiy updated KAFKA-2174: - Attachment: KAFKA-2174.patch Wrong TopicMetadata deserialization

[jira] [Updated] (KAFKA-2174) Wrong TopicMetadata deserialization

2015-05-06 Thread Alexey Ozeritskiy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Ozeritskiy updated KAFKA-2174: - Status: Patch Available (was: Open) Wrong TopicMetadata deserialization

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-06 Thread Sriharsha Chintalapani
Thanks Jay. I removed partitioner.metadata from KIP. I’ll send an updated patch. --  Harsha Sent with Airmail On May 5, 2015 at 6:31:47 AM, Sriharsha Chintalapani (harsh...@fastmail.fm) wrote: Thanks for the comments everyone. Hi Jay,      I do have a question regarding configurable interface

[jira] [Created] (KAFKA-2174) Wrong TopicMetadata deserialization

2015-05-06 Thread Alexey Ozeritskiy (JIRA)
Alexey Ozeritskiy created KAFKA-2174: Summary: Wrong TopicMetadata deserialization Key: KAFKA-2174 URL: https://issues.apache.org/jira/browse/KAFKA-2174 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-06 Thread Jiangjie Qin
Yes, that is the plan. On 5/5/15, 8:23 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Just a quick question, can we handle REQUEST TIMEOUT as disconnections and do a fresh MetaDataRequest and retry instead of failing the request? Thanks, Mayuresh On Mon, May 4, 2015 at 10:32 AM,

Review Request 33916: Patch for KAFKA-2163

2015-05-06 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33916/ --- Review request for kafka. Bugs: KAFKA-2163

[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets

2015-05-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Attachment: KAFKA-2163.patch Offsets manager cache should prevent stale-offset-cleanup while an offset

[jira] [Updated] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets

2015-05-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2163: -- Assignee: Joel Koshy Status: Patch Available (was: Open) Offsets manager cache should prevent

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-06 Thread Jun Rao
On April 18, 2015, 12:49 a.m., Jun Rao wrote: core/src/main/scala/kafka/api/FetchResponse.scala, lines 66-72 https://reviews.apache.org/r/33065/diff/1/?file=922627#file922627line66 I am wondering if we need both completed() and remaining() in Send. It seems that one of the two is

[jira] [Commented] (KAFKA-2150) FetcherThread backoff need to grab lock before wait on condition.

2015-05-06 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531695#comment-14531695 ] Sriharsha Chintalapani commented on KAFKA-2150: --- [~guozhang] Can you please

Re: [VOTE] KIP-4 Admin Commands / Phase-1

2015-05-06 Thread Joe Stein
+1 (binding) ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, May 5, 2015 at 11:16 AM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Hi all, This is a voting thread for KIP-4 Phase-1. It will include Wire protocol

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-06 Thread Joel Koshy
+1 with a minor comment: do we need an init method given it extends Configurable? Also, can you move this wiki out of drafts and add it to the table in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals? Thanks, Joel On Wed, May 06, 2015 at 07:46:46AM -0700,

Re: Need Access to Wiki Page To Create Page for Discussion

2015-05-06 Thread Jun Rao
What your wiki user id? Thanks, Jun On Wed, May 6, 2015 at 11:09 AM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi All, I need access to create Discussion or KIP document. Let me know what is process of getting access. Thanks, Bhavesh

[jira] [Commented] (KAFKA-2160) DelayedOperationPurgatory should remove the pair in watchersForKey with empty watchers list

2015-05-06 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531672#comment-14531672 ] Guozhang Wang commented on KAFKA-2160: -- Updated reviewboard

Re: Review Request 33731: Second Attempt to Fix KAFKA-2160

2015-05-06 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33731/ --- (Updated May 6, 2015, 11:31 p.m.) Review request for kafka. Summary

[jira] [Updated] (KAFKA-2160) DelayedOperationPurgatory should remove the pair in watchersForKey with empty watchers list

2015-05-06 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2160: - Attachment: KAFKA-2160_2015-05-06_16:31:48.patch DelayedOperationPurgatory should remove the

[jira] [Commented] (KAFKA-2163) Offsets manager cache should prevent stale-offset-cleanup while an offset load is in progress; otherwise we can lose consumer offsets

2015-05-06 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531529#comment-14531529 ] Joel Koshy commented on KAFKA-2163: --- Created reviewboard

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-06 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/#review82625 --- Thanks for the patch. Looks great overall! Some comments below.

[jira] [Updated] (KAFKA-2175) Reduce server log verbosity at info level

2015-05-06 Thread Todd Palino (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Palino updated KAFKA-2175: --- Status: Patch Available (was: Open) Reduce server log verbosity at info level

[jira] [Assigned] (KAFKA-2175) Reduce server log verbosity at info level

2015-05-06 Thread Todd Palino (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Palino reassigned KAFKA-2175: -- Assignee: Todd Palino (was: Neha Narkhede) Reduce server log verbosity at info level

[jira] [Created] (KAFKA-2175) Reduce server log verbosity at info level

2015-05-06 Thread Todd Palino (JIRA)
Todd Palino created KAFKA-2175: -- Summary: Reduce server log verbosity at info level Key: KAFKA-2175 URL: https://issues.apache.org/jira/browse/KAFKA-2175 Project: Kafka Issue Type: Improvement

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Guozhang Wang
Jiangjie, Just trying to figure the class reference hierarchy of this approach (A - B means A either has a member variable of B or take B as API parameter). Metadata will have interface that takes in KafkaClient as a parameter, so Metadata - KafkaClient 1. For producer: KafkaProducer - Sender,

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Ewen Cheslack-Postava
On Wed, May 6, 2015 at 10:40 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: Jun Ewen, Thanks a lot for the comments. Both of them look better than my original plan. Jun, one downside about not changing NetworkClient but use a different metadata implementation is that NetworkClient will

Need Access to Wiki Page To Create Page for Discussion

2015-05-06 Thread Bhavesh Mistry
Hi All, I need access to create Discussion or KIP document. Let me know what is process of getting access. Thanks, Bhavesh

[jira] [Resolved] (KAFKA-2027) kafka never notifies the zookeeper client when a partition moved with due to an auto-rebalance (when auto.leader.rebalance.enable=true)

2015-05-06 Thread Sampath Reddy Lambu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sampath Reddy Lambu resolved KAFKA-2027. Resolution: Invalid kafka never notifies the zookeeper client when a partition

[jira] [Closed] (KAFKA-2027) kafka never notifies the zookeeper client when a partition moved with due to an auto-rebalance (when auto.leader.rebalance.enable=true)

2015-05-06 Thread Sampath Reddy Lambu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sampath Reddy Lambu closed KAFKA-2027. -- kafka never notifies the zookeeper client when a partition moved with due to an

[jira] [Commented] (KAFKA-2027) kafka never notifies the zookeeper client when a partition moved with due to an auto-rebalance (when auto.leader.rebalance.enable=true)

2015-05-06 Thread Sampath Reddy Lambu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531073#comment-14531073 ] Sampath Reddy Lambu commented on KAFKA-2027: this was handled in Kafka-Node

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Jiangjie Qin
Jun Ewen, Thanks a lot for the comments. Both of them look better than my original plan. Jun, one downside about not changing NetworkClient but use a different metadata implementation is that NetworkClient will still have all the metadata related code there which makes it a little bit weird. I

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-06 Thread Jun Rao
Jiangjie, Yes, I think using metadata timeout to expire batches in the record accumulator makes sense. Thanks, Jun On Mon, May 4, 2015 at 10:32 AM, Jiangjie Qin j...@linkedin.com.invalid wrote: I incorporated Ewen and Guozhang’s comments in the KIP page. Want to speed up on this KIP because

Re: [DISCUSS] KIP-21 Configuration Management

2015-05-06 Thread Ashish Singh
Hey Jun, Where does the broker get the info, which zk it needs to talk to? On Wednesday, May 6, 2015, Jun Rao j...@confluent.io wrote: Ashish, 3. Just want to clarify. Why can't you store ZK connection config in ZK? This is a property for ZK clients, not ZK server. Thanks, Jun On Wed,

Re: Need Access to Wiki Page To Create Page for Discussion

2015-05-06 Thread Bhavesh Mistry
Hi Jun, The account id is Bmis13. Thanks, Bhavesh On Wed, May 6, 2015 at 4:52 PM, Jun Rao j...@confluent.io wrote: What your wiki user id? Thanks, Jun On Wed, May 6, 2015 at 11:09 AM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi All, I need access to create Discussion

Re: Review Request 32650: Patch for KAFKA-2000

2015-05-06 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32650/#review82775 --- Thanks for the updated patch.

Re: [DISCUSS] KIP-21 Configuration Management

2015-05-06 Thread Jun Rao
Ashish, 3. Just want to clarify. Why can't you store ZK connection config in ZK? This is a property for ZK clients, not ZK server. Thanks, Jun On Wed, May 6, 2015 at 5:48 PM, Ashish Singh asi...@cloudera.com wrote: I too would like to share some concerns that we came up with while

Re: [DISCUSS] KIP-21 Configuration Management

2015-05-06 Thread Ashish Singh
I too would like to share some concerns that we came up with while discussing the effect of moving configs to zookeeper will have. 1. Kafka will start to become a configuration management tool to some degree, and be subject to all the things such tools are commonly asked to do. Kafka'll likely

Re: Review Request 33378: Patch for KAFKA-2136

2015-05-06 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33378/ --- (Updated May 7, 2015, 1:32 a.m.) Review request for kafka and Joel Koshy.

Re: [DISCUSS] KIP-21 Configuration Management

2015-05-06 Thread Ashish Singh
This raises one more concern. Just putting it out there. 4. What if one decides to move to a different zk, I am sure there are other concerns in status quo for this question. However, this is adding to the concern, right? Not sure though how much big of a concern it practically is. On Wednesday,

Re: Review Request 33378: Patch for KAFKA-2136

2015-05-06 Thread Aditya Auradkar
On May 4, 2015, 4:51 p.m., Jun Rao wrote: core/src/main/scala/kafka/api/FetchResponse.scala, lines 152-153 https://reviews.apache.org/r/33378/diff/1/?file=937083#file937083line152 This is tricky since FetchRequest is used in the follower as well. When doing a rolling upgrade of

Re: Need Access to Wiki Page To Create Page for Discussion

2015-05-06 Thread Guozhang Wang
Bhavesh, I could not find Bmis13 when adding you to the wiki permission. Could you double check the account id? Guozhang On Wed, May 6, 2015 at 6:47 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Jun, The account id is Bmis13. Thanks, Bhavesh On Wed, May 6, 2015 at 4:52 PM,

Re: [DISCUSS] KIP-21 Configuration Management

2015-05-06 Thread Jun Rao
One of the Chef users confirmed that Chef integration could still work if all configs are moved to ZK. My rough understanding of how Chef works is that a user first registers a service host with a Chef server. After that, a Chef client will be run on the service host. The user can then push config

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-06 Thread Harsha
Thanks for the review Joel. I agree don't need a init method we can use configure. I'll update the KIP. -Harsha On Wed, May 6, 2015, at 04:45 PM, Joel Koshy wrote: +1 with a minor comment: do we need an init method given it extends Configurable? Also, can you move this wiki out of drafts and

[jira] [Commented] (KAFKA-2136) Client side protocol changes to return quota delays

2015-05-06 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531857#comment-14531857 ] Aditya A Auradkar commented on KAFKA-2136: -- Updated reviewboard

[jira] [Updated] (KAFKA-2136) Client side protocol changes to return quota delays

2015-05-06 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-2136: - Attachment: KAFKA-2136_2015-05-06_18:32:48.patch Client side protocol changes to return

[jira] [Updated] (KAFKA-2136) Client side protocol changes to return quota delays

2015-05-06 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-2136: - Attachment: KAFKA-2136_2015-05-06_18:35:54.patch Client side protocol changes to return

[jira] [Commented] (KAFKA-2136) Client side protocol changes to return quota delays

2015-05-06 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531865#comment-14531865 ] Aditya A Auradkar commented on KAFKA-2136: -- Updated reviewboard

Re: Review Request 33378: Patch for KAFKA-2136

2015-05-06 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33378/ --- (Updated May 7, 2015, 1:36 a.m.) Review request for kafka and Joel Koshy.

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Jiangjie Qin
Hey Guozhang, Good point about unify the handling of MetadataResponse and other ClientResponses. Thanks for all the feedbacks! I’ll take a shot and upload a patch for review. Jiangjie (Becket) Qin On 5/6/15, 2:19 PM, Guozhang Wang wangg...@gmail.com wrote: Jiangjie, Just trying to figure the

[jira] [Commented] (KAFKA-2172) Round-robin partition assignment strategy too restrictive

2015-05-06 Thread Bryan Baugher (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530767#comment-14530767 ] Bryan Baugher commented on KAFKA-2172: -- I originally asked out on the mailing list