[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer

2015-03-12 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1910: - Resolution: Fixed Status: Resolved (was: Patch Available) Refactor KafkaConsumer

[jira] [Resolved] (KAFKA-1969) NPE in unit test for new consumer

2015-03-12 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1969. -- Resolution: Fixed Assignee: Guozhang Wang NPE in unit test for new consumer

[jira] [Commented] (KAFKA-2003) Add upgrade tests

2015-03-12 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355768#comment-14355768 ] Gwen Shapira commented on KAFKA-2003: - [~anigam] I just took a look at KAFKA-1888 and

Re: Review Request 31927: Patch for KAFKA-1461

2015-03-12 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31927/#review76075 --- Thanks for the patch. I do see the unit test time goes down from 12

[jira] [Commented] (KAFKA-1930) Move server over to new metrics library

2015-03-12 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355319#comment-14355319 ] Gwen Shapira commented on KAFKA-1930: - 1. Completely agree about the client library

[jira] [Commented] (KAFKA-2009) Fix UncheckedOffset.removeOffset synchronization and trace logging issue in mirror maker

2015-03-12 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357343#comment-14357343 ] Jiangjie Qin commented on KAFKA-2009: - Updated reviewboard

[jira] [Updated] (KAFKA-2006) switch the broker server over to the new java protocol definitions

2015-03-12 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-2006: - Description: This was brought up during the the review KAFKA-1694 The latest patch for KAFKA-1694 now

[ANNOUNCEMENT] Apache Kafka 0.8.2.1 Released

2015-03-12 Thread Jun Rao
The Apache Kafka community is pleased to announce the release for Apache Kafka 0.8.2.1. The 0.8.2.1 release fixes 4 critical issues in 0.8.2.0. All of the changes in this release can be found: https://archive.apache.org/dist/kafka/0.8.2.1/RELEASE_NOTES.html Apache Kafka is high-throughput,

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Jiangjie Qin
Hey Joe Jay, Thanks for the comments on the voting thread. Since it seems we probably will have more discussion on this, I am just replying from the discussion thread here. I’ve updated the KIP page to make it less like half-baked, apologize for the rush... The contract in current KIP is: 1.

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357145#comment-14357145 ] Jun Rao commented on KAFKA-1910: [~guozhang], I still see the following transient unit

[jira] [Commented] (KAFKA-1830) Closing socket connection to /10.118.192.104. (kafka.network.Processor)

2015-03-12 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357198#comment-14357198 ] Guozhang Wang commented on KAFKA-1830: -- 0.8.2 release already changed the log level

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

2015-03-12 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357348#comment-14357348 ] Guozhang Wang commented on KAFKA-1910: -- Found the root cause: in handling

Re: Review Request 29301: Patch for KAFKA-1694

2015-03-12 Thread Andrii Biletskyi
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29301/ --- (Updated March 12, 2015, 11:04 a.m.) Review request for kafka. Bugs:

Re: Review Request 29301: Patch for KAFKA-1694

2015-03-12 Thread Andrii Biletskyi
On Feb. 3, 2015, 7:14 p.m., Guozhang Wang wrote: clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java, line 44 https://reviews.apache.org/r/29301/diff/7/?file=821315#file821315line44 How about just augmenting OFFSET_FETCH request to return offsets committed by

Re: 0.8.3 release plan

2015-03-12 Thread Harsha
I would like to include ssl/sasl 1) Kafka-1684 (Patch posted for a review) 2) Kafka-1686 (Patch depends on kafka-1684) 3) Kafka-1688 (work is in progress) Thanks, Harsha On Thu, Mar 12, 2015, at 04:35 AM, Guozhang Wang wrote: Gwen, Just for clarification, you were suggesting we should or

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Guozhang Wang
Folks, Just want to elaborate a bit more on the create-topic metadata and batching describe-topic based on config / metadata in my previous email as we work on KAFKA-1694. The main motivation is to have some sort of topic management mechanisms, which I think is quite important in a multi-tenant /

[jira] [Commented] (KAFKA-1054) Eliminate Compilation Warnings for 0.8 Final Release

2015-03-12 Thread Blake Smith (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358722#comment-14358722 ] Blake Smith commented on KAFKA-1054: To follow up on this, it looks like we can't kill

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Joe Stein
Guozhang and Tong, I really do like this idea and where your discussion will lead as it will be very useful for folks to have. I am really concerned though that we are scope creeping this KIP. Andrii is already working on following up on ~ 14 different items of feedback in regards to the core

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Tong Li
Guozhang, augmenting topic is fine, but as soon as we start doing that, other issues follow, for example, access control, who can access the topic, who can grant permissions. how the information (metadata) itself gets secured. Should the information be saved in ZK or a datastore? Will using

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Honghai Chen updated KAFKA-1646: Attachment: KAFKA-1646_20150312_200352.patch Improve consumer read performance for Windows

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359855#comment-14359855 ] Honghai Chen commented on KAFKA-1646: - Updated reviewboard

[jira] [Commented] (KAFKA-2011) Rebalance with auto.leader.rebalance.enable=false

2015-03-12 Thread K Zakee (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359954#comment-14359954 ] K Zakee commented on KAFKA-2011: Thanks Jiangjie. Though the ZK client session timeout

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359829#comment-14359829 ] Jay Kreps commented on KAFKA-1646: -- Yeah I was proposing doing this across the board

[jira] [Created] (KAFKA-2018) Add metadata to consumer registry info

2015-03-12 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-2018: Summary: Add metadata to consumer registry info Key: KAFKA-2018 URL: https://issues.apache.org/jira/browse/KAFKA-2018 Project: Kafka Issue Type: Sub-task

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359859#comment-14359859 ] Honghai Chen commented on KAFKA-1646: - Reduce the check of if OS.IsWindows from 7 to

Re: 0.8.3 release plan

2015-03-12 Thread Dana Powers
Re last beta, I think it was extremely useful. We identified big issues wrt API versioning and third-party client compatibility. Even if the server code doesnt get deployed widely, I think the beta period is still a good signal to third party client devs to rev up tests and make any updates

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-12 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359925#comment-14359925 ] Aditya Auradkar commented on KAFKA-1546: I ran a bunch of tests on my patch for

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Guozhang Wang
3) I think this is fine. 4) Hmm, error-message-only may NOT be better than blocking, as with the code written with close(=0), it will likely to just pollute the logs with repeating errors until someone gets notified. How about error-message-and-close(-1), i.e. record the error and force shutdown?

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Jun Rao
Andrii, A few more comments. 100. There are a few fields such as ReplicaAssignment, ReassignPartitionRequest, and PartitionsSerialized that are represented as a string, but contain composite structures in json. Could we flatten them out directly in the protocol definition as arrays/records?

Re: Review Request 29091: Improve 1646 fix by reduce check if Os.IsWindows

2015-03-12 Thread Qianlin Xia
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated March 13, 2015, 3:04 a.m.) Review request for kafka. Summary

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Jiangjie Qin
Hi Jay, Currently I can see use case for 4). We’ve seen some application in LinkedIn having issue of undeployment failure because of timeout, it was because many Kafka consumers were shutdown at the same time and caused serialized consumer rebalance until retries are exhausted. I guess in

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Joe Stein
Since we are for the first time defining a bunch of new request formats, I feel it is better to think through the its possible common use cases and try to incorporate them Agreed providing we are only talking about the fields and not the implementation of the functionality. I worry (only a

Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing

2015-03-12 Thread Todd Palino
I understand the desire to not bloat this one change with too much more work, and it's a good change to start with. That said, I have one note on your comments: I don't agree with this because right now you get back the current state of the partitions so you can (today) write whatever logic you

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.1 Released

2015-03-12 Thread Neha Narkhede
Thanks for driving this Jun and everyone for the contributions! On Wed, Mar 11, 2015 at 12:01 PM, Jun Rao jun...@apache.org wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 0.8.2.1. The 0.8.2.1 release fixes 4 critical issues in 0.8.2.0. All of the

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-12 Thread Todd Palino
We're getting off the KIP a little bit here, but while I understand the idea of not honoring the request timeout, I think that the broker ignoring it without telling the client is not a good implementation. It gets back to a larger discussion, which I was mentioning, of just negotiating with (or

[jira] [Updated] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2015-03-12 Thread Ashwin Jayaprakash (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Jayaprakash updated KAFKA-1716: -- Attachment: after-shutdown.log before-shutdown.log [~becket_qin] Here

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Jay Kreps
Yeah I want to second this. For the protocol we should really start by writing out the end state we want. Then we can figure out how to get there in small, reasonable steps to avoid boiling the ocean in implementation. -Jay On Thu, Mar 12, 2015 at 8:27 AM, Joe Stein joe.st...@stealth.ly wrote:

[jira] [Created] (KAFKA-2015) Enable ConsoleConsumer to use new consumer

2015-03-12 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-2015: Summary: Enable ConsoleConsumer to use new consumer Key: KAFKA-2015 URL: https://issues.apache.org/jira/browse/KAFKA-2015 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Guozhang Wang
The reason I want to bring it up sooner than later is that future changing a defined request protocol takes quite some effort: we need to bump up the version of the request, bump up the ZK path data version, and make sure server can handle old versions as well as new ones both from clients and

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-12 Thread Jay Kreps
Actually I think we are making the same argument. request.timeout.ms seems like it is a timeout after which your request will be sent back as an error. Actually this is not true. There are many many things that can cause a request to be slow--sitting in queue on the server, waiting on slow I/O, GC

RE: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-12 Thread Aditya Auradkar
Hi Bhavesh, One of the ideas behind this quota proposal is that we want to avoid enforcing quotas on the client side (both producer and consumer). We want to make it easy for people to write new clients if they choose to and implementing quotas is tricky enough without having to do bug-free in

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

2015-03-12 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359004#comment-14359004 ] Ewen Cheslack-Postava commented on KAFKA-1988: -- [~noxis] Because it wasn't a

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359359#comment-14359359 ] Joe Stein commented on KAFKA-1461: -- Here is my reasoning. Say you are an operations

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359368#comment-14359368 ] Jun Rao commented on KAFKA-1461: Hmm, I don't think we removed or changed any config.

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359402#comment-14359402 ] Jun Rao commented on KAFKA-1461: [~sriharsha], for the changes in RB 31366. We need to

[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations

2015-03-12 Thread Andrii Biletskyi (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359409#comment-14359409 ] Andrii Biletskyi commented on KAFKA-1694: - Hi, The list of changes from the

[jira] [Resolved] (KAFKA-2010) unit tests sometimes are slow during shutdown

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-2010. Resolution: Duplicate This is fixed in KAFKA-1461. The unit tests time went down from 12 mins to 8.5 mins

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-12 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359361#comment-14359361 ] Aditya A Auradkar commented on KAFKA-1546: -- Updated reviewboard

Re: Review Request 31967: Patch for KAFKA-1546

2015-03-12 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31967/ --- (Updated March 12, 2015, 8:42 p.m.) Review request for kafka. Bugs:

Re: [DISCUSS] KIP 16 - Replica lag tuning

2015-03-12 Thread Joe Stein
Hi Aditya, thanks for the writeup. Lets say a broker follower goes down. And it is down for an hour or two When the broker follower comes back up it will start sending fetch requests (lets say every 2ms which would be under a configured lets say 100ms (whatever)). Then right away the brokers

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359384#comment-14359384 ] Sriharsha Chintalapani commented on KAFKA-1461: --- Updated reviewboard

Re: Review Request 31927: Patch for KAFKA-1461

2015-03-12 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31927/ --- (Updated March 12, 2015, 8:54 p.m.) Review request for kafka. Bugs:

Re: Review Request 31927: Patch for KAFKA-1461

2015-03-12 Thread Sriharsha Chintalapani
On March 12, 2015, 8:42 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/AbstractFetcherThread.scala, lines 105-106 https://reviews.apache.org/r/31927/diff/2/?file=891478#file891478line105 This needs to be inside the partitionMapLock. Also, typo occured. Sorry about this I

RE: [DISCUSS] KIP 16 - Replica lag tuning

2015-03-12 Thread Aditya Auradkar
I will change the wording to reflect this. But yes, a broker follower should only enter the ISR once it is fully caught up. Caught up means that the follower has read from the log end offset from the broker. I'm using the log end offset from before the actual read operation to avoid these off

[DISCUSS] KIP 16 - Replica lag tuning

2015-03-12 Thread Aditya Auradkar
I wrote a KIP for this after some discussion on KAFKA-1546. https://cwiki.apache.org/confluence/display/KAFKA/KIP+16+:+Automated+Replica+Lag+Tuning The RB is here: https://reviews.apache.org/r/31967/ Thanks, Aditya

[jira] [Updated] (KAFKA-1546) Automate replica lag tuning

2015-03-12 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-1546: - Attachment: KAFKA-1546_2015-03-12_13:42:01.patch Automate replica lag tuning

Re: Review Request 31927: Patch for KAFKA-1461

2015-03-12 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31927/#review76282 --- Thanks for the patch. Looks good to me except for the following.

Re: Review Request 31967: Patch for KAFKA-1546

2015-03-12 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31967/ --- (Updated March 12, 2015, 8:42 p.m.) Review request for kafka. Bugs:

[jira] [Comment Edited] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359359#comment-14359359 ] Joe Stein edited comment on KAFKA-1461 at 3/12/15 8:42 PM: --- Here

[jira] [Issue Comment Deleted] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1461: - Comment: was deleted (was: Here is my reasoning. Say you are an operations person. And, in the next

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-12 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359371#comment-14359371 ] Aditya Auradkar commented on KAFKA-1546: [~jkreps]I'm going to test it like you

[jira] [Updated] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1461: -- Attachment: KAFKA-1461_2015-03-12_13:54:51.patch Replica fetcher thread does

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359394#comment-14359394 ] Jun Rao commented on KAFKA-1461: [~sriharsha], thanks for the patch. +1 on RB 31927 after

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359427#comment-14359427 ] Jun Rao commented on KAFKA-1461: Also, in AbstractFetcherThread, we probably should use

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359469#comment-14359469 ] Sriharsha Chintalapani commented on KAFKA-1461: --- Thanks [~junrao] I'll

Re: 0.8.3 release plan

2015-03-12 Thread Jun Rao
Since we have decided to only support security on the new clients, the security feature will only be available after the new consumer is done. So, for the next release, we probably should just target finishing the new consumer as the main feature. If security can be done in the same release,

[jira] [Commented] (KAFKA-1461) Replica fetcher thread does not implement any back-off behavior

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359316#comment-14359316 ] Jun Rao commented on KAFKA-1461: [~charmalloc], I don't think we need a KIP for this

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

2015-03-12 Thread Wojciech Kuranowski (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359229#comment-14359229 ] Wojciech Kuranowski commented on KAFKA-1988: Please update this issue.

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

2015-03-12 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1988: - Fix Version/s: (was: 0.8.2.1) 0.8.3

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication

2015-03-12 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359151#comment-14359151 ] Gwen Shapira commented on KAFKA-1684: - Thank you so much for going ahead with this.

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Jiangjie Qin
Hey Guozhang, Thanks for the comments. I updated the page as suggested. For 3), that’s right, I put this in java doc. Do you think we need to reject value other than -1? For 4), I think user will notice this easily because the thread will block and producer is not going to shutdown. About using

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Gwen Shapira
Found KIP-11 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface) It actually specifies changes to the Metadata protocol, so making sure both KIPs are consistent in this regard will be good. On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira gshap...@cloudera.com

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Gwen Shapira
Specifically for ownership, I think the plan is to add ACL (it sounds like you are describing ACL) via an external system (Argus, Sentry). I remember KIP-11 described this, but I can't find the KIP any longer. Regardless, I think KIP-4 focuses on getting information that already exists from Kafka

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

2015-03-12 Thread Wojciech Kuranowski (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358613#comment-14358613 ] Wojciech Kuranowski commented on KAFKA-1988: Why this fix is not included in

Re: Review Request 31806: Patch for KAFKA-1501

2015-03-12 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31806/#review76190 --- The patch does not apply, could you rebase?

Re: Review Request 31706: Patch for KAFKA-1997

2015-03-12 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31706/#review76226 --- Ship it! LGTM. Joel, do you want to take another look at it? -

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359658#comment-14359658 ] Honghai Chen commented on KAFKA-1646: - Hey [~jkreps] We have ask the NTFS owners,

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

2015-03-12 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359678#comment-14359678 ] Ewen Cheslack-Postava commented on KAFKA-1988: -- [~tongli] You're all good, it

[jira] [Resolved] (KAFKA-2011) Rebalance with auto.leader.rebalance.enable=false

2015-03-12 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin resolved KAFKA-2011. - Resolution: Won't Fix Rebalance with auto.leader.rebalance.enable=false

[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359497#comment-14359497 ] Jun Rao commented on KAFKA-902: --- Thanks for the patch. A few comments. 1. I feel that it's

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359506#comment-14359506 ] Jun Rao commented on KAFKA-1910: In BounceBrokerScheduler, is there a particular reason

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Andrii Biletskyi
Hi, As said above - I list again all comments from this thread so we can see what's left and finalize all pending issues. Comments from Jay: 1. This is much needed functionality, but there are a lot of the so let's really think these protocols through. We really want to end up with a set of well

Build failed in Jenkins: KafkaPreCommit #37

2015-03-12 Thread Apache Jenkins Server
See https://builds.apache.org/job/KafkaPreCommit/37/changes Changes: [junrao] kafka-1461; Replica fetcher thread does not implement any back-off behavior; patched by Sriharsha Chintalapani; reviewed by Jun Rao -- [...truncated 599 lines...]

[jira] [Commented] (KAFKA-2011) Rebalance with auto.leader.rebalance.enable=false

2015-03-12 Thread K Zakee (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359513#comment-14359513 ] K Zakee commented on KAFKA-2011: Increasing the timeout (to 60ms) has shown

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Guozhang Wang
Gwen, My main motivation is not authenticate via ownership, but rather query topics via ownership, and more generally query topics via patterns, where a pattern could be a config value, metadata k-v pair, etc. Does it make sense? Guozhang On Thu, Mar 12, 2015 at 12:26 PM, Gwen Shapira

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-12 Thread Andrii Biletskyi
Hi all, Today I uploaded the patch that covers some of the discussed and agreed items: - removed MaybeOf optional type - switched to java protocol definitions - simplified messages (normalized configs, removed topic marked for deletion) I also updated the KIP-4 with respective changes and wrote

[jira] [Created] (KAFKA-2016) RollingBounceTest takes long

2015-03-12 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-2016: -- Summary: RollingBounceTest takes long Key: KAFKA-2016 URL: https://issues.apache.org/jira/browse/KAFKA-2016 Project: Kafka Issue Type: Improvement Reporter:

[jira] [Commented] (KAFKA-2016) RollingBounceTest takes long

2015-03-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359556#comment-14359556 ] Jun Rao commented on KAFKA-2016: Perhaps we can first figure out where the time is spent

[jira] [Created] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

2015-03-12 Thread Onur Karaman (JIRA)
Onur Karaman created KAFKA-2017: --- Summary: Persist Coordinator State for Coordinator Failover Key: KAFKA-2017 URL: https://issues.apache.org/jira/browse/KAFKA-2017 Project: Kafka Issue Type:

[jira] [Commented] (KAFKA-2011) Rebalance with auto.leader.rebalance.enable=false

2015-03-12 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359549#comment-14359549 ] Jiangjie Qin commented on KAFKA-2011: - On the server side, I would suggest keeping

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Guozhang Wang
Hi Becket, Some comments on the wiki page: 1. There are a few typos, for example multivations, wiithin. 2. I think the main motivation could just be current close() needs to block on flushing all buffered data, however there are scenarios in which producers would like to close without blocking

Re: 0.8.3 release plan

2015-03-12 Thread Guozhang Wang
Gwen, Just for clarification, you were suggesting we should or should not include MM improvement in 0.8.3? I personally would prefer it (KAFKA-1650 and KAFKA-1997) to go into 0.8.3. I see Joe has made a pass over the tickets and mark them 0.8.3. We can probably do another pass and consider

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication

2015-03-12 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357665#comment-14357665 ] Sriharsha Chintalapani commented on KAFKA-1684: --- [~junrao] [~jkreps]

[jira] [Commented] (KAFKA-1694) kafka command line and centralized operations

2015-03-12 Thread Andrii Biletskyi (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358489#comment-14358489 ] Andrii Biletskyi commented on KAFKA-1694: - Updated reviewboard

[jira] [Updated] (KAFKA-1694) kafka command line and centralized operations

2015-03-12 Thread Andrii Biletskyi (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Biletskyi updated KAFKA-1694: Attachment: KAFKA-1694_2015-03-12_13:04:37.patch kafka command line and centralized

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-12 Thread Jiangjie Qin
Hi folks, Since there are actually a long interleaved discussions in KAFKA-1660 and KAFKA-1659, I summarized them in the rejected alternatives sections - the section exists for a reason and I shouldn’t have always put None there. . . It looks we took a while to agree upon the current interface.

Re: [jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-03-12 Thread Ewen Cheslack-Postava
I think the real problem is that the conditionals are spread across a bunch of different files. Seven special cases isn't necessarily unsustainable, but is difficult to manage now because they aren't isolated to one location. I think platform specific customizations are sustainable (as long as

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-12 Thread Bhavesh Mistry
Hi Aditya, I just wanted to give you use case of rate limiting that we have implemented with producer which is a work around: Use Case 1: 1) topic based rate limiting per producer instance (not across multiple instance of producers yet, we have producer which we send Heartbeat and regular

[jira] [Created] (KAFKA-2013) benchmark test for the purgatory

2015-03-12 Thread Yasuhiro Matsuda (JIRA)
Yasuhiro Matsuda created KAFKA-2013: --- Summary: benchmark test for the purgatory Key: KAFKA-2013 URL: https://issues.apache.org/jira/browse/KAFKA-2013 Project: Kafka Issue Type: Test