[jira] [Commented] (KAFKA-702) Deadlock between request handler/processor threads

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554807#comment-13554807 ] Neha Narkhede commented on KAFKA-702: - Agree that a longer term fix is to be able to bo

[jira] [Commented] (KAFKA-700) log client ip when we log each request on the broker

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554773#comment-13554773 ] Jun Rao commented on KAFKA-700: --- I was thinking of just putting the ip in RequestChannel.Requ

[jira] [Commented] (KAFKA-700) log client ip when we log each request on the broker

2013-01-15 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554769#comment-13554769 ] Sriram Subramanian commented on KAFKA-700: -- You want the ip in RequestOrReponse ob

[jira] [Updated] (KAFKA-702) Deadlock between request handler/processor threads

2013-01-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-702: Attachment: KAFKA-702-v1.patch TLDR: attached one-line patch should fix the deadlock, but there is a larger

[jira] [Updated] (KAFKA-699) Disallow clients to set replicaId in FetchRequest

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-699: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the r

[jira] [Commented] (KAFKA-700) log client ip when we log each request on the broker

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554755#comment-13554755 ] Jun Rao commented on KAFKA-700: --- This doesn't seem too hard. In Processor.read(), we can add

[jira] [Issue Comment Deleted] (KAFKA-699) Disallow clients to set replicaId in FetchRequest

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-699: -- Comment: was deleted (was: This doesn't seem too hard. In Processor.read(), we can add socketChannel.socket.get

[jira] [Resolved] (KAFKA-698) broker may expose uncommitted data to a consumer

2013-01-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-698. - Resolution: Fixed Fix Version/s: 0.8 > broker may expose uncommitted data to a consumer > -

[jira] [Commented] (KAFKA-699) Disallow clients to set replicaId in FetchRequest

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554590#comment-13554590 ] Neha Narkhede commented on KAFKA-699: - +1. Changes look good > Disallo

[jira] [Comment Edited] (KAFKA-652) Create testcases for clean shut-down

2013-01-15 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510791#comment-13510791 ] John Fung edited comment on KAFKA-652 at 1/16/13 12:36 AM: --- This

[jira] [Updated] (KAFKA-652) Create testcases for clean shut-down

2013-01-15 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-652: Attachment: kafka-652-v1.patch Uploaded kafka-652-v1.patch which contains the following testcases: testcase

[jira] [Updated] (KAFKA-696) Fix toString() API for all requests to make logging easier to read

2013-01-15 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-696: - Attachment: cleanup-v1.patch - Added ToString overrides to all the request object to make

[jira] [Updated] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-705: Attachment: shutdown-command shutdown_brokers_eat.py Run it with the --help option t

[jira] [Created] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

2013-01-15 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-705: --- Summary: Controlled shutdown doesn't seem to work on more than one broker in a cluster Key: KAFKA-705 URL: https://issues.apache.org/jira/browse/KAFKA-705 Project: Kafk

[jira] [Commented] (KAFKA-347) change number of partitions of a topic online

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554458#comment-13554458 ] Maxime Brugidou commented on KAFKA-347: --- Agreed. Except you also have to think about

[jira] [Assigned] (KAFKA-703) A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-703: --- Assignee: Sriram Subramanian > A fetch request in Fetch Purgatory can double count the byt

Re: Need mailing list moderators

2013-01-15 Thread Jun Rao
Sounds good. Thanks, Jun On Tue, Jan 15, 2013 at 11:31 AM, Alan Cabrera wrote: > Maybe this is only an issue for the private mailing list. May I add you > two? > > > Regards, > Alan > > On Jan 15, 2013, at 9:34 AM, Neha Narkhede wrote: > > > I am also moderating the mailing list. > > > > Than

[jira] [Updated] (KAFKA-631) Implement log compaction

2013-01-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-631: Attachment: KAFKA-631-v1.patch This patch implements more or less what was described above. Specific Change

Re: Need mailing list moderators

2013-01-15 Thread Neha Narkhede
Sure. Thanks, Neha On Tue, Jan 15, 2013 at 11:31 AM, Alan Cabrera wrote: > Maybe this is only an issue for the private mailing list. May I add you > two? > > > Regards, > Alan > > On Jan 15, 2013, at 9:34 AM, Neha Narkhede wrote: > > > I am also moderating the mailing list. > > > > Thanks, >

Re: Need mailing list moderators

2013-01-15 Thread Alan Cabrera
Maybe this is only an issue for the private mailing list. May I add you two? Regards, Alan On Jan 15, 2013, at 9:34 AM, Neha Narkhede wrote: > I am also moderating the mailing list. > > Thanks, > Neha > > > On Tue, Jan 15, 2013 at 7:48 AM, Jun Rao wrote: > >> I am already moderating the m

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

2013-01-15 Thread Neha Narkhede
>When the producer >tries to send to the old broker (which is either dead, or a slave now), >the broker will either not respond, or the response will contain an error >code. In this case, the broker sends a response with an error code to the producer, and then the producer retries the metadata r

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

2013-01-15 Thread Chris Riccomini
Hey Guys, Correct me if I'm wrong, but I believe in 0.8, the producer uses a metadata request to get topic/partition mappings from the broker. The broker then interacts with ZK (rather than having the producer do it using zk.connect). In the event that the master for a topic/partition fails, a ne

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

2013-01-15 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554178#comment-13554178 ] Sriram Subramanian commented on KAFKA-671: -- 2. I am not sure if it is an issue in

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554172#comment-13554172 ] Neha Narkhede commented on KAFKA-671: - 2. You maybe right. However, given that you set

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

2013-01-15 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554158#comment-13554158 ] Sriram Subramanian commented on KAFKA-671: -- 2. I don't think so since DelayedProdu

[jira] [Comment Edited] (KAFKA-671) DelayedProduce requests should not hold full producer request data

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554142#comment-13554142 ] Neha Narkhede edited comment on KAFKA-671 at 1/15/13 6:55 PM: --

[jira] [Commented] (KAFKA-671) DelayedProduce requests should not hold full producer request data

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554142#comment-13554142 ] Neha Narkhede commented on KAFKA-671: - Overall, the changes look good. Minor suggestion

[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554068#comment-13554068 ] Jun Rao commented on KAFKA-693: --- Thanks for the patch. Some comments: 10. ZookeeperConsumerC

[jira] [Commented] (KAFKA-347) change number of partitions of a topic online

2013-01-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554062#comment-13554062 ] Jay Kreps commented on KAFKA-347: - Well i think there are really two mappings here: key =>

Re: About kafka 0.8 producer auto detect broker

2013-01-15 Thread Neha Narkhede
> Can I guess that the auto detecting ability for producer is weakened in > Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8? In 0.8, we got rid of zookeeper from the producer and replaced its functionality with a metadata API. On startup and whenever a request fails, the produc

[jira] [Closed] (KAFKA-704) ConsumerFetcherThread can create illegal clientId

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-704. --- Fixed by https://issues.apache.org/jira/browse/KAFKA-697 > ConsumerFetcherThread can crea

[jira] [Closed] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-697. --- > ConsoleConsumer throws InvalidConfigException for "." in client id > ---

[jira] [Commented] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554048#comment-13554048 ] Jun Rao commented on KAFKA-697: --- Thanks for the latest patch. Committed to 0.8 and merged to

[jira] [Resolved] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-697. - Resolution: Fixed Committed patch to 0.8 branch > ConsoleConsumer throws InvalidC

[jira] [Commented] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554035#comment-13554035 ] Neha Narkhede commented on KAFKA-697: - +1 on the new patch, thanks Swapnil!

Re: Need mailing list moderators

2013-01-15 Thread Neha Narkhede
I am also moderating the mailing list. Thanks, Neha On Tue, Jan 15, 2013 at 7:48 AM, Jun Rao wrote: > I am already moderating the mailing list. > > Thanks, > > Jun > > On Tue, Jan 15, 2013 at 7:37 AM, Alan Cabrera wrote: > > > It seems that I am the only moderator for the Kafka mailing lists.

[jira] [Commented] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554000#comment-13554000 ] John Fung commented on KAFKA-697: - Thanks Swapnil for the latest patch. The patch is sanit

[jira] [Updated] (KAFKA-703) A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-703: -- Priority: Blocker (was: Major) > A fetch request in Fetch Purgatory can double count the bytes from the sam

[jira] [Resolved] (KAFKA-704) ConsumerFetcherThread can create illegal clientId

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-704. --- Resolution: Duplicate This is a duplicate of KAFKA-697. > ConsumerFetcherThread can create il

[jira] [Commented] (KAFKA-697) ConsoleConsumer throws InvalidConfigException for "." in client id

2013-01-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553968#comment-13553968 ] Jun Rao commented on KAFKA-697: --- The new patch looks good to me. We can just apply this patch

Re: Need mailing list moderators

2013-01-15 Thread Jun Rao
I am already moderating the mailing list. Thanks, Jun On Tue, Jan 15, 2013 at 7:37 AM, Alan Cabrera wrote: > It seems that I am the only moderator for the Kafka mailing lists. We > need some volunteers from the PMC to take my place, at least two. > > > Regards, > Alan > > >

Need mailing list moderators

2013-01-15 Thread Alan Cabrera
It seems that I am the only moderator for the Kafka mailing lists. We need some volunteers from the PMC to take my place, at least two. Regards, Alan

Re: About kafka 0.8 producer auto detect broker

2013-01-15 Thread Jun Rao
That's an experiment for a future release. The consumer logic in 0.8 is more or less the same as 0.7, ie, running in a decentralized way. Thanks, Jun On Tue, Jan 15, 2013 at 12:10 AM, gj1989lh wrote: > Hi, > How is the consumer rebalancing works? > In the design part of the official document,

[jira] [Commented] (KAFKA-347) change number of partitions of a topic online

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553873#comment-13553873 ] Maxime Brugidou commented on KAFKA-347: --- It's sort of out of scope but I think that u

[jira] [Resolved] (KAFKA-155) Support graceful Decommissioning of Broker

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou resolved KAFKA-155. --- Resolution: Fixed Fix Version/s: 0.8 > Support graceful Decommissioning of Broker >

[jira] [Commented] (KAFKA-155) Support graceful Decommissioning of Broker

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553867#comment-13553867 ] Maxime Brugidou commented on KAFKA-155: --- I think this is done in 0.8 branch with the

Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis

2013-01-15 Thread Jun Rao
All apis for 0.8 are documented in https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol However, there is no need for a producer to call the metadata api directly. Our producer client does this for you automatically. For your first question, if a new broker is added, cu

About kafka 0.8 producer auto detect broker

2013-01-15 Thread gj1989lh
Hi, How is the consumer rebalancing works? In the design part of the official document, it says each consumer decides which partitions it should own. But in below document https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination It says there will be a centralized consumer c

[jira] [Created] (KAFKA-704) ConsumerFetcherThread can create illegal clientId

2013-01-15 Thread ben fleis (JIRA)
ben fleis created KAFKA-704: --- Summary: ConsumerFetcherThread can create illegal clientId Key: KAFKA-704 URL: https://issues.apache.org/jira/browse/KAFKA-704 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Assignee: Maxime Brugidou > Consumer rebalance fails if no leader available for a partition

[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Status: Patch Available (was: Open) > Consumer rebalance fails if no leader available for a

[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553751#comment-13553751 ] Maxime Brugidou commented on KAFKA-693: --- Here is a patch: 1. AbstractFetchThread.add

[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-15 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: KAFKA-693.patch > Consumer rebalance fails if no leader available for a partitio