Re: What's new in Kafka 0.8.1?

Thanks for the writeup. In 0.8.1, we also significantly improved the partition reassignment tool. Jun On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps wrote: > Hi guys, > > I wrote up a quick blog post on the new stuff in 0.8.1: > > http://blog.empathybox.com/post/79427855885/whats-new-in-kafka-0-8-

Broker side consume-request filtering

Hi, I have a use case that might be relevant to the new consumer API but that would require most of the work on the broker. I would be surprised if it had not been discussed before but I was not able to find any directly related thread. Has there been any discussion about providing broker side con

Re: Review Request 19132: Patch for KAFKA-1302

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19132/ --- (Updated March 13, 2014, 6 p.m.) Review request for kafka. Bugs: KAFKA-1302

Re: Review Request 19132: Patch for KAFKA-1302

> On March 12, 2014, 10:05 p.m., Guozhang Wang wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java, > > line 368 > > > > > > Would suggest to add similar logging here, e.g.: > >

Re: Review Request 19132: Patch for KAFKA-1302

> On March 12, 2014, 6:19 p.m., Jay Kreps wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, > > line 236 > > > > > > Jun, the way this works is intentional and I do not agree with

[jira] [Updated] (KAFKA-1302) cleanup logging in new producer

[ https://issues.apache.org/jira/browse/KAFKA-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1302: --- Attachment: KAFKA-1302_2014-03-13_11:00:33.patch > cleanup logging in new producer >

[jira] [Commented] (KAFKA-1302) cleanup logging in new producer

[ https://issues.apache.org/jira/browse/KAFKA-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933646#comment-13933646 ] Jun Rao commented on KAFKA-1302: Updated reviewboard https://reviews.apache.org/r/19132/

Broker plugins

Hi, I have a use case for which it would be useful with pluggable processing functions in the broker. We have some data containing sensitive information which is legally ok to transmit over the internal network to the Kafka brokers and keep in volatile memory but not to flush to disk unconcealed/

Re: Review Request 19132: Patch for KAFKA-1302

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19132/#review37078 --- Ship it! Minor comment below clients/src/main/java/org/apache/kaf

[jira] [Updated] (KAFKA-1304) unregistered mbean exception in new producer

[ https://issues.apache.org/jira/browse/KAFKA-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1304: - Issue Type: Sub-task (was: Bug) Parent: KAFKA-1239 > unregistered mbean exception in new

Re: Review Request 19132: Patch for KAFKA-1302

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19132/#review37088 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/S

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (85 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1302 cleanup logging in new producer https://issues.apache.org/jira/browse/KAFKA-1302 KAFKA-1300 Added WaitForReplaction adm

Re: Review Request 19132: Patch for KAFKA-1302

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19132/#review37091 --- Ship it! I'm +1 with that one minor comment I gave... - Jay Kreps

reviewer for KAFKA-1028

Do we have a reviewer for https://issues.apache.org/jira/browse/KAFKA-1028 -Jay

Re: Broker plugins

In general, the preference has been to avoid having user code run on the brokers since that just opens a can of worms where the broker logic get's complicated trying to deal with errors that the user code can throw. The suggestion is to push any user specific processing to the client side. In this

[jira] [Commented] (KAFKA-1006) Consumer loses messages of a new topic with auto.offset.reset = largest

[ https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933999#comment-13933999 ] Neha Narkhede commented on KAFKA-1006: -- Instead of adding a new property, we can trea

[jira] [Updated] (KAFKA-1006) Consumer loses messages of a new topic with auto.offset.reset = largest

[ https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1006: - Labels: usability (was: ) > Consumer loses messages of a new topic with auto.offset.reset = larg

Re: reviewer for KAFKA-1028

I've been reviewing it so far but haven't gotten a chance to look at this last week. Will take a look today. On Thu, Mar 13, 2014 at 12:16 PM, Jay Kreps wrote: > Do we have a reviewer for > https://issues.apache.org/jira/browse/KAFKA-1028 > > -Jay >

Re: Broker plugins

Thanks for the response, indeed the encryption/concealment should ideally be done on the producer side but it's just not feasible in some of our applications so a middle layer would be needed. So far our thoughts have been around using Flume interceptors but that means introducing another "moving

[jira] [Resolved] (KAFKA-1302) cleanup logging in new producer

[ https://issues.apache.org/jira/browse/KAFKA-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1302. Resolution: Fixed Fix Version/s: 0.8.2 Thanks for the reviews. Committed to trunk. > cleanup loggin

Re: Review Request 18774: Patch for KAFKA-1240

> On March 7, 2014, 6:48 p.m., Jay Kreps wrote: > > core/src/main/scala/kafka/utils/Utils.scala, line 545 > > > > > > This cannot go in utils and utils cannot depend on the producer (for > > obvious reasons). This is

Re: Review Request 18774: Patch for KAFKA-1240

> On March 8, 2014, 12:48 a.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/utils/Utils.scala, line 545 > > > > > > Since this logging entry is from the Utils we will not know the id of > > the producer either

Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18022/ --- (Updated March 13, 2014, 11:03 p.m.) Review request for kafka. Changes --

Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18022/#review37130 --- Ship it! Minor review comment below core/src/main/scala/kafka/con

Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

> On March 13, 2014, 11:25 p.m., Neha Narkhede wrote: > > core/src/main/scala/kafka/consumer/ConsoleConsumer.scala, line 132 > > > > > > Should this default to true if offsets-storage is set to kafka? Right now it i

Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

> On March 13, 2014, 11:25 p.m., Neha Narkhede wrote: > > core/src/main/scala/kafka/consumer/ConsoleConsumer.scala, line 132 > > > > > > Should this default to true if offsets-storage is set to kafka? > > Joel Koshy

[jira] [Updated] (KAFKA-1305) Controller can hang on controlled shutdown with auto leader balance enabled

[ https://issues.apache.org/jira/browse/KAFKA-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1305: -- Description: This is relatively easy to reproduce especially when doing a rolling bounce. What happened

[jira] [Created] (KAFKA-1305) Controller can hang on controlled shutdown with auto leader balance enabled

Joel Koshy created KAFKA-1305: - Summary: Controller can hang on controlled shutdown with auto leader balance enabled Key: KAFKA-1305 URL: https://issues.apache.org/jira/browse/KAFKA-1305 Project: Kafka

[jira] [Assigned] (KAFKA-1305) Controller can hang on controlled shutdown with auto leader balance enabled

[ https://issues.apache.org/jira/browse/KAFKA-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-1305: Assignee: Neha Narkhede > Controller can hang on controlled shutdown with auto leader balan

[jira] [Commented] (KAFKA-657) Add an API to commit offsets

[ https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934469#comment-13934469 ] korebantic2 commented on KAFKA-657: --- All, I'm looking to update a client to support this

[jira] [Comment Edited] (KAFKA-657) Add an API to commit offsets

[ https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934469#comment-13934469 ] korebantic2 edited comment on KAFKA-657 at 3/14/14 3:20 AM: All