Re: offset storage as kafka with zookeeper 3.4.6

Is it mandatory to use the zookeeper that comes with kafka for offset storage to be migrated to kafka? If you want to move offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group then

[jira] [Created] (KAFKA-2263) Update Is it possible to delete a topic wiki FAQ answer

Stevo Slavic created KAFKA-2263: --- Summary: Update Is it possible to delete a topic wiki FAQ answer Key: KAFKA-2263 URL: https://issues.apache.org/jira/browse/KAFKA-2263 Project: Kafka Issue

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Hi Guys, This has been lying around for quite some time. Should I start a voting thread on this? On Thu, May 7, 2015 at 12:20 PM, Ashish Singh asi...@cloudera.com wrote: Had to change the title of the page and that surprisingly changed the link as well. KIP-23 is now available at here

Re: Review Request 34789: Patch for KAFKA-2168

On June 9, 2015, 7:58 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, lines 797-798 https://reviews.apache.org/r/34789/diff/8/?file=980075#file980075line797 Hmm, seekToBegining() is supposed to be a blocking call. Basically, at

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Discussion aside, was there any significant material change besides the additions below? If so, then we can avoid the overhead of another vote unless someone wants to down-vote these changes. Joel On Thu, Jun 11, 2015 at 06:36:36PM +, Aditya Auradkar wrote: Andrii, Do we need a new

Pending review requests

Hey Guys, I have a few JIRAs in patch available state for some time. I will really appreciate if someone can review them. 1. https://issues.apache.org/jira/browse/KAFKA-2132 2. https://issues.apache.org/jira/browse/KAFKA-2005 3. https://issues.apache.org/jira/browse/KAFKA-1722 4.

Re: Review Request 34789: Patch for KAFKA-2168

On June 3, 2015, 6:44 a.m., Ewen Cheslack-Postava wrote: clients/src/main/java/org/apache/kafka/clients/consumer/internals/BrokerResponse.java, line 15 https://reviews.apache.org/r/34789/diff/3/?file=976967#file976967line15 The classes named XResponse may be a bit confusing

[jira] [Updated] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2168: --- Attachment: KAFKA-2168_2015-06-11_14:09:59.patch New consumer poll() can block other calls

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582524#comment-14582524 ] Jason Gustafson commented on KAFKA-2168: Updated reviewboard

Re: Review Request 34789: Patch for KAFKA-2168

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/ --- (Updated June 11, 2015, 9:10 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582528#comment-14582528 ] Jay Kreps commented on KAFKA-2168: -- Hey [~guozhang], have you had a chance to look at

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582532#comment-14582532 ] Ashish K Singh commented on KAFKA-1367: --- [~jjkoshy] thanks for confirming. I will

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Maybe bring it up at the next KIP call, to make sure everyone is aware? On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh asi...@cloudera.com wrote: Hi Guys, This has been lying around for quite some time. Should I start a voting thread on this? On Thu, May 7, 2015 at 12:20 PM, Ashish Singh

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Jun, Can we add this as part of next KIP's agenda? On Thu, Jun 11, 2015 at 3:00 PM, Gwen Shapira gshap...@cloudera.com wrote: Maybe bring it up at the next KIP call, to make sure everyone is aware? On Thu, Jun 11, 2015 at 2:17 PM, Ashish Singh asi...@cloudera.com wrote: Hi Guys, This

[jira] [Updated] (KAFKA-2005) Generate html report for system tests

[ https://issues.apache.org/jira/browse/KAFKA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2005: --- Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Thanks for

[jira] [Updated] (KAFKA-2249) KafkaConfig does not preserve original Properties

[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2249: --- Status: In Progress (was: Patch Available) KafkaConfig does not preserve original Properties

[jira] [Updated] (KAFKA-2132) Move Log4J appender to a separate module

[ https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2132: Summary: Move Log4J appender to a separate module (was: Move Log4J appender to clients module)

[jira] [Commented] (KAFKA-2005) Generate html report for system tests

[ https://issues.apache.org/jira/browse/KAFKA-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582646#comment-14582646 ] Ashish K Singh commented on KAFKA-2005: --- Thanks [~junrao] for reviewing and

Re: Review Request 35347: Patch for KAFKA-2249

On June 11, 2015, 11:22 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaConfig.scala, line 485 https://reviews.apache.org/r/35347/diff/1/?file=982452#file982452line485 Is there a particular reason to change this to a long? It is used as LONG everywhere in the code. On

Re: Review Request 33614: Patch for KAFKA-2132

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33614/#review87638 --- Hey Ashish, I've left a few minor comments. Thanks!

[jira] [Commented] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)

[ https://issues.apache.org/jira/browse/KAFKA-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582698#comment-14582698 ] Gwen Shapira commented on KAFKA-2238: - [~aauradkar] I made significant changes to

[jira] [Commented] (KAFKA-2249) KafkaConfig does not preserve original Properties

[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581505#comment-14581505 ] Gwen Shapira commented on KAFKA-2249: - Created reviewboard

[jira] [Updated] (KAFKA-2249) KafkaConfig does not preserve original Properties

[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2249: Assignee: Gwen Shapira Status: Patch Available (was: Open) KafkaConfig does not preserve

[jira] [Updated] (KAFKA-2249) KafkaConfig does not preserve original Properties

[ https://issues.apache.org/jira/browse/KAFKA-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2249: Attachment: KAFKA-2249.patch KafkaConfig does not preserve original Properties

Re: Review Request 34492: Patch for KAFKA-2210

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34492/#review87532 --- core/src/main/scala/kafka/server/KafkaConfig.scala

Review Request 35347: Patch for KAFKA-2249

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35347/ --- Review request for kafka. Bugs: KAFKA-2249

Re: Review Request 35261: Patch for KAFKA-2232

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35261/#review87661 --- Thanks for the patch. A few comments below.

Re: [Discussion] New Consumer API / Protocol

Guozhang, Perhaps we can discuss this in our KIP hangout next week? Thanks, Jun On Tue, Jun 9, 2015 at 1:12 PM, Guozhang Wang wangg...@gmail.com wrote: This email is to kick-off some discussion around the changes we want to make on the new consumer APIs as well as their semantics. Here are

Re: Review Request 33614: Patch for KAFKA-2132

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33614/#review87670 --- Thanks for the patch. A few comments below. build.gradle

[jira] [Updated] (KAFKA-2132) Move Log4J appender to a separate module

[ https://issues.apache.org/jira/browse/KAFKA-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2132: --- Status: In Progress (was: Patch Available) Move Log4J appender to a separate module

Re: [DISCUSS] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

Thanks for submitting the KIP, Ashish! Few questions. 1. Can you specify more details around how you expect csv output to be used. Same for json. 2. If we add these options, would you still need the old format. If csv/json offers more convenience, should we have a plan to phase out the old

[jira] [Commented] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)

[ https://issues.apache.org/jira/browse/KAFKA-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582722#comment-14582722 ] Jun Rao commented on KAFKA-2238: [~auradkar], took a look at your patch. Currently, the

Re: [VOTE] KIP-25 System test improvements

+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Kafka, After a few rounds of discussion on KIP-25, there doesn't seem to be opposition, so I'd like to propose a vote. Thanks, Geoff On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson

[jira] [Resolved] (KAFKA-2263) Update Is it possible to delete a topic wiki FAQ answer

[ https://issues.apache.org/jira/browse/KAFKA-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-2263. Resolution: Fixed Thanks for reporting this. Updated the wiki. Update Is it possible to delete a topic

[jira] [Created] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

Jason Gustafson created KAFKA-2266: -- Summary: Client Selector can drop idle connections without notifying NetworkClient Key: KAFKA-2266 URL: https://issues.apache.org/jira/browse/KAFKA-2266 Project:

Review Request 35371: Patch for KAFKA-2266

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35371/ --- Review request for kafka. Bugs: KAFKA-2266

[jira] [Updated] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2266: --- Attachment: KAFKA-2266.patch Client Selector can drop idle connections without notifying

[jira] [Updated] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2266: --- Status: Patch Available (was: Open) Client Selector can drop idle connections without

[jira] [Commented] (KAFKA-2266) Client Selector can drop idle connections without notifying NetworkClient

[ https://issues.apache.org/jira/browse/KAFKA-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582762#comment-14582762 ] Jason Gustafson commented on KAFKA-2266: Created reviewboard

Re: Review Request 35371: Patch for KAFKA-2266

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35371/#review87667 --- Thanks for the patch. Good catch! A minor comment below.

Re: [VOTE] KIP-25 System test improvements

+1 Thanks, Jun On Wed, Jun 10, 2015 at 6:10 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi Kafka, After a few rounds of discussion on KIP-25, there doesn't seem to be opposition, so I'd like to propose a vote. Thanks, Geoff On Mon, Jun 8, 2015 at 10:56 PM, Geoffrey Anderson

offset storage as kafka with zookeeper 3.4.6

I am trying to migrate the offset storage to kafka (3 brokers of version 0.8.2.1) using the consumer property offsets.storage=kafka. I noticed that a new topic, __consumer_offsets got created. But nothing is being written to this topic, while the consumer offsets continue to reside on zookeeper.

[jira] [Created] (KAFKA-2264) SESSION_TIMEOUT_MS_CONFIG in ConsumerConfig should be int

Jun Rao created KAFKA-2264: -- Summary: SESSION_TIMEOUT_MS_CONFIG in ConsumerConfig should be int Key: KAFKA-2264 URL: https://issues.apache.org/jira/browse/KAFKA-2264 Project: Kafka Issue Type: Bug

Re: offset storage as kafka with zookeeper 3.4.6

If you want to move offsets from zookeeper to Kafka then yes you need to have a phase where all consumers in your group set dual commit to true. If you are starting a fresh consumer group then you can turn off dual-commit. I followed these steps to move the offsets from zookeeper to kafka: 1. Set

Re: Review Request 35201: Fix KAFKA-2253

On June 11, 2015, 1:07 a.m., Jun Rao wrote: core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 https://reviews.apache.org/r/35201/diff/2/?file=980805#file980805line264 Not sure if we need this check. Since all writes to watchersForKey are sync-ed, it's ok to

Re: Review Request 35201: Fix KAFKA-2253

On June 11, 2015, 1:07 a.m., Jun Rao wrote: core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 https://reviews.apache.org/r/35201/diff/2/?file=980805#file980805line264 Not sure if we need this check. Since all writes to watchersForKey are sync-ed, it's ok to

RE: [DISCUSS] KIP-25 System test improvements

Hi Geoffrey, Thanks for the writeup. Couple of questions: - Is it possible to configure suites using ducktape? For example: assume all the tests in system_tests have been migrated to ducktape. Can I run a subset of all tests grouped by functional areas i.e. replication, broker failure etc. -

[jira] [Updated] (KAFKA-2136) Client side protocol changes to return quota delays

[ https://issues.apache.org/jira/browse/KAFKA-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2136: --- Labels: quotas (was: ) Client side protocol changes to return quota delays

[jira] [Updated] (KAFKA-2171) System Test for Quotas

[ https://issues.apache.org/jira/browse/KAFKA-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2171: --- Labels: quotas (was: ) System Test for Quotas --

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[ https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2084: --- Labels: quotas (was: ) byte rate metrics per client ID (producer and consumer)

[jira] [Updated] (KAFKA-2241) AbstractFetcherThread.shutdown() should not block on ReadableByteChannel.read(buffer)

[ https://issues.apache.org/jira/browse/KAFKA-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2241: --- Labels: quotas (was: ) AbstractFetcherThread.shutdown() should not block on

[jira] [Updated] (KAFKA-2191) Measured rate should not be infinite

[ https://issues.apache.org/jira/browse/KAFKA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2191: Labels: quotas (was: ) Measured rate should not be infinite

[jira] [Updated] (KAFKA-2205) Generalize TopicConfigManager to handle multiple entity configs

[ https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-2205: --- Labels: quotas (was: ) Generalize TopicConfigManager to handle multiple entity configs

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

I've made two changes to the document: - Removed the TMR evolution piece since we agreed to retain this. - Added two new API's to the admin client spec. (Alter and Describe config). Please review. Aditya From: Ashish Singh [asi...@cloudera.com] Sent:

Re: Review Request 35201: Fix KAFKA-2253

On June 11, 2015, 1:07 a.m., Jun Rao wrote: core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 https://reviews.apache.org/r/35201/diff/2/?file=980805#file980805line264 Not sure if we need this check. Since all writes to watchersForKey are sync-ed, it's ok to

Re: Review Request 35231: Fix KAFKA-1740

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35231/#review87583 --- core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala

Re: [DISCUSS] KIP-25 System test improvements

Hi Aditya, (1) There are a currently a few different ways to target a specific test or subset of tests. If for example tests were organized like the current system tests, where suites are grouped by directory, you could run cd system_test_dir ducktape replication_testsuite/ You can also target

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, Do we need a new voting thread for this KIP? The last round of votes had 3 binding +1's but there's been a fair amount of discussion since then. Aditya From: Aditya Auradkar Sent: Thursday, June 11, 2015 10:32 AM To: dev@kafka.apache.org

[jira] [Created] (KAFKA-2265) creating a topic with large number of partitions takes a long time

Jun Rao created KAFKA-2265: -- Summary: creating a topic with large number of partitions takes a long time Key: KAFKA-2265 URL: https://issues.apache.org/jira/browse/KAFKA-2265 Project: Kafka Issue

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582433#comment-14582433 ] Joel Koshy commented on KAFKA-1367: --- [~singhashish] - yes that is a good summary.