Re: Review Request 33620: Patch for KAFKA-1690

2015-06-23 Thread Sriharsha Chintalapani
On May 22, 2015, 12:33 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, line 231 https://reviews.apache.org/r/33620/diff/8/?file=966794#file966794line231 Do we need config.values? Could we just pass in config.originals() as we do

[jira] [Commented] (KAFKA-2297) VerifiableProperties fails when constructed with Properties with numeric values

2015-06-23 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597942#comment-14597942 ] Gwen Shapira commented on KAFKA-2297: - Yeah, I think its a known issue -

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Labels: (was: quotas) Client Selector can drop connections on InvalidReceiveException without

[jira] [Created] (KAFKA-2297) VerifiableProperties fails when constructed with Properties with numeric values

2015-06-23 Thread Grant Overby (JIRA)
Grant Overby created KAFKA-2297: --- Summary: VerifiableProperties fails when constructed with Properties with numeric values Key: KAFKA-2297 URL: https://issues.apache.org/jira/browse/KAFKA-2297 Project:

[jira] [Updated] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

2015-06-23 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2168: --- Attachment: KAFKA-2168_2015-06-23_09:39:07.patch New consumer poll() can block other calls

[jira] [Created] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
Dong Lin created KAFKA-2298: --- Summary: Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient Key: KAFKA-2298 URL: https://issues.apache.org/jira/browse/KAFKA-2298

Re: [GitHub] kafka pull request: Kafka 2276

2015-06-23 Thread Gwen Shapira
Awesome, thanks :) On Tue, Jun 23, 2015 at 10:32 AM, Geoffrey Anderson ge...@confluent.io wrote: Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined

[jira] [Updated] (KAFKA-2294) javadoc compile error due to illegal p/ , build failing (jdk 8)

2015-06-23 Thread Jeff Maxwell (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Maxwell updated KAFKA-2294: Attachment: KAFKA-2294-1.patch Fixed close() javadoc javadoc compile error due to illegal p/ ,

[jira] [Commented] (KAFKA-2294) javadoc compile error due to illegal p/ , build failing (jdk 8)

2015-06-23 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598057#comment-14598057 ] Jakob Homan commented on KAFKA-2294: +1. javadoc compile error due to illegal p/ ,

Re: Review Request 34789: Patch for KAFKA-2168

2015-06-23 Thread Jason Gustafson
On June 23, 2015, 4:20 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java, line 355 https://reviews.apache.org/r/34789/diff/12/?file=990006#file990006line355 What's the logic to initiate connection to coordinator if the

[jira] [Commented] (KAFKA-2168) New consumer poll() can block other calls like position(), commit(), and close() indefinitely

2015-06-23 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597914#comment-14597914 ] Jason Gustafson commented on KAFKA-2168: Updated reviewboard

Re: Review Request 34789: Patch for KAFKA-2168

2015-06-23 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34789/ --- (Updated June 23, 2015, 4:39 p.m.) Review request for kafka. Bugs:

Re: [GitHub] kafka pull request: Kafka 2276

2015-06-23 Thread Geoffrey Anderson
Hi Gwen, That is indeed the plan, and my understanding is that the merge script Ismael has been working on helps committers with this step. I'm trying out the Github flow roughly as outlined here:

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Attachment: KAFKA-2298.patch Client Selector can drop connections on InvalidReceiveException without

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Status: Patch Available (was: Open) Client Selector can drop connections on InvalidReceiveException

[jira] [Commented] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598024#comment-14598024 ] Dong Lin commented on KAFKA-2298: - Created reviewboard

Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Dong Lin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/ --- Review request for kafka. Bugs: KAFKA-2298

high level consumer memory footprint

2015-06-23 Thread Kris K
Hi, I was just wondering if there is any difference in the memory footprint of a high level consumer when: 1. the consumer is live and continuously consuming messages with no backlogs 2. when the consumer is down for quite some time and needs to be brought up to clear the backlog. My test case

Re: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-23 Thread Neha Narkhede
+1 On Tue, Jun 23, 2015 at 11:15 AM, Ashish Singh asi...@cloudera.com wrote: Hey Guys, We had some discussion over mail and in KIP hangout. I will update the RB with proposed changes. On Sun, Jun 14, 2015 at 10:07 AM, Ashish Singh asi...@cloudera.com wrote: Hi Neha, Answers

RE: [VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-23 Thread Aditya Auradkar
Hey Ashish, Last hangout, I think we discussed adding a num-iterations parameter to the script. Do you plan to support that? Aditya From: Neha Narkhede [n...@confluent.io] Sent: Tuesday, June 23, 2015 11:24 AM To: dev@kafka.apache.org Cc: Jun Rao

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Dong Lin
On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 282 https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282 Do you think we should just move the disconnected.add() into the close

[VOTE] KIP-23 - Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-06-23 Thread Ashish Singh
Hey Guys, We had some discussion over mail and in KIP hangout. I will update the RB with proposed changes. On Sun, Jun 14, 2015 at 10:07 AM, Ashish Singh asi...@cloudera.com wrote: Hi Neha, Answers inline. On Thu, Jun 11, 2015 at 7:20 PM, Neha Narkhede n...@confluent.io wrote: Thanks

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/#review89010 ---

[jira] [Resolved] (KAFKA-2294) javadoc compile error due to illegal p/ , build failing (jdk 8)

2015-06-23 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved KAFKA-2294. Resolution: Fixed Fix Version/s: 0.9.0 Reviewer: Jakob Homan Assignee: Jeff

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Jason Gustafson
On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 282 https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282 Do you think we should just move the disconnected.add() into the close

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-06-23 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598297#comment-14598297 ] Sriharsha Chintalapani commented on KAFKA-1690: --- Updated reviewboard

[jira] [Updated] (KAFKA-1690) new java producer needs ssl support as a client

2015-06-23 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1690: -- Attachment: KAFKA-1690_2015-06-23_13:18:20.patch new java producer needs ssl

Re: Review Request 33620: Patch for KAFKA-1690

2015-06-23 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/ --- (Updated June 23, 2015, 8:18 p.m.) Review request for kafka. Bugs:

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Ewen Cheslack-Postava
There was some discussion on the KIP call today. I'll give my summary of what I heard here to make sure this thread has the complete context for ongoing discussion. * Where the project should live, and if in Kafka, where should connectors live? If some are in Kafka and some not, how many and

Re: Review Request 33620: Patch for KAFKA-1690

2015-06-23 Thread Gwen Shapira
On May 22, 2015, 12:33 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, line 231 https://reviews.apache.org/r/33620/diff/8/?file=966794#file966794line231 Do we need config.values? Could we just pass in config.originals() as we do

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Dong Lin
On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 282 https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282 Do you think we should just move the disconnected.add() into the close

Re: Review Request 33620: Patch for KAFKA-1690

2015-06-23 Thread Sriharsha Chintalapani
On May 22, 2015, 12:33 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java, line 294 https://reviews.apache.org/r/33620/diff/8/?file=966806#file966806line294 Not sure why we are looping back here. If the HandshakeStatus is

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Labels: quotas (was: ) Client Selector can drop connections on InvalidReceiveException without

[jira] [Commented] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

2015-06-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597558#comment-14597558 ] Manikumar Reddy commented on KAFKA-1901: Probably the easiest thing to do would be

[jira] [Comment Edited] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

2015-06-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597558#comment-14597558 ] Manikumar Reddy edited comment on KAFKA-1901 at 6/23/15 12:10 PM:

[jira] [Comment Edited] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

2015-06-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597558#comment-14597558 ] Manikumar Reddy edited comment on KAFKA-1901 at 6/23/15 12:13 PM:

Re: [GitHub] kafka pull request: Kafka 2276

2015-06-23 Thread Gwen Shapira
I'm unclear on the directory structure and few high level things (and I can't figure out how to comment on more than a single line in github): tests/kafkatest - isn't it redundant? do we expect any non-kafka tests? services/templates - those are basically all configuration files to be used by

Review Request 35820: Patch for KAFKA-1367

2015-06-23 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35820/ --- Review request for kafka. Bugs: KAFKA-1367

[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-23 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1367: -- Attachment: KAFKA-1367.patch Broker topic metadata not kept in sync with ZooKeeper

Re: Review Request 35820: Patch for KAFKA-1367

2015-06-23 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35820/ --- (Updated June 24, 2015, 5:10 a.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-23 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598880#comment-14598880 ] Ashish K Singh commented on KAFKA-1367: --- [~junrao], [~jjkoshy], [~nehanarkhede],

[jira] [Updated] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-23 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-1367: -- Status: Patch Available (was: Open) Broker topic metadata not kept in sync with ZooKeeper

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

2015-06-23 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598877#comment-14598877 ] Ashish K Singh commented on KAFKA-1367: --- Created reviewboard

[jira] [Created] (KAFKA-2296) Not able to delete topic on latest kafka

2015-06-23 Thread Andrew M (JIRA)
Andrew M created KAFKA-2296: --- Summary: Not able to delete topic on latest kafka Key: KAFKA-2296 URL: https://issues.apache.org/jira/browse/KAFKA-2296 Project: Kafka Issue Type: Bug Affects

[jira] [Created] (KAFKA-2295) Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer

2015-06-23 Thread Tathagata Das (JIRA)
Tathagata Das created KAFKA-2295: Summary: Dynamically loaded classes (encoders, etc.) may not be found by Kafka Producer Key: KAFKA-2295 URL: https://issues.apache.org/jira/browse/KAFKA-2295

[jira] [Commented] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598719#comment-14598719 ] Dong Lin commented on KAFKA-2298: - Updated reviewboard

[jira] [Updated] (KAFKA-2298) Client Selector can drop connections on InvalidReceiveException without notifying NetworkClient

2015-06-23 Thread Dong Lin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2298: Attachment: KAFKA-2298_2015-06-23_18:47:54.patch Client Selector can drop connections on

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Dong Lin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35791/ --- (Updated June 24, 2015, 1:48 a.m.) Review request for kafka. Bugs:

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Dong Lin
On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 282 https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282 Do you think we should just move the disconnected.add() into the close

trying to manage offset for ConsumerConnector

2015-06-23 Thread Shashank Singh
Hi Team I am trying to create a Kafka Consumer API which is high level as ConsumerConnector but is enable to manage offset to track the message read/consumed and save from duplicate message consumption. In addition to few other pages I was going through

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

2015-06-23 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598643#comment-14598643 ] Jay Kreps commented on KAFKA-1215: -- This is great! Rack-Aware replica assignment option

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Ewen Cheslack-Postava
On Mon, Jun 22, 2015 at 8:32 PM, Roshan Naik ros...@hortonworks.com wrote: Thanks Jay and Ewen for the response. @Jay 3. This has a built in notion of parallelism throughout. It was not obvious how it will look like or differ from existing systemsÅ  since all of existing ones do

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Ewen Cheslack-Postava
And, one more piece of follow up. Some folks were wondering about more specific details about what we had in mind for the framework. Along with a prototype I had been writing up some documentation. This isn't meant in any way to be finalized and I just wrote it up using the same tools we use

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Sriram Subramanian
I am still not convinced why a stream processing framework closely tied to Kafka will not help with this (since we are also referring to some basic transformations). The devil is in the details of the design and I would be able to better comment on it after that. I would love to see a detailed

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Gwen Shapira
Re: Flume vs. CopyCat I would love to have an automagically-parallelizing, schema-aware version of Flume with great reliability guarantees. Flume has good core architecture and I'm sure that if the Flume community is interested, it can be extended in that direction. However, the Apache way is

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

2015-06-23 Thread Gwen Shapira
One more reason to have CopyCat as a separate project is to sidestep the entire Why CopyCat and not X discussion :) On Tue, Jun 23, 2015 at 6:26 PM, Gwen Shapira gshap...@cloudera.com wrote: Re: Flume vs. CopyCat I would love to have an automagically-parallelizing, schema-aware version of

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

2015-06-23 Thread Allen Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598615#comment-14598615 ] Allen Wang commented on KAFKA-1215: --- We have a working solution now for rack aware

Re: Review Request 35791: Patch for KAFKA-2298

2015-06-23 Thread Joel Koshy
On June 23, 2015, 5:59 p.m., Jason Gustafson wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 282 https://reviews.apache.org/r/35791/diff/1/?file=990592#file990592line282 Do you think we should just move the disconnected.add() into the close