Re: Kafka/Hadoop consumers and producers

2013-07-03 Thread Sam Meder
I think it makes sense to kill the hadoop consumer/producer code in Kafka, given, as you said, Camus and the simplicity of the Hadoop producer. /Sam On Jul 2, 2013, at 5:01 PM, Jay Kreps jay.kr...@gmail.com wrote: We currently have a contrib package for consuming and producing messages from

Re: Kafka/Hadoop consumers and producers

2013-07-03 Thread Cosmin Lehene
If the Hadoop consumer/producers use-case will remain relevant for Kafka (I assume it will), it would make sense to have the core components (kafka input/output format at least) as part of Kafka so that it could be built, tested and versioned together to maintain compatibility. This would also

[jira] [Created] (KAFKA-958) Please compile list of key metrics on the broker and client side and put it on a wiki

2013-07-03 Thread Vadim (JIRA)
Vadim created KAFKA-958: --- Summary: Please compile list of key metrics on the broker and client side and put it on a wiki Key: KAFKA-958 URL: https://issues.apache.org/jira/browse/KAFKA-958 Project: Kafka

[jira] [Commented] (KAFKA-943) Move all configuration key string to constants

2013-07-03 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699095#comment-13699095 ] Jay Kreps commented on KAFKA-943: - Yeah, I think the core disagreement is that we are

Re: criteria for fixes on 0.8 beta

2013-07-03 Thread Jay Kreps
+1 on Jun's defn. On Tue, Jul 2, 2013 at 10:21 PM, Jun Rao jun...@gmail.com wrote: My vote is that a patch can go into 0.8 if (1) it fixes a critical issue or (2) the change is trivial and it makes the 0.8 experience better (e.g., improving log4j readability). kafka-946 may fall into (2).

[jira] [Commented] (KAFKA-943) Move all configuration key string to constants

2013-07-03 Thread Sam Meder (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699128#comment-13699128 ] Sam Meder commented on KAFKA-943: - So we also have a config management system external to

Re: Helping out

2013-07-03 Thread Jun Rao
Sam, Interested in taking a look at KAFKA-959https://issues.apache.org/jira/browse/KAFKA-959 ? Thanks, Jun On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder sam.me...@jivesoftware.comwrote: Hey, I now have roughly a day a week I can dedicate to working on Kafka, so I am looking for issues in

[jira] [Created] (KAFKA-960) Upgrade Metrics to 3.x

2013-07-03 Thread Cosmin Lehene (JIRA)
Cosmin Lehene created KAFKA-960: --- Summary: Upgrade Metrics to 3.x Key: KAFKA-960 URL: https://issues.apache.org/jira/browse/KAFKA-960 Project: Kafka Issue Type: Improvement Affects

[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x

2013-07-03 Thread Cosmin Lehene (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699191#comment-13699191 ] Cosmin Lehene commented on KAFKA-826: - https://issues.apache.org/jira/browse/KAFKA-960

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2013-07-03 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699211#comment-13699211 ] Joel Koshy commented on KAFKA-960: -- Given that there are API changes and mbean name

Re: Helping out

2013-07-03 Thread S Ahmed
Jun, Looks like the update would happen around here? On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao jun...@gmail.com wrote: Sam, Interested in taking a look at KAFKA-959https://issues.apache.org/jira/browse/KAFKA-959 ? Thanks, Jun On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder

Re: Helping out

2013-07-03 Thread S Ahmed
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala#L152 On Wed, Jul 3, 2013 at 2:18 PM, S Ahmed sahmed1...@gmail.com wrote: Jun, Looks like the update would happen around here? On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao

[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

2013-07-03 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Assignee: Guozhang Wang MirrorMaker needs to preserve the key in the source cluster

[jira] [Created] (KAFKA-961) state.change.logger: Error on broker 1 while processing LeaderAndIsr request correlationId 6 received from controller 1 epoch 1 for partition (page_visits,0)

2013-07-03 Thread Garrett Barton (JIRA)
Garrett Barton created KAFKA-961: Summary: state.change.logger: Error on broker 1 while processing LeaderAndIsr request correlationId 6 received from controller 1 epoch 1 for partition (page_visits,0) Key: KAFKA-961

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2013-07-03 Thread Cosmin Lehene (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699389#comment-13699389 ] Cosmin Lehene commented on KAFKA-960: - [~jjkoshy] I was going to set it back on 3.x on

[jira] [Resolved] (KAFKA-621) System Test 9051 : ConsoleConsumer doesn't receives any data for 20 topics but works for 10

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-621. - Resolution: Fixed Not an issue any more. System Test 9051 : ConsoleConsumer doesn't

[jira] [Closed] (KAFKA-619) Regression : System Test 900x (Migration Tool) - java.lang.ClassCastException: kafka.message.Message cannot be cast to [B

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-619. --- Regression : System Test 900x (Migration Tool) - java.lang.ClassCastException: kafka.message.Message

[jira] [Resolved] (KAFKA-624) Add 07 ConsoleConsumer to validate message content for Migration Tool testcases

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-624. - Resolution: Fixed This is handled in KAFKA-882 KAFKA-883 Add 07 ConsoleConsumer to

[jira] [Resolved] (KAFKA-606) System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched across replicas

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-606. - Resolution: Fixed Not an issue any more. System Test Transient Failure (case 0302 GC

[jira] [Closed] (KAFKA-606) System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched across replicas

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-606. --- System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched across replicas

[jira] [Resolved] (KAFKA-607) System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer receives less data

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-607. - Resolution: Fixed Not an issue any more. System Test Transient Failure (case 4011 Log

[jira] [Closed] (KAFKA-607) System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer receives less data

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-607. --- System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer receives less data

[jira] [Closed] (KAFKA-819) System Test : Add validation of log segment index to offset

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-819. --- System Test : Add validation of log segment index to offset

[jira] [Closed] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-792. --- Update multiple attributes in testcase__properties.json

[jira] [Closed] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-716. --- SimpleConsumerPerformance does not consume all available messages

[jira] [Resolved] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-716. - Resolution: Fixed Not an issue SimpleConsumerPerformance does not consume all available

[jira] [Commented] (KAFKA-961) state.change.logger: Error on broker 1 while processing LeaderAndIsr request correlationId 6 received from controller 1 epoch 1 for partition (page_visits,0)

2013-07-03 Thread Garrett Barton (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699489#comment-13699489 ] Garrett Barton commented on KAFKA-961: -- I found out what is going on. Since I'm doing

[jira] [Resolved] (KAFKA-729) Gzip compression codec complains about missing SnappyInputStream

2013-07-03 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-729. - Resolution: Fixed Not an issue any more. Gzip compression codec complains about missing

[jira] [Commented] (KAFKA-911) Bug in controlled shutdown logic in controller leads to controller not sending out some state change request

2013-07-03 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699500#comment-13699500 ] Sriram Subramanian commented on KAFKA-911: -- This has been fixed.

[jira] [Updated] (KAFKA-911) Bug in controlled shutdown logic in controller leads to controller not sending out some state change request

2013-07-03 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-911: - Resolution: Fixed Status: Resolved (was: Patch Available) Bug in controlled

[jira] [Commented] (KAFKA-184) Log retention size and file size should be a long

2013-07-03 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699510#comment-13699510 ] Sriram Subramanian commented on KAFKA-184: -- We have already fixed all the config

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2013-07-03 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699545#comment-13699545 ] Joel Koshy commented on KAFKA-960: -- Not really. However, my point is that given that going

[jira] [Commented] (KAFKA-961) state.change.logger: Error on broker 1 while processing LeaderAndIsr request correlationId 6 received from controller 1 epoch 1 for partition (page_visits,0)

2013-07-03 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699567#comment-13699567 ] Joel Koshy commented on KAFKA-961: -- Passing in null for time would definitely lead to that

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2013-07-03 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699608#comment-13699608 ] Jay Kreps commented on KAFKA-960: - Word. I actually think given our experience we might be

[jira] [Created] (KAFKA-962) Add list topics to ClientUtils

2013-07-03 Thread Jakob Homan (JIRA)
Jakob Homan created KAFKA-962: - Summary: Add list topics to ClientUtils Key: KAFKA-962 URL: https://issues.apache.org/jira/browse/KAFKA-962 Project: Kafka Issue Type: Improvement Affects

[jira] [Commented] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed

2013-07-03 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699641#comment-13699641 ] Joel Koshy commented on KAFKA-915: -- This failure is due to the fact that the leaderAndIsr

[jira] [Updated] (KAFKA-559) Garbage collect old consumer metadata entries

2013-07-03 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tejas Patil updated KAFKA-559: -- Attachment: KAFKA-559.v1.patch Attached KAFKA-559.v1.patch above. - It has been developed to work do

[jira] [Commented] (KAFKA-961) state.change.logger: Error on broker 1 while processing LeaderAndIsr request correlationId 6 received from controller 1 epoch 1 for partition (page_visits,0)

2013-07-03 Thread Garrett Barton (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699741#comment-13699741 ] Garrett Barton commented on KAFKA-961: -- Thanks for the reply! I personally lean

Re: Kafka/Hadoop consumers and producers

2013-07-03 Thread Felix GV
IMHO, I think Camus should probably be decoupled from Avro before the simpler contribs are deleted. We don't actually use the contribs, so I'm not saying this for our sake, but it seems like the right thing to do to provide simple examples for this type of stuff, no...? -- Felix On Wed, Jul 3,

Re: Helping out

2013-07-03 Thread Jun Rao
Yes. We will need to think a bit more for the more general case when there are messages for multiple topics. Thanks, Jun On Wed, Jul 3, 2013 at 11:18 AM, S Ahmed sahmed1...@gmail.com wrote: