Re: Is kafka suitable for our architecture?

2014-10-10 Thread Albert Vila
Hi We process data in real time, and we are taking a look at Storm and Spark streaming too, however our actions are atomic, done at a document level so I don't know if it fits on something like Storm/Spark. Regarding what you Christian said, isn't Kafka used for scenarios like the one I

Re: Is kafka suitable for our architecture?

2014-10-10 Thread cac...@gmail.com
Albert, you certainly can use Kafka (and it will probably work quite well) you'll just need to make sure your consumers are written to match the available options. I think I may not have a good picture of what you need to do. Is it that you have a stream of documents coming in and then each

Re: Scala+Kafka puzzler

2014-10-10 Thread Sean Owen
It's private to package kafka: https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/cluster/Broker.scala#L29 On Fri, Oct 10, 2014 at 10:47 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, Attached trivial Maven built project with Kafka code fails to

Re: Scala+Kafka puzzler

2014-10-10 Thread Stevo Slavić
Oh, right, well spotted, I was looking in wrong branch. In 0.8.2 branch it's no longer private. Thanks! Kind regards, Stevo Slavic On Fri, Oct 10, 2014 at 12:21 PM, Sean Owen so...@cloudera.com wrote: It's private to package kafka:

mount of messages produced and those consumed are not equal

2014-10-10 Thread Shuo Chen
Dear folks, I have run some basic tests using kafka. I have started 7 brokers, created a topic with 21 partitions and 3 replicas. 2 producers are started with 5 threads each and pushed 2419200 messages of 50 bytes in total. 3 consumers with 2 in one group and 1 in other are started to

Re: including KAFKA-1555 in 0.8.2?

2014-10-10 Thread Neha Narkhede
+1. On Thu, Oct 9, 2014 at 6:41 PM, Jun Rao jun...@gmail.com wrote: Hi, Everyone, I just committed KAFKA-1555 (min.isr support) to trunk. I felt that it's probably useful to include it in the 0.8.2 release. Any objections? Thanks, Jun

Re: Auto Purging Consumer Group Configuration [Especially Kafka Console Group]

2014-10-10 Thread Neha Narkhede
Bhavesh, All consumer groups with at least one alive consumer will have a registration node for that consumer instance under /consumers/group/ids. Any consumer group that is long gone will have no registered consumer instance in zookeeper. The right way to clean up defunct consumer groups is to

Re: mount of messages produced and those consumed are not equal

2014-10-10 Thread Guozhang Wang
Hi Shuo, What are your producer and consumer config values? Guozhang On Fri, Oct 10, 2014 at 3:47 AM, Shuo Chen chenatu2...@gmail.com wrote: Dear folks, I have run some basic tests using kafka. I have started 7 brokers, created a topic with 21 partitions and 3 replicas. 2 producers

Re: Auto Purging Consumer Group Configuration [Especially Kafka Console Group]

2014-10-10 Thread Guozhang Wang
There is a JIRA open for exactly this issue with patch available: KAFKA-559 https://issues.apache.org/jira/browse/KAFKA-559 On Fri, Oct 10, 2014 at 9:33 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Bhavesh, All consumer groups with at least one alive consumer will have a registration

Starting KafkaMetricsReporter from Kafka Consumer/Producer

2014-10-10 Thread ravi singh
In ProducerPerformance class we use CSVMetricsReporter for metrics reporting. Which I think is actually started with the help of below function: KafkaMetricsReporter.startReporters(verifiableProps) Similarly I wrote my own producer and I have a custom implementation of KafkaMetricsReporter. But

Installation and Java Examples

2014-10-10 Thread Mohit Anchlia
I am new to Kafka and very little familiarity with Scala. I see that the build requires sbt tool, but do I also need to install Scala separately? Is there a detailed documentation on software requirements on the broker machine. I am also looking for 3 different types of java examples 1) Follow

Re: Installation and Java Examples

2014-10-10 Thread Harsha
Mohit, Kafka uses gradle to build the project, check the README.md under source dir for details on how to build and run unit tests. You can find consumer and producer api here http://kafka.apache.org/documentation.html and also more details on consumer

Re: Starting KafkaMetricsReporter from Kafka Consumer/Producer

2014-10-10 Thread ravi singh
Never mind. I found the issue. Thanks, Ravi On Fri, Oct 10, 2014 at 11:47 AM, ravi singh rrs120...@gmail.com wrote: In ProducerPerformance class we use CSVMetricsReporter for metrics reporting. Which I think is actually started with the help of below function:

Re: Is kafka suitable for our architecture?

2014-10-10 Thread Christian Csar
On 10/10/2014 05:12 AM, Albert Vila wrote: Some comments below. On 10 October 2014 11:19, cac...@gmail.com cac...@gmail.com wrote: Albert, you certainly can use Kafka (and it will probably work quite well) you'll just need to make sure your consumers are written to match the available

kafka-web-console

2014-10-10 Thread Sa Li
Hi, all I am installing kafka-web-console on ubuntu server, when I sbt package it, it stuck on waiting for ivy.lock root@DO-mq-dev:/home/stuser/kafkaprj/kafka-web-console# sbt package Loading /usr/share/sbt/bin/sbt-launch-lib.bash [info] Loading project definition from