Re: serializable kafka producer pool
Bump... Looking for a Kafka Producer Object pool to use in Spark Streaming inside foreachPartition On Wed, Jan 14, 2015 at 8:40 PM, Josh J joshjd...@gmail.com wrote: Hi, Does anyone have a serializable kafka producer pool that uses the KafkaProducer.crateProducer() method? I'm trying to use the Spark borrow feature to cache the kafka producers. Thanks, Josh
generate specific throughput load
Hi, Is there a way to generate a specified amount of throughput? I'm using the Stats class here https://github.com/apache/kafka/blob/7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4/clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java to measure the throughput. Though I need to be able to precisely control the amount of load. For example, 1000 records per second. Thanks, Josh
serializable kafka producer pool
Hi, Does anyone have a serializable kafka producer pool that uses the KafkaProducer.crateProducer() method? I'm trying to use the Spark borrow feature to cache the kafka producers. Thanks, Josh
Kafka producer iOS and android
hi Is it possible to have iOS and android to run the code needed for Kafka producers ? I want to have mobile clients connect to Kafka broker Thanks, Josh
Re: Kafka producer iOS and android
Thanks for the tip. I would like to avoid hand rolling any code if possible. For example, on Android I would like to ask if people are able to include and use the kafka jars with no problem? And on iOS, if there is a way to include any C or other relevant code. On Mon, Oct 20, 2014 at 8:49 AM, Harsha ka...@harsha.io wrote: Hi Josh, Why not have Rest api service running where you post messages from your mobile clients. Rest api can run kafka producers accepting these messages pushing it into kafka brokers. Here is an example where we did similar service for kafka https://github.com/mozilla-metrics/bagheera . This project used kafka 0.7 but you can see how its implemented. Hope that helps. -Harsha On Mon, Oct 20, 2014, at 08:45 AM, Josh J wrote: hi Is it possible to have iOS and android to run the code needed for Kafka producers ? I want to have mobile clients connect to Kafka broker Thanks, Josh
Re: Kafka producer iOS and android
What is the use case requiring that? I'm looking for an open source library that I can use in Android and iOS, instead of hand rolling my own. On Mon, Oct 20, 2014 at 10:21 AM, Joe Stein joe.st...@stealth.ly wrote: What is the use case requiring that? If you try to integrate kafka in the two different mobile platforms you will get many separate development cycles and none will work in many mobile networked environments. You can HTTP/HTTPS POST the same Avro objects (or Thrift or ProtoBuf) from each platform. You also then have 1 integration cycle then easily maintained and supported cohesively over all. The Avro/Thrift/ProtoBuf objects and a HTTP/HTTPS layer, thats it. Wrapping the producer is pretty straight forward in any language now a days I think and it depends what language and how you operate production services to pick how to write that producer. That producer and interface should change much and you have to think a little about configurations but most all of that is server side depending on the message. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Oct 20, 2014 at 1:09 PM, Josh J joshjd...@gmail.com wrote: Thanks for the tip. I would like to avoid hand rolling any code if possible. For example, on Android I would like to ask if people are able to include and use the kafka jars with no problem? And on iOS, if there is a way to include any C or other relevant code. On Mon, Oct 20, 2014 at 8:49 AM, Harsha ka...@harsha.io wrote: Hi Josh, Why not have Rest api service running where you post messages from your mobile clients. Rest api can run kafka producers accepting these messages pushing it into kafka brokers. Here is an example where we did similar service for kafka https://github.com/mozilla-metrics/bagheera . This project used kafka 0.7 but you can see how its implemented. Hope that helps. -Harsha On Mon, Oct 20, 2014, at 08:45 AM, Josh J wrote: hi Is it possible to have iOS and android to run the code needed for Kafka producers ? I want to have mobile clients connect to Kafka broker Thanks, Josh
read N items from topic
hi, How do I read N items from a topic? I also would like to do this for a consumer group, so that each consumer can specify an N number of tuples to read, and each consumer reads distinct tuples. Thanks, Josh
rate of producers writing to a topic?
Hi, Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the topic in particular ways depending on the incoming write throughput. Thanks, Josh
Re: rate of producers writing to a topic?
Interested in the total number of tuples written per millisecond per topic. On Tue, Oct 7, 2014 at 3:56 PM, Josh J joshjd...@gmail.com wrote: Hi, Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the topic in particular ways depending on the incoming write throughput. Thanks, Josh
Re: rate of producers writing to a topic?
we've got a cron which iterates each topic+partition and writes an index of timestamps-byte offset What is byte offset? Is this reliable to gauge the number of tuples being written to a topic when there are many different consumers writing to the same topic ? On Tue, Oct 7, 2014 at 4:35 PM, Neil Harkins nhark...@gmail.com wrote: On Tue, Oct 7, 2014 at 3:56 PM, Josh J joshjd...@gmail.com wrote: Is there a way to monitor the rate of rates to a particular topic? I wish to monitor the frequency of incoming tuples in order to consume from the topic in particular ways depending on the incoming write throughput. we've got a cron which iterates each topic+partition and writes an index of timestamps-byte offset, so it was easy to hook from there into graphite. but graphite couldn't do 2000+ derivative()s for a reasonable pageload time, so we put the pre-calculated write rate into graphite. what i'd very much like to have is per-topic read/write stats by client ip. -neil
kafka high level consumer - threads guaranteed to read a single partition?
Hi, For the kafka high level consumer, if I create exactly the number of threads as the number of partitions, is there a guarantee that each thread will be the only thread that reads from a particular partition? I'm following this example https://github.com/bingoohuang/java-sandbox/blob/92318c6d3f2533bbadb253c59a201e4e70f72ad2/src/main/java/org/n3r/sandbox/kafka/ConsumerGroupExample.java. Assuming that the number of threads and partitions is fixed. Thanks, Josh
Re: consumer read from specific partition
You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example Any suggestions on where in the code to modify the high level producer to support reading from specific partitions ? Thanks, Josh . On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede neha.narkh...@gmail.com wrote: You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example . On Thu, Aug 14, 2014 at 3:23 AM, Sharninder sharnin...@gmail.com wrote: Implement the low level Simple Consumer. -- Sharninder On Thu, Aug 14, 2014 at 2:16 PM, Josh J joshjd...@gmail.com wrote: Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
Re: consumer read from specific partition
Is it possible to modify and use the high level consumer so that I can ignore processing certain partitions? On Mon, Aug 18, 2014 at 5:07 PM, Sharninder sharnin...@gmail.com wrote: On Mon, Aug 18, 2014 at 7:27 PM, Josh J joshjd...@gmail.com wrote: You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example Any suggestions on where in the code to modify the high level producer to support reading from specific partitions ? High level producer? I'm assuming you meant to write the high level consumer, in which case it isn't possible. The link above, which has an example for reading messages off a specific partition, is for the Simple consumer, which ironically, is more complex than the high level consumer. In short, if you have a usecase where you want to read from a specific partition, you will need to implement a simple consumer. -- Sharninder Josh . On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede neha.narkh...@gmail.com wrote: You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example . On Thu, Aug 14, 2014 at 3:23 AM, Sharninder sharnin...@gmail.com wrote: Implement the low level Simple Consumer. -- Sharninder On Thu, Aug 14, 2014 at 2:16 PM, Josh J joshjd...@gmail.com wrote: Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
Re: consumer read from specific partition
One tactic that might be worth exploring is to rely on the message key to facilitate this. It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant. The only other option I can think of if you are using the high-level API would be finer-grained topics. Yes, this works great ! My producer is bucketing the messages based on the key (the key is a timestamp and I simply mod the timestamp as a function of the number of buckets). I can then do the same function on the consumer when it reads the key. I'm essentially implementing consumer sliding window. Any suggestions or tips on where I would implement reading the message key? Thanks, Josh On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks jonathanbwe...@gmail.com wrote: One tactic that might be worth exploring is to rely on the message key to facilitate this. It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant. The only other option I can think of if you are using the high-level API would be finer-grained topics. Best Regards, -Jonathan On Aug 18, 2014, at 9:14 AM, Josh J joshjd...@gmail.com wrote: Is it possible to modify and use the high level consumer so that I can ignore processing certain partitions? On Mon, Aug 18, 2014 at 5:07 PM, Sharninder sharnin...@gmail.com wrote: On Mon, Aug 18, 2014 at 7:27 PM, Josh J joshjd...@gmail.com wrote: You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example Any suggestions on where in the code to modify the high level producer to support reading from specific partitions ? High level producer? I'm assuming you meant to write the high level consumer, in which case it isn't possible. The link above, which has an example for reading messages off a specific partition, is for the Simple consumer, which ironically, is more complex than the high level consumer. In short, if you have a usecase where you want to read from a specific partition, you will need to implement a simple consumer. -- Sharninder Josh . On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede neha.narkh...@gmail.com wrote: You can see an example of using the SimpleConsumer here https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example . On Thu, Aug 14, 2014 at 3:23 AM, Sharninder sharnin...@gmail.com wrote: Implement the low level Simple Consumer. -- Sharninder On Thu, Aug 14, 2014 at 2:16 PM, Josh J joshjd...@gmail.com wrote: Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
consumer read from specific partition
Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
Re: In the middle of a leadership election, there is currently no leader...
What is the Kafka version you are using? 0.8.1.1. And can you describe the steps to re-produce this issue? I'm running three kafka brokers using Docker. I was using the docker IP 172.17.42.1 for the kafka host address. It looks like kafka doesn't work nicely with the same local IP. I ended up using the eth0 interface on each kafka docker container. Thanks, Josh On Thu, Jul 17, 2014 at 7:03 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Josh, What is the Kafka version you are using? And can you describe the steps to re-produce this issue? Guozhang On Thu, Jul 17, 2014 at 9:01 AM, Josh J joshjd...@gmail.com wrote: Hi, I'm running three kafka brokers. I receive the following when using the producer panic: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes. Though the describe topic shows a leader for each partition. What's happening here to make it unavailable for writes? root@6e911d719c7b:/# $KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK_PORT_2181_TCP_ADDR Topic:topicPartitionCount:20ReplicationFactor:2Configs: Topic: topicPartition: 0Leader: 5341Replicas: 5341,29311 Isr: 5341,29311 Topic: topicPartition: 1Leader: 12452Replicas: 12452,5341Isr: 12452,5341 Topic: topicPartition: 2Leader: 29311Replicas: 29311,12452Isr: 29311,12452 Topic: topicPartition: 3Leader: 5341Replicas: 5341,12452 Isr: 5341,12452 Topic: topicPartition: 4Leader: 12452Replicas: 12452,29311Isr: 12452,29311 Topic: topicPartition: 5Leader: 29311Replicas: 29311,5341Isr: 29311,5341 Topic: topicPartition: 6Leader: 5341Replicas: 5341,29311 Isr: 5341,29311 Topic: topicPartition: 7Leader: 12452Replicas: 12452,5341Isr: 12452,5341 Topic: topicPartition: 8Leader: 29311Replicas: 29311,12452Isr: 29311,12452 Topic: topicPartition: 9Leader: 5341Replicas: 5341,12452 Isr: 5341,12452 Topic: topicPartition: 10Leader: 12452Replicas: 12452,29311Isr: 12452,29311 Topic: topicPartition: 11Leader: 29311Replicas: 29311,5341Isr: 29311,5341 Topic: topicPartition: 12Leader: 5341Replicas: 5341,29311Isr: 5341,29311 Topic: topicPartition: 13Leader: 12452Replicas: 12452,5341Isr: 12452,5341 Topic: topicPartition: 14Leader: 29311Replicas: 29311,12452Isr: 29311,12452 Topic: topicPartition: 15Leader: 5341Replicas: 5341,12452Isr: 5341,12452 Topic: topicPartition: 16Leader: 12452Replicas: 12452,29311Isr: 12452,29311 Topic: topicPartition: 17Leader: 29311Replicas: 29311,5341Isr: 29311,5341 Topic: topicPartition: 18Leader: 5341Replicas: 5341,29311Isr: 5341,29311 Topic: topicPartition: 19Leader: 12452Replicas: 12452,5341Isr: 12452,5341 Thanks, Josh -- -- Guozhang
