Re: How to verify offsets topic exists?

There's another related bug - triggering offsets topic creation through requesting metadata about that topic does not work in case of single broker clean (no topics created yet) Kafka cluster running. In that case sequence returned by KafkaApis.getAliveBrokers is empty, and

Re: mapping events to topics

Multiple topics is the model I would recommend for what you have described. LinkedIn has an environment where we have a wide mix, in a lot of different clusters. We have some topics that have one producer and one consumer (queuing). We have some topics that are multi-producer (tracking and

Frequent halts in Kafka Consumption

Hi, We are using Kafka cluster with 2 nodes (replication factor 2) - v0.8.1.1 and we have topics with 20 partitions. At consumer side we use v0.8.2.0 kafka library and it's still using ZookeeperConsumerConnection way of consuming. What we have seen recently is that ConsumerIterator gets stuck at

Re: mapping events to topics

Gwen, Thanks for your reply. I understand all of the points you've made. I think the challenge for us is that we have some consumers that are interested in messages of one type, but we also have a rules engine that is checking for events of many types and acting on them. If we put discrete

0.9.0 Beta Release

Greetings, We were really looking forward to the 0.8.3 release that was slated for October. Looks like that was pushed back in to the 0.9.0 release in November :/ What is the timeline we're looking at for a beta and release of 0.9.0? I'm trying to get a feel if the date is going to be pushed

Kafka Mirror to consume data from beginning of topics

Hi there, I am trying to setup a Kafka Mirror mechanism, but it seems the consumer from the source Kafka cluster only reads from new incoming data to the topics, i.e. it does not read historically saved data in the topics. Is there a way to define the consumer of Kafka Mirror to read from the

Re: 0.9.0 Beta Release

Hi Jarred, At the moment, we still believe that we are planning a release (without a beta, I think) in mid-Nov. On the other hand, we are all engineers, we tend to be optimistic about those things :) Gwen On Thu, Oct 8, 2015 at 10:31 AM, Jarred Ward wrote: > Greetings, > >

Kafka Mirror Maker, cannot read topic's offset/group

Hello, I am trying to setup mirroring between two kafka servers using MirrorMaker. The messages are successfully mirrored between the servers on all topics. I can list all topics on both the master and the slave (kafka-topics.sh --list). The problem that I am facing now is that when I try to

Documentation for 0.8.2 kafka-client api?

hi! where can I find a quickstart doc for kafka-client java api version 0.8.2 ? The documentation at http://kafka.apache.org/documentation.html does not seem to sync with the 0.8.2 API in the kafka-clients artifact. Specifically, I cannot find the class ConsumerConnector that is referenced here: 

Re: Documentation for 0.8.2 kafka-client api?

Hmm. Then the documentation needs to be fixed. at this site: Apache Kafka it is asking consumers to use kafka-clients jar. THere is no mention of old vs new consumer api. On Thursday, October 8, 2015 1:27 PM, Ewen Cheslack-Postava wrote: ConsumerConnector is

Re: log compaction scaling with ~100m messages

Just want to chime on this question as this does seem a good option to avoid some memory hungry K,V store in case we are ok with some async processing. There are cases where you want a combination of some near realtime and some offline processing of the same index and as kafka topic is much

Re: Documentation for 0.8.2 kafka-client api?

ConsumerConnector is part of the old consumer API (which is what is currently released; new consumer is coming in 0.9.0). That class is not in kafka-clients, it is in the core Kafka jar, which is named with the Scala version you want to use, e.g. kafka_2.10. -Ewen On Thu, Oct 8, 2015 at 1:24 PM,

Re: log compaction scaling with ~100m messages

Thank you for your response! Our use case is more similar to a traditional k/v store. We are doing a new process that is going to spit huge amounts of data. We are using kafka as a broker so that downstream clients can all consume from the kafka topic. What we would like to avoid, is writing

Re: log compaction scaling with ~100m messages

Hi, just want to pick this up again. You can always use more partitions to reduce the number of keys handled by a single broker and parallelize the compaction. So with sufficient number of machines and the ability to partition I don’t see you running into problems. Jan On 07.10.2015 05:34,