KAFKA-3112

Hi All, Does anyone know if KAFKA-3112 is merged to 0.9.0.1? Is there a place to check which version has this fix? Jira doesn’t show fix versions. https://issues.apache.org/jira/browse/KAFKA-3112 Thanks, Raj Tanneru Information contained in this e-mail message is confidential. This e-mail

Re: Get topic level detail from new consumer group command

you need to run describe topic command to get the topic details: ./kafka-topics.sh --zookeeper ":2181" --describe --topic From: ravi singh To: users@kafka.apache.org; d...@kafka.apache.org Sent: Friday, 6 May 2016 1:07 AM Subject: Get topic level detail from new

Reg: Kafka-Acls

Hi, When I run the command /bin/kafka-acls.sh --topic permissiontopic --add --allow-host {host} --allow-principal User:dev --operation Write --authorizer-properties zookeeper.connect={host:port} I am getting output as acls are set. But when i check under zookeeper using below command, it is not

Tool to look at what's stored in internal topics on kafka 0.9

When we are on kafka 0.8, all the consumer offsets are stored in ZK and we can use some ZK browser to see the contents in different ZK paths. On kafka 0.9, when everything moved to internal kafka topics, do we have a tool to browse through the contents in those topics?

[VOTE] 0.10.0.0 RC3

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give

Re: Hash partition of key with skew

Understood. Thanks. On Wed, May 4, 2016 at 3:47 PM, Wesley Chow wrote: > > We don’t do this on the Kafka side, but for a different system that has > similar distribution problems we manually maintain a map of “hot” keys. On > the Kafka side, we distribute keys with an even

Get topic level detail from new consumer group command

./bin/kafka-consumer-groups.sh --group batchprocessord_zero --bootstrap-server kafka-1-evilcorp.com:9092 --new-consumer --describe Running the above ConsumerGroupcommad will describe consumer for all the topics it's listening to. Is there any workaround to get *only topic level detail*? ​ --

ktable implemented as compacted topic?

hello! i would like to ask you if ktable is using as backend storage a compacted topic ? i have read here http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple thtat a ktable is at its base a compacted topic if i would like to have the messages from ktable be

Re: Kafka Consumer consuming large number of messages

When polling from kafka I am logging the number of records fetched. I have 200 partitions for the topic and 12 Consumers each with 4 threads, so around 4-5 partitions per Consumer, and if the Consumer is slow the number of messages it's consuming increases to around 7000 messages. Here is the

Re: Kafka-to-Kafka Connector ?

Thnx, that looks like a good option. I'm a bit concerned about running/monitoring an additional external app, an in-stream solution (Connector or Streams plugin) would be preferable. But mirroring may be good enough until we eventually upgrade to 0.10. On Thu, May 5, 2016 at 10:57 AM, tao xiao

Re: Kafka-to-Kafka Connector ?

You can use the built-in mirror maker to mirror data from one Kafka to the other. http://kafka.apache.org/documentation.html#basic_ops_mirror_maker On Thu, 5 May 2016 at 10:47 Dean Arnold wrote: > I'm developing a Streams plugin for Kafka 0.10, to be run in a dev sandbox,

Kafka-to-Kafka Connector ?

I'm developing a Streams plugin for Kafka 0.10, to be run in a dev sandbox, but pull data from a production 0.9 Kafka deployment. Is there a source connector that can be used from the 0.10 sandbox to connect to the 0.9 cluster ? Given the number of changes/features in 0.10, such a connector would

Re: Kafka just ate my homework?

Thanks - I'll check and see if the other topic got cleared too... What / where is the per topic retention set? I wasn't are of that - thanks for mentioning it. On Thu, May 5, 2016 at 9:49 AM, Salman Ahmed wrote: > On Kafka 0.8.2 release there were a few issues like

Consumer best practices

Hello! We are using Kafka 0.9.1. We have created a class CustomKafkaConsumer whose method receive has the pseudocoode public OurClassStructure[] receiveFromKafka() { //gte the message from topic ConsumerRecords received= org

Re: Kafka just ate my homework?

On Kafka 0.8.2 release there were a few issues like KAFKA-2477 that could have impacted you. Did you have issue with just one topic or all topics got data deleted? Also did you have a per topic level retention override configured on top of the base

Re: Consumer stopped after reading some messages

This may or may not help. I found it to be a clever workaround for some of the limitations in the 8.x version of the high level consumer... I ended up writing code that "waited" a lot because I couldn't be sure how quickly Kafka would respond... Nothing ever took minutes however... the waits

Subject=0.9.0 Client - Streaming vs Polling

What did you conclude? We also use the streaming methods (consumerConnector.createMessageStreams), I can't see what the equivalents are in 0.9 Cheers

Re: Kafka just ate my homework?

kafka_2.10-0.8.2.2 Thanks On Wed, May 4, 2016 at 9:42 PM, Salman Ahmed wrote: > Which version of Kafka are you on? > On Wed, May 4, 2016 at 4:54 PM John Bickerstaff > wrote: > > > Hi, > > > > I've been working with Kafka a lot recently and

Re: Backing up Kafka data and using it later?

John, I’m not as expert expert in Kafka but I would assume so. Best regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This

Re: Suggesstion for mixture of 0.8 and 0.9?

It's really not a good idea to use jdk 1.6 to rebuild the jar file. There can always but jdk 1.7 only features that are used in the code that won't play nicely with jdk 1.6. As Tom mentioned, you should really be looking at moving off of jdk 1.6 before looking at upgrading anything else in your

Consumer stopped after reading some messages

I am using high level consumer API ( Kafka API version 0.9.0.0 ) I am running consumers on a topic of 10 partitions. There are lot of unread messages in that topic. Initially all of them , consume from the topic and read messages. After sometime, all of them hangs and doesn't read any message at

Re: How to build kafka jar with all dependencies?

Hi Ravi, You might want to ask this question on the Kafka developers mailing list. Cheers, Jens On Wednesday, May 4, 2016, ravi singh wrote: > I used .*/gradlew jarAll* but still scala libs are missing from the jar? > ​It should be something ​very simple which I might be

Re: 0.8.x consumer group protocol

Thank you Dana. On Wed, May 4, 2016 at 4:25 PM, Dana Powers wrote: > 0.8 clients manage groups by connecting directly to zookeeper and > implementing shared group management code. There are no broker APIs used. > > 0.9 clients manage groups using new kafka broker APIs.

Re: Kafka Streams: KStream - KTable Left Join

Thanks, I'll try it. It would be nice if you could improve the error message. 2016-05-04 20:13 GMT+02:00 Matthias J. Sax : > +1 > > I had the same thought and put it on my personal agenda already. > > -Matthias > > On 05/04/2016 06:37 PM, Jay Kreps wrote: > > Is it