Re: Performance issue with KafkaStreams

Hello Caleb, `./gradlew jar` should be sufficient to run the SimpleBenchmark. Could you look into kafka-run-class.sh and see if "streams/build/libs /kafka-streams*.jar" is added to the dependent path? In trunk it is added. Guozhang On Sat, Sep 17, 2016 at 11:30 AM, Eno Thereska

Re: Exception while deserializing in kafka streams

Hello Walter, The WARN log entry should not be the cause of this issue. I double checked the 0.10.0.0 release and this issue should not really happen, so your observation is a bit weird to me. Could your add a log entry in the `configure` function which constructs the registry client to make

Re: Error kafka-stream method punctuate in context.forward()

Hello Hamza, Which Kafka version are you using with this application? Also could you share your code skeleton of the StatsByDay processor implementation? Guozhang On Fri, Sep 16, 2016 at 6:58 AM, Hamza HACHANI wrote: > Good morning, > > I have a problem with a

Kafka cluster behavior: broker no longer belonging to ISR, offsets reset to middle of partition

Hi, One of my Kafka 0.9.0.1 clusters (3 brokers, default.replication.factor=2) that has been working fine until yesterday. The message volume was pretty low. There were no obvious problems except The first symptom was *kafka-consumer-groups.sh* failing with an empty.head exception. When I

Re: Running Zookeeper on same instance as kafka brokers in multi node cluster

Hello, Initially we had our setup with both ZK and Kafka on the same nodes as well. Over time though this proved problematic as we both increased the usage of Kafka and had to scale out the cluster. ZK specifically can be very touchy if it gets IO bound - so when the Kafka cluster would get under

Running Zookeeper on same instance as kafka brokers in multi node cluster

Hi Team, What are the downsides of installing Zookeeper and kafka on same machine, in multi broker environment? We are trying to install Zookeeper and kafka in AWS world and its becoming difficult for us to maintain ZK and Kafka with some issues. Also re-provisioning ZK and Kafka instances

Re: Schema for jsonConverter

Thanks Shikar. I made this change and it's working now. Thanks, Sri > On Sep 19, 2016, at 2:25 PM, Shikhar Bhushan wrote: > > Hi Srikrishna, > > The issue is that you are using "name" to specify the field name for the > struct's fields. The correct key to use is

Re: Schema for jsonConverter

Hi Srikrishna, The issue is that you are using "name" to specify the field name for the struct's fields. The correct key to use is "field". Best, Shikhar On Thu, Sep 15, 2016 at 4:23 PM Gwen Shapira wrote: > ah, never mind - I just noticed you do use a schema... Maybe you

Consul / Zookeeper [was Re: any update on this?]

[+ dev list] I have not worked on KAFKA-1793 directly, but I believe most of the work so far has been in removing all zookeeper dependencies from clients. The two main areas for that are (1) consumer rebalancing, and (2) administrative commands. 1) Consumer rebalancing APIs were added to the

Re: Partition creation issues (0.9.0.1)

Bumping for visibility - appreciate any responses on this. Thanks, Apurva On Thu, Sep 15, 2016 at 3:57 PM, Apurva Sharma wrote: > > Kafka Topic Creation issues (via Kafka-Manager with > auto.create.topics.enable = false) > Version: 0.9.0.1 > > We created a topic

Benchmarking kafka performance

Good morning. Which benchmarking tools we should use to compare performance of 0.8 and 0.10 versions? Which metrics should we monitor ? Thanks in advance, Vadim

RE: Kafka usecase

Please find my response here. 1. Kafka can be used as a message store. 2. What is the message arrival rate per second? 20 per sec 3. What is the SLA for the messages to be processed? 500 ms per message 4. If your messages arrive faster than they are consumed, you will get a backlog of messages.

RE: any update on this?

Jens/Kant not aware of any shortfall with zookeeper so perhaps you can suggest advantages for Consul vs Zookeeper? Maven (I am building, testing and running kafka internally with maven) implements wagon-providers for URLConnection vs HttpURLConnection

Re: Time of derived records in Kafka Streams

Hi Elias, - out of order records: the timestamp is that of the out of order record, i.e., time goes backwards sometimes - joins: the same, the timestamp could be that of either record. We'll update the docs, thanks for your question. Eno > On 17 Sep 2016, at 00:43, Elias Levy

Re: any update on this?

Yes ofcourse the goal shouldn't be moving towards consul. It should just be flexible enough for users to pick any distributed coordinated system. On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se wrote: I think I read somewhere that the long-term goal is to make Kafka

Re: any update on this?

I think I read somewhere that the long-term goal is to make Kafka independent of Zookeeper alltogether. Maybe not worth spending time on migrating to Consul in that case. Cheers, Jens On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain wrote: > +2 watching. > > On Sat, Sep

Kafka with Flume

Hi, We are using flume with kafka source. Flume gets json messages from kafka topic with given morphline file. If any message that has not well formed with morphline file json structure is added to kafka topic, flume (consumer) always tries to consume that wrong message. When we looked the