need some help in understanding weird IllegalStateException: Log end offset should not change while restoring

2016-12-17 Thread Sachin Mittal
Hi, What I have noticed recently is that when running the streams application for a while it exits with some uncaught exception. The issue was not happening before and I started seeing recently only. Not sure what we did wrong. Version is kafka_2.10-0.10.0.1 Is this some bug in kafka. What could

getting intermittent TimeoutException at producer side in streams application

2016-12-17 Thread Sachin Mittal
Hi all, I have a simple stream application pipeline src.filter.aggragteByKey.mapValues.forEach >From time to time I get the following exception: Error sending record to topic test-stream-key-table-changelog org.apache.kafka.common.errors.TimeoutException: Batch containing 2 record(s) expired due t

Re: Running cluster of stream processing application

2016-12-17 Thread Sachin Mittal
Hi, Thanks for the suggestions. Before running the streams application in a standby cluster I was trying to get the graceful shutdown right. I have code something like this streams.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() { public void uncaughtException(

Re: checking consumer lag on KStreams app?

2016-12-17 Thread Sachin Mittal
Hi, Looks like the kafka-consumer-groups.sh command is still not working. I ran: bin/kafka-consumer-groups.sh --zookeeper 192.168.73.198:2181 --describe --group test I get the output like No topic available for consumer group provided GROUP TOPIC

Is running kafka streaming application advisable on high latency WAN setup

2016-12-17 Thread Sachin Mittal
Hi folks, I needed bit of feedback from you based on your experiences using kafka streaming application. We have a replicated kafka cluster running in a data center in one city. We are running a kafka streaming application which reads from a source topic from that cluster and commits the output in

Re: What does GetOffsetShell result represent

2016-12-17 Thread Sachin Mittal
Hi, Thanks for the lucid explanation. So say if I run this bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.73.198:9092 --topic test-stream --time -1 --offsets 6 I get output like: test-stream:0:81658712,81616045,81299569,80987584,80673168,80364832 This means these are the

Re: Kafka Connect gets into a rebalance loop

2016-12-17 Thread Ewen Cheslack-Postava
The message > Wasn't unable to resume work after last rebalance means that you previous iterations of the rebalance were somehow behind/out of sync with other members of the group, i.e. they had not read up to the same point in the config topic so it wouldn't be safe for this worker (or possibly

Re: __consumer_offsets topic acks

2016-12-17 Thread Ewen Cheslack-Postava
The default is -1 which means all replicas need to replicate the committed data before the ack will be sent to the consumer. See the offsets.commit.required.acks setting for the broker. min.insync.replicas applies to the offsets topic as well, but defaults to 1. You may want to increase this (eith

Re: Kafka connect distributed mode not distributing the work

2016-12-17 Thread Ewen Cheslack-Postava
Hi Manjunath, I think you're seeing a case of this issue: https://issues.apache. org/jira/browse/KAFKA-4553 where the way round robin assignment works with an even # of workers and connectors that generate only 1 task generates uneven work assignments because connectors aren't really equivalent to

Re: Producer connect timeouts

2016-12-17 Thread Ewen Cheslack-Postava
Without having dug back into the code to check, this sounds right. Connection management just fires off a request to connect and then subsequent poll() calls will handle any successful/failed connections. The timeouts wrt requests are handled somewhat differently (the connection request isn't expli

Re: What does GetOffsetShell result represent

2016-12-17 Thread Ewen Cheslack-Postava
The tool writes output in the format: :: So in the case of your example with --time -1 that returned test-window-stream:0:724, it is saying that test-window-stream has partition 0 with a valid log segment which has the first offset = 724. Note that --time -1 is a special code for "only give the l

Re: Regarding Connection Problem

2016-12-17 Thread Kenny Gorman
Here are some examples, hope they help: https://github.com/Eventador/examples/tree/master/node Thanks Kenny Gorman www.eventador.io Sent from my iPhone > On Dec 17, 2016, at 1:56 PM, Hans Jespersen wrote: > > I would recommend you use either the Blizzard node-rdkafka module ( see > https://

Re: Regarding Connection Problem

2016-12-17 Thread Hans Jespersen
I would recommend you use either the Blizzard node-rdkafka module ( see https://github.com/Blizzard/node-rdkafka ) or the Confluent kafka-rest-node module ( see https://github.com/confluentinc/kafka-rest-node

Re: effect of high IOWait on KStream app?

2016-12-17 Thread Eno Thereska
Yeah, the numbers for the streams tests seem to be low. For reference, here is what I get when I run it on my laptop, with Kafka co-located (Macbook pro, 16GB, SSD). These are rounded up with no decimal places: > Producer Performance [MB/sec write]: 40 > Consumer Performance [MB/sec read]: 126

Regarding Connection Problem

2016-12-17 Thread Chintan Bhatt
Hi I want to give continuous output (avg. temperature) generated from node.js to store on Hadoop and then retrieve it for visualization. please guide me how to give continuous output of node.js to kafka. -- CHINTAN BHATT -- DISCLAIMER

Re: effect of high IOWait on KStream app?

2016-12-17 Thread Jon Yeargers
stateDir=/tmp/kafka-streams-simple-benchmark numRecords=1000 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/ec2-user/kafka/streams/build/dependant-libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding

Issue with "stuck" consumer in 0.9 broker

2016-12-17 Thread Robert Quinlivan
I am running an 0.9 broker and I'm having trouble viewing and committing offsets. Upon starting up the broker, I see the following in the kafka.out log: [2016-12-17 14:56:14,389] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket) I have one clien

Re: effect of high IOWait on KStream app?

2016-12-17 Thread Jon Yeargers
I'd be happy to but the AWS AMI (default) i'm using is fighting this at every turn. Will keep trying. On Sat, Dec 17, 2016 at 2:46 AM, Eno Thereska wrote: > Jon, > > It's hard to tell. Would you be willing to run a simple benchmark and > report back the numbers? The benchmark is called SimpleBen

Re: Restarting a failed kafka application

2016-12-17 Thread Eno Thereska
Hi Sachin, I think Matthias meant that you can chance the compaction configuration parameters when you create the topic, but you're right, by default you shouldn't need to do anything since the topic will be eventually compacted automatically. Eno > On 17 Dec 2016, at 08:23, Sachin Mittal wr

Re: effect of high IOWait on KStream app?

2016-12-17 Thread Eno Thereska
Jon, It's hard to tell. Would you be willing to run a simple benchmark and report back the numbers? The benchmark is called SimpleBenchmark.java, it's included with the source, and it will start a couple of streams apps. It requires a ZK and a broker to be up. Then you run it: org.apache.kafka.

Re: KTable#through from window store to key value store

2016-12-17 Thread Eno Thereska
Ok, makes sense. We'll be putting out a KIP on updating the KTable semantics and one of the things we'll add is the ability to materialize any KTable without explicitely needing to call through. So you could do a .mapValues, you'll get a KTable back and then we'll probably have a .materialize AP

Re: Restarting a failed kafka application

2016-12-17 Thread Sachin Mittal
Understood. What I understand so far is that by default changelog topics are compacted. So why do we need to compact these topics ourselves. Won't stream application take care of this. Thanks Sachin On 17 Dec 2016 11:47 a.m., "Matthias J. Sax" wrote: > If a Streams application is shutdown via