Re: kafka.SocketServerStats write rate metrics seem wrong

No compression is configured, the spool files can be /usr/bin/strings'd to show the msgs. On Tue, Aug 5, 2014 at 2:13 PM, Guozhang Wang wrote: > Hello Neil, > > Did you use any compression on your producers? > > Guozhang > > > On Tue, Aug 5, 2014 at 12:23 PM, Neil Harkins wrote: >> >> Hi. >> I'

Re: Kafka 0.8 automatically changing leadership

The session expirations (in the log you pasted) lead to the broker losing its registration from zookeeper (which triggers the 'broker failure callback' in the controller) - that causes leader election and leaders to move. Session expirations are typically due to GC so you can take a look at https:/

Re: Kafka 0.8 automatically changing leadership

Joel, Thanks for responding. I see no "Broker failure callback" messages in the logs. However, I did find this: [2014-08-01 16:47:19,866] INFO Client session timed out, have not heard from server in 4213ms for sessionid 0x143f9e2c9956ee0, closing socket connection and attempting reconnect (org

Re: Apache webserver access logs + Kafka producer

You might be interested by something like Logstash http://logstash.org for logs and event processing. Regards, Florian Le 5 août 2014 23:17, "Jonathan Weeks" a écrit : > You can look at something like: > > https://github.com/harelba/tail2kafka > > (although I don’t know what the effort would be

Re: Apache webserver access logs + Kafka producer

You can look at something like: https://github.com/harelba/tail2kafka (although I don’t know what the effort would be to update it, as it doesn’t look like it has been updated in a couple years) We are using flume to gather logs, and then sending them to a kafka cluster via a flume kafka sink

Re: kafka.SocketServerStats write rate metrics seem wrong

Hello Neil, Did you use any compression on your producers? Guozhang On Tue, Aug 5, 2014 at 12:23 PM, Neil Harkins wrote: > Hi. > I'm seeing some odd numbers from kafka.SocketServerStats. > > Ideally, I'd like to have stats broken down per-topic, e.g. > what's our most written/read topics? For

Apache webserver access logs + Kafka producer

Hi, I want to collect apache web server logs in real time and send it to Kafka server. Is there any existing Producer available to do this operation, If not can you please provide a way to implement it. Regards, Sree.

kafka.SocketServerStats write rate metrics seem wrong

Hi. I'm seeing some odd numbers from kafka.SocketServerStats. Ideally, I'd like to have stats broken down per-topic, e.g. what's our most written/read topics? For write rates, I've got a separate process iterating topics every minute, doing this: (head_offset_now - head_offset_last) / (time_now -

Is there a way to stop/pause producers from writing?

Is there a tool or way using the Kafka admin api to tell a producer to stop/pause writing to a specific topic? The use case is basically we need to stop writing to a topic, let the consumers get caught up and then deploy some new code either for the producers or consumers. Our producers will have

Re: Consumer is never shutdown

Without looking at your complete code, I'm only going to make some guesses. Unless you really want to shutdown the consumer, why would you call consumer.shutdown()? A consumer will keep waiting for new messages unless otherwise specified. I'm guessing you used something like while iterator.hasNex

Re: high level consumer api blocked forever

Hi, every one. forget it, sorry it's my fault. when my thread throwed a exception i caught it and then call "createJavaConsumerConnector" with out close old one... then of course cause redundant consumers in consumer group some of which will never get messages. Thanks 2014-08-05 11:48 GMT+08:00