Re: kafka 0.8 producer issue

2015-12-04 Thread Raju Bairishetti
For setting up logs for kafka, we need to export KAFKA_LOG4J_OPS env variable before starting the kafka servers. if log4j.properties is presented in /etc/kafka/ directory then it will be directly picks otherwise it uses tools-log4j.properties from /etc/kafka. You can set the location of server log

Re: managing replication between nodes on specific NICs on a host

2015-12-04 Thread Lance Laursen
Hi, You're not going to be able to get it to do what you want. When a publisher queries a list of seed brokers for cluster information, the host names that are passed back in the metadata response are the same hostnames the brokers use to talk to and replicate between each other. Run zkCli.sh then

managing replication between nodes on specific NICs on a host

2015-12-04 Thread scott macfawn
I am attempting to see if i can maximize throughput on 1GB interfaces. Each of my brokers has 2 1GB interfaces on it, on two different subnet, with two different CNAMEs. Currently, I am seeing that one of the interfaces is taking 90% of the traffic where the other is basically taking 10% of the tra

Re: Kafka uncomitted offset

2015-12-04 Thread Gaurav Agarwal
Tx On Dec 4, 2015 9:11 PM, "scott macfawn" wrote: > I think what you are asking for is the "lag" on a topic for a specific > consumer. This is basically how far behind you are falling behind the > producer for a topic that is being written to the cluster. > > something like this should help you f

Re: Kafka uncomitted offset

2015-12-04 Thread scott macfawn
I think what you are asking for is the "lag" on a topic for a specific consumer. This is basically how far behind you are falling behind the producer for a topic that is being written to the cluster. something like this should help you figure that out. [path of kafka]/kafka-run-class.sh kafka.too

How to deploy Kafka on AWS

2015-12-04 Thread SpikyHawk SpikyHawk
Hi I am starting to analyze how to deploy Kafka (we are using AWS) and would like to hear what are you doing. I am particularly interested in knowing things like these: Do you use Docker for Kafka and/or ZooKeeper? If using Docker, which OS, CoreOS, other? Are you using some scheduler like Kuber

Re: Trying to understand the format of the LogSegment file.

2015-12-04 Thread Steve Graham
Very helpful, thanks Magnus. Should the documentation found in http://kafka.apache.org/documentation.html#messages be updated to reflect this format for messages? So, to close this one out, this is the section in the protocol guide (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+

Cannot send message to Kafka broker

2015-12-04 Thread chengxin Cai
Hi, I just try Kafka the first time. I am using hortonworks 2.3, and my setting of Kafka broker is "listeners =PLANTEXT://localhost:9092" I tried with "kafka-console-producer.sh --topic test --broker-lists localhost:9092", and I got lots of errors. Also, I try "netstat -a|grep 9092", cannot get