ConsumerOffsetChecker returns negative value for log lag

2014-10-18 Thread Yu Yang
Hi, I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool to check the lag of consumer groups. I found that for some partition, the tool returns negative value for the lag column. Is this a known issue that has been seen before? I find that the negative value prevents the consumer

Too much log for kafka.common.KafkaException

2014-10-18 Thread xingcan
Hi, all Recently, I upgrade my Kafka cluster to 0.8.1.1 and set replication with num.replica.fetchers=5. Last night, there's something wrong with the network. Soon, I found the server.log files (not data log!) on every node reached 4GB in an hour. I am not sure if it's my inappropriate

Re: Too much log for kafka.common.KafkaException

2014-10-18 Thread Ewen Cheslack-Postava
This looks very similar to the error and stacktrace I see when reproducing https://issues.apache.org/jira/browse/KAFKA-1196 -- that's an overflow where the data returned in a FetchResponse exceeds 2GB. (It triggers the error you're seeing because FetchResponse's size overflows to become negative,

Re: postgresql consumer

2014-10-18 Thread Sa Li
Hi, all I've just made a 3-node kafka cluster (9 brokers, 3 for each node), the performance test is OK. Now I am using tridentKafkaSpout, and being able to getting data from producer, see BrokerHosts zk = new ZkHosts(10.100.70.128:2181); TridentKafkaConfig spoutConf = new