I am seeing poor throughput when using the kafka output plugin. It appears that when I set the max buffer time it has a bad effect and seems to make it pause between messages for whatever the value is set too. Here is my config:
[hekad] maxprocs = 2 base_dir = "/tmp/heka" share_dir = "/tmp/heka/share/heka" poolsize = 300 …. [KafkaOutput] type = "KafkaOutput" message_matcher = 'Logger == "LogstreamerInput" && Fields[TopicName] != NIL' topic_variable = "Fields[TopicName]" addrs = [“broker:9092"] encoder = “MyEncoder" partitioner = "RoundRobin" #max_buffer_time = 1000 #max_buffered_bytes = 1000000 #required_acks= "WaitForLocal" #compression_codec=“Snappy" I am trying to figure out what is the best way to tune the Kafka output for small, high volume (1000/sec) messages. This is running on an edge that is in the same data center as the brokers. The ping time to the brokers is sub ms so it is not a network latency issue. It appears to be sending one message at a time, is this the expected behavior? How do you increase the batch.size to produce when sending to kafka, if that is even possible? Thank you for your help Chris
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
