Hi Kafka is just what we need for log aggregation and dispatching. I am play with it recently. But found that the docs is pretty poor. I got the following links to find docs and sources of Kafka. It's either lack of docs or inconsistent between different sources or even broken ( eg. some examples on [1]).
[1] http://incubator.apache.org/kafka/index.html [2] https://cwiki.apache.org/confluence/display/KAFKA/Index [3] https://github.com/apache/kafka [4] https://github.com/kafka-dev/kafka Also there's no clear way to get the latest Kafka release (no maven yet). Spent some time and finally find this http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating-candidate-7/link to get the lastest lib. For compression, there's no example to show how to use it. I tried with the following config it works. val props = new Properties(); props.put("zk.connect", "127.0.0.1:2181"); props.put("producer.type", "async"); props.put("batch.size", "50") props.put("serializer.class", "kafka.serializer.StringEncoder"); props.put("compression.codec", "1") //gzip val config = new ProducerConfig(props); But if I remove the serializer.class config : props.put("serializer.class", "kafka.serializer.StringEncoder"); The consumer-shell can no longer get the messages published by producer, so it's like there's something wrong, but no exception got. What serializer.class should I use when compression.codec=1 (gzip) is used ? Why Kafka doesn't throw exception when no serializer.class is configured ? Lack of documentation is frustrating and I have to spend many time digging the Kafka source code. Thanks for your help. Best Regards, Stone
