Hi, I recently experimented with the hadoop-consumer and I wrote a script that makes it easier to use.
I wrote a blog post describing what it does and the source code. I think it might be worth contributing back to Kafka. It might need some changes before it can be contributed. I don't know, it's the first time I would contribute anything to an open-source project :) Let me know what you think! Here's the blog post: http://felixgv.com/post/69/automating-incremental-imports-with-the-kafka-hadoop-consumer/ -- Felix