Suggestions on load testing a system that uses kafka as underlying message bus?

2015-09-19 Thread David Luu
I'd like to generate load against a system we have that uses kafka as the message bus. We have a custom JSON message format, and to properly load test the system, each set of messages for a particular scenario (i.e. user) needs to have a unique identifier, which it normally does. I think of using

missing topic error message not useful

2015-09-19 Thread Richard Lee
Hi there- The error message returned when trying to produce to a nonexistent kafka topic (with auto topic creation disabled) is non obvious. It returns an error message of the form: "org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after xx ms" where xx is a small n

Flume use case for Kafka & HDFS

2015-09-19 Thread Hemanth Abbina
I'm new to Flume and thinking to use Flume in the below scenario. Our system receives events as HTTP POST, and we need to store them in Kafka(for processing) as well as HDFS(as permanent store). Can we configure Flume as below ? * Source: HTTP (expecting JSON event as HTTP body, with a