Good day Kafka-users. To support our transition to Kafka as the central hub for data in our Big Data Platform, we created a new producer named Klogger (https://github.com/blackberry/klogger). It's a stripped down, high performance producer that can take a TCP port or file as an input, and produce to a defined Kafka topic. Klogger was specifically created to require minimal garbage collection by reusing objects, which is very important for our deployment. We have been running Klogger in production for almost a year now, across thousands of hosts, facilitating ingest rates over 40million lines per second.
Klogger is one of the utilities BlackBerry has created to work with Kafka. We have also created KaBoom (https://github.com/blackberry/kaboom) a high performance Kafka consumer and HDFS writer, and Krackle (https://github.com/blackberry/krackle) a lightweight Kafka producer/consumer library which is used by both Klogger and KaBoom. The project is still active internally, with frequent releases. Please have a look if you're interested. We welcome any feedback or contributions. Cheers, Todd Todd Snyder Senior Infrastructure Specialist, Big Data Platform tsny...@blackberry.com Always On, Always Connected.
