Common Form of Data Written to Kafka for Data Ingestion

2015-03-24 Thread Rendy Bambang Junior
Hi, I'm a new Kafka user. I'm planning to send web usage data from application to S3 for EMR and MongoDB using Kafka. What is common form to write as message in Kafka for data ingestion use case? I am doing a little homework and find Avro as one of the options. Thanks. Rendy

Re: 2 kafka cluster sharing same ZK Ensemble.

2015-03-27 Thread Rendy Bambang Junior
Based on documentation, as long as you define different folder zookeeper chroot at broker configuration, it should be OK. Cmiiw. Disclaimer: myself never tried this scheme. Rendy On Mar 28, 2015 2:14 AM, Shrikant Patel spa...@pdxinc.com wrote: Can 2 separate kafka cluster share same ZK

Differences between new and legacy scala producer API

2015-05-07 Thread Rendy Bambang Junior
Hi - Legacy scala api for producer is having keyed message along with topic, key, partkey, and message. Meanwhile new api has no partkey. Whats the difference between key and partkey? - In javadoc, new producer api send method is always async, does producer.type properties overriden? - Will scala

Kafka broker and producer max message default config

2015-05-12 Thread Rendy Bambang Junior
Hi, I see configuration for broker max.message.bytes 1,000,000 and configuration for producer max.request.size 1,048,576 Why is default config for broker is less than producer? If that is the case then there will be message sent by producer which is bigger than what broker could receive. Could

Re: Kafka broker and producer max message default config

2015-05-12 Thread Rendy Bambang Junior
purpose is, as the name implies, to limit the size of a request, which could potentially include many messages. This keeps the producer from sending very large requests to the broker. The limitation on message size is just a side effect. On Tue, May 12, 2015 at 12:33 AM, Rendy Bambang Junior