Re: Colocating Kafka Connect on Kafka Broker

2016-07-23 Thread Ewen Cheslack-Postava
Generally we discourage colocating services with Kafka. Kafka relies heavily on the page cache. It's generally light on CPU (except maybe if it has to recompress messages), but may not play well with other services. For very light installations, colocating some services (e.g. both ZK and Kafka),

Colocating Kafka Connect on Kafka Broker

2016-06-29 Thread Kristoffer Sjögren
Hi We want to use Kafka Connect to copy data to HDFS (using kafka-connect-hdfs) in parquet format and was wondering if its a good idea to collocate distributed Kafka Connect 1-1 on Kafka Brokers? Considering the parquet indexing process would steal (a lot of / too much?) computing resources from