The mirroring process (both the embedded consumer and the mirror-maker tool) needs a zookeeper connection string. It should be possible to write a mirroring tool that talks to a broker, but you would most likely then need to have an alternate mechanism to do some load-balancing across the mirror's consumers. In that case, you could have something as simple as rsync'ing your partitions from each source broker to designated brokers in your mirror cluster (I guess for that to work you would need a consistent mapping). The easiest approach of course would be to have all your application servers talk to a common Kafka cluster that would be an aggregate, but you probably have a good reason to keep them separate.
Thanks, Joel > In the mirroring setup, the consumer configuration takes only zookeeper > connection string. I am wondering whether it can point directly to a > broker. > Here is what I am trying to do. > > 1. run my application server and a kafka broker on the same machine. My > application sends logs to the kafka broker > 2. have an agggregate kafka cluster which collects logs from the kafka > brokers on all my servers through mirroring. > > Zookeeper is not useful on my application servers since each has only one > broker running. But I have to run it for mirroring to work. Anyway to avoid > running zookeeper on my application server? > > Thanks, > > -Xiaoyu >
