David, The benefit of the strategy used in Solr is that it simplifies client routing. The downside is potential additional RPC overhead and a bit more logic in the server. Technically, you can achieve what Solr does in the client layer too. You can run a proxy that runs the java version of Kafka producer and exposes a restful api. Then, your non-java client can talk to the proxy.
We do plan to support a restful api for the producer in the future. Doing the Solr strategy needs more thinking since currently, not every broker knows the leader of all partitions. Thanks, Jun ---------- Forwarded message ---------- From: David Arthur <[email protected]> Date: Mon, Feb 11, 2013 at 7:45 AM Subject: Clients and replica leaders To: "[email protected]" <[email protected]> In writing a client for 0.8, I now have to keep state of which topic+partition is owned by what broker. This is inherently a pain to deal with and has the downside that I must wait for an error before I am notified about a change in the broker topology. I would be nice if the clients didn't need to know so much about the brokers. In Apache Solr, which actually has a similar partition+replication strategy, each server (broker) can handle requests for any shard (partition) in the cluster. If the current server happens to be the leader then it will process the request; if not it will forward it to the correct server, wait for a response, then forward the response back to the client. Dumb clients will pay the extra cost of the additional hop, but do not need to know anything about the brokers. Smart clients will work basically like they would now with the added benefit of not getting an error when leader changes. Would a strategy like this work in 0.8? Do the brokers know about one another? -David
