Hi, I've been looking at Kafka for a while now, and am getting ready to start moving an implementation we have been working on into production.
Previously, I asked about support for large numbers of topics per kafka instance, and was not surprised to learn it would have a number of limitations such as the filesystem itself and zookeeper. Through testing we've determined a comfortable number of live topics at around 20k. In the future, I'd like to start digging into the Kafka internals to see if this number can't be increased…however in the short term we will be going with an unmodified version of Kafka which we will build some additional infrastructure to support our needs. Thus I have the following questions regarding the protocol… Our use case requires greater than 20k topics system-wide, thus I am looking to shard multiple instances of Kafka sharding by topic name and using a consistent hashing scheme to pick the correct physical instance. This poses a challenge, however. The routing to each sharded kafka instance has to be done somewhere, either in the client driver, above the client driver, or as a proxy in between. I've chosen to go with a proxy in between, as it centralizes the sharding knowledge into a single manageable tier. However, the design of the kafka protocol makes this very difficult to implement - here's why. While the kafka protocol supports pipelining, it doesn't support out-of-order responses because the responses do not indicate for which topic they are a response to. Therefore, to implement a proxy I would have no choice but to buffer responses from different sharded instances and re-order them into the client issued order, or else the client would be confused. I propose a modification to the protocol to allow the client to receive responses out of order. In the proxy use case, this would eliminate the need to buffer and re-order responses and allow the proxy to be mostly a request/response i/o router. Also, on the kafka side, this would also allow the kafka server to service requests out of order, which could, for some use cases, improve response times, although for that to be actually realized, I suspect the kafka code might have to support storing topics on multiple disk mount points and we would have to assume that Java NIO could handle true async disk requests to multiple disk subsystems. It probably does but I've never pushed Java async disk i/o that far…maybe you have? So…the point of my long-winded post is - I propose a modification to the protocol to implement out of order responses. This could be implemented in one of two ways - either embed the topic name in the response header, or, maybe preferably, force the client to issue a 64-bit token for each request, and the response can include that token in the response (the latter proposal makes the parsing easier, and shortens the response without unduly burdening the implementations with too much complexity). As I said, we are moving forward with a proxy based solution and it will most likely take the existing protocol and modify it in this way. What do you think of this effort? Thanks! Taylor
