Hey All, We have been discussing different extensions to the kafka network protocol to support richer APIs and some of the needs of the replication work (esp. KAFKA-48 and KAFKA-49). From past experience I can say one thing we really want to avoid is incremental or iterative design for our service apis--this always leads to a mess that gets carried into each client implementation. So to avoid this, I think it would be a good idea to centralize all the discussions on protocol changes. This will also allow us to have a single set of JIRAS to add the request object support separate from the features these fields support--this will avoid us having 7 different overlapping patches floating around that all work on the same code.
To kick this off I added a wiki here with all the various proposals I collected from different threads: https://cwiki.apache.org/confluence/display/KAFKA/New+Wire+Format+Proposal For reference the current state of the network apis has fantastic documentation here (thanks to Jeffrey and Taylor and others): https://cwiki.apache.org/confluence/display/KAFKA/Wire+Format The feedback I would like is whether or not this covers all the needs or there is anything missing we haven't thought of. One big question we should try to answer is whether it is better to do a one-time incompatible change or map the old request types back onto the new ones on the server side to retain compatibility for one or two releases to give a gentler migration path. For example I am not sure how feasible a non-backwards compatible release would be at LinkedIn given the large number of clients that would have to be simultaneously upgraded. -Jay
