Hey Guys, The new FetchRequest in the 0.8 branch has a fetch_size per topic/partition. Does this make sense? The purpose of the fetch_size is to allow the client to control how much data they need to fetch at once. I think setting this per-partition is a mistake since I don't really consider how many partitions or topics there are. For example, maybe there is a single partition on the server, so I configure 1MB which is the most memory I want to allocate, then someone adds many partitions on the server changing how much I fetch. Likewise based on how the partition balancing comes out I will have larger or smaller requests to a given server. Wouldn't it make more sense for there to be a single size at the top level of the request controlling the total size across all topic/partitions?
One detail is that to avoid "starvation" with a single size (i.e. if the first partition could always satisfy that size) the client should rotate the order of the topics and partitions in the fetch. Thoughts? -Jay
