Maybe support both? On Mar 29, 2012, at 1:52 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
> Hey Guys, > > The new FetchRequest in the 0.8 branch has a fetch_size per > topic/partition. Does this make sense? The purpose of the fetch_size > is to allow the client to control how much data they need to fetch at > once. I think setting this per-partition is a mistake since I don't > really consider how many partitions or topics there are. For example, > maybe there is a single partition on the server, so I configure 1MB > which is the most memory I want to allocate, then someone adds many > partitions on the server changing how much I fetch. Likewise based on > how the partition balancing comes out I will have larger or smaller > requests to a given server. Wouldn't it make more sense for there to > be a single size at the top level of the request controlling the total > size across all topic/partitions? > > One detail is that to avoid "starvation" with a single size (i.e. if > the first partition could always satisfy that size) the client should > rotate the order of the topics and partitions in the fetch. > > Thoughts? > > -Jay