Maybe support both?

On Mar 29, 2012, at 1:52 AM, Jay Kreps <jay.kr...@gmail.com> wrote:

> Hey Guys,
>
> The new FetchRequest in the 0.8 branch has a fetch_size per
> topic/partition. Does this make sense? The purpose of the fetch_size
> is to allow the client to control how much data they need to fetch at
> once. I think setting this per-partition is a mistake since I don't
> really consider how many partitions or topics there are. For example,
> maybe there is a single partition on the server, so I configure 1MB
> which is the most memory I want to allocate, then someone adds many
> partitions on the server changing how much I fetch. Likewise based on
> how the partition balancing comes out I will have larger or smaller
> requests to a given server. Wouldn't it make more sense for there to
> be a single size at the top level of the request controlling the total
> size across all topic/partitions?
>
> One detail is that to avoid "starvation" with a single size (i.e. if
> the first partition could always satisfy that size) the client should
> rotate the order of the topics and partitions in the fetch.
>
> Thoughts?
>
> -Jay

Reply via email to