A somewhat related request came up in another thread and I think it is reasonable to provide this. However, there are already some indicators that you can use: - The consumer iterator throws a MessageSizeTooLargeException if it cannot extract any messages out of the next chunk. - If you are using the simple consumer, the fetch response includes the high watermark - so if the HW > your fetch offset but there are no iterable messages, then you will know that your fetch size needs to increase.
Thanks, Joel On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote: > Hello Apache Kafka community, > > Couldn't broker return a special error code in FetchResponse for a given > partition(s) where it detects that there was something to return/read from > partition but actual FetchResponse contains no messages for that partition > since fetch size in FetchRequest for that partition is too small? > > I don't see it already in existing Kafka wire protocol docs > <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>. > It doesn't seem to be supported, since docs for broker and consumer > configuration mention that the maximums should be in sync, otherwise too > large message could block consumer. > > If it was available this feature would be useful for simple consumer API > users, in the cases when range of possible message sizes on same topic is > relatively large (e.g. most very small messages, but some small fraction of > relatively large messages). Would help with being able to soft and hard > maximums, so most of the time one could read with soft max, and only in > case of this error code is on, attempt a read using hard max limit. > > Assuming this feature is not there, I guess what would need to be done at > minimum is: > - minimal change in protocol and docs, just one new error code > - accompanying detection of this condition on broker side, so extend > handling of FetchRequest > - maybe in high level consumer make use of this error code in a special way > (e.g. just log a warning/error that there is potential misconfiguration). > > Kind regards, > Stevo Slavic.
