Garrick Dasbach created STORM-1471:
--------------------------------------
Summary: Make FetchRequestBuilder.minBytes a configurable
parameter in SpoutConfig
Key: STORM-1471
URL: https://issues.apache.org/jira/browse/STORM-1471
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Reporter: Garrick Dasbach
We currently have an issue in our storm cluster where our Kafka brokers are
under heavy load due to too many fetch requests from storm. We've narrowed the
problem to the way Fetch Requests are build in KafkaUtils. When using the
FetchRequestBuilder, storm provides overrides for all the properties except
minBytes. The default for that field is 0 (even though the Kafka default for
the high-level consumer is 1). When paired with a maxWait > 0, this creates a
situation where the broker can immediately return a response without waiting
(due to minBytes 0). This puts a heavy load on the brokers and defeats the
purpose of any long polling.
By making this a SpoutConfig option, it will allow the user to set that as
appropriate for their situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
