[
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221116#comment-15221116
]
ASF GitHub Bot commented on STORM-1674:
---------------------------------------
Github user moesol commented on the pull request:
https://github.com/apache/storm/pull/1287#issuecomment-204239216
KafkaConfig has socketTimeoutMs which I assume maps to socket.timeout.ms.
It's passed along to the SimpleConsumer's constructor.
I'm not sure yet where rebalance.max.retries etc. would need to get
applied. They are not a part of the FetchRequest. My vote is to keep this pull
request focused on fixing the issue at hand and have a different pull request
to implement the comprehensive list of configurations.
Also, I thought I saw some work to create a KafkaSpout that uses the Kafka
0.9 client. Perhaps the comprehensive list of configurations should be coupled
with that work.
> Idle KafkaSpout consumes more bandwidth than needed
> ---------------------------------------------------
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 0.9.3
> Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 10000
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
