[
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294320#comment-15294320
]
ASF GitHub Bot commented on STORM-1674:
---------------------------------------
Github user moesol commented on the pull request:
https://github.com/apache/storm/pull/1393#issuecomment-220727660
I think it's awesome that this change is getting in! I'll be looking out
for the next release that contains this PR.<br>
Robert Hastings, Lab SDL: 619-553-0826, Cell: 858-395-4436, Moebius:
858-455-5431x210
SIPR: hast...@spawar.navy.smil.mil
JWICS: hast...@spawar.navy.ic.gov
From: "P. Taylor Goetz" <notificati...@github.com>
To: "apache/storm" <st...@noreply.github.com>
Cc: "rhastings" <rhasti...@moesol.com>, "Mention"
<ment...@noreply.github.com>
Sent: Friday, May 20, 2016 1:03:11 PM
Subject: Re: [apache/storm] (STORM-1674) Idle KafkaSpout consumes more
bandwidth than needed (#1393)
Thanks @moesol I merged this into master, 1.x-branch and 1.0.x-branch.
Note that with 1.0.x-branch this created a conflict, as
KafkaConfig.minFetchByte had been introduced to do the same thing as your
KafkaConfig.fetchMinBytes variable. These are public variables, so they make up
part of the API.
Since 1.0.1 had already been released with the minFetchByte varibable, to
ensure backward compatibility I used the minFetchByte variable name across
branches.
While I prefer the variable name you chose, I think it's more important to
ensure backward compatibility. :)
Thanks again!
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
> Idle KafkaSpout consumes more bandwidth than needed
> ---------------------------------------------------
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 0.9.3, 0.10.1, 1.0.1
> Reporter: Robert Hastings
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 10000
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
