[
https://issues.apache.org/jira/browse/CHUKWA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456595#comment-13456595
]
Sourygna Luangsay commented on CHUKWA-664:
------------------------------------------
You were right: too many flushes due to compression.
Nonetheless, I don't think that compression is called for every chunk. I only
call compressionOutputStream.finish() method once every chunks are written on
the stream.
I have played a bit with Hadoop "io.file.buffer.size parameter" and I managed
to get bigger (and less numerous) TCP fragments if I increase this buffer
variable. So I guess that would fix the TCP incast problem (I have also tried
changing the "chukwaAgent.fileTailingAdaptor.maxReadSize" parameter and got
more interesting results).
The only trouble is that "io.file.buffer.size parameter" currently only works
if you load the native-hadoop library for compression. I have got to have a
better look at my code and Hadoop compression package and see if I can enable
it if native-hadoop is not loaded.
And do you have any idea why HTTP POST can't be seen when compression is
enabled?
> network compression between agent and collector
> -----------------------------------------------
>
> Key: CHUKWA-664
> URL: https://issues.apache.org/jira/browse/CHUKWA-664
> Project: Chukwa
> Issue Type: New Feature
> Components: Data Collection
> Affects Versions: 0.5.0, 0.6.0
> Reporter: Sourygna Luangsay
> Priority: Trivial
> Fix For: 0.6.0
>
> Attachments: chukwa-664.patch
>
>
> As suggested in
> http://mail-archives.apache.org/mod_mbox/incubator-chukwa-user/201207.mbox/%3C001b01cd69b4$13d9c100$3b8d4300$@com%3E
> , Chukwa should be able to compress network communications between agent and
> collector.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
