[
https://issues.apache.org/jira/browse/KAFKA-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
C. Scott Andreas updated KAFKA-132:
-----------------------------------
Attachment: patch.txt
Patch applied to core/src/main/scala/kafka/log/Log.scala in version 0.6.1
> Reduce Unnecessary Filesystem Writes for Logs Without Unflushed Events
> ----------------------------------------------------------------------
>
> Key: KAFKA-132
> URL: https://issues.apache.org/jira/browse/KAFKA-132
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.6
> Environment: Linux 2.6.38-10-server (Ubuntu 10.04.3 LTS), XFS
> Filesystem
> Reporter: C. Scott Andreas
> Priority: Minor
> Attachments: patch.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We noticed a fair amount of stress on our filesystem in an environment with a
> large number of topics but low message activity. After some investigation, we
> realized that a short log.flush.interval coupled with a large number of
> topics resulted in a lot of unnecessary disk activity, even without events to
> be written.
> This activity occurs because FileChannel.force(true) is called on the
> underlying FileMessageSet for each log, even when there are no messages to be
> written. This call forces unproductive writes to the underlying filesystem.
> This case is especially stressed in an environment with a large number of
> low-activity topics for which low latency is still important. Here is the
> before-and-after output of `iostat -x 2` on a system with 1044 topics and a
> timed flush interval of 100ms. Note the reduction in %util and writes/second.
> In the "before" output, we see 40-80% util and ~260 writes/second. In the
> "after" output, we see 10-15% util and ~65 writes/second.
> Pre-patch output:
> https://raw.github.com/gist/54d0f4c62753a6e2de1f/7ee1982bfa8e5c088bcf9ba953f01956443bd31e/iostat-pre-kafka-patch.txt
> Post-patch output:
> https://raw.github.com/gist/54d0f4c62753a6e2de1f/b939973c7fed642480856d9bdeb2e4cb0ada445b/iostat-post-kafka-patch.txt
> The proposed patch (see attached) skips calling the underlying FileMessageSet
> flush operation if the log's atomic counter indicates that there are no
> messages to be written.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
