[
https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881433#comment-13881433
]
dan jatnieks edited comment on CASSANDRA-6357 at 1/24/14 9:20 PM:
------------------------------------------------------------------
I did some stress write testing on a single node physical machine with multiple
rotating disks. The machine was a quad-core Intel Xeon E3-1230, 32Gb memory,
and 4 disks (2Tb, 5400 rpm).
Two scenarios were measured using stress-write for 10 million records
(cassandra-stress --num-keys 10000000 --columns=50 --operation=INSERT):
Base scenario:
* O/S on separate device
* Commitlog on separate device
* Data and flush directories on the same device
Flush scenario:
* O/S on separate device
* Commitlog on separate device
* Data directory on separate device
* Flush directory on separate device
The result of splitting the flush directory to another device on the
stress-write results was:
* an elapsed time improvement of about 57%
* a much lower, and consistent, 99.9th percentile latency (3308 ms vs 15963 ms)
See attached graph: [Stress Write Latency 99.9th
Percentile|^c6357-stress-write-latency-99th-1.png]
was (Author: djatnieks):
I did some stress write testing on a single node physical machine with multiple
rotating disks. The machine was a quad-code Intel Xeon E3-1230, 32Gb memory,
and 4 disks (2Tb, 5400 rpm).
Two scenarios were measured using stress-write for 10 million records
(cassandra-stress --num-keys 10000000 --columns=50 --operation=INSERT):
Base scenario:
* O/S on separate device
* Commitlog on separate device
* Data and flush directories on the same device
Flush scenario:
* O/S on separate device
* Commitlog on separate device
* Data directory on separate device
* Flush directory on separate device
The result of splitting the flush directory to another device on the
stress-write results was:
* an elapsed time improvement of about 57%
* a much lower, and consistent, 99.9th percentile latency (3308 ms vs 15963 ms)
See attached graph: [Stress Write Latency 99.9th
Percentile|^c6357-stress-write-latency-99th-1.png]
> Flush memtables to separate directory
> -------------------------------------
>
> Key: CASSANDRA-6357
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6357
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Patrick McFadin
> Assignee: Jonathan Ellis
> Attachments: 6357.txt, c6357-stress-write-latency-99th-1.png
>
>
> Flush writers are a critical element for keeping a node healthy. When several
> compactions run on systems with low performing data directories, IO becomes a
> premium. Once the disk subsystem is saturated, write IO is blocked which will
> cause flush writer threads to backup. Since memtables are large blocks of
> memory in the JVM, too much blocking can cause excessive GC over time
> degrading performance. In the worst case causing an OOM.
> Since compaction is running on the data directories. My proposal is to create
> a separate directory for flushing memtables. Potentially we can use the same
> methodology of keeping the commit log separate and minimize disk contention
> against the critical function of the flushwriter.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
