[
https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700684#comment-14700684
]
Stefania commented on CASSANDRA-8630:
-------------------------------------
Thanks for your analysis.
I repeated the tests, 3 identical runs each time, albeit with a smaller data
set. They still indicate it is the uncompressed case where something has gone
wrong, not the compressed case. And more specifically I traced the slowness to
mmap disk access.
Here are the results, because I am on a 64-bit machine
{{disk_access_mode=auto}} resolves to {{mmap}} (although I am not sure at which
version this behavior started so it may not be true for all versions). In the
'uncomp-std' test I forced the disk access mode to standard.
||Version||Run 1||Run 2||Run 3||Rounded AVG||
|8630 comp|17.91|18.31|17.94|18|
|8630 uncomp|28.06|28.95|28.02|28|
|8630 uncomp-std|19.31|18.09|18.9|19|
|TRUNK comp|17.95|17.64|17.72|18|
|TRUNK uncomp|20.81|20.01|18.81|20|
|2.2 comp|19.95|20.33|19.97|20|
|2.2 uncomp|19.14|19.18|20.1|19|
|2.1 comp|21.61|20.43|20.43|21|
|2.1 uncomp|20.4|19.67|19.71|20|
|2.0 comp|18.8|19.42|19.66|19|
|2.0 uncomp|19.48|19.55|19.68|20|
Notes:
* Reduced data to 1M entries, which corresponds to approximately 220 MB of
data. This allowed me to keep the machine _more or less_ idle during the tests.
* All tests done with Java 8 update 51 except for 2.0 which was done with Java
7 update 80.
* Tests performed on a 64-bit linux laptop with SSD
* Compaction strategy was the default strategy used by the stress tool:
SizedTieredCompactionStrategy
Next I need to understand why mmap is so slow, I think I must have broken
something when I moved the segments to the RAR however.
bq. I usually set the file read and write and contention thresholds to one
millisecond.
What parameters do you use to achieve this?
> Faster sequential IO (on compaction, streaming, etc)
> ----------------------------------------------------
>
> Key: CASSANDRA-8630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8630
> Project: Cassandra
> Issue Type: Improvement
> Components: Core, Tools
> Reporter: Oleg Anastasyev
> Assignee: Stefania
> Labels: compaction, performance
> Fix For: 3.x
>
> Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png,
> flight_recorder_001_files.tar.gz, flight_recorder_002_files.tar.gz
>
>
> When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot
> of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int).
> This is because default implementations of readShort,readLong, etc as well as
> their matching write* are implemented with numerous calls of byte by byte
> read and write.
> This makes a lot of syscalls as well.
> A quick microbench shows than just reimplementation of these methods in
> either way gives 8x speed increase.
> A patch attached implements RandomAccessReader.read<Type> and
> SequencialWriter.write<Type> methods in more efficient way.
> I also eliminated some extra byte copies in CompositeType.split and
> ColumnNameHelper.maxComponents, which were on my profiler's hotspot method
> list during tests.
> A stress tests on my laptop show that this patch makes compaction 25-30%
> faster on uncompressed sstables and 15% faster for compressed ones.
> A deployment to production shows much less CPU load for compaction.
> (I attached a cpu load graph from one of our production, orange is niced CPU
> load - i.e. compaction; yellow is user - i.e. not compaction related tasks)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
