[
https://issues.apache.org/jira/browse/CASSANDRA-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701515#comment-14701515
]
Ariel Weisberg edited comment on CASSANDRA-8630 at 8/18/15 4:24 PM:
--------------------------------------------------------------------
I guess I mixed up compressed and uncompressed?
JFR has an option to load settings from an XML file. You can export the xml
file from JMC and then modify the fields you care about. There are some options
in there for those thresholds that I don't know off hand. You can also set them
in the U and then export.
I ran without rate limiting (commented out the acquire) and the original
workload.
|Version|Run1||
|8630 uncompressed|202|
|8630 compressed|197|
|Trunk uncompressed|201|
|Trunk compressed|203|
Looking at the profile I think costs here are dominated by all the Java code
for merging. I'll run with JMC and no rate limiting and see where that gets me.
I suspect it's going to be very similar.
It kind of looks like a bit more than 50% of time is spent materializing data
and maybe 40% is spent rewriting it. The top 4 classes are 3 merge iterators
and ConcurrentHashMapV8 in concurrent linked hashmap. I guess cache maintenance
is part of the overhead of compaction.
I am hesitant to prognosticate on where to go to make compaction faster because
I know a lot of this stuff is already recorded or debunked. Optimizing single
threaded stuff does not seem like the right starting point to me though.
was (Author: aweisberg):
I guess I mixed up compressed and uncompressed?
JFR has an option to load settings from an XML file. You can export the xml
file from JMC and then modify the fields you care about. There are some options
in there for those thresholds that I don't know off hand. You can also set them
in the U and then export.
I ran without rate limiting (commented out the acquire) and the original
workload.
|Version|Run1||
|8630 uncompressed|202|
|8630 compressed|197|
|Trunk uncompressed|201|
|Trunk compressed|203|
Looking at the profile I think costs here are dominated by all the Java code
for merging. I'll run with JMC and no rate limiting and see where that gets me.
I suspect it's going to be very similar.
It kind of looks like a bit more than 50% of time is spent materializing data
and maybe 40% is spent rewriting it. The top 4 classes are 3 merge iterators
and ConcurrentHashMapV8 in concurrent linked hashmap. I guess cache maintenance
is part of the overhead of compaction.
It looks like we are optimizing the wrong part of the profile? Have things
changed since <2.2? I kind of think we are DOA if compaction within a single
column family is single threaded anyways. I think I heard of such a thing
existing, but no one was satisfied with it.
> Faster sequential IO (on compaction, streaming, etc)
> ----------------------------------------------------
>
> Key: CASSANDRA-8630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8630
> Project: Cassandra
> Issue Type: Improvement
> Components: Core, Tools
> Reporter: Oleg Anastasyev
> Assignee: Stefania
> Labels: compaction, performance
> Fix For: 3.x
>
> Attachments: 8630-FasterSequencialReadsAndWrites.txt, cpu_load.png,
> flight_recorder_001_files.tar.gz, flight_recorder_002_files.tar.gz,
> mmaped_uncomp_hotspot.png
>
>
> When node is doing a lot of sequencial IO (streaming, compacting, etc) a lot
> of CPU is lost in calls to RAF's int read() and DataOutputStream's write(int).
> This is because default implementations of readShort,readLong, etc as well as
> their matching write* are implemented with numerous calls of byte by byte
> read and write.
> This makes a lot of syscalls as well.
> A quick microbench shows than just reimplementation of these methods in
> either way gives 8x speed increase.
> A patch attached implements RandomAccessReader.read<Type> and
> SequencialWriter.write<Type> methods in more efficient way.
> I also eliminated some extra byte copies in CompositeType.split and
> ColumnNameHelper.maxComponents, which were on my profiler's hotspot method
> list during tests.
> A stress tests on my laptop show that this patch makes compaction 25-30%
> faster on uncompressed sstables and 15% faster for compressed ones.
> A deployment to production shows much less CPU load for compaction.
> (I attached a cpu load graph from one of our production, orange is niced CPU
> load - i.e. compaction; yellow is user - i.e. not compaction related tasks)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
