[
https://issues.apache.org/jira/browse/CASSANDRA-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943083#comment-14943083
]
Stefania commented on CASSANDRA-10407:
--------------------------------------
Thank you for the tests [~aboudreault]. It seems that when the patch was
committed the performance of std was really bad (44-45k ops/second) compared to
mmap (66-68k ops/second) and neither the patch nor readahead had any impact.
However, this has changed quite radically with 3.0 rc1 (52k vs 57-59k). It's
difficult to know why without bisecting but I think we should forget what the
code was like back then and focus on the current 3.0 head and aim to have std
without readahead as good as or better than mmap with readahead. We should also
bring mmap back to 66-68k ops/second.
I did some profiling by running mmap vs std locally and I've attached a couple
of _.nps_ files that can be opened with [visual
vm|http://visualvm.java.net/download.html]. I've also attached some screenshots.
Here are some things that I've noticed:
* In RAR.overrideLength() the call to channel.size() is expensive, we should
cache the size in the channel or avoid calling this (it is just to throw an
IllegalArgumentException)
* Most of the time we spend in {{reBuffer}} follows a call from
{{SSTableReader.getFileDataInput()}}, which is called when initializing
iterators. Here a new reader is created so we cannot avoid rebuffering, also we
spend a lot of time in dealing with checksums, more than we spend reading.
* CRAR is allocating buffers that are too big for the buffer pool (> 64kb) and
therefore we spend time calling {{allocateDirect()}}: {{INFO 07:57:16
Requested buffer size 65813 is bigger than 65536, allocating directly}}
We should really address these points first (the first one is trivial but the
other two I am not as familiar with). However, we could also in parallel try
running these tests to see if there are any differences with the STD case:
* 3.0 HEAD MMAP with readahead
* 3.0 HEAD STD without readahead and setting this in the yaml:
{{disk_optimization_page_cross_chance: 0.1}} - this is actually the current
default
* 3.0 HEAD STD without readahead and setting this in the yaml:
{{disk_optimization_page_cross_chance: 0.5}}
* 3.0 HEAD STD without readahead and setting this in the yaml:
{{disk_optimization_page_cross_chance: 0.9}}
[~benedict] WDYT regarding the points I've raised? Is it worth addressing them
before running more tests or would you expect a big difference in the tests
anyway? Also, is there anything that needs to be done to disable readahead,
other than calling {{blockdev --setra}}?
> Benchmark and evaluate CASSANDRA-8894 improvements
> --------------------------------------------------
>
> Key: CASSANDRA-10407
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10407
> Project: Cassandra
> Issue Type: Test
> Reporter: Aleksey Yeschenko
> Assignee: Alan Boudreault
> Fix For: 3.0.0 rc2
>
> Attachments: 3_0_head_mmap_wo_ra.nps, 3_0_head_std_wo_ra.nps,
> 8894_tiny.yaml, allocateDirect.png, flight-recordings.tar.gz,
> reBufferStandardTime.png, size.png, test-with-8894-tiny.json,
> test-without-8894-tiny.json
>
>
> The original ticket (CASSANDRA-8894) was committed to 3.0 alpha1 two months
> ago. We need to get proper performance tests before GA.
> See [~benedict]'s
> [comment|https://issues.apache.org/jira/browse/CASSANDRA-8894?focusedCommentId=14631203&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14631203]
> for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
