[
https://issues.apache.org/jira/browse/CASSANDRA-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610890#comment-14610890
]
Joshua McKenzie commented on CASSANDRA-9658:
--------------------------------------------
Thanks for the quick turnaround!
bq. The log statement is repeated 3 times in code you can just put it at the
end of the if/else conditions
I think that was originally in there to differentiate "auto" from other modes
in the log which requires either logging the way we do now or building a log
statement after depending on conf.disk_access_mode. Both seem equally
distasteful to me so I figured leave well enough alone.
bq. Use the FBUtiliies.getProtectedField method vs doing it yourself?
Didn't know we had that - updated.
bq. Should you log every file at info? should it be debug?
Yes. Mistake / left over from testing. Fixed.
bq. Bad import org.hsqldb.Database;
IDE got over-eager and I missed it on final pass. Removed.
bq. Is there anything we need todo if the delete fails? like kill -9? or stack
overflow on your recursive delete? Should we have a .deleteme file added that
can be checked on start? Or at least a log message if a Major exception is
thrown that users should expect to clean up themselves
I thought about that in passing but originally landed on the side of "if a node
hard-dies and an operator sees there's large disk usage on there, they should
turn up snapshots pretty quick and can correlate it with the log message". In
retrospect the log message should be more clear about it being "best effort" on
JVM shutdown w/certain caveats (crashing, stack overflow, OOM) if we go that
route.
Given the work taking place in CASSANDRA-7066, I think I'll peruse Stefania's
work on there and follow her lead w/an implementation to delete old snapshots
on startup. It's really the most robust solution and it may be a long road for
us to get buffered and memory-mapped into parity. Pulling this out of patch
available for now while I get that implemented.
CI currently running:
[testall|https://cassci.datastax.com/view/Dev/view/josh-mckenzie/job/josh-mckenzie-9658-testall/1/]
[dtest|https://cassci.datastax.com/view/Dev/view/josh-mckenzie/job/josh-mckenzie-9658-dtest/1/]
> Re-enable memory-mapped index file reads on Windows
> ---------------------------------------------------
>
> Key: CASSANDRA-9658
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9658
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Joshua McKenzie
> Assignee: Joshua McKenzie
> Labels: Windows, performance
> Fix For: 2.2.x
>
>
> It appears that the impact of buffered vs. memory-mapped index file reads has
> changed dramatically since last I tested. [Here's some results on various
> platforms we pulled together yesterday
> w/2.2-HEAD|https://docs.google.com/spreadsheets/d/1JaO2x7NsK4SSg_ZBqlfH0AwspGgIgFZ9wZ12fC4VZb0/edit#gid=0].
> TL;DR: On linux we see a 40% hit in performance from 108k ops/sec on reads to
> 64.8k ops/sec. While surprising in itself, the really unexpected result (to
> me) is on Windows - with standard access we're getting 16.8k ops/second on
> our bare-metal perf boxes vs. 184.7k ops/sec with memory-mapped index files,
> an over 10-fold increase in throughput. While testing w/standard access,
> CPU's on the stress machine and C* node are both sitting < 4%, network
> doesn't appear bottlenecked, resource monitor doesn't show anything
> interesting, and performance counters in the kernel show very little. Changes
> in thread count simply serve to increase median latency w/out impacting any
> other visible metric that we're measuring, so I'm at a loss as to why the
> disparity is so huge on the platform.
> The combination of my changes to get the 2.1 branch to behave on Windows
> along with [~benedict] and [~Stefania]'s changes in lifecycle and cleanup
> patterns on 2.2 should hopefully have us in a state where transitioning back
> to using memory-mapped I/O on Windows will only cause trouble on snapshot
> deletion. Fairly simple runs of stress w/compaction aren't popping up any
> obvious errors on file access or renaming - I'm going to do some much heavier
> testing (ccm multi-node clusters, long stress w/repair and compaction, etc)
> and see if there's any outstanding issues that need to be stamped out to call
> mmap'ed index files on Windows safe. The one thing we'll never be able to
> support is deletion of snapshots while a node is running and sstables are
> mapped, but for a > 10x throughput increase I think users would be willing to
> make that sacrifice.
> The combination of the powercfg profile change, the kernel timer resolution,
> and memory-mapped index files are giving some pretty interesting performance
> numbers on EC2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
