[
https://issues.apache.org/jira/browse/CASSANDRA-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736247#comment-17736247
]
Ariel Weisberg commented on CASSANDRA-18464:
--------------------------------------------
I am a little confused as to why memory mapped files are slow for sequential
writes from multiple threads and if the fix for this should be to switch to
direct IO?
Sure direct IO will bring all the bottlenecks for managing memory, and sending
commands to the disk, and managing concurrency to Cassandra where can address
them directly, but why can't we saturate an NVMe disk using memory mapped files?
I wrote a quick and dirty program that writes 1mb at a time to a memory mapped
buffer and syncs it every 2gb of append and that has no problem saturating the
2gb/sec disk on my MBP (apples and oranges I know). So if you remove
concurrency bottlenecks it seems like memory mapped files work on that
particular platform.
One thing I did observe running commit log stress is the throughput increases
as mutation goes up, so there is definitely some contention around each append.
It started at 1gb/sec and then settled down to 650mb/sec after about 10 seconds
which is an odd.
I'll take a look at the patch now.
> Enable Direct I/O For CommitLog Files
> -------------------------------------
>
> Key: CASSANDRA-18464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18464
> Project: Cassandra
> Issue Type: New Feature
> Components: Local/Commit Log
> Reporter: Josh McKenzie
> Assignee: Amit Pawar
> Priority: Normal
> Fix For: 5.x
>
> Attachments: EnableDirectIOForCommitLogUsingNativeAPI.patch,
> UseDirectIOFeatureForCommitLogFiles.patch
>
>
> Relocating from [dev@ email
> thread.|https://lists.apache.org/thread/j6ny17q2rhkp7jxvwxm69dd6v1dozjrg]
>
> I shared my investigation about Commitlog I/O issue on large core count
> system in my previous email dated July-22 and link to the thread is given
> below.
> [https://lists.apache.org/thread/xc5ocog2qz2v2gnj4xlw5hbthfqytx2n]
> Basically, two solutions looked possible to improve the CommitLog I/O.
> # Multi-threaded syncing
> # Using Direct-IO through JNA
> I worked on 2nd option considering the following benefit compared to the
> first one
> # Direct I/O read/write throughput is very high compared to non-Direct I/O.
> Learnt through FIO benchmarking.
> # Reduces kernel file cache uses which in-turn reduces kernel I/O activity
> for Commitlog files only.
> # Overall CPU usage reduced for flush activity. JVisualvm shows CPU usage <
> 30% for Commitlog syncer thread with Direct I/O feature
> # Direct I/O implementation is easier compared to multi-threaded
> As per the community suggestion, less in code complex is good to have. Direct
> I/O enablement looked promising but there was one issue.
> Java version 8 does not have native support to enable Direct I/O. So, JNA
> library usage is must. The same implementation should also work across other
> versions of Java (like 11 and beyond).
> I have completed Direct I/O implementation and summary of the attached patch
> changes are given below.
> # This implementation is not using Java file channels and file is opened
> through JNA to use Direct I/O feature.
> # New Segment are defined named “DirectIOSegment” for Direct I/O and
> “NonDirectIOSegment” for non-direct I/O (NonDirectIOSegment is test purpose
> only).
> # JNA write call is used to flush the changes.
> # New helper functions are defined in NativeLibrary.java and platform
> specific file. Currently tested on Linux only.
> # Patch allows user to configure optimum block size and alignment if
> default values are not OK for CommitLog disk.
> # Following configuration options are provided in Cassandra.yaml file
> a. use_jna_for_commitlog_io : to use jna feature
> b. use_direct_io_for_commitlog : to use Direct I/O feature.
> c. direct_io_minimum_block_alignment: 512 (default)
> d. nvme_disk_block_size: 32MiB (default and can be changed as per the
> required size)
> Test matrix is complex so CommitLog related testcases and TPCx-IOT benchmark
> was tested. It works with both Java 8 and 11 versions. Compressed and
> Encrypted based segments are not supported yet and it can be enabled later
> based on the Community feedback.
> Following improvement are seen with Direct I/O enablement.
> # 32 cores >= ~15%
> # 64 cores >= ~80%
> Also, another observation would like to share here. Reading Commitlog files
> with Direct I/O might help in reducing node bring-up time after the node
> crash.
> Tested with commit ID: 91f6a9aca8d3c22a03e68aa901a0b154d960ab07
> The attached patch enables Direct I/O feature for Commitlog files. Please
> check and share your feedback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
