[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651488#comment-14651488 ] Albert P Tobey commented on CASSANDRA-9946: --- Here's a script for pinning compaction to cores and ionice in one go: https://gist.github.com/tobert/97c52f80fdff2ba79ee9 Comment out the 'taskset' line to mess with ionice in isolation. FWIW I think it's possible to tune up CFQ to be acceptable. There isn't a lot of existing advice on the internet about how to do it, but it's doable. I've seen some references in various Redhat low-latency guides but have yet to try it out. Even if many users choose deadline/noop for peak throughput, others may prefer the performance tradeoff of CFQ if there is a payback of more predictable/smooth performance. That's not to mention the large number of setups that never tweak the disk scheduler at all. Setting compaction IO to idle class will benefit some folks and doesn't hurt those on noop/deadline. use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649393#comment-14649393 ] Joshua McKenzie commented on CASSANDRA-9946: I think the right Windows analog would be [SetThreadPriority|https://msdn.microsoft.com/en-us/library/windows/desktop/ms686277(v=vs.85).aspx], specifically using THREAD_MODE_BACKGROUND_BEGIN. I'm assuming we can use something like [GetCurrentThread|https://msdn.microsoft.com/en-us/library/windows/desktop/ms683182(v=vs.85).aspx] via jna and use the HANDLE from there but I'd have to tinker with it to see how it behaves; for some reason I have a recollection of getting tid's being dicey w/regards to java's internal threading model. use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648676#comment-14648676 ] Ariel Weisberg commented on CASSANDRA-9946: --- The IO scheduler is a per block device thing so it could also vary across devices. If you wanted to know you could grab the setting from the filesystem (read /sys) and maybe then correlate it with the mounted filesystem that contains the directory you are writing to. CFQ is not the most popular choice for databases. The standard recommendation is deadline or no-op. Disk controllers and SSDs are doing their own scheduling so the only useful thing the IO scheduler can do is balance reads and writes and maybe respect latency for reads. I am out of my league here because I have never worked on a workload with a random read component and that is where things get hard since unlike writes you can't buffer a random read they are latency sensitive. We definitely should have a recommendation for what IO scheduler people use and then do our benchmarking based on that. use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648616#comment-14648616 ] Ariel Weisberg commented on CASSANDRA-9946: --- Setting this is harmless, but the only IO scheduler it works with is CFS unless that has changed at some point. It also might not do much because the queue for the IO scheduler is small, but I don't have good experimental data on that other then that you certainly don't get completely fair IO scheduling. use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648707#comment-14648707 ] sankalp kohli commented on CASSANDRA-9946: -- CFQ is not the most popular choice for databases. The standard recommendation is deadline or no-op. +1 We are working on moving away from CFQ to no-op. We have seen some improvements with it. use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling
[ https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648652#comment-14648652 ] Jonathan Ellis commented on CASSANDRA-9946: --- CFQ is default on debian and RHEL. Is there a syscall that can check if it's enabled first? use ioprio_set on compaction threads by default instead of manually throttling -- Key: CASSANDRA-9946 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Assignee: Ariel Weisberg Fix For: 3.x Compaction throttling works as designed, but it has two drawbacks: * it requires manual tuning to choose the right value for a given machine * it does not allow compaction to burst above its limit if there is additional i/o capacity available while there are less application requests to serve Using ioprio_set instead solves both of these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
