[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405969#comment-15405969 ] Ariel Weisberg commented on CASSANDRA-12228: OK, I will update the comment for MCT to say that it is deprecated with an explanation. I also updated the fix version to 3.10. I created CASSANDRA-12372 for removing it from 4.0 when trunk is available for 4.0 commits. > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.10 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405953#comment-15405953 ] Sylvain Lebresne commented on CASSANDRA-12228: -- We even wrote stuff up: http://wiki.apache.org/cassandra/CompatibilityGuarantees, which while absolutely imperfect is better than nothing. In particular: bq. No removal/modifications of any configuration option, startup option, exposed metrics or general behavior of the Cassandra process > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405922#comment-15405922 ] Ariel Weisberg commented on CASSANDRA-12228: I know we have a compatibility policy for things like data artifacts and communication between nodes, but does that apply to configuration artifacts as well? > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405797#comment-15405797 ] Marcus Eriksson commented on CASSANDRA-12228: - New default looks good, and it makes sense to remove memtable_cleanup_threshold - but I'm not entirely sure we can/should do that in 3.x (def not in 3.9, maybe 3.10), should we make it deprecated it and remove in 4.0 instead? > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404434#comment-15404434 ] Ariel Weisberg commented on CASSANDRA-12228: I think the default can be really simple. If there are multiple data directories use a value of 1 so concurrency will be # of data directories and there will be a single memtable flush. If there is a single directory set it to 2 so there can be two concurrent memtable flushes to the single data directory. If the user specifies a # of concurrent memtable flushes we honor that by providing enough threads to wait on the flushes as well as enough threads per disk that each memtable can flush concurrently. I pushed an updated version reflecting this change. There is also a commit that removes memtable_cleanup_threshold. > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404078#comment-15404078 ] Marcus Eriksson commented on CASSANDRA-12228: - Since 6696 the configuration sets how many flush writers there should be *per data directory* - so factoring in the number of data directories in the calculation is probably wrong (ie, with 10 data directories and 2 memtable flush writers you would get 20 threads actually doing writing to disk and 2 threads waiting on the writing, meaning we can flush 2 memtables concurrently) > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404054#comment-15404054 ] T Jake Luciani commented on CASSANDRA-12228: I'm good with that, I'll wait to see what [~krummas] thinks > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403197#comment-15403197 ] Ariel Weisberg commented on CASSANDRA-12228: |[3.9|https://github.com/apache/cassandra/compare/cassandra-3.9...aweisberg:CASSANDRA-12228-3.9?expand=1]|[utest|https://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-12228-3.9-testall/1/]|[utest|https://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-12228-3.9-dtest/1/]| |[trunk|https://github.com/apache/cassandra/compare/trunk...aweisberg:CASSANDRA-12228-3.9?expand=1]|[utest|https://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-12228-trunk-testall/1/]|[utest|https://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-12228-trunk-dtest/1/]| > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402770#comment-15402770 ] Ariel Weisberg commented on CASSANDRA-12228: I was incorrect. JMXEnabledThreadPoolExecutor automatically sets core pool threads to max if you only specify core pool threads. So I am running into some other pain point around memory accounting unrelated to flushing performance. > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Marcus Eriksson >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400361#comment-15400361 ] Ariel Weisberg commented on CASSANDRA-12228: I think there is also an issue I am stilling working on nailing down where memory accounting releases memory pinned by memtables too early or is just off by too much causing the heap to fill up with memtables that are waiting for the post flush executor. I can see the heap going to double the limit in a heap dump and things are falling apart server side. > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Marcus Eriksson >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12228) Write performance regression in 3.x vs 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400133#comment-15400133 ] Ariel Weisberg commented on CASSANDRA-12228: There are some remaining issues with thread pool sizes. See [CASSANDRA-12071|https://issues.apache.org/jira/browse/CASSANDRA-12071?focusedCommentId=15400086=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15400086]. You still can't get multiple threads if you have a single disk. Seems like this would be a good place to address the related issue. I also don't think this is minor it's pretty crippling for performance and you can't work around it by changing configuration values. > Write performance regression in 3.x vs 3.0 > -- > > Key: CASSANDRA-12228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12228 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Marcus Eriksson >Priority: Minor > Fix For: 3.9 > > > I've been tracking down a performance issue in trunk vs cassandra-3.0 branch. > I think I've found it. CASSANDRA-6696 changed the default memtable flush > default to 1 vs the min of 2 in cassandra-3.0. > I don't see any technical reason for this and we should add back the min of 2 > sstable flushers per disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
