[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995176#comment-13995176 ] Jonathan Ellis commented on CASSANDRA-5936: --- Is this redundant wrt CASSANDRA-6696? Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Labels: compaction Fix For: 2.1 rc1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817818#comment-13817818 ] Jonathan Ellis commented on CASSANDRA-5936: --- bq. curious why this logic is only? run for L0 compactions This is actually flush logic, not compaction. So that makes more sense now. Created CASSANDRA-6323 for this since it's not the same problem that Marcus is talking about in the description here. Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805352#comment-13805352 ] Jonathan Ellis commented on CASSANDRA-5936: --- So basically, it promotes to the first level L where L+1 has overlapping data with it, or where L+2 has more than 10*sstable size bytes overlapping with it. There's also a special case that if there are other L0 sstables overlapping it, it stays in L0. I don't think that last part fits our design (remember that LDB stops writes when L0 is full). I'm also curious why this logic is only? run for L0 compactions. But it does sound like a good improvement. Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804705#comment-13804705 ] T Jake Luciani commented on CASSANDRA-5936: --- https://code.google.com/p/leveldb/source/browse/db/version_set.cc#507 Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802295#comment-13802295 ] T Jake Luciani commented on CASSANDRA-5936: --- [~mericsson] Have you looked at all at promoting any L0 sstable to a level with no overlaps? LevelDB does this and I think it would give LCS a lot more breathing room before it starts hitting the wall. It would also promote more cross level parallel compactions. Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5936) Improve the way we pick L0 compaction candidates
[ https://issues.apache.org/jira/browse/CASSANDRA-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802328#comment-13802328 ] Jonathan Ellis commented on CASSANDRA-5936: --- Can you link the LevelDB source in question? I don't remember seeing that. Curious when it decides to do that vs merging w/ L1. Improve the way we pick L0 compaction candidates Key: CASSANDRA-5936 URL: https://issues.apache.org/jira/browse/CASSANDRA-5936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1 We could improve the way we pick compaction candidates in level 0 in LCS. The most common way for us to get behind on compaction is after repairs, we should exploit the fact that the streamed sstables are most often very narrow in range since the other nodes in the ring will have a similar sstable-range-distribution. We should in theory be able to do 10 concurrent compactions involving L1 - ie, partition L0 in buckets defined by the sstables in L1 to only keep one L1 SSTable busy for every compaction (be it L1 to L2 or L0 to L1). we will need some heuristics on when to select candidates from the buckets and when to do it the old way (since L0 sstables can span several L1 sstables) -- This message was sent by Atlassian JIRA (v6.1#6144)
