[
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186704#comment-16186704
]
Jeff Jirsa commented on CASSANDRA-13123:
----------------------------------------
Hi folks,
Pretty sure this commit breaks {{CommitLogSegmentManagerTest}} - have seen a
pretty sharp rise in failures, and reverting this commit seems to solve them.
{code}
[junit] INFO 23:34:52 Initializing CommitLogTest.Standard1
[junit] INFO 23:34:52 Initializing CommitLogTest.Standard2
[junit] ------------- ---------------- ---------------
[junit] Testcase:
testShutdownWithPendingTasks(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):
FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:105)
[junit] at
org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testShutdownWithPendingTasks(CommitLogSegmentManagerTest.java:147)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$10.evaluate(BMUnitRunner.java:371)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
[junit]
[junit]
[junit] Testcase:
testCompressedCommitLogBackpressure(org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest):
FAILED
[junit] expected:<3> but was:<1>
[junit] junit.framework.AssertionFailedError: expected:<3> but was:<1>
[junit] at org.apache.cassandra.Util.spinAssertEquals(Util.java:535)
[junit] at
org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest.testCompressedCommitLogBackpressure(CommitLogSegmentManagerTest.java:112)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$9.evaluate(BMUnitRunner.java:342)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:241)
[junit] at
org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:75)
[junit]
[junit]
[junit] Test org.apache.cassandra.db.commitlog.CommitLogSegmentManagerTest
FAILED
[delete] Deleting directory
/Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/commitlog:0
[delete] Deleting directory
/Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/data:0
[delete] Deleting directory
/Users/jjirsa/Desktop/Dev/cassandra/build/test/cassandra/saved_caches:0
[junitreport] Processing
/Users/jjirsa/Desktop/Dev/cassandra/build/test/TESTS-TestSuites.xml to
/var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913
[junitreport] Loading stylesheet
jar:file:/usr/local/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 256ms
[junitreport] Deleting:
/var/folders/nq/4w83hn7s3h13dc5wxmvcdn9w0000gn/T/null1048031913
{code}
> Draining a node might fail to delete all inactive commitlogs
> ------------------------------------------------------------
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: Jan Urbański
> Assignee: Jan Urbański
> Fix For: 3.0.15, 3.11.1, 4.0
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt,
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down
> the CommitLogSegmentManager. This has the effect of discarding any pending
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind
> after a drain and a lengthy recovery after a restart. With a fleet of dozens
> of nodes, each of them leaving several GB of commitlogs after a drain and
> taking up to two minutes to recover them on restart, the additional time
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done
> in CASSANDRA-8844.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
