[
https://issues.apache.org/jira/browse/CASSANDRA-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119643#comment-13119643
]
Yang Yang commented on CASSANDRA-3269:
--------------------------------------
I added a warn() to CLS.turnOffIfNotWritten() :
void turnOffIfNotWritten(Integer cfId, Integer flushPosition)
{
Integer lastWritten = cfLastWrite.get(cfId);
String keypace = Schema.instance.getCF(cfId).left;
final ColumnFamilyStore cfs =
Table.open(keypace).getColumnFamilyStore(cfId);
logger.warn("turnOffIfNotWritten:" + cfs.columnFamily + " " +
lastWritten + "/" + flushPosition);
if (lastWritten == null || lastWritten < flushPosition)
cfLastWrite.remove(cfId);
}
I saw many cases where lastWritten is > than the flushPosition, this does not
make sense.
I think this is definitely going to lead to the un-purged commit logs symptoms,
IF no more writes are added onto the dirty CFs in the oldest segment. so the
analysis I gave before still holds, but we just need to find out why
lastWritten > flushPosition.
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:19,552 CommitLogSegment.java (line
205) turnOffIfNotWritten:ad_impression_session 96030/46355
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:22,688 CommitLogSegment.java (line
205) turnOffIfNotWritten:ad_ip_agent 3070922/46355
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:24,397 CommitLogSegment.java (line
205) turnOffIfNotWritten:multi_click_filter 4996295/46522
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:25,576 CommitLogSegment.java (line
205) turnOffIfNotWritten:session_limit_filter 5242578/59906
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:27,013 CommitLogSegment.java (line
205) turnOffIfNotWritten:measuredSession 5241697/80685
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:28,740 CommitLogSegment.java (line
205) turnOffIfNotWritten:IpFilter 5242052/96877
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:28,919 CommitLogSegment.java (line
205) turnOffIfNotWritten:session_limit_filter 5242475/870755
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:29,187 CommitLogSegment.java (line
205) turnOffIfNotWritten:measuredSession 5242807/890241
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:29,471 CommitLogSegment.java (line
205) turnOffIfNotWritten:IpFilter 5242120/890241
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:43:29,690 CommitLogSegment.java (line
205) turnOffIfNotWritten:IpFilter 3422607/768691
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:45:36,674 CommitLogSegment.java (line
205) turnOffIfNotWritten:ad_impression_session 3552787/165636
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:45:39,484 CommitLogSegment.java (line
205) turnOffIfNotWritten:ad_ip_agent 5242441/165636
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:45:40,832 CommitLogSegment.java (line
205) turnOffIfNotWritten:multi_click_filter 5241748/165636
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:45:41,670 CommitLogSegment.java (line
205) turnOffIfNotWritten:ad_ip_agent 5242691/1387810
WARN [COMMIT-LOG-WRITER] 2011-10-03 19:45:41,863 CommitLogSegment.java (line
205) turnOffIfNotWritten:multi_click_filter 5242353/1409533
> accumulated 100GB of commit logs
> --------------------------------
>
> Key: CASSANDRA-3269
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3269
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Yang Yang
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 1.0.0
>
> Attachments: 3269-v2.txt, 3269-v3.txt, 3269.patch
>
>
> I ran my cluster for about 2 days. the cluster has 2 nodes. I restarted one
> box several times, and the other one was always running. the one always
> running ended up accumulating 100GB of commit logs.
> this is 1.0.0 code from about Sept 15 in github. I kept the original setting
> for
> #commitlog_total_space_in_mb: 4096
> i.e. commented out
> here is some sample of the output:
> -rw-r--r-- 1 yyang yyang 134217857 2011-09-28 03:51
> CommitLog-1317181834810.log
> -rw-r--r-- 1 yyang yyang 134217869 2011-09-28 03:50
> CommitLog-1317181764105.log
> -rw-r--r-- 1 yyang yyang 134217783 2011-09-28 03:49
> CommitLog-1317181694633.log
> -rw-r--r-- 1 yyang yyang 134217750 2011-09-28 02:39
> CommitLog-1317176955102.log
> yyang@ip-10-71-21-46:/mnt/cass/log/cassandra$ ls -lt
> /mnt/cass/lib//cassandra/commitlog/|wc -l
> 727
> yyang@ip-10-71-21-46:/mnt/cass/log/cassandra$ du -s
> /mnt/cass/lib/cassandra/commitlog/
> 95095316 /mnt/cass/lib/cassandra/commitlog/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
