[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981084#comment-13981084
]
Bill Mitchell commented on CASSANDRA-6841:
--
Although not the author, I ran into this symptom of the
ConcurrentModificationException at line 309 in the CommitLogAllocator more than
once this week on 2.0.6, on an 8-core Windows desktop. After downloading and
installing 2.0.7 yesterday, I have not seen this problem reappear.
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 1.2.17, 2.0.7, 2.1 beta2
Attachments: 6841.12.txt
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935382#comment-13935382
]
Jonathan Ellis commented on CASSANDRA-6841:
---
committed. might want to doublecheck my merge to 2.1; i kept the changes to
Schema but the CL changes looked irrelevant
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7, 2.1 beta2
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
serialized/live bytes, 3276 ops)
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935813#comment-13935813
]
Benedict commented on CASSANDRA-6841:
-
LGTM as far as I can tell, but a bit tricky as git diff includes some of the
2.0 patches in the 2.1 git log.
The CL changes weren't necessary for 2.1 anyway, you're right, and it looks
like they're missing from 2.1 so I'm happy.
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7, 2.1 beta2
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933183#comment-13933183
]
Benedict commented on CASSANDRA-6841:
-
Updated the tree. Also converted one of those ISE to an IAE check prior to
modifying the map
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
serialized/live bytes, 3276 ops)
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933193#comment-13933193
]
Carl Yeksigian commented on CASSANDRA-6841:
---
+1
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
serialized/live bytes, 3276 ops)
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of Memtable-schema_columns@252242773(185191/630698
serialized/live bytes, 3614 ops)
ERROR
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13931897#comment-13931897
]
Benedict commented on CASSANDRA-6841:
-
Without investigating _thoroughly_ to rule out anything else, I would guess the
problem is with Keyspace.cfIdMap, which is not threadsafe. We should probably
create it with two ConcurrentHashMap, instead of the default (plain HashMap)
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Priority: Minor
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
serialized/live bytes, 3276 ops)
INFO [OptionalTasks:1] 2014-03-12
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932005#comment-13932005
]
Pas commented on CASSANDRA-6841:
To give more context. There was a single node ring used for running test of an
application that uses, among other stores, Cassandra as a backend. It has a lot
of small keyspaces (each with 1 or 2 CFs - supercolumn based), total data on
the node was/is less than 300G. We are in the process of migrating away from
supercolumns, so we're moving to CQL and composite primary keys and whatnot, so
there are a few keyspaces already with this new schema.
Then we tried to add a new node, setting its seed property to the old one. It
failed to bootstrap with errors (
https://issues.apache.org/jira/browse/CASSANDRA-6565?focusedCommentId=13931732page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13931732
), but finally after a few tries it started and went from BOOT to NORMAL. Then
we noticed that the new node has a different schema UUID (as indicated by
gossipinfo), and tried to do resetlocalschema. (Which happened, according to
system.log, but then we have observed no improvement, so we tried it again then
the gossip stage appeared to have been deadlocked; but a restart solved it. See
also
https://issues.apache.org/jira/browse/CASSANDRA-6799?focusedCommentId=13931690page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13931690
)
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1
[jira] [Commented] (CASSANDRA-6841) ConcurrentModificationException in commit-log-writer after local schema reset
[
https://issues.apache.org/jira/browse/CASSANDRA-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932336#comment-13932336
]
Carl Yeksigian commented on CASSANDRA-6841:
---
The patch LGTM, but it might help to have the two IllegalStateExceptions in the
ConcurrentBiMap.put method commented.
ConcurrentModificationException in commit-log-writer after local schema reset
-
Key: CASSANDRA-6841
URL: https://issues.apache.org/jira/browse/CASSANDRA-6841
Project: Cassandra
Issue Type: Bug
Environment: Linux 3.2.0 (Debian Wheezy) Cassandra 2.0.6, Oracle JVM
1.7.0_51
Almost default cassandra.yaml (IPs and cluster name changed)
This is the 2nd node in a 2-node ring. It has ~2500 keyspaces and very low
traffic. (Only new keyspaces see reads and writes.)
Reporter: Pas
Assignee: Benedict
Priority: Minor
Fix For: 2.0.7
{code}
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,013
MigrationManager.java (line 329) Starting local schema reset...
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,016
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,016 Memtable.java (line 331)
Writing Memtable-local@394448776(114/1140 serialized/live bytes, 3 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,182 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db (145 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159822)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,185
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,185 Memtable.java (line 331)
Writing Memtable-local@1087210140(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,357 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33159959)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,361
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,361 Memtable.java (line 331)
Writing Memtable-local@768887091(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,516 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160096)
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,517 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-398-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-400-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-399-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-397-Data.db')]
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,519
ColumnFamilyStore.java (line 785) Enqueuing flush of
Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,519 Memtable.java (line 331)
Writing Memtable-local@271993477(62/620 serialized/live bytes, 1 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:54,794 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-jb-401-Data.db (96 bytes)
for commitlog position ReplayPosition(segmentId=1394620057452,
position=33160233)
INFO [RMI TCP Connection(38)-192.168.36.171] 2014-03-12 11:37:54,799
MigrationManager.java (line 357) Local schema reset is complete.
INFO [CompactionExecutor:38] 2014-03-12 11:37:54,848 CompactionTask.java
(line 275) Compacted 4 sstables to
[/var/lib/cassandra/data/system/local/system-local-jb-402,]. 6,099 bytes to
5,821 (~95% of original) in 330ms = 0.016822MB/s. 4 total partitions merged
to 1. Partition merge counts were {4:1, }
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
785) Enqueuing flush of
Memtable-schema_columnfamilies@106276050(181506/509164 serialized/live bytes,
3276 ops)
INFO [FlushWriter:6] 2014-03-12 11:37:55,110 Memtable.java (line 331)
Writing Memtable-schema_columnfamilies@106276050(181506/509164
serialized/live bytes, 3276 ops)
INFO [OptionalTasks:1] 2014-03-12 11:37:55,110 ColumnFamilyStore.java (line
