[
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-2766:
--------------------------------------
Attachment: 2766.txt
looks like a straightforward case of "not using threadsafe collections means
there is no happens-before guarantee where a naive reading of the code would
expect one" (i.e., since buildFutures.add is always called before files.remove,
after files.isEmpty is true there should be no more changes to
buildFutures.add).
patch attached that changes both of these from Arraylist to CSLS.
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was
> actually started on this node (but it had gotten data through the regular
> data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but
> the node have finished one incoming stream before this (I guess from some
> other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java
> (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java
> (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java
> (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java
> (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line
> 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line
> 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line
> 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line
> 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line
> 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
