[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082097#comment-14082097
]
Marcus Eriksson commented on CASSANDRA-7593:
[~rhatch] I don't think that is related to this (NoSuchMethodError is likely an
environment issue)
+1 on the patch
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593-v2.txt, 7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082263#comment-14082263
]
Jonathan Ellis commented on CASSANDRA-7593:
---
[~rhatch] let's open a separate ticket for that.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593-v2.txt, 7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080817#comment-14080817
]
Marcus Eriksson commented on CASSANDRA-7593:
Still not working, we get clusteringSize() = 1 for rt.max and clusteringSize()
= 0 for min
when deserializing the RT we get to CSCNT.WithCollection.makeWith():
{code}
protected Composite makeWith(ByteBuffer[] components, int size,
Composite.EOC eoc, boolean isStatic)
{
if (size clusteringSize + 1 || eoc != Composite.EOC.NONE)
return new CompoundComposite(components, size,
isStatic).withEOC(eoc);
return new CompoundSparseCellName(components, clusteringSize,
idFor(components[clusteringSize]), isStatic);
}
{code}
so, if we have an EOC we wrap a CompoundComposite with size (which is 1 in this
case), and if we don't, we create a CSCN with clusteringSize (which is 0)
Attaching a patch that keeps clusteringSize in the compound composite, but have
to say that I don't know enough about this to say if it breaks everything else.
a way to repro:
# start cassandra2.0
# CREATE TABLE t (k int PRIMARY KEY,something text, a setvarchar, b
setvarchar);
# update t set b = null where k = 1;
# flush
# upgrade to 2.1
# upgradesstables
# check minmax column names (i hacked SSTableMetadataViewer to dump the
contents)
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080825#comment-14080825
]
Benedict commented on CASSANDRA-7593:
-
Might be better to just expose this in CSCNT, since we have access to it when
doing this.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080971#comment-14080971
]
Aleksey Yeschenko commented on CASSANDRA-7593:
--
bq. We don't allow deletions of that form right now, but I agree that it would
be nice to future-proof a bit here if we can do so safely.
Except I think we do, via Thrift (because stupid CASSANDRA-5435).
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081121#comment-14081121
]
Tyler Hobbs commented on CASSANDRA-7593:
Let's take a step back.
Moving {{clusteringSize()}} up to Composite creates problems for
CompoundComposite, as Marcus demonstrated, because the CompoundComposite isn't
capable of knowing its own clustering size. We would need to patch all of the
places where we create CompoundComposites (which Marcus started) in order to
avoid a lot of subtle bugs in the future.
Instead of moving {{clusteringSize()}} up to the Composite class, can't we
utilize the {{comparator}} in {{ColumnNameHelper.min/maxComponents()}}? For
example, what we have now is:
{code}
public static ListByteBuffer minComponents(ListByteBuffer minSeen,
Composite candidate, CellNameType comparator)
{
// For a cell name, no reason to look more than the clustering prefix
// (and comparing the collection element would actually crash)
int size = candidate instanceof CellName ?
((CellName)candidate).clusteringSize() : candidate.size();
if (minSeen.isEmpty())
return getComponents(candidate, size);
{code}
Instead, we can do:
{code}
int size = min(comparator.clusteringPrefixSize(), candidate.size());
if (minSeen.isEmpty())
return getComponents(candidate, size);
{code}
We would definitely need to do CASSANDRA-7651 along with this (especially due
to CASSANDRA-5435), but it seems like the least risky, most correct fix to me.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081135#comment-14081135
]
Benedict commented on CASSANDRA-7593:
-
Yes, this is what I meant when I said expose it in CSCNT; didn't spot it was
already exposed
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081818#comment-14081818
]
Russ Hatch commented on CASSANDRA-7593:
---
I'm still running tests, but this exception is cropping up:
{noformat}
INFO [GossipStage:1] 2014-07-31 18:29:57,613 StorageService.java (line 1511)
Node /127.0.0.1 state jump to normal
INFO [HANDSHAKE-/127.0.0.1] 2014-07-31 18:29:58,163 OutboundTcpConnection.java
(line 386) Handshaking version with /127.0.0.1
ERROR [Thrift:4] 2014-07-31 18:29:58,175 CassandraDaemon.java (line 199)
Exception in thread Thread[Thrift:4,5,main]
java.lang.NoSuchMethodError:
org.apache.cassandra.concurrent.TracingAwareExecutorService.maybeExecuteImmediately(Ljava/lang/Runnable;)V
at
org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:96)
at
org.apache.cassandra.service.AbstractReadExecutor$NeverSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:215)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1216)
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1157)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:253)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:223)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:61)
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:175)
at
org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
Attachments: 0001-keep-clusteringSize-in-CompoundComposite.patch,
7593-v2.txt, 7593.txt
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077643#comment-14077643
]
Marcus Eriksson commented on CASSANDRA-7593:
this issue affects more than the min/max col name collection:
{code}
oss/cassandra [cassandra-2.0●] » bin/cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.9-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.39.0]
Use HELP for help.
cqlsh CREATE KEYSPACE aa WITH replication = {'class': 'SimpleStrategy',
'replication_factor':1};
cqlsh use aa;
cqlsh:aa CREATE TABLE t (k int PRIMARY KEY,something text, a setvarchar, b
setvarchar);
cqlsh:aa INSERT INTO t (k, something, a, b) VALUES (1, 'bluh', {'a', 'aa'},
{'b', 'bb'});
cqlsh:aa INSERT INTO t (k, something, a, b) VALUES (2, 'bluh', {'a', 'aa'},
{'b', 'bb'});
oss/cassandra [cassandra-2.0●] » bin/nodetool flush
oss/cassandra [cassandra-2.0●] » bin/cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.9-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.39.0]
Use HELP for help.
cqlsh USE aa;
cqlsh:aa UPDATE t SET a=NULL WHERE k=1;
oss/cassandra [cassandra-2.0●] » bin/nodetool flush
oss/cassandra [cassandra-2.0●] » bin/cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.9-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.39.0]
Use HELP for help.
cqlsh USE aa;
cqlsh:aa SELECT * FROM t;
k | a | b | something
---+-+-+---
1 |null | {'b', 'bb'} | bluh
2 | {'a', 'aa'} | {'b', 'bb'} | bluh
(2 rows)
{code}
and upgrade the node to 2.1:
{code}
oss/cassandra [cassandra-2.1●] » bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc4-SNAPSHOT | CQL spec 3.2.0 | Native protocol
v3]
Use HELP for help.
cqlsh USE aa;
cqlsh:aa SELECT * FROM t;
k | a | b | something
---+-+-+---
1 |null | {'b', 'bb'} | bluh
2 | {'a', 'aa'} | {'b', 'bb'} | bluh
(2 rows)
cqlsh:aa
oss/cassandra [cassandra-2.1●] » bin/nodetool upgradesstables aa t
oss/cassandra [cassandra-2.1●] » bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc4-SNAPSHOT | CQL spec 3.2.0 | Native protocol
v3]
Use HELP for help.
cqlsh USE aa;
cqlsh:aa SELECT * FROM t;
k | a | b | something
---+-+-+---
1 |null | {'b', 'bb'} | bluh
2 | {'a', 'aa'} | {'b', 'bb'} | bluh
(2 rows)
cqlsh:aa
oss/cassandra [cassandra-2.1●] » bin/nodetool compact aa t
oss/cassandra [cassandra-2.1●] » bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc4-SNAPSHOT | CQL spec 3.2.0 | Native protocol
v3]
Use HELP for help.
cqlsh USE aa;
cqlsh:aa SELECT * FROM t;
k | a | b | something
---+-+-+---
1 | {'a', 'aa'} | {'b', 'bb'} | bluh
2 | {'a', 'aa'} | {'b', 'bb'} | bluh
(2 rows)
{code}
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077981#comment-14077981
]
Sylvain Lebresne commented on CASSANDRA-7593:
-
bq. In 2.0 we only make the end have an EOC (...) while in 2.1 both do
That's correct but that shouldn't be a problem. That is, whether or not the RT
min has an EOC or not shouldn't really matter (as far as anything CQL is
concerned at least). Why does the storage engine code breaks when the min RT
has no EOC in 2.1 is the right question.
bq. This shows itself when we deserialize the 2.0-RT in 2.1 since we get a
CompoundSparseCellName for RT.min and a BoundedComposite for RT.max and that
breaks the collection of min/max column names since the CSCN is empty and no
columns will be collected.
Not sure I follow. What parts breaks exactly? Wy having a CSCN instead of a
BoundComposite would matter for the collection of min/max column names?
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077989#comment-14077989
]
Tyler Hobbs commented on CASSANDRA-7593:
Assuming EOC.START on RT mins does seem like the correct fix, but I would like
to understand why having a normal CompoundSparseCellName (instead of a
BoundedComposite) for the RT min doesn't work.
bq. that breaks the collection of min/max column names since the CSCN is
empty and no columns will be collected.
Can you point out exactly where this happens?
You also mentioned:
bq. the clusteringSize() is 0 for min which means we don't collect it. They
both contain 'tokens' which is correct, problem is that when we deserialize the
RT from the sstable it seems we think the column is one of the 'ordinary'
columns in the table, and, they should have clusteringSize=0
Can you also clarify where/how it thinks this is an ordinary column?
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078016#comment-14078016
]
Sylvain Lebresne commented on CASSANDRA-7593:
-
bq. we get a CompoundSparseCellName for RT.min and a BoundedComposite for
RT.max and that breaks the collection of min/max column names
I see now, that's because ColumnNameHelper.minComponents special case CellNames
to only use the clusteringSize but for BoundedComposite, which are not
CellName, it will use the full size.
bq. in the grabbing of min/max column names, we only care about clustering
columns, so with or without the extra EOC we should not be fetching the whole
of this BoundedComposite into min/max - we should only be fetching up to the
number of clustering columns
I agree. And a simple way to do this would be to promote the
{{clusteringSize()}} method to the class Composite having it return the same
than {{size()}} for non-cellname composites (but BoundedComposite would forward
to it's wrapped composite properly). We'd then use that in
ColumnNameHelper.\{min/max\}Components instead of the {{instanceof CellName}}.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078043#comment-14078043
]
Sylvain Lebresne commented on CASSANDRA-7593:
-
bq. Assuming EOC.START on RT mins does seem like the correct fix
I definitively disagree btw. I see no reason to do that (outside of the fact
that it happens to fix this very particular issue). No assumptions are made
that a RT should have an EOC for any of its bounds and creating one just feels
wrong.
bq. This fix doesn't address the situation where we legitimately get an sstable
with only range tombstones where the lowerbound has fewer components to the
upper bounds (let's say we have a flood of DELETE from T where pk=K and a X
and (a,b) (Y, Z))
Yes, and imo the assumption made by min/max column collection that there is the
same number of component in minColumns and maxColumns in the metadata (the one
that breaks here) is wrong in the first place and we should remove it (make the
code work without that assumption). That said, since we don't currently support
the syntax above, for 2.1.0, we can probably stick to make the fix suggested
above (i.e. make sure we always use the clustering size only in
{{ColumnNameHelper.minComponents}}) as that's pretty simple. Removing the
assumption that both bound of a RT cannot have the same size will have to be
done before CASSANDRA-6237 however.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078050#comment-14078050
]
Marcus Eriksson commented on CASSANDRA-7593:
hmm, so the reappearing data issue i posted above might be unrelated, i'll dig
more into that tomorrow
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076068#comment-14076068
]
Marcus Eriksson commented on CASSANDRA-7593:
no, they should not be empty
We are inserting a RangeTombstone with start='token' and end='token' (ie,
delete the set for this row).
In 2.0 we only make the end have an EOC
(https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/cql3/Sets.java#L234)
while in 2.1 both do:
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/cql3/Sets.java#L252
+
https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/composites/AbstractComposite.java#L69
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076411#comment-14076411
]
Jonathan Ellis commented on CASSANDRA-7593:
---
bq. Could we assume EOC.START for RT.min in 2.1 when deserializing old sstables?
That sounds like the right fix to me.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076472#comment-14076472
]
Benedict commented on CASSANDRA-7593:
-
This fix doesn't address the situation where we legitimately get an sstable
with only range tombstones where the lowerbound has fewer components to the
upper bounds (let's say we have a flood of DELETE from T where pk=K and a X
and (a,b) (Y, Z))
Also, whilst we _may_ separately want to insert an EOC.START (since we now
populate it, so presumably it should be populated, though it would be good to
understand why this is now the case to document here for posterity), according
to the commenting in the grabbing of min/max column names, we only care about
clustering columns, so with or without the extra EOC we should not be fetching
the whole of this BoundedComposite into min/max - we should only be fetching up
to the number of clustering columns (0). Or we should be fetching the column
name (and potentially any further components for sets/maps/etc.) from CellName
as well.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074163#comment-14074163
]
Marcus Eriksson commented on CASSANDRA-7593:
Think I'm getting to the bottom of this now
Seems it is a matter of how we store/deserialize RangeTombstones in 2.0 vs 2.1;
If I, in 2.0 store a RT like this, by itself in an sstable;
{code}
update p set tokens=null where peer = '127.0.0.4' ;
{code}
schema is:
{code}
CREATE TABLE aa.p (
peer inet PRIMARY KEY,
data_center text,
host_id uuid,
preferred_ip inet,
rack text,
release_version text,
rpc_address inet,
schema_version uuid,
tokens settext
)
{code}
And then deserialize that, I get no EOC for RT.min, but i get EOC.end in RT.max
If i store the same tombstone in 2.1, i get EOC.start for RT.min and EOC.end
for RT.max
This shows itself when we deserialize the 2.0-RT in 2.1 since we get a
CompoundSparseCellName for RT.min and a BoundedComposite for RT.max and that
breaks the collection of min/max column names since the CSCN is empty and no
columns will be collected.
Could we assume EOC.START for RT.min in 2.1 when deserializing old sstables?
There is also a small bug when upgrading from 1.2 to 2.0, but it seems safe, in
1.2 we store a row marker with the RT, in 2.0 we don't, so the row marker looks
like a standard row which is included in the min/max collection (as a min
column name). But this is afaict not dangerous since we will only include more
sstable than we needed to
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074181#comment-14074181
]
Benedict commented on CASSANDRA-7593:
-
There's still something fishy going on, though, since the BoundedComposite
should also be empty (it should have a size of 0, just with an EOC component).
That doesn't change the fact that this bug could present itself in situations
where it legitimately does have a different clustering size to size, so needs
its own fix. I would say the correct answer may be to have a BoundedCellName,
which implements CellName, so that the collection check knows to not treat it
as a general composite but to grab its clustering size.
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071415#comment-14071415
]
Marcus Eriksson commented on CASSANDRA-7593:
Hmm, could you post datamodel and queries? And (full) logs from the node.
The code around the first AE has not changed in 2.1, so I guess the problem is
when we collect the max/min col names, either during compaction or flushing
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619)
~[na:1.7.0_60]
at
org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071497#comment-14071497
]
Benedict commented on CASSANDRA-7593:
-
There's a race condition on reloading statsMetaData, but I doubt it would
change the value here (though we should potentially check it doesn't cause any
other issues).
It's *possible* that a table only contains range tombstones, and that the
tombstones are not evenly defined (i.e. the min of the range has a different
number of components to the max). This is also certainly a bug, but I'm not
convinced it's the cause. We could do with expanding the text of the assertion
to give us some extra debug info
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
[jira] [Commented] (CASSANDRA-7593) Errors when upgrading through several versions to 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071977#comment-14071977
]
Marcus Eriksson commented on CASSANDRA-7593:
This happens for system.peers when upgrading sstables to the 2.1 version
Reason is that in 2.0 the range tombstone was a plain ByteBuffer, which, if
empty, was kept as an empty BB. In 2.1 this empty BB is converted to a
composite that has size=0 that we don't keep when compacting.
We probably need to change the way we handle RangeTombstones when collecting
min/max col names in 2.1
Errors when upgrading through several versions to 2.1
-
Key: CASSANDRA-7593
URL: https://issues.apache.org/jira/browse/CASSANDRA-7593
Project: Cassandra
Issue Type: Bug
Environment: java 1.7
Reporter: Russ Hatch
Assignee: Marcus Eriksson
Priority: Critical
Fix For: 2.1.0
I'm seeing two different errors cropping up in the dtest which upgrades a
cluster through several versions.
This is the more common error:
{noformat}
ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 -
Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347)
~[main/:na]
at
org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
~[main/:na]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[main/:na]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[main/:na]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302)
~[main/:na]
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514)
~[main/:na]
at
org.apache.cassandra.net.OutboundTcpConnectionPool.init(OutboundTcpConnectionPool.java:51)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689)
~[main/:na]
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663)
~[main/:na]
at
org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[main/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
{noformat}
The same test sometimes fails with this exception instead:
{noformat}
ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168
- Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530)
~[na:1.7.0_60]
