[jira] [Commented] (CASSANDRA-8611) give streaming_socket_timeout_in_ms a non-zero default
[ https://issues.apache.org/jira/browse/CASSANDRA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709372#comment-14709372 ] Eric Lubow commented on CASSANDRA-8611: --- I've seen streams hang for days on EC2 as well. This can be especially problematic when you are trying to add capacity. Typically if nothing has happened in an hour, then it's probably the result of a hung stream and waiting another hour doesn't serve to benefit much. The one thing to keep in mind for a timeout of two hours is that on smaller datasets, the timeout for the stream is going to be longer than the entire bootstrap of the machine would take. I think it would be safe to bring thing down to an hour which is also still very conservative. give streaming_socket_timeout_in_ms a non-zero default -- Key: CASSANDRA-8611 URL: https://issues.apache.org/jira/browse/CASSANDRA-8611 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jeremy Hanna Assignee: Benjamin Lerer Sometimes as mentioned in CASSANDRA-8472 streams will hang. We have streaming_socket_timeout_in_ms which can retry after a timeout. It would be good to make a default non-zero value. We don't want to paper over problems, but streams sometimes hang and you don't want long running streaming operations to just fail - as in repairs or bootstraps. streaming_socket_timeout_in_ms should be based on the tcp idle timeout so it shouldn't be a problem to set it to on the order of minutes. Also the socket should only be open during the actual streaming and not during operations such as merkle tree generation. We can set it to a conservative value and people can set it more aggressively as needed. Disabling as a default, in my opinion, is too conservative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6604) AssertionError: Verb COUNTER_MUTATION should not legally be dropped
[
https://issues.apache.org/jira/browse/CASSANDRA-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893996#comment-13893996
]
Eric Lubow commented on CASSANDRA-6604:
---
I'm not entirely sure this is cosmetic. We have a 20 node cluster running
1.2.13.2 (DSE 3.2.4) and it is not under a heavy load at all. We are seeing
this simultaneously with tens of thousands of dropped mutations under higher
traffic scenarios and just thousands of mutations dropped under lower traffic
scenarios. However, it is only happening on a single node. There is nothing
(visibly) wrong with that node other than higher CPU utilization that the other
nodes. Even with the higher CPU utilization, it is still not a stressed node
and yet we are seeing the dropped MUTATIONS and this error message (again only
on this node).
AssertionError: Verb COUNTER_MUTATION should not legally be dropped
---
Key: CASSANDRA-6604
URL: https://issues.apache.org/jira/browse/CASSANDRA-6604
Project: Cassandra
Issue Type: Bug
Reporter: Bartłomiej Romański
Fix For: 1.2.14, 2.0.5
We're seeing the following errors in our logs from time to time (about once
an hour):
{code}
ERROR [MutationStage:10] 2014-01-19 10:36:26,659 CassandraDaemon.java (line
187) Exception in thread Thread[MutationStage:10,5,main]
java.lang.AssertionError: Verb COUNTER_MUTATION should not legally be dropped
at
org.apache.cassandra.net.MessagingService.incrementDroppedMessages(MessagingService.java:779)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1925)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{code}
They always appear in groups. About 50-100 errors in a row.
We've got a 12-nodes cluster recently upgraded from 1.2.10 to 2.0.4. It's
under pretty heavy load.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (CASSANDRA-6542) nodetool removenode hangs
Eric Lubow created CASSANDRA-6542: - Summary: nodetool removenode hangs Key: CASSANDRA-6542 URL: https://issues.apache.org/jira/browse/CASSANDRA-6542 Project: Cassandra Issue Type: Bug Components: Core Environment: Ubuntu 12, 1.2.11 DSE Reporter: Eric Lubow Fix For: 1.2.11 Running *nodetool removenode $host-id* doesn't actually remove the node from the ring. I've let it run anywhere from 5 minutes to 3 days and there are no messages in the log about it hanging or failing, the command just sits there running. So the regular response has been to run *nodetool removenode $host-id*, give it about 10-15 minutes and then run *nodetool removenode force*. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4206) AssertionError: originally calculated column size of 629444349 but now it is 588008950
[
https://issues.apache.org/jira/browse/CASSANDRA-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859902#comment-13859902
]
Eric Lubow commented on CASSANDRA-4206:
---
We are seeing this as well with 1.2.11. As was mentioned above, knowing which
CF would very useful here. It seems to be happening to hints. The only other
major action we are seeing that is out of the ordinary is thousands of hint
SSTables being transferred at a time. Here is the Java error:
{quote}
ERROR [HintedHandoff:6] 2014-01-01 16:45:19,914 CassandraDaemon.java (line 191)
Exception in thread Thread[HintedHandoff:6,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of 1028119265 but
now it is 1028119453
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:436)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:502)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError:
originally calculated column size of 1028119265 but now it is 1028119453
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:432)
... 6 more
Caused by: java.lang.AssertionError: originally calculated column size of
1028119265 but now it is 1028119453
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
{quote}
Multithreaded compactions are set to false in our cluster on all nodes. We
also don't have any pending compactions in the cluster. Just seeing this error
a lot in the logs. The error seems to happen more frequently during bootstraps
or repairs that have a lot of work to do.
AssertionError: originally calculated column size of 629444349 but now it is
588008950
--
Key: CASSANDRA-4206
URL: https://issues.apache.org/jira/browse/CASSANDRA-4206
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.9
Environment: Debian Squeeze Linux, kernel 2.6.32, sun-java6-bin
6.26-0squeeze1
Reporter: Patrik Modesto
I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3
and one CF with two secondary indexes. I'm importing data into this CF using
Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX:
MaxRowSize: 1597
MeanRowSize: 369
And there are some tens of millions of rows.
It's write-heavy usage and there is a big pressure on each node, there are
quite some dropped mutations on each node. After ~12 hours of inserting I see
these assertion exceptiona on 3 out of four nodes:
{noformat}
ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: originally calculated column size of 629444349 but
now it is 588008950
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388)
at
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[
https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859661#comment-13859661
]
Eric Lubow commented on CASSANDRA-6494:
---
I am seeing this on a bootstrapping node on 1.2.11:
{quote}
ERROR 19:53:52,029 Exception in thread Thread[CompactionExecutor:93,1,main]
java.lang.RuntimeException: 01433f155800 is not defined as a collection
at
org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:128)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:219)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:98)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{quote}
Cassandra refuses to restart due to a corrupted commit log.
---
Key: CASSANDRA-6494
URL: https://issues.apache.org/jira/browse/CASSANDRA-6494
Project: Cassandra
Issue Type: Bug
Reporter: Shao-Chuan Wang
This is running on our production server. Please advise how to address this
issue. Thank you!
INFO 02:46:58,879 Finished reading
/mnt/cassandra/commitlog/CommitLog-3-1386069222785.log
ERROR 02:46:58,879 Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: 706167655f74616773 is not defined as a collection
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by:
[jira] [Comment Edited] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859672#comment-13859672 ] Eric Lubow edited comment on CASSANDRA-6494 at 12/31/13 8:26 PM: - We dropped a table named 'uniques' and recreated the same table named 'uniques' with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. was (Author: elubow): We dropped a table named 'uniques' and recreated the same table with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. Cassandra refuses to restart due to a corrupted commit log. --- Key: CASSANDRA-6494 URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 Project: Cassandra Issue Type: Bug Reporter: Shao-Chuan Wang This is running on our production server. Please advise how to address this issue. Thank you! INFO 02:46:58,879 Finished reading /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log ERROR 02:46:58,879 Exception encountered during startup java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ... 8 more Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) at edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) at edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) at edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) at edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) at edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) at org.apache.cassandra.db.Memtable.put(Memtable.java:160) at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13859672#comment-13859672 ] Eric Lubow commented on CASSANDRA-6494: --- We dropped a table named 'uniques' and recreated the same table with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. Cassandra refuses to restart due to a corrupted commit log. --- Key: CASSANDRA-6494 URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 Project: Cassandra Issue Type: Bug Reporter: Shao-Chuan Wang This is running on our production server. Please advise how to address this issue. Thank you! INFO 02:46:58,879 Finished reading /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log ERROR 02:46:58,879 Exception encountered during startup java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) ... 8 more Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) at edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) at edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) at edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) at edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) at edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) at org.apache.cassandra.db.Memtable.put(Memtable.java:160) at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 706167655f74616773 is not defined as a collection at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96)
[jira] [Commented] (CASSANDRA-4730) CommitLogReplayer should report the bad CRC checksum in the log
[ https://issues.apache.org/jira/browse/CASSANDRA-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698088#comment-13698088 ] Eric Lubow commented on CASSANDRA-4730: --- The one thing I would like to advocate against is that Cassandra should not fail to start if there is a corrupt CommitLog. It should throw a nasty error in the logs but not fail to load entirely. I think the default behavior should be leaning towards availability and the server should be able to start just ignoring the corrupt CommitLog. CommitLogReplayer should report the bad CRC checksum in the log --- Key: CASSANDRA-4730 URL: https://issues.apache.org/jira/browse/CASSANDRA-4730 Project: Cassandra Issue Type: Bug Affects Versions: 1.1.5 Environment: Cassandra 1.1.5 Reporter: Arya Goudarzi If commit log isn't fully fsynced, the record which fails the checksum is not replayed do this logic. It would be beneficial to log that as an error so that user can know easily what happened. ./src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java line 188 of 287 --65%-- col 54 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485333#comment-13485333 ] Eric Lubow commented on CASSANDRA-4417: --- We are getting this on DSE 2.2 (C* 1.1.5) on a new node during bootstrap. We upgraded the cluster from C* 1.0.10 about 10 days ago and upgradesstables was run on every node and we repaired the entire cluster. We ran We've been getting this error sporadically on various nodes at various points but it's not consistent. I've double and triple checked every node looking for sstable files named *-hd-* and I don't see any (assuming that's enough to tell that the sstable has been upgraded. If this error is an effect of requiring one to run upgradesstables, then how would it happen during a bootstrap? All nodes involved in this cluster are 1.1.5. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4417) invalid counter shard detected
[ https://issues.apache.org/jira/browse/CASSANDRA-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485333#comment-13485333 ] Eric Lubow edited comment on CASSANDRA-4417 at 10/27/12 2:22 AM: - We are getting this on DSE 2.2 (C* 1.1.5) on a new node during bootstrap. We upgraded the cluster from C* 1.0.10 about 10 days ago and upgradesstables was run on every node and we repaired the entire cluster. We ran We've been getting this error sporadically on various nodes at various points but it's not consistent. I've double and triple checked every node looking for sstable files named *- hd -* and I don't see any (assuming that's enough to tell that the sstable has been upgraded. If this error is an effect of requiring one to run upgradesstables, then how would it happen during a bootstrap? All nodes involved in this cluster are 1.1.5. was (Author: elubow): We are getting this on DSE 2.2 (C* 1.1.5) on a new node during bootstrap. We upgraded the cluster from C* 1.0.10 about 10 days ago and upgradesstables was run on every node and we repaired the entire cluster. We ran We've been getting this error sporadically on various nodes at various points but it's not consistent. I've double and triple checked every node looking for sstable files named *-hd-* and I don't see any (assuming that's enough to tell that the sstable has been upgraded. If this error is an effect of requiring one to run upgradesstables, then how would it happen during a bootstrap? All nodes involved in this cluster are 1.1.5. invalid counter shard detected --- Key: CASSANDRA-4417 URL: https://issues.apache.org/jira/browse/CASSANDRA-4417 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: Amazon Linux Reporter: Senthilvel Rangaswamy Seeing errors like these: 2012-07-06_07:00:27.22662 ERROR 07:00:27,226 invalid counter shard detected; (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 13) and (17bfd850-ac52-11e1--6ecd0b5b61e7, 1, 1) differ only in count; will pick highest to self-heal; this indicates a bug or corruption generated a bad counter shard What does it mean ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3865) Cassandra-cli returns 'command not found' instead of syntax error
[ https://issues.apache.org/jira/browse/CASSANDRA-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281691#comment-13281691 ] Eric Lubow commented on CASSANDRA-3865: --- This doesn't appear to be fixed. Welcome to Cassandra CLI version 1.0.8 Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. [default@linkcurrent] create column family social_poll_deltas ... with column_type = 'Standard' ... and comparator = 'CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)' ... and default_validation_class = 'UUIDType' ... and key_validation_class = 'UUIDType' ... and rows_cached = 0.0 ... and row_cache_save_period = 0 ... and row_cache_keys_to_save = 2147483647 ... and keys_cached = 20.0 ... and key_cache_save_period = 14400 ... and read_repair_chance = .25 ... and gc_grace = 864000 ... and min_compaction_threshold = 4 ... and max_compaction_threshold = 32 ... and replicate_on_write = true ... and row_cache_provider = 'SerializingCacheProvider' ... and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' ... and comment = 'Social poll totals and deltas '; Command not found: `create column family social_poll_deltas with column_type = 'Standard' and comparator = 'CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)' and default_validation_class = 'UUIDType' and key_validation_class = 'UUIDType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 14400 and read_repair_chance = .25 and gc_grace = 864000 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and row_cache_provider = 'SerializingCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' and comment = 'Social poll totals and deltas ';`. Type 'help;' or '?' for help. Cassandra-cli returns 'command not found' instead of syntax error - Key: CASSANDRA-3865 URL: https://issues.apache.org/jira/browse/CASSANDRA-3865 Project: Cassandra Issue Type: Bug Components: Core Environment: DSE 1.0.5 Reporter: Eric Lubow Priority: Trivial Labels: cassandra-cli When creating a column family from the output of 'show schema' with an index, there is a trailing comma after index_type: 0, The return from this is a 'command not found' This is misleading because the command is found, there is just a syntax error. 'Command not found: `create column family $cfname ...` -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3865) Cassandra-cli returns 'command not found' instead of syntax error
[ https://issues.apache.org/jira/browse/CASSANDRA-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281720#comment-13281720 ] Eric Lubow commented on CASSANDRA-3865: --- As an update, the problem with this (after MUCH trial and error), was that the read_repair_chance was .25 instead of 0.25. Cassandra-cli returns 'command not found' instead of syntax error - Key: CASSANDRA-3865 URL: https://issues.apache.org/jira/browse/CASSANDRA-3865 Project: Cassandra Issue Type: Bug Components: Core Environment: DSE 1.0.5 Reporter: Eric Lubow Assignee: Yuki Morishita Priority: Trivial Labels: cassandra-cli When creating a column family from the output of 'show schema' with an index, there is a trailing comma after index_type: 0, The return from this is a 'command not found' This is misleading because the command is found, there is just a syntax error. 'Command not found: `create column family $cfname ...` -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3865) Cassandra-cli returns 'command not found' instead of syntax error
[ https://issues.apache.org/jira/browse/CASSANDRA-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282150#comment-13282150 ] Eric Lubow commented on CASSANDRA-3865: --- As Yuki said, I'd be more interested in the fix being a complaint of a syntax error (or parse error) when attempting to create a column family as opposed to command not found (which is clearly ambiguous). This way I (as a user) know that the CLI is aware that I am attempting to create a column family and it's failing. Now not only do I know that I am on the right track with the command I'm trying to execute, but then I know roughly where the problem is. Fixing the parse error is just a band-aid. Cassandra-cli returns 'command not found' instead of syntax error - Key: CASSANDRA-3865 URL: https://issues.apache.org/jira/browse/CASSANDRA-3865 Project: Cassandra Issue Type: Bug Components: Core Environment: DSE 1.0.5 Reporter: Eric Lubow Assignee: Dave Brosius Priority: Trivial Labels: cassandra-cli Fix For: 1.0.11, 1.1.1 Attachments: parse_doubles_better.txt When creating a column family from the output of 'show schema' with an index, there is a trailing comma after index_type: 0, The return from this is a 'command not found' This is misleading because the command is found, there is just a syntax error. 'Command not found: `create column family $cfname ...` -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
