[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298522#comment-15298522
]
Stefano Ortolani commented on CASSANDRA-11470:
----------------------------------------------
I am not sure it's related, but I noticed another exception while checking my
logs.
Note that, unlike with CASSANDRA-11823, it's not related to creating a new
column family.
{code:xml}
WARN [CompactionExecutor:2006] 2016-05-23 18:23:31,694 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:eda6b9c36f8df6fe596492c3438d7a38e9b109a6
(123663388 bytes)
INFO [IndexSummaryManager:1] 2016-05-23 18:24:23,731
IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [CompactionExecutor:2006] 2016-05-23 18:24:56,669 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:05b6b424194dd19ab7cfbcd53c4979768cd859e9
(256286063 bytes)
WARN [CompactionExecutor:2006] 2016-05-23 18:26:23,575 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:04e9fac15552b9ae77c27a6cb8d0fd11fdcc24d7
(212445557 bytes)
INFO [CompactionExecutor:2005] 2016-05-23 18:29:26,839
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61041-big-Data.db')
to candidates
WARN [CompactionExecutor:2006] 2016-05-23 18:30:34,154 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:edbe6f178503be90911dbf29a55b97a4b095a9ec
(183852539 bytes)
INFO [CompactionExecutor:2006] 2016-05-23 18:31:21,080
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61042-big-Data.db')
to candidates
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,207
LogFile.java:173 - Unexpected files detected for sstable [ma-91034-big], record
[REMOVE:[/data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396/ma-91034-big,1463992176000,8][457420186]]:
last update time [00:00:00] should have been [08:29:36]
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,208
ScheduledReporter.java:119 - RuntimeException thrown from
GraphiteReporter#report. Exception was suppressed.
java.lang.RuntimeException: Failed to list files in
/data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:57)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.<init>(Directories.java:981)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:692)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:686)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:281)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
~[metrics-core-3.1.0.jar:3.1.0]
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
~[metrics-core-3.1.0.jar:3.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by:
org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException:
Some records failed verification. See earlier in log for details.
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:114)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:106)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
~[na:1.8.0_91]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[na:1.8.0_91]
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
~[na:1.8.0_91]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[na:1.8.0_91]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
~[na:1.8.0_91]
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
~[na:1.8.0_91]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
~[na:1.8.0_91]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[na:1.8.0_91]
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
~[na:1.8.0_91]
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:75)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
~[apache-cassandra-3.0.6.jar:3.0.6]
... 22 common frames omitted
INFO [CompactionExecutor:2005] 2016-05-23 18:32:27,284
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_3-37a9cb90b7ac11e48a9b6776bf484396/ma-98189-big-Data.db')
to candidates
INFO [CompactionExecutor:2005] 2016-05-23 18:34:42,562
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_3-37a9cb90b7ac11e48a9b6776bf484396/ma-98210-big-Data.db')
to candidates
{code}
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
> Reporter: Philip Thompson
> Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> ~[na:1.8.0_45]
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> ~[na:1.8.0_45]
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
> ~[na:1.8.0_45]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
> ~[na:1.8.0_45]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
> ~[na:1.8.0_45]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> ~[na:1.8.0_45]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> ~[na:1.8.0_45]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> ~[na:1.8.0_45]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:71)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:49)
> ~[main/:na]
> ... 25 common frames omitted
> {code}
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1092/testReport/materialized_views_test/TestMaterializedViews/base_replica_repair_test
> Failed on CassCI build trunk_dtest #1092
> I've attached the logs from the failure in build 1092.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
