[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299327#comment-15299327
]
Stefania commented on CASSANDRA-11470:
--
It's a different issue, thanks for reporting it! I've opened CASSANDRA-11889
for it.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298522#comment-15298522
]
Stefano Ortolani commented on CASSANDRA-11470:
--
I am not sure it's related, but I noticed another exception while checking my
logs.
Note that, unlike with CASSANDRA-11823, it's not related to creating a new
column family.
{code:xml}
WARN [CompactionExecutor:2006] 2016-05-23 18:23:31,694 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:eda6b9c36f8df6fe596492c3438d7a38e9b109a6
(123663388 bytes)
INFO [IndexSummaryManager:1] 2016-05-23 18:24:23,731
IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [CompactionExecutor:2006] 2016-05-23 18:24:56,669 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:05b6b424194dd19ab7cfbcd53c4979768cd859e9
(256286063 bytes)
WARN [CompactionExecutor:2006] 2016-05-23 18:26:23,575 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:04e9fac15552b9ae77c27a6cb8d0fd11fdcc24d7
(212445557 bytes)
INFO [CompactionExecutor:2005] 2016-05-23 18:29:26,839
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61041-big-Data.db')
to candidates
WARN [CompactionExecutor:2006] 2016-05-23 18:30:34,154 BigTableWriter.java:171
- Writing large partition
test_keyspace/test_columnfamily:edbe6f178503be90911dbf29a55b97a4b095a9ec
(183852539 bytes)
INFO [CompactionExecutor:2006] 2016-05-23 18:31:21,080
LeveledManifest.java:437 - Adding high-level (L3)
BigTableReader(path='/data/cassandra/data/test_keyspace/test_columnfamily_2-d29dd71045a811e59aff6776bf484396/ma-61042-big-Data.db')
to candidates
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,207
LogFile.java:173 - Unexpected files detected for sstable [ma-91034-big], record
[REMOVE:[/data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396/ma-91034-big,1463992176000,8][457420186]]:
last update time [00:00:00] should have been [08:29:36]
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-23 18:31:21,208
ScheduledReporter.java:119 - RuntimeException thrown from
GraphiteReporter#report. Exception was suppressed.
java.lang.RuntimeException: Failed to list files in
/data/cassandra/data/test_keyspace/test_columnfamily-3996ce80b7ac11e48a9b6776bf484396
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:57)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.(Directories.java:981)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:692)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
org.apache.cassandra.metrics.TableMetrics$33.getValue(TableMetrics.java:686)
~[apache-cassandra-3.0.6.jar:3.0.6]
at
com.codahale.metrics.graphite.GraphiteReporter.reportGauge(GraphiteReporter.java:281)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:158)
~[metrics-graphite-3.1.0.jar:3.1.0]
at
com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
~[metrics-core-3.1.0.jar:3.1.0]
at
com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
~[metrics-core-3.1.0.jar:3.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_91]
at
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288493#comment-15288493
]
Stefania commented on CASSANDRA-11470:
--
Thank you for letting us know, it's really appreciated, and for reporting the
concurrent modification exception in the table metrics (CASSANDRA-11823).
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288481#comment-15288481
]
Stefano Ortolani commented on CASSANDRA-11470:
--
[~Stefania] I can confirm that 3.0.6 seems to be fixing the bug (repairing and
no exceptions triggered so far). I have though seen a possible race between the
reporter and the data storage when creating a table (happened on 3/4 nodes out
of 13). Opened a ticket here: CASSANDRA-11823
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281036#comment-15281036
]
Stefania commented on CASSANDRA-11470:
--
This would make sense, since during anti-compactions several new sstable files
would be created and it is new files that are logged in the txn but do not yet
exist on disk, that would incorrectly cause the inconsistent disk state. This
patch should have rectified this. Please do let us know if this problem
persists with 3.0.6.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280906#comment-15280906
]
Stefano Ortolani commented on CASSANDRA-11470:
--
I see it happening quite consistently during AntiCompactions.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278665#comment-15278665
]
Stefano Ortolani commented on CASSANDRA-11470:
--
I am having the same exception triggered after a runtime exception thrown by
the graphitereporter:
{{code:xml}}
ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-10 18:41:21,672
ScheduledReporter.java:119 - RuntimeException thrown from
GraphiteReporter#report. Exception was suppressed.
java.lang.RuntimeException: Failed to list files in
/data/cassandra/data/schema_test/cf_test-e32122c0800c11e48a94fb3921529dd4
{{code}
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255493#comment-15255493
]
Stefania commented on CASSANDRA-11470:
--
The release version was incorrect, this patch was delivered in 3.0.6, not
3.0.5. Please try again with 3.0.6.
The patch has added an extended log error message that should help debugging
any more issues. This message should appear just before the exception. Please
make sure to include it if you still have problems with 3.0.6.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.6, 3.0.6
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253732#comment-15253732
]
Alexander Nechiporuk commented on CASSANDRA-11470:
--
Hi, probably issue still exists in 3.0.5, but appears in different place.
{code}
nodetool cfstats XXX.YYY
Keyspace: XXX
Read Count: 75443752
Read Latency: 27.975226176251148 ms.
Write Count: 66394239
Write Latency: 0.03455783252519846 ms.
Pending Flushes: 0
Table:
SSTable count: 778
SSTables in each level: [20/4, 109/10, 105/100, 542, 0, 0, 0,
0, 0]
Space used (live): 152332003329
Space used (total): 152332003329
error: Failed to list directory files in /place/cassandra/data//
inconsistent disk state for transaction
[/place/cassandra/data///ma_txn_compaction_2695f850-0874-11e6-b5e6-1335f8eceb06.log]
-- StackTrace --
java.lang.RuntimeException: Failed to list directory files in
/place/cassandra/data/XXX/YYY, inconsistent disk state for transaction
[/place/cassan
dra/data/XXX/YYY/ma_txn_compaction_2695f850-0874-11e6-b5e6-1335f8eceb06.log]
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:71)
at
org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:49)
at
org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
at
org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:691)
at
org.apache.cassandra.db.Directories$SSTableLister.listFiles(Directories.java:662)
at
org.apache.cassandra.db.Directories$TrueFilesSizeVisitor.(Directories.java:981)
at
org.apache.cassandra.db.Directories.getTrueAllocatedSizeIn(Directories.java:893)
at
org.apache.cassandra.db.Directories.trueSnapshotsSize(Directories.java:883)
at
org.apache.cassandra.db.ColumnFamilyStore.trueSnapshotsSize(ColumnFamilyStore.java:2332)
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:637)
at
org.apache.cassandra.metrics.TableMetrics$32.getValue(TableMetrics.java:634)
at
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge.getValue(CassandraMetricsRegistry.java:235)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234815#comment-15234815
]
Stefania commented on CASSANDRA-11470:
--
Thanks, I've changed commit message and entry in CHANGES.txt to
{{LogAwareFileLister should only use OLD sstable files in current folder to
determine disk consistency}}.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source)
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234797#comment-15234797
]
Marcus Eriksson commented on CASSANDRA-11470:
-
+1, but could you update the branches with better CHANGES.txt entry? Since the
bug is not related to materialized views it is a bit confusing.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234394#comment-15234394
]
Stefania commented on CASSANDRA-11470:
--
Thanks for rerunning. We can see several failures and they are caused by
missing NEW files, so looking at OLD files definitely fixes the problem.
I don't know why it only happens on trunk, IMO the problem exists in 3.0 as
well and so I've produced a patch for both branches:
||3.0||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/11470-3.0]|[patch|https://github.com/stef1927/cassandra/commits/11470]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-dtest/]|
The two patches are slightly different because on trunk we have a method in
LogTransaction that prints the txn contents, whilst on 3.0 we need to access
the records in order to print them.
[~krummas] would you be OK reviewing this?
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232365#comment-15232365
]
Philip Thompson commented on CASSANDRA-11470:
-
That makes sense. Running here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/63/
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231492#comment-15231492
]
Stefania commented on CASSANDRA-11470:
--
Thanks for setting this up so quickly. No failures, which is good I suppose.
However, I really would like to understand if this is because of the patch of
because of some other reason. Could we run it another 150 times perhaps,
against [this patch|https://github.com/stef1927/cassandra/commits/11470-debug],
which is just trunk + the log message. If we find a failure, this should tell
us what the problem is and confirm that indeed the patch has fixed it. If it
doesn't fail, then it may be that for some reason we are not able to reproduce
the problem when running the test in isolation. What do you think?
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230643#comment-15230643
]
Philip Thompson commented on CASSANDRA-11470:
-
Done so here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/62/console
It will run 450 times. Hopefully that's enough.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230178#comment-15230178
]
Stefania commented on CASSANDRA-11470:
--
CI results are fine. [~philipthompson] could you set-up a job that runs
{{base_replica_repair_test}} several times against the patch above?
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229849#comment-15229849
]
Stefania commented on CASSANDRA-11470:
--
I fixed a problem in {{LogAwareFileLister.classifyFiles}}, which would have
affected transactions spread across multiple disks. However, this is not what
is causing these tests to fail unless we are using multiple disks and I haven't
noticed.
I've also fixed another problem with the existing logic, in that we should only
check that OLD sstable files still exist to determine if there is a race, we
don't need to look at NEW sstable files. Since we cannot tell in advance how
many NEW sstables files will be created and when, this might explain the
problems we are seeing although I would have expected them on 3.0 as well.
Finally, if we fail to classify files due to an inconsistent disk state, I've
added a log message showing the list of files detected and the content of the
transaction.
Patch and CI results for trunk:
|[patch|https://github.com/stef1927/cassandra/commits/11470]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-dtest/]|
Depending on the CI results, we may need to run {{base_replica_repair_test}}
multiple times to see if the problem is fixed and to gather more information if
it isn't.
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
>
[jira] [Commented] (CASSANDRA-11470) dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220248#comment-15220248
]
Philip Thompson commented on CASSANDRA-11470:
-
This appears to have nothing to do with MV. I am now also seeing in in
readrepair_test on trunk
http://cassci.datastax.com/job/trunk_offheap_dtest/100/testReport/
> dtest failure in
> materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log,
> node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in
> the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception
> encountered during startup
> java.lang.RuntimeException: Failed to list files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725)
> ~[main/:na]
> at
> org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:320)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.restrictions.StatementRestrictions.(StatementRestrictions.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
> ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
> [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985,
> inconsistent disk state for transaction
> [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in
> /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown
> Source) ~[na:na]
> at
>
