[jira] [Commented] (CASSANDRA-12261) dtest failure in write_failures_test.TestWriteFailures.test_thrift

2016-08-29 Thread Benjamin Lerer (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445799#comment-15445799
 ] 

Benjamin Lerer commented on CASSANDRA-12261:


+1

> dtest failure in write_failures_test.TestWriteFailures.test_thrift
> --
>
> Key: CASSANDRA-12261
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12261
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
>  Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/14/testReport/write_failures_test/TestWriteFailures/test_thrift
> Failure is
> {code}
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,127 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-2-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,334 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-15-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,337 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-31-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,339 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-18-big-Data.db
>  as it does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12261) dtest failure in write_failures_test.TestWriteFailures.test_thrift

2016-07-28 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398519#comment-15398519
 ] 

Stefania commented on CASSANDRA-12261:
--

Thanks anyway. Should it happen again, the exception stack trace added by this 
patch should give us more details.

> dtest failure in write_failures_test.TestWriteFailures.test_thrift
> --
>
> Key: CASSANDRA-12261
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12261
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
>  Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/14/testReport/write_failures_test/TestWriteFailures/test_thrift
> Failure is
> {code}
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,127 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-2-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,334 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-15-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,337 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-31-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,339 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-18-big-Data.db
>  as it does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12261) dtest failure in write_failures_test.TestWriteFailures.test_thrift

2016-07-28 Thread Jim Witschey (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397687#comment-15397687
 ] 

Jim Witschey commented on CASSANDRA-12261:
--

bq. are you aware of any other tests with the same problem or a reasonably 
reliable way to reproduce this?

No and no, unfortunately.

> dtest failure in write_failures_test.TestWriteFailures.test_thrift
> --
>
> Key: CASSANDRA-12261
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12261
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
>  Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/14/testReport/write_failures_test/TestWriteFailures/test_thrift
> Failure is
> {code}
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,127 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-2-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,334 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-15-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,337 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-31-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,339 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-18-big-Data.db
>  as it does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12261) dtest failure in write_failures_test.TestWriteFailures.test_thrift

2016-07-27 Thread Stefania (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396899#comment-15396899
 ] 

Stefania commented on CASSANDRA-12261:
--

This test seems to have failed only once with this specific failure and it 
cannot be reproduced locally. [~mambocab], are you aware of any other tests 
with the same problem or a reasonably reliable way to reproduce this?

The sstable transaction tidier deletes the sstable data file but at present it 
does not check if the data file exists. I've added this check and, if the 
sstable was not new, a new error message that should tell us if the sstable 
tidier ran without data file. After all sstable tidiers have run, the 
transaction tidier does another pass to ensure no content files are left before 
deleting the actual transaction log file. Here we were missing a folder sync, 
so I think it's not 100% impossible that {{record.getExistingFiles()}} returned 
a wrong result including files that were deleted just before by the sstable 
tidiers, although this wouldn't explain why only data files were reported as 
non existing. I've added the folder sync call. Unless the same file is involved 
in two different transactions, which shouldn't happen, there shouldn't be any 
race between the call to record.getExistingFiles() and LogTransaction::delete 
in LogFile.deleteRecordFiles().

I've also added the exception call stack in LogTransaction.delete(), but only 
to the debug log file.

I've prepared the patches fro 3.9 and trunk:

||3.9||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/12261-3.9]|[patch|https://github.com/stef1927/cassandra/commits/12261]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12261-3.9-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12261-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12261-3.9-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12261-dtest/]|

This problem could technically happen in 3.0 as well, but I would rather not 
disrupt 3.0 unless we are sure there is an issue in the first place.

> dtest failure in write_failures_test.TestWriteFailures.test_thrift
> --
>
> Key: CASSANDRA-12261
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12261
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Stefania
>  Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/14/testReport/write_failures_test/TestWriteFailures/test_thrift
> Failure is
> {code}
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,127 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-2-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,334 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-15-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,337 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-31-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,339 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-18-big-Data.db
>  as it does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12261) dtest failure in write_failures_test.TestWriteFailures.test_thrift

2016-07-21 Thread Philip Thompson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387778#comment-15387778
 ] 

Philip Thompson commented on CASSANDRA-12261:
-

[~mambocab], can you take a look? This might just need moved to the bug queue

> dtest failure in write_failures_test.TestWriteFailures.test_thrift
> --
>
> Key: CASSANDRA-12261
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12261
> Project: Cassandra
>  Issue Type: Test
>Reporter: Philip Thompson
>Assignee: DS Test Eng
>  Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, 
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/14/testReport/write_failures_test/TestWriteFailures/test_thrift
> Failure is
> {code}
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,127 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-2-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,334 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-15-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,337 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-31-big-Data.db
>  as it does not exist
> Unexpected error in node3 log, error: 
> ERROR [NonPeriodicTasks:1] 2016-07-20 07:09:52,339 LogTransaction.java:205 - 
> Unable to delete 
> /tmp/dtest-CSPEFG/test/node3/data2/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mb-18-big-Data.db
>  as it does not exist
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)