[ https://issues.apache.org/jira/browse/CASSANDRA-18098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846429#comment-17846429 ]
Brandon Williams commented on CASSANDRA-18098: ---------------------------------------------- >From the failure in CASSANDRA-15439: {noformat} 19:25:25,570 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-19-big-Data.db 19:25:25,570 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-16-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-17-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-18-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-23-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-2-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-5-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-8-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-11-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-14-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-3-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-6-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-24-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-20-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-21-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-22-big-Data.db 19:25:25,571 bootstrap_test ERROR /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-25-big-Data.db INFO [NonPeriodicTasks:1] 2024-04-25 19:25:24,329 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-13-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:24,545 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-7-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:24,682 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-10-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:24,841 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-4-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:24,965 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-15-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,140 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-9-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,286 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-12-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,429 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data0/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-1-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,576 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-6-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,661 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data2/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-3-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:25,840 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-14-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:26,039 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-8-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:26,164 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-11-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:26,358 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-5-big INFO [NonPeriodicTasks:1] 2024-04-25 19:25:26,479 BigFormat.java:231 - Deleting sstable: /tmp/dtest-fezk9v8f/test/node1/data1/keyspace1/standard1-3a213a10033911efaffa37bdbd08956d/nb-2-big {noformat} If we look at the log timestamps, we can see the dtest failing at 19:25:25,570 and sstables 2, 3, 5, 6, 8, 11, and 14 are part of the existing set, but these are all deleted at a slightly later time in the corresponding system log, beginning with the deletion of sstable 6 at 19:25:25,576. >From the failure in CASSANDRA-19557: {noformat} 20:56:58,522 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-1-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-4-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-7-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-10-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-13-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-2-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-5-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-8-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-11-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-14-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-3-big-Data.db 20:56:58,523 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-6-big-Data.db 20:56:58,524 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-9-big-Data.db 20:56:58,524 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-12-big-Data.db 20:56:58,524 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-15-big-Data.db 20:56:58,524 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-16-big-Data.db 20:56:58,524 bootstrap_test ERROR /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-17-big-Data.db INFO [NonPeriodicTasks:1] 2024-05-03 20:56:58,521 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-12-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:58,700 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-9-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:58,870 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-15-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,040 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-6-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,176 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data2/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-3-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,336 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-10-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,473 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-7-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,600 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-13-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,736 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-4-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,862 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data0/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-1-big INFO [NonPeriodicTasks:1] 2024-05-03 20:56:59,948 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-11-big INFO [NonPeriodicTasks:1] 2024-05-03 20:57:00,041 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-8-big INFO [NonPeriodicTasks:1] 2024-05-03 20:57:00,122 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-14-big INFO [NonPeriodicTasks:1] 2024-05-03 20:57:00,206 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-5-big INFO [NonPeriodicTasks:1] 2024-05-03 20:57:00,293 BigFormat.java:324 - Deleting sstable: /tmp/dtest-zisjefyg/test/node1/data1/keyspace1/standard1-1b255f4def2540a60000000000000005/oa-2-big {noformat} Here we can observe the same behavior, but more egregious as all the sstables were deleted after the test failed. I think our removal semantics must have changed at point and the test just needs to wait a little longer before doing its check. I'll work on modifying it to accommodate this. > Test failure bootstrap_test.py::test_cleanup > -------------------------------------------- > > Key: CASSANDRA-18098 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18098 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest/python > Reporter: Yifan Cai > Assignee: Brandon Williams > Priority: Normal > Fix For: 4.0.x, 5.0.x, 5.x > > > The test failed a few times in the recent CI runs. For example, this log > captures a recent failure. > {code:none} > 20:02:01,364 ccm INFO node1: using Java 11 for the current invocation > 20:02:02,679 bootstrap_test ERROR --- > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data0/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-1-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data0/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-4-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data0/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-7-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data0/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-10-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data0/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-13-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data1/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-2-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data1/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-5-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data1/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-8-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data1/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-11-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data1/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-14-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-3-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-6-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-9-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-12-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-15-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-16-big-Data.db > 20:02:02,679 bootstrap_test ERROR > /tmp/dtest-8kcle23s/test/node1/data2/keyspace1/standard1-e36da130727b11edb08827a767e354f3/nb-17-big-Data.db > 20:02:02,679 bootstrap_test ERROR Current count is 17, basecount was 15 > -- generated xml file: /tmp/results/dtests/pytest_result_j11_with_vnodes.xml > --- > ===Flaky Test Report=== > test_materialized_views_auth passed 1 out of the required 1 times. Success! > test_cleanup failed and was not selected for rerun. > <class 'AssertionError'> > assert not True > + where True = <bound method Event.is_set of <threading.Event object at > 0x7f071d43cba8>>() > + where <bound method Event.is_set of <threading.Event object at > 0x7f071d43cba8>> = <threading.Event object at 0x7f071d43cba8>.is_set > [<TracebackEntry /home/cassandra/cassandra-dtest/bootstrap_test.py:912>] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org