Ruslan Fomkin created CASSANDRA-16946:
-----------------------------------------
Summary: Find and remove duplicate execution of dtests enabled by
inheritance
Key: CASSANDRA-16946
URL: https://issues.apache.org/jira/browse/CASSANDRA-16946
Project: Cassandra
Issue Type: Improvement
Components: Test/dtest/python
Reporter: Ruslan Fomkin
Patch to CASSANDRA-16841 helped to reveal that there are tests, which are
selected multiple times for execution. The discussion starts from this comment.
The reason for double test execution is having a test class inheriting another
test class, which means that test methods in the parent test class will be
selected for the execution two times: one as part of their class, i.e., the
parent class, and one as part of the child class, since it inherits them.
In particular I noticed two different cases:
1. A base class is a test class according to PyTest convention and it contains
test methods in it. It is imported in different dtest file, e.g., in upgrade
folder to be executed as upgrade test, where it is inherited in a child class,
which is also a test class. Just in the second file the test methods from the
base class will be present two times: once due to import and once due to
inheritance.
2. A base test class is inherited inside the same file and it is assumed in
the file that the tests need to be executed only once. However, the tests will
be selected two times, since the base test class will be selected to execution,
and the child test class will include them and thus the test will be selected
for the execution.
I illustrate this on a [commit before my
fix|https://github.com/apache/cassandra-dtest/commit/efb82f574cd99ea598713035bbc9a95e568d73ce].
1.
[upgrade_tests/bootstrap_upgrade_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/bootstrap_upgrade_test.py]
imports a test class {{TestBootstrap}} and then a test class
{{TestBootstrapUpgrade}} inherits from it. This results in executing tests from
{{TestBootstrap}} two times:
{code}
pytest --cassandra-dir=../cassandra/ upgrade_tests/bootstrap_upgrade_test.py
--collect-only --execute-upgrade-tests-only
...
collected 57 items
<Module 'upgrade_tests/bootstrap_upgrade_test.py'>
<Class 'TestBootstrap'>
<Instance '()'>
<Function 'test_simple_bootstrap_with_ssl'>
<Function 'test_simple_bootstrap'>
<Function 'test_bootstrap_on_write_survey'>
<Function 'test_simple_bootstrap_small_keepalive_period'>
<Function 'test_simple_bootstrap_nodata'>
<Function 'test_schema_removed_nodes'>
<Function 'test_read_from_bootstrapped_node'>
<Function 'test_bootstrap_waits_for_streaming_to_finish'>
<Function
'test_consistent_range_movement_true_with_replica_down_should_fail'>
<Function
'test_consistent_range_movement_false_with_replica_down_should_succeed'>
<Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
<Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
<Function 'test_rf_gt_nodes_multidc_should_succeed'>
<Function 'test_resumable_bootstrap'>
<Function 'test_bootstrap_with_reset_bootstrap_state'>
<Function 'test_manual_bootstrap'>
<Function 'test_local_quorum_bootstrap'>
<Function 'test_shutdown_wiped_node_cannot_join'>
<Function 'test_killed_wiped_node_cannot_join'>
<Function 'test_decommissioned_wiped_node_can_join'>
<Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
<Function 'test_failed_bootstrap_wiped_node_can_join'>
<Function
'test_node_cannot_join_as_hibernating_node_without_replace_address'>
<Function 'test_simultaneous_bootstrap'>
<Function 'test_cleanup'>
<Function 'test_bootstrap_binary_disabled'>
<Function 'test_invalid_host_id'>
<Function 'test_host_id_override'>
<Class 'TestBootstrapUpgrade'>
<Instance '()'>
<Function 'test_simple_bootstrap_with_ssl'>
<Function 'test_simple_bootstrap'>
<Function 'test_bootstrap_on_write_survey'>
<Function 'test_simple_bootstrap_small_keepalive_period'>
<Function 'test_simple_bootstrap_nodata'>
<Function 'test_schema_removed_nodes'>
<Function 'test_read_from_bootstrapped_node'>
<Function 'test_bootstrap_waits_for_streaming_to_finish'>
<Function
'test_consistent_range_movement_true_with_replica_down_should_fail'>
<Function
'test_consistent_range_movement_false_with_replica_down_should_succeed'>
<Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
<Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
<Function 'test_rf_gt_nodes_multidc_should_succeed'>
<Function 'test_resumable_bootstrap'>
<Function 'test_bootstrap_with_reset_bootstrap_state'>
<Function 'test_manual_bootstrap'>
<Function 'test_local_quorum_bootstrap'>
<Function 'test_shutdown_wiped_node_cannot_join'>
<Function 'test_killed_wiped_node_cannot_join'>
<Function 'test_decommissioned_wiped_node_can_join'>
<Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
<Function 'test_failed_bootstrap_wiped_node_can_join'>
<Function
'test_node_cannot_join_as_hibernating_node_without_replace_address'>
<Function 'test_simultaneous_bootstrap'>
<Function 'test_cleanup'>
<Function 'test_bootstrap_binary_disabled'>
<Function 'test_invalid_host_id'>
<Function 'test_host_id_override'>
<Function 'test_simple_bootstrap_mixed_versions'>
{code}
2.
[/sstable_generation_loading_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/sstable_generation_loading_test.py]
defines a base class {{TestBaseSStableLoader}}, which is inherited by
{{TestSSTableGenerationAndLoading}} without doing any changes to test
configurations. As result the test methods from the base class are selected for
execution two times:
{code}
pytest --cassandra-dir=../cassandra/ sstable_generation_loading_test.py
--collect-only
...
collected 24 items
<Module 'sstable_generation_loading_test.py'>
<Class 'TestBaseSStableLoader'>
<Instance '()'>
<Function 'test_sstableloader_compression_none_to_none'>
<Function 'test_sstableloader_compression_none_to_snappy'>
<Function 'test_sstableloader_compression_none_to_deflate'>
<Function 'test_sstableloader_compression_snappy_to_none'>
<Function 'test_sstableloader_compression_snappy_to_snappy'>
<Function 'test_sstableloader_compression_snappy_to_deflate'>
<Function 'test_sstableloader_compression_deflate_to_none'>
<Function 'test_sstableloader_compression_deflate_to_snappy'>
<Function 'test_sstableloader_compression_deflate_to_deflate'>
<Function 'test_sstableloader_with_mv'>
<Class 'TestSSTableGenerationAndLoading'>
<Instance '()'>
<Function 'test_sstableloader_compression_none_to_none'>
<Function 'test_sstableloader_compression_none_to_snappy'>
<Function 'test_sstableloader_compression_none_to_deflate'>
<Function 'test_sstableloader_compression_snappy_to_none'>
<Function 'test_sstableloader_compression_snappy_to_snappy'>
<Function 'test_sstableloader_compression_snappy_to_deflate'>
<Function 'test_sstableloader_compression_deflate_to_none'>
<Function 'test_sstableloader_compression_deflate_to_snappy'>
<Function 'test_sstableloader_compression_deflate_to_deflate'>
<Function 'test_sstableloader_uppercase_keyspace_name'>
<Function 'test_incompressible_data_in_compressed_table'>
<Function 'test_remove_index_file'>
<Function 'test_sstableloader_with_mv'>
<Function 'test_sstableloader_with_failing_2i'>
{code}
[This
commit|https://github.com/apache/cassandra-dtest/pull/152/commits/4952c6392a4cc5735a6c8488312defab930ac543]
fixes the above cases, however it is likely to be more of them in dtest repo.
It is possible to discover them in two ways:
1. Run pytest with collect only and see if there are duplications. It will
require to use additional options to enable marked tests.
2. Search for test classes inheriting other test classes, i.e., according the
pattern {{class Test*(Test*)}}
Note that there will be false positives, since there are cases to inherit on
purpose with different configuration. See [this
example|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/storage_engine_upgrade_test.py#L514]
It will be good to implement a way to prevent such issue in future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
