I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
includes tons of details (and a patch available for review) with my efforts to 
migrate dtests from nosetest to pytest (which ultimately ended up also 
including porting the ode from python 2.7 to python 3).

I'd love if people could pitch in in any way to help get this reviewed and 
committed so we can reduce the natural drift that will occur with a huge patch 
like this against the changes going into master. I apologize for sending this 
so close to the holidays, but I really have been working non-stop trying to get 
things into a completed and stable state.

The latest CircleCI runs I did took roughly 15 minutes to run all the dtests 
with only 6 failures remaining (when run with vnodes) and 12 failures remaining 
(when run without vnodes). For comparison the last ASF Jenkins Dtest job to 
successfully complete took nearly 10 hours (9:51) and we had 36 test failures. 
Of note, while I was working on this and trying to determine a baseline for the 
existing tests I found that the ASF Jenkins jobs were incorrectly configured 
due to a typo. The no-vnodes job is actually running with vnodes (meaning the 
no-vnodes job is identical to the with-vnodes ASF Jenkins job). There are some 
bootstrap tests that will 100% reliably hang both nosetest and pytest on test 
cleanup, however this test only runs in the no-vnodes configuration. I've 
debugged and fixed a lot of these cases across many test cases over the past 
few weeks and I no longer know of any tests that can hang CI.

Thanks and I'm optimistic about making testing great for the project and most 
importantly for the OSS C* community!

best,
kjellman

Some highlights that I quickly thought of (in no particular order): {also 
included in the JIRA}
-Migrate dtests from executing using the nosetest framework to pytest
-Port the entire code base from Python 2.7 to Python 3.6
-Update run_dtests.py to work with pytest
-Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
list of all available collected tests
-Update README.md for executing the dtests with pytest
-Add new debugging tips section to README.md to help with some basics of 
debugging python3 and pytest
-Migrate all existing Enviornment Variable usage as a means to control dtest 
operation modes to argparse command line options with documented help on each 
toggles intended usage
-Migration of old unitTest and nose based test structure to modern pytest 
fixture approach
-Automatic detection of physical system resources to automatically determine if 
@pytest.mark.resource_intensive annotated tests should be collected and run on 
the system where they are being executed
-new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
annotations
-Migration to python logging framework
-Upgrade thrift bindings to latest version with full python3 compatibility
-Remove deprecated cql and pycassa dependencies and migrate any remaining tests 
to fully remove those dependencies
-Fixed dozens of tests that would hang the pytest framework forever when run in 
CI enviornments
-Ran code nearly 300 times in CircleCI during the migration and to find, 
identify, and fix any tests capable of hanging CI
-Upgrade Tests do not yet run in CI and still need additional migration work 
(although all upgrade test classes compile successfully)

Reply via email to