Re: [Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Michael Kjellman]

no, i’m not. i just figured i should target python 3.6 if i was doing this work 
in the first place. the current Ubuntu LTS was pulling in a pretty old version. 
any concerns with using 3.6?

> On Jan 3, 2018, at 1:51 AM, Stefan Podkowinski  wrote:
> 
> The latest updates to your branch fixed the logging issue, thanks! Tests
> now seem to execute fine locally using pytest.
> 
> I was looking at the dockerfile and noticed that you explicitly use
> python 3.6 there. Are you aware of any issues with older python3
> versions, e.g. 3.5? Do I have to use 3.6 as well locally and do we have
> to do the same for jenkins?
> 
> 
>> On 02.01.2018 22:42, Michael Kjellman wrote:
>> I reproduced the NOTSET log issue locally... got a fix.. i'll push a commit 
>> up in a moment.
>> 
>>> On Jan 2, 2018, at 11:24 AM, Michael Kjellman 
>>>  wrote:
>>> 
>>> Comments Inline: Thanks for giving this a go!!
>>> 
 On Jan 2, 2018, at 6:10 AM, Stefan Podkowinski  wrote:
 
 I was giving this a try today with some mixed results. First of all,
 running pytest locally would fail with an "ccmlib.common.ArgumentError:
 Unknown log level NOTSET" error for each test. Although I created a new
 virtualenv for that as described in the readme (thanks for updating!)
 and use both of your dtest and cassandra branches. But I haven't patched
 ccm as described in the ticket, maybe that's why? Can you publish a
 patched ccm branch to gh?
>>> 
>>> 99% sure this is an issue parsing the logging level passed to pytest to the 
>>> python logger... could you paste the exact command you're using to invoke 
>>> pytest? should be a small change - i'm sure i just missed a invocation case.
>>> 
 
 The updated circle.yml is now using docker, which seems to be a good
 idea to reduce clutter in the yaml file and gives us more control over
 the test environment. Can you add the Dockerfile to the .circleci
 directory as well? I couldn't find it when I was trying to solve the
 pytest error mentioned above.
>>> 
>>> This is already tracked in a separate repo: 
>>> https://github.com/mkjellman/cassandra-test-docker/blob/master/Dockerfile
 
 Next thing I did was to push your trunk_circle branch to my gh repo to
 start a circleCI run. Finishing all dtests in 15 minutes sounds
 exciting, but requires a paid tier plan to get that kind of
 parallelization. Looks like the dtests have even been deliberately
 disabled for non-paid accounts, so I couldn't test this any further.
>>> 
>>> the plan of action (i already already mentioned this in previous emails) is 
>>> to get dtests working for the free circieci oss accounts as well. part of 
>>> this work (already included in this pytest effort) is to have fixtures that 
>>> look at the system resources and dynamically include tests as possible.
>>> 
 
 Running dtests from the pytest branch on builds.apache.org did not work
 either. At least the run_dtests.py arguments will need to be updated in
 cassandra-builds. We currently only use a single cassandra-dtest.sh
 script for all builds. Maybe we should create a new job template that
 would use an updated script with the wip-pytest dtest branch, to make
 this work and testable in parallel.
>>> 
>>> yes, i didn't touch cassandra-builds yet.. focused on getting circleci and 
>>> local runs working first... once we're happy with that and stable we can 
>>> make the changes to jenkins configs pretty easily...
>>> 
 
 
 
> On 21.12.2017 11:13, Michael Kjellman wrote:
> I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 
> which includes tons of details (and a patch available for review) with my 
> efforts to migrate dtests from nosetest to pytest (which ultimately ended 
> up also including porting the ode from python 2.7 to python 3).
> 
> I'd love if people could pitch in in any way to help get this reviewed 
> and committed so we can reduce the natural drift that will occur with a 
> huge patch like this against the changes going into master. I apologize 
> for sending this so close to the holidays, but I really have been working 
> non-stop trying to get things into a completed and stable state.
> 
> The latest CircleCI runs I did took roughly 15 minutes to run all the 
> dtests with only 6 failures remaining (when run with vnodes) and 12 
> failures remaining (when run without vnodes). For comparison the last ASF 
> Jenkins Dtest job to successfully complete took nearly 10 hours (9:51) 
> and we had 36 test failures. Of note, while I was working on this and 
> trying to determine a baseline for the existing tests I found that the 
> ASF Jenkins jobs were incorrectly configured due to a typo. The no-vnodes 
> job is actually running with vnodes (meaning the no-vnodes job is 
> identical to the 

Re: [Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Stefan Podkowinski]

The latest updates to your branch fixed the logging issue, thanks! Tests
now seem to execute fine locally using pytest.

I was looking at the dockerfile and noticed that you explicitly use
python 3.6 there. Are you aware of any issues with older python3
versions, e.g. 3.5? Do I have to use 3.6 as well locally and do we have
to do the same for jenkins?


On 02.01.2018 22:42, Michael Kjellman wrote:
> I reproduced the NOTSET log issue locally... got a fix.. i'll push a commit 
> up in a moment.
> 
>> On Jan 2, 2018, at 11:24 AM, Michael Kjellman  
>> wrote:
>>
>> Comments Inline: Thanks for giving this a go!!
>>
>>> On Jan 2, 2018, at 6:10 AM, Stefan Podkowinski  wrote:
>>>
>>> I was giving this a try today with some mixed results. First of all,
>>> running pytest locally would fail with an "ccmlib.common.ArgumentError:
>>> Unknown log level NOTSET" error for each test. Although I created a new
>>> virtualenv for that as described in the readme (thanks for updating!)
>>> and use both of your dtest and cassandra branches. But I haven't patched
>>> ccm as described in the ticket, maybe that's why? Can you publish a
>>> patched ccm branch to gh?
>>
>> 99% sure this is an issue parsing the logging level passed to pytest to the 
>> python logger... could you paste the exact command you're using to invoke 
>> pytest? should be a small change - i'm sure i just missed a invocation case.
>>
>>>
>>> The updated circle.yml is now using docker, which seems to be a good
>>> idea to reduce clutter in the yaml file and gives us more control over
>>> the test environment. Can you add the Dockerfile to the .circleci
>>> directory as well? I couldn't find it when I was trying to solve the
>>> pytest error mentioned above.
>>
>> This is already tracked in a separate repo: 
>> https://github.com/mkjellman/cassandra-test-docker/blob/master/Dockerfile
>>>
>>> Next thing I did was to push your trunk_circle branch to my gh repo to
>>> start a circleCI run. Finishing all dtests in 15 minutes sounds
>>> exciting, but requires a paid tier plan to get that kind of
>>> parallelization. Looks like the dtests have even been deliberately
>>> disabled for non-paid accounts, so I couldn't test this any further.
>>
>> the plan of action (i already already mentioned this in previous emails) is 
>> to get dtests working for the free circieci oss accounts as well. part of 
>> this work (already included in this pytest effort) is to have fixtures that 
>> look at the system resources and dynamically include tests as possible.
>>
>>>
>>> Running dtests from the pytest branch on builds.apache.org did not work
>>> either. At least the run_dtests.py arguments will need to be updated in
>>> cassandra-builds. We currently only use a single cassandra-dtest.sh
>>> script for all builds. Maybe we should create a new job template that
>>> would use an updated script with the wip-pytest dtest branch, to make
>>> this work and testable in parallel.
>>
>> yes, i didn't touch cassandra-builds yet.. focused on getting circleci and 
>> local runs working first... once we're happy with that and stable we can 
>> make the changes to jenkins configs pretty easily...
>>
>>>
>>>
>>>
>>> On 21.12.2017 11:13, Michael Kjellman wrote:
 I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
 includes tons of details (and a patch available for review) with my 
 efforts to migrate dtests from nosetest to pytest (which ultimately ended 
 up also including porting the ode from python 2.7 to python 3).

 I'd love if people could pitch in in any way to help get this reviewed and 
 committed so we can reduce the natural drift that will occur with a huge 
 patch like this against the changes going into master. I apologize for 
 sending this so close to the holidays, but I really have been working 
 non-stop trying to get things into a completed and stable state.

 The latest CircleCI runs I did took roughly 15 minutes to run all the 
 dtests with only 6 failures remaining (when run with vnodes) and 12 
 failures remaining (when run without vnodes). For comparison the last ASF 
 Jenkins Dtest job to successfully complete took nearly 10 hours (9:51) and 
 we had 36 test failures. Of note, while I was working on this and trying 
 to determine a baseline for the existing tests I found that the ASF 
 Jenkins jobs were incorrectly configured due to a typo. The no-vnodes job 
 is actually running with vnodes (meaning the no-vnodes job is identical to 
 the with-vnodes ASF Jenkins job). There are some bootstrap tests that will 
 100% reliably hang both nosetest and pytest on test cleanup, however this 
 test only runs in the no-vnodes configuration. I've debugged and fixed a 
 lot of these cases across many test cases over the past few weeks and I no 
 longer know of any tests that can hang CI.

 Thanks and I'm 

Re: [Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Michael Kjellman]

I reproduced the NOTSET log issue locally... got a fix.. i'll push a commit up 
in a moment.

> On Jan 2, 2018, at 11:24 AM, Michael Kjellman  
> wrote:
> 
> Comments Inline: Thanks for giving this a go!!
> 
>> On Jan 2, 2018, at 6:10 AM, Stefan Podkowinski  wrote:
>> 
>> I was giving this a try today with some mixed results. First of all,
>> running pytest locally would fail with an "ccmlib.common.ArgumentError:
>> Unknown log level NOTSET" error for each test. Although I created a new
>> virtualenv for that as described in the readme (thanks for updating!)
>> and use both of your dtest and cassandra branches. But I haven't patched
>> ccm as described in the ticket, maybe that's why? Can you publish a
>> patched ccm branch to gh?
> 
> 99% sure this is an issue parsing the logging level passed to pytest to the 
> python logger... could you paste the exact command you're using to invoke 
> pytest? should be a small change - i'm sure i just missed a invocation case.
> 
>> 
>> The updated circle.yml is now using docker, which seems to be a good
>> idea to reduce clutter in the yaml file and gives us more control over
>> the test environment. Can you add the Dockerfile to the .circleci
>> directory as well? I couldn't find it when I was trying to solve the
>> pytest error mentioned above.
> 
> This is already tracked in a separate repo: 
> https://github.com/mkjellman/cassandra-test-docker/blob/master/Dockerfile
>> 
>> Next thing I did was to push your trunk_circle branch to my gh repo to
>> start a circleCI run. Finishing all dtests in 15 minutes sounds
>> exciting, but requires a paid tier plan to get that kind of
>> parallelization. Looks like the dtests have even been deliberately
>> disabled for non-paid accounts, so I couldn't test this any further.
> 
> the plan of action (i already already mentioned this in previous emails) is 
> to get dtests working for the free circieci oss accounts as well. part of 
> this work (already included in this pytest effort) is to have fixtures that 
> look at the system resources and dynamically include tests as possible.
> 
>> 
>> Running dtests from the pytest branch on builds.apache.org did not work
>> either. At least the run_dtests.py arguments will need to be updated in
>> cassandra-builds. We currently only use a single cassandra-dtest.sh
>> script for all builds. Maybe we should create a new job template that
>> would use an updated script with the wip-pytest dtest branch, to make
>> this work and testable in parallel.
> 
> yes, i didn't touch cassandra-builds yet.. focused on getting circleci and 
> local runs working first... once we're happy with that and stable we can make 
> the changes to jenkins configs pretty easily...
> 
>> 
>> 
>> 
>> On 21.12.2017 11:13, Michael Kjellman wrote:
>>> I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
>>> includes tons of details (and a patch available for review) with my efforts 
>>> to migrate dtests from nosetest to pytest (which ultimately ended up also 
>>> including porting the ode from python 2.7 to python 3).
>>> 
>>> I'd love if people could pitch in in any way to help get this reviewed and 
>>> committed so we can reduce the natural drift that will occur with a huge 
>>> patch like this against the changes going into master. I apologize for 
>>> sending this so close to the holidays, but I really have been working 
>>> non-stop trying to get things into a completed and stable state.
>>> 
>>> The latest CircleCI runs I did took roughly 15 minutes to run all the 
>>> dtests with only 6 failures remaining (when run with vnodes) and 12 
>>> failures remaining (when run without vnodes). For comparison the last ASF 
>>> Jenkins Dtest job to successfully complete took nearly 10 hours (9:51) and 
>>> we had 36 test failures. Of note, while I was working on this and trying to 
>>> determine a baseline for the existing tests I found that the ASF Jenkins 
>>> jobs were incorrectly configured due to a typo. The no-vnodes job is 
>>> actually running with vnodes (meaning the no-vnodes job is identical to the 
>>> with-vnodes ASF Jenkins job). There are some bootstrap tests that will 100% 
>>> reliably hang both nosetest and pytest on test cleanup, however this test 
>>> only runs in the no-vnodes configuration. I've debugged and fixed a lot of 
>>> these cases across many test cases over the past few weeks and I no longer 
>>> know of any tests that can hang CI.
>>> 
>>> Thanks and I'm optimistic about making testing great for the project and 
>>> most importantly for the OSS C* community!
>>> 
>>> best,
>>> kjellman
>>> 
>>> Some highlights that I quickly thought of (in no particular order): {also 
>>> included in the JIRA}
>>> -Migrate dtests from executing using the nosetest framework to pytest
>>> -Port the entire code base from Python 2.7 to Python 3.6
>>> -Update run_dtests.py to work with pytest
>>> -Add --dtest-print-tests-only option to 

Re: [Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Michael Kjellman]

Comments Inline: Thanks for giving this a go!!

> On Jan 2, 2018, at 6:10 AM, Stefan Podkowinski  wrote:
> 
> I was giving this a try today with some mixed results. First of all,
> running pytest locally would fail with an "ccmlib.common.ArgumentError:
> Unknown log level NOTSET" error for each test. Although I created a new
> virtualenv for that as described in the readme (thanks for updating!)
> and use both of your dtest and cassandra branches. But I haven't patched
> ccm as described in the ticket, maybe that's why? Can you publish a
> patched ccm branch to gh?

99% sure this is an issue parsing the logging level passed to pytest to the 
python logger... could you paste the exact command you're using to invoke 
pytest? should be a small change - i'm sure i just missed a invocation case.

> 
> The updated circle.yml is now using docker, which seems to be a good
> idea to reduce clutter in the yaml file and gives us more control over
> the test environment. Can you add the Dockerfile to the .circleci
> directory as well? I couldn't find it when I was trying to solve the
> pytest error mentioned above.

This is already tracked in a separate repo: 
https://github.com/mkjellman/cassandra-test-docker/blob/master/Dockerfile
> 
> Next thing I did was to push your trunk_circle branch to my gh repo to
> start a circleCI run. Finishing all dtests in 15 minutes sounds
> exciting, but requires a paid tier plan to get that kind of
> parallelization. Looks like the dtests have even been deliberately
> disabled for non-paid accounts, so I couldn't test this any further.

the plan of action (i already already mentioned this in previous emails) is to 
get dtests working for the free circieci oss accounts as well. part of this 
work (already included in this pytest effort) is to have fixtures that look at 
the system resources and dynamically include tests as possible.

> 
> Running dtests from the pytest branch on builds.apache.org did not work
> either. At least the run_dtests.py arguments will need to be updated in
> cassandra-builds. We currently only use a single cassandra-dtest.sh
> script for all builds. Maybe we should create a new job template that
> would use an updated script with the wip-pytest dtest branch, to make
> this work and testable in parallel.

yes, i didn't touch cassandra-builds yet.. focused on getting circleci and 
local runs working first... once we're happy with that and stable we can make 
the changes to jenkins configs pretty easily...

> 
> 
> 
> On 21.12.2017 11:13, Michael Kjellman wrote:
>> I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
>> includes tons of details (and a patch available for review) with my efforts 
>> to migrate dtests from nosetest to pytest (which ultimately ended up also 
>> including porting the ode from python 2.7 to python 3).
>> 
>> I'd love if people could pitch in in any way to help get this reviewed and 
>> committed so we can reduce the natural drift that will occur with a huge 
>> patch like this against the changes going into master. I apologize for 
>> sending this so close to the holidays, but I really have been working 
>> non-stop trying to get things into a completed and stable state.
>> 
>> The latest CircleCI runs I did took roughly 15 minutes to run all the dtests 
>> with only 6 failures remaining (when run with vnodes) and 12 failures 
>> remaining (when run without vnodes). For comparison the last ASF Jenkins 
>> Dtest job to successfully complete took nearly 10 hours (9:51) and we had 36 
>> test failures. Of note, while I was working on this and trying to determine 
>> a baseline for the existing tests I found that the ASF Jenkins jobs were 
>> incorrectly configured due to a typo. The no-vnodes job is actually running 
>> with vnodes (meaning the no-vnodes job is identical to the with-vnodes ASF 
>> Jenkins job). There are some bootstrap tests that will 100% reliably hang 
>> both nosetest and pytest on test cleanup, however this test only runs in the 
>> no-vnodes configuration. I've debugged and fixed a lot of these cases across 
>> many test cases over the past few weeks and I no longer know of any tests 
>> that can hang CI.
>> 
>> Thanks and I'm optimistic about making testing great for the project and 
>> most importantly for the OSS C* community!
>> 
>> best,
>> kjellman
>> 
>> Some highlights that I quickly thought of (in no particular order): {also 
>> included in the JIRA}
>> -Migrate dtests from executing using the nosetest framework to pytest
>> -Port the entire code base from Python 2.7 to Python 3.6
>> -Update run_dtests.py to work with pytest
>> -Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
>> list of all available collected tests
>> -Update README.md for executing the dtests with pytest
>> -Add new debugging tips section to README.md to help with some basics of 
>> debugging python3 and pytest
>> -Migrate all existing Enviornment Variable usage as a 

Re: [Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Stefan Podkowinski]

I was giving this a try today with some mixed results. First of all,
running pytest locally would fail with an "ccmlib.common.ArgumentError:
Unknown log level NOTSET" error for each test. Although I created a new
virtualenv for that as described in the readme (thanks for updating!)
and use both of your dtest and cassandra branches. But I haven't patched
ccm as described in the ticket, maybe that's why? Can you publish a
patched ccm branch to gh?

The updated circle.yml is now using docker, which seems to be a good
idea to reduce clutter in the yaml file and gives us more control over
the test environment. Can you add the Dockerfile to the .circleci
directory as well? I couldn't find it when I was trying to solve the
pytest error mentioned above.

Next thing I did was to push your trunk_circle branch to my gh repo to
start a circleCI run. Finishing all dtests in 15 minutes sounds
exciting, but requires a paid tier plan to get that kind of
parallelization. Looks like the dtests have even been deliberately
disabled for non-paid accounts, so I couldn't test this any further.

Running dtests from the pytest branch on builds.apache.org did not work
either. At least the run_dtests.py arguments will need to be updated in
cassandra-builds. We currently only use a single cassandra-dtest.sh
script for all builds. Maybe we should create a new job template that
would use an updated script with the wip-pytest dtest branch, to make
this work and testable in parallel.



On 21.12.2017 11:13, Michael Kjellman wrote:
> I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
> includes tons of details (and a patch available for review) with my efforts 
> to migrate dtests from nosetest to pytest (which ultimately ended up also 
> including porting the ode from python 2.7 to python 3).
> 
> I'd love if people could pitch in in any way to help get this reviewed and 
> committed so we can reduce the natural drift that will occur with a huge 
> patch like this against the changes going into master. I apologize for 
> sending this so close to the holidays, but I really have been working 
> non-stop trying to get things into a completed and stable state.
> 
> The latest CircleCI runs I did took roughly 15 minutes to run all the dtests 
> with only 6 failures remaining (when run with vnodes) and 12 failures 
> remaining (when run without vnodes). For comparison the last ASF Jenkins 
> Dtest job to successfully complete took nearly 10 hours (9:51) and we had 36 
> test failures. Of note, while I was working on this and trying to determine a 
> baseline for the existing tests I found that the ASF Jenkins jobs were 
> incorrectly configured due to a typo. The no-vnodes job is actually running 
> with vnodes (meaning the no-vnodes job is identical to the with-vnodes ASF 
> Jenkins job). There are some bootstrap tests that will 100% reliably hang 
> both nosetest and pytest on test cleanup, however this test only runs in the 
> no-vnodes configuration. I've debugged and fixed a lot of these cases across 
> many test cases over the past few weeks and I no longer know of any tests 
> that can hang CI.
> 
> Thanks and I'm optimistic about making testing great for the project and most 
> importantly for the OSS C* community!
> 
> best,
> kjellman
> 
> Some highlights that I quickly thought of (in no particular order): {also 
> included in the JIRA}
> -Migrate dtests from executing using the nosetest framework to pytest
> -Port the entire code base from Python 2.7 to Python 3.6
> -Update run_dtests.py to work with pytest
> -Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> -Update README.md for executing the dtests with pytest
> -Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> -Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> -Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> -Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> -new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> -Migration to python logging framework
> -Upgrade thrift bindings to latest version with full python3 compatibility
> -Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> -Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> -Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> -Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all 

[Patch Available for Review!] CASSANDRA-14134: Migrate dtests to use pytest and python3

[Michael Kjellman]

I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which 
includes tons of details (and a patch available for review) with my efforts to 
migrate dtests from nosetest to pytest (which ultimately ended up also 
including porting the ode from python 2.7 to python 3).

I'd love if people could pitch in in any way to help get this reviewed and 
committed so we can reduce the natural drift that will occur with a huge patch 
like this against the changes going into master. I apologize for sending this 
so close to the holidays, but I really have been working non-stop trying to get 
things into a completed and stable state.

The latest CircleCI runs I did took roughly 15 minutes to run all the dtests 
with only 6 failures remaining (when run with vnodes) and 12 failures remaining 
(when run without vnodes). For comparison the last ASF Jenkins Dtest job to 
successfully complete took nearly 10 hours (9:51) and we had 36 test failures. 
Of note, while I was working on this and trying to determine a baseline for the 
existing tests I found that the ASF Jenkins jobs were incorrectly configured 
due to a typo. The no-vnodes job is actually running with vnodes (meaning the 
no-vnodes job is identical to the with-vnodes ASF Jenkins job). There are some 
bootstrap tests that will 100% reliably hang both nosetest and pytest on test 
cleanup, however this test only runs in the no-vnodes configuration. I've 
debugged and fixed a lot of these cases across many test cases over the past 
few weeks and I no longer know of any tests that can hang CI.

Thanks and I'm optimistic about making testing great for the project and most 
importantly for the OSS C* community!

best,
kjellman

Some highlights that I quickly thought of (in no particular order): {also 
included in the JIRA}
-Migrate dtests from executing using the nosetest framework to pytest
-Port the entire code base from Python 2.7 to Python 3.6
-Update run_dtests.py to work with pytest
-Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
list of all available collected tests
-Update README.md for executing the dtests with pytest
-Add new debugging tips section to README.md to help with some basics of 
debugging python3 and pytest
-Migrate all existing Enviornment Variable usage as a means to control dtest 
operation modes to argparse command line options with documented help on each 
toggles intended usage
-Migration of old unitTest and nose based test structure to modern pytest 
fixture approach
-Automatic detection of physical system resources to automatically determine if 
@pytest.mark.resource_intensive annotated tests should be collected and run on 
the system where they are being executed
-new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
annotations
-Migration to python logging framework
-Upgrade thrift bindings to latest version with full python3 compatibility
-Remove deprecated cql and pycassa dependencies and migrate any remaining tests 
to fully remove those dependencies
-Fixed dozens of tests that would hang the pytest framework forever when run in 
CI enviornments
-Ran code nearly 300 times in CircleCI during the migration and to find, 
identify, and fix any tests capable of hanging CI
-Upgrade Tests do not yet run in CI and still need additional migration work 
(although all upgrade test classes compile successfully)