[
https://issues.apache.org/jira/browse/CASSANDRA-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333349#comment-17333349
]
Andres de la Peña commented on CASSANDRA-16625:
-----------------------------------------------
[Here|https://github.com/adelapena/cassandra/commit/682aa6082f55f0d508eeaf3d31021ce9363a3f84]
is as simple patch adding jobs to run a specific Python dtest multiple times.
The new jobs are:
* j8_repeated-dtest-with-vnodes
* j8_repeated-dtest-no-vnodes
* 11_repeated-dtest-with-vnodes
* j11_repeated-dtest-no-vnodes
All these jobs are optional and require user approval to start. The name of the
test to run is defined by the env var {{REPEATED_DTEST_PYTHON_TEST_NAME}}. It
can be anything of the form:
* pending_range_test.py
* pending_range_test.py::TestPendingRangeMovements
* pending_range_test.py::TestPendingRangeMovements::test_pending_range
The number of iterations is set in the environment variable
{{REPEATED_DTEST_PYTHON_COUNT}}, which defaults to 100. Sadly I haven't found a
way to introduce these values in the CircleCI UI, so they should be set in the
{{.circleci/config.yml}}, as we do when we want to run a specific dtest branch.
All the jobs use the parallel executor, evenly distributing the test iterations
among the runners. For example, if we use the MIDRES config with a parallelism
of 25 and we want to run a certain test 100 times, each runner will run the
test 4 times.
[Here|https://app.circleci.com/pipelines/github/adelapena/cassandra/305/workflows/827ede69-9a87-4738-a326-4368e94daa0b/jobs/2737]
we can see 100 runs of {{pending_range_test.py}}, which was recently fixed by
CASSANDRA-16614. It has taken less than 3 minutes with HIGHRES.
Also,
[here|https://app.circleci.com/pipelines/github/adelapena/cassandra/304/workflows/e848906f-86e5-4779-a982-b4887570c012/jobs/2729]
we can see 100 runs of the class
{{cqlsh_tests/test_cqlsh.py::TestCqlshSmoke}}, taking around six minutes with
HIGHRES.
> Add a CircleCI job to run some tests repeatedly
> -----------------------------------------------
>
> Key: CASSANDRA-16625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16625
> Project: Cassandra
> Issue Type: Task
> Components: CI
> Reporter: Andres de la Peña
> Assignee: Andres de la Peña
> Priority: Normal
>
> I think it could be useful to have an optional CircleCI job to run some
> specific tests n times. That way, tickets could attach CircleCI runs showing
> that the changes don't make a certain ticket flaky or, conversely, that they
> fix a flaky test. Doing this systematically should mitigate the risk of
> introducing new flaky tests, and I guess it would be more convenient and easy
> to share than running the tests locally or on a private CI system.
> It would also be nice to have something similar in Jenkins, but I'm focusing
> this ticket on CircleCI because it's available also for non-committers, so
> assignees can run their tests before setting the tickets as ready for review.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
