[
https://issues.apache.org/jira/browse/CASSANDRA-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338922#comment-17338922
]
Andres de la Peña edited comment on CASSANDRA-16625 at 5/4/21, 12:15 PM:
-------------------------------------------------------------------------
I have added a loop-based CircleCI job for running repeated JUnit tests. That
job can be used for regular unit tests, long unit tests, jvm dtests, etc. The
Ant target is set in the env var
[{{REPEATED_UTEST_TARGET}}|https://github.com/apache/cassandra/blob/a63fc6b9dee772caa5014a543046a12a5d23eda5/.circleci/config-2_1.yml#L38-L44].
I still have to add some doc, but I think that the PR is ready for a first
round of review.
As an example, I have used the repeated test jobs to reproduce some of the
flaky tests that we have opened for 4.0:
||branch||config||run||
|CASSANDRA-16644|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16644]|[3270|https://app.circleci.com/pipelines/github/adelapena/cassandra/376/workflows/eb463f0a-ad93-488a-89a3-bb44fbc0c578/jobs/3270]|
|CASSANDRA-16627|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16627]|[3271|https://app.circleci.com/pipelines/github/adelapena/cassandra/375/workflows/1d48154f-b6eb-4d0b-bb88-fe40a8d40f27/jobs/3271]|
|CASSANDRA-16637|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16637]|[3255|https://app.circleci.com/pipelines/github/adelapena/cassandra/374/workflows/df3c4361-c6f3-4cbf-801f-dfedeef95652/jobs/3255]|
|CASSANDRA-16598|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16598]|[3260|https://app.circleci.com/pipelines/github/adelapena/cassandra/373/workflows/f1612e92-8a59-483b-bc79-d6f29a7f5766/jobs/3260]|
All the test failures reported in the tickets are reproduced. For some of the
runs the CircleCI UI is not able to properly parse and highlight the failed
test output, probably due to its large size. However, the failures can still be
seen in the downloadable test output file.
Unfortunately there isn't an easy way to specify the repeated tests parameters
in the CircleCI UI. It's however possible to [trigger a
pipeline|https://support.circleci.com/hc/en-us/articles/360050351292-How-to-trigger-a-workflow-via-CircleCI-API-v2]
with a POST request specifying parameters. The problem is that pipeline
parameters should be in the final {{config.yml}} file, but {{circleci config
process}} removes them from the configuration in {{config-2_1.yml}} (see
[here|https://github.com/CircleCI-Public/circleci-cli/issues/359]), which is a
CircleCI limitation if not a bug. We could generate patches to add the repeated
tests parameters to the resource files that are generated from the config, so
we can start repeated runs with a simple http request without editing the
config files. However patching automatically generated files is probably too
risky, given that they might change depending on the used circleCI CLI. wdyt?
was (Author: adelapena):
I have added a loop-based CircleCI job for running repeated JUnit tests. That
job can be used for regular unit tests, long unit tests, jvm dtests, etc. The
Ant target is set in the env var
[{{REPEATED_UTEST_TARGET}}|https://github.com/apache/cassandra/blob/a63fc6b9dee772caa5014a543046a12a5d23eda5/.circleci/config-2_1.yml#L38-L44].
I still have to add some doc, but I think that the PR is ready for a first
round of review.
As an example, I have used the repeated test jobs to reproduce some of the
flaky tests that we have opened for 4.0:
||branch||config||run||
|CASSANDRA-16644|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16644]|[3270|https://app.circleci.com/pipelines/github/adelapena/cassandra/376/workflows/eb463f0a-ad93-488a-89a3-bb44fbc0c578/jobs/3270]|
|CASSANDRA-16627|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16627]|[3271|https://app.circleci.com/pipelines/github/adelapena/cassandra/375/workflows/1d48154f-b6eb-4d0b-bb88-fe40a8d40f27/jobs/3271]|
|CASSANDRA-16637|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16637]|[3255|https://app.circleci.com/pipelines/github/adelapena/cassandra/374/workflows/df3c4361-c6f3-4cbf-801f-dfedeef95652/jobs/3255]|
|CASSANDRA-16598|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16598]|[3260|https://app.circleci.com/pipelines/github/adelapena/cassandra/373/workflows/f1612e92-8a59-483b-bc79-d6f29a7f5766/jobs/3260]|
All the test failures reported in the tickets are reproduced. For some of the
runs the CircleCI UI is not able to properly parse and highlight the failed
test output, probably due to its large size. However, the failures can still be
seen in the downloadable test output file.
Unfortunately there isn't an easy way to specify the repeated tests parameters
in the CircleCI UI. It's however possible to [trigger a
pipeline|https://circleci.com/docs/2.0/pipeline-variables/] with a POST request
specifying parameters. The problem is that pipeline parameters should be in the
final {{config.yml}} file, but {{circleci config process}} removes them from
the configuration in {{config-2_1.yml}} (see
[here|https://github.com/CircleCI-Public/circleci-cli/issues/359]), which is a
CircleCI limitation if not a bug. We could generate patches to add the repeated
tests parameters to the resource files that are generated from the config, so
we can start repeated runs with a simple http request without editing the
config files. However patching automatically generated files is probably too
risky, given that they might change depending on the used circleCI CLI. wdyt?
> Add a CircleCI job to run some tests repeatedly
> -----------------------------------------------
>
> Key: CASSANDRA-16625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16625
> Project: Cassandra
> Issue Type: Task
> Components: CI
> Reporter: Andres de la Peña
> Assignee: Andres de la Peña
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think it could be useful to have an optional CircleCI job to run some
> specific tests n times. That way, tickets could attach CircleCI runs showing
> that the changes don't make a certain ticket flaky or, conversely, that they
> fix a flaky test. Doing this systematically should mitigate the risk of
> introducing new flaky tests, and I guess it would be more convenient and easy
> to share than running the tests locally or on a private CI system.
> It would also be nice to have something similar in Jenkins, but I'm focusing
> this ticket on CircleCI because it's available also for non-committers, so
> assignees can run their tests before setting the tickets as ready for review.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
