[
https://issues.apache.org/jira/browse/CASSANDRA-17939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613054#comment-17613054
]
Andres de la Peña edited comment on CASSANDRA-17939 at 10/5/22 4:21 PM:
------------------------------------------------------------------------
Although the new jobs will run any new or modified {{*Test.java}} file, in some
cases we'll still need to manually run tests that are affected by the changes.
We can do that with the classic test multiplexer. This however can be awkward
because the multiplexer jobs only run a class at a time, so we'd need multiple
pushes if we want to run multiple not-new-nor-modified classes.
CASSANDRA-17938 aims to solve that by allowing to specify multiple classes in
the classic multiplexer. However, I think that we can run those sets of
not-new-nor-modified classes in the same companion classes that we are
proposing here. For that, we can have a env property per companion job allowing
to specify what test classes we want to run in addition to the automatically
detected new-or-modified tests. The list of env vars for manually specifying
the tests to be repeated would be:
{code:java}
REPEATED_UTESTS_CLASSES:
REPEATED_UTESTS_FQLTOOL_CLASSES:
REPEATED_UTESTS_LONG_CLASSES:
REPEATED_UTESTS_STRESS_CLASSES:
REPEATED_SIMULATOR_DTESTS_CLASSES:
REPEATED_JVM_DTESTS_CLASSES:
REPEATED_JVM_UPGRADE_DTESTS_CLASSES:
{code}
We could pass those variables to the {{generate.sh}} script, for example:
{code:java}
.circleci/generate.sh -m \
-e REPEATED_TESTS_COUNT=5 \
-e
REPEATED_UTESTS_CLASSES=org.apache.cassandra.cql3.ListsTest,org.apache.cassandra.cql3.validation.entities.DateTypeTest
\
-e REPEATED_UTESTS_COUNT=10 \
-e
REPEATED_JVM_DTESTS_CLASSES=org.apache.cassandra.distributed.test.AlterTest \
-e REPEATED_JVM_DTESTS_COUNT=10 \
-e
REPEATED_UTESTS_LONG_CLASSES=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest
\
-e REPEATED_UTESTS_LONG_COUNT=2
{code}
Those manually specified tests would be run together with the automatically
detected tests. For example, the above command with the [synthetic patch from
my previous
comment|https://github.com/adelapena/cassandra/commit/b8ec11dac7b18288c297ad4412476ab4dc7a50fe]
would produce this run:
||Patch||CI||
|[trunk|https://github.com/adelapena/cassandra/tree/17939-trunk-classes-test]|[j8_pre-commit|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/19d6741b-72f1-43d3-88a2-556b17ca1531]
[j11_pre-commit|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/43fe94a0-213e-41b0-b0bc-1db368e81bda]
[j8_separate|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/47fe1bf9-6454-41f0-89e5-4a18a38c5bc1]
[j11_separate|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/557e0782-1b2e-4ef8-b24d-60082be1b8ac]|
We can see in the [artifacts
tabs|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/47fe1bf9-6454-41f0-89e5-4a18a38c5bc1/jobs/21565/artifacts]
that the jobs are repeatedly running both the automatically detected tests and
the manually specified tests.
If we adopt this approach, the classic utest multiplexer would only be used for
debugging existing flakies, as originally intended, and the new jobs would be
what we use in regular tickets to make sure we are not introducing new flakies.
Indeed, we could remove the classic utest multiplexer from the pre-commit
workflows and leave it only in the separate workflows.
was (Author: adelapena):
Although the new jobs will run any new or modified {{*Test.java}} file, in some
cases we'll still need to manually run tests that are affected by the changes.
We can do that with the classic test multiplexer. This however can be awkward
because the multiplexer jobs only run a class at a time, so we'd need multiple
pushes if we want to run multiple not-new-nor-modified classes.
CASSANDRA-17938 aims to solve that by allowing to specify multiple classes in
the classic multiplexer. However, I think that we can run those sets of
not-new-nor-modified classes in the same companion classes that we are
proposing here. For that, we can have a env property per companion job allowing
to specify what test classes we want to run in addition to the automatically
detected new-or-modified tests. The list of env vars for manually specifying
the tests to be repated would be:
{code:java}
REPEATED_UTESTS_CLASSES:
REPEATED_UTESTS_FQLTOOL_CLASSES:
REPEATED_UTESTS_LONG_CLASSES:
REPEATED_UTESTS_STRESS_CLASSES:
REPEATED_SIMULATOR_DTESTS_CLASSES:
REPEATED_JVM_DTESTS_CLASSES:
REPEATED_JVM_UPGRADE_DTESTS_CLASSES:
{code}
We could pass those variables to the {{generate.sh}} script, for example:
{code:java}
.circleci/generate.sh -m \
-e REPEATED_TESTS_COUNT=5 \
-e
REPEATED_UTESTS_CLASSES=org.apache.cassandra.cql3.ListsTest,org.apache.cassandra.cql3.validation.entities.DateTypeTest
\
-e REPEATED_UTESTS_COUNT=10 \
-e
REPEATED_JVM_DTESTS_CLASSES=org.apache.cassandra.distributed.test.AlterTest \
-e REPEATED_JVM_DTESTS_COUNT=10 \
-e
REPEATED_UTESTS_LONG_CLASSES=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest
\
-e REPEATED_UTESTS_LONG_COUNT=2
{code}
Those manually specified tests would be run together with the automatically
detected tests. For example, the above command with the [synthetic patch from
my previous
comment|https://github.com/adelapena/cassandra/commit/b8ec11dac7b18288c297ad4412476ab4dc7a50fe]
would produce this run:
||Patch||CI||
|[trunk|https://github.com/adelapena/cassandra/tree/17939-trunk-classes-test]|[j8_pre-commit|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/19d6741b-72f1-43d3-88a2-556b17ca1531]
[j11_pre-commit|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/43fe94a0-213e-41b0-b0bc-1db368e81bda]
[j8_separate|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/47fe1bf9-6454-41f0-89e5-4a18a38c5bc1]
[j11_separate|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/557e0782-1b2e-4ef8-b24d-60082be1b8ac]|
We can see in the [artifacts
tabs|https://app.circleci.com/pipelines/github/adelapena/cassandra/2143/workflows/47fe1bf9-6454-41f0-89e5-4a18a38c5bc1/jobs/21565/artifacts]
that the jobs are repeatedly running both the automatically detected tests and
the manually specified tests.
If we adopt this approach, the classic utest multiplexer would only be used for
debugging existing flakies, as originally intended, and the new jobs would be
what we use in regular tickets to make sure we are not introducing new flakies.
> CircleCI: Automatically detect and repeat new or modified JUnit tests
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-17939
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17939
> Project: Cassandra
> Issue Type: Task
> Components: CI
> Reporter: Andres de la Peña
> Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The purpose of this ticket is adding a new CircleCI job that automatically
> detects new or modified test classes and runs them repeatedly. That way we
> wouldn't need to manually specify those tests with {{.circleci/generate.sh}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
