dcapwell commented on code in PR #2554: URL: https://github.com/apache/cassandra/pull/2554#discussion_r1289180917
########## .build/config/cassandra_ci.yaml: ########## @@ -0,0 +1,307 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all pipelines and jobs (test suites) in Apache Cassandra's CI. +# + +# CI consists of: +# 1. job: a set of commands to run against a list of files containing tests +# 2. pipeline: a list of jobs that can be run in arbitrary order +# pipelines contain a list of JDK's they have to be run across to certify correctness + +#----------------------------------------------------------------------------- +# PIPELINES +#----------------------------------------------------------------------------- +pipelines: + # All jobs in the pre-commit pipeline must run within constraints and pass + # before a commit is merged upstream. Committers are expected to validate + # and sign off on this if using non-reference CI environments. + # + # Failure to do so can lead to commits being reverted. + - name: pre-commit + jdk: + - 11 Review Comment: given this is a list, are the jobs within the pipline all run with this as a config? so this is equal to ``` for jdk in [11]: for job in jobs: job.apply(jdk=jdk) ``` ########## .build/config/cassandra_ci.yaml: ########## @@ -0,0 +1,307 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all pipelines and jobs (test suites) in Apache Cassandra's CI. +# + +# CI consists of: +# 1. job: a set of commands to run against a list of files containing tests +# 2. pipeline: a list of jobs that can be run in arbitrary order +# pipelines contain a list of JDK's they have to be run across to certify correctness + +#----------------------------------------------------------------------------- +# PIPELINES +#----------------------------------------------------------------------------- +pipelines: + # All jobs in the pre-commit pipeline must run within constraints and pass + # before a commit is merged upstream. Committers are expected to validate + # and sign off on this if using non-reference CI environments. + # + # Failure to do so can lead to commits being reverted. + - name: pre-commit + jdk: + - 11 + jobs: + - unit + - jvm-dtest + - python-dtest + - dtest + - dtest-large + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # The post-commit pipeline is a larger set of tests that include all supported JDKs. + # We expect different JDKs and variations on test suites to fail very rarely. + # + # Failures in these tests will be made visible on JIRA tickets shortly after + # test run on reference CI and committers are expected to prioritize + # rectifying any failures introduced by their work. + - name: post-commit + jdk: + - 11 + - 17 + jobs: + - unit + - unit-cdc + - compression + - test-oa + - test-system-keyspace-directory + - test-tries + - jvm-dtest + - jvm-dtest-upgrade + - dtest + - dtest-novnode + - dtest-offheap + - dtest-large + - dtest-large-novnode + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # These are longer-term, much more rarely changing pieces of infrastructure or + # testing. We expect these to fail even more rarely than post-commit. + - name: nightly + jdk: + - 11 + - 17 + jobs: + - stress-test + - fqltool-test + - test-burn + +#----------------------------------------------------------------------------- +# RESOURCE LIMITS, ALIASES, AND DEFAULT ENV VARS +#----------------------------------------------------------------------------- +# Downstream test orchestration needs to use <= the following values when running tests. +small_executor: &small_executor {cpu: 4, memory: 1g, storage: 5g} +medium_executor: &medium_executor {cpu: 4, memory: 6g, storage: 25g} +large_executor: &large_executor {cpu: 4, memory: 16g, storage: 50g} + +# Be opinionated and don't make people do ms <-> min conversions +timeout_1h: &16h_in_ms 57600000 +timeout_15m: &15m_in_ms 900000 +timeout_10m: &10m_in_ms 600000 +timeout_8m: &8m_in_ms 480000 + +# Be opinionated; discourage bespoke repeat amounts on suites where we can. +repeat_default: &repeat_default 500 +repeat_less: &repeat_less 100 +repeat_tiny: &repeat_tiny 25 + +# There are some general environment variables devs will often need to override for parameterized runs. +default_env_vars: &default_env_vars + ANT_HOME: /usr/share/ant + LANG: en_US.UTF-8 + KEEP_TEST_DIR: true + DEFAULT_DIR: /home/cassandra/cassandra-dtest + # Whether the repeated test iterations should stop on the first failure by default. + REPEATED_TESTS_STOP_ON_FAILURE: false + +#----------------------------------------------------------------------------- +# JOBS +# +# Parameters: +# job: the name +# resources: cpu: memory: storage: max allowable for the suite. +# env: +# TYPE: The type of test; this should translate into tests found under ${CASSANDRA_DIR}/test/${TYPE} in test_list +# TEST_FILTER: filter to run after test_list to narrow down tests (splits, upgrade vs. non, etc) +# TEST_TIMEOUT: We need to have the timeout at runtime to pass into ant in -Dtest.timeout=, so these are env vars in ms +# test_list: command to run in shell to generate full list of tests to run. Expected to output filename value to TEST_LIST= env var. +# repeat: Number of times to multiplex a test of the given suite on addition or change +# run: command to run in shell to execute tests +# +#----------------------------------------------------------------------------- + +#----------------------------------------------------------------------------- +# Single node JVM tests +#----------------------------------------------------------------------------- +jobs: + - &job_unit + name: unit + resources: *medium_executor + env: + <<: *default_env_vars + ANT_TEST_OPTS: -Dno-build-test=true + # TYPE lines up with the various targets in build.xml for usage by the <testclasslist> target + TYPE: unit + TEST_FILTER: "" + TEST_TIMEOUT: *8m_in_ms Review Comment: nit: I don't think we should define this here, its in `build.xml` so leave that as the source of truth... I can share code for how we made this generic in scripts in Jenkins and other environments. ########## .build/config/cassandra_ci.yaml: ########## @@ -0,0 +1,307 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all pipelines and jobs (test suites) in Apache Cassandra's CI. +# + +# CI consists of: +# 1. job: a set of commands to run against a list of files containing tests +# 2. pipeline: a list of jobs that can be run in arbitrary order +# pipelines contain a list of JDK's they have to be run across to certify correctness + +#----------------------------------------------------------------------------- +# PIPELINES +#----------------------------------------------------------------------------- +pipelines: + # All jobs in the pre-commit pipeline must run within constraints and pass + # before a commit is merged upstream. Committers are expected to validate + # and sign off on this if using non-reference CI environments. + # + # Failure to do so can lead to commits being reverted. + - name: pre-commit + jdk: + - 11 + jobs: + - unit + - jvm-dtest + - python-dtest + - dtest + - dtest-large + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # The post-commit pipeline is a larger set of tests that include all supported JDKs. + # We expect different JDKs and variations on test suites to fail very rarely. + # + # Failures in these tests will be made visible on JIRA tickets shortly after + # test run on reference CI and committers are expected to prioritize + # rectifying any failures introduced by their work. + - name: post-commit + jdk: + - 11 + - 17 + jobs: + - unit + - unit-cdc + - compression + - test-oa + - test-system-keyspace-directory + - test-tries + - jvm-dtest + - jvm-dtest-upgrade + - dtest + - dtest-novnode + - dtest-offheap + - dtest-large + - dtest-large-novnode + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # These are longer-term, much more rarely changing pieces of infrastructure or + # testing. We expect these to fail even more rarely than post-commit. + - name: nightly + jdk: + - 11 + - 17 + jobs: + - stress-test + - fqltool-test + - test-burn + +#----------------------------------------------------------------------------- +# RESOURCE LIMITS, ALIASES, AND DEFAULT ENV VARS +#----------------------------------------------------------------------------- +# Downstream test orchestration needs to use <= the following values when running tests. +small_executor: &small_executor {cpu: 4, memory: 1g, storage: 5g} +medium_executor: &medium_executor {cpu: 4, memory: 6g, storage: 25g} +large_executor: &large_executor {cpu: 4, memory: 16g, storage: 50g} + +# Be opinionated and don't make people do ms <-> min conversions +timeout_1h: &16h_in_ms 57600000 +timeout_15m: &15m_in_ms 900000 +timeout_10m: &10m_in_ms 600000 +timeout_8m: &8m_in_ms 480000 + +# Be opinionated; discourage bespoke repeat amounts on suites where we can. +repeat_default: &repeat_default 500 +repeat_less: &repeat_less 100 +repeat_tiny: &repeat_tiny 25 + +# There are some general environment variables devs will often need to override for parameterized runs. +default_env_vars: &default_env_vars + ANT_HOME: /usr/share/ant + LANG: en_US.UTF-8 + KEEP_TEST_DIR: true + DEFAULT_DIR: /home/cassandra/cassandra-dtest + # Whether the repeated test iterations should stop on the first failure by default. + REPEATED_TESTS_STOP_ON_FAILURE: false + +#----------------------------------------------------------------------------- +# JOBS +# +# Parameters: +# job: the name +# resources: cpu: memory: storage: max allowable for the suite. +# env: +# TYPE: The type of test; this should translate into tests found under ${CASSANDRA_DIR}/test/${TYPE} in test_list +# TEST_FILTER: filter to run after test_list to narrow down tests (splits, upgrade vs. non, etc) +# TEST_TIMEOUT: We need to have the timeout at runtime to pass into ant in -Dtest.timeout=, so these are env vars in ms +# test_list: command to run in shell to generate full list of tests to run. Expected to output filename value to TEST_LIST= env var. +# repeat: Number of times to multiplex a test of the given suite on addition or change +# run: command to run in shell to execute tests +# +#----------------------------------------------------------------------------- + +#----------------------------------------------------------------------------- +# Single node JVM tests +#----------------------------------------------------------------------------- +jobs: + - &job_unit + name: unit + resources: *medium_executor + env: + <<: *default_env_vars + ANT_TEST_OPTS: -Dno-build-test=true + # TYPE lines up with the various targets in build.xml for usage by the <testclasslist> target + TYPE: unit + TEST_FILTER: "" + TEST_TIMEOUT: *8m_in_ms + test_list: find "test/${TYPE}" -name "*Test.java" ${TEST_FILTER:-} | sed "s;^test/${TYPE}/;;" | sort + repeat: *repeat_default + run: ant testclasslist -Dtest.classlistprefix=${TYPE} -Dtest.timeout=${TEST_TIMEOUT} -Dtest.classlistfile=${TEST_LIST} ${ANT_TEST_OPTS} + + - <<: *job_unit + name: test-cdc + env_override: + TYPE: test-cdc + + - <<: *job_unit + name: test-compression + env_override: + TYPE: test-compression + + - <<: *job_unit + name: test-oa + env_override: + TYPE: test-oa + + - <<: *job_unit + name: test-system-keyspace-directory + env_override: + TYPE: test-system-keyspace-directory + + - <<: *job_unit + name: test-trie + env_override: + TYPE: test-trie + + - <<: *job_unit + name: microbench + resources: *small_executor + env_override: + TYPE: microbench + + - <<: *job_unit + name: fqltool-test + resources: *small_executor + env_override: + TYPE: fqltool-test + + - <<: *job_unit + name: stress-test + resources: *small_executor + env_override: + TYPE: stress-test + + - <<: *job_unit + name: long-test + env_override: + TYPE: long-test + TEST_TIMEOUT: *10m_in_ms + repeat: 100 + + - <<: *job_unit + name: cqlsh-test + run: ./pylib/cassandra-cqlsh-tests.sh $(pwd) + resources: *small_executor + env_override: + TYPE: cqlsh-test + + - <<: *job_unit + name: test-burn + resources: *large_executor + env_override: + TYPE: test-burn + TEST_TIMEOUT: *16h_in_ms + + + #----------------------------------------------------------------------------- + # JVM Multi-node tests + #----------------------------------------------------------------------------- + - <<: *job_unit + name: jvm_dtest + resources: *large_executor + env_override: + TYPE: distributed + TEST_FILTER: "| grep -v -e upgrade/ -e fuzz/" + # ms; 15 min + TEST_TIMEOUT: *15m_in_ms + repeat: 100 + + - <<: *job_unit + name: jvm-dtest-upgrade + resources: *large_executor + env_override: + TYPE: distributed + TEST_FILTER: "| grep upgrade/ -e fuzz/" + TEST_TIMEOUT: *15m_in_ms + repeat: 100 + + + #----------------------------------------------------------------------------- + # Python Dtests + #----------------------------------------------------------------------------- + - &python_dtest + name: python_dtest + resources: *large_executor + env: + <<: *default_env_vars + PYTHONIOENCODING: "utf-8" + PYTHONUNBUFFERED: true + CASS_DRIVER_NO_EXTENSIONS: true + CASS_DRIVER_NO_CYTHON: true + #Skip all syncing to disk to avoid performance issues in flaky CI environments + CASSANDRA_SKIP_SYNC: true + CCM_MAX_HEAP_SIZE: "1024M" + CCM_HEAP_NEWSIZE: "512M" + CCM_CONFIG_DIR: ${DIST_DIR}/.ccm + NUM_TOKENS: "16" + TMPDIR: "$(mktemp -d /tmp/run-python-dtest.XXXXXX)" Review Comment: I know you are trying to show an idea, but I would recommend that the commands be a script and anything that should be "tuned" is at this level and the rest treat them as internal details ########## .build/config/cassandra_ci.yaml: ########## @@ -0,0 +1,307 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all pipelines and jobs (test suites) in Apache Cassandra's CI. +# + +# CI consists of: +# 1. job: a set of commands to run against a list of files containing tests +# 2. pipeline: a list of jobs that can be run in arbitrary order +# pipelines contain a list of JDK's they have to be run across to certify correctness + +#----------------------------------------------------------------------------- +# PIPELINES +#----------------------------------------------------------------------------- +pipelines: + # All jobs in the pre-commit pipeline must run within constraints and pass + # before a commit is merged upstream. Committers are expected to validate + # and sign off on this if using non-reference CI environments. + # + # Failure to do so can lead to commits being reverted. + - name: pre-commit + jdk: + - 11 + jobs: + - unit + - jvm-dtest + - python-dtest + - dtest + - dtest-large + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # The post-commit pipeline is a larger set of tests that include all supported JDKs. + # We expect different JDKs and variations on test suites to fail very rarely. + # + # Failures in these tests will be made visible on JIRA tickets shortly after + # test run on reference CI and committers are expected to prioritize + # rectifying any failures introduced by their work. + - name: post-commit + jdk: + - 11 + - 17 + jobs: + - unit + - unit-cdc + - compression + - test-oa + - test-system-keyspace-directory + - test-tries + - jvm-dtest + - jvm-dtest-upgrade + - dtest + - dtest-novnode + - dtest-offheap + - dtest-large + - dtest-large-novnode + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # These are longer-term, much more rarely changing pieces of infrastructure or + # testing. We expect these to fail even more rarely than post-commit. + - name: nightly + jdk: + - 11 + - 17 + jobs: + - stress-test + - fqltool-test + - test-burn + +#----------------------------------------------------------------------------- +# RESOURCE LIMITS, ALIASES, AND DEFAULT ENV VARS +#----------------------------------------------------------------------------- +# Downstream test orchestration needs to use <= the following values when running tests. +small_executor: &small_executor {cpu: 4, memory: 1g, storage: 5g} +medium_executor: &medium_executor {cpu: 4, memory: 6g, storage: 25g} +large_executor: &large_executor {cpu: 4, memory: 16g, storage: 50g} + +# Be opinionated and don't make people do ms <-> min conversions +timeout_1h: &16h_in_ms 57600000 +timeout_15m: &15m_in_ms 900000 +timeout_10m: &10m_in_ms 600000 +timeout_8m: &8m_in_ms 480000 + +# Be opinionated; discourage bespoke repeat amounts on suites where we can. +repeat_default: &repeat_default 500 +repeat_less: &repeat_less 100 +repeat_tiny: &repeat_tiny 25 + +# There are some general environment variables devs will often need to override for parameterized runs. +default_env_vars: &default_env_vars + ANT_HOME: /usr/share/ant Review Comment: this can be iffy as it depends on the environment the tests are run on, but since this iteration wants to leave that out, then this kinda hard codes a very specific environment... feel we should delegate this to the runtime... "how" we document this is a requirement? no idea! ########## .build/config/cassandra_ci.yaml: ########## @@ -0,0 +1,307 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains definitions of all pipelines and jobs (test suites) in Apache Cassandra's CI. +# + +# CI consists of: +# 1. job: a set of commands to run against a list of files containing tests +# 2. pipeline: a list of jobs that can be run in arbitrary order +# pipelines contain a list of JDK's they have to be run across to certify correctness + +#----------------------------------------------------------------------------- +# PIPELINES +#----------------------------------------------------------------------------- +pipelines: + # All jobs in the pre-commit pipeline must run within constraints and pass + # before a commit is merged upstream. Committers are expected to validate + # and sign off on this if using non-reference CI environments. + # + # Failure to do so can lead to commits being reverted. + - name: pre-commit + jdk: + - 11 + jobs: + - unit + - jvm-dtest + - python-dtest + - dtest + - dtest-large + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # The post-commit pipeline is a larger set of tests that include all supported JDKs. + # We expect different JDKs and variations on test suites to fail very rarely. + # + # Failures in these tests will be made visible on JIRA tickets shortly after + # test run on reference CI and committers are expected to prioritize + # rectifying any failures introduced by their work. + - name: post-commit + jdk: + - 11 + - 17 + jobs: + - unit + - unit-cdc + - compression + - test-oa + - test-system-keyspace-directory + - test-tries + - jvm-dtest + - jvm-dtest-upgrade + - dtest + - dtest-novnode + - dtest-offheap + - dtest-large + - dtest-large-novnode + - dtest-upgrade + - dtest-upgrade-large + - long-test + - cqlsh-test + + # These are longer-term, much more rarely changing pieces of infrastructure or + # testing. We expect these to fail even more rarely than post-commit. + - name: nightly + jdk: + - 11 + - 17 + jobs: + - stress-test + - fqltool-test + - test-burn + +#----------------------------------------------------------------------------- +# RESOURCE LIMITS, ALIASES, AND DEFAULT ENV VARS +#----------------------------------------------------------------------------- +# Downstream test orchestration needs to use <= the following values when running tests. +small_executor: &small_executor {cpu: 4, memory: 1g, storage: 5g} +medium_executor: &medium_executor {cpu: 4, memory: 6g, storage: 25g} +large_executor: &large_executor {cpu: 4, memory: 16g, storage: 50g} + +# Be opinionated and don't make people do ms <-> min conversions +timeout_1h: &16h_in_ms 57600000 +timeout_15m: &15m_in_ms 900000 +timeout_10m: &10m_in_ms 600000 +timeout_8m: &8m_in_ms 480000 + +# Be opinionated; discourage bespoke repeat amounts on suites where we can. +repeat_default: &repeat_default 500 +repeat_less: &repeat_less 100 +repeat_tiny: &repeat_tiny 25 + +# There are some general environment variables devs will often need to override for parameterized runs. +default_env_vars: &default_env_vars + ANT_HOME: /usr/share/ant + LANG: en_US.UTF-8 + KEEP_TEST_DIR: true + DEFAULT_DIR: /home/cassandra/cassandra-dtest + # Whether the repeated test iterations should stop on the first failure by default. + REPEATED_TESTS_STOP_ON_FAILURE: false + +#----------------------------------------------------------------------------- +# JOBS +# +# Parameters: +# job: the name +# resources: cpu: memory: storage: max allowable for the suite. +# env: +# TYPE: The type of test; this should translate into tests found under ${CASSANDRA_DIR}/test/${TYPE} in test_list +# TEST_FILTER: filter to run after test_list to narrow down tests (splits, upgrade vs. non, etc) +# TEST_TIMEOUT: We need to have the timeout at runtime to pass into ant in -Dtest.timeout=, so these are env vars in ms +# test_list: command to run in shell to generate full list of tests to run. Expected to output filename value to TEST_LIST= env var. +# repeat: Number of times to multiplex a test of the given suite on addition or change +# run: command to run in shell to execute tests +# +#----------------------------------------------------------------------------- + +#----------------------------------------------------------------------------- +# Single node JVM tests +#----------------------------------------------------------------------------- +jobs: + - &job_unit + name: unit + resources: *medium_executor + env: + <<: *default_env_vars + ANT_TEST_OPTS: -Dno-build-test=true + # TYPE lines up with the various targets in build.xml for usage by the <testclasslist> target + TYPE: unit + TEST_FILTER: "" + TEST_TIMEOUT: *8m_in_ms + test_list: find "test/${TYPE}" -name "*Test.java" ${TEST_FILTER:-} | sed "s;^test/${TYPE}/;;" | sort + repeat: *repeat_default Review Comment: I don't think this should be top level, `repeat` is not a top level thing IMO, it leverages primitives defined. I would rework how repeat jobs are done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
