driazati commented on code in PR #11927: URL: https://github.com/apache/tvm/pull/11927#discussion_r908646634
########## docs/contribute/ci.rst: ########## @@ -23,14 +23,21 @@ Using TVM's CI .. contents:: :local: -TVM uses Jenkins for running Linux continuous integration (CI) tests on -`branches <https://ci.tlcpack.ai/job/tvm/>`_ and +TVM primarily uses Jenkins for running Linux continuous integration (CI) tests on +`branches <https://ci.tlcpack.ai/job/tvm/>`_ `pull requests <https://ci.tlcpack.ai/job/tvm/view/change-requests/>`_ through a build configuration specified in a `Jenkinsfile <https://github.com/apache/tvm/blob/main/Jenkinsfile>`_. -Non-critical jobs run in GitHub Actions for Windows and MacOS jobs. +Jenkins is the only CI step that is codified to block merging. TVM is also tested minimally +against Windows and MacOS using GitHub Actions. + +This page describes how contributors and committers can use TVM's CI to verify their code. You can +read more about the design of TVM CI in the Review Comment: link to repo? ########## ci/jenkins/README.md: ########## @@ -33,17 +37,20 @@ https://github.com/apache/tvm/actions has the logs for each of these workflows. ## Keeping CI Green -Developers rely on the TVM CI to get signal on their PRs before merging. -Occasionally breakages slip through and break `main`, which in turn causes -the same error to show up on an PR that is based on the broken commit(s). Broken -commits can be identified [through GitHub](https://github.com/apache/tvm/commits/main>) -via the commit status icon or via [Jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/activity?branch=main>). -In these situations it is possible to either revert the offending commit or -submit a forward fix to address the issue. It is up to the committer and commit -author which option to choose, keeping in mind that a broken CI affects all TVM -developers and should be fixed as soon as possible. +Developers rely on the TVM CI to get signal on their PRs before merging. Occasionally breakages +slip through and break `main`, which in turn causes the same error to show up on an unrelated PR +that is based on the broken commit(s). Broken commits can be identified [through +GitHub](https://github.com/apache/tvm/commits/main>) via the commit status icon or via +[Jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/activity?branch=main>). In these +situations it is possible to either revert the offending commit or submit a forward fix to address +the issue. It is up to the committer and commit author which option to choose. A broken CI affects +all TVM developers and should be fixed as soon as possible, while a revert may be especially painful +for the author of the offending PR when that PR is large. -Some tests are also flaky and fail for reasons unrelated to the PR. The [CI monitoring rotation](https://github.com/apache/tvm/wiki/CI-Monitoring-Runbook) watches for these failures and disables tests as necessary. It is the responsibility of those who wrote the test to ultimately fix and re-enable the test. +Some tests are also flaky and occasionally fail for reasons unrelated to the PR. The [CI monitoring +rotation](https://github.com/apache/tvm/wiki/CI-Monitoring-Runbook) watches for these failures and +disables tests as necessary. It is the responsibility of those who wrote the test to ultimately fix +and re-enable the test. ## Dealing with Flakiness Review Comment: This stuff should be either in .rst or in .md, can you delete one or the other? ########## ci/jenkins/README.md: ########## @@ -153,6 +160,7 @@ _venv/bin/python3 jenkins/generate.py # Infrastructure +While all TVM tests are contained within the apache/tvm repository, the infrastructure used to run the tests is donated by the TVM Community. To encourage collaboration, Jenkins runs in AWS on an EC2 instance fronted by an ELB which makes it available at https://ci.tlcpack.ai. These definitions are declared via Terraform in the [tlc-pack/ci-terraform](https://github.com/tlc-pack/ci-terraform) repository. The Terraform code references custom AMIs built in [tlc-pack/ci-packer](https://github.com/tlc-pack/ci-packer). [tlc-pack/ci](https://github.com/tlc-pack/ci) contains Ansible scripts to deploy the Jenkins head node and set it up to interact with AWS. The Jenkins head node has a number of autoscaling groups with labels that are used to run jobs (e.g. `CPU`, `GPU` or `ARM`) via the [EC2 Fleet](https://plugins.jenkins.io/ec2-fleet/) plugin. Review Comment: the tlcpack stuff is merged now, can you delete the diagram from here? ########## ci/jenkins/README.md: ########## @@ -153,6 +160,7 @@ _venv/bin/python3 jenkins/generate.py # Infrastructure +While all TVM tests are contained within the apache/tvm repository, the infrastructure used to run the tests is donated by the TVM Community. To encourage collaboration, Review Comment: the sentence is missing ########## ci/README.md: ########## @@ -0,0 +1,97 @@ +<!--- Licensed to the Apache Software Foundation (ASF) under one --> +<!--- or more contributor license agreements. See the NOTICE file --> +<!--- distributed with this work for additional information --> +<!--- regarding copyright ownership. The ASF licenses this file --> +<!--- to you under the Apache License, Version 2.0 (the --> +<!--- "License"); you may not use this file except in compliance --> +<!--- with the License. You may obtain a copy of the License at --> + +<!--- http://www.apache.org/licenses/LICENSE-2.0 --> + +<!--- Unless required by applicable law or agreed to in writing, --> +<!--- software distributed under the License is distributed on an --> +<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --> +<!--- KIND, either express or implied. See the License for the --> +<!--- specific language governing permissions and limitations --> +<!--- under the License. --> + +# Apache TVM Continuous Integration (CI) + +## Overview + +TVM's Continuous Integration is responsible for verifying the code in `apache/tvm` and testing PRs +before they merge to inform TVM contributors and committers. These jobs are essential to keeping the +TVM project in a healthy state and preventing breakages. CI in TVM is broken into these pieces: + - Lint scripts in [`tests/lint`](../tests/lint). + - The tests themselves, all of which live underneath [`tests`](../tests). + - Definitions of test suites, with each suite defined as a separate `task_` script in + [`tests/scripts`](../tests/scripts). + - The linux test sequence (in [`Jenkinsfile`](../Jenkinsfile)), which lints and builds TVM and runs test + suites using Docker on Linux. + - The Windows and Mac test sequences (in [`.github/actions`](../.github/actions)). + - GitHub Actions that support the code review process (in [`.github/actions`](../.github/actions)). + - Tools to reproduce the CI locally (in `tests/scripts`). + - Infrastructure-as-Code that configures the cloud services that provide Jenkins for the TVM CI (in the + [`tlc-pack/ci`](https://github.com/tlc-pack/ci) repo). + +## CI Documentation Index + +The CI documentation belongs with the implementation it describes. To make that concrete, the +documentation is split like so: +1. An overview of the CI is in this file. +1. User-facing documentation lives in `apache/tvm`'s `docs/contribute` sub-directory and is served on the + [TVM docs site](https://tvm.apache.org/docs/contribute/ci.html). +2. Documentation of the tools that run TVM's various regression tests locally and the test suites + are in this sub-directory. +3. Documentation of the cloud services and their configuration lives in the + [`tlc-pack/ci`](https://github.com/tlc-pack/ci) repo. + +## Jenkins + +Jenkins runs all of the linux-based TVM CI-enabled regression tests. This includes tests against accelerated hardware such as GPUs. It excludes those regression tests that run against hardware not available in the cloud (those tests aren't currently exercised in TVM CI). The tests run by Jenkins represent most of the merge-blocking tests (and passing Jenkins should mostly correlate with passing the remaining Windows/Mac builds). Review Comment: ```suggestion Jenkins runs all of the Linux-based TVM CI-enabled regression tests. This includes tests against accelerated hardware such as GPUs. It excludes those regression tests that run against hardware not available in the cloud (those tests aren't currently exercised in TVM CI). The tests run by Jenkins represent most of the merge-blocking tests (and passing Jenkins should mostly correlate with passing the remaining Windows/Mac builds). ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
