This is an automated email from the ASF dual-hosted git repository.
mousius pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm-rfcs.git
The following commit(s) were added to refs/heads/main by this push:
new 41e5ba0 [RFC][Roadmap] TVM Continuous Integration & Testing Roadmap
(#54)
41e5ba0 is described below
commit 41e5ba078b830299692c844e5ab3e5193d3d6129
Author: Denise Kutnick <[email protected]>
AuthorDate: Wed Feb 9 05:01:13 2022 -0500
[RFC][Roadmap] TVM Continuous Integration & Testing Roadmap (#54)
---
rfcs/0054-ci-testing-roadmap.md | 124 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 124 insertions(+)
diff --git a/rfcs/0054-ci-testing-roadmap.md b/rfcs/0054-ci-testing-roadmap.md
new file mode 100644
index 0000000..bf235b1
--- /dev/null
+++ b/rfcs/0054-ci-testing-roadmap.md
@@ -0,0 +1,124 @@
+- Feature Name: TVM CI & Testing Roadmap
+- Start Date: 2022-01-19
+- RFC PR: [apache/tvm-rfcs#0054](https://github.com/apache/tvm-rfcs/pull/0054)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+- Co-Authors: Denise Kutnick ([**@denise-k**](https://github.com/denise-k)),
Andrew Reusch
+ ([**@areusch**](https://github.com/areusch)), Leandro Nunes
([**@leandron**](https://github.com/leandron))
+
+# [RFC] TVM Continuous Integration & Testing Roadmap
+
+## Background and Context
+
+**Roadmap Name:** TVM Continuous Integration & Testing Roadmap
+
+**Roadmap Maintainers: `@areusch, @denise-k, @leandron` (add more here)**
+
+**Roadmap Description:** This roadmap tracks all efforts related to Continuous
Integration and Testing within TVM.
+
+## Scope
+
+**How are the tasks tracked in this roadmap grouped together? How can we think
about this grouping distinct from those made in other roadmaps?**
+
+Tasks in this roadmap may be grouped together using **vertical** or
**horizontal** tasks.
+* **Horizontal Tasks:** CI/Testing tasks which have effects on all of TVM are
considered horizontally-grouped tasks. For instance, if a task existed to
[parameterize all of TVM's unit
tests](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0007-parametrized-unit-tests.md),
this would be considered a **horizontal task**.
+* **Vertical Tasks:** CI/Testing tasks which are concentrated within a single
subproject of TVM are considered vertically-grouped tasks. For instance, adding
[Vulkan unit tests to TVM's CI](https://github.com/apache/tvm/pull/9093) is
considered a **vertical task**. Multiple vertical tasks in the same domain may
be grouped together into a larger roadmap item as needed. For example, the
aforementioned Vulkan example could be grouped into a roadmap item tracking all
Vulkan-related CI patches.
+
+**Is the proposed roadmap intended to represent a perpetually ongoing set of
efforts, or is there an end goal which will close/finalize the roadmap?**
+
+The CI & Testing roadmap represents a perpetually ongoing set of efforts
within TVM.
+
+**Does the proposed roadmap have any scope overlaps with any existing
roadmaps? If so, please list them.**
+
+This roadmap is intended to provide a unified view of all CI and Testing
efforts horizontally across TVM. Other roadmaps may vertically track certain
areas of TVM (e.g. MicroTVM) and may share CI/Testing tasks with this roadmap.
+
+## Themes
+
+**List 4-6 proposed “themes” of the roadmap, intended to convey the purpose of
the roadmap and the types of tasks that should be added.**
+
+- **For each theme, include a set of definitions specific to the proposed
roadmap.**
+- **For each theme, include a set of success criteria specific to the proposed
roadmap.**
+
+### Coverage
+
+#### Definition
+
+Test coverage can be defined in multiple ways:
+
+- Coverage of the matrix of end-to-end flows on TVM (from all different types
of frontends to all different types of hardware)
+- Operator Coverage (for all the different parameters that can be configured
for an operator)
+- API coverage (for user-facing APIs, internal APIs, and FFI)
+
+#### Goals and Success Criteria
+
+Maximize coverage such that any issues introduced to the code are caught
preemptively, rather than having users encounter and report bugs that could've
been caught with proper test coverage.
+
+### Ease of Use
+
+#### Definition
+
+Ease of use applies to both local testing and the CI environment.
+
+#### Goals and Success Criteria
+
+- Being able to easily run tests on a local machine
+- Being able to replicate the behavior/dependencies of CI on a local machine
+- Being able to maintain and update dependencies on CI machines in a
sustainable way
+
+### Latency
+
+#### Definition
+
+CI latency is the combination of build time, run time and wait time for
individual CI jobs. This includes:
+
+- How long one must wait for CI jobs to start
+- How long one must wait for TVM and various dependencies to build
+- How long one must wait for nodes to become available
+- How long one must wait to get test results once a CI job has started
+
+#### Goals & Success Criteria
+
+The goal of CI latency work is to continuously reduce the overall latency, and
to make sure that latency stays low as new features and tests are added.
+
+The success of CI latency work can be measured by % reductions in such latency
metrics over time.
+
+### Reporting
+
+#### Definition
+
+CI reporting can be defined in two parts:
+
+- **Infrastructure**: any software, services, or hardware used to generate
reporting artifacts
+- **Artifacts**: any raw data, metrics, and charts generated as an output from
CI runs
+
+#### Goals & Success Criteria
+
+Build up reporting infrastructure to generate useful artifacts for TVM users
and developers.
+
+### Stability
+
+#### Definition
+
+CI stability is defined by the overall availability and uptime of CI machines.
+
+#### Goals & Success Criteria
+
+- 100% uptime on baseline CI nodes
+- Fallback mechanism for CI node failure
+- Documented postmortem protocol for CI node failure
+
+### Security
+
+#### Definition
+
+Security in the context of TVM CI & Testing can be defined in two different
ways:
+
+- **TVM Security**: This is the security of TVM's codebase as a whole.
+- **CI Security**: This is the security of TVM's CI pipelines and machines.
+
+This roadmap will cover **CI Security**. **TVM Security** will be covered
separately in another roadmap.
+
+#### Goals & Success Criteria
+
+- System-level security (e.g. operating system updates) should be regularly
maintained on TVM's CI & Build machines, ensuring CI Security.
+- TVM's external dependencies should be updated on CI & Build machines in a
timely manner, so that any security patches within those dependencies are
installed, ensuring CI Security.
+- TVM's CI & Testing pipelines should automatically monitor the TVM codebase
and alert users if any security issues are found.
