This is an automated email from the ASF dual-hosted git repository.
tqchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new 0a5e910479 fix(infra): update tir-test for the in-tree TIRx split
(#20748)
0a5e910479 is described below
commit 0a5e9104797d02d4388b394be22d441444fed9c8
Author: Bohan Hou <[email protected]>
AuthorDate: Thu Oct 15 17:46:40 2026 -0400
fix(infra): update tir-test for the in-tree TIRx split (#20748)
After the in-tree TIRx split, compiler tests live in
`tirx/tests/python/tirx/`, while downstream registry correctness
coverage remains in
`tests/python/tirx/test_tirx_kernels_registry_correctness.py`. The
`tir-test` workflow still uses the old compiler package and test
location.
Load this worktree's `tvm_tirx` package and native libraries, preserve
the inherited `PYTHONPATH`, and run both test locations. Keep both
tirx-kernels import gates and registry correctness checks enabled by
default, with failure propagation and complete pytest logs.
`TIR_TEST_WITH_KERNELS=0` explicitly selects compiler-only coverage.
Handle `ProcessLookupError` when a terminating process disappears during
the IKET cleanup test's `/proc` read. The complete workflow exposed this
race; the new deterministic regression reproduces it against the
mainline helper.
Paired with the downstream namespace migration in
[mlc-ai/TIRx-kernels#12](https://github.com/mlc-ai/TIRx-kernels/pull/12).
Validation:
- Both downstream import gates passed for all 93 kernels.
- Combined compiler and downstream collection after rebasing onto main:
4,118 tests collected successfully.
- Paired tirx-kernels infrastructure and API regressions: 210 passed;
exported-host isolation checks: 4 passed.
- Full combined run after rebase on two B200 GPUs with eight xdist
workers: 3,804 passed, 299 skipped, 5 xfailed, and 10 CUDA out-of-memory
failures in 657.97 seconds. The failure logs show concurrent GPU
allocations of approximately 155 GiB.
- All 10 out-of-memory cases passed in a serial retry on GPU 3 in 35.80
seconds, with the same code and no test exclusions. This validates 3,814
passing cases across the full run and retry.
- All 32 IKET orchestration tests passed, including both new
process-exit regressions; the complete combined run also passed this
coverage.
- Bash syntax, skill validation, and pre-commit checks passed.
---
.agents/skills/tir-test/SKILL.md | 124 ++++++++++++++++------
tirx/tests/python/tirx/iket/test_orchestration.py | 13 ++-
2 files changed, 105 insertions(+), 32 deletions(-)
diff --git a/.agents/skills/tir-test/SKILL.md b/.agents/skills/tir-test/SKILL.md
index e862e6c40f..38e6aee623 100644
--- a/.agents/skills/tir-test/SKILL.md
+++ b/.agents/skills/tir-test/SKILL.md
@@ -1,17 +1,39 @@
-Run the full TIRX test suite.
+---
+name: tir-test
+description: Run TVM's in-tree TIRx tests and tirx-kernels import and
correctness checks.
+---
+
+Run the compiler suite in `tirx/tests/python/tirx/`, using the `tvm_tirx`
package
+in `tirx/python/`, together with the downstream correctness test in
+`tests/python/tirx/test_tirx_kernels_registry_correctness.py`.
+
+Both tirx-kernels import gates and its correctness coverage are enabled by
+default. Use `TIR_TEST_WITH_KERNELS=0` only for an explicitly requested
+compiler-only run, and report that reduced coverage.
## Steps
Run all commands below in the same Bash shell from the TVM repository root.
-1. Point Python at this workspace's repos and select every Blackwell GPU.
+1. Point Python at this worktree and select every Blackwell GPU.
`TIR_TEST_GPUS` may explicitly provide physical GPU IDs, and
`TIR_TEST_NUM_GPUS` may optionally cap the automatically selected count.
Otherwise,
selection uses all Blackwell GPUs, ordered by least memory in use.
```bash
- export WORKSPACE=/path/to/workspace
+ export TIR_TEST_TVM_ROOT="$(pwd -P)"
export TIR_TEST_NUM_GPUS="${TIR_TEST_NUM_GPUS:-}"
export TIR_TEST_XDIST_WORKERS="${TIR_TEST_XDIST_WORKERS:-16}"
+ export TIR_TEST_WITH_KERNELS="${TIR_TEST_WITH_KERNELS:-1}"
+
+ [[ -d "$TIR_TEST_TVM_ROOT/tirx/python/tvm_tirx" &&
+ -d "$TIR_TEST_TVM_ROOT/tirx/tests/python/tirx" ]] || {
+ echo "run from the TVM repository root containing tirx/" >&2
+ exit 2
+ }
+ [[ "$TIR_TEST_WITH_KERNELS" == 0 || "$TIR_TEST_WITH_KERNELS" == 1 ]] || {
+ echo "TIR_TEST_WITH_KERNELS must be 0 or 1" >&2
+ exit 2
+ }
[[ -z "$TIR_TEST_NUM_GPUS" || "$TIR_TEST_NUM_GPUS" =~ ^[1-9][0-9]*$ ]] || {
echo "TIR_TEST_NUM_GPUS must be empty or a positive integer" >&2
@@ -66,11 +88,10 @@ Run all commands below in the same Bash shell from the TVM
repository root.
export CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES="$(IFS=,; printf '%s' "${TIR_TEST_SELECTED_GPUS[*]}")"
- # Registry correctness tests skip a device with less than this much free
- # memory; 0 runs them regardless of what else is on the GPU.
- export TIRX_KERNEL_TEST_MIN_FREE_GB=0
- export PYTHONPATH="${WORKSPACE}/tirx-kernels:${WORKSPACE}/tvm/python"
- export TVM_LIBRARY_PATH="${WORKSPACE}/tvm/build/lib"
+ # Preserve reference packages and other paths supplied by branch-shell.
+ export
PYTHONPATH="${TIR_TEST_TVM_ROOT}/.local/python:${TIR_TEST_TVM_ROOT}/tirx/python:${TIR_TEST_TVM_ROOT}/python:${PYTHONPATH:-}"
+ export TVM_LIBRARY_PATH="${TIR_TEST_TVM_ROOT}/build/lib"
+ export TVM_TIRX_LIBRARY_PATH="${TIR_TEST_TVM_ROOT}/build/lib"
PYTHON_BIN="${PYTHON_BIN:-$(command -v python || true)}"
[[ -n "$PYTHON_BIN" && -x "$PYTHON_BIN" ]] || {
@@ -78,10 +99,33 @@ Run all commands below in the same Bash shell from the TVM
repository root.
exit 1
}
export PATH="$(dirname "$PYTHON_BIN"):$PATH"
- "$PYTHON_BIN" -c 'import pytest, torch, tvm' >/dev/null || {
- echo "PYTHON_BIN must provide pytest and torch and import this TVM
worktree" >&2
+ "$PYTHON_BIN" - <<'PY'
+ import os
+ from pathlib import Path
+
+ import pytest
+ import torch
+ import tvm_ffi
+ import tvm_tirx
+ import xdist
+ from tvm_tirx.base import _LOADED_LIBS, _RUNTIME_ONLY
+
+ root = Path(os.environ["TIR_TEST_TVM_ROOT"])
+ assert Path(tvm_tirx.__file__).resolve().is_relative_to(root /
"tirx/python"), (
+ f"imported another TIRx worktree: {tvm_tirx.__file__}"
+ )
+ assert not _RUNTIME_ONLY, "build the TIRx compiler before running its tests"
+ for name, library in _LOADED_LIBS.items():
+ assert Path(library._name).resolve().is_relative_to(root /
"build/lib"), (
+ f"loaded another worktree's {name}: {library._name}"
+ )
+ print(f"tvm_tirx: {tvm_tirx.__file__}")
+ print(f"tvm_ffi: {tvm_ffi.__file__}")
+ PY
+ if [[ $? -ne 0 ]]; then
+ echo "PYTHON_BIN must provide pytest, xdist, torch and this worktree's
TIRx build" >&2
exit 1
- }
+ fi
if ! command -v ninja >/dev/null; then
NINJA_BIN_DIR="$("$PYTHON_BIN" -c 'import ninja; print(ninja.BIN_DIR)'
2>/dev/null || true)"
if [[ -n "$NINJA_BIN_DIR" && -x "$NINJA_BIN_DIR/ninja" ]]; then
@@ -103,33 +147,50 @@ Run all commands below in the same Bash shell from the
TVM repository root.
`TIR_TEST_NUM_GPUS=4` to cap the count or `TIR_TEST_GPUS=3,4,5,6` to
override
selection with exact physical IDs.
-2. Import gate: bench workloads. Fail fast if any kernel listed in
- `workloads.yaml` fails to import.
- ```bash
- "$PYTHON_BIN" -m tirx_kernels.bench_suite --check-imports
- ```
- A non-zero exit means a pinned workload kernel failed to import. Fix it
before
- proceeding.
+ Use dependencies compatible with `tirx/pyproject.toml`. A non-editable FFI
+ installation in `.local/python/` takes priority over the inherited
environment.
+ If native libraries are missing or stale, build this worktree with the same
+ `PYTHONPATH` before testing.
-3. Full kernel import gate for correctness-suite coverage:
+2. Run downstream import gates and select the full test coverage:
```bash
- "$PYTHON_BIN" -m tirx_kernels.registry --strict
+ TIR_TEST_PATHS=(tirx/tests/python/tirx/)
+ if [[ "$TIR_TEST_WITH_KERNELS" == 1 ]]; then
+ TIR_TEST_KERNELS_ROOT="${TIR_TEST_KERNELS_ROOT:-$(dirname
"$TIR_TEST_TVM_ROOT")/tirx-kernels}"
+ [[ -d "$TIR_TEST_KERNELS_ROOT/tirx_kernels" ]] || {
+ echo "set TIR_TEST_KERNELS_ROOT to the tirx-kernels repository" >&2
+ exit 2
+ }
+ export PYTHONPATH="${TIR_TEST_KERNELS_ROOT}:${PYTHONPATH:-}"
+ "$PYTHON_BIN" -m tirx_kernels.bench_suite --check-imports || exit 1
+ "$PYTHON_BIN" -m tirx_kernels.registry --strict || exit 1
+
TIR_TEST_PATHS+=(tests/python/tirx/test_tirx_kernels_registry_correctness.py)
+ export TIRX_KERNEL_TEST_MIN_FREE_GB="${TIRX_KERNEL_TEST_MIN_FREE_GB:-0}"
+ else
+ echo "compiler-only run: tirx-kernels import and correctness checks
disabled"
+ fi
```
-4. Run the full test suite with xdist parallelism:
+3. Run the selected test suites with xdist parallelism and save the full log:
```bash
- "$PYTHON_BIN" -m pytest tests/python/tirx/ -n "$TIR_TEST_XDIST_WORKERS"
+ export TIR_TEST_LOG="${TIR_TEST_LOG:-/tmp/tir-test-$(date
+%Y%m%d-%H%M%S).log}"
+ set -o pipefail
+ "$PYTHON_BIN" -m pytest "${TIR_TEST_PATHS[@]}" -n "$TIR_TEST_XDIST_WORKERS"
2>&1 | tee "$TIR_TEST_LOG"
```
- `tests/python/tirx/conftest.py` assigns each xdist worker's PyTorch current
- device round-robin across the visible GPUs. Registry correctness tests also
- take a per-device lock, so at most one large registered kernel runs on each
- GPU concurrently. Tests that explicitly use `tvm.cuda(0)` still run on the
- first selected physical GPU. MegaMoE remains skipped because it requires its
- dedicated multi-process scheduler, not ordinary xdist device assignment.
+ `tirx/tests/python/tirx/conftest.py` uses `tvm_tirx.testing._pytest` to
assign
+ each xdist worker's PyTorch current device round-robin across the visible
+ GPUs. Tests that explicitly use `tvm_tirx.cuda(0)` still run on the first
+ selected physical GPU.
+
+ Registry correctness tests take a per-device lock to serialize large
workloads
+ on each GPU. MegaMoE remains skipped because it requires its dedicated
+ multi-process scheduler.
-5. Report results: selected physical GPUs, elapsed time, total passed, failed,
- skipped, and errors, plus both import-gate results.
+4. Report results: selected physical GPUs, elapsed time, total passed, failed,
+ skipped, and errors, plus the full log path. Report both downstream gates as
+ passed, failed, or explicitly disabled; disabled integration checks are not
+ passes.
## Failure triage rules
@@ -139,7 +200,8 @@ Always capture and read full logs.**
Classify every failure into one of these categories:
- **A - Environment/import error**: Module not found, missing dependency,
- collection error. These are not caused by code changes.
+ collection error. Check the source and native-library paths, FFI dependency,
+ and old `tvm` versus `tvm_tirx` namespaces before assigning a cause.
- **B - Real kernel correctness regression**: Assertion failures (`cosine_sim`,
numerical diff), `CUDA: unspecified launch failure`, or wrong results. These
MUST be investigated and fixed if caused by current changes.
diff --git a/tirx/tests/python/tirx/iket/test_orchestration.py
b/tirx/tests/python/tirx/iket/test_orchestration.py
index 4260a0796c..0d8f057e97 100644
--- a/tirx/tests/python/tirx/iket/test_orchestration.py
+++ b/tirx/tests/python/tirx/iket/test_orchestration.py
@@ -342,11 +342,22 @@ def _pid_is_running(pid: int) -> bool:
return False
try:
state = stat_path.read_text(encoding="utf-8").split()[2]
- except (FileNotFoundError, IndexError):
+ except (FileNotFoundError, ProcessLookupError, IndexError):
return False
return state != "Z"
[email protected]("error", [FileNotFoundError, ProcessLookupError])
+def test_pid_check_handles_process_exit_during_stat_read(monkeypatch, error):
+ def read_stat(*_args, **_kwargs):
+ raise error("process exited")
+
+ with monkeypatch.context() as patch:
+ patch.setattr(Path, "exists", lambda _path: True)
+ patch.setattr(Path, "read_text", read_stat)
+ assert not _pid_is_running(12345)
+
+
def test_timeout_kills_run_iket_workload_and_grandchild(tmp_path, monkeypatch):
parent_pid = tmp_path / "parent.pid"
child_pid = tmp_path / "child.pid"
