Aharrypotter opened a new pull request, #19515:
URL: https://github.com/apache/tvm/pull/19515
## Summary
Introduce a test runner that reuses the official ONNX Backend Test Suite to
systematically verify the Relax ONNX importer. This complements the existing
hand-written tests in `test_frontend_onnx.py` by providing spec-aligned
coverage of standard ONNX operator semantics.
Towards #19505
## Motivation
The existing `test_frontend_onnx.py` has 187 hand-written tests that
validate TVM-specific importer behavior (parameter handling, name sanitization,
dynamic shapes, Relax IR structure). However, it relies on ONNX Runtime as the
reference and cannot systematically cover all edge cases defined in the ONNX
specification.
The ONNX Backend Test Suite provides 1653+ node-level tests with protobuf
reference inputs/outputs. It is the industry standard for validating ONNX
importers/exporters (used by ONNX Runtime, TensorFlow, PyTorch). Reusing it
gives Relax a living, upstream-aligned correctness baseline.
## What this PR adds
- `tests/python/relax/test_frontend_onnx_backend.py` — a backend adapter
(`TVMRelaxBackend`) that implements the `onnx.backend.base.Backend` interface,
wiring `from_onnx()` → `DecomposeOpsForInference()` → `LegalizeOps()` →
`tvm.compile()` → `VirtualMachine`.
- `tests/python/relax/conftest.py` — registers the `onnx_backend` pytest
marker for selective test execution via `SKIP_SLOW_TESTS=1`.
## Coverage
72 operators with 388 test cases, all passing. Only operators where every
ONNX node test passes are included — no xfail markers.
Operators not yet covered include: cast (exotic dtypes), reduce ops (edge
cases), reshape/resize/attention (complex behavior), quantization, and several
others with known importer gaps. These can be added incrementally as the
importer improves.
## Test results
388 passed, 3216 skipped (CUDA variants + operators not yet in allowlist), 0
failed, 0 xfailed
## CI impact
- New test file is not added to any existing CI test shard by default
- `SKIP_SLOW_TESTS=1` allows developers to skip the full suite locally
- Full suite (388 tests) is lightweight on CPU-only runners
## Design decisions
- **Coexistence with existing tests**: `test_frontend_onnx.py` remains
unchanged. Backend tests cover standard ONNX semantics; hand-written tests
continue to cover TVM-specific behavior (dynamic shapes, Relax IR structure,
importer options).
- **Public API only**: uses `backend_test.include()` with `^`-anchored regex
patterns. No access to private ONNX APIs.
- **No xfail**: only include operators that fully pass. Uncovered operators
are documented in code comments and this PR description. Follow-up PRs can
expand coverage as importer gaps are fixed.
- **Prefix conflict handling**: `include()` patterns use
`^test_{op}(?:_.*)?(?:_cpu|_cuda)$`, which can cause false matches when a short
op name is a prefix of a longer one (e.g. `log` vs `log_softmax`). Affected ops
(`log`, `max`, `relu`) are excluded until a more precise matching strategy is
adopted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
