tqchen opened a new issue, #15354:
URL: https://github.com/apache/tvm/issues/15354
We have discussed this item in community forum and community meetups.
Capturing some of the takeaways here. Right now the set of tests we have are a
bit mixed together that causes things to slow down and mix of more recent
modules with legacy ones. In many cases, it is also tempting to just comparing
TVM results against other frameworks is easy to do in tests but it’s slow and
very fragile; many older tests do this.
The main approach we would like to take is to categorize the test cases. We
will start with TensorIR. Hopefully the lessons also expand to other modules
(at least those that are not legacy).
### Guidelines
There will be three categories of tests(using tir as example)
- tests/python/tir: This is the location for unit-tests
- The tests in this folder always use TVMScript/IRBuilder before and
after to test related module(pass)
- There is no explicit build involved
- Tests here should be fast
- tests/python/integration/tir: This is the location for tests that involves
build, but is also still fast
- The tests in this folder can call build
- We still encourage not calling schedule and other things, instead, if
we are testing say a CUDA build,
consider start from a already scheduled TVMScript
- Tests in this folder should be reasonably fast(e.g. finish in 1min or
less)
- tests/python/slow/tir: This is the location for tests that involves build,
runs end to end are slow
- Tests in this folder can take minutes to run
- The idea is that if a bug happens and is not caught by integration or
unit-test, likely it means we should update those tests
The idea is to gradually move tests into those three categories, and we can
move the slow tests to nightly, with the goal that if a regression is caught by
`tests/python/slow/tir`, it should really be covered by ` tests/python/tir` and
`tests/python/integration/tir` instead.
The categorization do not need to be done in a single shot. Since we can
start by having (legacy) tests in their original folder, and migrate old tests
and put new tests into those folders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
