| Issue |
164681
|
| Summary |
[lit] Add a way to prevent undesirably skipping tests
|
| Labels |
llvm-lit
|
| Assignees |
|
| Reporter |
ldionne
|
Lit uses "features" to enable and disable individual tests with e.g. `REQUIRES: floating-point-support` or `UNSUPPORTED: windows`. This is used to create often complex conditions that enable/disable parts of a test suite.
This is great, except that as these conditions grow in number and complexity, we can sometimes end up disabling parts of the test suite that shouldn't be disabled. When the test suite uses `UNSUPPORTED:` or `REQUIRES:`, tests can end up never running without anybody noticing.
It would be nice to have an easy way to detect when that happens. Here are a few possibilities:
1. Provide a way for `lit` to check the list of features defined in every test against a user-provided list of features. If the features don't match, make it an error. For example: `lit -sv libcxx/tests --check-features /path/to/list/of/features/txt`. That way, for example, libc++ pre-commit CI runners would pass the list of features they expect to `lit` and we'd see an error whenever that changes. It would require some maintenance but it might be manageable.
2. Provide a way for `lit` to check the list of skipped tests against a user-provided list. Same idea as (1), except we provide a list of tests instead of a list of features.
3. Alternative designs to explore might be to extend (or reuse) the support we already have for setting the list of tests to `XFAIL` via `--xfail LIST`. Perhaps we can do something similar like `--unsupported LIST`, however we'd have to think through the implications of controlling XFAILs/UNSUPPORTEDs solely via that mechanism.
![]()
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs
