Lunderberg commented on a change in pull request #7: URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698489600
########## File path: rfcs/0007-parametrized-unit-tests.md ########## @@ -0,0 +1,568 @@ +- Feature Name: Parametrized Unit Tests +- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD) +- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007) +- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010) + +# Summary +[summary]: #summary + +This RFC documents how to implement unit tests that depend on input +parameters, or have setup that depends on input parameters. + +# Motivation +[motivation]: #motivation + +Some unit tests should be tested along a variety of parameters for +better coverage. For example, a unit test that does not depend on +target-specific features should be tested on all targets that the test +platform supports. Alternatively, a unit test may need to pass +different array sizes to a function, in order to exercise different +code paths within that function. + +The simplest implementation would be to write a test function that +loops over all parameters, throwing an exception if any parameter +fails the test. However, this does not give full information to a +developer, as a failure from any parameter results in the entire test +to be marked as failing. A unit-test that fails for all targets +requires different debugging than a unit-test that fails on a single +specific target, and so this information should be exposed. + +This RFC adds functionality for implementing parameterized unit tests, +such that each set of parameters appears as a separate test result in +the final output. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +## Parameters + +To make a new parameter for unit tests to use, define it with the +`tvm.testing.parameter` function. For example, the following will +define a parameter named `array_size` that has three possible values. +This can appear either at global scope inside a test module to be +usable by all test functions in that module, or in a directory's +`conftest.py` to be usable by all tests in that directory. + +```python +array_size = tvm.testing.parameter(8, 256, 1024) +``` + +To use a parameter, define a test function that accepts the parameter +as an input. This test will be run once for each value of the +parameter. For example, the `test_function` below would be run three +times, each time with a different value of `array_size` according to +the earlier definition. These would show up in the output report as +`test_function[8]`, `test_function[256]`, and `test_function[1024]`, +with the name of the parameter as part of the function. + +```python +def test_function(array_size): + input_array = np.random.uniform(size=array_size) + # Test code here +``` + +If a parameter is used by a test function, but isn't declared as a +function argument, it will produce a `NameError` when accessed. This +happens even if the parameter is defined at module scope, and would +otherwise be accessible by the usual scoping rules. This is +intentional, as access of the global variable would otherwise access +an `array_size` function definition, rather than the specific +parameter value. + +```python +def test_function_broken(): + # Throws NameError, undefined variable "array_size" + input_array = np.random.uniform(size=array_size) + # Test code here +``` + +By default, a test function that accepts multiple parameters as +arguments will be run for all combinations of values of those +parameters. If only some combinations of parameters should be used, +the `tvm.testing.parameters` function can be used to simultaneously +define multiple parameters. A test function that accepts parameters +that were defined through `tvm.testing.parameters` will only be called +once for each set of parameters. + +```python +array_size = tvm.testing.parameter(8, 256, 1024) +dtype = tvm.testing.parameter('float32', 'int32') + +# Called 6 times, once for each combination of array_size and dtype. +def test_function1(array_size, dtype): + assert(True) + +test_data, reference_result = tvm.testing.parameters( + ('test_data_1.dat', 'result_1.txt'), + ('test_data_2.dat', 'result_2.txt'), + ('test_data_3.dat', 'result_3.txt'), +) + +# Called 3 times, once for each (test_data, reference_result) tuple. +def test_function3(test_data, reference_result): + assert(True) +``` + +## Fixtures + +Fixtures in pytest separate setup code from test code, and are used +for two primary purposes. The first is for improved readability when +debugging, so that a failure in the setup is distinguishable from a +failure in the test. The second is to avoid performing expensive test +setup that is shared across multiple tests, letting the test suite run +faster. + +For example, the following function first reads test data, and then +performs tests that use the test data. + +```python +# test_function_old() calls read_test_data(). If read_test_data() +# throws an error, test_function_old() shows as a failed test. + +def test_function_old(): + dataset = read_test_data() + assert(True) # Test succeeds +``` + +This can be pulled out into a separate setup function, which the test +function then accepts as an argument. In this usage, this is +equivalent to using a bare `@pytest.fixture` decorator. By default, +the fixture value is recalculated for every test function, to minimize +the potential for interaction between unit tests. + +```python [email protected] +def dataset(): + print('Prints once for each test function that uses dataset.') + return read_test_data() + +# test_function_new() accepts the dataset fixture. If +# read_test_data() throws an error, test_function_new() shows +# as unrunnable. +def test_function_new(dataset): + assert(True) # Test succeeds +``` + +If the fixture is more expensive to calculate, then it may be worth +caching the computed fixture. This is done with the +`cache_return_value=True` argument. + +```python [email protected](cache_return_value = True) +def dataset(): + print('Prints once no matter how many test functions use dataset.') + return download_test_data() + +def test_function(dataset): + assert(True) # Test succeeds +``` + +The caching can be disabled entirely by setting the environment +variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be +useful to re-run tests that failed, to check whether the failure is +due to modification/re-use of a cached value. + +A fixture can depend on parameters, or on other fixtures. This is Review comment: Updated as suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
