Lunderberg commented on a change in pull request #8638: URL: https://github.com/apache/tvm/pull/8638#discussion_r682749159
########## File path: docs/dev/pytest_target_parametrization.rst ########## @@ -0,0 +1,261 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Python Target Parametrization +============================= + +Summary +------- + +A single python function in the test suite can expand to several +parametrized unit tests, each of which tests a single target device. +In order for a test to be run, all of the following must be true. + +- The test exists in a file or directory that has been passed to + `pytest`. + +- The pytest marks applied to the function, either explicitly or + through target parametrization, must be compatible with the + expression passed to pytest's `-m` argument. + +- For parametrized tests using the `target` fixture, the target must + appear in the environment variable `TVM_TEST_TARGETS`. + +- For parametrized tests using the `target` fixture, the build + configuration in `config.cmake` must enable the corresponding + runtime. + +Unit-Test File Contents +----------------------- + +.. _pytest-marks: https://docs.pytest.org/en/6.2.x/mark.html + +Each test file has one or more test functions. The test functions can +be decorated with `pytest marks <pytest-marks>`_ to include metadata. +The most frequently applied marks are as follows. + +- ``@pytest.mark.gpu`` - Tags a function as using GPU + capabilities. This has no effect on its own, but can be paired with + command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict + which tests pytest will executed. Typically not called on its own. Review comment: Sounds good, change made. ########## File path: docs/dev/pytest_target_parametrization.rst ########## @@ -0,0 +1,261 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Python Target Parametrization +============================= + +Summary +------- + +A single python function in the test suite can expand to several +parametrized unit tests, each of which tests a single target device. +In order for a test to be run, all of the following must be true. + +- The test exists in a file or directory that has been passed to + `pytest`. + +- The pytest marks applied to the function, either explicitly or + through target parametrization, must be compatible with the + expression passed to pytest's `-m` argument. + +- For parametrized tests using the `target` fixture, the target must + appear in the environment variable `TVM_TEST_TARGETS`. + +- For parametrized tests using the `target` fixture, the build + configuration in `config.cmake` must enable the corresponding + runtime. + +Unit-Test File Contents +----------------------- + +.. _pytest-marks: https://docs.pytest.org/en/6.2.x/mark.html + +Each test file has one or more test functions. The test functions can +be decorated with `pytest marks <pytest-marks>`_ to include metadata. +The most frequently applied marks are as follows. + +- ``@pytest.mark.gpu`` - Tags a function as using GPU + capabilities. This has no effect on its own, but can be paired with + command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict + which tests pytest will executed. Typically not called on its own. + +- ``@tvm.testing.uses_gpu`` - Applies ``@pytest.mark.gpu``. Needed + only for tests that explicitly loop over Review comment: Sounds good, change made. ########## File path: docs/dev/pytest_target_parametrization.rst ########## @@ -0,0 +1,261 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Python Target Parametrization +============================= + +Summary +------- + +A single python function in the test suite can expand to several +parametrized unit tests, each of which tests a single target device. +In order for a test to be run, all of the following must be true. + +- The test exists in a file or directory that has been passed to + `pytest`. + +- The pytest marks applied to the function, either explicitly or + through target parametrization, must be compatible with the + expression passed to pytest's `-m` argument. + +- For parametrized tests using the `target` fixture, the target must + appear in the environment variable `TVM_TEST_TARGETS`. + +- For parametrized tests using the `target` fixture, the build + configuration in `config.cmake` must enable the corresponding + runtime. + +Unit-Test File Contents +----------------------- + +.. _pytest-marks: https://docs.pytest.org/en/6.2.x/mark.html + +Each test file has one or more test functions. The test functions can +be decorated with `pytest marks <pytest-marks>`_ to include metadata. +The most frequently applied marks are as follows. + +- ``@pytest.mark.gpu`` - Tags a function as using GPU + capabilities. This has no effect on its own, but can be paired with + command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict + which tests pytest will executed. Typically not called on its own. + +- ``@tvm.testing.uses_gpu`` - Applies ``@pytest.mark.gpu``. Needed + only for tests that explicitly loop over + ``tvm.testing.enabled_targets()``. + +- ``@tvm.testing.requires_gpu`` - Applies ``@tvm.testing.uses_gpu``, + and additionally marks that the test should be skipped + (``@pytest.mark.skipif``) entirely if no GPU is present. + +- ``@tvfm.testing.requires_RUNTIME`` - Several decorators + (e.g. ``@tvm.testing.requires_cuda``), each of which skips a test if + the specified runtime cannot be used. A runtime cannot be used if it + is disabled in the ``config.cmake``, or if a compatible device is + not present. For runtimes that use the GPU, this includes + ``@tvm.testing.requires_gpu``. + +A single test can loop over ``tvm.testing.enabled_targets()`` to +validate all enabled targets, based on the environment variable +``TVM_TEST_TARGETS``, the build configuration, and the physical +hardware present. Most current tests use this style, but it is not +recommended for new tests. The pytest output for this style silently +skips runtimes that are disabled in ``config.cmake``, or do not have a +device on which they can run. In addition, the test halts on the +first target to fail, which is ambiguous as to whether the error +occurs on a particular target, or on every target. + +.. code-block:: python + + # Old style, not recommend anymore Review comment: Sounds good, change made. ########## File path: docs/dev/pytest_target_parametrization.rst ########## @@ -0,0 +1,261 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Python Target Parametrization +============================= + +Summary +------- + +A single python function in the test suite can expand to several +parametrized unit tests, each of which tests a single target device. +In order for a test to be run, all of the following must be true. + +- The test exists in a file or directory that has been passed to + `pytest`. + +- The pytest marks applied to the function, either explicitly or + through target parametrization, must be compatible with the + expression passed to pytest's `-m` argument. + +- For parametrized tests using the `target` fixture, the target must + appear in the environment variable `TVM_TEST_TARGETS`. + +- For parametrized tests using the `target` fixture, the build + configuration in `config.cmake` must enable the corresponding + runtime. + +Unit-Test File Contents +----------------------- + +.. _pytest-marks: https://docs.pytest.org/en/6.2.x/mark.html + +Each test file has one or more test functions. The test functions can +be decorated with `pytest marks <pytest-marks>`_ to include metadata. +The most frequently applied marks are as follows. + +- ``@pytest.mark.gpu`` - Tags a function as using GPU + capabilities. This has no effect on its own, but can be paired with + command-line arguments ``-m gpu`` or ``-m 'not gpu'`` to restrict + which tests pytest will executed. Typically not called on its own. + +- ``@tvm.testing.uses_gpu`` - Applies ``@pytest.mark.gpu``. Needed + only for tests that explicitly loop over + ``tvm.testing.enabled_targets()``. + +- ``@tvm.testing.requires_gpu`` - Applies ``@tvm.testing.uses_gpu``, + and additionally marks that the test should be skipped + (``@pytest.mark.skipif``) entirely if no GPU is present. + +- ``@tvfm.testing.requires_RUNTIME`` - Several decorators + (e.g. ``@tvm.testing.requires_cuda``), each of which skips a test if + the specified runtime cannot be used. A runtime cannot be used if it + is disabled in the ``config.cmake``, or if a compatible device is + not present. For runtimes that use the GPU, this includes + ``@tvm.testing.requires_gpu``. + +A single test can loop over ``tvm.testing.enabled_targets()`` to +validate all enabled targets, based on the environment variable +``TVM_TEST_TARGETS``, the build configuration, and the physical +hardware present. Most current tests use this style, but it is not +recommended for new tests. The pytest output for this style silently +skips runtimes that are disabled in ``config.cmake``, or do not have a +device on which they can run. In addition, the test halts on the +first target to fail, which is ambiguous as to whether the error +occurs on a particular target, or on every target. + +.. code-block:: python + + # Old style, not recommend anymore + def test_function(): + for target,dev in tvm.testing.enabled_targets(): + # Test code goes here + +The recommended method to run a test on multiple targets is by +parametrizing the test. This can be done explicitly for a fixed list +of targets by decorating with +``@tvm.testing.parametrize_targets('target_1', 'target_2', ...)``, and +accepting ``target`` or ``dev`` as function arguments. The function will +be run once for each target listed, and the success/failure of each +target is reported separately. Each test run is decorated with the +``@tvm.testing.requires_RUNTIME`` that corresponds to the target being +used. As a result, if a target is disabled in ``config.cmake`` or does +not have appropriate hardware to run, it will be explicitly listed as +skipped. + +.. code-block:: python + + # New style, explicit listing of targets to use. + @tvm.testing.parametrize_target('llvm', 'cuda') + def test_function(target, dev): + # Test code goes here + +For tests that should run correctly on all targets, the decorator can +be omitted. Any test that accepts a ``target`` or ``dev`` argument +will automatically be parametrized over all targets specified in +``TVM_TEST_TARGETS``. The parametrization provides the same +pass/fail/skipped report for each target, while allowing the test +suite to be easily extended to cover additional targets. + +.. code-block:: python + + # New style, implicitly parametrized to run on all Review comment: Sounds good, change made. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
