jonkeane commented on a change in pull request #12140:
URL: https://github.com/apache/arrow/pull/12140#discussion_r784117244
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
+ └── testthat.R # runs tests when R CMD check runs (e.g. with
devtools::check())
+
+ Usually, most files in the ``R/`` sub-folder have a corresponding test
file in ``tests/testthat``.
+
+ **Running tests**
+
+ To run all tests in a package locally call
+
+ .. code-block:: R
+
+ devtools::test()
+
+ in the R console. Alternatively, you can use
+
+ .. code:: console
+
+ $ make test
+
+ in the shell.
+
+ You can run the tests in a single test file you have open with
+
+ .. code-block:: R
+
+ devtools::test_active_file()
+
+ All tests are also run as part of our continuous integration (CI)
pipelines.
+
+ **Good practice**
+
+ In general any change to source code needs to be accompanied by unit
tests. All tests are expected to pass before a pull request is merged.
+
+ * Add functionality -> add unit tests
+ * Modify functionality -> update unit tests
Review comment:
I like the simplicity of these bullet points, so maybe this should be in
a footnote or in a paragraph below this, but we should also include something
like:
If the new functionality is a user-facing or API change, you will almost
certainly need to change tests — if no tests need to be changed it might mean
the tests aren't right! If the new functionality is a refactor and no APIs are
changing, there might not need to be test changes.
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
+ └── testthat.R # runs tests when R CMD check runs (e.g. with
devtools::check())
Review comment:
This also has a bit of configuration about what kind of output the tests
will have, should we mention that as well? And possibly a note that this isn't
a file that gets touched often too?
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
Review comment:
```suggestion
├── testthat # test files live here
```
We mostly have unit tests, but also have integration tests (the pyarrow
bridge), end to end tests, functional tests too. Doing ~unit~ means that no one
will be confused (and I think "test files" is clear enough her on it's own)
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
+ └── testthat.R # runs tests when R CMD check runs (e.g. with
devtools::check())
+
+ Usually, most files in the ``R/`` sub-folder have a corresponding test
file in ``tests/testthat``.
+
+ **Running tests**
+
+ To run all tests in a package locally call
+
+ .. code-block:: R
+
+ devtools::test()
+
+ in the R console. Alternatively, you can use
+
+ .. code:: console
+
+ $ make test
+
+ in the shell.
+
+ You can run the tests in a single test file you have open with
+
+ .. code-block:: R
+
+ devtools::test_active_file()
+
+ All tests are also run as part of our continuous integration (CI)
pipelines.
+
+ **Good practice**
+
+ In general any change to source code needs to be accompanied by unit
tests. All tests are expected to pass before a pull request is merged.
+
+ * Add functionality -> add unit tests
+ * Modify functionality -> update unit tests
+ * Solve a bug -> add unit test before solving it, which helps prove the
bug and its fix
Review comment:
💯
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
+ └── testthat.R # runs tests when R CMD check runs (e.g. with
devtools::check())
+
+ Usually, most files in the ``R/`` sub-folder have a corresponding test
file in ``tests/testthat``.
+
+ **Running tests**
+
+ To run all tests in a package locally call
+
+ .. code-block:: R
+
+ devtools::test()
+
+ in the R console. Alternatively, you can use
+
+ .. code:: console
+
+ $ make test
+
+ in the shell.
+
+ You can run the tests in a single test file you have open with
+
+ .. code-block:: R
+
+ devtools::test_active_file()
+
+ All tests are also run as part of our continuous integration (CI)
pipelines.
+
+ **Good practice**
+
+ In general any change to source code needs to be accompanied by unit
tests. All tests are expected to pass before a pull request is merged.
+
+ * Add functionality -> add unit tests
+ * Modify functionality -> update unit tests
+ * Solve a bug -> add unit test before solving it, which helps prove the
bug and its fix
+ * Performance improvements should be reflected in benchmarks (which are
also tests)
+ * An exception could be refactoring functionality that is fully covered
by unit tests
+
+ **Testing helpers**
+
+ To complement the ``testthat`` functionality, the ``arrow`` R package
has defined a series of specific utility functions (called helpers), such as:
+
+ * Expectations - these start with ``expect_`` and are used to compare
objects
+ - for example, ``expect_altrep_roundtrip()`` compares the result
+ of a function ``fn`` run on a vector ``x`` with the result of the
+ same function run on the altrep version of ``x``. More generally,
+ expect_…_roundtrip() functions do …
+
+ .. TODO _fill in the blanks_
+ .. TODO
Review comment:
I presume these will be done in another commit? OR should we remove them
/ add Jiras for following up?
##########
File path: docs/source/developers/guide/step_by_step/testing.rst
##########
@@ -62,29 +62,103 @@ In this section we outline steps needed for unit testing
in Arrow.
If the tests start failing, try to recompile
PyArrow or C++.
-
+
.. note::
**Recompiling Cython**
If you only make changes to `.py` files, you do not need to
recompile PyArrow. However, you should recompile it if you make
changes in `.pyx` or `.pxd` files.
-
+
To do that run this command again:
.. code:: console
$ python setup.py build_ext --inplace
.. note::
-
+
**Recompiling C++**
Similarly, you will need to recompile the C++ code if you have
made changes to any C++ files. In this case,
- re-run the cmake commands again.
+ re-run the cmake commands again.
.. tab:: R tests
- .. TODO
+ We use `testthat <https://testthat.r-lib.org/index.html>`_ for unit
testing in R. More specifically, we use the `3rd edition of testthat
<https://testthat.r-lib.org/articles/third-edition.html>`_. On rare occasions
we might want the behaviour of the 2nd edition of testthat, which is indicated
by ``testthat::local_edition(2)``.
+
+ **Structure**
+
+ Expect the usual testthat folder structure:
+
+ .. code-block:: R
+
+ tests
+ ├── testthat # unit test files live here
+ └── testthat.R # runs tests when R CMD check runs (e.g. with
devtools::check())
+
+ Usually, most files in the ``R/`` sub-folder have a corresponding test
file in ``tests/testthat``.
+
+ **Running tests**
+
+ To run all tests in a package locally call
+
+ .. code-block:: R
+
+ devtools::test()
+
+ in the R console. Alternatively, you can use
+
+ .. code:: console
+
+ $ make test
+
+ in the shell.
+
+ You can run the tests in a single test file you have open with
+
+ .. code-block:: R
+
+ devtools::test_active_file()
+
+ All tests are also run as part of our continuous integration (CI)
pipelines.
+
+ **Good practice**
+
+ In general any change to source code needs to be accompanied by unit
tests. All tests are expected to pass before a pull request is merged.
+
+ * Add functionality -> add unit tests
+ * Modify functionality -> update unit tests
+ * Solve a bug -> add unit test before solving it, which helps prove the
bug and its fix
+ * Performance improvements should be reflected in benchmarks (which are
also tests)
+ * An exception could be refactoring functionality that is fully covered
by unit tests
+
+ **Testing helpers**
+
+ To complement the ``testthat`` functionality, the ``arrow`` R package
has defined a series of specific utility functions (called helpers), such as:
+
+ * Expectations - these start with ``expect_`` and are used to compare
objects
+ - for example, ``expect_altrep_roundtrip()`` compares the result
+ of a function ``fn`` run on a vector ``x`` with the result of the
+ same function run on the altrep version of ``x``. More generally,
+ expect_…_roundtrip() functions do …
+
+ .. TODO _fill in the blanks_
+ .. TODO
+
+ - Expect
+ * ``skip_`` - skips a unit test - think of them as acceptable fails.
Situations in which we might want to skip unit tests:
+
+ - ``skip_if_r_version()`` - this is a specific ``arrow`` skip. For
example, we use this to skip a unit test when the R version is 3.5.0 and below
(``skip_if_r_version(“3.5.0”)``). You will likely see it used when the
functionality we are testing depends on features introduced after version 3.5.0
of R (such as the alternative representation of vectors, Altrep, introduced in
R 3.5.0, but with significant additions in subsequent releases). As part of our
CI workflow we test against different versions of R and this is where this
feature comes in.
+ - ``skip_if_not_available()`` - another specific {arrow} skip. Arrow
(libarrow) has a series of additional features that can be switched on or off
(but this needs to happen at build time). If a unit test depends on such a
feature and this feature is not available (i.e. was not selected when libarrow
was built) the test is skipped, as opposed to having a failed test.
+ - ``skip_if_offline()`` - will not run tests that require an internet
connection
+ - ``skip_on_os()`` - for unit tests that are OS specific.
+
+ *Important*: Once the conditions for a ``skip_()`` statement is met, no
other line of code in the same ``test_that()`` test block will get executed.
Review comment:
```suggestion
*Important*: Once the conditions for a ``skip_()`` statement is met,
no other line of code in the same ``test_that()`` test block will get executed.
If the skip is outside of a `test_that()` code block, it will skip the rest of
the file.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
