Re: [PATCH] Documentation: kunit: Update kunit_tool page
On Thu, Apr 15, 2021 at 8:40 PM David Gow wrote: > > The kunit_tool documentation page was pretty minimal, and a bit > outdated. Update it and flesh it out a bit. > > In particular, > - Mention that .kunitconfig is now in the build directory > - Describe the use of --kunitconfig to specify a different config > framgent > - Mention the split functionality (i.e., commands other than 'run') > - Describe --raw_output and kunit.py parse > - Mention the globbing support > - Provide a quick overview of other options, including --build_dir and > --alltests > > Note that this does overlap a little with the new running_tips page. I > don't think it's a problem having both: this page is supposed to be a > bit more of a reference, rather than a list of useful tips, so the fact > that they both describe the same features isn't a problem. > > Signed-off-by: David Gow Reviewed-by: Daniel Latypov Looks good to me. I completely forgot to update this page when adding blobs about the new features in running_tips.rst... Two minor, optional nits. > --- > Documentation/dev-tools/kunit/kunit-tool.rst | 132 ++- > 1 file changed, 128 insertions(+), 4 deletions(-) > > diff --git a/Documentation/dev-tools/kunit/kunit-tool.rst > b/Documentation/dev-tools/kunit/kunit-tool.rst > index 29ae2fee8123..0b45affcd65c 100644 > --- a/Documentation/dev-tools/kunit/kunit-tool.rst > +++ b/Documentation/dev-tools/kunit/kunit-tool.rst > @@ -22,14 +22,19 @@ not require any virtualization support: it is just a > regular program. > What is a .kunitconfig? > === > > -It's just a defconfig that kunit_tool looks for in the base directory. > +It's just a defconfig that kunit_tool looks for in the build directory. nit: should we mention this is .kunit here? We don't mention this till the very bottom right now, which seems suboptimal. I assume most people are going to still be fiddling with that .kunitconfig rather than passing in --kunitconfig (I know I am). > kunit_tool uses it to generate a .config as you might expect. In addition, it > verifies that the generated .config contains the CONFIG options in the > .kunitconfig; the reason it does this is so that it is easy to be sure that a > CONFIG that enables a test actually ends up in the .config. > > -How do I use kunit_tool? > - > +It's also possible to pass a separate .kunitconfig fragment to kunit_tool, > +which is useful if you have several different groups of tests you wish > +to run independently, or if you want to use pre-defined test configs for > +certain subsystems. > + > +Getting Started with kunit_tool > +=== > > If a kunitconfig is present at the root directory, all you have to do is: > > @@ -48,10 +53,129 @@ However, you most likely want to use it with the > following options: > > .. note:: > This command will work even without a .kunitconfig file: if no > -.kunitconfig is present, a default one will be used instead. > + .kunitconfig is present, a default one will be used instead. > + > +If you wish to use a different .kunitconfig file (such as one provided for > +testing a particular subsystem), you can pass it as an option. > + > +.. code-block:: bash > + > + ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig > > For a list of all the flags supported by kunit_tool, you can run: > > .. code-block:: bash > > ./tools/testing/kunit/kunit.py run --help > + > +Configuring, Building, and Running Tests > + > + > +It's also possible to run just parts of the KUnit build process > independently, > +which is useful if you want to make manual changes to part of the process. > + > +A .config can be generated from a .kunitconfig by using the ``config`` > argument > +when running kunit_tool: > + > +.. code-block:: bash > + > + ./tools/testing/kunit/kunit.py config > + > +Similarly, if you just want to build a KUnit kernel from the current .config, > +you can use the ``build`` argument: > + > +.. code-block:: bash > + > + ./tools/testing/kunit/kunit.py build > + > +And, if you already have a built UML kernel with built-in KUnit tests, you > can > +run the kernel and display the test results with the ``exec`` argument: > + > +.. code-block:: bash > + > + ./tools/testing/kunit/kunit.py exec > + > +The ``run`` command which is discussed above is equivalent to running all > three > +of these in sequence. > + > +All of these commands accept a number of optional command-line arguments. The > +``--help`` flag will give a complete list of these, or keep readi
[PATCH v2] kunit: add unit test for filtering suites by names
This adds unit tests for kunit_filter_subsuite() and
kunit_filter_suites().
Note: what the executor means by "subsuite" is the array of suites
corresponding to each test file.
This patch lightly refactors executor.c to avoid the use of global
variables to make it testable.
It also includes a clever `kfree_at_end()` helper that makes this test
easier to write than it otherwise would have been.
Tested by running just the new tests using itself
$ ./tools/testing/kunit/kunit.py run '*exec*'
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
v1 -> v2:
* Fix missing free in kfree_subsuites_at_end()
---
lib/kunit/executor.c | 26
lib/kunit/executor_test.c | 133 ++
2 files changed, 148 insertions(+), 11 deletions(-)
create mode 100644 lib/kunit/executor_test.c
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 15832ed44668..96a4ae983786 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -19,7 +19,7 @@ MODULE_PARM_DESC(filter_glob,
"Filter which KUnit test suites run at boot-time, e.g. list*");
static struct kunit_suite * const *
-kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite, const char
*filter_glob)
{
int i, n = 0;
struct kunit_suite **filtered;
@@ -52,19 +52,14 @@ struct suite_set {
struct kunit_suite * const * const *end;
};
-static struct suite_set kunit_filter_suites(void)
+static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
+ const char *filter_glob)
{
int i;
struct kunit_suite * const **copy, * const *filtered_subsuite;
struct suite_set filtered;
- const size_t max = __kunit_suites_end - __kunit_suites_start;
-
- if (!filter_glob) {
- filtered.start = __kunit_suites_start;
- filtered.end = __kunit_suites_end;
- return filtered;
- }
+ const size_t max = suite_set->end - suite_set->start;
copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
filtered.start = copy;
@@ -74,7 +69,7 @@ static struct suite_set kunit_filter_suites(void)
}
for (i = 0; i < max; ++i) {
- filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ filtered_subsuite = kunit_filter_subsuite(suite_set->start[i],
filter_glob);
if (filtered_subsuite)
*copy++ = filtered_subsuite;
}
@@ -98,8 +93,13 @@ static void kunit_print_tap_header(struct suite_set
*suite_set)
int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
+ struct suite_set suite_set = {
+ .start = __kunit_suites_start,
+ .end = __kunit_suites_end,
+ };
- struct suite_set suite_set = kunit_filter_suites();
+ if (filter_glob)
+ suite_set = kunit_filter_suites(_set, filter_glob);
kunit_print_tap_header(_set);
@@ -115,4 +115,8 @@ int kunit_run_all_tests(void)
return 0;
}
+#if IS_BUILTIN(CONFIG_KUNIT_TEST)
+#include "executor_test.c"
+#endif
+
#endif /* IS_BUILTIN(CONFIG_KUNIT) */
diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
new file mode 100644
index ..cdbe54b16501
--- /dev/null
+++ b/lib/kunit/executor_test.c
@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for the KUnit executor.
+ *
+ * Copyright (C) 2021, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+
+static void kfree_at_end(struct kunit *test, const void *to_free);
+static struct kunit_suite *alloc_fake_suite(struct kunit *test,
+ const char *suite_name);
+
+static void filter_subsuite_test(struct kunit *test)
+{
+ struct kunit_suite *subsuite[3] = {NULL, NULL, NULL};
+ struct kunit_suite * const *filtered;
+
+ subsuite[0] = alloc_fake_suite(test, "suite1");
+ subsuite[1] = alloc_fake_suite(test, "suite2");
+
+ /* Want: suite1, suite2, NULL -> suite2, NULL */
+ filtered = kunit_filter_subsuite(subsuite, "suite2*");
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, filtered);
+ kfree_at_end(test, filtered);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, filtered[0]);
+ KUNIT_EXPECT_STREQ(test, (const char *)filtered[0]->name, "suite2");
+
+ KUNIT_EXPECT_FALSE(test, filtered[1]);
+}
+
+static void filter_subsuite_to_empty_test(struct kunit *test)
+{
+ struct kunit_suite *subsuite[3] = {NULL, NULL, NULL};
+ struct kunit_suite * const *filtered;
+
+ subsuite[0] = alloc_fake_suite(test, "suite1");
+ subsuite[1] = alloc_fake_suite(test, "suite2");
+
+ filtered = kunit_filter_subsuite(subsuite, "n
Re: [PATCH v5] lib: add basic KUnit test for lib/math
On Tue, Apr 13, 2021 at 5:33 PM Daniel Latypov wrote:
>
> On Mon, Apr 12, 2021 at 11:41 PM David Gow wrote:
> >
> > On Tue, Apr 13, 2021 at 3:07 AM Daniel Latypov wrote:
> > >
> > > Add basic test coverage for files that don't require any config options:
> > > * part of math.h (what seem to be the most commonly used macros)
> > > * gcd.c
> > > * lcm.c
> > > * int_sqrt.c
> > > * reciprocal_div.c
> > > (Ignored int_pow.c since it's a simple textbook algorithm.)
> > >
> > > These tests aren't particularly interesting, but they
> > > * provide short and simple examples of parameterized tests
> > > * provide a place to add tests for any new files in this dir
> > > * are written so adding new test cases to cover edge cases should be easy
> > > * looking at code coverage, we hit all the branches in the .c files
> > >
> > > Signed-off-by: Daniel Latypov
> >
> > This looks good to me. A few comments/observations below, but nothing
> > that I think should actually block this.
> >
> > Reviewed-by: David Gow
> >
> > -- David
> >
> > > ---
> > > Changes since v4:
> > > * add in test cases for some math.h macros (abs, round_up/round_down,
> > > div_round_down/closest)
> > > * use parameterized testing less to keep things terser
> > >
> > > Changes since v3:
> > > * fix `checkpatch.pl --strict` warnings
> > > * add test cases for gcd(0,0) and lcm(0,0)
> > > * minor: don't test both gcd(a,b) and gcd(b,a) when a == b
> > >
> > > Changes since v2: mv math_test.c => math_kunit.c
> > >
> > > Changes since v1:
> > > * Rebase and rewrite to use the new parameterized testing support.
> > > * misc: fix overflow in literal and inline int_sqrt format string.
> > > * related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
> > > for testing many inputs") was merged explaining the patterns shown here.
> > > * there's an in-flight patch to update it for parameterized testing.
> > >
> > > v1:
> > > https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
> > > ---
> > > lib/math/Kconfig | 5 +
> > > lib/math/Makefile | 2 +
> > > lib/math/math_kunit.c | 264 ++
> > > 3 files changed, 271 insertions(+)
> > > create mode 100644 lib/math/math_kunit.c
> > >
> > > diff --git a/lib/math/Kconfig b/lib/math/Kconfig
> > > index f19bc9734fa7..6ba8680439c1 100644
> > > --- a/lib/math/Kconfig
> > > +++ b/lib/math/Kconfig
> > > @@ -15,3 +15,8 @@ config PRIME_NUMBERS
> > >
> > > config RATIONAL
> > > bool
> > > +
> > > +config MATH_KUNIT_TEST
> > > + tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
> > > + default KUNIT_ALL_TESTS
> > > + depends on KUNIT
> >
> > This could have a description of the test and KUnit here, as mentioned
> > in the style guide doc:
> > https://www.kernel.org/doc/html/latest/dev-tools/kunit/style.html#test-kconfig-entries
> >
> > (I think it's sufficiently self explanatory that it's not essential,
> > but it could be nice to have a more detailed description of the things
> > being tested than just "lib/math".)
> >
>
> Done. I've left off the details about what the test tests so we have
> less places to go and update if/when new tests are added.
>
> > > diff --git a/lib/math/Makefile b/lib/math/Makefile
> > > index be6909e943bd..30abb7a8d564 100644
> > > --- a/lib/math/Makefile
> > > +++ b/lib/math/Makefile
> > > @@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
> > > reciprocal_div.o
> > > obj-$(CONFIG_CORDIC) += cordic.o
> > > obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
> > > obj-$(CONFIG_RATIONAL) += rational.o
> > > +
> > > +obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
> > > diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
> > > new file mode 100644
> > > index ..80a087a32884
> > > --- /dev/null
> > > +++ b/lib/math/math_kunit.c
> > > @@ -0,0 +1,264 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * Simple KUnit suite for math helper funcs that are always enabled.
> > > + *
> > > + * Copyright (C) 2020, Google LLC.
> > &g
[PATCH v6] lib: add basic KUnit test for lib/math
Add basic test coverage for files that don't require any config options:
* part of math.h (what seem to be the most commonly used macros)
* gcd.c
* lcm.c
* int_sqrt.c
* reciprocal_div.c
(Ignored int_pow.c since it's a simple textbook algorithm.)
These tests aren't particularly interesting, but they
* provide short and simple examples of parameterized tests
* provide a place to add tests for any new files in this dir
* are written so adding new test cases to cover edge cases should be easy
* looking at code coverage, we hit all the branches in the .c files
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
Changes since v5:
* add in test cases for roundup/rounddown
* address misc comments from David
Changes since v4:
* add in test cases for some math.h macros (abs, round_up/round_down,
div_round_down/closest)
* use parameterized testing less to keep things terser
Changes since v3:
* fix `checkpatch.pl --strict` warnings
* add test cases for gcd(0,0) and lcm(0,0)
* minor: don't test both gcd(a,b) and gcd(b,a) when a == b
Changes since v2: mv math_test.c => math_kunit.c
Changes since v1:
* Rebase and rewrite to use the new parameterized testing support.
* misc: fix overflow in literal and inline int_sqrt format string.
* related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
for testing many inputs") was merged explaining the patterns shown here.
* there's an in-flight patch to update it for parameterized testing.
---
lib/math/Kconfig | 12 ++
lib/math/Makefile | 2 +
lib/math/math_kunit.c | 291 ++
3 files changed, 305 insertions(+)
create mode 100644 lib/math/math_kunit.c
diff --git a/lib/math/Kconfig b/lib/math/Kconfig
index f19bc9734fa7..a974d4db0f9c 100644
--- a/lib/math/Kconfig
+++ b/lib/math/Kconfig
@@ -15,3 +15,15 @@ config PRIME_NUMBERS
config RATIONAL
bool
+
+config MATH_KUNIT_TEST
+ tristate "KUnit test for lib/math and math.h" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds unit tests for lib/math and math.h.
+
+ For more information on KUnit and unit tests in general, please
refer
+ to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ If unsure, say N.
diff --git a/lib/math/Makefile b/lib/math/Makefile
index be6909e943bd..30abb7a8d564 100644
--- a/lib/math/Makefile
+++ b/lib/math/Makefile
@@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
reciprocal_div.o
obj-$(CONFIG_CORDIC) += cordic.o
obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
obj-$(CONFIG_RATIONAL) += rational.o
+
+obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
new file mode 100644
index ..556c23b17c3c
--- /dev/null
+++ b/lib/math/math_kunit.c
@@ -0,0 +1,291 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple KUnit suite for math helper funcs that are always enabled.
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+static void abs_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, abs((char)0), (char)0);
+ KUNIT_EXPECT_EQ(test, abs((char)42), (char)42);
+ KUNIT_EXPECT_EQ(test, abs((char)-42), (char)42);
+
+ /* The expression in the macro is actually promoted to an int. */
+ KUNIT_EXPECT_EQ(test, abs((short)0), 0);
+ KUNIT_EXPECT_EQ(test, abs((short)42), 42);
+ KUNIT_EXPECT_EQ(test, abs((short)-42), 42);
+
+ KUNIT_EXPECT_EQ(test, abs(0), 0);
+ KUNIT_EXPECT_EQ(test, abs(42), 42);
+ KUNIT_EXPECT_EQ(test, abs(-42), 42);
+
+ KUNIT_EXPECT_EQ(test, abs(0L), 0L);
+ KUNIT_EXPECT_EQ(test, abs(42L), 42L);
+ KUNIT_EXPECT_EQ(test, abs(-42L), 42L);
+
+ KUNIT_EXPECT_EQ(test, abs(0LL), 0LL);
+ KUNIT_EXPECT_EQ(test, abs(42LL), 42LL);
+ KUNIT_EXPECT_EQ(test, abs(-42LL), 42LL);
+
+ /* Unsigned types get casted to signed. */
+ KUNIT_EXPECT_EQ(test, abs(0ULL), 0LL);
+ KUNIT_EXPECT_EQ(test, abs(42ULL), 42LL);
+}
+
+static void int_sqrt_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, int_sqrt(0UL), 0UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(1UL), 1UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(4UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(5UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(8UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(1UL << 30), 1UL << 15);
+}
+
+static void round_up_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, round_up(0, 1), 0);
+ KUNIT_EXPECT_EQ(test, round_up(1, 2), 2);
+ KUNIT_EXPECT_EQ(test, round_up(3, 2), 4);
+ KUNIT_EXPECT_EQ(test, round_up((1 << 30) - 1, 2), 1 << 30);
+ KUNIT_EXPECT_EQ(test, round_up((1 << 30) - 1, 1 << 29), 1 << 30);
+}
+
+static void round_down_test(str
[PATCH v4] Documentation: kunit: add tips for running KUnit
This is long overdue.
There are several things that aren't nailed down (in-tree
.kunitconfig's), or partially broken (GCOV on UML), but having them
documented, warts and all, is better than having nothing.
This covers a bunch of the more recent features
* kunit_filter_glob
* kunit.py run --kunitconfig
* slightly more detail on building tests as modules
* CONFIG_KUNIT_DEBUGFS
By my count, the only headline features now not mentioned are the KASAN
integration and KernelCI json output support (kunit.py run --json).
And then it also discusses how to get code coverage reports under UML
and non-UML since this is a question people have repeatedly asked.
Non-UML coverage collection is no different from normal, but we should
probably explicitly call this out.
As for UML, I was able to get it working again with two small hacks.*
E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
Overall coverage rate:
lines..: 15.1% (18294 of 120776 lines)
functions..: 16.8% (1860 of 11050 functions)
Note: this doesn't document --alltests since this is not stable yet.
Hopefully being run more frequently as part of KernelCI will help...
*Using gcc/gcov-6 and not using uml_abort() in os_dump_core().
I've documented these hacks in "Notes" but left TODOs for
brendanhigg...@google.com who tracked down the runtime issue in GCC.
To be clear: these are not issues specific to KUnit, but rather to UML.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
v3 -> v4:
* update instructions on how to remove uml_abort() call
v2 -> v3:
* Suggest --make_options=CC=/usr/bin/gcc-6 instead of manually editing
kunit_kernel.py
v1 -> v2:
Fix typos, drop --alltests, changed wordiing on config fragments.
---
Documentation/dev-tools/kunit/index.rst | 1 +
.../dev-tools/kunit/running_tips.rst | 259 ++
Documentation/dev-tools/kunit/start.rst | 2 +
3 files changed, 262 insertions(+)
create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
diff --git a/Documentation/dev-tools/kunit/index.rst
b/Documentation/dev-tools/kunit/index.rst
index 848478838347..7f7cf8d2ab20 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
style
faq
tips
+ running_tips
What is KUnit?
==
diff --git a/Documentation/dev-tools/kunit/running_tips.rst
b/Documentation/dev-tools/kunit/running_tips.rst
new file mode 100644
index ..7d99386cf94a
--- /dev/null
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -0,0 +1,259 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Tips For Running KUnit Tests
+
+
+Using ``kunit.py run`` ("kunit tool")
+=
+
+Running from any directory
+--
+
+It can be handy to create a bash function like:
+
+.. code-block:: bash
+
+ function run_kunit() {
+ ( cd "$(git rev-parse --show-toplevel)" &&
./tools/testing/kunit/kunit.py run $@ )
+ }
+
+.. note::
+ Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
+ the kernel root, hence the use of a subshell and ``cd``.
+
+Running a subset of tests
+-
+
+``kunit.py run`` accepts an optional glob argument to filter tests. Currently
+this only matches against suite names, but this may change in the future.
+
+Say that we wanted to run the sysctl tests, we could do so via:
+
+.. code-block:: bash
+
+ $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' >
.kunit/.kunitconfig
+ $ ./tools/testing/kunit/kunit.py run 'sysctl*'
+
+We're paying the cost of building more tests than we need this way, but it's
+easier than fiddling with ``.kunitconfig`` files or commenting out
+``kunit_suite``'s.
+
+However, if we wanted to define a set of tests in a less ad hoc way, the next
+tip is useful.
+
+Defining a set of tests
+---
+
+``kunit.py run`` (along with ``build``, and ``config``) supports a
+``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
+regular basis (especially if they have other dependencies), you can create a
+specific ``.kunitconfig`` for them.
+
+E.g. kunit has one for its tests:
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run
--kunitconfig=lib/kunit/.kunitconfig
+
+Alternatively, if you're following the convention of naming your
+file ``.kunitconfig``, you can just pass in the dir, e.g.
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
+
+.. note::
+ This is a relatively new feature (5.12+) so we don't have any
+ conventions yet about on what files should be checked in versus just
+ kept around locally. It's up to you and your maintainer to decide if a
+ config is useful eno
Re: [PATCH v3] Documentation: kunit: add tips for running KUnit
On Tue, Apr 13, 2021 at 9:22 PM David Gow wrote: > > On Wed, Apr 14, 2021 at 8:45 AM Daniel Latypov wrote: > > > > This is long overdue. > > > > There are several things that aren't nailed down (in-tree > > .kunitconfig's), or partially broken (GCOV on UML), but having them > > documented, warts and all, is better than having nothing. > > > > This covers a bunch of the more recent features > > * kunit_filter_glob > > * kunit.py run --kunitconfig > > * slightly more detail on building tests as modules > > * CONFIG_KUNIT_DEBUGFS > > > > By my count, the only headline features now not mentioned are the KASAN > > integration and KernelCI json output support (kunit.py run --json). > > > > And then it also discusses how to get code coverage reports under UML > > and non-UML since this is a question people have repeatedly asked. > > > > Non-UML coverage collection is no different from normal, but we should > > probably explicitly call this out. > > > > As for UML, I was able to get it working again with two small hacks.* > > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > > Overall coverage rate: > > lines..: 15.1% (18294 of 120776 lines) > > functions..: 16.8% (1860 of 11050 functions) > > > > Note: this doesn't document --alltests since this is not stable yet. > > Hopefully being run more frequently as part of KernelCI will help... > > > > *Using gcc/gcov-6 and not using uml_abort() in os_dump_core(). > > I've documented these hacks in "Notes" but left TODOs for > > brendanhigg...@google.com who tracked down the runtime issue in GCC. > > To be clear: these are not issues specific to KUnit, but rather to UML. > > > > Signed-off-by: Daniel Latypov > > --- > > I'm very happy with this now: all my issues with the previous versions > are addressed. I'm particularly excited to have code coverage > documented somewhere. I just realized I forgot to commit the updated wording you requested wrt uml_abort() in the patch itself. Sending a v4 now. > > Assuming Brendan's happy with the TODOs being there, I think this is > ready to go. > > I also built this with Sphinx and gave it a quick look, and it all > looks good there as well. > > Therefore, this is: > > Reviewed-by: David Gow > > Cheers, > -- David
Re: [PATCH v2] Documentation: dev-tools: Add Testing Overview
On Wed, Apr 14, 2021 at 1:15 AM David Gow wrote: > > The kernel now has a number of testing and debugging tools, and we've > seen a bit of confusion about what the differences between them are. > > Add a basic documentation outlining the testing tools, when to use each, > and how they interact. > > This is a pretty quick overview rather than the idealised "kernel > testing guide" that'd probably be optimal, but given the number of times > questions like "When do you use KUnit and when do you use Kselftest?" > are being asked, it seemed worth at least having something. Hopefully > this can form the basis for more detailed documentation later. > > Signed-off-by: David Gow Reviewed-by: Daniel Latypov Looks good to me. Some minor typos and nits about wording here and there. > --- > Thanks, everyone, for the comments on the doc. I've made a few of the > suggested changes. Please let me know what you think! > > -- David > > Changes since v1: > https://lore.kernel.org/linux-kselftest/20210410070529.4113432-1-david...@google.com/ > - Note KUnit's speed and that one should provide selftests for syscalls > - Mention lockdep as a Dynamic Analysis Tool > - Refer to "Dynamic Analysis Tools" instead of "Sanitizers" > - A number of minor formatting tweaks and rewordings for clarity. > > Not changed: > - I haven't included an exhaustive list of differences, advantages, etc, > between KUnit and kselftest: for now, the doc continues to focus on > the difference between 'in-kernel' and 'userspace' testing here. > - Similarly, I'm not linking out to docs defining and describing "Unit" > tests versus "End-to-end" tests. None of the existing documentation > elsewhere quite matches what we do in the kernel perfectly, so it > seems less confusing to focus on the 'in-kernel'/'userspace' > distinction, and leave other definitions as a passing mention for > those who are already familiar with the concepts. > - I haven't linked to any talk videos here: a few of them are linked on > (e.g.) the KUnit webpage, but I wanted to keep the Kernel documentation > more self-contained for now. No objection to adding them in a follow-up > patch if people feel strongly about it, though. > - The link from index.rst to this doc is unchanged. I personally think > that the link is prominent enough there: it's the first link, and > shows up a few times. One possibility if people disagreed would be to > merge this page with the index, but given not all dev-tools are going > to be testing-related, it seemed a bit arrogant. :-) > > Documentation/dev-tools/index.rst| 3 + > Documentation/dev-tools/testing-overview.rst | 117 +++ > 2 files changed, 120 insertions(+) > create mode 100644 Documentation/dev-tools/testing-overview.rst > > diff --git a/Documentation/dev-tools/index.rst > b/Documentation/dev-tools/index.rst > index 1b1cf4f5c9d9..f590e5860794 100644 > --- a/Documentation/dev-tools/index.rst > +++ b/Documentation/dev-tools/index.rst > @@ -7,6 +7,8 @@ be used to work on the kernel. For now, the documents have > been pulled > together without any significant effort to integrate them into a coherent > whole; patches welcome! > > +A brief overview of testing-specific tools can be found in > :doc:`testing-overview`. > + > .. class:: toc-title > >Table of contents > @@ -14,6 +16,7 @@ whole; patches welcome! > .. toctree:: > :maxdepth: 2 > > + testing-overview > coccinelle > sparse > kcov > diff --git a/Documentation/dev-tools/testing-overview.rst > b/Documentation/dev-tools/testing-overview.rst > new file mode 100644 > index ..ce36a8cdf6b5 > --- /dev/null > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -0,0 +1,117 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > + > +Kernel Testing Guide > + > + > + > +There are a number of different tools for testing the Linux kernel, so > knowing > +when to use each of them can be a challenge. This document provides a rough > +overview of their differences, and how they fit together. > + > + > +Writing and Running Tests > += > + > +The bulk of kernel tests are written using either the kselftest or KUnit > +frameworks. These both provide infrastructure to help make running tests and > +groups of tests easier, as well as providing helpers to aid in writing new > +tests. > + > +If you're looking to verify the behaviour of the Kernel — particularly > specific > +parts of the kernel — then you'll want to use KUnit or kselftest. > + > + > +The Differen
[PATCH v3] Documentation: kunit: add tips for running KUnit
This is long overdue.
There are several things that aren't nailed down (in-tree
.kunitconfig's), or partially broken (GCOV on UML), but having them
documented, warts and all, is better than having nothing.
This covers a bunch of the more recent features
* kunit_filter_glob
* kunit.py run --kunitconfig
* slightly more detail on building tests as modules
* CONFIG_KUNIT_DEBUGFS
By my count, the only headline features now not mentioned are the KASAN
integration and KernelCI json output support (kunit.py run --json).
And then it also discusses how to get code coverage reports under UML
and non-UML since this is a question people have repeatedly asked.
Non-UML coverage collection is no different from normal, but we should
probably explicitly call this out.
As for UML, I was able to get it working again with two small hacks.*
E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
Overall coverage rate:
lines..: 15.1% (18294 of 120776 lines)
functions..: 16.8% (1860 of 11050 functions)
Note: this doesn't document --alltests since this is not stable yet.
Hopefully being run more frequently as part of KernelCI will help...
*Using gcc/gcov-6 and not using uml_abort() in os_dump_core().
I've documented these hacks in "Notes" but left TODOs for
brendanhigg...@google.com who tracked down the runtime issue in GCC.
To be clear: these are not issues specific to KUnit, but rather to UML.
Signed-off-by: Daniel Latypov
---
v2 -> v3:
* Suggest --make_options=CC=/usr/bin/gcc-6 instead of manually editing
kunit_kernel.py
* update instructions on how to remove uml_abort() call
v1 -> v2:
Fix typos, drop --alltests, changed wordiing on config fragments.
---
Documentation/dev-tools/kunit/index.rst | 1 +
.../dev-tools/kunit/running_tips.rst | 258 ++
Documentation/dev-tools/kunit/start.rst | 2 +
3 files changed, 261 insertions(+)
create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
diff --git a/Documentation/dev-tools/kunit/index.rst
b/Documentation/dev-tools/kunit/index.rst
index 848478838347..7f7cf8d2ab20 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
style
faq
tips
+ running_tips
What is KUnit?
==
diff --git a/Documentation/dev-tools/kunit/running_tips.rst
b/Documentation/dev-tools/kunit/running_tips.rst
new file mode 100644
index ..e2e9af711d1b
--- /dev/null
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -0,0 +1,258 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Tips For Running KUnit Tests
+
+
+Using ``kunit.py run`` ("kunit tool")
+=
+
+Running from any directory
+--
+
+It can be handy to create a bash function like:
+
+.. code-block:: bash
+
+ function run_kunit() {
+ ( cd "$(git rev-parse --show-toplevel)" &&
./tools/testing/kunit/kunit.py run $@ )
+ }
+
+.. note::
+ Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
+ the kernel root, hence the use of a subshell and ``cd``.
+
+Running a subset of tests
+-
+
+``kunit.py run`` accepts an optional glob argument to filter tests. Currently
+this only matches against suite names, but this may change in the future.
+
+Say that we wanted to run the sysctl tests, we could do so via:
+
+.. code-block:: bash
+
+ $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' >
.kunit/.kunitconfig
+ $ ./tools/testing/kunit/kunit.py run 'sysctl*'
+
+We're paying the cost of building more tests than we need this way, but it's
+easier than fiddling with ``.kunitconfig`` files or commenting out
+``kunit_suite``'s.
+
+However, if we wanted to define a set of tests in a less ad hoc way, the next
+tip is useful.
+
+Defining a set of tests
+---
+
+``kunit.py run`` (along with ``build``, and ``config``) supports a
+``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
+regular basis (especially if they have other dependencies), you can create a
+specific ``.kunitconfig`` for them.
+
+E.g. kunit has one for its tests:
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run
--kunitconfig=lib/kunit/.kunitconfig
+
+Alternatively, if you're following the convention of naming your
+file ``.kunitconfig``, you can just pass in the dir, e.g.
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
+
+.. note::
+ This is a relatively new feature (5.12+) so we don't have any
+ conventions yet about on what files should be checked in versus just
+ kept around locally. It's up to you and your maintainer to decide if a
+ config is useful enough to submit (and therefore have t
Re: [PATCH v5] lib: add basic KUnit test for lib/math
On Mon, Apr 12, 2021 at 11:41 PM David Gow wrote:
>
> On Tue, Apr 13, 2021 at 3:07 AM Daniel Latypov wrote:
> >
> > Add basic test coverage for files that don't require any config options:
> > * part of math.h (what seem to be the most commonly used macros)
> > * gcd.c
> > * lcm.c
> > * int_sqrt.c
> > * reciprocal_div.c
> > (Ignored int_pow.c since it's a simple textbook algorithm.)
> >
> > These tests aren't particularly interesting, but they
> > * provide short and simple examples of parameterized tests
> > * provide a place to add tests for any new files in this dir
> > * are written so adding new test cases to cover edge cases should be easy
> > * looking at code coverage, we hit all the branches in the .c files
> >
> > Signed-off-by: Daniel Latypov
>
> This looks good to me. A few comments/observations below, but nothing
> that I think should actually block this.
>
> Reviewed-by: David Gow
>
> -- David
>
> > ---
> > Changes since v4:
> > * add in test cases for some math.h macros (abs, round_up/round_down,
> > div_round_down/closest)
> > * use parameterized testing less to keep things terser
> >
> > Changes since v3:
> > * fix `checkpatch.pl --strict` warnings
> > * add test cases for gcd(0,0) and lcm(0,0)
> > * minor: don't test both gcd(a,b) and gcd(b,a) when a == b
> >
> > Changes since v2: mv math_test.c => math_kunit.c
> >
> > Changes since v1:
> > * Rebase and rewrite to use the new parameterized testing support.
> > * misc: fix overflow in literal and inline int_sqrt format string.
> > * related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
> > for testing many inputs") was merged explaining the patterns shown here.
> > * there's an in-flight patch to update it for parameterized testing.
> >
> > v1:
> > https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
> > ---
> > lib/math/Kconfig | 5 +
> > lib/math/Makefile | 2 +
> > lib/math/math_kunit.c | 264 ++
> > 3 files changed, 271 insertions(+)
> > create mode 100644 lib/math/math_kunit.c
> >
> > diff --git a/lib/math/Kconfig b/lib/math/Kconfig
> > index f19bc9734fa7..6ba8680439c1 100644
> > --- a/lib/math/Kconfig
> > +++ b/lib/math/Kconfig
> > @@ -15,3 +15,8 @@ config PRIME_NUMBERS
> >
> > config RATIONAL
> > bool
> > +
> > +config MATH_KUNIT_TEST
> > + tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
> > + default KUNIT_ALL_TESTS
> > + depends on KUNIT
>
> This could have a description of the test and KUnit here, as mentioned
> in the style guide doc:
> https://www.kernel.org/doc/html/latest/dev-tools/kunit/style.html#test-kconfig-entries
>
> (I think it's sufficiently self explanatory that it's not essential,
> but it could be nice to have a more detailed description of the things
> being tested than just "lib/math".)
>
Done. I've left off the details about what the test tests so we have
less places to go and update if/when new tests are added.
> > diff --git a/lib/math/Makefile b/lib/math/Makefile
> > index be6909e943bd..30abb7a8d564 100644
> > --- a/lib/math/Makefile
> > +++ b/lib/math/Makefile
> > @@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
> > reciprocal_div.o
> > obj-$(CONFIG_CORDIC) += cordic.o
> > obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
> > obj-$(CONFIG_RATIONAL) += rational.o
> > +
> > +obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
> > diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
> > new file mode 100644
> > index ..80a087a32884
> > --- /dev/null
> > +++ b/lib/math/math_kunit.c
> > @@ -0,0 +1,264 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Simple KUnit suite for math helper funcs that are always enabled.
> > + *
> > + * Copyright (C) 2020, Google LLC.
> > + * Author: Daniel Latypov
> > + */
> > +
> > +#include
> > +#include
> > +#include
> > +#include
> > +#include
> > +
> > +static void abs_test(struct kunit *test)
> > +{
>
> There's something weird about taking the absolute values of char
> literals. I'm not sure if it's better to case integer literals (like
> with 'short' below), or keep it as-is.
I just thought it was amusing :)
Converting it to be like the short test below.
> > + KUNIT_EXPECT_EQ(test, abs('\0'), '\0');
&g
Re: [PATCH] kunit: add unit test for filtering suites by names
On Mon, Apr 12, 2021 at 10:00 PM David Gow wrote:
>
> On Tue, Apr 13, 2021 at 8:08 AM Daniel Latypov wrote:
> >
> > This adds unit tests for kunit_filter_subsuite() and
> > kunit_filter_suites().
> >
> > Note: what the executor means by "subsuite" is the array of suites
> > corresponding to each test file.
> >
> > This patch lightly refactors executor.c to avoid the use of global
> > variables to make it testable.
> > It also includes a clever `kfree_at_end()` helper that makes this test
> > easier to write than it otherwise would have been.
> >
> > Tested by running just the new tests using itself
> > $ ./tools/testing/kunit/kunit.py run '*exec*'
> >
> > Signed-off-by: Daniel Latypov
>
> I really like this test, thanks.
>
> A few small notes below, including what I think is a missing
> kfree_at_end() call.
>
> Assuming that one issue is fixed (or I'm mistaken):
> Reviewed-by: David Gow
>
> -- David
>
> > ---
> > lib/kunit/executor.c | 26
> > lib/kunit/executor_test.c | 132 ++
> > 2 files changed, 147 insertions(+), 11 deletions(-)
> > create mode 100644 lib/kunit/executor_test.c
> >
> > diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> > index 15832ed44668..96a4ae983786 100644
> > --- a/lib/kunit/executor.c
> > +++ b/lib/kunit/executor.c
> > @@ -19,7 +19,7 @@ MODULE_PARM_DESC(filter_glob,
> > "Filter which KUnit test suites run at boot-time, e.g.
> > list*");
> >
> > static struct kunit_suite * const *
> > -kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
> > +kunit_filter_subsuite(struct kunit_suite * const * const subsuite, const
> > char *filter_glob)
> > {
> > int i, n = 0;
> > struct kunit_suite **filtered;
> > @@ -52,19 +52,14 @@ struct suite_set {
> > struct kunit_suite * const * const *end;
> > };
> >
> > -static struct suite_set kunit_filter_suites(void)
> > +static struct suite_set kunit_filter_suites(const struct suite_set
> > *suite_set,
> > + const char *filter_glob)
> > {
> > int i;
> > struct kunit_suite * const **copy, * const *filtered_subsuite;
> > struct suite_set filtered;
> >
> > - const size_t max = __kunit_suites_end - __kunit_suites_start;
> > -
> > - if (!filter_glob) {
> > - filtered.start = __kunit_suites_start;
> > - filtered.end = __kunit_suites_end;
> > - return filtered;
> > - }
> > + const size_t max = suite_set->end - suite_set->start;
> >
> > copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
> > filtered.start = copy;
> > @@ -74,7 +69,7 @@ static struct suite_set kunit_filter_suites(void)
> > }
> >
> > for (i = 0; i < max; ++i) {
> > - filtered_subsuite =
> > kunit_filter_subsuite(__kunit_suites_start[i]);
> > + filtered_subsuite =
> > kunit_filter_subsuite(suite_set->start[i], filter_glob);
> > if (filtered_subsuite)
> > *copy++ = filtered_subsuite;
> > }
> > @@ -98,8 +93,13 @@ static void kunit_print_tap_header(struct suite_set
> > *suite_set)
> > int kunit_run_all_tests(void)
> > {
> > struct kunit_suite * const * const *suites;
> > + struct suite_set suite_set = {
> > + .start = __kunit_suites_start,
> > + .end = __kunit_suites_end,
> > + };
> >
> > - struct suite_set suite_set = kunit_filter_suites();
> > + if (filter_glob)
> > + suite_set = kunit_filter_suites(_set, filter_glob);
> >
> > kunit_print_tap_header(_set);
> >
> > @@ -115,4 +115,8 @@ int kunit_run_all_tests(void)
> > return 0;
> > }
> >
> > +#if IS_BUILTIN(CONFIG_KUNIT_TEST)
> > +#include "executor_test.c"
> > +#endif
> > +
> > #endif /* IS_BUILTIN(CONFIG_KUNIT) */
> > diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
> > new file mode 100644
> > index ..8e925395beeb
> > --- /dev/null
> > +++ b/lib/kunit/executor_test.c
> > @@ -0,0 +1,132 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * KUnit test for the KUnit executor.
> > + *
> > + * Copyright (C) 2021, Google
Re: [PATCH v4 2/3] mm/slub, kunit: add a KUnit test for SLUB debugging functionality
On Tue, Apr 13, 2021 at 3:07 AM wrote:
>
> From: Oliver Glitta
>
> SLUB has resiliency_test() function which is hidden behind #ifdef
> SLUB_RESILIENCY_TEST that is not part of Kconfig, so nobody
> runs it. KUnit should be a proper replacement for it.
>
> Try changing byte in redzone after allocation and changing
> pointer to next free node, first byte, 50th byte and redzone
> byte. Check if validation finds errors.
>
> There are several differences from the original resiliency test:
> Tests create own caches with known state instead of corrupting
> shared kmalloc caches.
>
> The corruption of freepointer uses correct offset, the original
> resiliency test got broken with freepointer changes.
>
> Scratch changing random byte test, because it does not have
> meaning in this form where we need deterministic results.
>
> Add new option CONFIG_SLUB_KUNIT_TEST in Kconfig.
> Because the test deliberatly modifies non-allocated objects, it depends on
nit: *deliberately
> !KASAN which would have otherwise prevented that.
>
> Use kunit_resource to count errors in cache and silence bug reports.
> Count error whenever slab_bug() or slab_fix() is called or when
> the count of pages is wrong.
>
> Signed-off-by: Oliver Glitta
Acked-by: Daniel Latypov
Looks good to me!
My one minor suggestion: perhaps let's log a summary of the error or
the func name in slab_add_kunit_errors().
> ---
> Changes since v3
>
> Use kunit_resource to silence bug reports and count errors suggested by
> Marco Elver.
> Make the test depends on !KASAN thanks to report from the kernel test robot.
>
> Changes since v2
>
> Use bit operation & instead of logical && as reported by kernel test
> robot and Dan Carpenter
>
> Changes since v1
>
> Conversion from kselftest to KUnit test suggested by Marco Elver.
> Error silencing.
> Error counting improvements.
> lib/Kconfig.debug | 12
> lib/Makefile | 1 +
> lib/slub_kunit.c | 150 ++
> mm/slab.h | 1 +
> mm/slub.c | 50 ++--
> 5 files changed, 209 insertions(+), 5 deletions(-)
> create mode 100644 lib/slub_kunit.c
>
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 2779c29d9981..9b8a0d754278 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -2371,6 +2371,18 @@ config BITS_TEST
>
> If unsure, say N.
>
> +config SLUB_KUNIT_TEST
> + tristate "KUnit test for SLUB cache error detection" if
> !KUNIT_ALL_TESTS
> + depends on SLUB_DEBUG && KUNIT && !KASAN
> + default KUNIT_ALL_TESTS
> + help
> + This builds SLUB allocator unit test.
> + Tests SLUB cache debugging functionality.
> + For more information on KUnit and unit tests in general please refer
> + to the KUnit documentation in Documentation/dev-tools/kunit/.
> +
> + If unsure, say N.
> +
> config TEST_UDELAY
> tristate "udelay test driver"
> help
> diff --git a/lib/Makefile b/lib/Makefile
> index b5307d3eec1a..1e59c6714ed8 100644
> --- a/lib/Makefile
> +++ b/lib/Makefile
> @@ -352,5 +352,6 @@ obj-$(CONFIG_LIST_KUNIT_TEST) += list-test.o
> obj-$(CONFIG_LINEAR_RANGES_TEST) += test_linear_ranges.o
> obj-$(CONFIG_BITS_TEST) += test_bits.o
> obj-$(CONFIG_CMDLINE_KUNIT_TEST) += cmdline_kunit.o
> +obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o
>
> obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) += devmem_is_allowed.o
> diff --git a/lib/slub_kunit.c b/lib/slub_kunit.c
> new file mode 100644
> index ..cb9ae9f7e8a6
> --- /dev/null
> +++ b/lib/slub_kunit.c
> @@ -0,0 +1,150 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include "../mm/slab.h"
> +
> +static struct kunit_resource resource;
> +static int slab_errors;
> +
> +static void test_clobber_zone(struct kunit *test)
> +{
> + struct kmem_cache *s = kmem_cache_create("TestSlub_RZ_alloc", 64, 0,
> + SLAB_RED_ZONE, NULL);
> + u8 *p = kmem_cache_alloc(s, GFP_KERNEL);
> +
> + p[64] = 0x12;
> +
> + validate_slab_cache(s);
> + KUNIT_EXPECT_EQ(test, 2, slab_errors);
> +
> + kmem_cache_free(s, p);
> + kmem_cache_destroy(s);
Might not be worth doing for now:
I see kmem_cache_destroy() has a `if (err) { pr_err(...); dump_stack(); }` call.
Does it make sense to cause that to fail the test?
I see it's defined in mm/slab_common.c, so we might not want to touch
that in this patch.
> +}
> +
> +static void test_next_pointer(struct kunit *te
[PATCH v2] Documentation: kunit: add tips for running KUnit
This is long overdue.
There are several things that aren't nailed down (in-tree
.kunitconfig's), or partially broken (GCOV on UML), but having them
documented, warts and all, is better than having nothing.
This covers a bunch of the more recent features
* kunit_filter_glob
* kunit.py run --kunitconfig
* slightly more detail on building tests as modules
* CONFIG_KUNIT_DEBUGFS
By my count, the only headline features now not mentioned are the KASAN
integration and KernelCI json output support (kunit.py run --json).
And then it also discusses how to get code coverage reports under UML
and non-UML since this is a question people have repeatedly asked.
Non-UML coverage collection is no different from normal, but we should
probably explicitly call this out.
As for UML, I was able to get it working again with two small hacks.*
E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
Overall coverage rate:
lines..: 15.1% (18294 of 120776 lines)
functions..: 16.8% (1860 of 11050 functions)
Note: this doesn't document --alltests since this is not stable yet.
Hopefully being run more frequently as part of KernelCI will help...
*Using gcc/gcov-6 and not using uml_abort() in os_dump_core().
I've documented these hacks in "Notes" but left TODOs for
brendanhigg...@google.com who tracked down the runtime issue in GCC.
To be clear: these are not issues specific to KUnit, but rather to UML.
Signed-off-by: Daniel Latypov
---
Documentation/dev-tools/kunit/index.rst | 1 +
.../dev-tools/kunit/running_tips.rst | 260 ++
Documentation/dev-tools/kunit/start.rst | 2 +
3 files changed, 263 insertions(+)
create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
diff --git a/Documentation/dev-tools/kunit/index.rst
b/Documentation/dev-tools/kunit/index.rst
index 848478838347..7f7cf8d2ab20 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
style
faq
tips
+ running_tips
What is KUnit?
==
diff --git a/Documentation/dev-tools/kunit/running_tips.rst
b/Documentation/dev-tools/kunit/running_tips.rst
new file mode 100644
index ..52cc62d1c83b
--- /dev/null
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -0,0 +1,260 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Tips For Running KUnit Tests
+
+
+Using ``kunit.py run`` ("kunit tool")
+=
+
+Running from any directory
+--
+
+It can be handy to create a bash function like:
+
+.. code-block:: bash
+
+ function run_kunit() {
+ ( cd "$(git rev-parse --show-toplevel)" &&
./tools/testing/kunit/kunit.py run $@ )
+ }
+
+.. note::
+ Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
+ the kernel root, hence the use of a subshell and ``cd``.
+
+Running a subset of tests
+-
+
+``kunit.py run`` accepts an optional glob argument to filter tests. Currently
+this only matches against suite names, but this may change in the future.
+
+Say that we wanted to run the sysctl tests, we could do so via:
+
+.. code-block:: bash
+
+ $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' >
.kunit/.kunitconfig
+ $ ./tools/testing/kunit/kunit.py run 'sysctl*'
+
+We're paying the cost of building more tests than we need this way, but it's
+easier than fiddling with ``.kunitconfig`` files or commenting out
+``kunit_suite``'s.
+
+However, if we wanted to define a set of tests in a less ad hoc way, the next
+tip is useful.
+
+Defining a set of tests
+---
+
+``kunit.py run`` (along with ``build``, and ``config``) supports a
+``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
+regular basis (especially if they have other dependencies), you can create a
+specific ``.kunitconfig`` for them.
+
+E.g. kunit has one for its tests:
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run
--kunitconfig=lib/kunit/.kunitconfig
+
+Alternatively, if you're following the convention of naming your
+file ``.kunitconfig``, you can just pass in the dir, e.g.
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
+
+.. note::
+ This is a relatively new feature (5.12+) so we don't have any
+ conventions yet about on what files should be checked in versus just
+ kept around locally. It's up to you and your maintainer to decide if a
+ config is useful enough to submit (and therefore have to maintain).
+
+.. note::
+ Having ``.kunitconfig`` fragments in a parent and child directory is
+ iffy. There's discussion about adding an "import" statement in these
+ files to make it possible to have a to
[PATCH] kunit: add unit test for filtering suites by names
This adds unit tests for kunit_filter_subsuite() and
kunit_filter_suites().
Note: what the executor means by "subsuite" is the array of suites
corresponding to each test file.
This patch lightly refactors executor.c to avoid the use of global
variables to make it testable.
It also includes a clever `kfree_at_end()` helper that makes this test
easier to write than it otherwise would have been.
Tested by running just the new tests using itself
$ ./tools/testing/kunit/kunit.py run '*exec*'
Signed-off-by: Daniel Latypov
---
lib/kunit/executor.c | 26
lib/kunit/executor_test.c | 132 ++
2 files changed, 147 insertions(+), 11 deletions(-)
create mode 100644 lib/kunit/executor_test.c
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 15832ed44668..96a4ae983786 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -19,7 +19,7 @@ MODULE_PARM_DESC(filter_glob,
"Filter which KUnit test suites run at boot-time, e.g. list*");
static struct kunit_suite * const *
-kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite, const char
*filter_glob)
{
int i, n = 0;
struct kunit_suite **filtered;
@@ -52,19 +52,14 @@ struct suite_set {
struct kunit_suite * const * const *end;
};
-static struct suite_set kunit_filter_suites(void)
+static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
+ const char *filter_glob)
{
int i;
struct kunit_suite * const **copy, * const *filtered_subsuite;
struct suite_set filtered;
- const size_t max = __kunit_suites_end - __kunit_suites_start;
-
- if (!filter_glob) {
- filtered.start = __kunit_suites_start;
- filtered.end = __kunit_suites_end;
- return filtered;
- }
+ const size_t max = suite_set->end - suite_set->start;
copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
filtered.start = copy;
@@ -74,7 +69,7 @@ static struct suite_set kunit_filter_suites(void)
}
for (i = 0; i < max; ++i) {
- filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ filtered_subsuite = kunit_filter_subsuite(suite_set->start[i],
filter_glob);
if (filtered_subsuite)
*copy++ = filtered_subsuite;
}
@@ -98,8 +93,13 @@ static void kunit_print_tap_header(struct suite_set
*suite_set)
int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
+ struct suite_set suite_set = {
+ .start = __kunit_suites_start,
+ .end = __kunit_suites_end,
+ };
- struct suite_set suite_set = kunit_filter_suites();
+ if (filter_glob)
+ suite_set = kunit_filter_suites(_set, filter_glob);
kunit_print_tap_header(_set);
@@ -115,4 +115,8 @@ int kunit_run_all_tests(void)
return 0;
}
+#if IS_BUILTIN(CONFIG_KUNIT_TEST)
+#include "executor_test.c"
+#endif
+
#endif /* IS_BUILTIN(CONFIG_KUNIT) */
diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
new file mode 100644
index ..8e925395beeb
--- /dev/null
+++ b/lib/kunit/executor_test.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for the KUnit executor.
+ *
+ * Copyright (C) 2021, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+
+static void kfree_at_end(struct kunit *test, const void *to_free);
+static struct kunit_suite *alloc_fake_suite(struct kunit *test,
+ const char *suite_name);
+
+static void filter_subsuite_test(struct kunit *test)
+{
+ struct kunit_suite *subsuite[3] = {NULL, NULL, NULL};
+ struct kunit_suite * const *filtered;
+
+ subsuite[0] = alloc_fake_suite(test, "suite1");
+ subsuite[1] = alloc_fake_suite(test, "suite2");
+
+ /* Want: suite1, suite2, NULL -> suite2, NULL */
+ filtered = kunit_filter_subsuite(subsuite, "suite2*");
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, filtered);
+ kfree_at_end(test, filtered);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, filtered[0]);
+ KUNIT_EXPECT_STREQ(test, (const char *)filtered[0]->name, "suite2");
+
+ KUNIT_EXPECT_FALSE(test, filtered[1]);
+}
+
+static void filter_subsuite_to_empty_test(struct kunit *test)
+{
+ struct kunit_suite *subsuite[3] = {NULL, NULL, NULL};
+ struct kunit_suite * const *filtered;
+
+ subsuite[0] = alloc_fake_suite(test, "suite1");
+ subsuite[1] = alloc_fake_suite(test, "suite2");
+
+ filtered = kunit_filter_subsuite(subsuite, "not_found");
+ kfree_at_end(test, filtered); /* just in case */
+
+ KUNI
Re: [PATCH] Documentation: kunit: add tips for running KUnit
On Mon, Apr 12, 2021 at 1:42 PM Brendan Higgins wrote: > > On Mon, Apr 12, 2021 at 10:27 AM Daniel Latypov wrote: > > > > hOn Fri, Apr 9, 2021 at 9:10 PM David Gow wrote: > > > > > > Thanks for writing this: it's good to have these things documented at > > > last! > > > > > > There are definitely a few things this document points out which still > > > need deciding, which does make this document lean a bit into "design > > > discussion" territory in a few of the notes. This doesn't bother me -- > > > it's an accurate description of the state of things -- but I wouldn't > > > want this documentation held up too long because of these sorts of > > > TODOs (and can definitely see how having too many of them might > > > discourage KUnit use a bit). Particularly things like the > > > ".kunitconfig" fragment file feature stuff: I feel that's something > > > better discussed on patches adding/using the feature than in the > > > documentation / reviews of the documentation, so I'd rather drop or > > > simplify those '..note:'s than bokeshed about it here (something I'm a > > > little guilty of below). > > > > I don't think we'll actually make progress on any of those in the near > > future though. > > So I figured it'd be best to accurately represent the state of the > > world ~somewhere. > > > > But it did feel a bit strange to do it here, so I'm not against removing it. > > I actually like the accurate and upfront way that you spelled these things > out. > > > > Otherwise, a few minor comments and nitpicks: > > > > > > -- David > > > > > > On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov > > > wrote: > > > > > > > > This is long overdue. > > > > > > > > There are several things that aren't nailed down (in-tree > > > > .kunitconfig's), or partially broken (GCOV on UML), but having them > > > > documented, warts and all, is better than having nothing. > > > > > > > > This covers a bunch of the more recent features > > > > * kunit_filter_glob > > > > * kunit.py run --kunitconfig > > > > * kunit.py run --alltests > > > > * slightly more detail on building tests as modules > > > > * CONFIG_KUNIT_DEBUGFS > > > > > > > > By my count, the only headline features now not mentioned are the KASAN > > > > integration and KernelCI json output support (kunit.py run --json). > > > > > > > > And then it also discusses how to get code coverage reports under UML > > > > and non-UML since this is a question people have repeatedly asked. > > > > > > > > Non-UML coverage collection is no differnt from normal, but we should > > > > probably explicitly call thsi out. > > > > > > Nit: typos in 'different' and 'this'. > > Fixed. > > > > > > > > > > > As for UML, I was able to get it working again with two small hacks.* > > > > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > > > > Overall coverage rate: > > > > lines..: 15.1% (18294 of 120776 lines) > > > > functions..: 16.8% (1860 of 11050 functions) > > > > > > > > *Switching to use gcc/gcov-6 and not using uml_abort(). > > > > I've documented these hacks in "Notes" but left TODOs for > > > > brendanhigg...@google.com who tracked down the runtime issue in GCC. > > > > To be clear: these are not issues specific to KUnit, but rather to UML. > > > > > > (We should probably note where uml_abort() needs to be replaced if > > > we're mentioning this, though doing so below in the more detailed > > > section may be more useful.) > > > > Updated to > > *Using gcc/gcov-6 and not using uml_abort() in os_dump_core(). > > > > I figured we'd be more precise in the documentation itself. > > > > > > > > > > > > > Signed-off-by: Daniel Latypov > > > > --- > > > > Documentation/dev-tools/kunit/index.rst | 1 + > > > > .../dev-tools/kunit/running_tips.rst | 278 ++ > > > > Documentation/dev-tools/kunit/start.rst | 2 + > > > > 3 files changed, 281 insertions(+) > > > > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst > > > > > > > > diff --git a/Documentation/dev-tools/kunit/index.rst > > &g
[PATCH v5] lib: add basic KUnit test for lib/math
Add basic test coverage for files that don't require any config options:
* part of math.h (what seem to be the most commonly used macros)
* gcd.c
* lcm.c
* int_sqrt.c
* reciprocal_div.c
(Ignored int_pow.c since it's a simple textbook algorithm.)
These tests aren't particularly interesting, but they
* provide short and simple examples of parameterized tests
* provide a place to add tests for any new files in this dir
* are written so adding new test cases to cover edge cases should be easy
* looking at code coverage, we hit all the branches in the .c files
Signed-off-by: Daniel Latypov
---
Changes since v4:
* add in test cases for some math.h macros (abs, round_up/round_down,
div_round_down/closest)
* use parameterized testing less to keep things terser
Changes since v3:
* fix `checkpatch.pl --strict` warnings
* add test cases for gcd(0,0) and lcm(0,0)
* minor: don't test both gcd(a,b) and gcd(b,a) when a == b
Changes since v2: mv math_test.c => math_kunit.c
Changes since v1:
* Rebase and rewrite to use the new parameterized testing support.
* misc: fix overflow in literal and inline int_sqrt format string.
* related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
for testing many inputs") was merged explaining the patterns shown here.
* there's an in-flight patch to update it for parameterized testing.
v1: https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
---
lib/math/Kconfig | 5 +
lib/math/Makefile | 2 +
lib/math/math_kunit.c | 264 ++
3 files changed, 271 insertions(+)
create mode 100644 lib/math/math_kunit.c
diff --git a/lib/math/Kconfig b/lib/math/Kconfig
index f19bc9734fa7..6ba8680439c1 100644
--- a/lib/math/Kconfig
+++ b/lib/math/Kconfig
@@ -15,3 +15,8 @@ config PRIME_NUMBERS
config RATIONAL
bool
+
+config MATH_KUNIT_TEST
+ tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ depends on KUNIT
diff --git a/lib/math/Makefile b/lib/math/Makefile
index be6909e943bd..30abb7a8d564 100644
--- a/lib/math/Makefile
+++ b/lib/math/Makefile
@@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
reciprocal_div.o
obj-$(CONFIG_CORDIC) += cordic.o
obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
obj-$(CONFIG_RATIONAL) += rational.o
+
+obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
new file mode 100644
index ..80a087a32884
--- /dev/null
+++ b/lib/math/math_kunit.c
@@ -0,0 +1,264 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple KUnit suite for math helper funcs that are always enabled.
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+static void abs_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, abs('\0'), '\0');
+ KUNIT_EXPECT_EQ(test, abs('a'), 'a');
+ KUNIT_EXPECT_EQ(test, abs(-'a'), 'a');
+
+ /* The expression in the macro is actually promoted to an int. */
+ KUNIT_EXPECT_EQ(test, abs((short)0), 0);
+ KUNIT_EXPECT_EQ(test, abs((short)42), 42);
+ KUNIT_EXPECT_EQ(test, abs((short)-42), 42);
+
+ KUNIT_EXPECT_EQ(test, abs(0), 0);
+ KUNIT_EXPECT_EQ(test, abs(42), 42);
+ KUNIT_EXPECT_EQ(test, abs(-42), 42);
+
+ KUNIT_EXPECT_EQ(test, abs(0L), 0L);
+ KUNIT_EXPECT_EQ(test, abs(42L), 42L);
+ KUNIT_EXPECT_EQ(test, abs(-42L), 42L);
+
+ KUNIT_EXPECT_EQ(test, abs(0LL), 0LL);
+ KUNIT_EXPECT_EQ(test, abs(42LL), 42LL);
+ KUNIT_EXPECT_EQ(test, abs(-42LL), 42LL);
+
+ /* Unsigned types get casted to signed. */
+ KUNIT_EXPECT_EQ(test, abs(0ULL), 0LL);
+ KUNIT_EXPECT_EQ(test, abs(42ULL), 42LL);
+}
+
+static void int_sqrt_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, int_sqrt(0UL), 0UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(1UL), 1UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(4UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(5UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(8UL), 2UL);
+ KUNIT_EXPECT_EQ(test, int_sqrt(1UL << 30), 1UL << 15);
+}
+
+static void round_up_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, round_up(0, 1), 0);
+ KUNIT_EXPECT_EQ(test, round_up(1, 2), 2);
+ KUNIT_EXPECT_EQ(test, round_up(3, 2), 4);
+ KUNIT_EXPECT_EQ(test, round_up((1 << 30) - 1, 2), 1 << 30);
+ KUNIT_EXPECT_EQ(test, round_up((1 << 30) - 1, 1 << 29), 1 << 30);
+}
+
+static void round_down_test(struct kunit *test)
+{
+ KUNIT_EXPECT_EQ(test, round_down(0, 1), 0);
+ KUNIT_EXPECT_EQ(test, round_down(1, 2), 0);
+ KUNIT_EXPECT_EQ(test, round_down(3, 2), 2);
+ KUNIT_EXPECT_EQ(test, round_down((1 << 30) - 1, 2), (1 << 30) - 2);
+ KUNIT_EXPECT_EQ(test, round_down((1 << 30) - 1, 1 << 29), 1 <&l
Re: [PATCH] Documentation: kunit: add tips for running KUnit
hOn Fri, Apr 9, 2021 at 9:10 PM David Gow wrote:
>
> Thanks for writing this: it's good to have these things documented at last!
>
> There are definitely a few things this document points out which still
> need deciding, which does make this document lean a bit into "design
> discussion" territory in a few of the notes. This doesn't bother me --
> it's an accurate description of the state of things -- but I wouldn't
> want this documentation held up too long because of these sorts of
> TODOs (and can definitely see how having too many of them might
> discourage KUnit use a bit). Particularly things like the
> ".kunitconfig" fragment file feature stuff: I feel that's something
> better discussed on patches adding/using the feature than in the
> documentation / reviews of the documentation, so I'd rather drop or
> simplify those '..note:'s than bokeshed about it here (something I'm a
> little guilty of below).
I don't think we'll actually make progress on any of those in the near
future though.
So I figured it'd be best to accurately represent the state of the
world ~somewhere.
But it did feel a bit strange to do it here, so I'm not against removing it.
>
> Otherwise, a few minor comments and nitpicks:
>
> -- David
>
> On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov wrote:
> >
> > This is long overdue.
> >
> > There are several things that aren't nailed down (in-tree
> > .kunitconfig's), or partially broken (GCOV on UML), but having them
> > documented, warts and all, is better than having nothing.
> >
> > This covers a bunch of the more recent features
> > * kunit_filter_glob
> > * kunit.py run --kunitconfig
> > * kunit.py run --alltests
> > * slightly more detail on building tests as modules
> > * CONFIG_KUNIT_DEBUGFS
> >
> > By my count, the only headline features now not mentioned are the KASAN
> > integration and KernelCI json output support (kunit.py run --json).
> >
> > And then it also discusses how to get code coverage reports under UML
> > and non-UML since this is a question people have repeatedly asked.
> >
> > Non-UML coverage collection is no differnt from normal, but we should
> > probably explicitly call thsi out.
>
> Nit: typos in 'different' and 'this'.
Fixed.
>
> >
> > As for UML, I was able to get it working again with two small hacks.*
> > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
> > Overall coverage rate:
> > lines..: 15.1% (18294 of 120776 lines)
> > functions..: 16.8% (1860 of 11050 functions)
> >
> > *Switching to use gcc/gcov-6 and not using uml_abort().
> > I've documented these hacks in "Notes" but left TODOs for
> > brendanhigg...@google.com who tracked down the runtime issue in GCC.
> > To be clear: these are not issues specific to KUnit, but rather to UML.
>
> (We should probably note where uml_abort() needs to be replaced if
> we're mentioning this, though doing so below in the more detailed
> section may be more useful.)
Updated to
*Using gcc/gcov-6 and not using uml_abort() in os_dump_core().
I figured we'd be more precise in the documentation itself.
>
> >
> > Signed-off-by: Daniel Latypov
> > ---
> > Documentation/dev-tools/kunit/index.rst | 1 +
> > .../dev-tools/kunit/running_tips.rst | 278 ++
> > Documentation/dev-tools/kunit/start.rst | 2 +
> > 3 files changed, 281 insertions(+)
> > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
> >
> > diff --git a/Documentation/dev-tools/kunit/index.rst
> > b/Documentation/dev-tools/kunit/index.rst
> > index 848478838347..7f7cf8d2ab20 100644
> > --- a/Documentation/dev-tools/kunit/index.rst
> > +++ b/Documentation/dev-tools/kunit/index.rst
> > @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
> > style
> > faq
> > tips
> > + running_tips
> >
> > What is KUnit?
> > ==
> > diff --git a/Documentation/dev-tools/kunit/running_tips.rst
> > b/Documentation/dev-tools/kunit/running_tips.rst
> > new file mode 100644
> > index ..d38e665e530f
> > --- /dev/null
> > +++ b/Documentation/dev-tools/kunit/running_tips.rst
> > @@ -0,0 +1,278 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +
> > +Tips For Running KUnit Tests
> > +
> > +
> > +Using ``kunit.py run`` ("kunit tool")
> > +=
> > +
> > +Running from any directory
> > +-
Re: [PATCH] Documentation: dev-tools: Add Testing Overview
On Sat, Apr 10, 2021 at 12:05 AM David Gow wrote: > > The kernel now has a number of testing and debugging tools, and we've > seen a bit of confusion about what the differences between them are. > > Add a basic documentation outlining the testing tools, when to use each, > and how they interact. > > This is a pretty quick overview rather than the idealised "kernel > testing guide" that'd probably be optimal, but given the number of times > questions like "When do you use KUnit and when do you use Kselftest?" > are being asked, it seemed worth at least having something. Hopefully > this can form the basis for more detailed documentation later. > > Signed-off-by: David Gow > --- > Documentation/dev-tools/index.rst| 3 + > Documentation/dev-tools/testing-overview.rst | 102 +++ > 2 files changed, 105 insertions(+) > create mode 100644 Documentation/dev-tools/testing-overview.rst > > diff --git a/Documentation/dev-tools/index.rst > b/Documentation/dev-tools/index.rst > index 1b1cf4f5c9d9..f590e5860794 100644 > --- a/Documentation/dev-tools/index.rst > +++ b/Documentation/dev-tools/index.rst > @@ -7,6 +7,8 @@ be used to work on the kernel. For now, the documents have > been pulled > together without any significant effort to integrate them into a coherent > whole; patches welcome! > > +A brief overview of testing-specific tools can be found in > :doc:`testing-overview`. > + > .. class:: toc-title > >Table of contents > @@ -14,6 +16,7 @@ whole; patches welcome! > .. toctree:: > :maxdepth: 2 > > + testing-overview > coccinelle > sparse > kcov > diff --git a/Documentation/dev-tools/testing-overview.rst > b/Documentation/dev-tools/testing-overview.rst > new file mode 100644 > index ..8452adcb8608 > --- /dev/null > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -0,0 +1,102 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > + > +Kernel Testing Guide > + > + > + > +There are a number of different tools for testing the Linux kernel, so > knowing > +when to use each of them can be a challenge. This document provides a rough > +overview of their differences, and how they fit together. > + > + > +Writing and Running Tests > += > + > +The bulk of kernel tests are written using either the :doc:`kselftest > +` or :doc:`KUnit ` frameworks. These both provide > +infrastructure to help make running tests and groups of tests easier, as well > +as providing helpers to aid in writing new tests. > + > +If you're looking to verify the behaviour of the Kernel — particularly > specific > +parts of the kernel — then you'll want to use `KUnit` or `kselftest`. > + > + > +The Difference Between KUnit and kselftest > +-- This section does a good job, but on a pragmatic level, there are a few more reasons to pick one or the other. E.g. the edit/build/test cycle will likely always be faster in KUnit. I'd also initially drafted up a _very_ long list of reasons to prefer kselftest as well. But looking back at them, a lot will hopefully be mitigated soon, or naturally get better with more usage/time, and you touched on that it can be easier to set up state from userspace already down below. > + > +:doc:`KUnit ` is an entirely in-kernel system for "white box" > +testing: because test code is part of the kernel, it can access internal > +structures and functions which aren't exposed to userspace. > + > +`KUnit` tests therefore are best written against small, self-contained parts > +of the kernel, which can be tested in isolation. This aligns well with the > +concept of Unit testing. Nit: we have "Unit testing" here and "'system' or 'end-to-end' testing." Perhaps: 'unit' testing > + > +For example, a KUnit test might test an individual kernel function (or even a > +single codepath through a function, such as an error handling case), rather > +than a feature as a whole. > + > +There is a KUnit test style guide which may give further pointers Seems like this sentence got truncated? Hmm, I'm not sure what this would be referring to however. I'm not sure there's a doc that touches on what's amenable to being unit tested. > + > + > +:doc:`kselftest `, on the other hand, is largely implemented in > +userspace, and tests are normal userspace scripts or programs. > + > +This makes it easier to write more complicated tests, or tests which need to > +manipulate the overall system state more (e.g., spawning processes, etc.). > +However, it's not possible to call kernel functions directly unless they're Saying it's not possible to call kernel code before mentioning the use of kernel modules to call kernel code directly is a bit confusing. Perhaps instead: However, it's not possible to call kernel functions directly unless you write a companion kernel module for the test. If your test is mostly or entirely inside a kernel module, `KUnit` may be the better
[PATCH] Documentation: kunit: add tips for running KUnit
This is long overdue.
There are several things that aren't nailed down (in-tree
.kunitconfig's), or partially broken (GCOV on UML), but having them
documented, warts and all, is better than having nothing.
This covers a bunch of the more recent features
* kunit_filter_glob
* kunit.py run --kunitconfig
* kunit.py run --alltests
* slightly more detail on building tests as modules
* CONFIG_KUNIT_DEBUGFS
By my count, the only headline features now not mentioned are the KASAN
integration and KernelCI json output support (kunit.py run --json).
And then it also discusses how to get code coverage reports under UML
and non-UML since this is a question people have repeatedly asked.
Non-UML coverage collection is no differnt from normal, but we should
probably explicitly call thsi out.
As for UML, I was able to get it working again with two small hacks.*
E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
Overall coverage rate:
lines..: 15.1% (18294 of 120776 lines)
functions..: 16.8% (1860 of 11050 functions)
*Switching to use gcc/gcov-6 and not using uml_abort().
I've documented these hacks in "Notes" but left TODOs for
brendanhigg...@google.com who tracked down the runtime issue in GCC.
To be clear: these are not issues specific to KUnit, but rather to UML.
Signed-off-by: Daniel Latypov
---
Documentation/dev-tools/kunit/index.rst | 1 +
.../dev-tools/kunit/running_tips.rst | 278 ++
Documentation/dev-tools/kunit/start.rst | 2 +
3 files changed, 281 insertions(+)
create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
diff --git a/Documentation/dev-tools/kunit/index.rst
b/Documentation/dev-tools/kunit/index.rst
index 848478838347..7f7cf8d2ab20 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
style
faq
tips
+ running_tips
What is KUnit?
==
diff --git a/Documentation/dev-tools/kunit/running_tips.rst
b/Documentation/dev-tools/kunit/running_tips.rst
new file mode 100644
index ..d38e665e530f
--- /dev/null
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -0,0 +1,278 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Tips For Running KUnit Tests
+
+
+Using ``kunit.py run`` ("kunit tool")
+=
+
+Running from any directory
+--
+
+It can be handy to create a bash function like:
+
+.. code-block:: bash
+
+ function run_kunit() {
+ ( cd "$(git rev-parse --show-toplevel)" &&
./tools/testing/kunit/kunit.py run $@ )
+ }
+
+.. note::
+ Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
+ the kernel root, hence the use of a subshell and ``cd``.
+
+Running a subset of tests
+-
+
+``kunit.py run`` accepts an optional glob argument to filter tests. Currently
+this only matches against suite names, but this may change in the future.
+
+Say that we wanted to run the sysctl tests, we could do so via:
+
+.. code-block:: bash
+
+ $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' >
.kunit/.kunitconfig
+ $ ./tools/testing/kunit/kunit.py run 'sysctl*'
+
+We're paying the cost of building more tests than we need this way, but it's
+easier than fiddling with ``.kunitconfig`` files or commenting out
+``kunit_suite``'s.
+
+However, if we wanted to define a set of tests in a less ad hoc way, the next
+tip is useful.
+
+Defining a set of tests
+---
+
+``kunit.py run`` (along with ``build``, and ``config``) supports a
+``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
+regular basis (especially if they have other dependencies), you can create a
+specific ``.kunitconfig`` for them.
+
+E.g. kunit has own for its tests:
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run
--kunitconfig=lib/kunit/.kunitconfig
+
+Alternatively, if you're following the convention of naming your
+file ``.kunitconfig``, you can just pass in the dir, e.g.
+
+.. code-block:: bash
+
+ $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
+
+.. note::
+ This is a relatively new feature (5.12+) so we don't have any
+ conventions yet about on what files should be checked in versus just
+ kept around locally. But if the tests don't have any dependencies
+ (beyond ``CONFIG_KUNIT``), it's probably not worth writing and
+ maintaining a ``.kunitconfig`` fragment. Running with
+ ``CONFIG_KUNIT_ALL_TESTS=y`` is probably easier.
+
+.. note::
+ Having ``.kunitconfig`` fragments in a parent and child directory is
+ iffy. There's discussion about adding an "import" statement in these
+ files to make it possible to have a top-level config run te
Re: [PATCH v4] lib: add basic KUnit test for lib/math
On Fri, Apr 9, 2021 at 8:30 AM Andy Shevchenko
wrote:
>
> On Thu, Apr 08, 2021 at 06:40:01PM -0700, Daniel Latypov wrote:
> > Add basic test coverage for files that don't require any config options:
> > * gcd.c
> > * lcm.c
> > * int_sqrt.c
> > * reciprocal_div.c
> > (Ignored int_pow.c since it's a simple textbook algorithm.)
>
> What about adding math.h test cases?
>
> We have some macros there and it might be a good idea to test them, for
> example
> that round_up() and roundup() produces the same output for the same (power of
> two divisor) input.
I completely overlooked the macros in math.h, sounds like a good idea.
Grepping around, seems like abs() and DIV_ROUND_UP/CLOSEST() are among
the more popular macros:
$ ag -s '\bDIV_ROUND_UP\(' | wc -l
2946
$ ag -s '\babs\(' | wc -l
923
$ ag -s '\bDIV_ROUND_CLOSEST\(' | wc -l
864
$ ag -s '\bround_up\(' | wc -l
727
$ ag -s '\broundup\(' | wc -l
620
$ ag -s '\bround_down\(' | wc -l
371
$ ag -s 'rounddown\(' | wc -l
131
>
> > These tests aren't particularly interesting, but they
> > * provide short and simple examples of parameterized tests
> > * provide a place to add tests for any new files in this dir
> > * are written so adding new test cases to cover edge cases should be easy
>
> Yes, that's why I think macros also can be a good example how to test *macro*.
Yeah, there's more to cover there since they have a range of types
they can work on.
On another note, the parameterized test arrays all use unsigned long,
so abs() sticks out even more.
I'm thinking of something like
static void test_abs(struct kunit *test)
{
KUNIT_EXPECT_EQ(test, abs('a'), 'a');
KUNIT_EXPECT_EQ(test, abs(-'a'), 'a');
...
}
and then maybe use parameters for the other macros but also throw in
an additional test case like
static void test_div_round_up_diff_types(struct kunit *test)
{
KUNIT_EXPECT_EQ(test, DIV_ROUND_UP((char) 42, (char) 10), (char) 4);
KUNIT_EXPECT_EQ(test, DIV_ROUND_UP((int) 42, (int) 10), (int) 4);
KUNIT_EXPECT_EQ(test, DIV_ROUND_UP((long) 42, (long) 10), (long) 4);
...
}
>
> --
> With Best Regards,
> Andy Shevchenko
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kunit-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kunit-dev/YHBzA7SwH194ywRv%40smile.fi.intel.com.
[PATCH v4] lib: add basic KUnit test for lib/math
Add basic test coverage for files that don't require any config options:
* gcd.c
* lcm.c
* int_sqrt.c
* reciprocal_div.c
(Ignored int_pow.c since it's a simple textbook algorithm.)
These tests aren't particularly interesting, but they
* provide short and simple examples of parameterized tests
* provide a place to add tests for any new files in this dir
* are written so adding new test cases to cover edge cases should be easy
Signed-off-by: Daniel Latypov
---
Changes since v3:
* fix `checkpatch.pl --strict` warnings
* add test cases for gcd(0,0) and lcm(0,0)
* minor: don't test both gcd(a,b) and gcd(b,a) when a == b
Changes since v2: mv math_test.c => math_kunit.c
Changes since v1:
* Rebase and rewrite to use the new parameterized testing support.
* misc: fix overflow in literal and inline int_sqrt format string.
* related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
for testing many inputs") was merged explaining the patterns shown here.
* there's an in-flight patch to update it for parameterized testing.
v1: https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
---
lib/math/Kconfig | 5 +
lib/math/Makefile | 2 +
lib/math/math_kunit.c | 214 ++
3 files changed, 221 insertions(+)
create mode 100644 lib/math/math_kunit.c
diff --git a/lib/math/Kconfig b/lib/math/Kconfig
index f19bc9734fa7..6ba8680439c1 100644
--- a/lib/math/Kconfig
+++ b/lib/math/Kconfig
@@ -15,3 +15,8 @@ config PRIME_NUMBERS
config RATIONAL
bool
+
+config MATH_KUNIT_TEST
+ tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ depends on KUNIT
diff --git a/lib/math/Makefile b/lib/math/Makefile
index be6909e943bd..30abb7a8d564 100644
--- a/lib/math/Makefile
+++ b/lib/math/Makefile
@@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
reciprocal_div.o
obj-$(CONFIG_CORDIC) += cordic.o
obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
obj-$(CONFIG_RATIONAL) += rational.o
+
+obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
new file mode 100644
index ..fed15ade8fb2
--- /dev/null
+++ b/lib/math/math_kunit.c
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple KUnit suite for math helper funcs that are always enabled.
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+/* Generic test case for unsigned long inputs. */
+struct test_case {
+ unsigned long a, b;
+ unsigned long result;
+};
+
+static struct test_case gcd_cases[] = {
+ {
+ .a = 0, .b = 0,
+ .result = 0,
+ },
+ {
+ .a = 0, .b = 1,
+ .result = 1,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 4,
+ .result = 2,
+ },
+ {
+ .a = 3, .b = 5,
+ .result = 1,
+ },
+ {
+ .a = 3 * 9, .b = 3 * 5,
+ .result = 3,
+ },
+ {
+ .a = 3 * 5 * 7, .b = 3 * 5 * 11,
+ .result = 15,
+ },
+ {
+ .a = 1 << 21,
+ .b = (1 << 21) - 1,
+ .result = 1,
+ },
+};
+
+KUNIT_ARRAY_PARAM(gcd, gcd_cases, NULL);
+
+static void gcd_test(struct kunit *test)
+{
+ const char *message_fmt = "gcd(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->a, test_param->b),
+ message_fmt, test_param->a,
+ test_param->b);
+
+ if (test_param->a == test_param->b)
+ return;
+
+ /* gcd(a,b) == gcd(b,a) */
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->b, test_param->a),
+ message_fmt, test_param->b,
+ test_param->a);
+}
+
+static struct test_case lcm_cases[] = {
+ {
+ .a = 0, .b = 0,
+ .result = 0,
+ },
+ {
+ .a = 0, .b = 1,
+ .result = 0,
+ },
+ {
+ .a = 1, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 3 * 5, .b = 3 * 7,
+ .result = 3 * 5 * 7,
+ },
+};
+
+KUNIT_ARRAY_PARAM(lcm, lcm_cases, NULL);
+
+static void lcm_test(struct kunit *test)
+{
+ const char *message_fmt = "lcm(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXP
Re: [PATCH v3 1/2] kunit: add a KUnit test for SLUB debugging functionality
On Thu, Apr 8, 2021 at 3:30 AM Marco Elver wrote:
>
> On Tue, 6 Apr 2021 at 12:57, Vlastimil Babka wrote:
> >
> >
> > On 4/1/21 11:24 PM, Marco Elver wrote:
> > > On Thu, 1 Apr 2021 at 21:04, Daniel Latypov wrote:
> > >> > }
> > >> > #else
> > >> > static inline bool slab_add_kunit_errors(void) { return false;
> > >> > }
> > >> > #endif
> > >> >
> > >> > And anywhere you want to increase the error count, you'd call
> > >> > slab_add_kunit_errors().
> > >> >
> > >> > Another benefit of this approach is that if KUnit is disabled, there is
> > >> > zero overhead and no additional code generated (vs. the current
> > >> > approach).
> > >>
> > >> The resource approach looks really good, but...
> > >> You'd be picking up a dependency on
> > >> https://lore.kernel.org/linux-kselftest/20210311152314.3814916-2-dlaty...@google.com/
> > >> current->kunit_test will always be NULL unless CONFIG_KASAN=y &&
> > >> CONFIG_KUNIT=y at the moment.
> > >> My patch drops the CONFIG_KASAN requirement and opens it up to all tests.
> > >
> > > Oh, that's a shame, but hopefully it'll be in -next soon.
> > >
> > >> At the moment, it's just waiting another look over from Brendan or David.
> > >> Any ETA on that, folks? :)
> > >>
> > >> So if you don't want to get blocked on that for now, I think it's fine
> > >> to add:
> > >> #ifdef CONFIG_SLUB_KUNIT_TEST
> > >> int errors;
> > >> #endif
> > >
> > > Until kunit fixes setting current->kunit_test, a cleaner workaround
> > > that would allow to do the patch with kunit_resource, is to just have
> > > an .init/.exit function that sets it ("current->kunit_test = test;").
> > > And then perhaps add a note ("FIXME: ...") to remove it once the above
> > > patch has landed.
> > >
> > > At least that way we get the least intrusive change for mm/slub.c, and
> > > the test is the only thing that needs a 2-line patch to clean up
> > > later.
> >
> > So when testing internally Oliver's new version with your suggestions
> > (thanks
> > again for those), I got lockdep splats because slab_add_kunit_errors is
> > called
> > also from irq disabled contexts, and kunit_find_named_resource will call
> > spin_lock(>lock) that's not irq safe. Can we make the lock irq safe? I
> > tried the change below and it makde the problem go away. If you agree, the
> > question is how to proceed - make it part of Oliver's patch series and let
> > Andrew pick it all with eventually kunit team's acks on this patch, or
> > whatnot.
>
> From what I can tell it should be fine to make it irq safe (ack for
> your patch below). Regarding patch logistics, I'd probably add it to
> the series. If that ends up not working, we'll find out sooner or
> later.
>
> (FYI, the prerequisite patch for current->kunit_test is in -next now.)
Yep.
There's also two follow-up patches in
https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=kunit
>
> KUnit maintainers, do you have any preferences?
Poked offline and Brendan and David seemed fine either way.
So probably just include it in this patch series for convenience.
Brendan also mentioned KUnit used to use spin_lock_irqsave/restore()
but had been told to not use it until necessary.
See
https://lore.kernel.org/linux-kselftest/20181016235120.138227-3-brendanhigg...@google.com/
So I think there's no objections to the patch itself either.
But I'd wait for Brendan to chime in to confirm.
>
> > 8<
> >
> > commit ab28505477892e9824c57ac338c88aec2ec0abce
> > Author: Vlastimil Babka
> > Date: Tue Apr 6 12:28:07 2021 +0200
> >
> > kunit: make test->lock irq safe
> >
> > diff --git a/include/kunit/test.h b/include/kunit/test.h
> > index 49601c4b98b8..524d4789af22 100644
> > --- a/include/kunit/test.h
> > +++ b/include/kunit/test.h
> > @@ -515,8 +515,9 @@ kunit_find_resource(struct kunit *test,
> > void *match_data)
> > {
> > struct kunit_resource *res, *found = NULL;
> > + unsigned long flags;
> >
> > - spin_lock(>lock);
> > + spin_lock_irqsave(>lock, flags);
> >
> > list_for_each_entry_reverse(res, >resources, node) {
> > if (mat
[PATCH] Documentation: kunit: add tips for using current->kunit_test
As of commit 359a376081d4 ("kunit: support failure from dynamic analysis
tools"), we can use current->kunit_test to find the current kunit test.
Mention this in tips.rst and give an example of how this can be used in
conjunction with `test->priv` to pass around state and specifically
implement something like mocking.
There's a lot more we could go into on that topic, but given that
example is already longer than every other "tip" on this page, we just
point to the API docs and leave filling in the blanks as an exercise to
the reader.
Also give an example of kunit_fail_current_test().
Signed-off-by: Daniel Latypov
---
Documentation/dev-tools/kunit/tips.rst | 78 +-
1 file changed, 76 insertions(+), 2 deletions(-)
diff --git a/Documentation/dev-tools/kunit/tips.rst
b/Documentation/dev-tools/kunit/tips.rst
index a6ca0af14098..8d8c238f7f79 100644
--- a/Documentation/dev-tools/kunit/tips.rst
+++ b/Documentation/dev-tools/kunit/tips.rst
@@ -78,8 +78,82 @@ Similarly to the above, it can be useful to add
test-specific logic.
void test_only_hook(void) { }
#endif
-TODO(dlaty...@google.com): add an example of using ``current->kunit_test`` in
-such a hook when it's not only updated for ``CONFIG_KASAN=y``.
+This test-only code can be made more useful by accessing the current kunit
+test, see below.
+
+Accessing the current test
+--
+
+In some cases, you need to call test-only code from outside the test file, e.g.
+like in the example above or if you're providing a fake implementation of an
+ops struct.
+There is a ``kunit_test`` field in ``task_struct``, so you can access it via
+``current->kunit_test``.
+
+Here's a slightly in-depth example of how one could implement "mocking":
+
+.. code-block:: c
+
+ #include /* for current */
+
+ struct test_data {
+ int foo_result;
+ int want_foo_called_with;
+ };
+
+ static int fake_foo(int arg)
+ {
+ struct kunit *test = current->kunit_test;
+ struct test_data *test_data = test->priv;
+
+ KUNIT_EXPECT_EQ(test, test_data->want_foo_called_with, arg);
+ return test_data->foo_result;
+ }
+
+ static void example_simple_test(struct kunit *test)
+ {
+ /* Assume priv is allocated in the suite's .init */
+ struct test_data *test_data = test->priv;
+
+ test_data->foo_result = 42;
+ test_data->want_foo_called_with = 1;
+
+ /* In a real test, we'd probably pass a pointer to fake_foo
somewhere
+* like an ops struct, etc. instead of calling it directly. */
+ KUNIT_EXPECT_EQ(test, fake_foo(1), 42);
+ }
+
+
+Note: here we're able to get away with using ``test->priv``, but if you wanted
+something more flexible you could use a named ``kunit_resource``, see
:doc:`api/test`.
+
+Failing the current test
+
+
+But sometimes, you might just want to fail the current test. In that case, we
+have ``kunit_fail_current_test(fmt, args...)`` which is defined in
and
+doesn't require pulling in .
+
+E.g. say we had an option to enable some extra debug checks on some data
structure:
+
+.. code-block:: c
+
+ #include
+
+ #ifdef CONFIG_EXTRA_DEBUG_CHECKS
+ static void validate_my_data(struct data *data)
+ {
+ if (is_valid(data))
+ return;
+
+ kunit_fail_current_test("data %p is invalid", data);
+
+ /* Normal, non-KUnit, error reporting code here. */
+ }
+ #else
+ static void my_debug_function(void) { }
+ #endif
+
Customizing error messages
--
base-commit: 0a50438c84363bd37fe18fe432888ae9a074dcab
--
2.31.0.208.g409f899ff0-goog
Re: linux-next: build warning after merge of the kunit-next tree
Thanks for the catch.
Should be addressed by
https://lore.kernel.org/linux-kselftest/20210406172901.1729216-1-dlaty...@google.com/
When I was testing the CONFIG_KUNIT=n case, I added it to a file that
wasn't being compiled (CONFIG_UBSAN=y is not sufficient for
lib/ubsan.c to be compiled...).
On Tue, Apr 6, 2021 at 3:24 AM Stephen Rothwell wrote:
>
> Hi all,
>
> After merging the kunit-next tree, today's linux-next build (x86_64
> allmodconfig) produced this warning:
>
> In file included from lib/kunit/test.c:10:
> include/kunit/test-bug.h:22:28: warning: '__kunit_fail_current_test' defined
> but not used [-Wunused-function]
>22 | static __printf(3, 4) void __kunit_fail_current_test(const char
> *file, int line,
> |^
>
> Introduced by commit
>
> 359a376081d4 ("kunit: support failure from dynamic analysis tools")
>
> --
> Cheers,
> Stephen Rothwell
[PATCH] kunit: fix -Wunused-function warning for __kunit_fail_current_test
When CONFIG_KUNIT is not enabled, __kunit_fail_current_test() an empty
static function.
But GCC complains about unused static functions, *unless* they're static inline.
So add inline to make GCC happy.
Signed-off-by: Daniel Latypov
Fixes: 359a376081d4 ("kunit: support failure from dynamic analysis tools")
---
include/kunit/test-bug.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
index ce6f6edc7801..5fc58081d511 100644
--- a/include/kunit/test-bug.h
+++ b/include/kunit/test-bug.h
@@ -19,8 +19,8 @@ extern __printf(3, 4) void __kunit_fail_current_test(const
char *file, int line,
#else
-static __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
- const char *fmt, ...)
+static inline __printf(3, 4) void __kunit_fail_current_test(const char *file,
int line,
+ const char *fmt,
...)
{
}
base-commit: 359a376081d4fadfb073e3ddeb6bd6dc94d98341
--
2.31.0.208.g409f899ff0-goog
Re: [PATCH v4 1/2] kunit: support failure from dynamic analysis tools
On Fri, Apr 2, 2021 at 10:53 AM Shuah Khan wrote:
>
> On 4/2/21 2:55 AM, Brendan Higgins wrote:
> > On Thu, Mar 11, 2021 at 7:23 AM Daniel Latypov wrote:
> >>
> >> From: Uriel Guajardo
> >>
> >> Add a kunit_fail_current_test() function to fail the currently running
> >> test, if any, with an error message.
> >>
> >> This is largely intended for dynamic analysis tools like UBSAN and for
> >> fakes.
> >> E.g. say I had a fake ops struct for testing and I wanted my `free`
> >> function to complain if it was called with an invalid argument, or
> >> caught a double-free. Most return void and have no normal means of
> >> signalling failure (e.g. super_operations, iommu_ops, etc.).
> >>
> >> Key points:
> >> * Always update current->kunit_test so anyone can use it.
> >>* commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
> >>CONFIG_KASAN=y
> >>
> >> * Create a new header so non-test code doesn't have
> >> to include all of (e.g. lib/ubsan.c)
> >>
> >> * Forward the file and line number to make it easier to track down
> >> failures
> >>
> >> * Declare the helper function for nice __printf() warnings about mismatched
> >> format strings even when KUnit is not enabled.
> >>
> >> Example output from kunit_fail_current_test("message"):
> >> [15:19:34] [FAILED] example_simple_test
> >> [15:19:34] # example_simple_test: initializing
> >> [15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24:
> >> message
> >> [15:19:34] not ok 1 - example_simple_test
> >>
> >> Co-developed-by: Daniel Latypov
> >> Signed-off-by: Daniel Latypov
> >> Signed-off-by: Uriel Guajardo
> >> Reviewed-by: Alan Maguire
> >
> > Reviewed-by: Brendan Higgins
> >
>
> Please run checkpatch on your patches in the future. I am seeing
> a few checkpatch readability type improvements that can be made.
>
> Please make changes and send v2 with Brendan's Reviewed-by.
Thanks for the catch.
checkpatch.pl --strict should now be happy (aside from complaining
about line wrapping)
v5 here:
https://lore.kernel.org/linux-kselftest/20210402212131.835276-1-dlaty...@google.com
Note: Brendan didn't give an explicit Reviewed-by on the second patch,
not sure if that was intentional.
>
> thanks,
> -- Shuah
[PATCH v5 1/2] kunit: support failure from dynamic analysis tools
From: Uriel Guajardo
Add a kunit_fail_current_test() function to fail the currently running
test, if any, with an error message.
This is largely intended for dynamic analysis tools like UBSAN and for
fakes.
E.g. say I had a fake ops struct for testing and I wanted my `free`
function to complain if it was called with an invalid argument, or
caught a double-free. Most return void and have no normal means of
signalling failure (e.g. super_operations, iommu_ops, etc.).
Key points:
* Always update current->kunit_test so anyone can use it.
* commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
CONFIG_KASAN=y
* Create a new header so non-test code doesn't have
to include all of (e.g. lib/ubsan.c)
* Forward the file and line number to make it easier to track down
failures
* Declare the helper function for nice __printf() warnings about mismatched
format strings even when KUnit is not enabled.
Example output from kunit_fail_current_test("message"):
[15:19:34] [FAILED] example_simple_test
[15:19:34] # example_simple_test: initializing
[15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24: message
[15:19:34] not ok 1 - example_simple_test
Co-developed-by: Daniel Latypov
Signed-off-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Reviewed-by: Alan Maguire
Reviewed-by: Brendan Higgins
---
include/kunit/test-bug.h | 29 +
lib/kunit/test.c | 39 +++
2 files changed, 64 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
new file mode 100644
index ..ce6f6edc7801
--- /dev/null
+++ b/include/kunit/test-bug.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API allowing dynamic analysis tools to interact with KUnit tests
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Uriel Guajardo
+ */
+
+#ifndef _KUNIT_TEST_BUG_H
+#define _KUNIT_TEST_BUG_H
+
+#define kunit_fail_current_test(fmt, ...) \
+ __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
+
+#if IS_BUILTIN(CONFIG_KUNIT)
+
+extern __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...);
+
+#else
+
+static __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...)
+{
+}
+
+#endif
+
+#endif /* _KUNIT_TEST_BUG_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index ec9494e914ef..2f6cc0123232 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -7,6 +7,7 @@
*/
#include
+#include
#include
#include
#include
@@ -16,6 +17,40 @@
#include "string-stream.h"
#include "try-catch-impl.h"
+#if IS_BUILTIN(CONFIG_KUNIT)
+/*
+ * Fail the current test and print an error message to the log.
+ */
+void __kunit_fail_current_test(const char *file, int line, const char *fmt,
...)
+{
+ va_list args;
+ int len;
+ char *buffer;
+
+ if (!current->kunit_test)
+ return;
+
+ kunit_set_failure(current->kunit_test);
+
+ /* kunit_err() only accepts literals, so evaluate the args first. */
+ va_start(args, fmt);
+ len = vsnprintf(NULL, 0, fmt, args) + 1;
+ va_end(args);
+
+ buffer = kunit_kmalloc(current->kunit_test, len, GFP_KERNEL);
+ if (!buffer)
+ return;
+
+ va_start(args, fmt);
+ vsnprintf(buffer, len, fmt, args);
+ va_end(args);
+
+ kunit_err(current->kunit_test, "%s:%d: %s", file, line, buffer);
+ kunit_kfree(current->kunit_test, buffer);
+}
+EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
+#endif
+
/*
* Append formatted message to log, size of which is limited to
* KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -273,9 +308,7 @@ static void kunit_try_run_case(void *data)
struct kunit_suite *suite = ctx->suite;
struct kunit_case *test_case = ctx->test_case;
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = test;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT) */
/*
* kunit_run_case_internal may encounter a fatal error; if it does,
@@ -624,9 +657,7 @@ void kunit_cleanup(struct kunit *test)
spin_unlock(>lock);
kunit_remove_resource(test, res);
}
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = NULL;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT)*/
}
EXPORT_SYMBOL_GPL(kunit_cleanup);
--
2.31.0.208.g409f899ff0-goog
[PATCH v5 2/2] kunit: ubsan integration
From: Uriel Guajardo
Integrates UBSAN into the KUnit testing framework. It fails KUnit tests
whenever it reports undefined behavior.
When CONFIG_KUNIT=n, nothing is printed or even formatted, so this has
no behavioral impact outside of tests.
kunit_fail_current_test() effectively does a pr_err() as well, so
there's some slight duplication, but it also ensures an error is
recorded in the debugfs entry for the running KUnit test.
Print a shorter version of the message to make it less spammy.
Co-developed-by: Daniel Latypov
Signed-off-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Reviewed-by: Alan Maguire
---
lib/ubsan.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/ubsan.c b/lib/ubsan.c
index 26229973049d..bdc380ff5d5c 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -14,6 +14,7 @@
#include
#include
#include
+#include
#include "ubsan.h"
@@ -141,6 +142,8 @@ static void ubsan_prologue(struct source_location *loc,
const char *reason)
"\n");
pr_err("UBSAN: %s in %s:%d:%d\n", reason, loc->file_name,
loc->line & LINE_MASK, loc->column & COLUMN_MASK);
+
+ kunit_fail_current_test("%s in %s", reason, loc->file_name);
}
static void ubsan_epilogue(void)
--
2.31.0.208.g409f899ff0-goog
[PATCH v5 0/2] kunit: fail tests on UBSAN errors
v1 by Uriel is here: [1].
Since it's been a while, I've dropped the Reviewed-By's.
It depended on commit 83c4e7a0363b ("KUnit: KASAN Integration") which
hadn't been merged yet, so that caused some kerfuffle with applying them
previously and the series was reverted.
This revives the series but makes the kunit_fail_current_test() function
take a format string and logs the file and line number of the failing
code, addressing Alan Maguire's comments on the previous version.
As a result, the patch that makes UBSAN errors was tweaked slightly to
include an error message.
v2 -> v3:
Try and fail to make kunit_fail_current_test() work on CONFIG_KUNIT=m
s/_/__ on the helper func to match others in test.c
v3 -> v4:
Revert to only enabling kunit_fail_current_test() for CONFIG_KUNIT=y
v4 -> v5:
Delete blank line to make checkpatch.pl --strict happy
[1]
https://lore.kernel.org/linux-kselftest/20200806174326.3577537-1-urielguajard...@gmail.com/
Uriel Guajardo (2):
kunit: support failure from dynamic analysis tools
kunit: ubsan integration
include/kunit/test-bug.h | 29 +
lib/kunit/test.c | 39 +++
lib/ubsan.c | 3 +++
3 files changed, 67 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
base-commit: 1678e493d530e7977cce34e59a86bb86f3c5631e
--
2.31.0.208.g409f899ff0-goog
Re: [PATCH] kunit: make KUNIT_EXPECT_STREQ() quote values, don't print literals
On Fri, Apr 2, 2021 at 12:19 PM Shuah Khan wrote:
>
> On 4/2/21 1:09 PM, Daniel Latypov wrote:
> > On Fri, Apr 2, 2021 at 10:47 AM Shuah Khan
> > wrote:
> >>
> >> On 4/2/21 3:35 AM, Brendan Higgins wrote:
> >>> On Fri, Feb 5, 2021 at 2:18 PM Daniel Latypov wrote:
> >>>>
> >>>> Before:
> >>>>>Expected str == "world", but
> >>>>>str == hello
> >>>>>"world" == world
> >>>>
> >>>> After:
> >>>>>Expected str == "world", but
> >>>>>str == "hello"
> >>>>
> >>>>
> >>>> Note: like the literal ellision for integers, this doesn't handle the
> >>>> case of
> >>>> KUNIT_EXPECT_STREQ(test, "hello", "world")
> >>>> since we don't expect it to realistically happen in checked in tests.
> >>>> (If you really wanted a test to fail, KUNIT_FAIL("msg") exists)
> >>>>
> >>>> In that case, you'd get:
> >>>>>Expected "hello" == "world", but
> >>>>
> >>>>
> >>>> Signed-off-by: Daniel Latypov
> >>>
> >>> Reviewed-by: Brendan Higgins
> >>>
> >>
> >> Hi Daniel,
> >>
> >> Please run checkpatch on your patches in the future. I am seeing
> >> a few checkpatch readability type improvements that can be made.
> >>
> >> Please make changes and send v2 with Brendan's Reviewed-by.
> >
> > Are there some flags you'd like me to pass to checkpatch?
> >
> > $ ./scripts/checkpatch.pl --git HEAD
> > total: 0 errors, 0 warnings, 42 lines checked
> >
>
> My commit script uses --strict which shows readability errors.
Oh neat, TIL.
I'll make sure to use that in the future, thanks!
v2:
https://lore.kernel.org/linux-kselftest/20210402193357.819176-1-dlaty...@google.com/
>
> > Commit f66884e8b831 ("kunit: make KUNIT_EXPECT_STREQ() quote values,
> > don't print literals") has no obvious style problems and is ready for
> > submission.
> >
> > I just rebased onto linus/master again since I know checkpatch.pl's
> > default behavior had changed recently, but I didn't see any errors
> > there.
> >
> > I know this commit made some lines go just over 80 characters, so
> > $ ./scripts/checkpatch.pl --max-line-length=80 --git HEAD
> > ...
> > total: 0 errors, 4 warnings, 42 lines checked
> >
>
> Don't worry about line wrap warns. I just ignore them. :)
>
> thanks,
> -- Shuah
>
>
>
[PATCH v2] kunit: make KUNIT_EXPECT_STREQ() quote values, don't print literals
Before:
> Expected str == "world", but
> str == hello
> "world" == world
After:
> Expected str == "world", but
> str == "hello"
Note: like the literal ellision for integers, this doesn't handle the
case of
KUNIT_EXPECT_STREQ(test, "hello", "world")
since we don't expect it to realistically happen in checked in tests.
(If you really wanted a test to fail, KUNIT_FAIL("msg") exists)
In that case, you'd get:
> Expected "hello" == "world", but
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
lib/kunit/assert.c | 30 --
1 file changed, 24 insertions(+), 6 deletions(-)
diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c
index e0ec7d6fed6f..27f95c8dc93f 100644
--- a/lib/kunit/assert.c
+++ b/lib/kunit/assert.c
@@ -156,6 +156,22 @@ void kunit_binary_ptr_assert_format(const struct
kunit_assert *assert,
}
EXPORT_SYMBOL_GPL(kunit_binary_ptr_assert_format);
+/* Checks if KUNIT_EXPECT_STREQ() args were string literals.
+ * Note: `text` will have ""s where as `value` will not.
+ */
+static bool is_str_literal(const char *text, const char *value)
+{
+ int len;
+
+ len = strlen(text);
+ if (len < 2)
+ return false;
+ if (text[0] != '\"' || text[len - 1] != '\"')
+ return false;
+
+ return strncmp(text + 1, value, len - 2) == 0;
+}
+
void kunit_binary_str_assert_format(const struct kunit_assert *assert,
struct string_stream *stream)
{
@@ -168,12 +184,14 @@ void kunit_binary_str_assert_format(const struct
kunit_assert *assert,
binary_assert->left_text,
binary_assert->operation,
binary_assert->right_text);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %s\n",
- binary_assert->left_text,
- binary_assert->left_value);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %s",
- binary_assert->right_text,
- binary_assert->right_value);
+ if (!is_str_literal(binary_assert->left_text,
binary_assert->left_value))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
\"%s\"\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ if (!is_str_literal(binary_assert->right_text,
binary_assert->right_value))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
\"%s\"",
+ binary_assert->right_text,
+ binary_assert->right_value);
kunit_assert_print_msg(assert, stream);
}
EXPORT_SYMBOL_GPL(kunit_binary_str_assert_format);
base-commit: 1678e493d530e7977cce34e59a86bb86f3c5631e
--
2.31.0.208.g409f899ff0-goog
Re: [PATCH] kunit: tool: make --kunitconfig accept dirs, add lib/kunit fragment
On Fri, Apr 2, 2021 at 11:00 AM Shuah Khan wrote: > > On 4/2/21 3:32 AM, Brendan Higgins wrote: > >> TL;DR > >> $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit > >> > >> Per suggestion from Ted [1], we can reduce the amount of typing by > >> assuming a convention that these files are named '.kunitconfig'. > >> > >> In the case of [1], we now have > >> $ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4 > >> > >> Also add in such a fragment for kunit itself so we can give that as an > >> example more close to home (and thus less likely to be accidentally > >> broken). > >> > >> [1] https://lore.kernel.org/linux-ext4/ycnf4yp1db97z...@mit.edu/ > >> > >> Signed-off-by: Daniel Latypov > > > > Reviewed-by: Brendan Higgins > > > > Should this be captured in documentation. Especially since this > is file is .* file. > > Do you want to include doc in this patch? Might be better that way. It definitely should be documented, yes. The only real example hadn't landed yet when I sent this patch (fs/ext4/.kunitconfig was going in through the ext4 tree), but now it's in linus/master. There's still some uncertainties about what best practices for this feature should be, i.e. * how granular should these be? * how should configs in parent dirs be handled? Should they be supersets of all the subdirs? * E.g. should fs/.kunitconfig be a superset of fs/ext4/.kunitconfig and any other hypothetical subdir configs? * Should we wait on saying "you should do this" until we have "import" statements/other mechanisms to make this less manual? * how should we handle non-UML tests, like the KASAN tests? * ideally, kunit.py run will eventually support running tests on x86 (using qemu) If it's fine with you, I was hoping to come back and add a section to kunit/start.rst when we've had some of those questions more figured out. > > thanks, > -- Shuah
Re: [PATCH] kunit: make KUNIT_EXPECT_STREQ() quote values, don't print literals
On Fri, Apr 2, 2021 at 10:47 AM Shuah Khan wrote:
>
> On 4/2/21 3:35 AM, Brendan Higgins wrote:
> > On Fri, Feb 5, 2021 at 2:18 PM Daniel Latypov wrote:
> >>
> >> Before:
> >>> Expected str == "world", but
> >>> str == hello
> >>> "world" == world
> >>
> >> After:
> >>> Expected str == "world", but
> >>> str == "hello"
> >>
> >>
> >> Note: like the literal ellision for integers, this doesn't handle the
> >> case of
> >>KUNIT_EXPECT_STREQ(test, "hello", "world")
> >> since we don't expect it to realistically happen in checked in tests.
> >> (If you really wanted a test to fail, KUNIT_FAIL("msg") exists)
> >>
> >> In that case, you'd get:
> >>> Expected "hello" == "world", but
> >>
> >>
> >> Signed-off-by: Daniel Latypov
> >
> > Reviewed-by: Brendan Higgins
> >
>
> Hi Daniel,
>
> Please run checkpatch on your patches in the future. I am seeing
> a few checkpatch readability type improvements that can be made.
>
> Please make changes and send v2 with Brendan's Reviewed-by.
Are there some flags you'd like me to pass to checkpatch?
$ ./scripts/checkpatch.pl --git HEAD
total: 0 errors, 0 warnings, 42 lines checked
Commit f66884e8b831 ("kunit: make KUNIT_EXPECT_STREQ() quote values,
don't print literals") has no obvious style problems and is ready for
submission.
I just rebased onto linus/master again since I know checkpatch.pl's
default behavior had changed recently, but I didn't see any errors
there.
I know this commit made some lines go just over 80 characters, so
$ ./scripts/checkpatch.pl --max-line-length=80 --git HEAD
...
total: 0 errors, 4 warnings, 42 lines checked
I can go and line wrap these but had figured they were more readable
this way if checkpatch.pl no longer complained by default.
Thanks,
Daniel
>
> thanks,
> -- Shuah
Re: [PATCH v3 1/2] kunit: add a KUnit test for SLUB debugging functionality
On Thu, Apr 1, 2021 at 2:16 AM 'Marco Elver' via KUnit Development
wrote:
>
> [Note, if you'd like me to see future versions, please Cc me, otherwise
> it's unlikely I see it in time. Also add kunit-...@googlegroups.com if
> perhaps a KUnit dev should have another look, too.]
>
> On Wed, Mar 31, 2021 at 10:51AM +0200, glit...@gmail.com wrote:
> > From: Oliver Glitta
> >
> > SLUB has resiliency_test() function which is hidden behind #ifdef
> > SLUB_RESILIENCY_TEST that is not part of Kconfig, so nobody
> > runs it. KUnit should be a proper replacement for it.
> >
> > Try changing byte in redzone after allocation and changing
> > pointer to next free node, first byte, 50th byte and redzone
> > byte. Check if validation finds errors.
> >
> > There are several differences from the original resiliency test:
> > Tests create own caches with known state instead of corrupting
> > shared kmalloc caches.
> >
> > The corruption of freepointer uses correct offset, the original
> > resiliency test got broken with freepointer changes.
> >
> > Scratch changing random byte test, because it does not have
> > meaning in this form where we need deterministic results.
> >
> > Add new option CONFIG_SLUB_KUNIT_TEST in Kconfig.
> >
> > Add a counter field "errors" to struct kmem_cache to count number
> > of errors detected in cache.
> >
> > Silence bug report in SLUB test. Add SLAB_SILENT_ERRORS debug flag.
> > Add SLAB_SILENT_ERRORS flag to SLAB_NEVER_MERGE, SLAB_DEBUG_FLAGS,
> > SLAB_FLAGS_PERMITTED macros.
> >
> > Signed-off-by: Oliver Glitta
> > ---
> > Changes since v2
> >
> > Use bit operation & instead of logical && as reported by kernel test
> > robot and Dan Carpenter
> >
> > Changes since v1
> >
> > Conversion from kselftest to KUnit test suggested by Marco Elver.
> > Error silencing.
> > Error counting improvements.
> >
> > include/linux/slab.h | 2 +
> > include/linux/slub_def.h | 2 +
> > lib/Kconfig.debug| 5 ++
> > lib/Makefile | 1 +
> > lib/test_slub.c | 124 +++
> > mm/slab.h| 7 ++-
> > mm/slab_common.c | 2 +-
> > mm/slub.c| 64 +---
> > 8 files changed, 184 insertions(+), 23 deletions(-)
> > create mode 100644 lib/test_slub.c
> >
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 7ae604076767..ed1a5a64d028 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -25,6 +25,8 @@
> > */
> > /* DEBUG: Perform (expensive) checks on alloc/free */
> > #define SLAB_CONSISTENCY_CHECKS ((slab_flags_t __force)0x0100U)
> > +/* DEBUG: Silent bug reports */
> > +#define SLAB_SILENT_ERRORS ((slab_flags_t __force)0x0200U)
>
> This flag wouldn't be necessary if you do the design using
> kunit_resource (see below).
>
> (But perhaps I missed a conversation that said that this flag is
> generally useful, but if so, it should probably be in a separate patch
> justifying why it is required beyond the test.)
>
> > /* DEBUG: Red zone objs in a cache */
> > #define SLAB_RED_ZONE((slab_flags_t __force)0x0400U)
> > /* DEBUG: Poison objects */
> > diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
> > index dcde82a4434c..e4b51bb5bb83 100644
> > --- a/include/linux/slub_def.h
> > +++ b/include/linux/slub_def.h
> > @@ -133,6 +133,8 @@ struct kmem_cache {
> > unsigned int usersize; /* Usercopy region size */
> >
> > struct kmem_cache_node *node[MAX_NUMNODES];
> > +
> > + int errors; /* Number of errors in cache */
>
> So, I think it's bad design to add a new field 'errors', just for the
> test. This will increase kmem_cache size for all builds, which is
> unnecessary.
>
> Is there use to retrieve 'errors' elsewhere?
>
> While you could guard this with #ifdef CONFIG_SLUB_DEBUG or so, there's
> a better design option if this is just for the KUnit test's benefit: use
> kunit_resource.
>
> The way it'd work is that for each test (you can add a common init
> function) you add a named resource, in this case just an 'int' I guess,
> that slab would be able to retrieve if this test is being run.
>
> In the test somewhere, you could add something like this:
>
>
> static struct kunit_resource resource;
> static int slab_errors;
>
> ..
>
> static int test_init(struct kunit *test)
> {
> slab_errors = 0;
> kunit_add_named_resource(test, NULL, NULL, ,
> "slab_errors", _errors);
> return 0;
> }
>
> .. tests now check slab_errors .
>
> and then in slub.c you'd have:
>
> #if IS_ENABLED(CONFIG_KUNIT)
> static bool slab_add_kunit_errors(void)
> {
> struct kunit_resource *resource;
>
> if (likely(!current->kunit_test))
> return false;
>
[PATCH v4 1/2] kunit: support failure from dynamic analysis tools
From: Uriel Guajardo
Add a kunit_fail_current_test() function to fail the currently running
test, if any, with an error message.
This is largely intended for dynamic analysis tools like UBSAN and for
fakes.
E.g. say I had a fake ops struct for testing and I wanted my `free`
function to complain if it was called with an invalid argument, or
caught a double-free. Most return void and have no normal means of
signalling failure (e.g. super_operations, iommu_ops, etc.).
Key points:
* Always update current->kunit_test so anyone can use it.
* commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
CONFIG_KASAN=y
* Create a new header so non-test code doesn't have
to include all of (e.g. lib/ubsan.c)
* Forward the file and line number to make it easier to track down
failures
* Declare the helper function for nice __printf() warnings about mismatched
format strings even when KUnit is not enabled.
Example output from kunit_fail_current_test("message"):
[15:19:34] [FAILED] example_simple_test
[15:19:34] # example_simple_test: initializing
[15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24: message
[15:19:34] not ok 1 - example_simple_test
Co-developed-by: Daniel Latypov
Signed-off-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Reviewed-by: Alan Maguire
---
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 39 +++
2 files changed, 65 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
new file mode 100644
index ..e88b74a4fd85
--- /dev/null
+++ b/include/kunit/test-bug.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API allowing dynamic analysis tools to interact with KUnit tests
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Uriel Guajardo
+ */
+
+#ifndef _KUNIT_TEST_BUG_H
+#define _KUNIT_TEST_BUG_H
+
+#define kunit_fail_current_test(fmt, ...) \
+ __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
+
+#if IS_BUILTIN(CONFIG_KUNIT)
+
+extern __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...);
+
+#else
+
+static __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...)
+{
+}
+
+#endif
+
+
+#endif /* _KUNIT_TEST_BUG_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index ec9494e914ef..2f6cc0123232 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -7,6 +7,7 @@
*/
#include
+#include
#include
#include
#include
@@ -16,6 +17,40 @@
#include "string-stream.h"
#include "try-catch-impl.h"
+#if IS_BUILTIN(CONFIG_KUNIT)
+/*
+ * Fail the current test and print an error message to the log.
+ */
+void __kunit_fail_current_test(const char *file, int line, const char *fmt,
...)
+{
+ va_list args;
+ int len;
+ char *buffer;
+
+ if (!current->kunit_test)
+ return;
+
+ kunit_set_failure(current->kunit_test);
+
+ /* kunit_err() only accepts literals, so evaluate the args first. */
+ va_start(args, fmt);
+ len = vsnprintf(NULL, 0, fmt, args) + 1;
+ va_end(args);
+
+ buffer = kunit_kmalloc(current->kunit_test, len, GFP_KERNEL);
+ if (!buffer)
+ return;
+
+ va_start(args, fmt);
+ vsnprintf(buffer, len, fmt, args);
+ va_end(args);
+
+ kunit_err(current->kunit_test, "%s:%d: %s", file, line, buffer);
+ kunit_kfree(current->kunit_test, buffer);
+}
+EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
+#endif
+
/*
* Append formatted message to log, size of which is limited to
* KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -273,9 +308,7 @@ static void kunit_try_run_case(void *data)
struct kunit_suite *suite = ctx->suite;
struct kunit_case *test_case = ctx->test_case;
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = test;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT) */
/*
* kunit_run_case_internal may encounter a fatal error; if it does,
@@ -624,9 +657,7 @@ void kunit_cleanup(struct kunit *test)
spin_unlock(>lock);
kunit_remove_resource(test, res);
}
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = NULL;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT)*/
}
EXPORT_SYMBOL_GPL(kunit_cleanup);
--
2.31.0.rc2.261.g7f71774620-goog
[PATCH v4 2/2] kunit: ubsan integration
From: Uriel Guajardo
Integrates UBSAN into the KUnit testing framework. It fails KUnit tests
whenever it reports undefined behavior.
When CONFIG_KUNIT=n, nothing is printed or even formatted, so this has
no behavioral impact outside of tests.
kunit_fail_current_test() effectively does a pr_err() as well, so
there's some slight duplication, but it also ensures an error is
recorded in the debugfs entry for the running KUnit test.
Print a shorter version of the message to make it less spammy.
Co-developed-by: Daniel Latypov
Signed-off-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Reviewed-by: Alan Maguire
---
lib/ubsan.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/ubsan.c b/lib/ubsan.c
index 26229973049d..bdc380ff5d5c 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -14,6 +14,7 @@
#include
#include
#include
+#include
#include "ubsan.h"
@@ -141,6 +142,8 @@ static void ubsan_prologue(struct source_location *loc,
const char *reason)
"\n");
pr_err("UBSAN: %s in %s:%d:%d\n", reason, loc->file_name,
loc->line & LINE_MASK, loc->column & COLUMN_MASK);
+
+ kunit_fail_current_test("%s in %s", reason, loc->file_name);
}
static void ubsan_epilogue(void)
--
2.31.0.rc2.261.g7f71774620-goog
[PATCH v4 0/2] kunit: fail tests on UBSAN errors
v1 by Uriel is here: [1].
Since it's been a while, I've dropped the Reviewed-By's.
It depended on commit 83c4e7a0363b ("KUnit: KASAN Integration") which
hadn't been merged yet, so that caused some kerfuffle with applying them
previously and the series was reverted.
This revives the series but makes the kunit_fail_current_test() function
take a format string and logs the file and line number of the failing
code, addressing Alan Maguire's comments on the previous version.
As a result, the patch that makes UBSAN errors was tweaked slightly to
include an error message.
v2 -> v3:
Try and fail to make kunit_fail_current_test() work on CONFIG_KUNIT=m
s/_/__ on the helper func to match others in test.c
v3 -> v4:
Revert to only enabling kunit_fail_current_test() for CONFIG_KUNIT=y
[1]
https://lore.kernel.org/linux-kselftest/20200806174326.3577537-1-urielguajard...@gmail.com/
Uriel Guajardo (2):
kunit: support failure from dynamic analysis tools
kunit: ubsan integration
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 39 +++
lib/ubsan.c | 3 +++
3 files changed, 68 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
base-commit: a74e6a014c9d4d4161061f770c9b4f98372ac778
--
2.31.0.rc2.261.g7f71774620-goog
Re: [PATCH] kunit: tool: Fix a python tuple typing error
On Mon, Feb 22, 2021 at 9:49 PM 'David Gow' via KUnit Development
wrote:
>
> The first argument to namedtuple() should match the name of the type,
> which wasn't the case for KconfigEntryBase.
>
> Fixing this is enough to make mypy show no python typing errors again.
Ah, this is something apparently only newer versions of mypy detect.
On 0.782 I didn't see it, but after pip install --upgrade to 0.812, I
see the error.
While I'm here, I also upgraded my pytype install and checked.
It's happy w/ or w/o this patch.
So while this is in some sense an error only mypy cares about, this
fix does make the code more stylistically correct and should
definitely go in.
>
> Fixes 97752c39bd ("kunit: kunit_tool: Allow .kunitconfig to disable config
> items")
> Signed-off-by: David Gow
Reviewed-by: Daniel Latypov
> ---
> tools/testing/kunit/kunit_config.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/testing/kunit/kunit_config.py
> b/tools/testing/kunit/kunit_config.py
> index 0b550cbd667d..1e2683dcc0e7 100644
> --- a/tools/testing/kunit/kunit_config.py
> +++ b/tools/testing/kunit/kunit_config.py
> @@ -13,7 +13,7 @@ from typing import List, Set
> CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_(\w+) is not set$'
> CONFIG_PATTERN = r'^CONFIG_(\w+)=(\S+|".*")$'
>
> -KconfigEntryBase = collections.namedtuple('KconfigEntry', ['name', 'value'])
> +KconfigEntryBase = collections.namedtuple('KconfigEntryBase', ['name',
> 'value'])
>
> class KconfigEntry(KconfigEntryBase):
>
> --
> 2.30.0.617.g56c4b15f3c-goog
>
> --
> You received this message because you are subscribed to the Google Groups
> "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kunit-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kunit-dev/20210223054930.234-1-davidgow%40google.com.
Re: [PATCH] ext4: add .kunitconfig fragment to enable ext4-specific tests
On Tue, Feb 9, 2021 at 6:33 PM Theodore Ts'o wrote: > > On Tue, Feb 09, 2021 at 05:32:06PM -0800, Daniel Latypov wrote: > > > > After [2]: > > $ ./tools/testing/kunit.py run --kunitconfig=fs/ext4/.kunitconfig > > Any chance that in the future this might become: > > $ ./tools/testing/kunit.py run --kunitconfig=fs/ext4 For future reference, this patch would make ^ work. https://lore.kernel.org/linux-kselftest/2021025241.201145-1-dlaty...@google.com (This time w/o the typo in the path to kunit.py :) > > Or better yet, syntactic sugar like: > > $ ./tools/testing/kunit.py test fs/ext4 > > would be really nice. > > - Ted
[PATCH] kunit: tool: make --kunitconfig accept dirs, add lib/kunit fragment
TL;DR
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
Per suggestion from Ted [1], we can reduce the amount of typing by
assuming a convention that these files are named '.kunitconfig'.
In the case of [1], we now have
$ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4
Also add in such a fragment for kunit itself so we can give that as an
example more close to home (and thus less likely to be accidentally
broken).
[1] https://lore.kernel.org/linux-ext4/ycnf4yp1db97z...@mit.edu/
Signed-off-by: Daniel Latypov
---
lib/kunit/.kunitconfig | 3 +++
tools/testing/kunit/kunit.py | 4 +++-
tools/testing/kunit/kunit_kernel.py| 2 ++
tools/testing/kunit/kunit_tool_test.py | 6 ++
4 files changed, 14 insertions(+), 1 deletion(-)
create mode 100644 lib/kunit/.kunitconfig
diff --git a/lib/kunit/.kunitconfig b/lib/kunit/.kunitconfig
new file mode 100644
index ..9235b7d42d38
--- /dev/null
+++ b/lib/kunit/.kunitconfig
@@ -0,0 +1,3 @@
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index d5144fcb03ac..5da8fb3762f9 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -184,7 +184,9 @@ def add_common_opts(parser) -> None:
help='Run all KUnit tests through allyesconfig',
action='store_true')
parser.add_argument('--kunitconfig',
-help='Path to Kconfig fragment that enables KUnit
tests',
+help='Path to Kconfig fragment that enables KUnit
tests.'
+' If given a directory, (e.g. lib/kunit),
"/.kunitconfig" '
+'will get automatically appended.',
metavar='kunitconfig')
def add_build_opts(parser) -> None:
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index f309a33256cd..89a7d4024e87 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -132,6 +132,8 @@ class LinuxSourceTree(object):
return
if kunitconfig_path:
+ if os.path.isdir(kunitconfig_path):
+ kunitconfig_path =
os.path.join(kunitconfig_path, KUNITCONFIG_PATH)
if not os.path.exists(kunitconfig_path):
raise ConfigError(f'Specified kunitconfig
({kunitconfig_path}) does not exist')
else:
diff --git a/tools/testing/kunit/kunit_tool_test.py
b/tools/testing/kunit/kunit_tool_test.py
index 1ad3049e9069..2e809dd956a7 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -251,6 +251,12 @@ class LinuxSourceTreeTest(unittest.TestCase):
with tempfile.NamedTemporaryFile('wt') as kunitconfig:
tree = kunit_kernel.LinuxSourceTree('',
kunitconfig_path=kunitconfig.name)
+ def test_dir_kunitconfig(self):
+ with tempfile.TemporaryDirectory('') as dir:
+ with open(os.path.join(dir, '.kunitconfig'), 'w') as f:
+ pass
+ tree = kunit_kernel.LinuxSourceTree('',
kunitconfig_path=dir)
+
# TODO: add more test cases.
base-commit: b12b47249688915e987a9a2a393b522f86f6b7ab
--
2.30.0.617.g56c4b15f3c-goog
Re: [PATCH v3 1/2] kunit: support failure from dynamic analysis tools
On Thu, Feb 11, 2021 at 1:33 PM 'Brendan Higgins' via KUnit
Development wrote:
>
> On Thu, Feb 11, 2021 at 12:58 PM Daniel Latypov wrote:
> >
> > On Thu, Feb 11, 2021 at 7:40 AM Alan Maguire
> > wrote:
> > >
> > > On Thu, 11 Feb 2021, David Gow wrote:
> > >
> > > > On Wed, Feb 10, 2021 at 6:14 AM Daniel Latypov
> > > > wrote:
> > > > >
> > > > > From: Uriel Guajardo
> > > > >
> > > > > Add a kunit_fail_current_test() function to fail the currently running
> > > > > test, if any, with an error message.
> > > > >
> > > > > This is largely intended for dynamic analysis tools like UBSAN and for
> > > > > fakes.
> > > > > E.g. say I had a fake ops struct for testing and I wanted my `free`
> > > > > function to complain if it was called with an invalid argument, or
> > > > > caught a double-free. Most return void and have no normal means of
> > > > > signalling failure (e.g. super_operations, iommu_ops, etc.).
> > > > >
> > > > > Key points:
> > > > > * Always update current->kunit_test so anyone can use it.
> > > > > * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it
> > > > > for
> > > > > CONFIG_KASAN=y
> > > > >
> > > > > * Create a new header so non-test code doesn't have
> > > > > to include all of (e.g. lib/ubsan.c)
> > > > >
> > > > > * Forward the file and line number to make it easier to track down
> > > > > failures
> > > > >
> > > > > * Declare the helper function for nice __printf() warnings about
> > > > > mismatched
> > > > > format strings even when KUnit is not enabled.
> > > > >
> > > > > Example output from kunit_fail_current_test("message"):
> > > > > [15:19:34] [FAILED] example_simple_test
> > > > > [15:19:34] # example_simple_test: initializing
> > > > > [15:19:34] # example_simple_test:
> > > > > lib/kunit/kunit-example-test.c:24: message
> > > > > [15:19:34] not ok 1 - example_simple_test
> > > > >
> > > > > Co-developed-by: Daniel Latypov
> > > > > Signed-off-by: Uriel Guajardo
> > > > > Signed-off-by: Daniel Latypov
> > > > > ---
> > > > > include/kunit/test-bug.h | 30 ++
> > > > > lib/kunit/test.c | 37 +
> > > > > 2 files changed, 63 insertions(+), 4 deletions(-)
> > > > > create mode 100644 include/kunit/test-bug.h
> > > > >
> > > > > diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
> > > > > new file mode 100644
> > > > > index ..18b1034ec43a
> > > > > --- /dev/null
> > > > > +++ b/include/kunit/test-bug.h
> > > > > @@ -0,0 +1,30 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0 */
> > > > > +/*
> > > > > + * KUnit API allowing dynamic analysis tools to interact with KUnit
> > > > > tests
> > > > > + *
> > > > > + * Copyright (C) 2020, Google LLC.
> > > > > + * Author: Uriel Guajardo
> > > > > + */
> > > > > +
> > > > > +#ifndef _KUNIT_TEST_BUG_H
> > > > > +#define _KUNIT_TEST_BUG_H
> > > > > +
> > > > > +#define kunit_fail_current_test(fmt, ...) \
> > > > > + __kunit_fail_current_test(__FILE__, __LINE__, fmt,
> > > > > ##__VA_ARGS__)
> > > > > +
> > > > > +#if IS_ENABLED(CONFIG_KUNIT)
> > > >
> > > > As the kernel test robot has pointed out on the second patch, this
> > > > probably should be IS_BUILTIN(), otherwise this won't build if KUnit
> > > > is a module, and the code calling it isn't.
> > > >
> > > > This does mean that things like UBSAN integration won't work if KUnit
> > > > is a module, which is a shame.
> > > >
> > > > (It's worth noting that the KASAN integration worked around this by
> > > > only calling inline functions, which would therefore be built-in even
> > > > if the rest of KUnit was built as a module. I don't think it's quite
> > > > as convenient to do that h
Re: [PATCH v3 1/2] kunit: support failure from dynamic analysis tools
On Thu, Feb 11, 2021 at 7:40 AM Alan Maguire wrote:
>
> On Thu, 11 Feb 2021, David Gow wrote:
>
> > On Wed, Feb 10, 2021 at 6:14 AM Daniel Latypov wrote:
> > >
> > > From: Uriel Guajardo
> > >
> > > Add a kunit_fail_current_test() function to fail the currently running
> > > test, if any, with an error message.
> > >
> > > This is largely intended for dynamic analysis tools like UBSAN and for
> > > fakes.
> > > E.g. say I had a fake ops struct for testing and I wanted my `free`
> > > function to complain if it was called with an invalid argument, or
> > > caught a double-free. Most return void and have no normal means of
> > > signalling failure (e.g. super_operations, iommu_ops, etc.).
> > >
> > > Key points:
> > > * Always update current->kunit_test so anyone can use it.
> > > * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
> > > CONFIG_KASAN=y
> > >
> > > * Create a new header so non-test code doesn't have
> > > to include all of (e.g. lib/ubsan.c)
> > >
> > > * Forward the file and line number to make it easier to track down
> > > failures
> > >
> > > * Declare the helper function for nice __printf() warnings about
> > > mismatched
> > > format strings even when KUnit is not enabled.
> > >
> > > Example output from kunit_fail_current_test("message"):
> > > [15:19:34] [FAILED] example_simple_test
> > > [15:19:34] # example_simple_test: initializing
> > > [15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24:
> > > message
> > > [15:19:34] not ok 1 - example_simple_test
> > >
> > > Co-developed-by: Daniel Latypov
> > > Signed-off-by: Uriel Guajardo
> > > Signed-off-by: Daniel Latypov
> > > ---
> > > include/kunit/test-bug.h | 30 ++
> > > lib/kunit/test.c | 37 +
> > > 2 files changed, 63 insertions(+), 4 deletions(-)
> > > create mode 100644 include/kunit/test-bug.h
> > >
> > > diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
> > > new file mode 100644
> > > index ..18b1034ec43a
> > > --- /dev/null
> > > +++ b/include/kunit/test-bug.h
> > > @@ -0,0 +1,30 @@
> > > +/* SPDX-License-Identifier: GPL-2.0 */
> > > +/*
> > > + * KUnit API allowing dynamic analysis tools to interact with KUnit tests
> > > + *
> > > + * Copyright (C) 2020, Google LLC.
> > > + * Author: Uriel Guajardo
> > > + */
> > > +
> > > +#ifndef _KUNIT_TEST_BUG_H
> > > +#define _KUNIT_TEST_BUG_H
> > > +
> > > +#define kunit_fail_current_test(fmt, ...) \
> > > + __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
> > > +
> > > +#if IS_ENABLED(CONFIG_KUNIT)
> >
> > As the kernel test robot has pointed out on the second patch, this
> > probably should be IS_BUILTIN(), otherwise this won't build if KUnit
> > is a module, and the code calling it isn't.
> >
> > This does mean that things like UBSAN integration won't work if KUnit
> > is a module, which is a shame.
> >
> > (It's worth noting that the KASAN integration worked around this by
> > only calling inline functions, which would therefore be built-in even
> > if the rest of KUnit was built as a module. I don't think it's quite
> > as convenient to do that here, though.)
> >
>
> Right, static inline'ing __kunit_fail_current_test() seems problematic
> because it calls other exported functions; more below
>
> > > +
> > > +extern __printf(3, 4) void __kunit_fail_current_test(const char *file,
> > > int line,
> > > + const char *fmt, ...);
> > > +
> > > +#else
> > > +
> > > +static __printf(3, 4) void __kunit_fail_current_test(const char *file,
> > > int line,
> > > + const char *fmt, ...)
> > > +{
> > > +}
> > > +
> > > +#endif
> > > +
> > > +
> > > +#endif /* _KUNIT_TEST_BUG_H */
> > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> > > index ec9494e914ef..5794059505cf 100644
> > > --- a/lib/kunit/test.c
> > > +++ b/lib/kunit/test.c
> > > @@ -7,6 +7,7 @@
> > > */
> > >
> > >
Re: [PATCH] ext4: add .kunitconfig fragment to enable ext4-specific tests
On Tue, Feb 9, 2021 at 6:33 PM Theodore Ts'o wrote: > > On Tue, Feb 09, 2021 at 05:32:06PM -0800, Daniel Latypov wrote: > > > > After [2]: > > $ ./tools/testing/kunit.py run --kunitconfig=fs/ext4/.kunitconfig > > Any chance that in the future this might become: > > $ ./tools/testing/kunit.py run --kunitconfig=fs/ext4 I've been in favor of something like that for a while, but haven't gotten folks to agree on the details. Using bazel-like syntax for a bit, I'd really like it if we had some easy way to do $ kunit test //fs/... # run all fs tests across all subdirs But since there's the possibility of having tests w/ incompatible requirements, I don't know that kunit.py can support it. (Tbh, I think just concatenating fragments would probably just work 99% of the time so kunit.py could get away with doing that). So --kunitconfig= is currently a compromise to give us a less controversial way of providing one-liners for testing a whole subdirectory. I don't think there'd be too much opposition for --kunitconfig to automatically append ".kunitconfig" when passed a directory. But there might be some, since a reader might think --kunitconfig=dir/ means it's recursing over all subdirs. > > Or better yet, syntactic sugar like: > > $ ./tools/testing/kunit.py test fs/ext4 The positional argument for run/exec is probably going to be taken by: https://lore.kernel.org/linux-kselftest/20210206000854.2037923-1-dlaty...@google.com/ https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit=5d31f71efcb6bce56ca3ab92eed0c8f2dbcc6f9a So we'd see something like: $ ./tools/testing/kunit.py run --kunitconfig=fs/ext4 '*inode*' Or if we set and followed naming conventions: $ ./tools/testing/kunit.py run --alltests "ext4-*" (this would take a lot longer to build however...) Filtering could also let us curate only a few, less granular .kunitconfig fragments (at the cost of higher build time). E.g. $ ./tools/testing/kunit.py run --kunitconfig=fs/ "ext4-*" > > would be really nice. > > - Ted
[PATCH] ext4: add .kunitconfig fragment to enable ext4-specific tests
As of [1], we no longer want EXT4_KUNIT_TESTS and others to `select`
their deps. This means it can get harder to get all the right things
selected as we gain more tests w/ more deps over time.
This patch (and [2]) proposes we store kunitconfig fragments in-tree to
represent sets of tests. (N.B. right now we only have one ext4 test).
There's still a discussion to be had about how to have a hierarchy of
these files (e.g. if one wanted to test all of fs/, not just fs/ext4).
But this fragment would likely be a leaf node and isn't blocked on
deciding if we want `import` statements and the like.
Usage
=
Before [2] (on its way to being merged):
$ cp fs/ext4/.kunitconfig .kunit/
$ ./tools/testing/kunit.py run
After [2]:
$ ./tools/testing/kunit.py run --kunitconfig=fs/ext4/.kunitconfig
".kunitconfig" vs "kunitconfig"
===
See also: commit 14ee5cfd4512 ("kunit: Rename 'kunitconfig' to '.kunitconfig'").
* The bit about .gitignore exluding it by default is now a con, however.
* But there are a lot of directories with files that begin with "k" and
so this could cause some annoyance w/ tab completion*
* This is the name kunit.py expects right now, so some people are used
to .kunitconfig over "kunitconfig"
[1]
https://lore.kernel.org/linux-ext4/20210122110234.2825685-1-ge...@linux-m68k.org/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit=243180f5924ed27ea417db39feb7f9691777688e
* 372/5556 directories isn't too much, but still not a small number:
$ find -type f -name 'k*' | xargs dirname | sort -u | wc -l
372
Signed-off-by: Daniel Latypov
---
fs/ext4/.kunitconfig | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 fs/ext4/.kunitconfig
diff --git a/fs/ext4/.kunitconfig b/fs/ext4/.kunitconfig
new file mode 100644
index ..bf51da7cd9fc
--- /dev/null
+++ b/fs/ext4/.kunitconfig
@@ -0,0 +1,3 @@
+CONFIG_KUNIT=y
+CONFIG_EXT4_FS=y
+CONFIG_EXT4_KUNIT_TESTS=y
base-commit: 88bb507a74ea7d75fa49edd421eaa710a7d80598
--
2.30.0.478.g8a0d178c01-goog
[PATCH v3 1/2] kunit: support failure from dynamic analysis tools
From: Uriel Guajardo
Add a kunit_fail_current_test() function to fail the currently running
test, if any, with an error message.
This is largely intended for dynamic analysis tools like UBSAN and for
fakes.
E.g. say I had a fake ops struct for testing and I wanted my `free`
function to complain if it was called with an invalid argument, or
caught a double-free. Most return void and have no normal means of
signalling failure (e.g. super_operations, iommu_ops, etc.).
Key points:
* Always update current->kunit_test so anyone can use it.
* commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
CONFIG_KASAN=y
* Create a new header so non-test code doesn't have
to include all of (e.g. lib/ubsan.c)
* Forward the file and line number to make it easier to track down
failures
* Declare the helper function for nice __printf() warnings about mismatched
format strings even when KUnit is not enabled.
Example output from kunit_fail_current_test("message"):
[15:19:34] [FAILED] example_simple_test
[15:19:34] # example_simple_test: initializing
[15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24: message
[15:19:34] not ok 1 - example_simple_test
Co-developed-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Signed-off-by: Daniel Latypov
---
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 37 +
2 files changed, 63 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
new file mode 100644
index ..18b1034ec43a
--- /dev/null
+++ b/include/kunit/test-bug.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API allowing dynamic analysis tools to interact with KUnit tests
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Uriel Guajardo
+ */
+
+#ifndef _KUNIT_TEST_BUG_H
+#define _KUNIT_TEST_BUG_H
+
+#define kunit_fail_current_test(fmt, ...) \
+ __kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
+
+#if IS_ENABLED(CONFIG_KUNIT)
+
+extern __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...);
+
+#else
+
+static __printf(3, 4) void __kunit_fail_current_test(const char *file, int
line,
+ const char *fmt, ...)
+{
+}
+
+#endif
+
+
+#endif /* _KUNIT_TEST_BUG_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index ec9494e914ef..5794059505cf 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -7,6 +7,7 @@
*/
#include
+#include
#include
#include
#include
@@ -16,6 +17,38 @@
#include "string-stream.h"
#include "try-catch-impl.h"
+/*
+ * Fail the current test and print an error message to the log.
+ */
+void __kunit_fail_current_test(const char *file, int line, const char *fmt,
...)
+{
+ va_list args;
+ int len;
+ char *buffer;
+
+ if (!current->kunit_test)
+ return;
+
+ kunit_set_failure(current->kunit_test);
+
+ /* kunit_err() only accepts literals, so evaluate the args first. */
+ va_start(args, fmt);
+ len = vsnprintf(NULL, 0, fmt, args) + 1;
+ va_end(args);
+
+ buffer = kunit_kmalloc(current->kunit_test, len, GFP_KERNEL);
+ if (!buffer)
+ return;
+
+ va_start(args, fmt);
+ vsnprintf(buffer, len, fmt, args);
+ va_end(args);
+
+ kunit_err(current->kunit_test, "%s:%d: %s", file, line, buffer);
+ kunit_kfree(current->kunit_test, buffer);
+}
+EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
+
/*
* Append formatted message to log, size of which is limited to
* KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -273,9 +306,7 @@ static void kunit_try_run_case(void *data)
struct kunit_suite *suite = ctx->suite;
struct kunit_case *test_case = ctx->test_case;
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = test;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT) */
/*
* kunit_run_case_internal may encounter a fatal error; if it does,
@@ -624,9 +655,7 @@ void kunit_cleanup(struct kunit *test)
spin_unlock(>lock);
kunit_remove_resource(test, res);
}
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = NULL;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT)*/
}
EXPORT_SYMBOL_GPL(kunit_cleanup);
--
2.30.0.478.g8a0d178c01-goog
[PATCH v3 0/2] kunit: fail tests on UBSAN errors
v1 by Uriel is here: [1].
Since it's been a while, I've dropped the Reviewed-By's.
It depended on commit 83c4e7a0363b ("KUnit: KASAN Integration") which
hadn't been merged yet, so that caused some kerfuffle with applying them
previously and the series was reverted.
This revives the series but makes the kunit_fail_current_test() function
take a format string and logs the file and line number of the failing
code, addressing Alan Maguire's comments on the previous version.
As a result, the patch that makes UBSAN errors was tweaked slightly to
include an error message.
v2 -> v3:
Fix kunit_fail_current_test() so it works w/ CONFIG_KUNIT=m
s/_/__ on the helper func to match others in test.c
[1]
https://lore.kernel.org/linux-kselftest/20200806174326.3577537-1-urielguajard...@gmail.com/
Uriel Guajardo (2):
kunit: support failure from dynamic analysis tools
kunit: ubsan integration
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 37 +
lib/ubsan.c | 3 +++
3 files changed, 66 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
base-commit: 1e0d27fce010b0a4a9e595506b6ede75934c31be
--
2.30.0.478.g8a0d178c01-goog
Re: [PATCH v2 1/2] kunit: support failure from dynamic analysis tools
On Tue, Feb 9, 2021 at 2:12 PM Alan Maguire wrote:
>
> On Tue, 9 Feb 2021, Daniel Latypov wrote:
>
> > On Tue, Feb 9, 2021 at 9:26 AM Alan Maguire wrote:
> > >
> > > On Fri, 5 Feb 2021, Daniel Latypov wrote:
> > >
> > > > From: Uriel Guajardo
> > > >
> > > > Add a kunit_fail_current_test() function to fail the currently running
> > > > test, if any, with an error message.
> > > >
> > > > This is largely intended for dynamic analysis tools like UBSAN and for
> > > > fakes.
> > > > E.g. say I had a fake ops struct for testing and I wanted my `free`
> > > > function to complain if it was called with an invalid argument, or
> > > > caught a double-free. Most return void and have no normal means of
> > > > signalling failure (e.g. super_operations, iommu_ops, etc.).
> > > >
> > > > Key points:
> > > > * Always update current->kunit_test so anyone can use it.
> > > > * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
> > > > CONFIG_KASAN=y
> > > >
> > > > * Create a new header so non-test code doesn't have
> > > > to include all of (e.g. lib/ubsan.c)
> > > >
> > > > * Forward the file and line number to make it easier to track down
> > > > failures
> > > >
> > >
> > > Thanks for doing this!
> > >
> > > > * Declare it as a function for nice __printf() warnings about mismatched
> > > > format strings even when KUnit is not enabled.
> > > >
> > >
> > > One thing I _think_ this assumes is that KUnit is builtin;
> > > don't we need an
> >
> > Ah, you're correct.
> > Also going to rename it to have two _ to match other functions used in
> > macros like __kunit_test_suites_init.
> >
>
> Great! If you're sending out an updated version with these
> changes, feel free to add
>
> Reviewed-by: Alan Maguire
Oops, there was a race in sending v3 and seeing this in my inbox.
If you could reply to the v3 that'd be great. I've already amended the
commit locally.
Thanks!
[PATCH v3 2/2] kunit: ubsan integration
From: Uriel Guajardo
Integrates UBSAN into the KUnit testing framework. It fails KUnit tests
whenever it reports undefined behavior.
When CONFIG_KUNIT=n, nothing is printed or even formatted, so this has
no behavioral impact outside of tests.
kunit_fail_current_test() effectively does a pr_err() as well, so
there's some slight duplication, but it also ensures an error is
recorded in the debugfs entry for the running KUnit test.
Print a shorter version of the message to make it less spammy.
Co-developed-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Signed-off-by: Daniel Latypov
---
lib/ubsan.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/ubsan.c b/lib/ubsan.c
index bec38c64d6a6..1ec7d6f1fe63 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -14,6 +14,7 @@
#include
#include
#include
+#include
#include "ubsan.h"
@@ -141,6 +142,8 @@ static void ubsan_prologue(struct source_location *loc,
const char *reason)
"\n");
pr_err("UBSAN: %s in %s:%d:%d\n", reason, loc->file_name,
loc->line & LINE_MASK, loc->column & COLUMN_MASK);
+
+ kunit_fail_current_test("%s in %s", reason, loc->file_name);
}
static void ubsan_epilogue(void)
--
2.30.0.478.g8a0d178c01-goog
Re: [PATCH v2 1/2] kunit: support failure from dynamic analysis tools
On Tue, Feb 9, 2021 at 9:26 AM Alan Maguire wrote:
>
> On Fri, 5 Feb 2021, Daniel Latypov wrote:
>
> > From: Uriel Guajardo
> >
> > Add a kunit_fail_current_test() function to fail the currently running
> > test, if any, with an error message.
> >
> > This is largely intended for dynamic analysis tools like UBSAN and for
> > fakes.
> > E.g. say I had a fake ops struct for testing and I wanted my `free`
> > function to complain if it was called with an invalid argument, or
> > caught a double-free. Most return void and have no normal means of
> > signalling failure (e.g. super_operations, iommu_ops, etc.).
> >
> > Key points:
> > * Always update current->kunit_test so anyone can use it.
> > * commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
> > CONFIG_KASAN=y
> >
> > * Create a new header so non-test code doesn't have
> > to include all of (e.g. lib/ubsan.c)
> >
> > * Forward the file and line number to make it easier to track down
> > failures
> >
>
> Thanks for doing this!
>
> > * Declare it as a function for nice __printf() warnings about mismatched
> > format strings even when KUnit is not enabled.
> >
>
> One thing I _think_ this assumes is that KUnit is builtin;
> don't we need an
Ah, you're correct.
Also going to rename it to have two _ to match other functions used in
macros like __kunit_test_suites_init.
I had been having some recent issues with getting QEMU to work on my
machine so I hadn't tested it before.
Somehow I've finally fixed it and can now say that it works w/
CONFIG_KUNIT=m after making the change
# modprobe kunit
# modprobe kunit-example-test
[ 27.689840] # Subtest: example
[ 27.689994] 1..1
[ 27.692337] # example_simple_test: initializing
[ 27.692862] # example_simple_test:
lib/kunit/kunit-example-test.c:31: example failure message: 42
[ 27.693158] not ok 1 - example_simple_test
[ 27.693654] not ok 1 - example
>
> EXPORT_SYMBOL_GPL(_kunit_fail_current_test);
>
> ?
>
> Without it, if an analysis tool (or indeed if KUnit) is built
> as a module, it won't be possible to use this functionality.
>
> > Example output from kunit_fail_current_test("message"):
> > [15:19:34] [FAILED] example_simple_test
> > [15:19:34] # example_simple_test: initializing
> > [15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24:
> > message
> > [15:19:34] not ok 1 - example_simple_test
> >
> > Co-developed-by: Daniel Latypov
> > Signed-off-by: Uriel Guajardo
> > Signed-off-by: Daniel Latypov
> > ---
> > include/kunit/test-bug.h | 30 ++
> > lib/kunit/test.c | 36
> > 2 files changed, 62 insertions(+), 4 deletions(-)
> > create mode 100644 include/kunit/test-bug.h
> >
> > diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
> > new file mode 100644
> > index ..4963ed52c2df
> > --- /dev/null
> > +++ b/include/kunit/test-bug.h
> > @@ -0,0 +1,30 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * KUnit API allowing dynamic analysis tools to interact with KUnit tests
> > + *
> > + * Copyright (C) 2020, Google LLC.
>
> nit; might want to update copyright year.
>
> > + * Author: Uriel Guajardo
> > + */
> > +
> > +#ifndef _KUNIT_TEST_BUG_H
> > +#define _KUNIT_TEST_BUG_H
> > +
> > +#define kunit_fail_current_test(fmt, ...) \
> > + _kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
> > +
> > +#if IS_ENABLED(CONFIG_KUNIT)
> > +
> > +extern __printf(3, 4) void _kunit_fail_current_test(const char *file, int
> > line,
> > + const char *fmt, ...);
> > +
> > +#else
> > +
> > +static __printf(3, 4) void _kunit_fail_current_test(const char *file, int
> > line,
> > + const char *fmt, ...)
> > +{
> > +}
> > +
> > +#endif
> > +
> > +
> > +#endif /* _KUNIT_TEST_BUG_H */
> > diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> > index ec9494e914ef..7b16aae0ccae 100644
> > --- a/lib/kunit/test.c
> > +++ b/lib/kunit/test.c
> > @@ -7,6 +7,7 @@
> > */
> >
> > #include
> > +#include
> > #include
> > #include
> > #include
> > @@ -16,6 +17,37 @@
> > #include "string-stream.h"
> > #include "try-catch-impl.h"
> >
> > +
[PATCH v2] kunit: don't show `1 == 1` in failed assertion messages
Currently, given something (fairly dystopian) like
> KUNIT_EXPECT_EQ(test, 2 + 2, 5)
KUnit will prints a failure message like this.
> Expected 2 + 2 == 5, but
> 2 + 2 == 4
> 5 == 5
With this patch, the output just becomes
> Expected 2 + 2 == 5, but
> 2 + 2 == 4
This patch is slightly hacky, but it's quite common* to compare an
expression to a literal integer value, so this can make KUnit less
chatty in many cases. (This patch also fixes variants like
KUNIT_EXPECT_GT, LE, et al.).
It also allocates an additional string briefly, but given this only
happens on test failures, it doesn't seem too bad a tradeoff.
Also, in most cases it'll realize the lengths are unequal and bail out
before the allocation.
We could save the result of the formatted string to avoid wasting this
extra work, but it felt cleaner to leave it as-is.
Edge case: for something silly and unrealistic like
> KUNIT_EXPECT_EQ(test, 4, 5);
It'll generate this message with a trailing "but"
> Expected 4 == 5, but
>
It didn't feel worth adding a check up-front to see if both sides are
literals to handle this better.
*A quick grep suggests 100+ comparisons to an integer literal as the
right hand side.
Signed-off-by: Daniel Latypov
Tested-by: David Gow
Reviewed-by: Brendan Higgins
---
lib/kunit/assert.c | 39 +--
1 file changed, 33 insertions(+), 6 deletions(-)
diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c
index 33acdaa28a7d..e0ec7d6fed6f 100644
--- a/lib/kunit/assert.c
+++ b/lib/kunit/assert.c
@@ -85,6 +85,29 @@ void kunit_ptr_not_err_assert_format(const struct
kunit_assert *assert,
}
EXPORT_SYMBOL_GPL(kunit_ptr_not_err_assert_format);
+/* Checks if `text` is a literal representing `value`, e.g. "5" and 5 */
+static bool is_literal(struct kunit *test, const char *text, long long value,
+ gfp_t gfp)
+{
+ char *buffer;
+ int len;
+ bool ret;
+
+ len = snprintf(NULL, 0, "%lld", value);
+ if (strlen(text) != len)
+ return false;
+
+ buffer = kunit_kmalloc(test, len+1, gfp);
+ if (!buffer)
+ return false;
+
+ snprintf(buffer, len+1, "%lld", value);
+ ret = strncmp(buffer, text, len) == 0;
+
+ kunit_kfree(test, buffer);
+ return ret;
+}
+
void kunit_binary_assert_format(const struct kunit_assert *assert,
struct string_stream *stream)
{
@@ -97,12 +120,16 @@ void kunit_binary_assert_format(const struct kunit_assert
*assert,
binary_assert->left_text,
binary_assert->operation,
binary_assert->right_text);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld\n",
- binary_assert->left_text,
- binary_assert->left_value);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld",
- binary_assert->right_text,
- binary_assert->right_value);
+ if (!is_literal(stream->test, binary_assert->left_text,
+ binary_assert->left_value, stream->gfp))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
%lld\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ if (!is_literal(stream->test, binary_assert->right_text,
+ binary_assert->right_value, stream->gfp))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld",
+ binary_assert->right_text,
+ binary_assert->right_value);
kunit_assert_print_msg(assert, stream);
}
EXPORT_SYMBOL_GPL(kunit_binary_assert_format);
base-commit: 1e0d27fce010b0a4a9e595506b6ede75934c31be
--
2.30.0.478.g8a0d178c01-goog
[PATCH v4 2/3] kunit: tool: add support for filtering suites by glob
This allows running different subsets of tests, e.g.
$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'
This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
tools/testing/kunit/kunit.py | 21 -
tools/testing/kunit/kunit_kernel.py| 4 +++-
tools/testing/kunit/kunit_tool_test.py | 15 +--
3 files changed, 28 insertions(+), 12 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 02871a363f76..d5144fcb03ac 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -28,12 +28,12 @@ KunitBuildRequest = namedtuple('KunitBuildRequest',
['jobs', 'build_dir', 'alltests',
'make_options'])
KunitExecRequest = namedtuple('KunitExecRequest',
- ['timeout', 'build_dir', 'alltests'])
+ ['timeout', 'build_dir', 'alltests',
'filter_glob'])
KunitParseRequest = namedtuple('KunitParseRequest',
['raw_output', 'input_data', 'build_dir',
'json'])
KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs',
- 'build_dir', 'alltests', 'json',
- 'make_options'])
+ 'build_dir', 'alltests',
'filter_glob',
+ 'json', 'make_options'])
KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
@@ -93,6 +93,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree,
test_start = time.time()
result = linux.run_kernel(
timeout=None if request.alltests else request.timeout,
+filter_glob=request.filter_glob,
build_dir=request.build_dir)
test_end = time.time()
@@ -149,7 +150,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
return build_result
exec_request = KunitExecRequest(request.timeout, request.build_dir,
- request.alltests)
+ request.alltests, request.filter_glob)
exec_result = exec_tests(linux, exec_request)
if exec_result.status != KunitStatus.SUCCESS:
return exec_result
@@ -200,6 +201,14 @@ def add_exec_opts(parser) -> None:
type=int,
default=300,
metavar='timeout')
+ parser.add_argument('filter_glob',
+ help='maximum number of seconds to allow for all
tests '
+ 'to run. This does not include time taken to build
the '
+ 'tests.',
+ type=str,
+ nargs='?',
+ default='',
+ metavar='filter_glob')
def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
@@ -266,6 +275,7 @@ def main(argv, linux=None):
cli_args.jobs,
cli_args.build_dir,
cli_args.alltests,
+ cli_args.filter_glob,
cli_args.json,
cli_args.make_options)
result = run_tests(linux, request)
@@ -307,7 +317,8 @@ def main(argv, linux=None):
exec_request = KunitExecRequest(cli_args.timeout,
cli_args.build_dir,
- cli_args.alltests)
+ cli_args.alltests,
+ cli_args.filter_glob)
exec_result = exec_tests(linux, exec_request)
parse_request = KunitParseRequest(cli_args.raw_output,
exec_result.result,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 0b461663e7d9..71a5f5c1750b 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -203,8 +203,10 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', timeout=None) ->
Iterator[str]:
+ def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
args.extend(['mem=1G', 'console=tty'])
+ if filter_glob:
+
[PATCH v4 3/3] kunit: tool: fix unintentional statefulness in run_kernel()
This is a bug that has been present since the first version of this
code.
Using [] as a default parameter is dangerous, since it's mutable.
Example using the REPL:
>>> def bad(param = []):
... param.append(len(param))
... print(param)
...
>>> bad()
[0]
>>> bad()
[0, 1]
This wasn't a concern in the past since it would just keep appending the
same values to it.
E.g. before, `args` would just grow in size like:
[mem=1G', 'console=tty']
[mem=1G', 'console=tty', mem=1G', 'console=tty']
But with now filter_glob, this is more dangerous, e.g.
run_kernel(filter_glob='my-test*') # default modified here
run_kernel() # filter_glob still applies here!
That earlier `filter_glob` will affect all subsequent calls that don't
specify `args`.
Note: currently the kunit tool only calls run_kernel() at most once, so
it's not possible to trigger any negative side-effects right now.
Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
tools/testing/kunit/kunit_kernel.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 71a5f5c1750b..f309a33256cd 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -203,7 +203,9 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ def run_kernel(self, args=None, build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ if not args:
+ args = []
args.extend(['mem=1G', 'console=tty'])
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
--
2.30.0.478.g8a0d178c01-goog
[PATCH v4 0/3] kunit: support running subsets of test suites from kunit.py
When using `kunit.py run` to run tests, users must populate a `kunitconfig` file to select the options the tests are hidden behind and all their dependencies. The patch [1] to allow specifying a path to kunitconfig promises to make this nicer as we can have checked in files corresponding to different sets of tests. But it's still annoying 1) when trying to run a subet of tests 2) when you want to run tests that don't have such a pre-existing kunitconfig and selecting all the necessary options is tricky. This patch series aims to alleviate both: 1) `kunit.py run 'my-suite-*'` I.e. use my current kunitconfig, but just run suites that match this glob 2) `kunit.py run --alltests 'my-suite-*'` I.e. use allyesconfig so I don't have to worry about writing a kunitconfig at all. See the first commit message for more details and discussion about future work. This patch series also includes a bugfix for a latent bug that can't be triggered right now but has worse consequences as a result of the changes needed to plumb in this suite name glob. [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlaty...@google.com/ --- v1 -> v2: Fix free of `suites` subarray in suite_set. Found by Dan Carpenter and kernel test robot. v2 -> v3: Add MODULE_PARM_DESC() for kunit.filter_glob. v3 -> v4: Rebase on top of kunit_tool_test.py and typing fixes for merging. Daniel Latypov (3): kunit: add kunit.filter_glob cmdline option to filter suites kunit: tool: add support for filtering suites by glob kunit: tool: fix unintentional statefulness in run_kernel() lib/kunit/Kconfig | 1 + lib/kunit/executor.c | 93 +++--- tools/testing/kunit/kunit.py | 21 -- tools/testing/kunit/kunit_kernel.py| 6 +- tools/testing/kunit/kunit_tool_test.py | 15 +++-- 5 files changed, 115 insertions(+), 21 deletions(-) base-commit: aa919f3b019d0e10e0c035598546b30cca7bcb19 -- 2.30.0.478.g8a0d178c01-goog
[PATCH v2 2/2] kunit: ubsan integration
From: Uriel Guajardo
Integrates UBSAN into the KUnit testing framework. It fails KUnit tests
whenever it reports undefined behavior.
When CONFIG_KUNIT=n, nothing is printed or even formatted, so this has
no behavioral impact outside of tests.
kunit_fail_current_test() effectively does a pr_err() as well, so
there's some slight duplication, but it also ensures an error is
recorded in the debugfs entry for the running KUnit test.
Print a shorter version of the message to make it less spammy.
Co-developed-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Signed-off-by: Daniel Latypov
---
lib/ubsan.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/ubsan.c b/lib/ubsan.c
index bec38c64d6a6..1ec7d6f1fe63 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -14,6 +14,7 @@
#include
#include
#include
+#include
#include "ubsan.h"
@@ -141,6 +142,8 @@ static void ubsan_prologue(struct source_location *loc,
const char *reason)
"\n");
pr_err("UBSAN: %s in %s:%d:%d\n", reason, loc->file_name,
loc->line & LINE_MASK, loc->column & COLUMN_MASK);
+
+ kunit_fail_current_test("%s in %s", reason, loc->file_name);
}
static void ubsan_epilogue(void)
--
2.30.0.478.g8a0d178c01-goog
[PATCH v4 1/3] kunit: add kunit.filter_glob cmdline option to filter suites
E.g. specifying this would run suites with "list" in their name.
kunit.filter_glob=list*
Note: the executor prints out a TAP header that includes the number of
suites we intend to run.
So unless we want to report empty results for filtered-out suites, we
need to do the filtering here in the executor.
It's also probably better in the executor since we most likely don't
want any filtering to apply to tests built as modules.
This code does add a CONFIG_GLOB=y dependency for CONFIG_KUNIT=y.
But the code seems light enough that it shouldn't be an issue.
For now, we only filter on suite names so we don't have to create copies
of the suites themselves, just the array (of arrays) holding them.
The name is rather generic since in the future, we could consider
extending it to a syntax like:
kunit.filter_glob=.
E.g. to run all the del list tests
kunit.filter_glob=list-kunit-test.*del*
But at the moment, it's far easier to manually comment out test cases in
test files as opposed to messing with sets of Kconfig entries to select
specific suites.
So even just doing this makes using kunit far less annoying.
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
lib/kunit/Kconfig| 1 +
lib/kunit/executor.c | 93 +++-
2 files changed, 85 insertions(+), 9 deletions(-)
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 00909e6a2443..0b5dfb001bac 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -4,6 +4,7 @@
menuconfig KUNIT
tristate "KUnit - Enable support for unit tests"
+ select GLOB if KUNIT=y
help
Enables support for kernel unit tests (KUnit), a lightweight unit
testing and mocking framework for the Linux kernel. These tests are
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index a95742a4ece7..15832ed44668 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
#include
+#include
+#include
/*
* These symbols point to the .kunit_test_suites section and are defined in
@@ -11,14 +13,81 @@ extern struct kunit_suite * const * const
__kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
-static void kunit_print_tap_header(void)
+static char *filter_glob;
+module_param(filter_glob, charp, 0);
+MODULE_PARM_DESC(filter_glob,
+ "Filter which KUnit test suites run at boot-time, e.g. list*");
+
+static struct kunit_suite * const *
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+{
+ int i, n = 0;
+ struct kunit_suite **filtered;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ ++n;
+ }
+
+ if (n == 0)
+ return NULL;
+
+ filtered = kmalloc_array(n + 1, sizeof(*filtered), GFP_KERNEL);
+ if (!filtered)
+ return NULL;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ filtered[n++] = subsuite[i];
+ }
+ filtered[n] = NULL;
+
+ return filtered;
+}
+
+struct suite_set {
+ struct kunit_suite * const * const *start;
+ struct kunit_suite * const * const *end;
+};
+
+static struct suite_set kunit_filter_suites(void)
+{
+ int i;
+ struct kunit_suite * const **copy, * const *filtered_subsuite;
+ struct suite_set filtered;
+
+ const size_t max = __kunit_suites_end - __kunit_suites_start;
+
+ if (!filter_glob) {
+ filtered.start = __kunit_suites_start;
+ filtered.end = __kunit_suites_end;
+ return filtered;
+ }
+
+ copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
+ filtered.start = copy;
+ if (!copy) { /* won't be able to run anything, return an empty set */
+ filtered.end = copy;
+ return filtered;
+ }
+
+ for (i = 0; i < max; ++i) {
+ filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ if (filtered_subsuite)
+ *copy++ = filtered_subsuite;
+ }
+ filtered.end = copy;
+ return filtered;
+}
+
+static void kunit_print_tap_header(struct suite_set *suite_set)
{
struct kunit_suite * const * const *suites, * const *subsuite;
int num_of_suites = 0;
- for (suites = __kunit_suites_start;
-suites < __kunit_suites_end;
-suites++)
+ for (suites = suite_set->start; suites < suite_set->end; suites++)
for (subsuite = *suites; *subsuite != NULL; subsuite++)
num_of_suites++;
@@ -30,12 +99,18 @@ int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
- kunit_print_tap_header();
+
[PATCH v2 1/2] kunit: support failure from dynamic analysis tools
From: Uriel Guajardo
Add a kunit_fail_current_test() function to fail the currently running
test, if any, with an error message.
This is largely intended for dynamic analysis tools like UBSAN and for
fakes.
E.g. say I had a fake ops struct for testing and I wanted my `free`
function to complain if it was called with an invalid argument, or
caught a double-free. Most return void and have no normal means of
signalling failure (e.g. super_operations, iommu_ops, etc.).
Key points:
* Always update current->kunit_test so anyone can use it.
* commit 83c4e7a0363b ("KUnit: KASAN Integration") only updated it for
CONFIG_KASAN=y
* Create a new header so non-test code doesn't have
to include all of (e.g. lib/ubsan.c)
* Forward the file and line number to make it easier to track down
failures
* Declare it as a function for nice __printf() warnings about mismatched
format strings even when KUnit is not enabled.
Example output from kunit_fail_current_test("message"):
[15:19:34] [FAILED] example_simple_test
[15:19:34] # example_simple_test: initializing
[15:19:34] # example_simple_test: lib/kunit/kunit-example-test.c:24: message
[15:19:34] not ok 1 - example_simple_test
Co-developed-by: Daniel Latypov
Signed-off-by: Uriel Guajardo
Signed-off-by: Daniel Latypov
---
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 36
2 files changed, 62 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
diff --git a/include/kunit/test-bug.h b/include/kunit/test-bug.h
new file mode 100644
index ..4963ed52c2df
--- /dev/null
+++ b/include/kunit/test-bug.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API allowing dynamic analysis tools to interact with KUnit tests
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Uriel Guajardo
+ */
+
+#ifndef _KUNIT_TEST_BUG_H
+#define _KUNIT_TEST_BUG_H
+
+#define kunit_fail_current_test(fmt, ...) \
+ _kunit_fail_current_test(__FILE__, __LINE__, fmt, ##__VA_ARGS__)
+
+#if IS_ENABLED(CONFIG_KUNIT)
+
+extern __printf(3, 4) void _kunit_fail_current_test(const char *file, int line,
+ const char *fmt, ...);
+
+#else
+
+static __printf(3, 4) void _kunit_fail_current_test(const char *file, int line,
+ const char *fmt, ...)
+{
+}
+
+#endif
+
+
+#endif /* _KUNIT_TEST_BUG_H */
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index ec9494e914ef..7b16aae0ccae 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -7,6 +7,7 @@
*/
#include
+#include
#include
#include
#include
@@ -16,6 +17,37 @@
#include "string-stream.h"
#include "try-catch-impl.h"
+/*
+ * Fail the current test and print an error message to the log.
+ */
+void _kunit_fail_current_test(const char *file, int line, const char *fmt, ...)
+{
+ va_list args;
+ int len;
+ char *buffer;
+
+ if (!current->kunit_test)
+ return;
+
+ kunit_set_failure(current->kunit_test);
+
+ /* kunit_err() only accepts literals, so evaluate the args first. */
+ va_start(args, fmt);
+ len = vsnprintf(NULL, 0, fmt, args) + 1;
+ va_end(args);
+
+ buffer = kunit_kmalloc(current->kunit_test, len, GFP_KERNEL);
+ if (!buffer)
+ return;
+
+ va_start(args, fmt);
+ vsnprintf(buffer, len, fmt, args);
+ va_end(args);
+
+ kunit_err(current->kunit_test, "%s:%d: %s", file, line, buffer);
+ kunit_kfree(current->kunit_test, buffer);
+}
+
/*
* Append formatted message to log, size of which is limited to
* KUNIT_LOG_SIZE bytes (including null terminating byte).
@@ -273,9 +305,7 @@ static void kunit_try_run_case(void *data)
struct kunit_suite *suite = ctx->suite;
struct kunit_case *test_case = ctx->test_case;
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = test;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT) */
/*
* kunit_run_case_internal may encounter a fatal error; if it does,
@@ -624,9 +654,7 @@ void kunit_cleanup(struct kunit *test)
spin_unlock(>lock);
kunit_remove_resource(test, res);
}
-#if (IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT))
current->kunit_test = NULL;
-#endif /* IS_ENABLED(CONFIG_KASAN) && IS_ENABLED(CONFIG_KUNIT)*/
}
EXPORT_SYMBOL_GPL(kunit_cleanup);
--
2.30.0.478.g8a0d178c01-goog
[PATCH] kunit: make KUNIT_EXPECT_STREQ() quote values, don't print literals
Before:
> Expected str == "world", but
> str == hello
> "world" == world
After:
> Expected str == "world", but
> str == "hello"
Note: like the literal ellision for integers, this doesn't handle the
case of
KUNIT_EXPECT_STREQ(test, "hello", "world")
since we don't expect it to realistically happen in checked in tests.
(If you really wanted a test to fail, KUNIT_FAIL("msg") exists)
In that case, you'd get:
> Expected "hello" == "world", but
Signed-off-by: Daniel Latypov
---
lib/kunit/assert.c | 30 --
1 file changed, 24 insertions(+), 6 deletions(-)
diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c
index e0ec7d6fed6f..176ef547fa94 100644
--- a/lib/kunit/assert.c
+++ b/lib/kunit/assert.c
@@ -156,6 +156,22 @@ void kunit_binary_ptr_assert_format(const struct
kunit_assert *assert,
}
EXPORT_SYMBOL_GPL(kunit_binary_ptr_assert_format);
+/* Checks if KUNIT_EXPECT_STREQ() args were string literals.
+ * Note: `text` will have ""s where as `value` will not.
+ */
+static bool is_str_literal(const char *text, const char *value)
+{
+ int len;
+
+ len = strlen(text);
+ if (len < 2)
+ return false;
+ if (text[0] != '\"' || text[len-1] != '\"')
+ return false;
+
+ return strncmp(text+1, value, len-2) == 0;
+}
+
void kunit_binary_str_assert_format(const struct kunit_assert *assert,
struct string_stream *stream)
{
@@ -168,12 +184,14 @@ void kunit_binary_str_assert_format(const struct
kunit_assert *assert,
binary_assert->left_text,
binary_assert->operation,
binary_assert->right_text);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %s\n",
- binary_assert->left_text,
- binary_assert->left_value);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %s",
- binary_assert->right_text,
- binary_assert->right_value);
+ if (!is_str_literal(binary_assert->left_text,
binary_assert->left_value))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
\"%s\"\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ if (!is_str_literal(binary_assert->right_text,
binary_assert->right_value))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
\"%s\"",
+ binary_assert->right_text,
+ binary_assert->right_value);
kunit_assert_print_msg(assert, stream);
}
EXPORT_SYMBOL_GPL(kunit_binary_str_assert_format);
base-commit: 1e0d27fce010b0a4a9e595506b6ede75934c31be
prerequisite-patch-id: 290f8022f30763cbfb6aec969b038a6f60a57482
--
2.30.0.478.g8a0d178c01-goog
[PATCH v2 0/2] kunit: fail tests on UBSAN errors
v1 by Uriel is here: [1].
Since it's been a while, I've dropped the Reviewed-By's.
It depended on commit 83c4e7a0363b ("KUnit: KASAN Integration") which
hadn't been merged yet, so that caused some kerfuffle with applying them
previously and the series was reverted.
This revives the series but makes the kunit_fail_current_test() function
take a format string and logs the file and line number of the failing
code, addressing Alan Maguire's comments on the previous version.
As a result, the patch that makes UBSAN errors was tweaked slightly to
include an error message.
[1]
https://lore.kernel.org/linux-kselftest/20200806174326.3577537-1-urielguajard...@gmail.com/
Uriel Guajardo (2):
kunit: support failure from dynamic analysis tools
kunit: ubsan integration
include/kunit/test-bug.h | 30 ++
lib/kunit/test.c | 36
lib/ubsan.c | 3 +++
3 files changed, 65 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/test-bug.h
base-commit: 1e0d27fce010b0a4a9e595506b6ede75934c31be
--
2.30.0.478.g8a0d178c01-goog
[PATCH v3 3/3] kunit: tool: fix unintentional statefulness in run_kernel()
This is a bug that has been present since the first version of this
code.
Using [] as a default parameter is dangerous, since it's mutable.
Example using the REPL:
>>> def bad(param = []):
... param.append(len(param))
... print(param)
...
>>> bad()
[0]
>>> bad()
[0, 1]
This wasn't a concern in the past since it would just keep appending the
same values to it.
E.g. before, `args` would just grow in size like:
[mem=1G', 'console=tty']
[mem=1G', 'console=tty', mem=1G', 'console=tty']
But with now filter_glob, this is more dangerous, e.g.
run_kernel(filter_glob='my-test*') # default modified here
run_kernel() # filter_glob still applies here!
That earlier `filter_glob` will affect all subsequent calls that don't
specify `args`.
Note: currently the kunit tool only calls run_kernel() at most once, so
it's not possible to trigger any negative side-effects right now.
Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
tools/testing/kunit/kunit_kernel.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 71b1942f5ccd..6dd3cf6e8efa 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,7 +199,9 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ def run_kernel(self, args=None, build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ if not args:
+ args = []
args.extend(['mem=1G', 'console=tty'])
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
--
2.30.0.478.g8a0d178c01-goog
[PATCH v3 2/3] kunit: tool: add support for filtering suites by glob
This allows running different subsets of tests, e.g.
$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'
This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
tools/testing/kunit/kunit.py| 21 -
tools/testing/kunit/kunit_kernel.py | 4 +++-
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e808a47c839b..a15ee33bb1f5 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -28,12 +28,12 @@ KunitBuildRequest = namedtuple('KunitBuildRequest',
['jobs', 'build_dir', 'alltests',
'make_options'])
KunitExecRequest = namedtuple('KunitExecRequest',
- ['timeout', 'build_dir', 'alltests'])
+ ['timeout', 'build_dir', 'alltests',
'filter_glob'])
KunitParseRequest = namedtuple('KunitParseRequest',
['raw_output', 'input_data', 'build_dir',
'json'])
KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs',
- 'build_dir', 'alltests', 'json',
- 'make_options'])
+ 'build_dir', 'alltests',
'filter_glob',
+ 'json', 'make_options'])
KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
@@ -93,6 +93,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree,
test_start = time.time()
result = linux.run_kernel(
timeout=None if request.alltests else request.timeout,
+filter_glob=request.filter_glob,
build_dir=request.build_dir)
test_end = time.time()
@@ -149,7 +150,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
return build_result
exec_request = KunitExecRequest(request.timeout, request.build_dir,
- request.alltests)
+ request.alltests, request.filter_glob)
exec_result = exec_tests(linux, exec_request)
if exec_result.status != KunitStatus.SUCCESS:
return exec_result
@@ -197,6 +198,14 @@ def add_exec_opts(parser) -> None:
type=int,
default=300,
metavar='timeout')
+ parser.add_argument('filter_glob',
+ help='maximum number of seconds to allow for all
tests '
+ 'to run. This does not include time taken to build
the '
+ 'tests.',
+ type=str,
+ nargs='?',
+ default='',
+ metavar='filter_glob')
def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
@@ -263,6 +272,7 @@ def main(argv, linux=None):
cli_args.jobs,
cli_args.build_dir,
cli_args.alltests,
+ cli_args.filter_glob,
cli_args.json,
cli_args.make_options)
result = run_tests(linux, request)
@@ -304,7 +314,8 @@ def main(argv, linux=None):
exec_request = KunitExecRequest(cli_args.timeout,
cli_args.build_dir,
- cli_args.alltests)
+ cli_args.alltests,
+ cli_args.filter_glob)
exec_result = exec_tests(linux, exec_request)
parse_request = KunitParseRequest(cli_args.raw_output,
exec_result.result,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2076a5a2d060..71b1942f5ccd 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,8 +199,10 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', timeout=None) ->
Iterator[str]:
+ def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
args.extend(['mem=1G', 'console=tty'])
+ if filter_glob:
+ args.append('kunit.filter_glob='+filter_glob)
se
[PATCH v3 1/3] kunit: add kunit.filter_glob cmdline option to filter suites
E.g. specifying this would run suites with "list" in their name.
kunit.filter_glob=list*
Note: the executor prints out a TAP header that includes the number of
suites we intend to run.
So unless we want to report empty results for filtered-out suites, we
need to do the filtering here in the executor.
It's also probably better in the executor since we most likely don't
want any filtering to apply to tests built as modules.
This code does add a CONFIG_GLOB=y dependency for CONFIG_KUNIT=y.
But the code seems light enough that it shouldn't be an issue.
For now, we only filter on suite names so we don't have to create copies
of the suites themselves, just the array (of arrays) holding them.
The name is rather generic since in the future, we could consider
extending it to a syntax like:
kunit.filter_glob=.
E.g. to run all the del list tests
kunit.filter_glob=list-kunit-test.*del*
But at the moment, it's far easier to manually comment out test cases in
test files as opposed to messing with sets of Kconfig entries to select
specific suites.
So even just doing this makes using kunit far less annoying.
Signed-off-by: Daniel Latypov
Reviewed-by: Brendan Higgins
---
lib/kunit/Kconfig| 1 +
lib/kunit/executor.c | 93 +++-
2 files changed, 85 insertions(+), 9 deletions(-)
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 00909e6a2443..0b5dfb001bac 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -4,6 +4,7 @@
menuconfig KUNIT
tristate "KUnit - Enable support for unit tests"
+ select GLOB if KUNIT=y
help
Enables support for kernel unit tests (KUnit), a lightweight unit
testing and mocking framework for the Linux kernel. These tests are
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index a95742a4ece7..15832ed44668 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
#include
+#include
+#include
/*
* These symbols point to the .kunit_test_suites section and are defined in
@@ -11,14 +13,81 @@ extern struct kunit_suite * const * const
__kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
-static void kunit_print_tap_header(void)
+static char *filter_glob;
+module_param(filter_glob, charp, 0);
+MODULE_PARM_DESC(filter_glob,
+ "Filter which KUnit test suites run at boot-time, e.g. list*");
+
+static struct kunit_suite * const *
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+{
+ int i, n = 0;
+ struct kunit_suite **filtered;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ ++n;
+ }
+
+ if (n == 0)
+ return NULL;
+
+ filtered = kmalloc_array(n + 1, sizeof(*filtered), GFP_KERNEL);
+ if (!filtered)
+ return NULL;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ filtered[n++] = subsuite[i];
+ }
+ filtered[n] = NULL;
+
+ return filtered;
+}
+
+struct suite_set {
+ struct kunit_suite * const * const *start;
+ struct kunit_suite * const * const *end;
+};
+
+static struct suite_set kunit_filter_suites(void)
+{
+ int i;
+ struct kunit_suite * const **copy, * const *filtered_subsuite;
+ struct suite_set filtered;
+
+ const size_t max = __kunit_suites_end - __kunit_suites_start;
+
+ if (!filter_glob) {
+ filtered.start = __kunit_suites_start;
+ filtered.end = __kunit_suites_end;
+ return filtered;
+ }
+
+ copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
+ filtered.start = copy;
+ if (!copy) { /* won't be able to run anything, return an empty set */
+ filtered.end = copy;
+ return filtered;
+ }
+
+ for (i = 0; i < max; ++i) {
+ filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ if (filtered_subsuite)
+ *copy++ = filtered_subsuite;
+ }
+ filtered.end = copy;
+ return filtered;
+}
+
+static void kunit_print_tap_header(struct suite_set *suite_set)
{
struct kunit_suite * const * const *suites, * const *subsuite;
int num_of_suites = 0;
- for (suites = __kunit_suites_start;
-suites < __kunit_suites_end;
-suites++)
+ for (suites = suite_set->start; suites < suite_set->end; suites++)
for (subsuite = *suites; *subsuite != NULL; subsuite++)
num_of_suites++;
@@ -30,12 +99,18 @@ int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
- kunit_print_tap_header();
+
[PATCH v3 0/3] kunit: support running subsets of test suites from kunit.py
When using `kunit.py run` to run tests, users must populate a `kunitconfig` file to select the options the tests are hidden behind and all their dependencies. The patch [1] to allow specifying a path to kunitconfig promises to make this nicer as we can have checked in files corresponding to different sets of tests. But it's still annoying 1) when trying to run a subet of tests 2) when you want to run tests that don't have such a pre-existing kunitconfig and selecting all the necessary options is tricky. This patch series aims to alleviate both: 1) `kunit.py run 'my-suite-*'` I.e. use my current kunitconfig, but just run suites that match this glob 2) `kunit.py run --alltests 'my-suite-*'` I.e. use allyesconfig so I don't have to worry about writing a kunitconfig at all. See the first commit message for more details and discussion about future work. This patch series also includes a bugfix for a latent bug that can't be triggered right now but has worse consequences as a result of the changes needed to plumb in this suite name glob. [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlaty...@google.com/ --- v1 -> v2: Fix free of `suites` subarray in suite_set. Found by Dan Carpenter and kernel test robot. v2 -> v3: Add MODULE_PARM_DESC() for kunit.filter_glob. Daniel Latypov (3): kunit: add kunit.filter_glob cmdline option to filter suites kunit: tool: add support for filtering suites by glob kunit: tool: fix unintentional statefulness in run_kernel() lib/kunit/Kconfig | 1 + lib/kunit/executor.c| 93 ++--- tools/testing/kunit/kunit.py| 21 +-- tools/testing/kunit/kunit_kernel.py | 6 +- 4 files changed, 106 insertions(+), 15 deletions(-) base-commit: 88bb507a74ea7d75fa49edd421eaa710a7d80598 -- 2.30.0.478.g8a0d178c01-goog
Re: [PATCH 1/3] kunit: add kunit.filter_glob cmdline option to filter suites
On Wed, Feb 3, 2021 at 11:13 PM Dan Carpenter wrote:
>
> Hi Daniel,
>
> url:
> https://github.com/0day-ci/linux/commits/Daniel-Latypov/kunit-support-running-subsets-of-test-suites-from/20210204-074405
> base: 88bb507a74ea7d75fa49edd421eaa710a7d80598
> config: x86_64-randconfig-m001-20210202 (attached as .config)
> compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
>
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot
> Reported-by: Dan Carpenter
>
> smatch warnings:
> lib/kunit/executor.c:110 kunit_run_all_tests() error: double free of
> 'suite_set.start'
>
> vim +110 lib/kunit/executor.c
>
> 8c0d884986ba22 Brendan Higgins 2020-08-04 96 int kunit_run_all_tests(void)
> aac35468ca20a3 Alan Maguire2020-08-04 97 {
> aac35468ca20a3 Alan Maguire2020-08-04 98 struct kunit_suite *
> const * const *suites;
> aac35468ca20a3 Alan Maguire2020-08-04 99
> d5554dd78a454b Daniel Latypov 2021-02-03 100 struct suite_set
> suite_set = kunit_filter_suites();
> 45dcbb6f5ef78b Brendan Higgins 2020-08-04 101
> d5554dd78a454b Daniel Latypov 2021-02-03 102
> kunit_print_tap_header(_set);
> d5554dd78a454b Daniel Latypov 2021-02-03 103
> d5554dd78a454b Daniel Latypov 2021-02-03 104 for (suites =
> suite_set.start; suites < suite_set.end; suites++)
> aac35468ca20a3 Alan Maguire2020-08-04 105
> __kunit_test_suites_init(*suites);
> aac35468ca20a3 Alan Maguire2020-08-04 106
> d5554dd78a454b Daniel Latypov 2021-02-03 107 if (filter_glob) { /*
> a copy was made of each array */
> d5554dd78a454b Daniel Latypov 2021-02-03 108 for (suites =
> suite_set.start; suites < suite_set.end; suites++)
>
>
> This will free "suite_set.start" will in the first iteration through the
> loop
Ah, the loop is supposed to contain `kfree(*suites)`.
I'll fix the patch and resend.
I'm not familiar with conventions but it feels like adding Reported-by
on the amended patch would almost imply the report suggested the need
for the ability to filter suites.
So I'll add an informal attribution in the cover letter.
Thanks!
>
> d5554dd78a454b Daniel Latypov 2021-02-03 109
> kfree(suites);
> d5554dd78a454b Daniel Latypov 2021-02-03 @110
> kfree(suite_set.start);
>
> ^^^
> and then double free it.
>
> d5554dd78a454b Daniel Latypov 2021-02-03 111 }
> d5554dd78a454b Daniel Latypov 2021-02-03 112
> aac35468ca20a3 Alan Maguire2020-08-04 113 return 0;
> aac35468ca20a3 Alan Maguire2020-08-04 114 }
>
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-...@lists.01.org
[PATCH v2 1/3] kunit: add kunit.filter_glob cmdline option to filter suites
E.g. specifying this would run suites with "list" in their name.
kunit.filter_glob=list*
Note: the executor prints out a TAP header that includes the number of
suites we intend to run.
So unless we want to report empty results for filtered-out suites, we
need to do the filtering here in the executor.
It's also probably better in the executor since we most likely don't
want any filtering to apply to tests built as modules.
This code does add a CONFIG_GLOB=y dependency for CONFIG_KUNIT=y.
But the code seems light enough that it shouldn't be an issue.
For now, we only filter on suite names so we don't have to create copies
of the suites themselves, just the array (of arrays) holding them.
The name is rather generic since in the future, we could consider
extending it to a syntax like:
kunit.filter_glob=.
E.g. to run all the del list tests
kunit.filter_glob=list-kunit-test.*del*
But at the moment, it's far easier to manually comment out test cases in
test files as opposed to messing with sets of Kconfig entries to select
specific suites.
So even just doing this makes using kunit far less annoying.
Signed-off-by: Daniel Latypov
---
lib/kunit/Kconfig| 1 +
lib/kunit/executor.c | 91 +++-
2 files changed, 83 insertions(+), 9 deletions(-)
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 00909e6a2443..0b5dfb001bac 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -4,6 +4,7 @@
menuconfig KUNIT
tristate "KUnit - Enable support for unit tests"
+ select GLOB if KUNIT=y
help
Enables support for kernel unit tests (KUnit), a lightweight unit
testing and mocking framework for the Linux kernel. These tests are
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index a95742a4ece7..996efb80dba6 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
#include
+#include
+#include
/*
* These symbols point to the .kunit_test_suites section and are defined in
@@ -11,14 +13,79 @@ extern struct kunit_suite * const * const
__kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
-static void kunit_print_tap_header(void)
+static char *filter_glob;
+module_param(filter_glob, charp, 0);
+
+static struct kunit_suite * const *
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+{
+ int i, n = 0;
+ struct kunit_suite **filtered;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ ++n;
+ }
+
+ if (n == 0)
+ return NULL;
+
+ filtered = kmalloc_array(n + 1, sizeof(*filtered), GFP_KERNEL);
+ if (!filtered)
+ return NULL;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ filtered[n++] = subsuite[i];
+ }
+ filtered[n] = NULL;
+
+ return filtered;
+}
+
+struct suite_set {
+ struct kunit_suite * const * const *start;
+ struct kunit_suite * const * const *end;
+};
+
+static struct suite_set kunit_filter_suites(void)
+{
+ int i;
+ struct kunit_suite * const **copy, * const *filtered_subsuite;
+ struct suite_set filtered;
+
+ const size_t max = __kunit_suites_end - __kunit_suites_start;
+
+ if (!filter_glob) {
+ filtered.start = __kunit_suites_start;
+ filtered.end = __kunit_suites_end;
+ return filtered;
+ }
+
+ copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
+ filtered.start = copy;
+ if (!copy) { /* won't be able to run anything, return an empty set */
+ filtered.end = copy;
+ return filtered;
+ }
+
+ for (i = 0; i < max; ++i) {
+ filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ if (filtered_subsuite)
+ *copy++ = filtered_subsuite;
+ }
+ filtered.end = copy;
+ return filtered;
+}
+
+static void kunit_print_tap_header(struct suite_set *suite_set)
{
struct kunit_suite * const * const *suites, * const *subsuite;
int num_of_suites = 0;
- for (suites = __kunit_suites_start;
-suites < __kunit_suites_end;
-suites++)
+ for (suites = suite_set->start; suites < suite_set->end; suites++)
for (subsuite = *suites; *subsuite != NULL; subsuite++)
num_of_suites++;
@@ -30,12 +97,18 @@ int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
- kunit_print_tap_header();
+ struct suite_set suite_set = kunit_filter_suites();
+
+ kunit_print_tap_header(_set);
+
+ for (suites = suite_set.start; suites < suite_set.end; suites++)
+
[PATCH v2 3/3] kunit: tool: fix unintentional statefulness in run_kernel()
This is a bug that has been present since the first version of this
code.
Using [] as a default parameter is dangerous, since it's mutable.
Example using the REPL:
>>> def bad(param = []):
... param.append(len(param))
... print(param)
...
>>> bad()
[0]
>>> bad()
[0, 1]
This wasn't a concern in the past since it would just keep appending the
same values to it.
E.g. before, `args` would just grow in size like:
[mem=1G', 'console=tty']
[mem=1G', 'console=tty', mem=1G', 'console=tty']
But with now filter_glob, this is more dangerous, e.g.
run_kernel(filter_glob='my-test*') # default modified here
run_kernel() # filter_glob still applies here!
That earlier `filter_glob` will affect all subsequent calls that don't
specify `args`.
Note: currently the kunit tool only calls run_kernel() at most once, so
it's not possible to trigger any negative side-effects right now.
Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit_kernel.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 71b1942f5ccd..6dd3cf6e8efa 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,7 +199,9 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ def run_kernel(self, args=None, build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ if not args:
+ args = []
args.extend(['mem=1G', 'console=tty'])
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
--
2.30.0.365.g02bc693789-goog
[PATCH v2 2/3] kunit: tool: add support for filtering suites by glob
This allows running different subsets of tests, e.g.
$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'
This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit.py| 21 -
tools/testing/kunit/kunit_kernel.py | 4 +++-
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e808a47c839b..a15ee33bb1f5 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -28,12 +28,12 @@ KunitBuildRequest = namedtuple('KunitBuildRequest',
['jobs', 'build_dir', 'alltests',
'make_options'])
KunitExecRequest = namedtuple('KunitExecRequest',
- ['timeout', 'build_dir', 'alltests'])
+ ['timeout', 'build_dir', 'alltests',
'filter_glob'])
KunitParseRequest = namedtuple('KunitParseRequest',
['raw_output', 'input_data', 'build_dir',
'json'])
KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs',
- 'build_dir', 'alltests', 'json',
- 'make_options'])
+ 'build_dir', 'alltests',
'filter_glob',
+ 'json', 'make_options'])
KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
@@ -93,6 +93,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree,
test_start = time.time()
result = linux.run_kernel(
timeout=None if request.alltests else request.timeout,
+filter_glob=request.filter_glob,
build_dir=request.build_dir)
test_end = time.time()
@@ -149,7 +150,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
return build_result
exec_request = KunitExecRequest(request.timeout, request.build_dir,
- request.alltests)
+ request.alltests, request.filter_glob)
exec_result = exec_tests(linux, exec_request)
if exec_result.status != KunitStatus.SUCCESS:
return exec_result
@@ -197,6 +198,14 @@ def add_exec_opts(parser) -> None:
type=int,
default=300,
metavar='timeout')
+ parser.add_argument('filter_glob',
+ help='maximum number of seconds to allow for all
tests '
+ 'to run. This does not include time taken to build
the '
+ 'tests.',
+ type=str,
+ nargs='?',
+ default='',
+ metavar='filter_glob')
def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
@@ -263,6 +272,7 @@ def main(argv, linux=None):
cli_args.jobs,
cli_args.build_dir,
cli_args.alltests,
+ cli_args.filter_glob,
cli_args.json,
cli_args.make_options)
result = run_tests(linux, request)
@@ -304,7 +314,8 @@ def main(argv, linux=None):
exec_request = KunitExecRequest(cli_args.timeout,
cli_args.build_dir,
- cli_args.alltests)
+ cli_args.alltests,
+ cli_args.filter_glob)
exec_result = exec_tests(linux, exec_request)
parse_request = KunitParseRequest(cli_args.raw_output,
exec_result.result,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2076a5a2d060..71b1942f5ccd 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,8 +199,10 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', timeout=None) ->
Iterator[str]:
+ def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
args.extend(['mem=1G', 'console=tty'])
+ if filter_glob:
+ args.append('kunit.filter_glob='+filter_glob)
self._ops.linux_bin(ar
[PATCH v2 0/3] kunit: support running subsets of test suites from kunit.py
When using `kunit.py run` to run tests, users must populate a `kunitconfig` file to select the options the tests are hidden behind and all their dependencies. The patch [1] to allow specifying a path to kunitconfig promises to make this nicer as we can have checked in files corresponding to different sets of tests. But it's still annoying 1) when trying to run a subet of tests 2) when you want to run tests that don't have such a pre-existing kunitconfig and selecting all the necessary options is tricky. This patch series aims to alleviate both: 1) `kunit.py run 'my-suite-*'` I.e. use my current kunitconfig, but just run suites that match this glob 2) `kunit.py run --alltests 'my-suite-*'` I.e. use allyesconfig so I don't have to worry about writing a kunitconfig at all. See the first commit message for more details and discussion about future work. This patch series also includes a bugfix for a latent bug that can't be triggered right now but has worse consequences as a result of the changes needed to plumb in this suite name glob. [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlaty...@google.com/ --- v1 -> v2: Fix free of `suites` subarray in suite_set. Found by Dan Carpenter and kernel test robot. Daniel Latypov (3): kunit: add kunit.filter_glob cmdline option to filter suites kunit: tool: add support for filtering suites by glob kunit: tool: fix unintentional statefulness in run_kernel() lib/kunit/Kconfig | 1 + lib/kunit/executor.c| 91 ++--- tools/testing/kunit/kunit.py| 21 +-- tools/testing/kunit/kunit_kernel.py | 6 +- 4 files changed, 104 insertions(+), 15 deletions(-) base-commit: 88bb507a74ea7d75fa49edd421eaa710a7d80598 -- 2.30.0.365.g02bc693789-goog
[PATCH 3/3] kunit: tool: fix unintentional statefulness in run_kernel()
This is a bug that has been present since the first version of this
code.
Using [] as a default parameter is dangerous, since it's mutable.
Example using the REPL:
>>> def bad(param = []):
... param.append(len(param))
... print(param)
...
>>> bad()
[0]
>>> bad()
[0, 1]
This wasn't a concern in the past since it would just keep appending the
same values to it.
E.g. before, `args` would just grow in size like:
[mem=1G', 'console=tty']
[mem=1G', 'console=tty', mem=1G', 'console=tty']
But with now filter_glob, this is more dangerous, e.g.
run_kernel(filter_glob='my-test*') # default modified here
run_kernel() # filter_glob still applies here!
That earlier `filter_glob` will affect all subsequent calls that don't
specify `args`.
Note: currently the kunit tool only calls run_kernel() at most once, so
it's not possible to trigger any negative side-effects right now.
Fixes: 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests")
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit_kernel.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 71b1942f5ccd..6dd3cf6e8efa 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,7 +199,9 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ def run_kernel(self, args=None, build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
+ if not args:
+ args = []
args.extend(['mem=1G', 'console=tty'])
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
--
2.30.0.365.g02bc693789-goog
[PATCH 2/3] kunit: tool: add support for filtering suites by glob
This allows running different subsets of tests, e.g.
$ ./tools/testing/kunit/kunit.py build
$ ./tools/testing/kunit/kunit.py exec 'list*'
$ ./tools/testing/kunit/kunit.py exec 'kunit*'
This passes the "kunit_filter.glob" commandline option to the UML
kernel, which currently only supports filtering by suite name.
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit.py| 21 -
tools/testing/kunit/kunit_kernel.py | 4 +++-
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e808a47c839b..a15ee33bb1f5 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -28,12 +28,12 @@ KunitBuildRequest = namedtuple('KunitBuildRequest',
['jobs', 'build_dir', 'alltests',
'make_options'])
KunitExecRequest = namedtuple('KunitExecRequest',
- ['timeout', 'build_dir', 'alltests'])
+ ['timeout', 'build_dir', 'alltests',
'filter_glob'])
KunitParseRequest = namedtuple('KunitParseRequest',
['raw_output', 'input_data', 'build_dir',
'json'])
KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs',
- 'build_dir', 'alltests', 'json',
- 'make_options'])
+ 'build_dir', 'alltests',
'filter_glob',
+ 'json', 'make_options'])
KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0]
@@ -93,6 +93,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree,
test_start = time.time()
result = linux.run_kernel(
timeout=None if request.alltests else request.timeout,
+filter_glob=request.filter_glob,
build_dir=request.build_dir)
test_end = time.time()
@@ -149,7 +150,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
return build_result
exec_request = KunitExecRequest(request.timeout, request.build_dir,
- request.alltests)
+ request.alltests, request.filter_glob)
exec_result = exec_tests(linux, exec_request)
if exec_result.status != KunitStatus.SUCCESS:
return exec_result
@@ -197,6 +198,14 @@ def add_exec_opts(parser) -> None:
type=int,
default=300,
metavar='timeout')
+ parser.add_argument('filter_glob',
+ help='maximum number of seconds to allow for all
tests '
+ 'to run. This does not include time taken to build
the '
+ 'tests.',
+ type=str,
+ nargs='?',
+ default='',
+ metavar='filter_glob')
def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
@@ -263,6 +272,7 @@ def main(argv, linux=None):
cli_args.jobs,
cli_args.build_dir,
cli_args.alltests,
+ cli_args.filter_glob,
cli_args.json,
cli_args.make_options)
result = run_tests(linux, request)
@@ -304,7 +314,8 @@ def main(argv, linux=None):
exec_request = KunitExecRequest(cli_args.timeout,
cli_args.build_dir,
- cli_args.alltests)
+ cli_args.alltests,
+ cli_args.filter_glob)
exec_result = exec_tests(linux, exec_request)
parse_request = KunitParseRequest(cli_args.raw_output,
exec_result.result,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2076a5a2d060..71b1942f5ccd 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -199,8 +199,10 @@ class LinuxSourceTree(object):
return False
return self.validate_config(build_dir)
- def run_kernel(self, args=[], build_dir='', timeout=None) ->
Iterator[str]:
+ def run_kernel(self, args=[], build_dir='', filter_glob='',
timeout=None) -> Iterator[str]:
args.extend(['mem=1G', 'console=tty'])
+ if filter_glob:
+ args.append('kunit.filter_glob='+filter_glob)
self._ops.linux_bin(ar
[PATCH 0/3] kunit: support running subsets of test suites from
When using `kunit.py run` to run tests, users must populate a `kunitconfig` file to select the options the tests are hidden behind and all their dependencies. The patch [1] to allow specifying a path to kunitconfig promises to make this nicer as we can have checked in files corresponding to different sets of tests. But it's still annoying 1) when trying to run a subet of tests 2) when you want to run tests that don't have such a pre-existing kunitconfig and selecting all the necessary options is tricky. This patch series aims to alleviate both: 1) `kunit.py run 'my-suite-*'` I.e. use my current kunitconfig, but just run suites that match this glob 2) `kunit.py run --alltests 'my-suite-*'` I.e. use allyesconfig so I don't have to worry about writing a kunitconfig at all (this is a bit overkill, but it works!) See the first commit message for more details and discussion about future work. This patch series also includes a bugfix for a latent bug that can't be triggered right now but has worse consequences as a result of the changes needed to plumb in this suite name glob. [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlaty...@google.com/ Daniel Latypov (3): kunit: add kunit.filter_glob cmdline option to filter suites kunit: tool: add support for filtering suites by glob kunit: tool: fix unintentional statefulness in run_kernel() lib/kunit/Kconfig | 1 + lib/kunit/executor.c| 85 ++--- tools/testing/kunit/kunit.py| 21 +-- tools/testing/kunit/kunit_kernel.py | 6 +- 4 files changed, 99 insertions(+), 14 deletions(-) base-commit: 88bb507a74ea7d75fa49edd421eaa710a7d80598 -- 2.30.0.365.g02bc693789-goog
[PATCH 1/3] kunit: add kunit.filter_glob cmdline option to filter suites
E.g. specifying this would run suites with "list" in their name.
kunit.filter_glob=list*
Note: the executor prints out a TAP header that includes the number of
suites we intend to run.
So unless we want to report empty results for filtered-out suites, we
need to do the filtering here in the executor.
It's also probably better in the executor since we most likely don't
want any filtering to apply to tests built as modules.
This code does add a CONFIG_GLOB=y dependency for CONFIG_KUNIT=y.
But the code seems light enough that it shouldn't be an issue.
For now, we only filter on suite names so we don't have to create copies
of the suites themselves, just the array (of arrays) holding them.
The name is rather generic since in the future, we could consider
extending it to a syntax like:
kunit.filter_glob=.
E.g. to run all the del list tests
kunit.filter_glob=list-kunit-test.*del*
But at the moment, it's far easier to manually comment out test cases in
test files as opposed to messing with sets of Kconfig entries to select
specific suites.
So even just doing this makes using kunit far less annoying.
Signed-off-by: Daniel Latypov
---
lib/kunit/Kconfig| 1 +
lib/kunit/executor.c | 91 +++-
2 files changed, 83 insertions(+), 9 deletions(-)
diff --git a/lib/kunit/Kconfig b/lib/kunit/Kconfig
index 00909e6a2443..0b5dfb001bac 100644
--- a/lib/kunit/Kconfig
+++ b/lib/kunit/Kconfig
@@ -4,6 +4,7 @@
menuconfig KUNIT
tristate "KUnit - Enable support for unit tests"
+ select GLOB if KUNIT=y
help
Enables support for kernel unit tests (KUnit), a lightweight unit
testing and mocking framework for the Linux kernel. These tests are
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index a95742a4ece7..25a891285816 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
#include
+#include
+#include
/*
* These symbols point to the .kunit_test_suites section and are defined in
@@ -11,14 +13,79 @@ extern struct kunit_suite * const * const
__kunit_suites_end[];
#if IS_BUILTIN(CONFIG_KUNIT)
-static void kunit_print_tap_header(void)
+static char *filter_glob;
+module_param(filter_glob, charp, 0);
+
+static struct kunit_suite * const *
+kunit_filter_subsuite(struct kunit_suite * const * const subsuite)
+{
+ int i, n = 0;
+ struct kunit_suite **filtered;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ ++n;
+ }
+
+ if (n == 0)
+ return NULL;
+
+ filtered = kmalloc_array(n + 1, sizeof(*filtered), GFP_KERNEL);
+ if (!filtered)
+ return NULL;
+
+ n = 0;
+ for (i = 0; subsuite[i] != NULL; ++i) {
+ if (glob_match(filter_glob, subsuite[i]->name))
+ filtered[n++] = subsuite[i];
+ }
+ filtered[n] = NULL;
+
+ return filtered;
+}
+
+struct suite_set {
+ struct kunit_suite * const * const *start;
+ struct kunit_suite * const * const *end;
+};
+
+static struct suite_set kunit_filter_suites(void)
+{
+ int i;
+ struct kunit_suite * const **copy, * const *filtered_subsuite;
+ struct suite_set filtered;
+
+ const size_t max = __kunit_suites_end - __kunit_suites_start;
+
+ if (!filter_glob) {
+ filtered.start = __kunit_suites_start;
+ filtered.end = __kunit_suites_end;
+ return filtered;
+ }
+
+ copy = kmalloc_array(max, sizeof(*filtered.start), GFP_KERNEL);
+ filtered.start = copy;
+ if (!copy) { /* won't be able to run anything, return an empty set */
+ filtered.end = copy;
+ return filtered;
+ }
+
+ for (i = 0; i < max; ++i) {
+ filtered_subsuite =
kunit_filter_subsuite(__kunit_suites_start[i]);
+ if (filtered_subsuite)
+ *copy++ = filtered_subsuite;
+ }
+ filtered.end = copy;
+ return filtered;
+}
+
+static void kunit_print_tap_header(struct suite_set *suite_set)
{
struct kunit_suite * const * const *suites, * const *subsuite;
int num_of_suites = 0;
- for (suites = __kunit_suites_start;
-suites < __kunit_suites_end;
-suites++)
+ for (suites = suite_set->start; suites < suite_set->end; suites++)
for (subsuite = *suites; *subsuite != NULL; subsuite++)
num_of_suites++;
@@ -30,12 +97,18 @@ int kunit_run_all_tests(void)
{
struct kunit_suite * const * const *suites;
- kunit_print_tap_header();
+ struct suite_set suite_set = kunit_filter_suites();
+
+ kunit_print_tap_header(_set);
+
+ for (suites = suite_set.start; suites < suite_set.end; suites++)
+
[PATCH v2] kunit: make kunit_tool accept optional path to .kunitconfig fragment
Currently running tests via KUnit tool means tweaking a .kunitconfig
file, which you'd keep around locally and never commit.
This changes makes it so users can pass in a path to a kunitconfig.
One of the imagined use cases is having kunitconfig fragments in-tree
to formalize interesting sets of tests for features/subsystems, e.g.
$ ./tools/testing/kunit/kunit.py run --kunticonfig=fs/ext4/kunitconfig
For now, this hypothetical fs/ext4/kunitconfig would contain
CONFIG_KUNIT=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_KUNIT_TESTS=y
At the moment, it's not hard to manually whip up this file, but as more
and more tests get added, this will get tedious.
It also opens the door to documenting how to run all the tests relevant
to a specific subsystem or feature as a simple one-liner.
This can be seen as an analogue to tools/testing/selftests/*/config
But in the case of KUnit, the tests live in the same directory as the
code-under-test, so it feels more natural to allow the kunitconfig
fragments to live anywhere. (Though, people could create a separate
directory if wanted; this patch imposes no restrictions on the path).
Signed-off-by: Daniel Latypov
---
Changes since v1: change from a positional arg to a flag --kunitconfig.
Ensure that it gets added for `kunit.py config` and all other commands.
---
tools/testing/kunit/kunit.py | 9 +---
tools/testing/kunit/kunit_kernel.py| 12 ++
tools/testing/kunit/kunit_tool_test.py | 32 ++
3 files changed, 46 insertions(+), 7 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e808a47c839b..02871a363f76 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -182,6 +182,9 @@ def add_common_opts(parser) -> None:
parser.add_argument('--alltests',
help='Run all KUnit tests through allyesconfig',
action='store_true')
+ parser.add_argument('--kunitconfig',
+help='Path to Kconfig fragment that enables KUnit
tests',
+metavar='kunitconfig')
def add_build_opts(parser) -> None:
parser.add_argument('--jobs',
@@ -256,7 +259,7 @@ def main(argv, linux=None):
os.mkdir(cli_args.build_dir)
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitRequest(cli_args.raw_output,
cli_args.timeout,
@@ -274,7 +277,7 @@ def main(argv, linux=None):
os.mkdir(cli_args.build_dir)
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitConfigRequest(cli_args.build_dir,
cli_args.make_options)
@@ -286,7 +289,7 @@ def main(argv, linux=None):
sys.exit(1)
elif cli_args.subcommand == 'build':
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitBuildRequest(cli_args.jobs,
cli_args.build_dir,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2076a5a2d060..0b461663e7d9 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -123,7 +123,7 @@ def get_outfile_path(build_dir) -> str:
class LinuxSourceTree(object):
"""Represents a Linux kernel source tree with KUnit tests."""
- def __init__(self, build_dir: str, load_config=True,
defconfig=DEFAULT_KUNITCONFIG_PATH) -> None:
+ def __init__(self, build_dir: str, load_config=True,
kunitconfig_path='') -> None:
signal.signal(signal.SIGINT, self.signal_handler)
self._ops = LinuxSourceTreeOperations()
@@ -131,9 +131,13 @@ class LinuxSourceTree(object):
if not load_config:
return
- kunitconfig_path = get_kunitconfig_path(build_dir)
- if not os.path.exists(kunitconfig_path):
- shutil.copyfile(defconfig, kunitconfig_path)
+ if kunitconfig_path:
+ if not os.path.exists(kunitconfig_path):
+ raise ConfigError(f'Specified kunitconfig
({kunitconfig_path}) does not exist')
+ else:
+ kuni
Re: [PATCH] kunit: don't show `1 == 1` in failed assertion messages
On Thu, Jan 28, 2021 at 8:51 PM David Gow wrote:
>
> On Fri, Jan 29, 2021 at 10:26 AM Daniel Latypov wrote:
> >
> > Currently, given something (fairly dystopian) like
> > > KUNIT_EXPECT_EQ(test, 2 + 2, 5)
> >
> > KUnit will prints a failure message like this.
> > > Expected 2 + 2 == 5, but
> > > 2 + 2 == 4
> > > 5 == 5
> >
> > With this patch, the output just becomes
> > > Expected 2 + 2 == 5, but
> > > 2 + 2 == 4
> >
> > This patch is slightly hacky, but it's quite common* to compare an
> > expression to a literal integer value, so this can make KUnit less
> > chatty in many cases. (This patch also fixes variants like
> > KUNIT_EXPECT_GT, LE, et al.).
> >
> > It also allocates an additional string briefly, but given this only
> > happens on test failures, it doesn't seem too bad a tradeoff.
> > Also, in most cases it'll realize the lengths are unequal and bail out
> > before the allocation.
> >
> > We could save the result of the formatted string to avoid wasting this
> > extra work, but it felt cleaner to leave it as-is.
> >
> > Edge case: for something silly and unrealistic like
> > > KUNIT_EXPECT_EQ(test, 4, 5);
> >
> > It'll generate this message with a trailing "but"
> > > Expected 2 + 2 == 5, but
> > >
>
> I assume this is supposed to say "Expected 4 == 5" here.
> (I tested it to make sure, and that's what it did here.)
Ah yes, too much copy-paste.
>
> Personally, I'd ideally like to get rid of the ", but", or even add a
> "but 4 != 5" style second line. Particularly in case the next line in
> the output might be confused for the rest of a sentence.
Given the apparent interest in other types (STR_EQ) of literal
ellision, maybe this should be done.
But I'd be tempted to have that change come later once at least the
str_eq version is in place.
>
> That being said, this is a pretty silly edge case: I'd be worried if
> we ever saw that case in an actual submitted test. People might see it
> a bit while debugging, though: particularly if they're using
> KUNIT_EXPECT_EQ(test, 1, 2) as a way of forcing a test to fail. (I've
> done this while testing tooling, for instance.)
Same/Agreed on all points.
>
> >
> > It didn't feel worth adding a check up-front to see if both sides are
> > literals to handle this better.
> >
> > *A quick grep suggests 100+ comparisons to an integer literal as the
> > right hand side.
> >
> > Signed-off-by: Daniel Latypov
> > ---
>
> I tested this, and it works well: the results are definitely more
> human readable. I could see it making things slightly more complicated
> for people who wanted to automatically parse assertion errors, but
> no-one is doing that, and the extra complexity is pretty minimal
> anyway.
Hmm, machine parsing of the contents of failures is interesting.
But in general, that feels that requires a more structured format.
I hate to invoke it, but the tooling I've seen that's parsed the
"expected" and "actual" values has represented them as XML elements.
>
> One thing which might be worth doing is expanding this to
> KUNIT_EXPECT_STREQ() and/or KUNIT_EXPECT_PTR_EQ(). These have slightly
> more complicated formatting (quotes, leading 0s, etc), though.
> Comparing pointer literals is pretty unlikely to show up, though, so I
> don't think it's as important. (I thought that maybe the KASAN shadow
> memory tests might use them, but a quick look didn't reveal any.)
>
Ack. Actually, the string literal check was smaller, see below.
I debated sending a patch out for that, but this case mattered more
and I wasn't sure if it would be acceptable or not.
It felt it would be incongruous to only handle strings and not the
much more common integer case.
So if the hackier, more costly integer comparison seems fine, I might
actually go and send out the str patch that I already have sitting
around anyways.
+/* Checks if KUNIT_EXPECT_STREQ() args were string literals.
+ * Note: `text` will have ""s where as `value` will not.
+ */
+static bool is_str_literal(const char *text, const char *value)
+{
+ int len;
+
+ len = strlen(text);
+ if (len < 2) return false;
+ if (text[0] != '\"' || text[len-1] != '\"') return false;
+
+ return strncmp(text+1, value, len-2) == 0;
+}
+
This produces
[10:05:59] Expected str == "world", but
[10:05:59] str == hello
One misgiving I had was whether we should "fix" the string printing to
quote the values or not before adding `is_str_literal()` in.
Having just "str == hello" where neither
Re: [PATCH] kunit: make kunit_tool accept optional path to .kunitconfig fragment
On Thu, Jan 28, 2021 at 10:33 PM David Gow wrote:
>
> On Sat, Jan 23, 2021 at 8:17 AM Daniel Latypov wrote:
> >
> > Currently running tests via KUnit tool means tweaking a .kunitconfig
> > file, which you'd keep around locally and never commit.
> > This changes makes it so users can pass in a path to a kunitconfig.
> >
> > One of the imagined use cases is having kunitconfig fragments in-tree
> > to formalize interesting sets of tests for features/subsystems, e.g.
> > $ ./tools/testing/kunit/kunit.py run fs/ext4/kunitconfig
> >
> > For now, this hypothetical fs/ext4/kunitconfig would contain
> > CONFIG_KUNIT=y
> > CONFIG_EXT4_FS=y
> > CONFIG_EXT4_KUNIT_TESTS=y
> >
> > At the moment, it's not hard to manually whip up this file, but as more
> > and more tests get added, this will get tedious.
> >
> > It also opens the door to documenting how to run all the tests relevant
> > to a specific subsystem or feature as a simple one-liner.
> >
> > This can be seen as an analogue to tools/testing/selftests/*/config
> > But in the case of KUnit, the tests live in the same directory as the
> > code-under-test, so it feels more natural to allow the kunitconfig
> > fragments to live anywhere. (Though, people could create a separate
> > directory if wanted; this patch imposes no restrictions on the path).
> >
> > Signed-off-by: Daniel Latypov
> > ---
>
> Really glad this is finally happening. I tried it out, and it seemed
> to work pretty well.
>
> I was wondering whether a positional argument like this was best, or
> whether it'd be better to have an explicitly named argument
> (--kunitconfig=path). Thinking about it though, I'm quite happy with
> having this as-is: the only real other contender for a coveted
> positional argument spot would've been the name of a test or test
> suite (e.g., kunit.py run ext4_inode_test), and that's not really
> possible with the kunit_tool architecture as-is.
Same, I was on the fence about this for a good while.
I had originally intended to make it a flag, but
* as you noted, that would require bigger changes to kunit_tool, but
also KUnit (the C code) itself to handle this.
* I felt that the kunitconfig fragment essentially takes the place of it.
* E.g. If I want to run a specific test, I can manually create a
fragment for just that.
* It's sadly more work than just specifying a single test for
bisection, but it would work.
That said, adding in support for running specific tests/suites by name
seems like it would be a somewhat smaller change than I thought.
So I might lean more towards making this a flag now.
Will wait to get Brendan's opinion and then send out a v2.
>
> One other comment below (should this work for kunit.py config?),
> otherwise it looks good.
>
> -- David
>
> > tools/testing/kunit/kunit.py | 9 ++---
> > tools/testing/kunit/kunit_kernel.py| 12
> > tools/testing/kunit/kunit_tool_test.py | 25 +
> > 3 files changed, 39 insertions(+), 7 deletions(-)
> >
> > diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> > index e808a47c839b..3204a23bd16e 100755
> > --- a/tools/testing/kunit/kunit.py
> > +++ b/tools/testing/kunit/kunit.py
> > @@ -188,6 +188,9 @@ def add_build_opts(parser) -> None:
> > help='As in the make command, "Specifies the
> > number of '
> > 'jobs (commands) to run simultaneously."',
> > type=int, default=8, metavar='jobs')
> > + parser.add_argument('kunitconfig',
> > +help='Path to Kconfig fragment that enables
> > KUnit tests',
> > +type=str, nargs='?', metavar='kunitconfig')
> >
>
> Should this maybe be in add_common_opts()? I'd assume that we want
> kunit.py config to accept this custom kunitconfig path as well.
Good point.
Moved it over into add_common_opts() and including this minimal test
case as well since `kunit.py config` is probably a very key command
that should work with custom fragments.
diff --git a/tools/testing/kunit/kunit_tool_test.py
b/tools/testing/kunit/kunit_tool_test.py
index 533fe41b5123..a74877ee2c90 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -424,5 +424,12 @@ class KUnitMainTest(unittest.TestCase):
# Just verify that we parsed and initialized it correctly here.
mock_linux_init.assert_called_once_with('.kunit',
kunitconfig_path='mykunitconfig')
+ @mock.patch.object(kunit_kernel, 'LinuxSourceTree')
+ def test_config_kunitcon
[PATCH] kunit: don't show `1 == 1` in failed assertion messages
Currently, given something (fairly dystopian) like
> KUNIT_EXPECT_EQ(test, 2 + 2, 5)
KUnit will prints a failure message like this.
> Expected 2 + 2 == 5, but
> 2 + 2 == 4
> 5 == 5
With this patch, the output just becomes
> Expected 2 + 2 == 5, but
> 2 + 2 == 4
This patch is slightly hacky, but it's quite common* to compare an
expression to a literal integer value, so this can make KUnit less
chatty in many cases. (This patch also fixes variants like
KUNIT_EXPECT_GT, LE, et al.).
It also allocates an additional string briefly, but given this only
happens on test failures, it doesn't seem too bad a tradeoff.
Also, in most cases it'll realize the lengths are unequal and bail out
before the allocation.
We could save the result of the formatted string to avoid wasting this
extra work, but it felt cleaner to leave it as-is.
Edge case: for something silly and unrealistic like
> KUNIT_EXPECT_EQ(test, 4, 5);
It'll generate this message with a trailing "but"
> Expected 2 + 2 == 5, but
>
It didn't feel worth adding a check up-front to see if both sides are
literals to handle this better.
*A quick grep suggests 100+ comparisons to an integer literal as the
right hand side.
Signed-off-by: Daniel Latypov
---
lib/kunit/assert.c | 39 +--
1 file changed, 33 insertions(+), 6 deletions(-)
diff --git a/lib/kunit/assert.c b/lib/kunit/assert.c
index 33acdaa28a7d..e0ec7d6fed6f 100644
--- a/lib/kunit/assert.c
+++ b/lib/kunit/assert.c
@@ -85,6 +85,29 @@ void kunit_ptr_not_err_assert_format(const struct
kunit_assert *assert,
}
EXPORT_SYMBOL_GPL(kunit_ptr_not_err_assert_format);
+/* Checks if `text` is a literal representing `value`, e.g. "5" and 5 */
+static bool is_literal(struct kunit *test, const char *text, long long value,
+ gfp_t gfp)
+{
+ char *buffer;
+ int len;
+ bool ret;
+
+ len = snprintf(NULL, 0, "%lld", value);
+ if (strlen(text) != len)
+ return false;
+
+ buffer = kunit_kmalloc(test, len+1, gfp);
+ if (!buffer)
+ return false;
+
+ snprintf(buffer, len+1, "%lld", value);
+ ret = strncmp(buffer, text, len) == 0;
+
+ kunit_kfree(test, buffer);
+ return ret;
+}
+
void kunit_binary_assert_format(const struct kunit_assert *assert,
struct string_stream *stream)
{
@@ -97,12 +120,16 @@ void kunit_binary_assert_format(const struct kunit_assert
*assert,
binary_assert->left_text,
binary_assert->operation,
binary_assert->right_text);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld\n",
- binary_assert->left_text,
- binary_assert->left_value);
- string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld",
- binary_assert->right_text,
- binary_assert->right_value);
+ if (!is_literal(stream->test, binary_assert->left_text,
+ binary_assert->left_value, stream->gfp))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s ==
%lld\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ if (!is_literal(stream->test, binary_assert->right_text,
+ binary_assert->right_value, stream->gfp))
+ string_stream_add(stream, KUNIT_SUBSUBTEST_INDENT "%s == %lld",
+ binary_assert->right_text,
+ binary_assert->right_value);
kunit_assert_print_msg(assert, stream);
}
EXPORT_SYMBOL_GPL(kunit_binary_assert_format);
base-commit: e5ff2cb9cf67a542f2ec7fb87e24934c88b32678
--
2.30.0.365.g02bc693789-goog
[PATCH] Documentation: kunit: add tips.rst for small examples
./usage.rst contains fairly long examples and explanations of things
like how to fake a class and how to use parameterized tests (and how you
could do table-driven tests yourself).
It's not exactly necessary information, so we add a new page with more
digestible tips like "use kunit_kzalloc() instead of kzalloc() so you
don't have to worry about calling kfree() yourself" and the like.
Change start.rst to point users to this new page first and let them know
that usage.rst is more of optional further reading.
Signed-off-by: Daniel Latypov
---
Documentation/dev-tools/kunit/index.rst | 2 +
Documentation/dev-tools/kunit/start.rst | 4 +-
Documentation/dev-tools/kunit/tips.rst | 115
3 files changed, 120 insertions(+), 1 deletion(-)
create mode 100644 Documentation/dev-tools/kunit/tips.rst
diff --git a/Documentation/dev-tools/kunit/index.rst
b/Documentation/dev-tools/kunit/index.rst
index c234a3ab3c34..848478838347 100644
--- a/Documentation/dev-tools/kunit/index.rst
+++ b/Documentation/dev-tools/kunit/index.rst
@@ -13,6 +13,7 @@ KUnit - Unit Testing for the Linux Kernel
api/index
style
faq
+ tips
What is KUnit?
==
@@ -88,6 +89,7 @@ How do I use it?
* :doc:`start` - for new users of KUnit
+* :doc:`tips` - for short examples of best practices
* :doc:`usage` - for a more detailed explanation of KUnit features
* :doc:`api/index` - for the list of KUnit APIs used for testing
* :doc:`kunit-tool` - for more information on the kunit_tool helper script
diff --git a/Documentation/dev-tools/kunit/start.rst
b/Documentation/dev-tools/kunit/start.rst
index 454f307813ea..c09e2747c958 100644
--- a/Documentation/dev-tools/kunit/start.rst
+++ b/Documentation/dev-tools/kunit/start.rst
@@ -233,5 +233,7 @@ Congrats! You just wrote your first KUnit test!
Next Steps
==
-* Check out the :doc:`usage` page for a more
+* Check out the :doc:`tips` page for tips on
+writing idiomatic KUnit tests.
+* Optional: see the :doc:`usage` page for a more
in-depth explanation of KUnit.
diff --git a/Documentation/dev-tools/kunit/tips.rst
b/Documentation/dev-tools/kunit/tips.rst
new file mode 100644
index ..a6ca0af14098
--- /dev/null
+++ b/Documentation/dev-tools/kunit/tips.rst
@@ -0,0 +1,115 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Tips For Writing KUnit Tests
+
+
+Exiting early on failed expectations
+
+
+``KUNIT_EXPECT_EQ`` and friends will mark the test as failed and continue
+execution. In some cases, it's unsafe to continue and you can use the
+``KUNIT_ASSERT`` variant to exit on failure.
+
+.. code-block:: c
+
+ void example_test_user_alloc_function(struct kunit *test)
+ {
+ void *object = alloc_some_object_for_me();
+
+ /* Make sure we got a valid pointer back. */
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, object);
+ do_something_with_object(object);
+ }
+
+Allocating memory
+-
+
+Where you would use ``kzalloc``, you should prefer ``kunit_kzalloc`` instead.
+KUnit will ensure the memory is freed once the test completes.
+
+This is particularly useful since it lets you use the ``KUNIT_ASSERT_EQ``
+macros to exit early from a test without having to worry about remembering to
+call ``kfree``.
+
+Example:
+
+.. code-block:: c
+
+ void example_test_allocation(struct kunit *test)
+ {
+ char *buffer = kunit_kzalloc(test, 16, GFP_KERNEL);
+ /* Ensure allocation succeeded. */
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer);
+
+ KUNIT_ASSERT_STREQ(test, buffer, "");
+ }
+
+
+Testing static functions
+
+
+If you don't want to expose functions or variables just for testing, one option
+is to conditionally ``#include`` the test file at the end of your .c file, e.g.
+
+.. code-block:: c
+
+ /* In my_file.c */
+
+ static int do_interesting_thing();
+
+ #ifdef CONFIG_MY_KUNIT_TEST
+ #include "my_kunit_test.c"
+ #endif
+
+Injecting test-only code
+
+
+Similarly to the above, it can be useful to add test-specific logic.
+
+.. code-block:: c
+
+ /* In my_file.h */
+
+ #ifdef CONFIG_MY_KUNIT_TEST
+ /* Defined in my_kunit_test.c */
+ void test_only_hook(void);
+ #else
+ void test_only_hook(void) { }
+ #endif
+
+TODO(dlaty...@google.com): add an example of using ``current->kunit_test`` in
+such a hook when it's not only updated for ``CONFIG_KASAN=y``.
+
+Customizing error messages
+--
+
+Each of the ``KUNIT_EXPECT`` and ``KUNIT_ASSERT`` macros have a ``_MSG``
variant.
+These take a format string and arguments to provide additional context to the
automatically generated err
[PATCH] kunit: make kunit_tool accept optional path to .kunitconfig fragment
Currently running tests via KUnit tool means tweaking a .kunitconfig
file, which you'd keep around locally and never commit.
This changes makes it so users can pass in a path to a kunitconfig.
One of the imagined use cases is having kunitconfig fragments in-tree
to formalize interesting sets of tests for features/subsystems, e.g.
$ ./tools/testing/kunit/kunit.py run fs/ext4/kunitconfig
For now, this hypothetical fs/ext4/kunitconfig would contain
CONFIG_KUNIT=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_KUNIT_TESTS=y
At the moment, it's not hard to manually whip up this file, but as more
and more tests get added, this will get tedious.
It also opens the door to documenting how to run all the tests relevant
to a specific subsystem or feature as a simple one-liner.
This can be seen as an analogue to tools/testing/selftests/*/config
But in the case of KUnit, the tests live in the same directory as the
code-under-test, so it feels more natural to allow the kunitconfig
fragments to live anywhere. (Though, people could create a separate
directory if wanted; this patch imposes no restrictions on the path).
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit.py | 9 ++---
tools/testing/kunit/kunit_kernel.py| 12
tools/testing/kunit/kunit_tool_test.py | 25 +
3 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index e808a47c839b..3204a23bd16e 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -188,6 +188,9 @@ def add_build_opts(parser) -> None:
help='As in the make command, "Specifies the
number of '
'jobs (commands) to run simultaneously."',
type=int, default=8, metavar='jobs')
+ parser.add_argument('kunitconfig',
+help='Path to Kconfig fragment that enables KUnit
tests',
+type=str, nargs='?', metavar='kunitconfig')
def add_exec_opts(parser) -> None:
parser.add_argument('--timeout',
@@ -256,7 +259,7 @@ def main(argv, linux=None):
os.mkdir(cli_args.build_dir)
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitRequest(cli_args.raw_output,
cli_args.timeout,
@@ -274,7 +277,7 @@ def main(argv, linux=None):
os.mkdir(cli_args.build_dir)
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitConfigRequest(cli_args.build_dir,
cli_args.make_options)
@@ -286,7 +289,7 @@ def main(argv, linux=None):
sys.exit(1)
elif cli_args.subcommand == 'build':
if not linux:
- linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir)
+ linux =
kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_path=cli_args.kunitconfig)
request = KunitBuildRequest(cli_args.jobs,
cli_args.build_dir,
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2076a5a2d060..0b461663e7d9 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -123,7 +123,7 @@ def get_outfile_path(build_dir) -> str:
class LinuxSourceTree(object):
"""Represents a Linux kernel source tree with KUnit tests."""
- def __init__(self, build_dir: str, load_config=True,
defconfig=DEFAULT_KUNITCONFIG_PATH) -> None:
+ def __init__(self, build_dir: str, load_config=True,
kunitconfig_path='') -> None:
signal.signal(signal.SIGINT, self.signal_handler)
self._ops = LinuxSourceTreeOperations()
@@ -131,9 +131,13 @@ class LinuxSourceTree(object):
if not load_config:
return
- kunitconfig_path = get_kunitconfig_path(build_dir)
- if not os.path.exists(kunitconfig_path):
- shutil.copyfile(defconfig, kunitconfig_path)
+ if kunitconfig_path:
+ if not os.path.exists(kunitconfig_path):
+ raise ConfigError(f'Specified kunitconfig
({kunitconfig_path}) does not exist')
+ else:
+ kunitconfig_path = get_kunitconfig_path(build_dir)
+ if not os.path.exists(kunitconfig
[PATCH] KUnit: Docs: make start.rst example Kconfig follow style.rst
The primary change is that we want to encourage people to respect KUNIT_ALL_TESTS to make it easy to run all the relevant tests for a given config. Signed-off-by: Daniel Latypov --- Documentation/dev-tools/kunit/start.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst index 454f307813ea..560f27af4619 100644 --- a/Documentation/dev-tools/kunit/start.rst +++ b/Documentation/dev-tools/kunit/start.rst @@ -196,8 +196,9 @@ Now add the following to ``drivers/misc/Kconfig``: .. code-block:: kconfig config MISC_EXAMPLE_TEST - bool "Test for my example" + tristate "Test for my example" if !KUNIT_ALL_TESTS depends on MISC_EXAMPLE && KUNIT=y + default KUNIT_ALL_TESTS and the following to ``drivers/misc/Makefile``: base-commit: 146620506274bd24d52fb1c589110a30eed8240b -- 2.30.0.296.g2bfb1c46d8-goog
[PATCH v3] lib: add basic KUnit test for lib/math
Add basic test coverage for files that don't require any config options:
* gcd.c
* lcm.c
* int_sqrt.c
* reciprocal_div.c
(Ignored int_pow.c since it's a simple textbook algorithm.)
These tests aren't particularly interesting, but they
* provide short and simple examples of parameterized tests
* provide a place to add tests for any new files in this dir
* are written so adding new test cases to cover edge cases should be easy
Signed-off-by: Daniel Latypov
---
Changes since v2: mv math_test.c => math_kunit.c
Changes since v1:
* Rebase and rewrite to use the new parameterized testing support.
* misc: fix overflow in literal and inline int_sqrt format string.
* related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
for testing many inputs") was merged explaining the patterns shown here.
* there's an in-flight patch to update it for parameterized testing.
v1: https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
---
lib/math/Kconfig | 5 ++
lib/math/Makefile | 2 +
lib/math/math_kunit.c | 197 ++
3 files changed, 204 insertions(+)
create mode 100644 lib/math/math_kunit.c
diff --git a/lib/math/Kconfig b/lib/math/Kconfig
index f19bc9734fa7..6ba8680439c1 100644
--- a/lib/math/Kconfig
+++ b/lib/math/Kconfig
@@ -15,3 +15,8 @@ config PRIME_NUMBERS
config RATIONAL
bool
+
+config MATH_KUNIT_TEST
+ tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ depends on KUNIT
diff --git a/lib/math/Makefile b/lib/math/Makefile
index be6909e943bd..30abb7a8d564 100644
--- a/lib/math/Makefile
+++ b/lib/math/Makefile
@@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
reciprocal_div.o
obj-$(CONFIG_CORDIC) += cordic.o
obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
obj-$(CONFIG_RATIONAL) += rational.o
+
+obj-$(CONFIG_MATH_KUNIT_TEST) += math_kunit.o
diff --git a/lib/math/math_kunit.c b/lib/math/math_kunit.c
new file mode 100644
index ..cb2637a24942
--- /dev/null
+++ b/lib/math/math_kunit.c
@@ -0,0 +1,197 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple KUnit suite for math helper funcs that are always enabled.
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+/* Generic test case for unsigned long inputs. */
+struct test_case {
+ unsigned long a, b;
+ unsigned long result;
+};
+
+static struct test_case gcd_cases[] = {
+ {
+ .a = 0, .b = 1,
+ .result = 1,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 4,
+ .result = 2,
+ },
+ {
+ .a = 3, .b = 5,
+ .result = 1,
+ },
+ {
+ .a = 3*9, .b = 3*5,
+ .result = 3,
+ },
+ {
+ .a = 3*5*7, .b = 3*5*11,
+ .result = 15,
+ },
+ {
+ .a = 1 << 21,
+ .b = (1 << 21) - 1,
+ .result = 1,
+ },
+};
+KUNIT_ARRAY_PARAM(gcd, gcd_cases, NULL);
+
+static void gcd_test(struct kunit *test)
+{
+ const char *message_fmt = "gcd(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->a, test_param->b),
+ message_fmt, test_param->a,
+ test_param->b);
+
+ /* gcd(a,b) == gcd(b,a) */
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->b, test_param->a),
+ message_fmt, test_param->b,
+ test_param->a);
+}
+
+
+static struct test_case lcm_cases[] = {
+ {
+ .a = 0, .b = 1,
+ .result = 0,
+ },
+ {
+ .a = 1, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 3*5, .b = 3*7,
+ .result = 3*5*7,
+ },
+};
+KUNIT_ARRAY_PARAM(lcm, lcm_cases, NULL);
+
+static void lcm_test(struct kunit *test)
+{
+ const char *message_fmt = "lcm(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ lcm(test_param->a, test_param->b),
+ message_fmt, test_param->a,
+ test_param->b);
+
+ /* lcm(a,b) == lcm(b,a) */
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ lcm(test_param->b, test_param->a),
+ message_fmt, test_param->b,
+
Re: [PATCH v2] lib: add basic KUnit test for lib/math
On Mon, Jan 18, 2021 at 2:36 AM Andy Shevchenko wrote: > > On Thu, Jan 14, 2021 at 10:27:00AM -0800, Daniel Latypov wrote: > > Add basic test coverage for files that don't require any config options: > > * gcd.c > > * lcm.c > > * int_sqrt.c > > * reciprocal_div.c > > (Ignored int_pow.c since it's a simple textbook algorithm.) > > > > These tests aren't particularly interesting, but they > > * provide a place to add tests for any new files in this dir > > * are written so adding new test cases to cover edge cases should be easy > > > > lib/math/math_test.c | 197 +++ > > Can it have _kunit instead of _test? Sounds good, will send a v3 shortly. > > > -- > With Best Regards, > Andy Shevchenko > >
[PATCH v4 2/3] kunit: tool: fix minor typing issue with None status
The code to handle aggregating statuses didn't check that the status
actually got set to some non-None value.
Default the value to SUCCESS instead of adding a bunch of `is None`
checks.
This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
fail test suites if one of them is empty").
Also slightly simplify the code and add type annotations.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
tools/testing/kunit/kunit_parser.py | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py
b/tools/testing/kunit/kunit_parser.py
index 8b5eb9507765..2e4cb5206fa7 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -12,13 +12,13 @@ from collections import namedtuple
from datetime import datetime
from enum import Enum, auto
from functools import reduce
-from typing import Iterator, List, Optional, Tuple
+from typing import Iterable, Iterator, List, Optional, Tuple
TestResult = namedtuple('TestResult', ['status','suites','log'])
class TestSuite(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.cases = [] # type: List[TestCase]
@@ -30,7 +30,7 @@ class TestSuite(object):
class TestCase(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.log = [] # type: List[str]
@@ -224,12 +224,11 @@ def parse_ok_not_ok_test_suite(lines: List[str],
else:
return False
-def bubble_up_errors(to_status, status_container_list) -> TestStatus:
- status_list = map(to_status, status_container_list)
- return reduce(max_status, status_list, TestStatus.SUCCESS)
+def bubble_up_errors(statuses: Iterable[TestStatus]) -> TestStatus:
+ return reduce(max_status, statuses, TestStatus.SUCCESS)
def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
- max_test_case_status = bubble_up_errors(lambda x: x.status,
test_suite.cases)
+ max_test_case_status = bubble_up_errors(x.status for x in
test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
def parse_test_suite(lines: List[str], expected_suite_index: int) ->
Optional[TestSuite]:
@@ -282,8 +281,8 @@ def parse_test_plan(lines: List[str]) -> Optional[int]:
else:
return None
-def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
- return bubble_up_errors(lambda x: x.status, test_suite_list)
+def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus:
+ return bubble_up_errors(x.status for x in test_suites)
def parse_test_result(lines: List[str]) -> TestResult:
consume_non_diagnositic(lines)
--
2.30.0.296.g2bfb1c46d8-goog
[PATCH v4 3/3] kunit: tool: move kunitconfig parsing into __init__, make it optional
LinuxSourceTree will unceremoniously crash if the user doesn't call read_kunitconfig() first in a number of functions. And currently every place we create an instance, the caller also calls create_kunitconfig() and read_kunitconfig(). Move these instead into __init__() so they can't be forgotten and to reduce copy-paste. The https://github.com/google/pytype type-checker complained that _config wasn't initialized. With this, kunit_tool now type checks under both pytype and mypy. Add an optional boolean that can be used to disable this for use cases in the future where we might not need/want to load the config. Signed-off-by: Daniel Latypov Reviewed-by: Brendan Higgins --- tools/testing/kunit/kunit.py| 20 tools/testing/kunit/kunit_kernel.py | 25 + 2 files changed, 17 insertions(+), 28 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 5521e0a8201e..e808a47c839b 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -256,10 +256,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitRequest(cli_args.raw_output, cli_args.timeout, @@ -277,10 +274,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitConfigRequest(cli_args.build_dir, cli_args.make_options) @@ -292,10 +286,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'build': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitBuildRequest(cli_args.jobs, cli_args.build_dir, @@ -309,10 +300,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'exec': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) exec_request = KunitExecRequest(cli_args.timeout, cli_args.build_dir, diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index e77ee06aa407..2076a5a2d060 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,28 +123,29 @@ def get_outfile_path(build_dir) -> str: class LinuxSourceTree(object): """Represents a Linux kernel source tree with KUnit tests.""" - def __init__(self) -> None: - self._ops = LinuxSourceTreeOperations() + def __init__(self, build_dir: str, load_config=True, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: signal.signal(signal.SIGINT, self.signal_handler) - def clean(self) -> bool: - try: - self._ops.make_mrproper() - except ConfigError as e: - logging.error(e) - return False - return True + self._ops = LinuxSourceTreeOperations() + + if not load_config: + return - def create_kunitconfig(self, build_dir, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: kunitconfig_path = get_kunitconfig_path(build_dir) if not os.path.exists(kunitconfig_path): shutil.copyfile(defconfig, kunitconfig_path) - def read_kunitconfig(self, build_dir) -> None: - kunitconfig_path = get_kunitconfig_path(build_dir) self._kconfig = kunit_config.Kconfig() self._kconfig.read_from_file(kunitconfig_path) + def clean(self) -> bool: + try: + self._ops.make_mrproper() + except ConfigError as e: + lo
[PATCH v4 1/3] kunit: tool: surface and address more typing issues
The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.
That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.
So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.
Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.
This commit fixes all but one error, where `TestSuite.status` might be
None.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
Changes since v1: none here, reworked last patch.
Changes since v2: rebased onto torvalds/master.
Changes since v3: rebased again torvalds/master.
---
tools/testing/kunit/kunit.py| 14 -
tools/testing/kunit/kunit_config.py | 7 +++--
tools/testing/kunit/kunit_json.py | 2 +-
tools/testing/kunit/kunit_kernel.py | 37 ---
tools/testing/kunit/kunit_parser.py | 46 ++---
5 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 21516e293d17..5521e0a8201e 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -43,9 +43,9 @@ class KunitStatus(Enum):
BUILD_FAILURE = auto()
TEST_FAILURE = auto()
-def get_kernel_root_path():
- parts = sys.argv[0] if not __file__ else __file__
- parts = os.path.realpath(parts).split('tools/testing/kunit')
+def get_kernel_root_path() -> str:
+ path = sys.argv[0] if not __file__ else __file__
+ parts = os.path.realpath(path).split('tools/testing/kunit')
if len(parts) != 2:
sys.exit(1)
return parts[0]
@@ -171,7 +171,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
exec_result.elapsed_time))
return parse_result
-def add_common_opts(parser):
+def add_common_opts(parser) -> None:
parser.add_argument('--build_dir',
help='As in the make command, it specifies the
build '
'directory.',
@@ -183,13 +183,13 @@ def add_common_opts(parser):
help='Run all KUnit tests through allyesconfig',
action='store_true')
-def add_build_opts(parser):
+def add_build_opts(parser) -> None:
parser.add_argument('--jobs',
help='As in the make command, "Specifies the
number of '
'jobs (commands) to run simultaneously."',
type=int, default=8, metavar='jobs')
-def add_exec_opts(parser):
+def add_exec_opts(parser) -> None:
parser.add_argument('--timeout',
help='maximum number of seconds to allow for all
tests '
'to run. This does not include time taken to build
the '
@@ -198,7 +198,7 @@ def add_exec_opts(parser):
default=300,
metavar='timeout')
-def add_parse_opts(parser):
+def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
action='store_true')
parser.add_argument('--json',
diff --git a/tools/testing/kunit/kunit_config.py
b/tools/testing/kunit/kunit_config.py
index 02ffc3a3e5dc..bdd60230764b 100644
--- a/tools/testing/kunit/kunit_config.py
+++ b/tools/testing/kunit/kunit_config.py
@@ -8,6 +8,7 @@
import collections
import re
+from typing import List, Set
CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_(\w+) is not set$'
CONFIG_PATTERN = r'^CONFIG_(\w+)=(\S+|".*")$'
@@ -30,10 +31,10 @@ class KconfigParseError(Exception):
class Kconfig(object):
"""Represents defconfig or .config specified using the Kconfig
language."""
- def __init__(self):
- self._entries = []
+ def __init__(self) -> None:
+ self._entries = [] # type: List[KconfigEntry]
- def entries(self):
+ def entries(self) -> Set[KconfigEntry]:
return set(self._entries)
def add_entry(self, entry: KconfigEntry) -> None:
diff --git a/tools/testing/kunit/kunit_json.py
b/tools/testing/kunit/kunit_json.py
index 624b31b2dbd6..f5cca5c38cac 100644
--- a/tools/testing/kunit/kunit_json.py
+++ b/tools/testing/kunit/kunit_json.py
@@ -13,7 +13,7 @@ import kunit_parser
from kunit_parser import TestStatus
-def get_json_result(test_result, def_config, build_dir, json_path):
+def get_json_result(test_result, def_config, build_dir, json_path) -> str:
sub_groups = []
# Each test suite is mapped to a KernelCI sub_group
diff --
[PATCH v2] lib: add basic KUnit test for lib/math
Add basic test coverage for files that don't require any config options:
* gcd.c
* lcm.c
* int_sqrt.c
* reciprocal_div.c
(Ignored int_pow.c since it's a simple textbook algorithm.)
These tests aren't particularly interesting, but they
* provide a place to add tests for any new files in this dir
* are written so adding new test cases to cover edge cases should be easy
Signed-off-by: Daniel Latypov
---
Changes since v1:
* Rebase and rewrite to use the new parameterized testing support.
* misc: fix overflow in literal and inline int_sqrt format string.
* related: commit 1f0e943df68a ("Documentation: kunit: provide guidance
for testing many inputs") was merged explaining the patterns shown here.
* there's an in-flight patch to update it for parameterized testing.
v1: https://lore.kernel.org/lkml/20201019224556.3536790-1-dlaty...@google.com/
---
lib/math/Kconfig | 5 ++
lib/math/Makefile| 2 +
lib/math/math_test.c | 197 +++
3 files changed, 204 insertions(+)
create mode 100644 lib/math/math_test.c
diff --git a/lib/math/Kconfig b/lib/math/Kconfig
index f19bc9734fa7..6ba8680439c1 100644
--- a/lib/math/Kconfig
+++ b/lib/math/Kconfig
@@ -15,3 +15,8 @@ config PRIME_NUMBERS
config RATIONAL
bool
+
+config MATH_KUNIT_TEST
+ tristate "KUnit test for lib/math" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ depends on KUNIT
diff --git a/lib/math/Makefile b/lib/math/Makefile
index be6909e943bd..fba6fe90f50b 100644
--- a/lib/math/Makefile
+++ b/lib/math/Makefile
@@ -4,3 +4,5 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o
reciprocal_div.o
obj-$(CONFIG_CORDIC) += cordic.o
obj-$(CONFIG_PRIME_NUMBERS)+= prime_numbers.o
obj-$(CONFIG_RATIONAL) += rational.o
+
+obj-$(CONFIG_MATH_KUNIT_TEST) += math_test.o
diff --git a/lib/math/math_test.c b/lib/math/math_test.c
new file mode 100644
index ..cb2637a24942
--- /dev/null
+++ b/lib/math/math_test.c
@@ -0,0 +1,197 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple KUnit suite for math helper funcs that are always enabled.
+ *
+ * Copyright (C) 2020, Google LLC.
+ * Author: Daniel Latypov
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+/* Generic test case for unsigned long inputs. */
+struct test_case {
+ unsigned long a, b;
+ unsigned long result;
+};
+
+static struct test_case gcd_cases[] = {
+ {
+ .a = 0, .b = 1,
+ .result = 1,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 4,
+ .result = 2,
+ },
+ {
+ .a = 3, .b = 5,
+ .result = 1,
+ },
+ {
+ .a = 3*9, .b = 3*5,
+ .result = 3,
+ },
+ {
+ .a = 3*5*7, .b = 3*5*11,
+ .result = 15,
+ },
+ {
+ .a = 1 << 21,
+ .b = (1 << 21) - 1,
+ .result = 1,
+ },
+};
+KUNIT_ARRAY_PARAM(gcd, gcd_cases, NULL);
+
+static void gcd_test(struct kunit *test)
+{
+ const char *message_fmt = "gcd(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->a, test_param->b),
+ message_fmt, test_param->a,
+ test_param->b);
+
+ /* gcd(a,b) == gcd(b,a) */
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ gcd(test_param->b, test_param->a),
+ message_fmt, test_param->b,
+ test_param->a);
+}
+
+
+static struct test_case lcm_cases[] = {
+ {
+ .a = 0, .b = 1,
+ .result = 0,
+ },
+ {
+ .a = 1, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 2, .b = 2,
+ .result = 2,
+ },
+ {
+ .a = 3*5, .b = 3*7,
+ .result = 3*5*7,
+ },
+};
+KUNIT_ARRAY_PARAM(lcm, lcm_cases, NULL);
+
+static void lcm_test(struct kunit *test)
+{
+ const char *message_fmt = "lcm(%lu, %lu)";
+ const struct test_case *test_param = test->param_value;
+
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ lcm(test_param->a, test_param->b),
+ message_fmt, test_param->a,
+ test_param->b);
+
+ /* lcm(a,b) == lcm(b,a) */
+ KUNIT_EXPECT_EQ_MSG(test, test_param->result,
+ lcm(test_param->b, test_param->a),
+ message_fmt, test_param->b,
+ test_param->a);
+}
+
+static struct test_case int_sqrt_cases[] = {
+ {
+
[PATCH v3 2/3] kunit: tool: fix minor typing issue with None status
The code to handle aggregating statuses didn't check that the status
actually got set to some non-None value.
Default the value to SUCCESS instead of adding a bunch of `is None`
checks.
This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
fail test suites if one of them is empty").
Also slightly simplify the code and add type annotations.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
tools/testing/kunit/kunit_parser.py | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py
b/tools/testing/kunit/kunit_parser.py
index 8b5eb9507765..2e4cb5206fa7 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -12,13 +12,13 @@ from collections import namedtuple
from datetime import datetime
from enum import Enum, auto
from functools import reduce
-from typing import Iterator, List, Optional, Tuple
+from typing import Iterable, Iterator, List, Optional, Tuple
TestResult = namedtuple('TestResult', ['status','suites','log'])
class TestSuite(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.cases = [] # type: List[TestCase]
@@ -30,7 +30,7 @@ class TestSuite(object):
class TestCase(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.log = [] # type: List[str]
@@ -224,12 +224,11 @@ def parse_ok_not_ok_test_suite(lines: List[str],
else:
return False
-def bubble_up_errors(to_status, status_container_list) -> TestStatus:
- status_list = map(to_status, status_container_list)
- return reduce(max_status, status_list, TestStatus.SUCCESS)
+def bubble_up_errors(statuses: Iterable[TestStatus]) -> TestStatus:
+ return reduce(max_status, statuses, TestStatus.SUCCESS)
def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
- max_test_case_status = bubble_up_errors(lambda x: x.status,
test_suite.cases)
+ max_test_case_status = bubble_up_errors(x.status for x in
test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
def parse_test_suite(lines: List[str], expected_suite_index: int) ->
Optional[TestSuite]:
@@ -282,8 +281,8 @@ def parse_test_plan(lines: List[str]) -> Optional[int]:
else:
return None
-def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
- return bubble_up_errors(lambda x: x.status, test_suite_list)
+def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus:
+ return bubble_up_errors(x.status for x in test_suites)
def parse_test_result(lines: List[str]) -> TestResult:
consume_non_diagnositic(lines)
--
2.29.2.729.g45daf8777d-goog
[PATCH v3 3/3] kunit: tool: move kunitconfig parsing into __init__, make it optional
LinuxSourceTree will unceremoniously crash if the user doesn't call read_kunitconfig() first in a number of functions. And currently every place we create an instance, the caller also calls create_kunitconfig() and read_kunitconfig(). Move these instead into __init__() so they can't be forgotten and to reduce copy-paste. The https://github.com/google/pytype type-checker complained that _config wasn't initialized. With this, kunit_tool now type checks under both pytype and mypy. Add an optional boolean that can be used to disable this for use cases in the future where we might not need/want to load the config. Signed-off-by: Daniel Latypov --- tools/testing/kunit/kunit.py| 20 tools/testing/kunit/kunit_kernel.py | 25 + 2 files changed, 17 insertions(+), 28 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 5521e0a8201e..e808a47c839b 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -256,10 +256,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitRequest(cli_args.raw_output, cli_args.timeout, @@ -277,10 +274,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitConfigRequest(cli_args.build_dir, cli_args.make_options) @@ -292,10 +286,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'build': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitBuildRequest(cli_args.jobs, cli_args.build_dir, @@ -309,10 +300,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'exec': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) exec_request = KunitExecRequest(cli_args.timeout, cli_args.build_dir, diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index db7ed84ea410..bf7a784ac6eb 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -123,28 +123,29 @@ def get_outfile_path(build_dir) -> str: class LinuxSourceTree(object): """Represents a Linux kernel source tree with KUnit tests.""" - def __init__(self) -> None: - self._ops = LinuxSourceTreeOperations() + def __init__(self, build_dir: str, load_config=True, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: signal.signal(signal.SIGINT, self.signal_handler) - def clean(self) -> bool: - try: - self._ops.make_mrproper() - except ConfigError as e: - logging.error(e) - return False - return True + self._ops = LinuxSourceTreeOperations() + + if not load_config: + return - def create_kunitconfig(self, build_dir, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: kunitconfig_path = get_kunitconfig_path(build_dir) if not os.path.exists(kunitconfig_path): shutil.copyfile(defconfig, kunitconfig_path) - def read_kunitconfig(self, build_dir) -> None: - kunitconfig_path = get_kunitconfig_path(build_dir) self._kconfig = kunit_config.Kconfig() self._kconfig.read_from_file(kunitconfig_path) + def clean(self) -> bool: + try: + self._ops.make_mrproper() + except ConfigError as e: + logging.error(e) +
[PATCH v3 1/3] kunit: tool: surface and address more typing issues
The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.
That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.
So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.
Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.
This commit fixes all but one error, where `TestSuite.status` might be
None.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
tools/testing/kunit/kunit.py| 14 -
tools/testing/kunit/kunit_config.py | 7 +++--
tools/testing/kunit/kunit_json.py | 2 +-
tools/testing/kunit/kunit_kernel.py | 37 ---
tools/testing/kunit/kunit_parser.py | 46 ++---
5 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 21516e293d17..5521e0a8201e 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -43,9 +43,9 @@ class KunitStatus(Enum):
BUILD_FAILURE = auto()
TEST_FAILURE = auto()
-def get_kernel_root_path():
- parts = sys.argv[0] if not __file__ else __file__
- parts = os.path.realpath(parts).split('tools/testing/kunit')
+def get_kernel_root_path() -> str:
+ path = sys.argv[0] if not __file__ else __file__
+ parts = os.path.realpath(path).split('tools/testing/kunit')
if len(parts) != 2:
sys.exit(1)
return parts[0]
@@ -171,7 +171,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
exec_result.elapsed_time))
return parse_result
-def add_common_opts(parser):
+def add_common_opts(parser) -> None:
parser.add_argument('--build_dir',
help='As in the make command, it specifies the
build '
'directory.',
@@ -183,13 +183,13 @@ def add_common_opts(parser):
help='Run all KUnit tests through allyesconfig',
action='store_true')
-def add_build_opts(parser):
+def add_build_opts(parser) -> None:
parser.add_argument('--jobs',
help='As in the make command, "Specifies the
number of '
'jobs (commands) to run simultaneously."',
type=int, default=8, metavar='jobs')
-def add_exec_opts(parser):
+def add_exec_opts(parser) -> None:
parser.add_argument('--timeout',
help='maximum number of seconds to allow for all
tests '
'to run. This does not include time taken to build
the '
@@ -198,7 +198,7 @@ def add_exec_opts(parser):
default=300,
metavar='timeout')
-def add_parse_opts(parser):
+def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
action='store_true')
parser.add_argument('--json',
diff --git a/tools/testing/kunit/kunit_config.py
b/tools/testing/kunit/kunit_config.py
index 02ffc3a3e5dc..bdd60230764b 100644
--- a/tools/testing/kunit/kunit_config.py
+++ b/tools/testing/kunit/kunit_config.py
@@ -8,6 +8,7 @@
import collections
import re
+from typing import List, Set
CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_(\w+) is not set$'
CONFIG_PATTERN = r'^CONFIG_(\w+)=(\S+|".*")$'
@@ -30,10 +31,10 @@ class KconfigParseError(Exception):
class Kconfig(object):
"""Represents defconfig or .config specified using the Kconfig
language."""
- def __init__(self):
- self._entries = []
+ def __init__(self) -> None:
+ self._entries = [] # type: List[KconfigEntry]
- def entries(self):
+ def entries(self) -> Set[KconfigEntry]:
return set(self._entries)
def add_entry(self, entry: KconfigEntry) -> None:
diff --git a/tools/testing/kunit/kunit_json.py
b/tools/testing/kunit/kunit_json.py
index 624b31b2dbd6..f5cca5c38cac 100644
--- a/tools/testing/kunit/kunit_json.py
+++ b/tools/testing/kunit/kunit_json.py
@@ -13,7 +13,7 @@ import kunit_parser
from kunit_parser import TestStatus
-def get_json_result(test_result, def_config, build_dir, json_path):
+def get_json_result(test_result, def_config, build_dir, json_path) -> str:
sub_groups = []
# Each test suite is mapped to a KernelCI sub_group
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 57c1724b7e5d..db7ed84ea410 100644
--- a/tools/testing/kunit/ku
[PATCH] Documentation: kunit: include example of a parameterized test
Commit fadb08e7c750 ("kunit: Support for Parameterized Testing")
introduced support but lacks documentation for how to use it.
This patch builds on commit 1f0e943df68a ("Documentation: kunit: provide
guidance for testing many inputs") to show a minimal example of the new
feature.
Signed-off-by: Daniel Latypov
---
Documentation/dev-tools/kunit/usage.rst | 57 +
1 file changed, 57 insertions(+)
diff --git a/Documentation/dev-tools/kunit/usage.rst
b/Documentation/dev-tools/kunit/usage.rst
index d9fdc14f0677..650f99590df5 100644
--- a/Documentation/dev-tools/kunit/usage.rst
+++ b/Documentation/dev-tools/kunit/usage.rst
@@ -522,6 +522,63 @@ There's more boilerplate involved, but it can:
* E.g. if we wanted to also test ``sha256sum``, we could add a ``sha256``
field and reuse ``cases``.
+* be converted to a "parameterized test", see below.
+
+Parameterized Testing
+~
+
+The table-driven testing pattern is common enough that KUnit has special
+support for it.
+
+Reusing the same ``cases`` array from above, we can write the test as a
+"parameterized test" with the following.
+
+.. code-block:: c
+
+ // This is copy-pasted from above.
+ struct sha1_test_case {
+ const char *str;
+ const char *sha1;
+ };
+ struct sha1_test_case cases[] = {
+ {
+ .str = "hello world",
+ .sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
+ },
+ {
+ .str = "hello world!",
+ .sha1 = "430ce34d020724ed75a196dfc2ad67c2d169",
+ },
+ };
+
+ // Need a helper function to generate a name for each test case.
+ static void case_to_desc(const struct sha1_test_case *t, char *desc)
+ {
+ strcpy(desc, t->str);
+ }
+ // Creates `sha1_gen_params()` to iterate over `cases`.
+ KUNIT_ARRAY_PARAM(sha1, cases, case_to_desc);
+
+ // Looks no different from a normal test.
+ static void sha1_test(struct kunit *test)
+ {
+ // This function can just contain the body of the for-loop.
+ // The former `cases[i]` is accessible under test->param_value.
+ char out[40];
+ struct sha1_test_case *test_param = (struct sha1_test_case
*)(test->param_value);
+
+ sha1sum(test_param->str, out);
+ KUNIT_EXPECT_STREQ_MSG(test, (char *)out, test_param->sha1,
+ "sha1sum(%s)", test_param->str);
+ }
+
+ // Instead of KUNIT_CASE, we use KUNIT_CASE_PARAM and pass in the
+ // function declared by KUNIT_ARRAY_PARAM.
+ static struct kunit_case sha1_test_cases[] = {
+ KUNIT_CASE_PARAM(sha1_test, sha1_gen_params),
+ {}
+ };
+
.. _kunit-on-non-uml:
KUnit on non-UML architectures
base-commit: 5f6b99d0287de2c2d0b5e7abcb0092d553ad804a
--
2.29.2.684.gfbc64c5ab5-goog
Re: [PATCH] kunit: Print test statistics on failure
On Thu, Dec 10, 2020 at 11:23 PM David Gow wrote:
>
> When a number of tests fail, it can be useful to get higher-level
> statistics of how many tests are failing (or how many parameters are
> failing in parameterised tests), and in what cases or suites. This is
> already done by some non-KUnit tests, so add support for automatically
> generating these for KUnit tests.
>
> This change adds a 'kunit_stats_enabled' switch which has three values:
> - 0: No stats are printed (current behaviour)
> - 1: Stats are printed only for tests/suites with more than one
> subtests, and at least one failure (new default)
> - 2: Always print test statistics
I personally prefer having less options here, if possible.
a) I wonder if 0 can be dropped in favor of just using the default (1).
I don't know that showing test counts on failures (ideally a rare
occurrence) in the diagnostics would be enough of an annoyance that
people would want to turn it off.
b) And/or perhaps strike a compromise between 1/2.
We print the test stats whenever there are >1 subtests, regardless of
# of failures.
If we take both suggestions, then we just have one way, which is
appealing to me personally, but I don't know that people would like
that.
>
> For parameterised tests, the summary line looks as follows:
> "# inode_test_xtimestamp_decoding: 0 / 16 test parameters failed"
> For test suites, it looks like this:
> "# ext4_inode_test: (0 / 1) tests failed (0 / 16 test parameters)"
>
> kunit_tool is also updated to correctly ignore diagnostic lines, so that
> these statistics do not prevent the result from parsing.
>
> Signed-off-by: David Gow
> ---
>
> This is largely a follow-up to the discussion here:
>
> https://lore.kernel.org/linux-kselftest/CABVgOSmy4n_LGwDS7yWfoLftcQzxv6S+iXx9Y=opcgg2gu0...@mail.gmail.com/T/#t
>
> Does this seem like a sensible addition?
>
> Cheers,
> -- David
>
> lib/kunit/test.c| 71 +
> tools/testing/kunit/kunit_parser.py | 2 +-
> 2 files changed, 72 insertions(+), 1 deletion(-)
>
> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> index ec9494e914ef..711e269366a7 100644
> --- a/lib/kunit/test.c
> +++ b/lib/kunit/test.c
> @@ -9,6 +9,7 @@
> #include
> #include
> #include
> +#include
> #include
> #include
>
> @@ -16,6 +17,40 @@
> #include "string-stream.h"
> #include "try-catch-impl.h"
>
> +/*
> + * KUnit statistic mode:
> + * 0 - disabled
> + * 1 - only when there is at least one failure, and more than one subtest
> + * 2 - enabled
> + */
> +static int kunit_stats_enabled = 1;
> +core_param(kunit_stats_enabled, kunit_stats_enabled, int, 0644);
> +
> +static bool kunit_should_print_stats(int num_failures, int num_subtests)
> +{
> + if (kunit_stats_enabled == 0)
> + return false;
> +
> + if (kunit_stats_enabled == 2)
> + return true;
> +
> + return (num_failures > 0 && num_subtests > 1);
> +}
> +
> +static void kunit_print_test_stats(struct kunit *test,
> + size_t num_failures, size_t num_subtests)
> +{
> + if (!kunit_should_print_stats(num_failures, num_subtests))
> + return;
> +
> + kunit_log(KERN_INFO, test,
> + KUNIT_SUBTEST_INDENT
> + "# %s: %lu / %lu test parameters failed",
> + test->name,
> + num_failures,
> + num_subtests);
> +}
> +
> /*
> * Append formatted message to log, size of which is limited to
> * KUNIT_LOG_SIZE bytes (including null terminating byte).
> @@ -346,15 +381,37 @@ static void kunit_run_case_catch_errors(struct
> kunit_suite *suite,
> test_case->success = test->success;
> }
>
> +static void kunit_print_suite_stats(struct kunit_suite *suite,
> + size_t num_failures,
> + size_t total_param_failures,
> + size_t total_params)
> +{
> + size_t num_cases = kunit_suite_num_test_cases(suite);
> +
> + if (!kunit_should_print_stats(num_failures, num_cases))
> + return;
> +
> + kunit_log(KERN_INFO, suite,
> + "# %s: (%lu / %lu) tests failed (%lu / %lu test
> parameters)",
> + suite->name,
> + num_failures,
> + num_cases,
> + total_param_failures,
> + total_params);
> +}
> +
> int kunit_run_tests(struct kunit_suite *suite)
> {
> char param_desc[KUNIT_PARAM_DESC_SIZE];
> struct kunit_case *test_case;
> + size_t num_suite_failures = 0;
> + size_t total_param_failures = 0, total_params = 0;
>
> kunit_print_subtest_start(suite);
>
> kunit_suite_for_each_test_case(suite, test_case) {
> struct kunit test = { .param_value = NULL, .param_index = 0 };
> + size_t num_params = 0, num_failures = 0;
> bool test_success = true;
[PATCH v2 3/3] kunit: tool: move kunitconfig parsing into __init__, make it optional
LinuxSourceTree will unceremoniously crash if the user doesn't call read_kunitconfig() first in a number of functions. And currently every place we create an instance, the caller also calls create_kunitconfig() and read_kunitconfig(). Move these instead into __init__() so they can't be forgotten and to reduce copy-paste. The https://github.com/google/pytype type-checker complained that _config wasn't initialized. With this, kunit_tool now type checks under both pytype and mypy. Add an optional boolean that can be used to disable this for use cases in the future where we might not need/want to load the config. Signed-off-by: Daniel Latypov --- tools/testing/kunit/kunit.py| 20 tools/testing/kunit/kunit_kernel.py | 25 + 2 files changed, 17 insertions(+), 28 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 08951a114654..b58fb3733cfa 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -256,10 +256,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitRequest(cli_args.raw_output, cli_args.timeout, @@ -277,10 +274,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitConfigRequest(cli_args.build_dir, cli_args.make_options) @@ -292,10 +286,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'build': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitBuildRequest(cli_args.jobs, cli_args.build_dir, @@ -309,10 +300,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'exec': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) exec_request = KunitExecRequest(cli_args.timeout, cli_args.build_dir, diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index bda7c4fd4d3e..b2d71b1e85e4 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -129,28 +129,29 @@ def get_outfile_path(build_dir) -> str: class LinuxSourceTree(object): """Represents a Linux kernel source tree with KUnit tests.""" - def __init__(self) -> None: - self._ops = LinuxSourceTreeOperations() + def __init__(self, build_dir: str, load_config=True, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: signal.signal(signal.SIGINT, self.signal_handler) - def clean(self) -> bool: - try: - self._ops.make_mrproper() - except ConfigError as e: - logging.error(e) - return False - return True + self._ops = LinuxSourceTreeOperations() + + if not load_config: + return - def create_kunitconfig(self, build_dir, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: kunitconfig_path = get_kunitconfig_path(build_dir) if not os.path.exists(kunitconfig_path): shutil.copyfile(defconfig, kunitconfig_path) - def read_kunitconfig(self, build_dir) -> None: - kunitconfig_path = get_kunitconfig_path(build_dir) self._kconfig = kunit_config.Kconfig() self._kconfig.read_from_file(kunitconfig_path) + def clean(self) -> bool: + try: + self._ops.make_mrproper() + except ConfigError as e: + logging.error(e) +
[PATCH v2 1/3] kunit: tool: surface and address more typing issues
The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.
That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.
So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.
Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.
This commit fixes all but one error, where `TestSuite.status` might be
None.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
tools/testing/kunit/kunit.py| 14 -
tools/testing/kunit/kunit_config.py | 7 +++--
tools/testing/kunit/kunit_json.py | 2 +-
tools/testing/kunit/kunit_kernel.py | 37 ---
tools/testing/kunit/kunit_parser.py | 46 ++---
5 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index d4f7846d0745..08951a114654 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -43,9 +43,9 @@ class KunitStatus(Enum):
BUILD_FAILURE = auto()
TEST_FAILURE = auto()
-def get_kernel_root_path():
- parts = sys.argv[0] if not __file__ else __file__
- parts = os.path.realpath(parts).split('tools/testing/kunit')
+def get_kernel_root_path() -> str:
+ path = sys.argv[0] if not __file__ else __file__
+ parts = os.path.realpath(path).split('tools/testing/kunit')
if len(parts) != 2:
sys.exit(1)
return parts[0]
@@ -171,7 +171,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
exec_result.elapsed_time))
return parse_result
-def add_common_opts(parser):
+def add_common_opts(parser) -> None:
parser.add_argument('--build_dir',
help='As in the make command, it specifies the
build '
'directory.',
@@ -183,13 +183,13 @@ def add_common_opts(parser):
help='Run all KUnit tests through allyesconfig',
action='store_true')
-def add_build_opts(parser):
+def add_build_opts(parser) -> None:
parser.add_argument('--jobs',
help='As in the make command, "Specifies the
number of '
'jobs (commands) to run simultaneously."',
type=int, default=8, metavar='jobs')
-def add_exec_opts(parser):
+def add_exec_opts(parser) -> None:
parser.add_argument('--timeout',
help='maximum number of seconds to allow for all
tests '
'to run. This does not include time taken to build
the '
@@ -198,7 +198,7 @@ def add_exec_opts(parser):
default=300,
metavar='timeout')
-def add_parse_opts(parser):
+def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
action='store_true')
parser.add_argument('--json',
diff --git a/tools/testing/kunit/kunit_config.py
b/tools/testing/kunit/kunit_config.py
index 02ffc3a3e5dc..bdd60230764b 100644
--- a/tools/testing/kunit/kunit_config.py
+++ b/tools/testing/kunit/kunit_config.py
@@ -8,6 +8,7 @@
import collections
import re
+from typing import List, Set
CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_(\w+) is not set$'
CONFIG_PATTERN = r'^CONFIG_(\w+)=(\S+|".*")$'
@@ -30,10 +31,10 @@ class KconfigParseError(Exception):
class Kconfig(object):
"""Represents defconfig or .config specified using the Kconfig
language."""
- def __init__(self):
- self._entries = []
+ def __init__(self) -> None:
+ self._entries = [] # type: List[KconfigEntry]
- def entries(self):
+ def entries(self) -> Set[KconfigEntry]:
return set(self._entries)
def add_entry(self, entry: KconfigEntry) -> None:
diff --git a/tools/testing/kunit/kunit_json.py
b/tools/testing/kunit/kunit_json.py
index 624b31b2dbd6..f5cca5c38cac 100644
--- a/tools/testing/kunit/kunit_json.py
+++ b/tools/testing/kunit/kunit_json.py
@@ -13,7 +13,7 @@ import kunit_parser
from kunit_parser import TestStatus
-def get_json_result(test_result, def_config, build_dir, json_path):
+def get_json_result(test_result, def_config, build_dir, json_path) -> str:
sub_groups = []
# Each test suite is mapped to a KernelCI sub_group
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2e3cc0fac726..bda7c4fd4d3e 100644
--- a/tools/testing/kunit/ku
[PATCH v2 2/3] kunit: tool: fix minor typing issue with None status
The code to handle aggregating statuses didn't check that the status
actually got set to some non-None value.
Default the value to SUCCESS instead of adding a bunch of `is None`
checks.
This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
fail test suites if one of them is empty").
Also slightly simplify the code and add type annotations.
Signed-off-by: Daniel Latypov
Reviewed-by: David Gow
---
tools/testing/kunit/kunit_parser.py | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py
b/tools/testing/kunit/kunit_parser.py
index 24954bbc9baf..97e070506c31 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -12,13 +12,13 @@ from collections import namedtuple
from datetime import datetime
from enum import Enum, auto
from functools import reduce
-from typing import Iterator, List, Optional, Tuple
+from typing import Iterable, Iterator, List, Optional, Tuple
TestResult = namedtuple('TestResult', ['status','suites','log'])
class TestSuite(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.cases = [] # type: List[TestCase]
@@ -30,7 +30,7 @@ class TestSuite(object):
class TestCase(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.log = [] # type: List[str]
@@ -223,12 +223,11 @@ def parse_ok_not_ok_test_suite(lines: List[str],
else:
return False
-def bubble_up_errors(to_status, status_container_list) -> TestStatus:
- status_list = map(to_status, status_container_list)
- return reduce(max_status, status_list, TestStatus.SUCCESS)
+def bubble_up_errors(statuses: Iterable[TestStatus]) -> TestStatus:
+ return reduce(max_status, statuses, TestStatus.SUCCESS)
def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
- max_test_case_status = bubble_up_errors(lambda x: x.status,
test_suite.cases)
+ max_test_case_status = bubble_up_errors(x.status for x in
test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
def parse_test_suite(lines: List[str], expected_suite_index: int) ->
Optional[TestSuite]:
@@ -281,8 +280,8 @@ def parse_test_plan(lines: List[str]) -> Optional[int]:
else:
return None
-def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
- return bubble_up_errors(lambda x: x.status, test_suite_list)
+def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus:
+ return bubble_up_errors(x.status for x in test_suites)
def parse_test_result(lines: List[str]) -> TestResult:
consume_non_diagnositic(lines)
--
2.29.2.684.gfbc64c5ab5-goog
[PATCH] kunit: tool: simplify kconfig is_subset_of() logic
Don't use an O(nm) algorithm* and make it more readable by using a dict.
*Most obviously, it does a nested for-loop over the entire other config.
A bit more subtle, it calls .entries(), which constructs a set from the
list for _every_ outer iteration.
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit_config.py | 13 ++---
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/tools/testing/kunit/kunit_config.py
b/tools/testing/kunit/kunit_config.py
index 02ffc3a3e5dc..f1101075d458 100644
--- a/tools/testing/kunit/kunit_config.py
+++ b/tools/testing/kunit/kunit_config.py
@@ -40,15 +40,14 @@ class Kconfig(object):
self._entries.append(entry)
def is_subset_of(self, other: 'Kconfig') -> bool:
+ other_dict = {e.name: e.value for e in other.entries()}
for a in self.entries():
- found = False
- for b in other.entries():
- if a.name != b.name:
+ b = other_dict.get(a.name)
+ if b is None:
+ if a.value == 'n':
continue
- if a.value != b.value:
- return False
- found = True
- if a.value != 'n' and found == False:
+ return False
+ elif a.value != b:
return False
return True
base-commit: c6f7e1510b872c281ff603a3108c084b6548d35c
--
2.29.2.576.ga3fc446d84-goog
Re: [PATCH 3/3] kunit: tool: move kunitconfig parsing into __init__
On Thu, Dec 3, 2020 at 7:57 PM David Gow wrote: > > On Fri, Dec 4, 2020 at 3:41 AM Daniel Latypov wrote: > > > > LinuxSourceTree will unceremoniously crash if the user doesn't call > > read_kunitconfig() first in a number of functions. > > This patch seems to partly be reverting the changes here, right: > https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/tools/testing/kunit?h=kunit=fcdb0bc08ced274078f371e1e0fe6421a97fa9f2 > (That patch moved the reading of kunitconfig out of __init__) Yes. > > My overall concern is that, really, there are some operations that > shouldn't need a kunitconfig (even if they do at the moment), so we'd > ideally want at least some of the operations currently under > LinuxSourceTree to be able to be run without first reading a > kunitconfig. Most notably, it'd be nice if kunit.py exec (and hence > LinuxSourceTree::run_kernel()) didn't need a kunitconfig, as the > kernel ought to already be built at this point. > > Now, this is all a little bit hypothetical, as we haven't bothered to > make kunit.py exec work without a kunitconfig thus far, but I'm a > touch hesitant to make it harder to bypass the kunitconfig reading > anyway. Fair point. So one alternative to this to make type-checkers happy is to declare _config instead of sneakily setting it in some random later method. Then in all the places that rely on _config, we'd need to add in checks that it's in fact set to give a better error message (so it's clear to the user that it's an internal tool bug and has nothing to do with them). The copy-paste of create+read_kunitconfig() is annoying, which is why I went with this. How about __init__ takes an optional argument that can disable this parsing? E.g. def __init__(kconfig = None): if kconfig is not None: self._config = kconfig else: // create and read Or if we don't like the idea of requiring users who don't want a kconfig to pass in a dummy, def __init__(load_kconfig=True): if not load_kconfig: self._config = None ... > > > > > Adn currently every place we create an instance, the caller also calls > > create_kunitconfig() and read_kunitconfig(). > > > > Move these instead into __init__() so they can't be forgotten and to > > reduce copy-paste. > > This seems to now be missing the create_kunitconfig() stuff (see below). Ah good catch. Completely unintentional. I'm sure I had the create_kunitconfig() stuff in __init__() at some point but must have inadvertently removed it somehow later on. > > > > The https://github.com/google/pytype type-checker complained that > > _config wasn't initialized. With this, kunit_tool now type checks > > under both pytype and mypy. > > > > Signed-off-by: Daniel Latypov > > --- > > tools/testing/kunit/kunit.py| 20 > > tools/testing/kunit/kunit_kernel.py | 19 +++ > > 2 files changed, 11 insertions(+), 28 deletions(-) > > > > diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py > > index 08951a114654..b58fb3733cfa 100755 > > --- a/tools/testing/kunit/kunit.py > > +++ b/tools/testing/kunit/kunit.py > > @@ -256,10 +256,7 @@ def main(argv, linux=None): > > os.mkdir(cli_args.build_dir) > > > > if not linux: > > - linux = kunit_kernel.LinuxSourceTree() > > - > > - linux.create_kunitconfig(cli_args.build_dir) > > - linux.read_kunitconfig(cli_args.build_dir) > > + linux = > > kunit_kernel.LinuxSourceTree(cli_args.build_dir) > > > > request = KunitRequest(cli_args.raw_output, > >cli_args.timeout, > > @@ -277,10 +274,7 @@ def main(argv, linux=None): > > os.mkdir(cli_args.build_dir) > > > > if not linux: > > - linux = kunit_kernel.LinuxSourceTree() > > - > > - linux.create_kunitconfig(cli_args.build_dir) > > - linux.read_kunitconfig(cli_args.build_dir) > > + linux = > > kunit_kernel.LinuxSourceTree(cli_args.build_dir) > > > > request = KunitConfigRequest(cli_args.build_dir, > > cli_args.make_options) > > @@ -292,10 +286,7 @@ def main(argv, linux=None): > > sys.exit(1) > > elif cli_args.subcommand == 'build': > > if not linux: > > - linux = kunit_kernel.LinuxSourceTree() > > - > > - linux.create_kun
Re: [PATCH 2/3] kunit: tool: fix minor typing issue with None status
On Thu, Dec 3, 2020 at 8:17 PM David Gow wrote:
>
> On Fri, Dec 4, 2020 at 3:41 AM Daniel Latypov wrote:
> >
>
> This seems good to me, but I have a few questions, particularly around
> the description.
>
> > The code to handle aggregating statuses didn't check that the status
> > actually got set.
>
> I don't understand what you mean here. Does this refer to
> Test{Case,Suite}::status potentially being None, and that not being
> supported properly in bubble_up_{suite_,test_case_,}errors(), or
> something else? Either way, I can't see any additional code to "check"
> that the status has been set. As far as I can tell everything except
> the default to SUCCESS is a no-op, or am I missing something?
mypy (rightly) sees the type is TestStatus or None and complains we
don't bother handling None, so we risk a crash in the tool.
The status will be none until we explicitly assign a value to it
later, which we always do currently, afaict.
This change just avoids that by giving it a default value to make mypy
happy, which shouldn't change behaviour at all right now.
There is no other (potential) behavioural change.
>
> > Default the value to SUCCESS.
>
> I'm a little iffy about defaulting this to success, but I think it's
> okay for now: the skip test support will eventually change this to a
> SKIPPED value.
Sounds good!
>
> >
> > This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
> > fail test suites if one of them is empty").
> >
> > Also slightly simplify the code and add type annotations.
> >
> > Signed-off-by: Daniel Latypov
> > ---
>
> Otherwise, the actual code changes all seem sensible, and it worked
> fine when I tested it, so:
>
> Reviewed-by: David Gow
>
> -- David
[PATCH 2/3] kunit: tool: fix minor typing issue with None status
The code to handle aggregating statuses didn't check that the status
actually got set.
Default the value to SUCCESS.
This sorta follows the precedent in commit 3fc48259d525 ("kunit: Don't
fail test suites if one of them is empty").
Also slightly simplify the code and add type annotations.
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit_parser.py | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py
b/tools/testing/kunit/kunit_parser.py
index 24954bbc9baf..97e070506c31 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -12,13 +12,13 @@ from collections import namedtuple
from datetime import datetime
from enum import Enum, auto
from functools import reduce
-from typing import Iterator, List, Optional, Tuple
+from typing import Iterable, Iterator, List, Optional, Tuple
TestResult = namedtuple('TestResult', ['status','suites','log'])
class TestSuite(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.cases = [] # type: List[TestCase]
@@ -30,7 +30,7 @@ class TestSuite(object):
class TestCase(object):
def __init__(self) -> None:
- self.status = None # type: Optional[TestStatus]
+ self.status = TestStatus.SUCCESS
self.name = ''
self.log = [] # type: List[str]
@@ -223,12 +223,11 @@ def parse_ok_not_ok_test_suite(lines: List[str],
else:
return False
-def bubble_up_errors(to_status, status_container_list) -> TestStatus:
- status_list = map(to_status, status_container_list)
- return reduce(max_status, status_list, TestStatus.SUCCESS)
+def bubble_up_errors(statuses: Iterable[TestStatus]) -> TestStatus:
+ return reduce(max_status, statuses, TestStatus.SUCCESS)
def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
- max_test_case_status = bubble_up_errors(lambda x: x.status,
test_suite.cases)
+ max_test_case_status = bubble_up_errors(x.status for x in
test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
def parse_test_suite(lines: List[str], expected_suite_index: int) ->
Optional[TestSuite]:
@@ -281,8 +280,8 @@ def parse_test_plan(lines: List[str]) -> Optional[int]:
else:
return None
-def bubble_up_suite_errors(test_suite_list: List[TestSuite]) -> TestStatus:
- return bubble_up_errors(lambda x: x.status, test_suite_list)
+def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus:
+ return bubble_up_errors(x.status for x in test_suites)
def parse_test_result(lines: List[str]) -> TestResult:
consume_non_diagnositic(lines)
--
2.29.2.576.ga3fc446d84-goog
[PATCH 1/3] kunit: tool: surface and address more typing issues
The authors of this tool were more familiar with a different
type-checker, https://github.com/google/pytype.
That's open source, but mypy seems more prevalent (and runs faster).
And unlike pytype, mypy doesn't try to infer types so it doesn't check
unanotated functions.
So annotate ~all functions in kunit tool to increase type-checking
coverage.
Note: per https://www.python.org/dev/peps/pep-0484/, `__init__()` should
be annotated as `-> None`.
Doing so makes mypy discover a number of new violations.
Exclude main() since we reuse `request` for the different types of
requests, which mypy isn't happy about.
This commit fixes all but one error, where `TestSuite.status` might be
None.
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit.py| 14 -
tools/testing/kunit/kunit_config.py | 7 +++--
tools/testing/kunit/kunit_json.py | 2 +-
tools/testing/kunit/kunit_kernel.py | 37 ---
tools/testing/kunit/kunit_parser.py | 46 ++---
5 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index d4f7846d0745..08951a114654 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -43,9 +43,9 @@ class KunitStatus(Enum):
BUILD_FAILURE = auto()
TEST_FAILURE = auto()
-def get_kernel_root_path():
- parts = sys.argv[0] if not __file__ else __file__
- parts = os.path.realpath(parts).split('tools/testing/kunit')
+def get_kernel_root_path() -> str:
+ path = sys.argv[0] if not __file__ else __file__
+ parts = os.path.realpath(path).split('tools/testing/kunit')
if len(parts) != 2:
sys.exit(1)
return parts[0]
@@ -171,7 +171,7 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
exec_result.elapsed_time))
return parse_result
-def add_common_opts(parser):
+def add_common_opts(parser) -> None:
parser.add_argument('--build_dir',
help='As in the make command, it specifies the
build '
'directory.',
@@ -183,13 +183,13 @@ def add_common_opts(parser):
help='Run all KUnit tests through allyesconfig',
action='store_true')
-def add_build_opts(parser):
+def add_build_opts(parser) -> None:
parser.add_argument('--jobs',
help='As in the make command, "Specifies the
number of '
'jobs (commands) to run simultaneously."',
type=int, default=8, metavar='jobs')
-def add_exec_opts(parser):
+def add_exec_opts(parser) -> None:
parser.add_argument('--timeout',
help='maximum number of seconds to allow for all
tests '
'to run. This does not include time taken to build
the '
@@ -198,7 +198,7 @@ def add_exec_opts(parser):
default=300,
metavar='timeout')
-def add_parse_opts(parser):
+def add_parse_opts(parser) -> None:
parser.add_argument('--raw_output', help='don\'t format output from
kernel',
action='store_true')
parser.add_argument('--json',
diff --git a/tools/testing/kunit/kunit_config.py
b/tools/testing/kunit/kunit_config.py
index 02ffc3a3e5dc..bdd60230764b 100644
--- a/tools/testing/kunit/kunit_config.py
+++ b/tools/testing/kunit/kunit_config.py
@@ -8,6 +8,7 @@
import collections
import re
+from typing import List, Set
CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_(\w+) is not set$'
CONFIG_PATTERN = r'^CONFIG_(\w+)=(\S+|".*")$'
@@ -30,10 +31,10 @@ class KconfigParseError(Exception):
class Kconfig(object):
"""Represents defconfig or .config specified using the Kconfig
language."""
- def __init__(self):
- self._entries = []
+ def __init__(self) -> None:
+ self._entries = [] # type: List[KconfigEntry]
- def entries(self):
+ def entries(self) -> Set[KconfigEntry]:
return set(self._entries)
def add_entry(self, entry: KconfigEntry) -> None:
diff --git a/tools/testing/kunit/kunit_json.py
b/tools/testing/kunit/kunit_json.py
index 624b31b2dbd6..f5cca5c38cac 100644
--- a/tools/testing/kunit/kunit_json.py
+++ b/tools/testing/kunit/kunit_json.py
@@ -13,7 +13,7 @@ import kunit_parser
from kunit_parser import TestStatus
-def get_json_result(test_result, def_config, build_dir, json_path):
+def get_json_result(test_result, def_config, build_dir, json_path) -> str:
sub_groups = []
# Each test suite is mapped to a KernelCI sub_group
diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 2e3cc0fac726..bda7c4fd4d3e 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/test
[PATCH 3/3] kunit: tool: move kunitconfig parsing into __init__
LinuxSourceTree will unceremoniously crash if the user doesn't call read_kunitconfig() first in a number of functions. Adn currently every place we create an instance, the caller also calls create_kunitconfig() and read_kunitconfig(). Move these instead into __init__() so they can't be forgotten and to reduce copy-paste. The https://github.com/google/pytype type-checker complained that _config wasn't initialized. With this, kunit_tool now type checks under both pytype and mypy. Signed-off-by: Daniel Latypov --- tools/testing/kunit/kunit.py| 20 tools/testing/kunit/kunit_kernel.py | 19 +++ 2 files changed, 11 insertions(+), 28 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 08951a114654..b58fb3733cfa 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -256,10 +256,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitRequest(cli_args.raw_output, cli_args.timeout, @@ -277,10 +274,7 @@ def main(argv, linux=None): os.mkdir(cli_args.build_dir) if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitConfigRequest(cli_args.build_dir, cli_args.make_options) @@ -292,10 +286,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'build': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) request = KunitBuildRequest(cli_args.jobs, cli_args.build_dir, @@ -309,10 +300,7 @@ def main(argv, linux=None): sys.exit(1) elif cli_args.subcommand == 'exec': if not linux: - linux = kunit_kernel.LinuxSourceTree() - - linux.create_kunitconfig(cli_args.build_dir) - linux.read_kunitconfig(cli_args.build_dir) + linux = kunit_kernel.LinuxSourceTree(cli_args.build_dir) exec_request = KunitExecRequest(cli_args.timeout, cli_args.build_dir, diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index bda7c4fd4d3e..79793031d2c4 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -129,10 +129,15 @@ def get_outfile_path(build_dir) -> str: class LinuxSourceTree(object): """Represents a Linux kernel source tree with KUnit tests.""" - def __init__(self) -> None: - self._ops = LinuxSourceTreeOperations() + def __init__(self, build_dir: str, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: signal.signal(signal.SIGINT, self.signal_handler) + self._ops = LinuxSourceTreeOperations() + + kunitconfig_path = get_kunitconfig_path(build_dir) + self._kconfig = kunit_config.Kconfig() + self._kconfig.read_from_file(kunitconfig_path) + def clean(self) -> bool: try: self._ops.make_mrproper() @@ -141,16 +146,6 @@ class LinuxSourceTree(object): return False return True - def create_kunitconfig(self, build_dir, defconfig=DEFAULT_KUNITCONFIG_PATH) -> None: - kunitconfig_path = get_kunitconfig_path(build_dir) - if not os.path.exists(kunitconfig_path): - shutil.copyfile(defconfig, kunitconfig_path) - - def read_kunitconfig(self, build_dir) -> None: - kunitconfig_path = get_kunitconfig_path(build_dir) - self._kconfig = kunit_config.Kconfig() - self._kconfig.read_from_file(kunitconfig_path) - def validate_config(self, build_dir) -> bool: kconfig_path = get_kconfig_path(build_dir) validated_kconfig = kunit_config.Kconfig() -- 2.29.2.576.ga3fc446d84-goog
Re: [PATCH v2 1/4] kunit: tool: fix unit test cleanup handling
On Wed, Dec 2, 2020 at 7:05 PM David Gow wrote: > > On Thu, Dec 3, 2020 at 3:09 AM Daniel Latypov wrote: > > > > * Stop leaking file objects. > > * Use self.addCleanup() to ensure we call cleanup functions even if > > setUp() fails. > > * use mock.patch.stopall instead of more error-prone manual approach > > > > Signed-off-by: Daniel Latypov > > --- > > This patch hasn't changed since v1, right? > > It's still: > Reviewed-by: David Gow Oops, yes. It's entirely unchanged. The only change to the entire series was a rebase + drop the second patch in favor of revamping the test_data_path() one. > > Cheers, > -- David
Re: [PATCH 2/5] kunit: tool: fix unit test so it can run from non-root dir
On Tue, Dec 1, 2020 at 8:41 PM David Gow wrote:
>
> On Wed, Dec 2, 2020 at 3:00 AM Daniel Latypov wrote:
> >
> > On Mon, Nov 30, 2020 at 11:33 PM David Gow wrote:
> > >
> > > On Tue, Dec 1, 2020 at 7:33 AM Daniel Latypov wrote:
> > > >
> > > > get_absolute_path() makes an attempt to allow for this.
> > > > But that doesn't work as soon as os.chdir() gets called.
> > >
> > > Can we explain why this doesn't work? It's because the test_data/
> > > files are accessed with relative paths, so chdir breaks access to
> > > them, right?
> >
> > Correct.
> > Because it actually returns a relative path.
> >
> > (I forgot that I called out that get_absolute_path() gives relative
> > paths in the last patch, and not this one. I can update the commit
> > desc if we want a v2 of this)
> >
> > >
> > > >
> > > > So make it so that os.chdir() does nothing to avoid this.
> > > >
> > > > Note: mock.patch.object() doesn't seem to work in setUpModule(), hence
> > > > the introduction of a new base class instead.
> > > >
> > > > Fixes: 5578d008d9e0 ("kunit: tool: fix running kunit_tool from outside
> > > > kernel tree")
> > > > Signed-off-by: Daniel Latypov
> > > > ---
> > >
> > > I don't like this: disabling a real system call seems like overkill to
> > > work around a path issue like this.
> > >
> > > Wouldn't it be better to either:
> > > (a) stop kunit_tool from needing to chdir entirely; or
> >
> > a) is doable, but would require plumbing cwd=get_kernel_root_path() to
> > all the subprocess calls to make, etc.
> > I'm not sure fixing a internal test-only issue necessarily justifies
> > taking that path instead of the easier "just add a chdir" we opted for
> > in 5578d008d9e0 ("kunit: tool: fix running kunit_tool from outside
> > kernel tree").
> >
> > > (b) have get_absolute_path() / test_data_path() produce an absolute path.
> > >
> > > The latter really seems like the most sensible approach: have the
> > > script read its own path and read files relative to that.
> >
> > b) is not that simple, sadly.
> >
> > Say I invoke
> > $ python3 kunit_tool_test.py
> > then __file__ = kunit_tool_test.py.
> >
> > So __file__ is a relative path, but the code assumed it was an
> > absolute one and any change of directory breaks things.
> > Working around that would require something like caching the result of
> > os.path.abspath(__file__) somewhere.
>
> So, to clarify, __file__ is a relative path based on the cwd when the
> script is initially run, right?
Yes, on my box at least.
https://docs.python.org/3/reference/import.html#__file__ doesn't not
seem to stipulate an absolute path, either.
>
> In any case, caching the result of os.path.abspath(__file__) seems
> like the most sensible solution to me. There's global state anyway
> (the current directory), we might as well have it in an explicit
> variable, IMHO.
Ok, sent out a v2 and squash this change with the test_data_path() patch.
Not really a fan of the adding the global state, but I see your point
about there maybe being need for more chdir calls and I don't see a
better way of keeping track of the absolute path.
> >
> > I can see the point about not mocking out something like os.chdir().
> > But on the other hand, do we have any other legitimate reason to call
> > it besides that one place in kunit.py?
> > If we do have something that relies on doing an os.chdir(), it should
> > ideally notice that it didn't work and manifest in a unit test failure
> > somehow.
>
> Certainly there doesn't seem to be any other need to chdir() in
> kunit_tool at the moment, but I could see us doing so when adding
> other features. More to the point, if both kunit.py and
> kunit_tool_test.py rely on or manipulate the current directory as part
> of their state, that seems like it's asking for some trouble down the
> line.
>
> If we use an absolute path for the test data, that's something that
> seems unlikely to ever need further changes or cause issues.
> >
> > Alternatively, we could make get_kernel_root_path() return ""/None to
> > avoid the chdir call.
> > kunit.py: if get_kernel_root_path():
> > kunit.py: os.chdir(get_kernel_root_path())
> >
> > There'd be no adverse affects atm for stubbing that out, and I don't forsee
> > any.
> > But if we want to be even safer, then perhaps we have
> >
> > def chdir_to_kernel_root():
> >...
> >
> > and mock out just that func in the unit test?
>
> I'd be okay with this, even if I'd prefer us to use an absolute path
> for the test_data as well. Having something like this might even give
> us the opportunity to verify that we're actually trying to change to
> the kernel directory in cases where we need to, but that seems like
> it's out-of-scope for a simple fix.
>
> -- David
[PATCH v2 1/4] kunit: tool: fix unit test cleanup handling
* Stop leaking file objects.
* Use self.addCleanup() to ensure we call cleanup functions even if
setUp() fails.
* use mock.patch.stopall instead of more error-prone manual approach
Signed-off-by: Daniel Latypov
---
tools/testing/kunit/kunit_tool_test.py | 14 ++
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/tools/testing/kunit/kunit_tool_test.py
b/tools/testing/kunit/kunit_tool_test.py
index 497ab51bc170..3fbe1acd531a 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -288,19 +288,17 @@ class StrContains(str):
class KUnitMainTest(unittest.TestCase):
def setUp(self):
path =
get_absolute_path('test_data/test_is_test_passed-all_passed.log')
- file = open(path)
- all_passed_log = file.readlines()
- self.print_patch = mock.patch('builtins.print')
- self.print_mock = self.print_patch.start()
+ with open(path) as file:
+ all_passed_log = file.readlines()
+
+ self.print_mock = mock.patch('builtins.print').start()
+ self.addCleanup(mock.patch.stopall)
+
self.linux_source_mock = mock.Mock()
self.linux_source_mock.build_reconfig =
mock.Mock(return_value=True)
self.linux_source_mock.build_um_kernel =
mock.Mock(return_value=True)
self.linux_source_mock.run_kernel =
mock.Mock(return_value=all_passed_log)
- def tearDown(self):
- self.print_patch.stop()
- pass
-
def test_config_passes_args_pass(self):
kunit.main(['config', '--build_dir=.kunit'],
self.linux_source_mock)
assert self.linux_source_mock.build_reconfig.call_count == 1
base-commit: 509a15421674b9e1a3e1916939d0d0efd3e578da
--
2.29.2.576.ga3fc446d84-goog
