[RFC v4 14/17] MAINTAINERS: add entry for KUnit the unit testing framework

[Brendan Higgins via dri-devel]

Add myself as maintainer of KUnit, the Linux kernel's unit testing
framework.

Signed-off-by: Brendan Higgins 
---
 MAINTAINERS | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 8c68de3cfd80e..ff2cc9fcb49ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8267,6 +8267,16 @@ S:   Maintained
 F: tools/testing/selftests/
 F: Documentation/dev-tools/kselftest*
 
+KERNEL UNIT TESTING FRAMEWORK (KUnit)
+M: Brendan Higgins 
+L: kunit-...@googlegroups.com
+W: https://google.github.io/kunit-docs/third_party/kernel/docs/
+S: Maintained
+F: Documentation/kunit/
+F: include/kunit/
+F: kunit/
+F: tools/testing/kunit/
+
 KERNEL USERMODE HELPER
 M: Luis Chamberlain 
 L: linux-ker...@vger.kernel.org
-- 
2.21.0.rc0.258.g878e2cd30e-goog

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 07/17] kunit: test: add initial tests

[Brendan Higgins via dri-devel]

Add a test for string stream along with a simpler example.

Signed-off-by: Brendan Higgins 
---
 kunit/Kconfig  | 12 ++
 kunit/Makefile |  4 ++
 kunit/example-test.c   | 88 ++
 kunit/string-stream-test.c | 61 ++
 4 files changed, 165 insertions(+)
 create mode 100644 kunit/example-test.c
 create mode 100644 kunit/string-stream-test.c

diff --git a/kunit/Kconfig b/kunit/Kconfig
index 64480092b2c24..5cb500355c873 100644
--- a/kunit/Kconfig
+++ b/kunit/Kconfig
@@ -13,4 +13,16 @@ config KUNIT
  special hardware. For more information, please see
  Documentation/kunit/
 
+config KUNIT_TEST
+   bool "KUnit test for KUnit"
+   depends on KUNIT
+   help
+ Enables KUnit test to test KUnit.
+
+config KUNIT_EXAMPLE_TEST
+   bool "Example test for KUnit"
+   depends on KUNIT
+   help
+ Enables example KUnit test to demo features of KUnit.
+
 endmenu
diff --git a/kunit/Makefile b/kunit/Makefile
index 6ddc622ee6b1c..60a9ea6cb4697 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -1,3 +1,7 @@
 obj-$(CONFIG_KUNIT) += test.o \
string-stream.o \
kunit-stream.o
+
+obj-$(CONFIG_KUNIT_TEST) +=string-stream-test.o
+
+obj-$(CONFIG_KUNIT_EXAMPLE_TEST) +=example-test.o
diff --git a/kunit/example-test.c b/kunit/example-test.c
new file mode 100644
index 0..352f64a423e7c
--- /dev/null
+++ b/kunit/example-test.c
@@ -0,0 +1,88 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Example KUnit test to show how to use KUnit.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#include 
+
+/*
+ * This is the most fundamental element of KUnit, the test case. A test case
+ * makes a set EXPECTATIONs and ASSERTIONs about the behavior of some code; if
+ * any expectations or assertions are not met, the test fails; otherwise, the
+ * test passes.
+ *
+ * In KUnit, a test case is just a function with the signature
+ * `void (*)(struct kunit *)`. `struct kunit` is a context object that stores
+ * information about the current test.
+ */
+static void example_simple_test(struct kunit *test)
+{
+   /*
+* This is an EXPECTATION; it is how KUnit tests things. When you want
+* to test a piece of code, you set some expectations about what the
+* code should do. KUnit then runs the test and verifies that the code's
+* behavior matched what was expected.
+*/
+   KUNIT_EXPECT_EQ(test, 1 + 1, 2);
+}
+
+/*
+ * This is run once before each test case, see the comment on
+ * example_test_module for more information.
+ */
+static int example_test_init(struct kunit *test)
+{
+   kunit_info(test, "initializing");
+
+   return 0;
+}
+
+/*
+ * Here we make a list of all the test cases we want to add to the test module
+ * below.
+ */
+static struct kunit_case example_test_cases[] = {
+   /*
+* This is a helper to create a test case object from a test case
+* function; its exact function is not important to understand how to
+* use KUnit, just know that this is how you associate test cases with a
+* test module.
+*/
+   KUNIT_CASE(example_simple_test),
+   {},
+};
+
+/*
+ * This defines a suite or grouping of tests.
+ *
+ * Test cases are defined as belonging to the suite by adding them to
+ * `kunit_cases`.
+ *
+ * Often it is desirable to run some function which will set up things which
+ * will be used by every test; this is accomplished with an `init` function
+ * which runs before each test case is invoked. Similarly, an `exit` function
+ * may be specified which runs after every test case and can be used to for
+ * cleanup. For clarity, running tests in a test module would behave as 
follows:
+ *
+ * module.init(test);
+ * module.test_case[0](test);
+ * module.exit(test);
+ * module.init(test);
+ * module.test_case[1](test);
+ * module.exit(test);
+ * ...;
+ */
+static struct kunit_module example_test_module = {
+   .name = "example",
+   .init = example_test_init,
+   .test_cases = example_test_cases,
+};
+
+/*
+ * This registers the above test module telling KUnit that this is a suite of
+ * tests that need to be run.
+ */
+module_test(example_test_module);
diff --git a/kunit/string-stream-test.c b/kunit/string-stream-test.c
new file mode 100644
index 0..6cfef69568011
--- /dev/null
+++ b/kunit/string-stream-test.c
@@ -0,0 +1,61 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for struct string_stream.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#include 
+#include 
+#include 
+
+static void string_stream_test_get_string(struct kunit *test)
+{
+   struct string_stream *stream = new_string_stream();
+   char *output;
+
+   stream->add(stream, "Foo");
+   stream->add(stream, " %s", 

Re: [RFC v3 17/19] of: unittest: migrate tests to run on KUnit

[Brendan Higgins via dri-devel]

On Thu, Feb 14, 2019 at 12:10 PM Rob Herring  wrote:
>
> On Tue, Feb 12, 2019 at 7:44 PM Brendan Higgins
>  wrote:
> >
> > On Wed, Nov 28, 2018 at 12:56 PM Rob Herring  wrote:
> > >
> > > On Wed, Nov 28, 2018 at 1:38 PM Brendan Higgins
> > >  wrote:
> > > >
> > > > Migrate tests without any cleanup, or modifying test logic in anyway to
> > > > run under KUnit using the KUnit expectation and assertion API.
> > >
> > > Nice! You beat me to it. This is probably going to conflict with what
> > > is in the DT tree for 4.21. Also, please Cc the DT list for
> > > drivers/of/ changes.
> > >
> > > Looks good to me, but a few mostly formatting comments below.
> >
> > I just realized that we never talked about your other comments, and I
> > still have some questions. (Sorry, it was the last thing I looked at
> > while getting v4 ready.) No worries if you don't get to it before I
> > send v4 out, I just didn't want you to think I was ignoring you.
> >
> > >
> > > >
> > > > Signed-off-by: Brendan Higgins 
> > > > ---
> > > >  drivers/of/Kconfig|1 +
> > > >  drivers/of/unittest.c | 1405 ++---
> > > >  2 files changed, 752 insertions(+), 654 deletions(-)
> > > >
> > 
> > > > diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
> > > > index 41b49716ac75f..a5ef44730ffdb 100644
> > > > --- a/drivers/of/unittest.c
> > > > +++ b/drivers/of/unittest.c
> > 
> > > > -
> > > > -static void __init of_unittest_find_node_by_name(void)
> > > > +static void of_unittest_find_node_by_name(struct kunit *test)
> > >
> > > Why do we have to drop __init everywhere? The tests run later?
> >
> > From the standpoint of a unit test __init doesn't really make any
> > sense, right? I know that right now we are running as part of a
> > kernel, but the goal should be that a unit test is not part of a
> > kernel and we just include what we need.
>
> Well, the test only runs during boot and better to free the space when
> done with it. There was some desire to make it a kernel module and
> then we'd also need to get rid of __init too.
>
> > Even so, that's the future. For now, I did not put the KUnit
> > infrastructure in the .init section because I didn't think it belonged
> > there. In practice, KUnit only knows how to run during the init phase
> > of the kernel, but I don't think it should be restricted there. You
> > should be able to run tests whenever you want because you should be
> > able to test anything right? I figured any restriction on that is
> > misleading and will potentially get in the way at worst, and
> > unnecessary at best especially since people shouldn't build a
> > production kernel with all kinds of unit tests inside.
>
> More folks will run things if they can be enabled on production
> kernels. If size is the only issue, modules mitigate that. However,
> there's probably APIs to test which we don't want to export to
> modules.
>
> I think in general, we change things in the kernel when needed, not
> for something in the future. Changing __init is simple enough to do
> later.
>
> OTOH, things get copied and maybe this we don't want copied, so we can
> remove it if you want to.

Mmmm...I just realized that the patch I sent you the other day makes
this patch unhappy because unflatten_device_tree is in the .init
section. So I will need to fix that. I still think that the correct
course of action is to make KUnit non init. Luis pointed out in
another thread that to be 100% sure that everything will be properly
initialized, KUnit must be able to run after all initialization takes
place.

>
> > 
> > > >
> > > > -static void __init of_unittest_property_string(void)
> > > > +static void of_unittest_property_string(struct kunit *test)
> > > >  {
> > > > const char *strings[4];
> > > > struct device_node *np;
> > > > int rc;
> > > >
> > > > np = 
> > > > of_find_node_by_path("/testcase-data/phandle-tests/consumer-a");
> > > > -   if (!np) {
> > > > -   pr_err("No testcase data in device tree\n");
> > > > -   return;
> > > > -   }
> > > > -
> > > > -   rc = of_property_match_string(np, "phandle-list-names", 
> > > > "first");
> > > > -   unittest(rc == 0, "first expected:0 got:%i\n", rc);
> > > > -   rc = of_property_match_string(np, "phandle-list-names", 
> > > > "second");
> > > > -   unittest(rc == 1, "second expected:1 got:%i\n", rc);
> > > > -   rc = of_property_match_string(np, "phandle-list-names", 
> > > > "third");
> > > > -   unittest(rc == 2, "third expected:2 got:%i\n", rc);
> > > > -   rc = of_property_match_string(np, "phandle-list-names", 
> > > > "fourth");
> > > > -   unittest(rc == -ENODATA, "unmatched string; rc=%i\n", rc);
> > > > -   rc = of_property_match_string(np, "missing-property", "blah");
> > > > -   unittest(rc == -EINVAL, "missing property; rc=%i\n", rc);
> > > > -   rc = of_property_match_string(np, "empty-property", "blah");
> > > > -   

[RFC v4 16/17] of: unittest: split out a couple of test cases from unittest

[Brendan Higgins via dri-devel]

Split out a couple of test cases that these features in base.c from the
unittest.c monolith. The intention is that we will eventually split out
all test cases and group them together based on what portion of device
tree they test.

Signed-off-by: Brendan Higgins 
---
 drivers/of/Makefile  |   2 +-
 drivers/of/base-test.c   | 214 
 drivers/of/test-common.c | 175 
 drivers/of/test-common.h |  16 ++
 drivers/of/unittest.c| 345 +--
 5 files changed, 407 insertions(+), 345 deletions(-)
 create mode 100644 drivers/of/base-test.c
 create mode 100644 drivers/of/test-common.c
 create mode 100644 drivers/of/test-common.h

diff --git a/drivers/of/Makefile b/drivers/of/Makefile
index 663a4af0cccd5..4a4bd527d586c 100644
--- a/drivers/of/Makefile
+++ b/drivers/of/Makefile
@@ -8,7 +8,7 @@ obj-$(CONFIG_OF_PROMTREE) += pdt.o
 obj-$(CONFIG_OF_ADDRESS)  += address.o
 obj-$(CONFIG_OF_IRQ)+= irq.o
 obj-$(CONFIG_OF_NET)   += of_net.o
-obj-$(CONFIG_OF_UNITTEST) += unittest.o
+obj-$(CONFIG_OF_UNITTEST) += unittest.o base-test.o test-common.o
 obj-$(CONFIG_OF_MDIO)  += of_mdio.o
 obj-$(CONFIG_OF_RESERVED_MEM) += of_reserved_mem.o
 obj-$(CONFIG_OF_RESOLVE)  += resolver.o
diff --git a/drivers/of/base-test.c b/drivers/of/base-test.c
new file mode 100644
index 0..3d3f4f1b74800
--- /dev/null
+++ b/drivers/of/base-test.c
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Unit tests for functions defined in base.c.
+ */
+#include 
+
+#include 
+
+#include "test-common.h"
+
+static void of_unittest_find_node_by_name(struct kunit *test)
+{
+   struct device_node *np;
+   const char *options, *name;
+
+   np = of_find_node_by_path("/testcase-data");
+   name = kasprintf(GFP_KERNEL, "%pOF", np);
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   KUNIT_EXPECT_STREQ_MSG(test, "/testcase-data", name,
+  "find /testcase-data failed\n");
+   of_node_put(np);
+   kfree(name);
+
+   /* Test if trailing '/' works */
+   KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("/testcase-data/"), NULL,
+   "trailing '/' on /testcase-data/ should fail\n");
+
+   np = of_find_node_by_path("/testcase-data/phandle-tests/consumer-a");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   name = kasprintf(GFP_KERNEL, "%pOF", np);
+   KUNIT_EXPECT_STREQ_MSG(
+   test, "/testcase-data/phandle-tests/consumer-a", name,
+   "find /testcase-data/phandle-tests/consumer-a failed\n");
+   of_node_put(np);
+   kfree(name);
+
+   np = of_find_node_by_path("testcase-alias");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   name = kasprintf(GFP_KERNEL, "%pOF", np);
+   KUNIT_EXPECT_STREQ_MSG(test, "/testcase-data", name,
+  "find testcase-alias failed\n");
+   of_node_put(np);
+   kfree(name);
+
+   /* Test if trailing '/' works on aliases */
+   KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("testcase-alias/"), NULL,
+   "trailing '/' on testcase-alias/ should fail\n");
+
+   np = of_find_node_by_path("testcase-alias/phandle-tests/consumer-a");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   name = kasprintf(GFP_KERNEL, "%pOF", np);
+   KUNIT_EXPECT_STREQ_MSG(
+   test, "/testcase-data/phandle-tests/consumer-a", name,
+   "find testcase-alias/phandle-tests/consumer-a failed\n");
+   of_node_put(np);
+   kfree(name);
+
+   KUNIT_EXPECT_EQ_MSG(
+   test,
+   np = of_find_node_by_path("/testcase-data/missing-path"), NULL,
+   "non-existent path returned node %pOF\n", np);
+   of_node_put(np);
+
+   KUNIT_EXPECT_EQ_MSG(
+   test, np = of_find_node_by_path("missing-alias"), NULL,
+   "non-existent alias returned node %pOF\n", np);
+   of_node_put(np);
+
+   KUNIT_EXPECT_EQ_MSG(
+   test,
+   np = of_find_node_by_path("testcase-alias/missing-path"), NULL,
+   "non-existent alias with relative path returned node %pOF\n",
+   np);
+   of_node_put(np);
+
+   np = of_find_node_opts_by_path("/testcase-data:testoption", );
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   KUNIT_EXPECT_STREQ_MSG(test, "testoption", options,
+  "option path test failed\n");
+   of_node_put(np);
+
+   np = of_find_node_opts_by_path("/testcase-data:test/option", );
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   KUNIT_EXPECT_STREQ_MSG(test, "test/option", options,
+  "option path test, subcase #1 failed\n");
+   of_node_put(np);
+
+   np = 
of_find_node_opts_by_path("/testcase-data/testcase-device1:test/option", 
);
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   KUNIT_EXPECT_STREQ_MSG(test, "test/option", 

[RFC v4 15/17] of: unittest: migrate tests to run on KUnit

[Brendan Higgins via dri-devel]

Migrate tests without any cleanup, or modifying test logic in anyway to
run under KUnit using the KUnit expectation and assertion API.

Signed-off-by: Brendan Higgins 
---
 drivers/of/Kconfig|1 +
 drivers/of/unittest.c | 1310 +
 2 files changed, 671 insertions(+), 640 deletions(-)

diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig
index ad3fcad4d75b8..f309399deac20 100644
--- a/drivers/of/Kconfig
+++ b/drivers/of/Kconfig
@@ -15,6 +15,7 @@ if OF
 config OF_UNITTEST
bool "Device Tree runtime unit tests"
depends on !SPARC
+   depends on KUNIT
select IRQ_DOMAIN
select OF_EARLY_FLATTREE
select OF_RESOLVE
diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
index effa4e2b9d992..96de69ccb3e63 100644
--- a/drivers/of/unittest.c
+++ b/drivers/of/unittest.c
@@ -26,186 +26,189 @@
 
 #include 
 
+#include 
+
 #include "of_private.h"
 
-static struct unittest_results {
-   int passed;
-   int failed;
-} unittest_results;
-
-#define unittest(result, fmt, ...) ({ \
-   bool failed = !(result); \
-   if (failed) { \
-   unittest_results.failed++; \
-   pr_err("FAIL %s():%i " fmt, __func__, __LINE__, ##__VA_ARGS__); 
\
-   } else { \
-   unittest_results.passed++; \
-   pr_debug("pass %s():%i\n", __func__, __LINE__); \
-   } \
-   failed; \
-})
-
-static void __init of_unittest_find_node_by_name(void)
+static void of_unittest_find_node_by_name(struct kunit *test)
 {
struct device_node *np;
const char *options, *name;
 
np = of_find_node_by_path("/testcase-data");
name = kasprintf(GFP_KERNEL, "%pOF", np);
-   unittest(np && !strcmp("/testcase-data", name),
-   "find /testcase-data failed\n");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
+   KUNIT_EXPECT_STREQ_MSG(test, "/testcase-data", name,
+  "find /testcase-data failed\n");
of_node_put(np);
kfree(name);
 
/* Test if trailing '/' works */
-   np = of_find_node_by_path("/testcase-data/");
-   unittest(!np, "trailing '/' on /testcase-data/ should fail\n");
+   KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("/testcase-data/"), NULL,
+   "trailing '/' on /testcase-data/ should fail\n");
 
np = of_find_node_by_path("/testcase-data/phandle-tests/consumer-a");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
name = kasprintf(GFP_KERNEL, "%pOF", np);
-   unittest(np && !strcmp("/testcase-data/phandle-tests/consumer-a", name),
+   KUNIT_EXPECT_STREQ_MSG(
+   test, "/testcase-data/phandle-tests/consumer-a", name,
"find /testcase-data/phandle-tests/consumer-a failed\n");
of_node_put(np);
kfree(name);
 
np = of_find_node_by_path("testcase-alias");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
name = kasprintf(GFP_KERNEL, "%pOF", np);
-   unittest(np && !strcmp("/testcase-data", name),
-   "find testcase-alias failed\n");
+   KUNIT_EXPECT_STREQ_MSG(test, "/testcase-data", name,
+  "find testcase-alias failed\n");
of_node_put(np);
kfree(name);
 
/* Test if trailing '/' works on aliases */
-   np = of_find_node_by_path("testcase-alias/");
-   unittest(!np, "trailing '/' on testcase-alias/ should fail\n");
+   KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("testcase-alias/"), NULL,
+   "trailing '/' on testcase-alias/ should fail\n");
 
np = of_find_node_by_path("testcase-alias/phandle-tests/consumer-a");
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
name = kasprintf(GFP_KERNEL, "%pOF", np);
-   unittest(np && !strcmp("/testcase-data/phandle-tests/consumer-a", name),
+   KUNIT_EXPECT_STREQ_MSG(
+   test, "/testcase-data/phandle-tests/consumer-a", name,
"find testcase-alias/phandle-tests/consumer-a failed\n");
of_node_put(np);
kfree(name);
 
-   np = of_find_node_by_path("/testcase-data/missing-path");
-   unittest(!np, "non-existent path returned node %pOF\n", np);
+   KUNIT_EXPECT_EQ_MSG(
+   test,
+   np = of_find_node_by_path("/testcase-data/missing-path"), NULL,
+   "non-existent path returned node %pOF\n", np);
of_node_put(np);
 
-   np = of_find_node_by_path("missing-alias");
-   unittest(!np, "non-existent alias returned node %pOF\n", np);
+   KUNIT_EXPECT_EQ_MSG(
+   test, np = of_find_node_by_path("missing-alias"), NULL,
+   "non-existent alias returned node %pOF\n", np);
of_node_put(np);
 
-   np = of_find_node_by_path("testcase-alias/missing-path");
-   unittest(!np, "non-existent alias with relative path returned node 
%pOF\n", np);
+   KUNIT_EXPECT_EQ_MSG(
+   test,
+ 

[RFC v4 13/17] Documentation: kunit: add documentation for KUnit

[Brendan Higgins via dri-devel]

Add documentation for KUnit, the Linux kernel unit testing framework.
- Add intro and usage guide for KUnit
- Add API reference

Signed-off-by: Felix Guo 
Signed-off-by: Brendan Higgins 
---
 Documentation/index.rst   |   1 +
 Documentation/kunit/api/index.rst |  16 ++
 Documentation/kunit/api/test.rst  |  15 +
 Documentation/kunit/faq.rst   |  46 +++
 Documentation/kunit/index.rst |  80 ++
 Documentation/kunit/start.rst | 180 
 Documentation/kunit/usage.rst | 447 ++
 7 files changed, 785 insertions(+)
 create mode 100644 Documentation/kunit/api/index.rst
 create mode 100644 Documentation/kunit/api/test.rst
 create mode 100644 Documentation/kunit/faq.rst
 create mode 100644 Documentation/kunit/index.rst
 create mode 100644 Documentation/kunit/start.rst
 create mode 100644 Documentation/kunit/usage.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index c858c2e66e361..9512de536b34a 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -65,6 +65,7 @@ merged much easier.
kernel-hacking/index
trace/index
maintainer/index
+   kunit/index
 
 Kernel API documentation
 
diff --git a/Documentation/kunit/api/index.rst 
b/Documentation/kunit/api/index.rst
new file mode 100644
index 0..c31c530088153
--- /dev/null
+++ b/Documentation/kunit/api/index.rst
@@ -0,0 +1,16 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=
+API Reference
+=
+.. toctree::
+
+   test
+
+This section documents the KUnit kernel testing API. It is divided into 3
+sections:
+
+= 
==
+:doc:`test`   documents all of the standard testing API
+  excluding mocking or mocking related 
features.
+= 
==
diff --git a/Documentation/kunit/api/test.rst b/Documentation/kunit/api/test.rst
new file mode 100644
index 0..7c926014f047c
--- /dev/null
+++ b/Documentation/kunit/api/test.rst
@@ -0,0 +1,15 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+Test API
+
+
+This file documents all of the standard testing API excluding mocking or 
mocking
+related features.
+
+.. kernel-doc:: include/kunit/test.h
+   :internal:
+
+.. kernel-doc:: include/kunit/kunit-stream.h
+   :internal:
+
diff --git a/Documentation/kunit/faq.rst b/Documentation/kunit/faq.rst
new file mode 100644
index 0..cb8e4fb2257a0
--- /dev/null
+++ b/Documentation/kunit/faq.rst
@@ -0,0 +1,46 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=
+Frequently Asked Questions
+=
+
+How is this different from Autotest, kselftest, etc?
+
+KUnit is a unit testing framework. Autotest, kselftest (and some others) are
+not.
+
+A `unit test `_ is supposed to
+test a single unit of code in isolation, hence the name. A unit test should be
+the finest granularity of testing and as such should allow all possible code
+paths to be tested in the code under test; this is only possible if the code
+under test is very small and does not have any external dependencies outside of
+the test's control like hardware.
+
+There are no testing frameworks currently available for the kernel that do not
+require installing the kernel on a test machine or in a VM and all require
+tests to be written in userspace and run on the kernel under test; this is true
+for Autotest, kselftest, and some others, disqualifying any of them from being
+considered unit testing frameworks.
+
+What is the difference between a unit test and these other kinds of tests?
+==
+Most existing tests for the Linux kernel would be categorized as an integration
+test, or an end-to-end test.
+
+- A unit test is supposed to test a single unit of code in isolation, hence the
+  name. A unit test should be the finest granularity of testing and as such
+  should allow all possible code paths to be tested in the code under test; 
this
+  is only possible if the code under test is very small and does not have any
+  external dependencies outside of the test's control like hardware.
+- An integration test tests the interaction between a minimal set of 
components,
+  usually just two or three. For example, someone might write an integration
+  test to test the interaction between a driver and a piece of hardware, or to
+  test the interaction between the userspace libraries the kernel provides and
+  the kernel itself; however, one of these tests would probably not test the
+  entire kernel along with hardware interactions and interactions with the
+  userspace.
+- An end-to-end test usually tests the entire system from the 

[RFC v4 10/17] kunit: test: add test managed resource tests

[Brendan Higgins via dri-devel]

Tests how tests interact with test managed resources in their lifetime.

Signed-off-by: Avinash Kondareddy 
Signed-off-by: Brendan Higgins 
---
 kunit/test-test.c | 121 ++
 1 file changed, 121 insertions(+)

diff --git a/kunit/test-test.c b/kunit/test-test.c
index 0b4ad6690310d..bb34431398526 100644
--- a/kunit/test-test.c
+++ b/kunit/test-test.c
@@ -127,3 +127,124 @@ static struct kunit_module kunit_try_catch_test_module = {
.test_cases = kunit_try_catch_test_cases,
 };
 module_test(kunit_try_catch_test_module);
+
+/*
+ * Context for testing test managed resources
+ * is_resource_initialized is used to test arbitrary resources
+ */
+struct kunit_test_resource_context {
+   struct kunit test;
+   bool is_resource_initialized;
+};
+
+static int fake_resource_init(struct kunit_resource *res, void *context)
+{
+   struct kunit_test_resource_context *ctx = context;
+
+   res->allocation = >is_resource_initialized;
+   ctx->is_resource_initialized = true;
+   return 0;
+}
+
+static void fake_resource_free(struct kunit_resource *res)
+{
+   bool *is_resource_initialized = res->allocation;
+
+   *is_resource_initialized = false;
+}
+
+static void kunit_resource_test_init_resources(struct kunit *test)
+{
+   struct kunit_test_resource_context *ctx = test->priv;
+
+   kunit_init_test(>test, "testing_test_init_test");
+
+   KUNIT_EXPECT_TRUE(test, list_empty(>test.resources));
+}
+
+static void kunit_resource_test_alloc_resource(struct kunit *test)
+{
+   struct kunit_test_resource_context *ctx = test->priv;
+   struct kunit_resource *res;
+   kunit_resource_free_t free = fake_resource_free;
+
+   res = kunit_alloc_resource(>test,
+  fake_resource_init,
+  fake_resource_free,
+  ctx);
+
+   KUNIT_ASSERT_NOT_ERR_OR_NULL(test, res);
+   KUNIT_EXPECT_EQ(test, >is_resource_initialized, res->allocation);
+   KUNIT_EXPECT_TRUE(test, list_is_last(>node, >test.resources));
+   KUNIT_EXPECT_EQ(test, free, res->free);
+}
+
+static void kunit_resource_test_free_resource(struct kunit *test)
+{
+   struct kunit_test_resource_context *ctx = test->priv;
+   struct kunit_resource *res = kunit_alloc_resource(>test,
+ fake_resource_init,
+ fake_resource_free,
+ ctx);
+
+   kunit_free_resource(>test, res);
+
+   KUNIT_EXPECT_EQ(test, false, ctx->is_resource_initialized);
+   KUNIT_EXPECT_TRUE(test, list_empty(>test.resources));
+}
+
+#define KUNIT_RESOURCE_NUM 5
+static void kunit_resource_test_cleanup_resources(struct kunit *test)
+{
+   int i;
+   struct kunit_test_resource_context *ctx = test->priv;
+   struct kunit_resource *resources[KUNIT_RESOURCE_NUM];
+
+   for (i = 0; i < KUNIT_RESOURCE_NUM; i++) {
+   resources[i] = kunit_alloc_resource(>test,
+   fake_resource_init,
+   fake_resource_free,
+   ctx);
+   }
+
+   kunit_cleanup(>test);
+
+   KUNIT_EXPECT_TRUE(test, list_empty(>test.resources));
+}
+
+static int kunit_resource_test_init(struct kunit *test)
+{
+   struct kunit_test_resource_context *ctx =
+   kzalloc(sizeof(*ctx), GFP_KERNEL);
+
+   if (!ctx)
+   return -ENOMEM;
+   test->priv = ctx;
+
+   kunit_init_test(>test, "test_test_context");
+   return 0;
+}
+
+static void kunit_resource_test_exit(struct kunit *test)
+{
+   struct kunit_test_resource_context *ctx = test->priv;
+
+   kunit_cleanup(>test);
+   kfree(ctx);
+}
+
+static struct kunit_case kunit_resource_test_cases[] = {
+   KUNIT_CASE(kunit_resource_test_init_resources),
+   KUNIT_CASE(kunit_resource_test_alloc_resource),
+   KUNIT_CASE(kunit_resource_test_free_resource),
+   KUNIT_CASE(kunit_resource_test_cleanup_resources),
+   {},
+};
+
+static struct kunit_module kunit_resource_test_module = {
+   .name = "kunit-resource-test",
+   .init = kunit_resource_test_init,
+   .exit = kunit_resource_test_exit,
+   .test_cases = kunit_resource_test_cases,
+};
+module_test(kunit_resource_test_module);
-- 
2.21.0.rc0.258.g878e2cd30e-goog

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 05/17] kunit: test: add the concept of expectations

[Brendan Higgins via dri-devel]

Add support for expectations, which allow properties to be specified and
then verified in tests.

Signed-off-by: Brendan Higgins 
---
 include/kunit/test.h | 415 +++
 kunit/test.c |  34 
 2 files changed, 449 insertions(+)

diff --git a/include/kunit/test.h b/include/kunit/test.h
index 75cd3c3ab1b4b..a36ad1a502c66 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -273,4 +273,419 @@ void __printf(3, 4) kunit_printk(const char *level,
 #define kunit_err(test, fmt, ...) \
kunit_printk(KERN_ERR, test, fmt, ##__VA_ARGS__)
 
+static inline struct kunit_stream *kunit_expect_start(struct kunit *test,
+ const char *file,
+ const char *line)
+{
+   struct kunit_stream *stream = kunit_new_stream(test);
+
+   stream->add(stream, "EXPECTATION FAILED at %s:%s\n\t", file, line);
+
+   return stream;
+}
+
+static inline void kunit_expect_end(struct kunit *test,
+   bool success,
+   struct kunit_stream *stream)
+{
+   if (!success)
+   test->fail(test, stream);
+   else
+   stream->clear(stream);
+}
+
+#define KUNIT_EXPECT_START(test) \
+   kunit_expect_start(test, __FILE__, __stringify(__LINE__))
+
+#define KUNIT_EXPECT_END(test, success, stream) \
+   kunit_expect_end(test, success, stream)
+
+#define KUNIT_EXPECT_MSG(test, success, message, fmt, ...) do {
   \
+   struct kunit_stream *__stream = KUNIT_EXPECT_START(test);  \
+  \
+   __stream->add(__stream, message);  \
+   __stream->add(__stream, fmt, ##__VA_ARGS__);   \
+   KUNIT_EXPECT_END(test, success, __stream); \
+} while (0)
+
+#define KUNIT_EXPECT(test, success, message) do { \
+   struct kunit_stream *__stream = KUNIT_EXPECT_START(test);  \
+  \
+   __stream->add(__stream, message);  \
+   KUNIT_EXPECT_END(test, success, __stream); \
+} while (0)
+
+/**
+ * KUNIT_SUCCEED() - A no-op expectation. Only exists for code clarity.
+ * @test: The test context object.
+ *
+ * The opposite of KUNIT_FAIL(), it is an expectation that cannot fail. In 
other
+ * words, it does nothing and only exists for code clarity. See
+ * KUNIT_EXPECT_TRUE() for more information.
+ */
+#define KUNIT_SUCCEED(test) do {} while (0)
+
+/**
+ * KUNIT_FAIL() - Always causes a test to fail when evaluated.
+ * @test: The test context object.
+ * @fmt: an informational message to be printed when the assertion is made.
+ * @...: string format arguments.
+ *
+ * The opposite of KUNIT_SUCCEED(), it is an expectation that always fails. In
+ * other words, it always results in a failed expectation, and consequently
+ * always causes the test case to fail when evaluated. See KUNIT_EXPECT_TRUE()
+ * for more information.
+ */
+#define KUNIT_FAIL(test, fmt, ...) \
+   KUNIT_EXPECT_MSG(test, false, "", fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_TRUE() - Causes a test failure when the expression is not true.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails when this does
+ * not evaluate to true.
+ *
+ * This and expectations of the form `KUNIT_EXPECT_*` will cause the test case
+ * to fail when the specified condition is not met; however, it will not 
prevent
+ * the test case from continuing to run; this is otherwise known as an
+ * *expectation failure*.
+ */
+#define KUNIT_EXPECT_TRUE(test, condition)\
+   KUNIT_EXPECT(test, (condition),\
+  "Expected " #condition " is true, but is false.")
+
+#define KUNIT_EXPECT_TRUE_MSG(test, condition, fmt, ...)  \
+   KUNIT_EXPECT_MSG(test, (condition),\
+   "Expected " #condition " is true, but is 
false.\n",\
+   fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_FALSE() - Makes a test failure when the expression is not 
false.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails when this does
+ * not evaluate to false.
+ *
+ * Sets an expectation that @condition evaluates to false. See
+ * KUNIT_EXPECT_TRUE() for more information.
+ */
+#define KUNIT_EXPECT_FALSE(test, condition)   \
+   KUNIT_EXPECT(test, !(condition),   \
+  "Expected " 

Re: [RFC v3 14/19] Documentation: kunit: add documentation for KUnit

[Brendan Higgins via dri-devel]

On Thu, Feb 14, 2019 at 9:26 AM Luis Chamberlain  wrote:
>
> On Wed, Feb 13, 2019 at 04:17:13PM -0800, Brendan Higgins wrote:
> > On Wed, Feb 13, 2019 at 1:55 PM Kieran Bingham
> >  wrote:
> > Oh, yep, you are right. Does that mean we should bother at all with a 
> > defconfig?
>
> If one wanted a qemu enabled type of kernel and also for kuniut one
> could imply run:
>
> make kvmconfig
> make kunitconfig
>
> That would get what you suggest above of default "bells and whistles"
> and keep the kuniut as a fragment.
>
> Hm, actually the kvmconfig doesn't really enable the required fragments
> for qemu, so perhaps one would be good. It would have the serial stuff
> for instance.
>
> > Luis, I know you said you wanted one. I am thinking just stick with
> > the UML one? The downside there is we then get stuck having to
> > maintain the fragment and the defconfig. I right now (in the new
> > revision I am working on) have the Python kunit_tool copy the
> > defconfig if no kunitconfig is provided and a flag is set. It would be
> > pretty straightforward to make it merge in the fragment instead.
>
> Up to you in the end.

I don't really have any opinions on the matter; I don't really use
defconfigs in any of my workflows. So, I just want whatever is easier
for people. The thing that makes the most sense to me would be to
provide a "merge-kunitconfig" option similar to what kselftest does,
but I don't intend on doing that in the initial patchset, unless
someone really thinks that I should do it. So in the meantime, I guess
provide both since that gives people options?

In anycase, I just (finally) sent out v4, so I suggest we continue the
discussion over there.

Cheers
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 11/17] kunit: tool: add Python wrappers for running KUnit tests

[Brendan Higgins via dri-devel]

From: Felix Guo 

The ultimate goal is to create minimal isolated test binaries; in the
meantime we are using UML to provide the infrastructure to run tests, so
define an abstract way to configure and run tests that allow us to
change the context in which tests are built without affecting the user.
This also makes pretty and dynamic error reporting, and a lot of other
nice features easier.

kunit_config.py:
  - parse .config and Kconfig files.

kunit_kernel.py: provides helper functions to:
  - configure the kernel using kunitconfig.
  - build the kernel with the appropriate configuration.
  - provide function to invoke the kernel and stream the output back.

Signed-off-by: Felix Guo 
Signed-off-by: Brendan Higgins 
---
Changes Since Last Version
 - Added support for building and running tests in an external
   directory.
 - Squashed with most other kunit_tool commits, since most did not
   represent a coherent new feature.
---
 tools/testing/kunit/.gitignore  |   3 +
 tools/testing/kunit/kunit.py|  78 +++
 tools/testing/kunit/kunit_config.py |  66 +
 tools/testing/kunit/kunit_kernel.py | 148 
 tools/testing/kunit/kunit_parser.py | 119 ++
 5 files changed, 414 insertions(+)
 create mode 100644 tools/testing/kunit/.gitignore
 create mode 100755 tools/testing/kunit/kunit.py
 create mode 100644 tools/testing/kunit/kunit_config.py
 create mode 100644 tools/testing/kunit/kunit_kernel.py
 create mode 100644 tools/testing/kunit/kunit_parser.py

diff --git a/tools/testing/kunit/.gitignore b/tools/testing/kunit/.gitignore
new file mode 100644
index 0..c791ff59a37a9
--- /dev/null
+++ b/tools/testing/kunit/.gitignore
@@ -0,0 +1,3 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
\ No newline at end of file
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
new file mode 100755
index 0..7413ec7351a20
--- /dev/null
+++ b/tools/testing/kunit/kunit.py
@@ -0,0 +1,78 @@
+#!/usr/bin/python3
+# SPDX-License-Identifier: GPL-2.0
+#
+# A thin wrapper on top of the KUnit Kernel
+#
+# Copyright (C) 2019, Google LLC.
+# Author: Felix Guo 
+# Author: Brendan Higgins 
+
+import argparse
+import sys
+import os
+import time
+
+import kunit_config
+import kunit_kernel
+import kunit_parser
+
+parser = argparse.ArgumentParser(description='Runs KUnit tests.')
+
+parser.add_argument('--raw_output', help='don\'t format output from kernel',
+   action='store_true')
+
+parser.add_argument('--timeout', help='maximum number of seconds to allow for '
+   'all tests to run. This does not include time taken to '
+   'build the tests.', type=int, default=300,
+   metavar='timeout')
+
+parser.add_argument('--jobs',
+   help='As in the make command, "Specifies  the number of '
+   'jobs (commands) to run simultaneously."',
+   type=int, default=8, metavar='jobs')
+
+parser.add_argument('--build_dir',
+   help='As in the make command, it specifies the build '
+   'directory.',
+   type=str, default=None, metavar='build_dir')
+
+cli_args = parser.parse_args()
+
+linux = kunit_kernel.LinuxSourceTree()
+
+build_dir = None
+if cli_args.build_dir:
+   build_dir = cli_args.build_dir
+
+config_start = time.time()
+success = linux.build_reconfig(build_dir)
+config_end = time.time()
+if not success:
+   quit()
+
+kunit_parser.print_with_timestamp('Building KUnit Kernel ...')
+
+build_start = time.time()
+
+success = linux.build_um_kernel(jobs=cli_args.jobs, build_dir=build_dir)
+build_end = time.time()
+if not success:
+   quit()
+
+kunit_parser.print_with_timestamp('Starting KUnit Kernel ...')
+test_start = time.time()
+
+if cli_args.raw_output:
+   kunit_parser.raw_output(linux.run_kernel(timeout=cli_args.timeout,
+build_dir=build_dir))
+else:
+   kunit_parser.parse_run_tests(linux.run_kernel(timeout=cli_args.timeout,
+ build_dir=build_dir))
+
+test_end = time.time()
+
+kunit_parser.print_with_timestamp((
+   "Elapsed time: %.3fs total, %.3fs configuring, %.3fs " +
+   "building, %.3fs running.\n") % (test_end - config_start,
+   config_end - config_start, build_end - build_start,
+   test_end - test_start))
diff --git a/tools/testing/kunit/kunit_config.py 
b/tools/testing/kunit/kunit_config.py
new file mode 100644
index 0..167f47d9ab8e4
--- /dev/null
+++ b/tools/testing/kunit/kunit_config.py
@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Builds a .config from a kunitconfig.
+#
+# Copyright (C) 2019, Google LLC.
+# Author: Felix Guo 
+# Author: Brendan Higgins 
+
+import collections
+import re
+
+CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_\w+ is not set$'
+CONFIG_PATTERN = r'^CONFIG_\w+=\S+$'
+

[RFC v4 09/17] kunit: test: add the concept of assertions

[Brendan Higgins via dri-devel]

Add support for assertions which are like expectations except the test
terminates if the assertion is not satisfied.

Signed-off-by: Brendan Higgins 
---
 include/kunit/test.h   | 397 -
 kunit/string-stream-test.c |  12 +-
 kunit/test-test.c  |   2 +
 kunit/test.c   |  33 +++
 4 files changed, 435 insertions(+), 9 deletions(-)

diff --git a/include/kunit/test.h b/include/kunit/test.h
index cd02dca96eb61..c42c67a9729fd 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -84,9 +84,10 @@ struct kunit;
  * @name: the name of the test case.
  *
  * A test case is a function with the signature, ``void (*)(struct kunit *)``
- * that makes expectations (see KUNIT_EXPECT_TRUE()) about code under test. 
Each
- * test case is associated with a  kunit_module and will be run after 
the
- * module's init function and followed by the module's exit function.
+ * that makes expectations and assertions (see KUNIT_EXPECT_TRUE() and
+ * KUNIT_ASSERT_TRUE()) about code under test. Each test case is associated 
with
+ * a  kunit_module and will be run after the module's init function and
+ * followed by the module's exit function.
  *
  * A test case should be static and should only be created with the 
KUNIT_CASE()
  * macro; additionally, every array of test cases should be terminated with an
@@ -712,4 +713,394 @@ static inline void kunit_expect_binary(struct kunit *test,
KUNIT_EXPECT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);  \
 } while (0)
 
+static inline struct kunit_stream *kunit_assert_start(struct kunit *test,
+   const char *file,
+   const char *line)
+{
+   struct kunit_stream *stream = kunit_new_stream(test);
+
+   stream->add(stream, "ASSERTION FAILED at %s:%s\n\t", file, line);
+
+   return stream;
+}
+
+static inline void kunit_assert_end(struct kunit *test,
+  bool success,
+  struct kunit_stream *stream)
+{
+   if (!success) {
+   test->fail(test, stream);
+   test->abort(test);
+   } else {
+   stream->clear(stream);
+   }
+}
+
+#define KUNIT_ASSERT_START(test) \
+   kunit_assert_start(test, __FILE__, __stringify(__LINE__))
+
+#define KUNIT_ASSERT_END(test, success, stream) \
+   kunit_assert_end(test, success, stream)
+
+#define KUNIT_ASSERT(test, success, message) do { \
+   struct kunit_stream *__stream = KUNIT_ASSERT_START(test);  \
+  \
+   __stream->add(__stream, message);  \
+   KUNIT_ASSERT_END(test, success, __stream); \
+} while (0)
+
+#define KUNIT_ASSERT_MSG(test, success, message, fmt, ...) do {
   \
+   struct kunit_stream *__stream = KUNIT_ASSERT_START(test);  \
+  \
+   __stream->add(__stream, message);  \
+   __stream->add(__stream, fmt, ##__VA_ARGS__);   \
+   KUNIT_ASSERT_END(test, success, __stream); \
+} while (0)
+
+#define KUNIT_ASSERT_FAILURE(test, fmt, ...) \
+   KUNIT_ASSERT_MSG(test, false, "", fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_TRUE() - Sets an assertion that @condition is true.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails and aborts when
+ * this does not evaluate to true.
+ *
+ * This and assertions of the form `KUNIT_ASSERT_*` will cause the test case to
+ * fail *and immediately abort* when the specified condition is not met. Unlike
+ * an expectation failure, it will prevent the test case from continuing to 
run;
+ * this is otherwise known as an *assertion failure*.
+ */
+#define KUNIT_ASSERT_TRUE(test, condition)\
+   KUNIT_ASSERT(test, (condition),\
+  "Asserted " #condition " is true, but is false.")
+
+#define KUNIT_ASSERT_TRUE_MSG(test, condition, fmt, ...)  \
+   KUNIT_ASSERT_MSG(test, (condition),\
+   "Asserted " #condition " is true, but is 
false.\n",\
+   fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_FALSE() - Sets an assertion that @condition is false.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression.
+ *
+ * Sets an assertion that the value that @condition evaluates to is false. This
+ * is the same as KUNIT_EXPECT_FALSE(), except it causes an assertion failure
+ * (see KUNIT_ASSERT_TRUE()) when the assertion is 

[RFC v4 12/17] kunit: defconfig: add defconfigs for building KUnit tests

[Brendan Higgins via dri-devel]

Add defconfig for UML and a fragment that can be used to configure other
architectures for building KUnit tests. Add option to kunit_tool to use
a defconfig to create the kunitconfig.

Signed-off-by: Brendan Higgins 
---
Changes Since Last Version
 - This patch is new adding default configs to build KUnit.
 - NOTE: there is still some discussion to be had here about whether we
   should go with a defconfig, a config fragment, or both.
---
 arch/um/configs/kunit_defconfig  |  8 
 tools/testing/kunit/configs/all_tests.config |  8 
 tools/testing/kunit/kunit.py | 17 +++--
 tools/testing/kunit/kunit_kernel.py  |  3 ++-
 4 files changed, 33 insertions(+), 3 deletions(-)
 create mode 100644 arch/um/configs/kunit_defconfig
 create mode 100644 tools/testing/kunit/configs/all_tests.config

diff --git a/arch/um/configs/kunit_defconfig b/arch/um/configs/kunit_defconfig
new file mode 100644
index 0..bfe49689038f1
--- /dev/null
+++ b/arch/um/configs/kunit_defconfig
@@ -0,0 +1,8 @@
+CONFIG_OF=y
+CONFIG_OF_UNITTEST=y
+CONFIG_OF_OVERLAY=y
+CONFIG_I2C=y
+CONFIG_I2C_MUX=y
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
diff --git a/tools/testing/kunit/configs/all_tests.config 
b/tools/testing/kunit/configs/all_tests.config
new file mode 100644
index 0..bfe49689038f1
--- /dev/null
+++ b/tools/testing/kunit/configs/all_tests.config
@@ -0,0 +1,8 @@
+CONFIG_OF=y
+CONFIG_OF_UNITTEST=y
+CONFIG_OF_OVERLAY=y
+CONFIG_I2C=y
+CONFIG_I2C_MUX=y
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 7413ec7351a20..63e9fb3b60200 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -11,6 +11,7 @@ import argparse
 import sys
 import os
 import time
+import shutil
 
 import kunit_config
 import kunit_kernel
@@ -36,14 +37,26 @@ parser.add_argument('--build_dir',
'directory.',
type=str, default=None, metavar='build_dir')
 
-cli_args = parser.parse_args()
+parser.add_argument('--defconfig',
+   help='Uses a default kunitconfig.',
+   action='store_true')
 
-linux = kunit_kernel.LinuxSourceTree()
+def create_default_kunitconfig():
+   if not os.path.exists(kunit_kernel.KUNITCONFIG_PATH):
+   shutil.copyfile('arch/um/configs/kunit_defconfig',
+   kunit_kernel.KUNITCONFIG_PATH)
+
+cli_args = parser.parse_args()
 
 build_dir = None
 if cli_args.build_dir:
build_dir = cli_args.build_dir
 
+if cli_args.defconfig:
+   create_default_kunitconfig()
+
+linux = kunit_kernel.LinuxSourceTree()
+
 config_start = time.time()
 success = linux.build_reconfig(build_dir)
 config_end = time.time()
diff --git a/tools/testing/kunit/kunit_kernel.py 
b/tools/testing/kunit/kunit_kernel.py
index 07c0abf2f47df..bf38768353313 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -14,6 +14,7 @@ import os
 import kunit_config
 
 KCONFIG_PATH = '.config'
+KUNITCONFIG_PATH = 'kunitconfig'
 
 class ConfigError(Exception):
"""Represents an error trying to configure the Linux kernel."""
@@ -81,7 +82,7 @@ class LinuxSourceTree(object):
 
def __init__(self):
self._kconfig = kunit_config.Kconfig()
-   self._kconfig.read_from_file('kunitconfig')
+   self._kconfig.read_from_file(KUNITCONFIG_PATH)
self._ops = LinuxSourceTreeOperations()
 
def clean(self):
-- 
2.21.0.rc0.258.g878e2cd30e-goog

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 06/17] kbuild: enable building KUnit

[Brendan Higgins via dri-devel]

Add KUnit to root Kconfig and Makefile allowing it to actually be built.

Signed-off-by: Brendan Higgins 
---
Changes Since Last Version
 - Rewrote patch description. This was previously called "[RFC v3 06/19]
   arch: um: enable running kunit from User Mode Linux," which was
   incorrect since this patch does not have any UML specific bits in it.
---
 Kconfig  | 2 ++
 Makefile | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/Kconfig b/Kconfig
index 48a80beab6853..10428501edb78 100644
--- a/Kconfig
+++ b/Kconfig
@@ -30,3 +30,5 @@ source "crypto/Kconfig"
 source "lib/Kconfig"
 
 source "lib/Kconfig.debug"
+
+source "kunit/Kconfig"
diff --git a/Makefile b/Makefile
index 3142e67d03f1b..d10308eb7f214 100644
--- a/Makefile
+++ b/Makefile
@@ -958,7 +958,7 @@ endif
 PHONY += prepare0
 
 ifeq ($(KBUILD_EXTMOD),)
-core-y += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
+core-y += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/ kunit/
 
 vmlinux-dirs   := $(patsubst %/,%,$(filter %/, $(init-y) $(init-m) \
 $(core-y) $(core-m) $(drivers-y) $(drivers-m) \
-- 
2.21.0.rc0.258.g878e2cd30e-goog

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 04/17] kunit: test: add test_stream a std::stream like logger

[Brendan Higgins via dri-devel]

A lot of the expectation and assertion infrastructure prints out fairly
complicated test failure messages, so add a C++ style log library for
for logging test results.

Signed-off-by: Brendan Higgins 
---
 include/kunit/kunit-stream.h |  50 
 include/kunit/test.h |   2 +
 kunit/Makefile   |   3 +-
 kunit/kunit-stream.c | 153 +++
 kunit/test.c |   8 ++
 5 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 include/kunit/kunit-stream.h
 create mode 100644 kunit/kunit-stream.c

diff --git a/include/kunit/kunit-stream.h b/include/kunit/kunit-stream.h
new file mode 100644
index 0..15ad83a6b7aae
--- /dev/null
+++ b/include/kunit/kunit-stream.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#ifndef _KUNIT_KUNIT_STREAM_H
+#define _KUNIT_KUNIT_STREAM_H
+
+#include 
+#include 
+
+struct kunit;
+
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ * @set_level: sets the level that this string should be printed at.
+ * @add: adds the formatted input to the internal buffer.
+ * @append: adds the contents of other to this.
+ * @commit: prints out the internal buffer to the user.
+ * @clear: clears the internal buffer.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
+struct kunit_stream {
+   void (*set_level)(struct kunit_stream *this, const char *level);
+   void (*add)(struct kunit_stream *this, const char *fmt, ...);
+   void (*append)(struct kunit_stream *this, struct kunit_stream *other);
+   void (*commit)(struct kunit_stream *this);
+   void (*clear)(struct kunit_stream *this);
+   /* private: internal use only. */
+   struct kunit *test;
+   spinlock_t lock; /* Guards level. */
+   const char *level;
+   struct string_stream *internal_stream;
+};
+
+/**
+ * kunit_new_stream() - constructs a new  kunit_stream.
+ * @test: The test context object.
+ *
+ * Constructs a new test managed  kunit_stream.
+ */
+struct kunit_stream *kunit_new_stream(struct kunit *test);
+
+#endif /* _KUNIT_KUNIT_STREAM_H */
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 21abc9e953969..75cd3c3ab1b4b 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -11,6 +11,7 @@
 
 #include 
 #include 
+#include 
 
 struct kunit_resource;
 
@@ -171,6 +172,7 @@ struct kunit {
void (*vprintk)(const struct kunit *test,
const char *level,
struct va_format *vaf);
+   void (*fail)(struct kunit *test, struct kunit_stream *stream);
 };
 
 int kunit_init_test(struct kunit *test, const char *name);
diff --git a/kunit/Makefile b/kunit/Makefile
index 275b565a0e81f..6ddc622ee6b1c 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_KUNIT) += test.o \
-   string-stream.o
+   string-stream.o \
+   kunit-stream.o
diff --git a/kunit/kunit-stream.c b/kunit/kunit-stream.c
new file mode 100644
index 0..bc88638aef3b1
--- /dev/null
+++ b/kunit/kunit-stream.c
@@ -0,0 +1,153 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#include 
+#include 
+#include 
+
+static const char *kunit_stream_get_level(struct kunit_stream *this)
+{
+   unsigned long flags;
+   const char *level;
+
+   spin_lock_irqsave(>lock, flags);
+   level = this->level;
+   spin_unlock_irqrestore(>lock, flags);
+
+   return level;
+}
+
+static void kunit_stream_set_level(struct kunit_stream *this, const char 
*level)
+{
+   unsigned long flags;
+
+   spin_lock_irqsave(>lock, flags);
+   this->level = level;
+   spin_unlock_irqrestore(>lock, flags);
+}
+
+static void kunit_stream_add(struct kunit_stream *this, const char *fmt, ...)
+{
+   va_list args;
+   struct string_stream *stream = this->internal_stream;
+
+   va_start(args, fmt);
+   if (stream->vadd(stream, fmt, args) < 0)
+   kunit_err(this->test, "Failed to allocate fragment: %s", fmt);
+
+   va_end(args);
+}
+
+static void kunit_stream_append(struct kunit_stream *this,
+   struct kunit_stream *other)
+{
+   struct string_stream *other_stream = other->internal_stream;
+   const char *other_content;
+
+   other_content = other_stream->get_string(other_stream);
+
+   if (!other_content) {
+   kunit_err(this->test,
+ "Failed to get string from second argument for 

[RFC v4 17/17] of: unittest: split up some super large test cases

[Brendan Higgins via dri-devel]

Split up the super large test cases of_unittest_find_node_by_name and
of_unittest_dynamic into properly sized and defined test cases.

Signed-off-by: Brendan Higgins 
---
 drivers/of/base-test.c | 297 ++---
 1 file changed, 249 insertions(+), 48 deletions(-)

diff --git a/drivers/of/base-test.c b/drivers/of/base-test.c
index 3d3f4f1b74800..7b44c967ed2fd 100644
--- a/drivers/of/base-test.c
+++ b/drivers/of/base-test.c
@@ -8,10 +8,10 @@
 
 #include "test-common.h"
 
-static void of_unittest_find_node_by_name(struct kunit *test)
+static void of_test_find_node_by_name_basic(struct kunit *test)
 {
struct device_node *np;
-   const char *options, *name;
+   const char *name;
 
np = of_find_node_by_path("/testcase-data");
name = kasprintf(GFP_KERNEL, "%pOF", np);
@@ -20,11 +20,21 @@ static void of_unittest_find_node_by_name(struct kunit 
*test)
   "find /testcase-data failed\n");
of_node_put(np);
kfree(name);
+}
 
+static void of_test_find_node_by_name_trailing_slash(struct kunit *test)
+{
/* Test if trailing '/' works */
KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("/testcase-data/"), NULL,
"trailing '/' on /testcase-data/ should fail\n");
 
+}
+
+static void of_test_find_node_by_name_multiple_components(struct kunit *test)
+{
+   struct device_node *np;
+   const char *name;
+
np = of_find_node_by_path("/testcase-data/phandle-tests/consumer-a");
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
name = kasprintf(GFP_KERNEL, "%pOF", np);
@@ -33,6 +43,12 @@ static void of_unittest_find_node_by_name(struct kunit *test)
"find /testcase-data/phandle-tests/consumer-a failed\n");
of_node_put(np);
kfree(name);
+}
+
+static void of_test_find_node_by_name_with_alias(struct kunit *test)
+{
+   struct device_node *np;
+   const char *name;
 
np = of_find_node_by_path("testcase-alias");
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
@@ -41,10 +57,23 @@ static void of_unittest_find_node_by_name(struct kunit 
*test)
   "find testcase-alias failed\n");
of_node_put(np);
kfree(name);
+}
 
+static void of_test_find_node_by_name_with_alias_and_slash(struct kunit *test)
+{
/* Test if trailing '/' works on aliases */
KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("testcase-alias/"), NULL,
-   "trailing '/' on testcase-alias/ should fail\n");
+  "trailing '/' on testcase-alias/ should fail\n");
+}
+
+/*
+ * TODO(brendanhigg...@google.com): This looks like a duplicate of
+ * of_test_find_node_by_name_multiple_components
+ */
+static void of_test_find_node_by_name_multiple_components_2(struct kunit *test)
+{
+   struct device_node *np;
+   const char *name;
 
np = of_find_node_by_path("testcase-alias/phandle-tests/consumer-a");
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
@@ -54,17 +83,33 @@ static void of_unittest_find_node_by_name(struct kunit 
*test)
"find testcase-alias/phandle-tests/consumer-a failed\n");
of_node_put(np);
kfree(name);
+}
+
+static void of_test_find_node_by_name_missing_path(struct kunit *test)
+{
+   struct device_node *np;
 
KUNIT_EXPECT_EQ_MSG(
test,
np = of_find_node_by_path("/testcase-data/missing-path"), NULL,
"non-existent path returned node %pOF\n", np);
of_node_put(np);
+}
+
+static void of_test_find_node_by_name_missing_alias(struct kunit *test)
+{
+   struct device_node *np;
 
KUNIT_EXPECT_EQ_MSG(
test, np = of_find_node_by_path("missing-alias"), NULL,
"non-existent alias returned node %pOF\n", np);
of_node_put(np);
+}
+
+static void of_test_find_node_by_name_missing_alias_with_relative_path(
+   struct kunit *test)
+{
+   struct device_node *np;
 
KUNIT_EXPECT_EQ_MSG(
test,
@@ -72,12 +117,24 @@ static void of_unittest_find_node_by_name(struct kunit 
*test)
"non-existent alias with relative path returned node %pOF\n",
np);
of_node_put(np);
+}
+
+static void of_test_find_node_by_name_with_option(struct kunit *test)
+{
+   struct device_node *np;
+   const char *options;
 
np = of_find_node_opts_by_path("/testcase-data:testoption", );
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
KUNIT_EXPECT_STREQ_MSG(test, "testoption", options,
   "option path test failed\n");
of_node_put(np);
+}
+
+static void of_test_find_node_by_name_with_option_and_slash(struct kunit *test)
+{
+   struct device_node *np;
+   const char *options;
 
np = of_find_node_opts_by_path("/testcase-data:test/option", );
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, np);
@@ -90,11 +147,22 @@ 

[RFC v4 02/17] kunit: test: add test resource management API

[Brendan Higgins via dri-devel]

Create a common API for test managed resources like memory and test
objects. A lot of times a test will want to set up infrastructure to be
used in test cases; this could be anything from just wanting to allocate
some memory to setting up a driver stack; this defines facilities for
creating "test resources" which are managed by the test infrastructure
and are automatically cleaned up at the conclusion of the test.

Signed-off-by: Brendan Higgins 
---
 include/kunit/test.h | 109 +++
 kunit/test.c |  95 +
 2 files changed, 204 insertions(+)

diff --git a/include/kunit/test.h b/include/kunit/test.h
index 23c2ebedd6dd9..21abc9e953969 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -12,6 +12,69 @@
 #include 
 #include 
 
+struct kunit_resource;
+
+typedef int (*kunit_resource_init_t)(struct kunit_resource *, void *);
+typedef void (*kunit_resource_free_t)(struct kunit_resource *);
+
+/**
+ * struct kunit_resource - represents a *test managed resource*
+ * @allocation: for the user to store arbitrary data.
+ * @free: a user supplied function to free the resource. Populated by
+ * kunit_alloc_resource().
+ *
+ * Represents a *test managed resource*, a resource which will automatically be
+ * cleaned up at the end of a test case.
+ *
+ * Example:
+ *
+ * .. code-block:: c
+ *
+ * struct kunit_kmalloc_params {
+ * size_t size;
+ * gfp_t gfp;
+ * };
+ *
+ * static int kunit_kmalloc_init(struct kunit_resource *res, void *context)
+ * {
+ * struct kunit_kmalloc_params *params = context;
+ * res->allocation = kmalloc(params->size, params->gfp);
+ *
+ * if (!res->allocation)
+ * return -ENOMEM;
+ *
+ * return 0;
+ * }
+ *
+ * static void kunit_kmalloc_free(struct kunit_resource *res)
+ * {
+ * kfree(res->allocation);
+ * }
+ *
+ * void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp)
+ * {
+ * struct kunit_kmalloc_params params;
+ * struct kunit_resource *res;
+ *
+ * params.size = size;
+ * params.gfp = gfp;
+ *
+ * res = kunit_alloc_resource(test, kunit_kmalloc_init,
+ * kunit_kmalloc_free, );
+ * if (res)
+ * return res->allocation;
+ * else
+ * return NULL;
+ * }
+ */
+struct kunit_resource {
+   void *allocation;
+   kunit_resource_free_t free;
+
+   /* private: internal use only. */
+   struct list_head node;
+};
+
 struct kunit;
 
 /**
@@ -104,6 +167,7 @@ struct kunit {
const char *name; /* Read only after initialization! */
spinlock_t lock; /* Gaurds all mutable test state. */
bool success; /* Protected by lock. */
+   struct list_head resources; /* Protected by lock. */
void (*vprintk)(const struct kunit *test,
const char *level,
struct va_format *vaf);
@@ -127,6 +191,51 @@ int kunit_run_tests(struct kunit_module *module);
} \
late_initcall(module_kunit_init##module)
 
+/**
+ * kunit_alloc_resource() - Allocates a *test managed resource*.
+ * @test: The test context object.
+ * @init: a user supplied function to initialize the resource.
+ * @free: a user supplied function to free the resource.
+ * @context: for the user to pass in arbitrary data.
+ *
+ * Allocates a *test managed resource*, a resource which will automatically be
+ * cleaned up at the end of a test case. See  kunit_resource for an
+ * example.
+ */
+struct kunit_resource *kunit_alloc_resource(struct kunit *test,
+   kunit_resource_init_t init,
+   kunit_resource_free_t free,
+   void *context);
+
+void kunit_free_resource(struct kunit *test, struct kunit_resource *res);
+
+/**
+ * kunit_kmalloc() - Like kmalloc() except the allocation is *test managed*.
+ * @test: The test context object.
+ * @size: The size in bytes of the desired memory.
+ * @gfp: flags passed to underlying kmalloc().
+ *
+ * Just like `kmalloc(...)`, except the allocation is managed by the test case
+ * and is automatically cleaned up after the test case concludes. See 
+ * kunit_resource for more information.
+ */
+void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp);
+
+/**
+ * kunit_kzalloc() - Just like kunit_kmalloc(), but zeroes the allocation.
+ * @test: The test context object.
+ * @size: The size in bytes of the desired memory.
+ * @gfp: flags passed to underlying kmalloc().
+ *
+ * See kzalloc() and kunit_kmalloc() for more information.
+ */
+static inline void *kunit_kzalloc(struct kunit *test, size_t size, gfp_t gfp)
+{
+   return kunit_kmalloc(test, size, gfp | __GFP_ZERO);
+}
+
+void kunit_cleanup(struct 

[RFC v4 08/17] kunit: test: add support for test abort

[Brendan Higgins via dri-devel]

Add support for aborting/bailing out of test cases. Needed for
implementing assertions.

Signed-off-by: Brendan Higgins 
---
Changes Since Last Version
 - This patch is new introducing a new cross-architecture way to abort
   out of a test case (needed for KUNIT_ASSERT_*, see next patch for
   details).
 - On a side note, this is not a complete replacement for the UML abort
   mechanism, but covers the majority of necessary functionality. UML
   architecture specific featurs have been dropped from the initial
   patchset.
---
 include/kunit/test.h |  24 +
 kunit/Makefile   |   3 +-
 kunit/test-test.c| 127 ++
 kunit/test.c | 208 +--
 4 files changed, 353 insertions(+), 9 deletions(-)
 create mode 100644 kunit/test-test.c

diff --git a/include/kunit/test.h b/include/kunit/test.h
index a36ad1a502c66..cd02dca96eb61 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -151,6 +151,26 @@ struct kunit_module {
struct kunit_case *test_cases;
 };
 
+struct kunit_try_catch_context {
+   struct kunit *test;
+   struct kunit_module *module;
+   struct kunit_case *test_case;
+   struct completion *try_completion;
+   int try_result;
+};
+
+struct kunit_try_catch {
+   void (*run)(struct kunit_try_catch *try_catch);
+   void (*throw)(struct kunit_try_catch *try_catch);
+   struct kunit_try_catch_context context;
+   void (*try)(struct kunit_try_catch_context *context);
+   void (*catch)(struct kunit_try_catch_context *context);
+};
+
+void kunit_try_catch_init(struct kunit_try_catch *try_catch);
+
+void kunit_generic_try_catch_init(struct kunit_try_catch *try_catch);
+
 /**
  * struct kunit - represents a running instance of a test.
  * @priv: for user to store arbitrary data. Commonly used to pass data created
@@ -166,13 +186,17 @@ struct kunit {
 
/* private: internal use only. */
const char *name; /* Read only after initialization! */
+   struct kunit_try_catch try_catch;
spinlock_t lock; /* Gaurds all mutable test state. */
bool success; /* Protected by lock. */
+   bool death_test; /* Protected by lock. */
struct list_head resources; /* Protected by lock. */
+   void (*set_death_test)(struct kunit *test, bool death_test);
void (*vprintk)(const struct kunit *test,
const char *level,
struct va_format *vaf);
void (*fail)(struct kunit *test, struct kunit_stream *stream);
+   void (*abort)(struct kunit *test);
 };
 
 int kunit_init_test(struct kunit *test, const char *name);
diff --git a/kunit/Makefile b/kunit/Makefile
index 60a9ea6cb4697..e4c300f67479a 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -2,6 +2,7 @@ obj-$(CONFIG_KUNIT) +=  test.o \
string-stream.o \
kunit-stream.o
 
-obj-$(CONFIG_KUNIT_TEST) +=string-stream-test.o
+obj-$(CONFIG_KUNIT_TEST) +=test-test.o \
+   string-stream-test.o
 
 obj-$(CONFIG_KUNIT_EXAMPLE_TEST) +=example-test.o
diff --git a/kunit/test-test.c b/kunit/test-test.c
new file mode 100644
index 0..a936c041f1c8f
--- /dev/null
+++ b/kunit/test-test.c
@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for core test infrastructure.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+#include 
+
+struct kunit_try_catch_test_context {
+   struct kunit_try_catch *try_catch;
+   bool function_called;
+};
+
+void kunit_test_successful_try(struct kunit_try_catch_context *context)
+{
+   struct kunit_try_catch_test_context *ctx = context->test->priv;
+
+   ctx->function_called = true;
+}
+
+void kunit_test_no_catch(struct kunit_try_catch_context *context)
+{
+   KUNIT_FAIL(context->test, "Catch should not be called.");
+}
+
+static void kunit_test_try_catch_successful_try_no_catch(struct kunit *test)
+{
+   struct kunit_try_catch_test_context *ctx = test->priv;
+   struct kunit_try_catch *try_catch = ctx->try_catch;
+
+   try_catch->try = kunit_test_successful_try;
+   try_catch->catch = kunit_test_no_catch;
+   try_catch->run(try_catch);
+
+   KUNIT_EXPECT_TRUE(test, ctx->function_called);
+}
+
+void kunit_test_unsuccessful_try(struct kunit_try_catch_context *context)
+{
+   struct kunit_try_catch *try_catch = container_of(context,
+struct kunit_try_catch,
+context);
+
+   try_catch->throw(try_catch);
+   KUNIT_FAIL(context->test, "This line should never be reached.");
+}
+
+void kunit_test_catch(struct kunit_try_catch_context *context)
+{
+   struct kunit_try_catch_test_context *ctx = context->test->priv;
+
+   ctx->function_called = true;
+}
+
+static 

Re: [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest

[Brendan Higgins via dri-devel]

On Thu, Feb 14, 2019 at 6:05 PM Frank Rowand  wrote:
>
> On 2/14/19 4:56 PM, Brendan Higgins wrote:
> > On Thu, Feb 14, 2019 at 3:57 PM Frank Rowand  wrote:
> >>
> >> On 12/5/18 3:54 PM, Brendan Higgins wrote:
> >>> On Tue, Dec 4, 2018 at 2:58 AM Frank Rowand  
> >>> wrote:
> 
>  Hi Brendan,
> 
>  On 11/28/18 11:36 AM, Brendan Higgins wrote:
> > Split out a couple of test cases that these features in base.c from the
> > unittest.c monolith. The intention is that we will eventually split out
> > all test cases and group them together based on what portion of device
> > tree they test.
> 
>  Why does splitting this file apart improve the implementation?
> >>>
> >>> This is in preparation for patch 19/19 and other hypothetical future
> >>> patches where test cases are split up and grouped together by what
> >>> portion of DT they test (for example the parsing tests and the
> >>> platform/device tests would probably go separate files as well). This
> >>> patch by itself does not do anything useful, but I figured it made
> >>> patch 19/19 (and, if you like what I am doing, subsequent patches)
> >>> easier to review.
> >>
> >> I do not see any value in splitting the devicetree tests into
> >> multiple files.
> >>
> >> Please help me understand what the benefits of such a split are.
>
> Note that my following comments are specific to the current devicetree
> unittests, and may not apply to the general case of unit tests in other
> subsystems.
>
Note taken.
>
> > Sorry, I thought it made sense in context of what I am doing in the
> > following patch. All I am trying to do is to provide an effective way
> > of grouping test cases. To be clear, the idea, assuming you agree, is
>
> Looking at _just_ the first few fragments of the following patch, the
> change is to break down a moderate size function of related tests,
> of_unittest_find_node_by_name(), into a lot of extremely small functions.

Hmm...I wouldn't call that a moderate function. By my standards those
functions are pretty large. In any case, I want to limit the
discussion to specifically what a test case should look like, and the
general consensus outside of the kernel is that unit test cases should
be very very small. The reason is that each test case is supposed to
test one specific property; it should be obvious what that property
is; and it should be obvious what is needed to exercise that property.

> Then to find the execution order of the many small functions requires
> finding the array of_test_find_node_by_name_cases[].  Then I have to

Execution order shouldn't matter. Each test case should be totally
hermetic. Obviously in this case we depend on the preceeding test case
to clean up properly, but that is something I am working on.

> chase off into the kunit test runner core, where I find that the set
> of tests in of_test_find_node_by_name_cases[] is processed by a
> late_initcall().  So now the order of the various test groupings,

That's fair. You are not the only one to complain about that. The
late_initcall is a hack which I plan on replacing shortly (and yes I
know that me planning on doing something doesn't mean much in this
discussion, but that's what I got); regardless, order shouldn't
matter.

> declared via module_test(), are subject to the fragile orderings
> of initcalls.
>
> There are ordering dependencies within the devicetree unittests.

There is now in the current devicetree unittests, but, if I may be so
bold, that is something that I would like to fix.

>
> I do not like breaking the test cases down into such small atoms.
>
> I do not see any value __for devicetree unittests__ of having
> such small atoms.

I imagine it probably makes less sense in the context of a strict
dependency order, but that is something that I want to do away with.
Ideally, when you look at a test case you shouldn't need to think
about anything other than the code under test and the test case
itself; so in my universe, a smaller test case should mean less you
need to think about.

I don't want to get hung up on size too much because I don't think
this is what it is really about. I think you and I can agree that a
test should be as simple and complete as possible. The ideal test
should cover all behavior, and should be obviously correct (since
otherwise we would have to test the test too). Obviously, these two
goals are at odds, so the compromise I attempt to make is to make a
bunch of test cases which are separately simple enough to be obviously
correct at first glance, and the sum total of all the tests provides
the necessary coverage. Additionally, because each test case is
independent of every other test case, they can be reasoned about
individually, and it is not necessary to reason about them as a group.
Hypothetically, this should give you the best of both worlds.

So even if I failed in execution, I think the principle is good.

>
> It makes it harder for me to read the source of the tests and
> understand 

Re: [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest

[Brendan Higgins via dri-devel]

On Thu, Feb 14, 2019 at 3:57 PM Frank Rowand  wrote:
>
> On 12/5/18 3:54 PM, Brendan Higgins wrote:
> > On Tue, Dec 4, 2018 at 2:58 AM Frank Rowand  wrote:
> >>
> >> Hi Brendan,
> >>
> >> On 11/28/18 11:36 AM, Brendan Higgins wrote:
> >>> Split out a couple of test cases that these features in base.c from the
> >>> unittest.c monolith. The intention is that we will eventually split out
> >>> all test cases and group them together based on what portion of device
> >>> tree they test.
> >>
> >> Why does splitting this file apart improve the implementation?
> >
> > This is in preparation for patch 19/19 and other hypothetical future
> > patches where test cases are split up and grouped together by what
> > portion of DT they test (for example the parsing tests and the
> > platform/device tests would probably go separate files as well). This
> > patch by itself does not do anything useful, but I figured it made
> > patch 19/19 (and, if you like what I am doing, subsequent patches)
> > easier to review.
>
> I do not see any value in splitting the devicetree tests into
> multiple files.
>
> Please help me understand what the benefits of such a split are.

Sorry, I thought it made sense in context of what I am doing in the
following patch. All I am trying to do is to provide an effective way
of grouping test cases. To be clear, the idea, assuming you agree, is
that we would follow up with several other patches like this one and
the subsequent patch, one which would pull out a couple test
functions, as I have done here, and another that splits those
functions up into a bunch of proper test cases.

I thought that having that many unrelated test cases in a single file
would just be a pain to sort through deal with, review, whatever.

This is not something I feel particularly strongly about, it is just
pretty atypical from my experience to have so many unrelated test
cases in a single file.

Maybe you would prefer that I break up the test cases first, and then
we split up the file as appropriate?

I just assumed that we would agree it would be way too much stuff for
a single file, so I went ahead and broke it up first, because I
thought it would make it easier to review in that order rather than
the other way around.

Cheers
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[RFC v4 01/17] kunit: test: add KUnit test runner core

[Brendan Higgins via dri-devel]

Add core facilities for defining unit tests; this provides a common way
to define test cases, functions that execute code which is under test
and determine whether the code under test behaves as expected; this also
provides a way to group together related test cases in test suites (here
we call them test_modules).

Just define test cases and how to execute them for now; setting
expectations on code will be defined later.

Signed-off-by: Brendan Higgins 
---
 include/kunit/test.h | 165 ++
 kunit/Kconfig|  16 +
 kunit/Makefile   |   1 +
 kunit/test.c | 168 +++
 4 files changed, 350 insertions(+)
 create mode 100644 include/kunit/test.h
 create mode 100644 kunit/Kconfig
 create mode 100644 kunit/Makefile
 create mode 100644 kunit/test.c

diff --git a/include/kunit/test.h b/include/kunit/test.h
new file mode 100644
index 0..23c2ebedd6dd9
--- /dev/null
+++ b/include/kunit/test.h
@@ -0,0 +1,165 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Base unit test (KUnit) API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#ifndef _KUNIT_TEST_H
+#define _KUNIT_TEST_H
+
+#include 
+#include 
+
+struct kunit;
+
+/**
+ * struct kunit_case - represents an individual test case.
+ * @run_case: the function representing the actual test case.
+ * @name: the name of the test case.
+ *
+ * A test case is a function with the signature, ``void (*)(struct kunit *)``
+ * that makes expectations (see KUNIT_EXPECT_TRUE()) about code under test. 
Each
+ * test case is associated with a  kunit_module and will be run after 
the
+ * module's init function and followed by the module's exit function.
+ *
+ * A test case should be static and should only be created with the 
KUNIT_CASE()
+ * macro; additionally, every array of test cases should be terminated with an
+ * empty test case.
+ *
+ * Example:
+ *
+ * .. code-block:: c
+ *
+ * void add_test_basic(struct kunit *test)
+ * {
+ * KUNIT_EXPECT_EQ(test, 1, add(1, 0));
+ * KUNIT_EXPECT_EQ(test, 2, add(1, 1));
+ * KUNIT_EXPECT_EQ(test, 0, add(-1, 1));
+ * KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX));
+ * KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN));
+ * }
+ *
+ * static struct kunit_case example_test_cases[] = {
+ * KUNIT_CASE(add_test_basic),
+ * {},
+ * };
+ *
+ */
+struct kunit_case {
+   void (*run_case)(struct kunit *test);
+   const char name[256];
+
+   /* private: internal use only. */
+   bool success;
+};
+
+/**
+ * KUNIT_CASE - A helper for creating a  kunit_case
+ * @test_name: a reference to a test case function.
+ *
+ * Takes a symbol for a function representing a test case and creates a
+ *  kunit_case object from it. See the documentation for
+ *  kunit_case for an example on how to use it.
+ */
+#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
+
+/**
+ * struct kunit_module - describes a related collection of  kunit_case 
s.
+ * @name: the name of the test. Purely informational.
+ * @init: called before every test case.
+ * @exit: called after every test case.
+ * @test_cases: a null terminated array of test cases.
+ *
+ * A kunit_module is a collection of related  kunit_case s, such that
+ * @init is called before every test case and @exit is called after every test
+ * case, similar to the notion of a *test fixture* or a *test class* in other
+ * unit testing frameworks like JUnit or Googletest.
+ *
+ * Every  kunit_case must be associated with a kunit_module for KUnit to
+ * run it.
+ */
+struct kunit_module {
+   const char name[256];
+   int (*init)(struct kunit *test);
+   void (*exit)(struct kunit *test);
+   struct kunit_case *test_cases;
+};
+
+/**
+ * struct kunit - represents a running instance of a test.
+ * @priv: for user to store arbitrary data. Commonly used to pass data created
+ * in the init function (see  kunit_module).
+ *
+ * Used to store information about the current context under which the test is
+ * running. Most of this data is private and should only be accessed indirectly
+ * via public functions; the one exception is @priv which can be used by the
+ * test writer to store arbitrary data.
+ */
+struct kunit {
+   void *priv;
+
+   /* private: internal use only. */
+   const char *name; /* Read only after initialization! */
+   spinlock_t lock; /* Gaurds all mutable test state. */
+   bool success; /* Protected by lock. */
+   void (*vprintk)(const struct kunit *test,
+   const char *level,
+   struct va_format *vaf);
+};
+
+int kunit_init_test(struct kunit *test, const char *name);
+
+int kunit_run_tests(struct kunit_module *module);
+
+/**
+ * module_test() - used to register a  kunit_module with KUnit.
+ * @module: a statically allocated  kunit_module.
+ *
+ * 

[RFC v4 03/17] kunit: test: add string_stream a std::stream like string builder

[Brendan Higgins via dri-devel]

A number of test features need to do pretty complicated string printing
where it may not be possible to rely on a single preallocated string
with parameters.

So provide a library for constructing the string as you go similar to
C++'s std::string.

Signed-off-by: Brendan Higgins 
---
Changes Since Last Version
 - None. There was some discussion about maybe trying to generalize this
   or replace it with something existing, but it didn't seem feasible to
   generalize this, and there wasn't really anything that is a great
   replacement.
---
 include/kunit/string-stream.h |  44 ++
 kunit/Makefile|   3 +-
 kunit/string-stream.c | 149 ++
 3 files changed, 195 insertions(+), 1 deletion(-)
 create mode 100644 include/kunit/string-stream.h
 create mode 100644 kunit/string-stream.c

diff --git a/include/kunit/string-stream.h b/include/kunit/string-stream.h
new file mode 100644
index 0..280ee67559588
--- /dev/null
+++ b/include/kunit/string-stream.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#ifndef _KUNIT_STRING_STREAM_H
+#define _KUNIT_STRING_STREAM_H
+
+#include 
+#include 
+#include 
+#include 
+
+struct string_stream_fragment {
+   struct list_head node;
+   char *fragment;
+};
+
+struct string_stream {
+   size_t length;
+   struct list_head fragments;
+
+   /* length and fragments are protected by this lock */
+   spinlock_t lock;
+   struct kref refcount;
+   int (*add)(struct string_stream *this, const char *fmt, ...);
+   int (*vadd)(struct string_stream *this, const char *fmt, va_list args);
+   char *(*get_string)(struct string_stream *this);
+   void (*clear)(struct string_stream *this);
+   bool (*is_empty)(struct string_stream *this);
+};
+
+struct string_stream *new_string_stream(void);
+
+void destroy_string_stream(struct string_stream *stream);
+
+void string_stream_get(struct string_stream *stream);
+
+int string_stream_put(struct string_stream *stream);
+
+#endif /* _KUNIT_STRING_STREAM_H */
diff --git a/kunit/Makefile b/kunit/Makefile
index 5efdc4dea2c08..275b565a0e81f 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -1 +1,2 @@
-obj-$(CONFIG_KUNIT) += test.o
+obj-$(CONFIG_KUNIT) += test.o \
+   string-stream.o
diff --git a/kunit/string-stream.c b/kunit/string-stream.c
new file mode 100644
index 0..e90fb595a5607
--- /dev/null
+++ b/kunit/string-stream.c
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins 
+ */
+
+#include 
+#include 
+#include 
+
+static int string_stream_vadd(struct string_stream *this,
+  const char *fmt,
+  va_list args)
+{
+   struct string_stream_fragment *fragment;
+   int len;
+   va_list args_for_counting;
+   unsigned long flags;
+
+   /* Make a copy because `vsnprintf` could change it */
+   va_copy(args_for_counting, args);
+
+   /* Need space for null byte. */
+   len = vsnprintf(NULL, 0, fmt, args_for_counting) + 1;
+
+   va_end(args_for_counting);
+
+   fragment = kmalloc(sizeof(*fragment), GFP_KERNEL);
+   if (!fragment)
+   return -ENOMEM;
+
+   fragment->fragment = kmalloc(len, GFP_KERNEL);
+   if (!fragment->fragment) {
+   kfree(fragment);
+   return -ENOMEM;
+   }
+
+   len = vsnprintf(fragment->fragment, len, fmt, args);
+   spin_lock_irqsave(>lock, flags);
+   this->length += len;
+   list_add_tail(>node, >fragments);
+   spin_unlock_irqrestore(>lock, flags);
+   return 0;
+}
+
+static int string_stream_add(struct string_stream *this, const char *fmt, ...)
+{
+   va_list args;
+   int result;
+
+   va_start(args, fmt);
+   result = string_stream_vadd(this, fmt, args);
+   va_end(args);
+   return result;
+}
+
+static void string_stream_clear(struct string_stream *this)
+{
+   struct string_stream_fragment *fragment, *fragment_safe;
+   unsigned long flags;
+
+   spin_lock_irqsave(>lock, flags);
+   list_for_each_entry_safe(fragment,
+fragment_safe,
+>fragments,
+node) {
+   list_del(>node);
+   kfree(fragment->fragment);
+   kfree(fragment);
+   }
+   this->length = 0;
+   spin_unlock_irqrestore(>lock, flags);
+}
+
+static char *string_stream_get_string(struct string_stream *this)
+{
+   struct string_stream_fragment *fragment;
+   size_t buf_len = this->length + 1; /* +1 for null 

[RFC v4 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework

[Brendan Higgins via dri-devel]

This patch set proposes KUnit, a lightweight unit testing and mocking
framework for the Linux kernel.

Unlike Autotest and kselftest, KUnit is a true unit testing framework;
it does not require installing the kernel on a test machine or in a VM
and does not require tests to be written in userspace running on a host
kernel. Additionally, KUnit is fast: From invocation to completion KUnit
can run several dozen tests in under a second. Currently, the entire
KUnit test suite for KUnit runs in under a second from the initial
invocation (build time excluded).

KUnit is heavily inspired by JUnit, Python's unittest.mock, and
Googletest/Googlemock for C++. KUnit provides facilities for defining
unit test cases, grouping related test cases into test suites, providing
common infrastructure for running tests, mocking, spying, and much more.

## What's so special about unit testing?

A unit test is supposed to test a single unit of code in isolation,
hence the name. There should be no dependencies outside the control of
the test; this means no external dependencies, which makes tests orders
of magnitudes faster. Likewise, since there are no external dependencies,
there are no hoops to jump through to run the tests. Additionally, this
makes unit tests deterministic: a failing unit test always indicates a
problem. Finally, because unit tests necessarily have finer granularity,
they are able to test all code paths easily solving the classic problem
of difficulty in exercising error handling code.

## Is KUnit trying to replace other testing frameworks for the kernel?

No. Most existing tests for the Linux kernel are end-to-end tests, which
have their place. A well tested system has lots of unit tests, a
reasonable number of integration tests, and some end-to-end tests. KUnit
is just trying to address the unit test space which is currently not
being addressed.

## More information on KUnit

There is a bunch of documentation near the end of this patch set that
describes how to use KUnit and best practices for writing unit tests.
For convenience I am hosting the compiled docs here:
https://google.github.io/kunit-docs/third_party/kernel/docs/
Additionally for convenience, I have applied these patches to a branch:
https://kunit.googlesource.com/linux/+/kunit/rfc/5.0-rc5/v4
The repo may be cloned with:
git clone https://kunit.googlesource.com/linux
This patchset is on the kunit/rfc/5.0-rc5/v4 branch.

## Changes Since Last Version

 - Got KUnit working on (hypothetically) all architectures (tested on
   x86), as per Rob's (and other's) request
 - Punting all KUnit features/patches depending on UML for now.
 - Broke out UML specific support into arch/um/* as per "[RFC v3 01/19]
   kunit: test: add KUnit test runner core", as requested by Luis.
 - Added support to kunit_tool to allow it to build kernels in external
   directories, as suggested by Kieran.
 - Added a UML defconfig, and a config fragment for KUnit as suggested
   by Kieran and Luis.
 - Cleaned up, and reformatted a bunch of stuff.

-- 
2.21.0.rc0.258.g878e2cd30e-goog

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC v3 17/19] of: unittest: migrate tests to run on KUnit

[Brendan Higgins via dri-devel]

On Wed, Nov 28, 2018 at 12:56 PM Rob Herring  wrote:
>
> On Wed, Nov 28, 2018 at 1:38 PM Brendan Higgins
>  wrote:
> >
> > Migrate tests without any cleanup, or modifying test logic in anyway to
> > run under KUnit using the KUnit expectation and assertion API.
>
> Nice! You beat me to it. This is probably going to conflict with what
> is in the DT tree for 4.21. Also, please Cc the DT list for
> drivers/of/ changes.
>
> Looks good to me, but a few mostly formatting comments below.

I just realized that we never talked about your other comments, and I
still have some questions. (Sorry, it was the last thing I looked at
while getting v4 ready.) No worries if you don't get to it before I
send v4 out, I just didn't want you to think I was ignoring you.

>
> >
> > Signed-off-by: Brendan Higgins 
> > ---
> >  drivers/of/Kconfig|1 +
> >  drivers/of/unittest.c | 1405 ++---
> >  2 files changed, 752 insertions(+), 654 deletions(-)
> >

> > diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
> > index 41b49716ac75f..a5ef44730ffdb 100644
> > --- a/drivers/of/unittest.c
> > +++ b/drivers/of/unittest.c

> > -
> > -static void __init of_unittest_find_node_by_name(void)
> > +static void of_unittest_find_node_by_name(struct kunit *test)
>
> Why do we have to drop __init everywhere? The tests run later?

From the standpoint of a unit test __init doesn't really make any
sense, right? I know that right now we are running as part of a
kernel, but the goal should be that a unit test is not part of a
kernel and we just include what we need.

Even so, that's the future. For now, I did not put the KUnit
infrastructure in the .init section because I didn't think it belonged
there. In practice, KUnit only knows how to run during the init phase
of the kernel, but I don't think it should be restricted there. You
should be able to run tests whenever you want because you should be
able to test anything right? I figured any restriction on that is
misleading and will potentially get in the way at worst, and
unnecessary at best especially since people shouldn't build a
production kernel with all kinds of unit tests inside.

>
> >  {
> > struct device_node *np;
> > const char *options, *name;
> >

> >
> >
> > -   np = of_find_node_by_path("/testcase-data/missing-path");
> > -   unittest(!np, "non-existent path returned node %pOF\n", np);
> > +   KUNIT_EXPECT_EQ_MSG(test,
> > +   
> > of_find_node_by_path("/testcase-data/missing-path"),
> > +   NULL,
> > +   "non-existent path returned node %pOF\n", np);
>
> 1 tab indent would help with less vertical code (in general, not this
> one so much).

Will do.

>
> > of_node_put(np);
> >
> > -   np = of_find_node_by_path("missing-alias");
> > -   unittest(!np, "non-existent alias returned node %pOF\n", np);
> > +   KUNIT_EXPECT_EQ_MSG(test, of_find_node_by_path("missing-alias"), 
> > NULL,
> > +   "non-existent alias returned node %pOF\n", np);
> > of_node_put(np);
> >
> > -   np = of_find_node_by_path("testcase-alias/missing-path");
> > -   unittest(!np, "non-existent alias with relative path returned node 
> > %pOF\n", np);
> > +   KUNIT_EXPECT_EQ_MSG(test,
> > +   
> > of_find_node_by_path("testcase-alias/missing-path"),
> > +   NULL,
> > +   "non-existent alias with relative path returned 
> > node %pOF\n",
> > +   np);
> > of_node_put(np);
> >

> >
> > -static void __init of_unittest_property_string(void)
> > +static void of_unittest_property_string(struct kunit *test)
> >  {
> > const char *strings[4];
> > struct device_node *np;
> > int rc;
> >
> > np = 
> > of_find_node_by_path("/testcase-data/phandle-tests/consumer-a");
> > -   if (!np) {
> > -   pr_err("No testcase data in device tree\n");
> > -   return;
> > -   }
> > -
> > -   rc = of_property_match_string(np, "phandle-list-names", "first");
> > -   unittest(rc == 0, "first expected:0 got:%i\n", rc);
> > -   rc = of_property_match_string(np, "phandle-list-names", "second");
> > -   unittest(rc == 1, "second expected:1 got:%i\n", rc);
> > -   rc = of_property_match_string(np, "phandle-list-names", "third");
> > -   unittest(rc == 2, "third expected:2 got:%i\n", rc);
> > -   rc = of_property_match_string(np, "phandle-list-names", "fourth");
> > -   unittest(rc == -ENODATA, "unmatched string; rc=%i\n", rc);
> > -   rc = of_property_match_string(np, "missing-property", "blah");
> > -   unittest(rc == -EINVAL, "missing property; rc=%i\n", rc);
> > -   rc = of_property_match_string(np, "empty-property", "blah");
> > -   unittest(rc == -ENODATA, "empty property; rc=%i\n", rc);
> > -   rc = 

Re: [RFC v3 14/19] Documentation: kunit: add documentation for KUnit

[Brendan Higgins via dri-devel]

On Wed, Feb 13, 2019 at 1:55 PM Kieran Bingham
 wrote:
>
> Hi Brendan,
>
> On 12/02/2019 22:10, Brendan Higgins wrote:
> > On Mon, Feb 11, 2019 at 4:16 AM Kieran Bingham
> >  wrote:
> >>
> >> Hi Brendan,
> >>
> >> On 09/02/2019 00:56, Brendan Higgins wrote:
> >>> On Thu, Dec 6, 2018 at 4:16 AM Kieran Bingham
> >>>  wrote:
> 
>  Hi Brendan,
> 
>  On 03/12/2018 23:53, Brendan Higgins wrote:
> > On Thu, Nov 29, 2018 at 7:45 PM Luis Chamberlain  
> > wrote:
> >>
> >> On Thu, Nov 29, 2018 at 01:56:37PM +, Kieran Bingham wrote:
> >>> Hi Brendan,
> >>>
> >>> Please excuse the top posting, but I'm replying here as I'm following
> >>> the section "Creating a kunitconfig" in Documentation/kunit/start.rst.
> >>>
> >>> Could the three line kunitconfig file live under say
> >>>arch/um/configs/kunit_defconfig?
> 
> 
>  Further consideration to this topic - I mentioned putting it in
>    arch/um/configs
> 
>  - but I think this is wrong.
> 
>  We now have a location for config-fragments, which is essentially what
>  this is, under kernel/configs
> 
>  So perhaps an addition as :
> 
>   kernel/configs/kunit.config
> 
>  Would be more appropriate - and less (UM) architecture specific.
> >>>
> >>> Sorry for the long radio silence.
> >>>
> >>> I just got around to doing this and I found that there are some
> >>> configs that are desirable to have when running KUnit under x86 in a
> >>> VM, but not UML.
> >>
> >> Should this behaviour you mention be handled by the KCONFIG depends flags?
> >>
> >> depends on (KUMIT & UML)
> >> or
> >> depends on (KUNIT & !UML)
> >>
> >> or such?
> >
> > Not really. Anything that is strictly necessary to run KUnit on an
> > architectures should of course be turned on as a dependency like you
> > suggest, but I am talking about stuff that you would probably want to
> > get yourself going, but is by no means necessary.
> >
> >>
> >> An example of which configs you are referring to would help to
> >> understand the issue perhaps.
> >>
> >
> > For example, you might want to enable a serial console that is known
> > to work with a fairly generic qemu setup when building for x86:
> > CONFIG_SERIAL_8250=y
> > CONFIG_SERIAL_8250_CONSOLE=y
> >
> > Obviously not a dependency, and not even particularly useful to people
> > who know what they are doing, but to someone who is new or just wants
> > something to work out of the box would probably want that.
>
> It sounds like that would be a config fragment for qemu ?
>
> Although - perhaps this is already covered by the following fragment:
>kernel/configs/kvm_guest.config
>

Oh, yep, you are right. Does that mean we should bother at all with a defconfig?

Luis, I know you said you wanted one. I am thinking just stick with
the UML one? The downside there is we then get stuck having to
maintain the fragment and the defconfig. I right now (in the new
revision I am working on) have the Python kunit_tool copy the
defconfig if no kunitconfig is provided and a flag is set. It would be
pretty straightforward to make it merge in the fragment instead.

All that being said, I think I am going to drop the arch/x86
defconfig, since I think we all agree that it is not very useful, but
keep the UML defconfig and the fragment. That will at least given
something concrete to discuss.

>
> >>> So should we have one that goes in with
> >>> config-fragments and others that go into architectures? Another idea,
> >>> it would be nice to have a KUnit config that runs all known tests
> >>
> >> This might also be a config option added to the tests directly like
> >> COMPILE_TEST perhaps?
> >
> > That just allows a bunch of drivers to be compiled, it does not
> > actually go through and turn the configs on, right? I mean, there is
> > no a priori way to know that there is a configuration which spans all
> > possible options available under COMPILE_TEST, right? Maybe I
> > misunderstand what you are suggesting...
>
> Bah - you're right of course. I was mis-remembering the functionality of
> COMPILE_TEST as if it were some sort of 'select' but it's just an enable..
>
> Sorry for the confusion.
>

No problem, I thought for a second that was a good example too (and I
wish it were. It would make my life so much easier!). I remember
getting emails with a COMPILE_TEST config attached that demonstrates
an invalid build caused by my changes, presumably that email bot just
tries random configs with a new change until it finds one that breaks.

>
> >> (Not sure what that would be called though ... KUNIT_RUNTIME_TEST?)
> >>
> >> I think that might be more maintainable as otherwise each new test would
> >> have to modify the {min,def}{config,fragment} ...
> >>
> >
> > Looking at kselftest-merge, they just start out with a set of
> > fragments in which the union should contain all tests and then merge
> > it with a base .config (probably intended to be 

Re: [RFC v3 14/19] Documentation: kunit: add documentation for KUnit

[Brendan Higgins via dri-devel]

On Mon, Feb 11, 2019 at 4:16 AM Kieran Bingham
 wrote:
>
> Hi Brendan,
>
> On 09/02/2019 00:56, Brendan Higgins wrote:
> > On Thu, Dec 6, 2018 at 4:16 AM Kieran Bingham
> >  wrote:
> >>
> >> Hi Brendan,
> >>
> >> On 03/12/2018 23:53, Brendan Higgins wrote:
> >>> On Thu, Nov 29, 2018 at 7:45 PM Luis Chamberlain  
> >>> wrote:
> 
>  On Thu, Nov 29, 2018 at 01:56:37PM +, Kieran Bingham wrote:
> > Hi Brendan,
> >
> > Please excuse the top posting, but I'm replying here as I'm following
> > the section "Creating a kunitconfig" in Documentation/kunit/start.rst.
> >
> > Could the three line kunitconfig file live under say
> >arch/um/configs/kunit_defconfig?
> >>
> >>
> >> Further consideration to this topic - I mentioned putting it in
> >>   arch/um/configs
> >>
> >> - but I think this is wrong.
> >>
> >> We now have a location for config-fragments, which is essentially what
> >> this is, under kernel/configs
> >>
> >> So perhaps an addition as :
> >>
> >>  kernel/configs/kunit.config
> >>
> >> Would be more appropriate - and less (UM) architecture specific.
> >
> > Sorry for the long radio silence.
> >
> > I just got around to doing this and I found that there are some
> > configs that are desirable to have when running KUnit under x86 in a
> > VM, but not UML.
>
> Should this behaviour you mention be handled by the KCONFIG depends flags?
>
> depends on (KUMIT & UML)
> or
> depends on (KUNIT & !UML)
>
> or such?

Not really. Anything that is strictly necessary to run KUnit on an
architectures should of course be turned on as a dependency like you
suggest, but I am talking about stuff that you would probably want to
get yourself going, but is by no means necessary.

>
> An example of which configs you are referring to would help to
> understand the issue perhaps.
>

For example, you might want to enable a serial console that is known
to work with a fairly generic qemu setup when building for x86:
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y

Obviously not a dependency, and not even particularly useful to people
who know what they are doing, but to someone who is new or just wants
something to work out of the box would probably want that.

>
> > So should we have one that goes in with
> > config-fragments and others that go into architectures? Another idea,
> > it would be nice to have a KUnit config that runs all known tests
>
> This might also be a config option added to the tests directly like
> COMPILE_TEST perhaps?

That just allows a bunch of drivers to be compiled, it does not
actually go through and turn the configs on, right? I mean, there is
no a priori way to know that there is a configuration which spans all
possible options available under COMPILE_TEST, right? Maybe I
misunderstand what you are suggesting...

>
> (Not sure what that would be called though ... KUNIT_RUNTIME_TEST?)
>
> I think that might be more maintainable as otherwise each new test would
> have to modify the {min,def}{config,fragment} ...
>

Looking at kselftest-merge, they just start out with a set of
fragments in which the union should contain all tests and then merge
it with a base .config (probably intended to be $(ARCH)_defconfig).
However, I don't know if that is the state of the art.

>
> > (this probably won't work in practice once we start testing mutually
> > exclusive things or things with lots of ifdeffery, but it probably
> > something we should try to maintain as best as we can?); this probably
> > shouldn't go in with the fragments, right?
>
> Sounds like we agree there :)

Totally. Long term we will need something a lot more sophisticated
than anything under discussion here. I was talking about this with
Luis on another thread:
https://groups.google.com/forum/#!topic/kunit-dev/EQ1x0SzrUus (feel
free to chime in!). Nevertheless, that's a really hard problem and I
figure some variant of defconfigs and config fragments will work well
enough until we reach that point.

>
> >
> > I will be sending another revision out soon, but I figured I might be
> > able to catch you before I did so.
>
> Thanks for thinking of me.

How can I forget? You have been super helpful!

> I hope I managed to reply in time to help and not hinder your progress.

Yep, no trouble at all. You are the one helping me :-)

Thanks!
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel