On 5/20/21 9:53 PM, Willian Rampazzo wrote: > Conceptually speaking, acceptance tests "are a series of specific tests > conducted by the customer in an attempt to uncover product errors before > accepting the software from the developer. Conducted by the end-user rather > than software engineers, acceptance testing can range from an informal > “test drive” to a planned and systematically executed series of scripted > tests" [1]. Every time Pressman refers to the term "acceptance testing," he > also refers to user's agreement in the final state of an implemented feature. > Today, QEMU is not implementing user acceptance tests as described by > Pressman. > > There are other three possible terms we could use to describe what is > currently > QEMU "acceptance" tests: > > 1 - Integration tests: > - "Integration testing is a systematic technique for constructing the > software architecture while at the same time conducting tests to > uncover errors associated with interfacing. The objective is to take > unit-tested components and build a program structure that has been > dictated by design." [2] > * Note: Sommerville does not have a clear definition of integration > testing. He refers to incremental integration of components inside > the system testing (see [3]). > > 2 - Validation tests: > - "Validation testing begins at the culmination of integration testing, > when individual components have been exercised, the software is > completely assembled as a package, and interfacing errors have been > uncovered and corrected. At the validation or system level, the > distinction between different software categories disappears. Testing > focuses on user-visible actions and user-recognizable output from the > system." [4] > - "where you expect the system to perform correctly using a set of test > cases that reflect the system’s expected use." [5] > * Note: the definition of "validation testing" from Sommerville reflects > the same definition found around the Internet, as one of the processes > inside the "Verification & Validation (V&V)." In this concept, > validation testing is a high-level definition that covers unit > testing, > functional testing, integration testing, system testing, and > acceptance > testing. > > 3 - System tests: > - "verifies that all elements mesh properly and that overall system > function and performance is achieved." [6] > - "involves integrating components to create a version of the system and > then testing the integrated system. System testing checks that > components are compatible, interact correctly, and transfer the right > data at the right time across their interfaces." [7] > > The tests implemented inside the QEMU "acceptance" directory depend on the > software completely assembled and, sometimes, on other elements, like > operating > system images. In this case, the proposal here is to rename the current > "acceptance" directory to "system."
Are user-mode tests using Avocado also system tests? https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html > [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. p. 430. > [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 398. > [3] Sommerville, Ian (2016). Software Engineering. p. 240-242. > [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 407. > [5] Sommerville, Ian (2016). Software Engineering. p. 227. > [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 377. > [7] Sommerville, Ian (2016). Software Engineering. p. 240. > > Signed-off-by: Willian Rampazzo <willi...@redhat.com> > --- > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index f718b61fa7..c5de3c9fd5 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -52,7 +52,7 @@ include: > # Avoid recompiling by hiding ninja with NINJA=":" > - make NINJA=":" $MAKE_CHECK_ARGS > > -.acceptance_template: &acceptance_definition > +.system_template: &system_definition .system_test_template: &system_test_definition ? > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 8f220e15d1..c580292bb5 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),) > @echo " $(MAKE) check-tcg Run TCG tests" > @echo " $(MAKE) check-softfloat Run FPU emulation tests" > endif > - @echo " $(MAKE) check-acceptance Run all acceptance (functional) > tests" > + @echo " $(MAKE) check-system Run all system tests"