Hi Stephen, On 2 December 2015 at 15:18, Stephen Warren <swar...@wwwdotorg.org> wrote: > This tool aims to test U-Boot by executing U-Boot shell commands using the > console interface. A single top-level script exists to execute or attach > to the U-Boot console, run the entire script of tests against it, and > summarize the results. Advantages of this approach are: > > - Testing is performed in the same way a user or script would interact > with U-Boot; there can be no disconnect. > - There is no need to write or embed test-related code into U-Boot itself. > It is asserted that writing test-related code in Python is simpler and > more flexible that writing it all in C. > - It is reasonably simple to interact with U-Boot in this way. > > A few simple tests are provided as examples. Soon, we should convert as > many as possible of the other tests in test/* and test/cmd_ut.c too. > > In the future, I hope to publish (out-of-tree) the hook scripts, relay > control utilities, and udev rules I will use for my own HW setup. > > See README.md for more details! > > Signed-off-by: Stephen Warren <swar...@wwwdotorg.org> > Signed-off-by: Stephen Warren <swar...@nvidia.com> > --- > v2: Many fixes and tweaks have been squashed in. Separated out some of' > the tests into separate commits, and added some more tests. > --- > test/py/.gitignore | 1 + > test/py/README.md | 300 > +++++++++++++++++++++++++++++++++++ > test/py/conftest.py | 278 ++++++++++++++++++++++++++++++++ > test/py/multiplexed_log.css | 76 +++++++++ > test/py/multiplexed_log.py | 193 ++++++++++++++++++++++ > test/py/pytest.ini | 9 ++ > test/py/test.py | 24 +++ > test/py/test_000_version.py | 13 ++ > test/py/test_help.py | 6 + > test/py/test_unknown_cmd.py | 8 + > test/py/uboot_console_base.py | 185 +++++++++++++++++++++ > test/py/uboot_console_exec_attach.py | 36 +++++ > test/py/uboot_console_sandbox.py | 31 ++++ > test/py/ubspawn.py | 97 +++++++++++ > 14 files changed, 1257 insertions(+) > create mode 100644 test/py/.gitignore > create mode 100644 test/py/README.md > create mode 100644 test/py/conftest.py > create mode 100644 test/py/multiplexed_log.css > create mode 100644 test/py/multiplexed_log.py > create mode 100644 test/py/pytest.ini > create mode 100755 test/py/test.py > create mode 100644 test/py/test_000_version.py > create mode 100644 test/py/test_help.py > create mode 100644 test/py/test_unknown_cmd.py > create mode 100644 test/py/uboot_console_base.py > create mode 100644 test/py/uboot_console_exec_attach.py > create mode 100644 test/py/uboot_console_sandbox.py > create mode 100644 test/py/ubspawn.py
This is a huge step forward for testing in U-Boot. Congratulations on putting this together! Tested on chromebook_link, sandbox Tested-by: Simon Glass <s...@chromium.org> I've made various comments in the series as I think it needs a little tuning. I'm also interested in how we can arrange for the existing unit tests to be run (and results supported) by this framework. One concern I have is about the ease of running and writing tests. It is pretty easy at present to run a particular driver model test: ./u-boot -d test.dtb -c "ut dm uclass" and we can run this in gdb and figure out where things are going wrong (I do this quite a bit). Somehow we need to preserve this ease of use. The tests should be accessible. I'm not sure how you intend to make that work. > > diff --git a/test/py/.gitignore b/test/py/.gitignore > new file mode 100644 > index 000000000000..0d20b6487c61 > --- /dev/null > +++ b/test/py/.gitignore > @@ -0,0 +1 @@ > +*.pyc > diff --git a/test/py/README.md b/test/py/README.md > new file mode 100644 > index 000000000000..23a403eb8d88 > --- /dev/null > +++ b/test/py/README.md > @@ -0,0 +1,300 @@ > +# U-Boot pytest suite > + > +## Introduction > + > +This tool aims to test U-Boot by executing U-Boot shell commands using the > +console interface. A single top-level script exists to execute or attach to > the > +U-Boot console, run the entire script of tests against it, and summarize the > +results. Advantages of this approach are: > + > +- Testing is performed in the same way a user or script would interact with > + U-Boot; there can be no disconnect. > +- There is no need to write or embed test-related code into U-Boot itself. > + It is asserted that writing test-related code in Python is simpler and more > + flexible that writing it all in C. > +- It is reasonably simple to interact with U-Boot in this way. > + > +## Requirements > + > +The test suite is implemented using pytest. Interaction with the U-Boot > console > +involves executing some binary and interacting with its stdin/stdout. You > will > +need to implement various "hook" scripts that are called by the test suite at > +the appropriate time. > + > +On Debian or Debian-like distributions, the following packages are required. > +Similar package names should exist in other distributions. > + > +| Package | Version tested (Ubuntu 14.04) | > +| -------------- | ----------------------------- | > +| python | 2.7.5-5ubuntu3 | > +| python-pytest | 2.5.1-1 | > + > +The test script supports either: > + > +- Executing a sandbox port of U-Boot on the local machine as a sub-process, > + and interacting with it over stdin/stdout. > +- Executing an external "hook" scripts to flash a U-Boot binary onto a > + physical board, attach to the board's console stream, and reset the board. > + Further details are described later. > + > +### Using `virtualenv` to provide requirements > + > +Older distributions (e.g. Ubuntu 10.04) may not provide all the required > +packages, or may provide versions that are too old to run the test suite. One > +can use the Python `virtualenv` script to locally install more up-to-date > +versions of the required packages without interfering with the OS > installation. > +For example: > + > +```bash > +$ cd /path/to/u-boot > +$ sudo apt-get install python python-virtualenv > +$ virtualenv venv > +$ . ./venv/bin/activate > +$ pip install pytest > +``` > + > +## Testing sandbox > + > +To run the testsuite on the sandbox port (U-Boot built as a native user-space > +application), simply execute: > + > +``` > +./test/py/test.py --bd sandbox --build > +``` > + > +The `--bd` option tells the test suite which board type is being tested. This > +lets the test suite know which features the board has, and hence exactly what > +can be tested. Can we use -b to fit in with buildman and patman? > + > +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may > +omit this option and build U-Boot yourself, in whatever way you choose, > before > +running the test script. > + > +The test script will attach to U-Boot, execute all valid tests for the board, > +then print a summary of the test process. A complete log of the test session > +will be written to `${build_dir}/test-log.html`. This is best viewed in a web > +browser, but may be read directly as plain text, perhaps with the aid of the > +`html2text` utility. > + > +## Command-line options > + > +- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For > + example, `sandbox` or `seaboard`. -b? > +- `--board-identity`, `--id` set the identity of the board to be tested. > + This allows differentiation between multiple instances of the same type of > + physical board that are attached to the same host machine. This parameter > is > + not interpreted by the test script in any way, but rather is simply passed > + to the hook scripts described below, and may be used in any site-specific > + way deemed necessary. > +- `--build` indicates that the test script should compile U-Boot itself > + before running the tests. If using this option, make sure that any > + environment variables required by the build process are already set, such > as > + `$CROSS_COMPILE`. > +- `--build-dir` sets the directory containing the compiled U-Boot binaries. > + If omitted, this is `${source_dir}/build-${board_type}`. -d? > +- `--result-dir` sets the directory to write results, such as log files, > + into. If omitted, the build directory is used. -r? > +- `--persistent-data-dir` sets the directory used to store persistent test > + data. This is test data that may be re-used across test runs, such as file- > + system images. -d? > + > +`pytest` also implements a number of its own command-line options. Please see > +`pytest` documentation for complete details. Execute `py.test --version` for > +a brief summary. Note that U-Boot's test.py script passes all command-line > +arguments directly to `pytest` for processing. > + > +## Testing real hardware > + > +The tools and techniques used to interact with real hardware will vary > +radically between different host and target systems, and the whims of the > user. > +For this reason, the test suite does not attempt to directly interact with > real > +hardware in any way. Rather, it executes a standardized set of "hook" scripts > +via `$PATH`. These scripts implement certain actions on behalf of the test > +suite. This keeps the test suite simple and isolated from system variances > +unrelated to U-Boot features. > + > +### Hook scripts > + > +#### Environment variables > + > +The following environment variables are set when running hook scripts: > + > +- `UBOOT_BOARD_TYPE` the board type being tested. Shouldn't these be U_BOOT_BOARD_TYPE, etc.? > +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was > + specified. > +- `UBOOT_SOURCE_DIR` the U-Boot source directory. > +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. > +- `UBOOT_BUILD_DIR` the U-Boot build directory. > +- `UBOOT_RESULT_DIR` the test result directory. > +- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory. > + > +#### `uboot-test-console` > + > +This script provides access to the U-Boot console. The script's stdin/stdout > +should be connected to the board's console. This process should continue to > run > +indefinitely, until killed. The test suite will run this script in parallel > +with all other hooks. > + > +This script may be implemented e.g. by exec()ing `cu`, `conmux`, etc. > + > +If you are able to run U-Boot under a hardware simulator such as qemu, then > +you would likely spawn that simulator from this script. However, note that > +`uboot-test-reset` may be called multiple times per test script run, and must How aobut u-boot-test-reset, etc.? > +cause U-Boot to start execution from scratch each time. Hopefully your > +simulator includes a virtual reset button! If not, you can launch the > +simulator from `uboot-test-reset` instead, while arranging for this console > +process to always communicate with the current simulator instance. > + > +#### `uboot-test-flash` > + > +Prior to running the test suite against a board, some arrangement must be > made > +so that the board executes the particular U-Boot binary to be tested. Often, > +this involves writing the U-Boot binary to the board's flash ROM. The test > +suite calls this hook script for that purpose. > + > +This script should perform the entire flashing process synchronously; the > +script should only exit once flashing is complete, and a board reset will > +cause the newly flashed U-Boot binary to be executed. > + > +It is conceivable that this script will do nothing. This might be useful in > +the following cases: > + > +- Some other process has already written the desired U-Boot binary into the > + board's flash prior to running the test suite. > +- The board allows U-Boot to be downloaded directly into RAM, and executed > + from there. Use of this feature will reduce wear on the board's flash, so > + may be preferable if available, and if cold boot testing of U-Boot is not > + required. If this feature is used, the `uboot-test-reset` script should > + peform this download, since the board could conceivably be reset multiple > + times in a single test run. > + > +It is up to the user to determine if those situations exist, and to code this > +hook script appropriately. > + > +This script will typically be implemented by calling out to some SoC- or > +board-specific vendor flashing utility. > + > +#### `uboot-test-reset` > + > +Whenever the test suite needs to reset the target board, this script is > +executed. This is guaranteed to happen at least once, prior to executing the > +first test function. If any test fails, the test infra-structure will execute > +this script again to restore U-Boot to an operational state before running > the > +next test function. > + > +This script will likely be implemented by communicating with some form of > +relay or electronic switch attached to the board's reset signal. > + > +The semantics of this script require that when it is executed, U-Boot will > +start running from scratch. If the U-Boot binary to be tested has been > written > +to flash, pulsing the board's reset signal is likely all this script need do. > +However, in some scenarios, this script may perform other actions. For > +example, it may call out to some SoC- or board-specific vendor utility in > order > +to download the U-Boot binary directly into RAM and execute it. This would > +avoid the need for `uboot-test-flash` to actually write U-Boot to flash, thus > +saving wear on the flash chip(s). > + > +### Board-type-specific configuration > + > +Each board has a different configuration and behaviour. Many of these > +differences can be automatically detected by parsing the `.config` file in > the > +build directory. However, some differences can't yet be handled > automatically. > + > +For each board, an optional Python module `uboot_board_${board_type}` may > exist > +to provide board-specific information to the test script. Any global value > +defined in these modules is available for use by any test function. The data > +contained in these scripts must be purely derived from U-Boot source code. > +Hence, these configuration files are part of the U-Boot source tree too. > + > +### Execution environment configuration > + > +Each user's hardware setup may enable testing different subsets of the > features > +implemented by a particular board's configuration of U-Boot. For example, a > +U-Boot configuration may support USB device mode and USB Mass Storage, but > this > +can only be tested if a USB cable is connected between the board and the host > +machine running the test script. > + > +For each board, optional Python modules `uboot_boardenv_${board_type}` and > +`uboot_boardenv_${board_type}_${board_identity}` may exist to provide > +board-specific and board-identity-specific information to the test script. > Any > +global value defined in these modules is available for use by any test > +function. The data contained in these is specific to a particular user's > +hardware configuration. Hence, these configuration files are not part of the > +U-Boot source tree, and should be installed outside of the source tree. Users > +should set `$PYTHONPATH` prior to running the test script to allow these > +modules to be loaded. > + > +### Board module parameter usage > + > +The test scripts rely on the following variables being defined by the board > +module: > + > +- None at present. > + > +### U-Boot `.config` feature usage > + > +The test scripts rely on various U-Boot `.config` features, either directly > in > +order to test those features, or indirectly in order to query information > from > +the running U-Boot instance in order to test other features. > + > +One example is that testing of the `md` command requires knowledge of a RAM > +address to use for the test. This data is parsed from the output of the > +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. > + > +For a complete list of dependencies, please search the test scripts for > +instances of: > + > +- `buildconfig.get(...` > +- `@pytest.mark.buildconfigspec(...` > + > +### Complete invocation example > + > +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and > +any required environment configuration Python modules into $HOME/ubtest/py, > +then you would likely invoke the test script as follows: > + > +If U-Boot has already been built: > + > +```bash > +PATH=$HOME/ubtest/bin:$PATH \ > + PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ > + ./test/py/test.py --bd seaboard > +``` > + > +If you want the test script to compile U-Boot for you too, then you likely > +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as > +follow: > + > +```bash > +CROSS_COMPILE=arm-none-eabi- \ > + PATH=$HOME/ubtest/bin:$PATH \ > + PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ > + ./test/py/test.py --bd seaboard --build > +``` > + > +## Writing tests > + > +Please refer to the pytest documentation for details of writing pytest tests. > +Details specific to the U-Boot test suite are described below. > + > +A test fixture named `uboot_console` should be used by each test function. > This > +provides the means to interact with the U-Boot console, and retrieve board > and > +environment configuration information. > + > +The function `uboot_console.run_command()` executes a shell command on the > +U-Boot console, and returns all output from that command. This allows > +validation or interpretation of the command output. This function validates > +that certain strings are not seen on the U-Boot console. These include shell > +error messages and the U-Boot sign-on message (in order to detect unexpected > +board resets). See the source of `uboot_console_base.py` for a complete list > of > +"bad" strings. Some test scenarios are expected to trigger these strings. Use > +`uboot_console.disable_check()` to temporarily disable checking for specific > +strings. See `test_unknown_cmd.py` for an example. > + > +Board- and board-environment configuration values may be accessed as > sub-fields > +of the `uboot_console.config` object, for example > +`uboot_console.config.ram_base`. > + > +Build configuration values (from `.config`) may be accessed via the > dictionary > +`uboot_console.config.buildconfig`, with keys equal to the Kconfig variable > +names. > diff --git a/test/py/conftest.py b/test/py/conftest.py > new file mode 100644 > index 000000000000..b6efe03a60f8 > --- /dev/null > +++ b/test/py/conftest.py > @@ -0,0 +1,278 @@ > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +import atexit > +import errno > +import os > +import os.path > +import pexpect > +import pytest > +from _pytest.runner import runtestprotocol > +import ConfigParser > +import StringIO > +import sys > + > +log = None > +console = None > + > +def mkdir_p(path): > + try: > + os.makedirs(path) > + except OSError as exc: > + if exc.errno == errno.EEXIST and os.path.isdir(path): > + pass > + else: > + raise > + > +def pytest_addoption(parser): > + parser.addoption("--build-dir", default=None, > + help="U-Boot build directory (O=)") You seem to use double quote consistently throughout rather than a single quote. That is different from the existing Python in the U-Boot tree. It might be worth swapping it out for consistency. > + parser.addoption("--result-dir", default=None, > + help="U-Boot test result/tmp directory") > + parser.addoption("--persistent-data-dir", default=None, > + help="U-Boot test persistent generated data directory") > + parser.addoption("--board-type", "--bd", "-B", default="sandbox", > + help="U-Boot board type") > + parser.addoption("--board-identity", "--id", default="na", > + help="U-Boot board identity/instance") > + parser.addoption("--build", default=False, action="store_true", > + help="Compile U-Boot before running tests") > + > +def pytest_configure(config): This series should have function comments throughout on non-trivial functions - e.g. purpose of the function and a description of the parameters and return value. > + global log > + global console > + global ubconfig > + > + test_py_dir = os.path.dirname(os.path.abspath(__file__)) > + source_dir = os.path.dirname(os.path.dirname(test_py_dir)) > + > + board_type = config.getoption("board_type") > + board_type_fn = board_type.replace("-", "_") > + > + board_identity = config.getoption("board_identity") > + board_identity_fn = board_identity.replace("-", "_") > + > + build_dir = config.getoption("build_dir") > + if not build_dir: > + build_dir = source_dir + "/build-" + board_type > + mkdir_p(build_dir) > + > + result_dir = config.getoption("result_dir") > + if not result_dir: > + result_dir = build_dir > + mkdir_p(result_dir) > + > + persistent_data_dir = config.getoption("persistent_data_dir") > + if not persistent_data_dir: > + persistent_data_dir = build_dir + "/persistent-data" > + mkdir_p(persistent_data_dir) > + > + import multiplexed_log > + log = multiplexed_log.Logfile(result_dir + "/test-log.html") > + > + if config.getoption("build"): > + if build_dir != source_dir: > + o_opt = "O=%s" % build_dir > + else: > + o_opt = "" > + cmds = ( > + ["make", o_opt, "-s", board_type + "_defconfig"], > + ["make", o_opt, "-s", "-j8"], > + ) > + runner = log.get_runner("make", sys.stdout) > + for cmd in cmds: > + runner.run(cmd, cwd=source_dir) > + runner.close() > + > + class ArbitraryAttrContainer(object): > + pass > + > + ubconfig = ArbitraryAttrContainer() > + ubconfig.brd = dict() > + ubconfig.env = dict() > + > + modules = [ > + (ubconfig.brd, "uboot_board_" + board_type_fn), > + (ubconfig.env, "uboot_boardenv_" + board_type_fn), > + (ubconfig.env, "uboot_boardenv_" + board_type_fn + "_" + > + board_identity_fn), > + ] > + for (sub_config, mod_name) in modules: > + try: > + mod = __import__(mod_name) > + except ImportError: > + continue > + sub_config.update(mod.__dict__) > + > + ubconfig.buildconfig = dict() > + > + for conf_file in (".config", "include/autoconf.mk"): > + dot_config = build_dir + "/" + conf_file > + if not os.path.exists(dot_config): > + raise Exception(conf_file + " does not exist; " + > + "try passing --build option?") > + > + with open(dot_config, "rt") as f: > + ini_str = "[root]\n" + f.read() > + ini_sio = StringIO.StringIO(ini_str) > + parser = ConfigParser.RawConfigParser() > + parser.readfp(ini_sio) > + ubconfig.buildconfig.update(parser.items("root")) > + > + ubconfig.test_py_dir = test_py_dir > + ubconfig.source_dir = source_dir > + ubconfig.build_dir = build_dir > + ubconfig.result_dir = result_dir > + ubconfig.persistent_data_dir = persistent_data_dir > + ubconfig.board_type = board_type > + ubconfig.board_identity = board_identity > + > + env_vars = ( > + "board_type", > + "board_identity", > + "source_dir", > + "test_py_dir", > + "build_dir", > + "result_dir", > + "persistent_data_dir", > + ) > + for v in env_vars: > + os.environ["UBOOT_" + v.upper()] = getattr(ubconfig, v) > + > + if board_type == "sandbox": > + import uboot_console_sandbox > + console = uboot_console_sandbox.ConsoleSandbox(log, ubconfig) > + else: > + import uboot_console_exec_attach > + console = uboot_console_exec_attach.ConsoleExecAttach(log, ubconfig) > + > +def pytest_generate_tests(metafunc): > + subconfigs = { > + "brd": console.config.brd, > + "env": console.config.env, > + } > + for fn in metafunc.fixturenames: > + parts = fn.split("__") > + if len(parts) < 2: > + continue > + if parts[0] not in subconfigs: > + continue > + subconfig = subconfigs[parts[0]] > + vals = [] > + val = subconfig.get(fn, []) > + if val: > + vals = (val, ) > + else: > + vals = subconfig.get(fn + "s", []) > + metafunc.parametrize(fn, vals) > + > +@pytest.fixture(scope="session") > +def uboot_console(request): > + return console > + > +tests_not_run = set() > +tests_failed = set() > +tests_skipped = set() > +tests_passed = set() > + > +def pytest_itemcollected(item): > + tests_not_run.add(item.name) > + > +def cleanup(): > + if console: > + console.close() > + if log: > + log.status_pass("%d passed" % len(tests_passed)) > + if tests_skipped: > + log.status_skipped("%d skipped" % len(tests_skipped)) > + for test in tests_skipped: > + log.status_skipped("... " + test) > + if tests_failed: > + log.status_fail("%d failed" % len(tests_failed)) > + for test in tests_failed: > + log.status_fail("... " + test) > + if tests_not_run: > + log.status_fail("%d not run" % len(tests_not_run)) > + for test in tests_not_run: > + log.status_fail("... " + test) > + log.close() > +atexit.register(cleanup) > + > +def setup_boardspec(item): > + mark = item.get_marker("boardspec") > + if not mark: > + return > + required_boards = [] > + for board in mark.args: > + if board.startswith("!"): > + if ubconfig.board_type == board[1:]: > + pytest.skip("board not supported") > + return > + else: > + required_boards.append(board) > + if required_boards and ubconfig.board_type not in required_boards: > + pytest.skip("board not supported") > + > +def setup_buildconfigspec(item): > + mark = item.get_marker("buildconfigspec") > + if not mark: > + return > + for option in mark.args: > + if not ubconfig.buildconfig.get("config_" + option.lower(), None): > + pytest.skip(".config feature not enabled") > + > +def pytest_runtest_setup(item): > + log.start_section(item.name) > + setup_boardspec(item) > + setup_buildconfigspec(item) > + > +def pytest_runtest_protocol(item, nextitem): > + reports = runtestprotocol(item, nextitem=nextitem) > + failed = None > + skipped = None > + for report in reports: > + if report.outcome == "failed": > + failed = report > + break > + if report.outcome == "skipped": > + if not skipped: > + skipped = report > + > + if failed: > + tests_failed.add(item.name) > + elif skipped: > + tests_skipped.add(item.name) > + else: > + tests_passed.add(item.name) > + tests_not_run.remove(item.name) > + > + try: > + if failed: > + msg = "FAILED:\n" + str(failed.longrepr) > + log.status_fail(msg) > + elif skipped: > + msg = "SKIPPED:\n" + str(skipped.longrepr) > + log.status_skipped(msg) > + else: > + log.status_pass("OK") > + except: > + # If something went wrong with logging, it's better to let the test > + # process continue, which may report other exceptions that triggered > + # the logging issue (e.g. console.log wasn't created). Hence, just > + # squash the exception. If the test setup failed due to e.g. syntax > + # error somewhere else, this won't be seen. However, once that issue > + # is fixed, if this exception still exists, it will then be logged as > + # part of the test's stdout. > + import traceback > + print "Exception occurred while logging runtest status:" > + traceback.print_exc() > + # FIXME: Can we force a test failure here? > + > + log.end_section(item.name) > + > + if failed: > + console.cleanup_spawn() > + > + return reports > diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css > new file mode 100644 > index 000000000000..96d87ebe034b > --- /dev/null > +++ b/test/py/multiplexed_log.css > @@ -0,0 +1,76 @@ > +/* > + * Copyright (c) 2015 Stephen Warren > + * > + * SPDX-License-Identifier: GPL-2.0 > + */ > + > +body { > + background-color: black; > + color: #ffffff; > +} > + > +.implicit { > + color: #808080; > +} > + > +.section { > + border-style: solid; > + border-color: #303030; > + border-width: 0px 0px 0px 5px; > + padding-left: 5px > +} > + > +.section-header { > + background-color: #303030; > + margin-left: -5px; > + margin-top: 5px; > +} > + > +.section-trailer { > + display: none; > +} > + > +.stream { > + border-style: solid; > + border-color: #303030; > + border-width: 0px 0px 0px 5px; > + padding-left: 5px > +} > + > +.stream-header { > + background-color: #303030; > + margin-left: -5px; > + margin-top: 5px; > +} > + > +.stream-trailer { > + display: none; > +} > + > +.error { > + color: #ff0000 > +} > + > +.warning { > + color: #ffff00 > +} > + > +.info { > + color: #808080 > +} > + > +.action { > + color: #8080ff > +} > + > +.status-pass { > + color: #00ff00 > +} > + > +.status-skipped { > + color: #ffff00 > +} > + > +.status-fail { > + color: #ff0000 > +} > diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py > new file mode 100644 > index 000000000000..58b9a9c50ecf > --- /dev/null > +++ b/test/py/multiplexed_log.py > @@ -0,0 +1,193 @@ > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +import cgi > +import os.path > +import shutil > +import subprocess > + > +mod_dir = os.path.dirname(os.path.abspath(__file__)) > + > +class LogfileStream(object): > + def __init__(self, logfile, name, chained_file): > + self.logfile = logfile > + self.name = name > + self.chained_file = chained_file > + > + def close(self): > + pass > + > + def write(self, data, implicit=False): > + self.logfile.write(self, data, implicit) > + if self.chained_file: > + self.chained_file.write(data) > + > + def flush(self): > + self.logfile.flush() > + if self.chained_file: > + self.chained_file.flush() > + > +class RunAndLog(object): > + def __init__(self, logfile, name, chained_file): > + self.logfile = logfile > + self.name = name > + self.chained_file = chained_file > + > + def close(self): > + pass > + > + def run(self, cmd, cwd=None): > + msg = "+" + " ".join(cmd) + "\n" > + if self.chained_file: > + self.chained_file.write(msg) > + self.logfile.write(self, msg) > + > + try: > + p = subprocess.Popen(cmd, cwd=cwd, > + stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) > + (output, stderr) = p.communicate() > + status = p.returncode > + except subprocess.CalledProcessError as cpe: > + output = cpe.output > + status = cpe.returncode > + self.logfile.write(self, output) > + if status: > + if self.chained_file: > + self.chained_file.write(output) > + raise Exception("command failed; exit code " + str(status)) > + > +class SectionCtxMgr(object): > + def __init__(self, log, marker): > + self.log = log > + self.marker = marker > + > + def __enter__(self): > + self.log.start_section(self.marker) > + > + def __exit__(self, extype, value, traceback): > + self.log.end_section(self.marker) > + > +class Logfile(object): > + def __init__(self, fn): > + self.f = open(fn, "wt") > + self.last_stream = None > + self.linebreak = True > + self.blocks = [] > + self.cur_evt = 1 > + shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn)) > + self.f.write("""\ > +<html> > +<head> > +<link rel="stylesheet" type="text/css" href="multiplexed_log.css"> > +</head> > +<body> > +<tt> > +""") > + > + def close(self): > + self.f.write("""\ > +</tt> > +</body> > +</html> > +""") > + self.f.close() > + > + def _escape(self, data): > + data = data.replace(chr(13), "") > + data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or > + c for c in data) > + data = cgi.escape(data) > + data = data.replace(" ", " ") > + self.linebreak = data[-1:-1] == "\n" > + data = data.replace(chr(10), "<br/>\n") > + return data > + > + def _terminate_stream(self): > + self.cur_evt += 1 > + if not self.last_stream: > + return > + if not self.linebreak: > + self.f.write("<br/>\n") > + self.f.write("<div class=\"stream-trailer\" id=\"" + > + self.last_stream.name + "\">End stream: " + > + self.last_stream.name + "</div>\n") > + self.f.write("</div>\n") > + self.last_stream = None > + > + def _note(self, note_type, msg): > + self._terminate_stream() > + self.f.write("<div class=\"" + note_type + "\">\n") > + self.f.write(self._escape(msg)) > + self.f.write("<br/>\n") > + self.f.write("</div>\n") > + self.linebreak = True > + > + def start_section(self, marker): > + self._terminate_stream() > + self.blocks.append(marker) > + blk_path = "/".join(self.blocks) > + self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n") > + self.f.write("<div class=\"section-header\" id=\"" + blk_path + > + "\">Section: " + blk_path + "</div>\n") > + > + def end_section(self, marker): > + if (not self.blocks) or (marker != self.blocks[-1]): > + raise Exception("Block nesting mismatch: \"%s\" \"%s\"" % > + (marker, "/".join(self.blocks))) > + self._terminate_stream() > + blk_path = "/".join(self.blocks) > + self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" + > + blk_path + "\">End section: " + blk_path + "</div>\n") > + self.f.write("</div>\n") > + self.blocks.pop() > + > + def section(self, marker): > + return SectionCtxMgr(self, marker) > + > + def error(self, msg): > + self._note("error", msg) > + > + def warning(self, msg): > + self._note("warning", msg) > + > + def info(self, msg): > + self._note("info", msg) > + > + def action(self, msg): > + self._note("action", msg) > + > + def status_pass(self, msg): > + self._note("status-pass", msg) > + > + def status_skipped(self, msg): > + self._note("status-skipped", msg) > + > + def status_fail(self, msg): > + self._note("status-fail", msg) > + > + def get_stream(self, name, chained_file=None): > + return LogfileStream(self, name, chained_file) > + > + def get_runner(self, name, chained_file=None): > + return RunAndLog(self, name, chained_file) > + > + _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c != 10) + > + "".join(chr(c) for c in range(127, 256))) > + > + def write(self, stream, data, implicit=False): > + if stream != self.last_stream: > + self._terminate_stream() > + self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name) > + self.f.write("<div class=\"stream-header\" id=\"" + stream.name + > + "\">Stream: " + stream.name + "</div>\n") > + if implicit: > + self.f.write("<span class=\"implicit\">") > + self.f.write(self._escape(data)) > + if implicit: > + self.f.write("</span>") > + self.last_stream = stream > + > + def flush(self): > + self.f.flush() > diff --git a/test/py/pytest.ini b/test/py/pytest.ini > new file mode 100644 > index 000000000000..1bdff810d36e > --- /dev/null > +++ b/test/py/pytest.ini > @@ -0,0 +1,9 @@ > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +[pytest] > +markers = > + boardspec: U-Boot: Describes the set of boards a test can/can't run on. > + buildconfigspec: U-Boot: Describes Kconfig/config-header constraints. > diff --git a/test/py/test.py b/test/py/test.py > new file mode 100755 > index 000000000000..7768216a2335 > --- /dev/null > +++ b/test/py/test.py > @@ -0,0 +1,24 @@ > +#!/usr/bin/env python > + > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +import os > +import os.path > +import sys > + > +sys.argv.pop(0) > + > +args = ["py.test", os.path.dirname(__file__)] > +args.extend(sys.argv) > + > +try: > + os.execvp("py.test", args) > +except: > + import traceback > + traceback.print_exc() > + print >>sys.stderr, """ > +exec(py.test) failed; perhaps you are missing some dependencies? > +See test/md/README.md for the list.""" > diff --git a/test/py/test_000_version.py b/test/py/test_000_version.py > new file mode 100644 > index 000000000000..360c8fd726e0 > --- /dev/null > +++ b/test/py/test_000_version.py > @@ -0,0 +1,13 @@ > +# Copyright (c) 2015 Stephen Warren > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +# pytest runs tests the order of their module path, which is related to the > +# filename containing the test. This file is named such that it is sorted > +# first, simply as a very basic sanity check of the functionality of the > U-Boot > +# command prompt. > + > +def test_version(uboot_console): > + with uboot_console.disable_check("main_signon"): > + response = uboot_console.run_command("version") > + uboot_console.validate_main_signon_in_text(response) > diff --git a/test/py/test_help.py b/test/py/test_help.py > new file mode 100644 > index 000000000000..3cc896ee7af8 > --- /dev/null > +++ b/test/py/test_help.py > @@ -0,0 +1,6 @@ > +# Copyright (c) 2015 Stephen Warren > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +def test_help(uboot_console): > + uboot_console.run_command("help") > diff --git a/test/py/test_unknown_cmd.py b/test/py/test_unknown_cmd.py > new file mode 100644 > index 000000000000..ba12de56a294 > --- /dev/null > +++ b/test/py/test_unknown_cmd.py > @@ -0,0 +1,8 @@ > +# Copyright (c) 2015 Stephen Warren > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +def test_unknown_command(uboot_console): > + with uboot_console.disable_check("unknown_command"): > + response = uboot_console.run_command("non_existent_cmd") > + assert("Unknown command 'non_existent_cmd' - try 'help'" in response) > diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py > new file mode 100644 > index 000000000000..9f13fead2e7e > --- /dev/null > +++ b/test/py/uboot_console_base.py > @@ -0,0 +1,185 @@ > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 > + > +import multiplexed_log > +import os > +import pytest > +import re > +import sys > + > +pattern_uboot_spl_signon = re.compile("(U-Boot SPL > \\d{4}\\.\\d{2}-[^\r\n]*)") > +pattern_uboot_main_signon = re.compile("(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)") > +pattern_stop_autoboot_prompt = re.compile("Hit any key to stop autoboot: ") > +pattern_unknown_command = re.compile("Unknown command '.*' - try 'help'") > +pattern_error_notification = re.compile("## Error: ") > + > +class ConsoleDisableCheck(object): > + def __init__(self, console, check_type): > + self.console = console > + self.check_type = check_type > + > + def __enter__(self): > + self.console.disable_check_count[self.check_type] += 1 > + > + def __exit__(self, extype, value, traceback): > + self.console.disable_check_count[self.check_type] -= 1 > + > +class ConsoleBase(object): > + def __init__(self, log, config, max_fifo_fill): > + self.log = log > + self.config = config > + self.max_fifo_fill = max_fifo_fill > + > + self.logstream = self.log.get_stream("console", sys.stdout) > + > + # Array slice removes leading/trailing quotes > + self.prompt = self.config.buildconfig["config_sys_prompt"][1:-1] > + self.prompt_escaped = re.escape(self.prompt) > + self.p = None > + self.disable_check_count = { > + "spl_signon": 0, > + "main_signon": 0, > + "unknown_command": 0, > + "error_notification": 0, > + } > + > + self.at_prompt = False > + self.at_prompt_logevt = None > + self.ram_base = None > + > + def close(self): > + if self.p: > + self.p.close() > + self.logstream.close() > + > + def run_command(self, cmd, wait_for_echo=True, send_nl=True, > wait_for_prompt=True): > + self.ensure_spawned() > + > + if self.at_prompt and \ > + self.at_prompt_logevt != self.logstream.logfile.cur_evt: > + self.logstream.write(self.prompt, implicit=True) > + > + bad_patterns = [] > + bad_pattern_ids = [] > + if (self.disable_check_count["spl_signon"] == 0 and > + self.uboot_spl_signon): > + bad_patterns.append(self.uboot_spl_signon_escaped) > + bad_pattern_ids.append("SPL signon") > + if self.disable_check_count["main_signon"] == 0: > + bad_patterns.append(self.uboot_main_signon_escaped) > + bad_pattern_ids.append("U-Boot main signon") > + if self.disable_check_count["unknown_command"] == 0: > + bad_patterns.append(pattern_unknown_command) > + bad_pattern_ids.append("Unknown command") > + if self.disable_check_count["error_notification"] == 0: > + bad_patterns.append(pattern_error_notification) > + bad_pattern_ids.append("Error notification") > + try: > + self.at_prompt = False > + if send_nl: > + cmd += "\n" > + while cmd: > + # Limit max outstanding data, so UART FIFOs don't overflow > + chunk = cmd[:self.max_fifo_fill] > + cmd = cmd[self.max_fifo_fill:] > + self.p.send(chunk) > + if not wait_for_echo: > + continue > + chunk = re.escape(chunk) > + chunk = chunk.replace("\\\n", "[\r\n]") > + m = self.p.expect([chunk] + bad_patterns) > + if m != 0: > + self.at_prompt = False > + raise Exception("Bad pattern found on console: " + > + bad_pattern_ids[m - 1]) > + if not wait_for_prompt: > + return > + m = self.p.expect([self.prompt_escaped] + bad_patterns) > + if m != 0: > + self.at_prompt = False > + raise Exception("Bad pattern found on console: " + > + bad_pattern_ids[m - 1]) > + self.at_prompt = True > + self.at_prompt_logevt = self.logstream.logfile.cur_evt > + # Only strip \r\n; space/TAB might be significant if testing > + # indentation. > + return self.p.before.strip("\r\n") > + except Exception as ex: > + self.log.error(str(ex)) > + self.cleanup_spawn() > + raise > + > + def ctrlc(self): > + self.run_command(chr(3), wait_for_echo=False, send_nl=False) > + > + def ensure_spawned(self): > + if self.p: > + return > + try: > + self.at_prompt = False > + self.log.action("Starting U-Boot") > + self.p = self.get_spawn() > + # Real targets can take a long time to scroll large amounts of > + # text if LCD is enabled. This value may need tweaking in the > + # future, possibly per-test to be optimal. This works for "help" > + # on board "seaboard". > + self.p.timeout = 30000 > + self.p.logfile_read = self.logstream Also I have found that tests fail on chromebook_link because it cannot keep up with the pace of keyboard input. I'm not sure what the solution is - maybe the best thing is to implement buffering in the serial uclass, assuming that fixes it. For now I disabled LCD output. I think it would be worth adding a test that checks for the banner and the prompt, so we know that other test failures are not due to this problem. > + if self.config.buildconfig.get("CONFIG_SPL", False) == "y": > + self.p.expect([pattern_uboot_spl_signon]) > + self.uboot_spl_signon = self.p.after > + self.uboot_spl_signon_escaped = re.escape(self.p.after) > + else: > + self.uboot_spl_signon = None > + self.p.expect([pattern_uboot_main_signon]) > + self.uboot_main_signon = self.p.after > + self.uboot_main_signon_escaped = re.escape(self.p.after) > + while True: > + match = self.p.expect([self.prompt_escaped, > + pattern_stop_autoboot_prompt]) > + if match == 1: > + self.p.send(chr(3)) # CTRL-C > + continue > + break > + self.at_prompt = True > + self.at_prompt_logevt = self.logstream.logfile.cur_evt > + except Exception as ex: > + self.log.error(str(ex)) > + self.cleanup_spawn() > + raise > + > + def cleanup_spawn(self): > + try: > + if self.p: > + self.p.close() > + except: > + pass > + self.p = None > + > + def validate_main_signon_in_text(self, text): > + assert(self.uboot_main_signon in text) > + > + def disable_check(self, check_type): > + return ConsoleDisableCheck(self, check_type) > + > + def find_ram_base(self): > + if self.config.buildconfig.get("config_cmd_bdi", "n") != "y": > + pytest.skip("bdinfo command not supported") > + if self.ram_base == -1: > + pytest.skip("Previously failed to find RAM bank start") > + if self.ram_base is not None: > + return self.ram_base > + > + with self.log.section("find_ram_base"): > + response = self.run_command("bdinfo") > + for l in response.split("\n"): > + if "-> start" in l: > + self.ram_base = int(l.split("=")[1].strip(), 16) > + break > + if self.ram_base is None: > + self.ram_base = -1 > + raise Exception("Failed to find RAM bank start in `bdinfo`") > + > + return self.ram_base > diff --git a/test/py/uboot_console_exec_attach.py > b/test/py/uboot_console_exec_attach.py > new file mode 100644 > index 000000000000..0267ae4dc070 > --- /dev/null > +++ b/test/py/uboot_console_exec_attach.py > @@ -0,0 +1,36 @@ > +# Copyright (c) 2015 Stephen Warren > +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. > +# > +# SPDX-License-Identifier: GPL-2.0 It would be useful to have a short description at the top of each file / class explaining what it is for. > + > +from ubspawn import Spawn > +from uboot_console_base import ConsoleBase > + > +def cmdline(app, args): > + return app + ' "' + '" "'.join(args) + '"' > + > +class ConsoleExecAttach(ConsoleBase): > + def __init__(self, log, config): > + # The max_fifo_fill value might need tweaking per-board/-SoC? > + # 1 would be safe anywhere, but is very slow (a pexpect issue?). > + # 16 is a common FIFO size. > + # HW flow control would mean this could be infinite. > + super(ConsoleExecAttach, self).__init__(log, config, > max_fifo_fill=16) > + > + self.log.action("Flashing U-Boot") > + cmd = ["uboot-test-flash", config.board_type, config.board_identity] > + runner = self.log.get_runner(cmd[0]) > + runner.run(cmd) > + runner.close() > + [snip] Regards, Simon _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot