Dear Wiki user, You have subscribed to a wiki page or wiki category on "Devicemap Wiki" for change notification.
The "DataSpec2" page has been changed by rezan: https://wiki.apache.org/devicemap/DataSpec2?action=diff&rev1=41&rev2=42 + = Test Suites = + + Test suites can be configured to test input against a client on a specific domain. + + Test suite attributes: + + Domain:: + :: Type: string + :: Required. + + DomainVersion:: + :: Type: string + :: Required. + + Tests:: + :: Type: a list of tests + :: Required. + + Test attributes: + + Input:: + :: Type: string + :: Required. + + ResultPatternId:: + :: Type: string + :: Required. + + ResultAttributes:: + :: Type: a list of attributes and values + :: Optional. + + When a client runs a test suite, it needs to load the associated domain and then iterate thru each test, + running the input string against the domain and then checking the resulting PatternId and attributes against + the expected results. + + A client must report several metrics on completion of the test suite: + + * Number of tests attempted + * Number of tests failed + * Total runtime in ms, including domain loadtime + * [Optional] Amount of memory required to run the test suite in bytes + + A client passes a test suite if there are no failed tests. + + A reference domain and test suite will be created along with a reference client. + All clients should pass the reference test suite to be considered a valid client. + All domains should provide a rigorous test suite which passes on the reference client to be considered a valid domain. + Any valid clients can run any valid domain. + + + = Format = The pattern and attribute files are JSON objects. These files will contain:
