> -----Original Message----- > From: Dylan Baker [mailto:[email protected]] > Sent: Friday, December 12, 2014 6:04 PM > To: [email protected] > Cc: Mason, Michael W; Chad Versace > Subject: Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 > tests (v3) > > On Saturday, December 13, 2014 12:53:42 AM Mason, Michael W wrote: > > A couple questions... > > > > First, deqp-gles3 has many command line options. Is there any way in > > piglit to pass them to deqp-gles3 from the piglit > command line? I don't see a way. One option in particular is important when > testing on Freon. I have to pass "--deqp-gl- > config-name=rgba8888d24s8" to get the images to display full screen. No > doubt we'll want to use other options in the > future. > > There is no way currently. It wouldn't be that hard to implement the feature > however. The other option is to do what we > do for several of the integrated suites and use a mixture of environment > variables and piglit.conf settings, depending on > how often you want to change those settings.
Using the env variable/piglit.conf method to pass options to deqp-gles3 works great. See my suggestions below. > > > > > Second, how do you kill a piglit test run once it's started? I tried > > multiple ctrl-C's without success. I couldn't get it to stop. > Is this a piglit limitation or something to do with the piglit/deqp combo? > > > > BTW, the latest patch does work, but I'd like to understand if the above > > limitations are expected or not before giving it my > blessing. > > As Matt pointed out ctrl+\ will kill all tests. ctrl+c will kill a single > running test. > > > > Mike > > > > -----Original Message----- > > From: Chad Versace [mailto:[email protected]] > > Sent: Wednesday, December 10, 2014 9:39 AM > > To: [email protected] > > Cc: Chad Versace; Baker, DylanX C; Mason, Michael W > > Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests > > (v3) > > > > Google opensourced the drawElements Quality Product Testsuite (dEQP) as > > part of the Android Lollipop release. It's git > repo lives in the Android tree at > [https://android.googlesource.com/platform/external/deqp/]. > > > > This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 > > tests. It queries the 'deqp-gles3' executable at > runtime for the list of testcases. The module attempts queries the > 'deqp-gles3' executable only if the environment > variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is > set. > > > > Why do we need to use Pigit as the testrunner for dEQP? (After all, > > dEQP has its own testrunner). Because dEQP runs all tests in a single > > process. If test > > 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining > > tests never run. Piglit doesn't suffer from > that problem, because it runs each test as a separate process. > > > > Tested on Intel Ivybridge by running the command > > piglit run -t dEQP-GLES3/info/vendor \ > > tests/deqp_gles3.py tests/quick.py results with and without > > PIGLIT_DEQP_GLES3_EXE set and with and without > piglit.conf:deqp-gles.exe set. > > > > Cc: Dylan Baker <[email protected]> > > Cc: Michael W Mason <[email protected]> > > Signed-off-by: Chad Versace <[email protected]> > > --- > > > > v2: > > - Remove unused imports (dcbaker) > > - Remove extra newlines (dcbaker) > > - Don't use 'global' keyword (dcbaker) > > - Use global constant for DEQP_GLES3_EXE (dcbaker) > > - Fix DEQPTest.interpret_result to correctly parse test output. > > (mwmason) > > - Use 'super' keyword (dcbaker) > > > > v3: > > - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason) > > > > piglit.conf.example | 7 +++ > > tests/deqp_gles3.py | 136 > > ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 143 insertions(+) > > create mode 100644 tests/deqp_gles3.py > > > > diff --git a/piglit.conf.example b/piglit.conf.example index > > a864c01..0ac0f4f 100644 > > --- a/piglit.conf.example > > +++ b/piglit.conf.example > > @@ -33,6 +33,13 @@ bindir=/home/usr/oclconform testA testB > > > > +;[deqp-gles3] > > +; > > +; Path to the deqp-gles3 executable. You can also set this with the > > +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the > > +dEQP-GLES3 tests if and ; only if this option is set. > > +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3 > > + > > ; Section for specific oclconform test. One of these sections is required > > for ; each test list in the oclconform section and > must be called: > > ; oclconform-$testname > > diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode > > 100644 index 0000000..24035c6 > > --- /dev/null > > +++ b/tests/deqp_gles3.py > > @@ -0,0 +1,136 @@ > > +# Copyright 2014 Intel Corporation > > +# > > +# Permission is hereby granted, free of charge, to any person > > +obtaining a copy # of this software and associated documentation > > +files (the "Software"), to deal # in the Software without > > +restriction, including without limitation the rights # to use, copy, > > +modify, merge, publish, distribute, sublicense, and/or sell # copies > > +of the Software, and to permit persons to whom the Software is # furnished > > to do so, subject to the following > conditions: > > +# > > +# The above copyright notice and this permission notice shall be > > +included in # all copies or substantial portions of the Software. > > +# > > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > > +EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > > +MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND > > +NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS > > +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN > > +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN > > +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # > > SOFTWARE. > > + > > +import ConfigParser > > +import os > > +import subprocess > > + > > +# Piglit modules > > +import framework > > +from framework.profile import Test, TestProfile > > + > > +__all__ = ['profile'] > > + > > +def get_option(env_varname, config_option): > > + """Query the given environment variable and then piglit.conf for the > > + option. Return None if the option is unset. > > + """ > > + opt = os.environ.get(env_varname, None) > > + if opt is not None: > > + return opt > > + > > + try: > > + opt = framework.core.PIGLIT_CONFIG.get(config_option[0], > > + config_option[1]) > > + except ConfigParser.NoSectionError, ConfigParser.NoOptionError: > > + pass > > + > > + return opt > > + > > + > > +# Path to the deqp-gles3 executable. > > +DEQP_GLES3_EXE = get_option('PIGLIT_DEQP_GLES3_EXE', ('deqp-gles3', 'exe')) Add an environment variable for options to pass to deqp-gles3: # Command line options to pass to deqp-gles3 executable. DEQP_GLES3_OPT = get_option('PIGLIT_DEQP_GLES3_OPT', ('deqp-gles3', 'options')) > > + > > + > > +def gen_caselist_txt(): > > + """Generate dEQP-GLES3-cases.txt and return its path.""" > > + # dEQP is stupid (2014-12-07): > > + # 1. To generate the caselist file, dEQP requires that the process's > > + # current directory must be that same as that of the executable. > > + # Otherwise, it fails to find its data files. > > + # 2. dEQP creates the caselist file in the process's current > > directory > > + # and provides no option to change its location. > > + # 3. dEQP creates a GL context when generating the caselist. > > Therefore, > > + # the caselist must be generated on the test target rather than > > the > > + # build host. In other words, when the build host and test target > > + # differ then we cannot pre-generate the caselist on the build > > host: > > + # we must *dynamically* generate it during the testrun. > > + basedir = os.path.dirname(DEQP_GLES3_EXE) > > + caselist_path = os.path.join(basedir, 'dEQP-GLES3-cases.txt') > > + subprocess.check_call( > > + [DEQP_GLES3_EXE, '--deqp-runmode=txt-caselist'], > > + cwd=basedir) > > + assert os.path.exists(caselist_path) > > + return caselist_path > > + > > + > > +def iter_deqp_test_cases(): > > + """Iterate over original dEQP GLES3 testcase names.""" > > + caselist_path = gen_caselist_txt() > > + with open(caselist_path) as caselist_file: > > + for i, line in enumerate(caselist_file): > > + if line.startswith('GROUP:'): > > + continue > > + elif line.startswith('TEST:'): > > + yield line[len('TEST:'):].strip() > > + else: > > + raise Exception('{0}:{1}: ill-formed > > +line'.format(caselist_path, i)) > > + > > + > > +class DEQPTest(Test): > > + > > + RESULT_MAP = {"Pass" : "pass", > > + "Fail" : "fail", > > + "QualityWarning" : "warn", > > + "InternalError" : "fail", > > + "Crash" : "crash", > > + "NotSupported" : "skip"} > > + > > + def __init__(self, case_name): > > + command = [ > > + DEQP_GLES3_EXE, > > + '--deqp-case=' + case_name, > > + '--deqp-visibility=hidden', > > + ] Change command to this: command = [ DEQP_GLES3_EXE, '--deqp-case=' + case_name, DEQP_GLES3_OPT ] The --deqp-visibility option isn't required, so let's leave that to the user. > > + super(DEQPTest, self).__init__(command) > > + > > + # dEQP's working directory must be the same as that of the > > executable, > > + # otherwise it cannot find its data files (2014-12-07). > > + self.cwd = os.path.dirname(DEQP_GLES3_EXE) > > + > > + def interpret_result(self): > > + if self.result['returncode'] != 0: > > + self.result['result'] = 'fail' > > + return > > + > > + for line in self.result['out'].split('\n'): > > + line = line.lstrip() > > + for (k, v) in DEQPTest.RESULT_MAP.iteritems(): > > + if line.startswith(k): > > + self.result['result'] = v > > + return > > + > > + # We failed to parse the test output. Fallback to 'fail'. > > + self.result['result'] = 'fail' > > + > > +def add_tests(): > > + if DEQP_GLES3_EXE is None: > > + return > > + > > + for deqp_testname in iter_deqp_test_cases(): > > + # dEQP uses '.' as the testgroup separator. > > + piglit_testname = deqp_testname.replace('.', '/') > > + > > + test = DEQPTest(deqp_testname) > > + profile.test_list[piglit_testname] = test > > + > > + > > +profile = TestProfile() > > +add_tests() > > -- > > 2.2.0 > > > > _______________________________________________ > > Piglit mailing list > > [email protected] > > http://lists.freedesktop.org/mailman/listinfo/piglit > > _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
