On 05/29/2012 01:47 PM, Pauli Nieminen wrote: > On Tue, May 29, 2012 at 11:17:26AM -0700, Chad Versace wrote: >> On 05/29/2012 06:30 AM, Pauli Nieminen wrote: >>> On Fri, May 25, 2012 at 01:40:10PM -0700, Paul Berry wrote: >>>> In the last few months Chad Versace and I have put a lot of effort >>>> into reworking the core of Piglit with an eye towards being able to >>>> effectively test GLES, Wayland, Android, and core profiles. A lot of >>>> that work is still in progress, and since others are starting to join >>>> in this effort, it seems like a good time to lay out our long term >>>> plans so that we can coordinate. >>>> >>>> Once we've had a chance to discuss these plans on the mailing list and >>>> come to a consensus, we're hoping to upload the revised description to >>>> the Piglit website (http://piglit.freedesktop.org/) so that it's easy >>>> to find and refer to. >>>> >>>> Background: >>>> >>>> As of roughly January 2012, Piglit was almost exclusively geared >>>> toward testing desktop OpenGL versions 1.0 through 3.0, running on >>>> Linux, Mac OSX, and Windows. It used GLEW >>>> (http://glew.sourceforge.net/) to dispatch to OpenGL functions, and >>>> GLUT (http://www.opengl.org/resources/libraries/glut/) to interact >>>> with the window system. Neither GLEW nor GLUT are compatible with >>>> GLES, Wayland, Android, or core profiles. >>>> >>>> Only a few GLES tests existed, but they were compiled separately from >>>> the GL tests, and they were not integrated tightly into the framework. >>>> Furthermore, tests that in theory should have been able to be run on >>>> both GL and GLES were only compiled for GL. >>>> >>>> Long term plan (and status): >>>> >>>> (5) In order for the framework to know which window systems, GL APIs, >>>> and extensions are relevant to any particular test, we will add a >>>> declarative mechanism for each test to specify the test requirements. >>>> The declarative syntax will use macros or inline functions. For >>>> example, if a test should run on desktop GL implementations that >>>> support GL_EXT_framebuffer_object, desktop GL implementations version >>>> 3.0 or later, and on ES2 implementations, the test would contain a >>>> line that looks something like "test_params.requirements = >>>> OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")), >>>> GL_API_VERSION(3, 0), GLES2_API)". This will replace the imperative >>>> mechanism that we currently use to limit a test to certain GL versions >>>> and extensions (the piglit_require_xxx() functions). Status: not yet >>>> started. >>>> >>> >>> This makes it only slightly easier that using piglit_is_supported >>> functions. That was mostly reason why I added runtime checks for >>> platform to GLX and EGL code. >>> >>> Of course it is nice to write common and possible complex requirements >>> for tests in standard simple to write fashion. But most likely tests >>> will still have to do runtime branching depending on if it is running >>> on GLES or GL. >> >> Having the test requirements stated declaratively allows us to do several >> things >> that are difficult, or not possible, with the piglit_is_supported functions. >> Again, >> these things have been part of Ken's, Paul's, and my plan for some time, >> but we forgot to document it. >> >> 1) Filtering tests by GL API. >> >> Eventually, a large portion of tests will be capable of running under GL and >> GLES. >> Once we reach that point, developers will normally not want to run all those >> tests twice, first as GL and then as GLES. The framework will need to be >> intelligent >> enough to, by request, run only GL test cases, only GLES test cases, or run >> GL and GLES test >> cases (in which case some tests will be ran multiple times). >> >> To allow this, the plan is to add some standard command-line option to test >> executables and to the Piglit python scripts, such as --gl-api. If >> --gl-api=gles2 >> is given to a test executable, then it will run only its GLES2 test cases. >> Given >> --gl-api=all, it will run test cases for all APIs. This plan is not yet set >> in >> stone, and we're receptive to suggestions. >> >> Similarly, if --gl-api=gles2 is given to piglit-print-commands.py, then the >> script >> will 1) invoke each test executable with a flag that requests it to print >> out info >> about its requirements, and 2) filter out tests that do no support GLES2. >> This >> will give developers fine grained control over which tests they run, control >> that would be cumbersome to provide with the piglit-run.py's -x option. And >> this >> gives QA the ability to filter out inappropriate test cases when running >> Piglit >> on Android. >> >> 2) Filtering window system platforms. >> >> Once support for the --gl-api option is implemented, the next logical step >> is to add a --platform={all,glx,wayland,x11_egl} option that behaves >> analogously. >> > > yes. Standard checks are good way. If I understand your proposal > correctly tests binary is supposed to tell about if it is run able in > given platform. > > Wouldn't that makes it simpler to have same tests set in all > platforms and GL flavors? Tests automatically skip when given > environment doesn't match what test requires. Fork-exec-skip should be > fast enough not to affect total runtime much.
It actually does take a while. Running all the 3.0+/1.30+ tests on pre-Sandybridge skips quite a few tests already...and more all the time. Plus, we'd eventually like to include FBConfigs/visuals in the list of requirements (i.e. RGBA, at least N bits of depth, stencil), and that will create a -lot- of combinations. The idea is to improve the runner infrastructure to select reasonable combinations of tests rather than always running them all. _______________________________________________ Piglit mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/piglit
