-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 05/29/2012 06:30 AM, Pauli Nieminen wrote:
> On Fri, May 25, 2012 at 01:40:10PM -0700, Paul Berry wrote:
>> In the last few months Chad Versace and I have put a lot of effort
>> into reworking the core of Piglit with an eye towards being able to
>> effectively test GLES, Wayland, Android, and core profiles. A lot of
>> that work is still in progress, and since others are starting to join
>> in this effort, it seems like a good time to lay out our long term
>> plans so that we can coordinate.
>>
>> Once we've had a chance to discuss these plans on the mailing list and
>> come to a consensus, we're hoping to upload the revised description to
>> the Piglit website (http://piglit.freedesktop.org/) so that it's easy
>> to find and refer to.
>>
>> Background:
>>
>> As of roughly January 2012, Piglit was almost exclusively geared
>> toward testing desktop OpenGL versions 1.0 through 3.0, running on
>> Linux, Mac OSX, and Windows. It used GLEW
>> (http://glew.sourceforge.net/) to dispatch to OpenGL functions, and
>> GLUT (http://www.opengl.org/resources/libraries/glut/) to interact
>> with the window system. Neither GLEW nor GLUT are compatible with
>> GLES, Wayland, Android, or core profiles.
>>
>> Only a few GLES tests existed, but they were compiled separately from
>> the GL tests, and they were not integrated tightly into the framework.
>> Furthermore, tests that in theory should have been able to be run on
>> both GL and GLES were only compiled for GL.
>>
>> Long term plan (and status):
>>
>> (5) In order for the framework to know which window systems, GL APIs,
>> and extensions are relevant to any particular test, we will add a
>> declarative mechanism for each test to specify the test requirements.
>> The declarative syntax will use macros or inline functions. For
>> example, if a test should run on desktop GL implementations that
>> support GL_EXT_framebuffer_object, desktop GL implementations version
>> 3.0 or later, and on ES2 implementations, the test would contain a
>> line that looks something like "test_params.requirements =
>> OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")),
>> GL_API_VERSION(3, 0), GLES2_API)". This will replace the imperative
>> mechanism that we currently use to limit a test to certain GL versions
>> and extensions (the piglit_require_xxx() functions). Status: not yet
>> started.
>>
>
> This makes it only slightly easier that using piglit_is_supported
> functions. That was mostly reason why I added runtime checks for
> platform to GLX and EGL code.
>
> Of course it is nice to write common and possible complex requirements
> for tests in standard simple to write fashion. But most likely tests
> will still have to do runtime branching depending on if it is running
> on GLES or GL.
Having the test requirements stated declaratively allows us to do several things
that are difficult, or not possible, with the piglit_is_supported functions.
Again,
these things have been part of Ken's, Paul's, and my plan for some time,
but we forgot to document it.
1) Filtering tests by GL API.
Eventually, a large portion of tests will be capable of running under GL and
GLES.
Once we reach that point, developers will normally not want to run all those
tests twice, first as GL and then as GLES. The framework will need to be
intelligent
enough to, by request, run only GL test cases, only GLES test cases, or run GL
and GLES test
cases (in which case some tests will be ran multiple times).
To allow this, the plan is to add some standard command-line option to test
executables and to the Piglit python scripts, such as --gl-api. If
--gl-api=gles2
is given to a test executable, then it will run only its GLES2 test cases. Given
- --gl-api=all, it will run test cases for all APIs. This plan is not yet set in
stone, and we're receptive to suggestions.
Similarly, if --gl-api=gles2 is given to piglit-print-commands.py, then the
script
will 1) invoke each test executable with a flag that requests it to print out
info
about its requirements, and 2) filter out tests that do no support GLES2. This
will give developers fine grained control over which tests they run, control
that would be cumbersome to provide with the piglit-run.py's -x option. And this
gives QA the ability to filter out inappropriate test cases when running Piglit
on Android.
2) Filtering window system platforms.
Once support for the --gl-api option is implemented, the next logical step
is to add a --platform={all,glx,wayland,x11_egl} option that behaves
analogously.
>> (6) Since standard C doesn't allow functions to be called inside a
>> global initializer, to facilitate (5) we will replace the main()
>> function in tests/util/piglit-framework.c with a piglit_run() function
>> which takes as its input a data structure describing the test
>> requirements (as well as other initialization info like the initial
>> window size). Each individual test will contain a main() function
>> that populates the data structure and then calls piglit_run(). To
>> avoid having to write a lot of boilerplate code in each individual
>> test, we'll provide a macro that creates a standard main() function
>> for those tests that don't need a lot of customization. Status: not
>> yet started.
>>
>
> There is failry simple way to have global initializer functions with
> gcc and msvc. But I still fail to see why we would need anything
> except piglit-framework.c having main. The framework code can check
> for test_params.requirements set by test code. It would work exactly
> same as piglit_width etc variables work.
You're right. The need for another global initializer function does not
neccessitate that each test define main(). The initialization of the
requirements struct could just as well go into a function named
piglit_require().
However, we have to remove main() from libpiglitutil, anyway.
The EGL tests, glean, and libpiglitutil define main(). The EGL tests and glean
link to libpiglitutil. Surprisingly, the linker doesn't complain. This situation
is very fragile. If you modify the libraries or tests in a way that changes
the order in which the linker resolves things, link failure occurs. I've
encountered
such link failures twice already.
This situation is absurd. main() needs to be removed from libpiglitutil. This
gives
us two options: a) that each test define main(), or b) that we create a new
static library,
libpiglitutil_main, that tests link to.
I don't see a strong argument for either (a) or (b), but I think (a) is the
simpler approach.
Which do you prefer?
>> (7) In order to allow the majority of piglit tests to run with core
>> profiles, forward-compatible contexts, and GLES (which don't support
>> depreceated functionality), we will write new wrapper functions in
>> tests/util which provide low-level testing operations using whatever
>> functionality is present. For example, we might have a function which
>> replaces any references to "glVertex" in a GLSL program with
>> references to a custom vertex attribute, for those implementations
>> that don't support the deprecated "glVertex" feature. Status: a few
>> wrapper functions already exist, but many more need to be written.
>>
>> (8) In order to allow a single test invocation to run in a number of
>> configurations sequentially (e.g. to test a certain piece of
>> functionality using both Wayland and X windows), we will extend the
>> protocol used by tests to report results so that a single invocation
>> of a test can report multiple test results. Status: not yet started.
>>
>> Implementation Strategy and immediate plans:
>>
>> The above description doesn't address what order we plan on
>> implementing these features. We would like to stay flexible on the
>> specific implementation order, so that we can adapt to unexpected
>> dependencies and to allow people to contribute to the effort as their
>> time becomes available. However, we would like to avoid regressions
>> as much as possible, since a regression in piglit is likely to be
>> misinterpreted as a regression in Mesa.
>>
>> Here is what we expect to be working on soon:
>>
>> (a) Replace the main() function in tests/util/piglit-framework.c with
>> individual main() functions in each test (see (6) above).
>>
>> (b) Add the necessary information to gl.spec and enumext.spec to
>> indicate which functions are avaiable in GLES, and to describe GLES
>> extensions (see (1) above).
>>
>> (c) Modify piglit-dispatch code generation scripts to consume the
>> information from (b) and dispatch properly to GLES.
>>
>> (d) Add Windows and MacOS X support to Waffle.
>>
>> (e) Add basic keyboard input support to Waffle. This is necessary
>> because most tests recognize "ESC" as a command to exit the test (if
>> not run in "-auto" mode), and because a few tests recognize other keys
>> (e.g. fbo tests that use keyboard input to cycle between framebuffer
>> formats).
>>
>> (f) Remove the compile-time switch that enables Waffle, and switch
>> over to using Waffle for all desktop GL tests. Note: since we want to
>> keep Piglit viable as a testing platform for Mesa while these tasks
>> are in progress, we can't do this until after doing (d) and (e).
>> However, we have already switched over to exclusively using Waffle for
>> GLES testing.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJPxRK1AAoJEAIvNt057x8ixmwP/3wscDzMEzjyTxq0S+FYBPJ+
nVA0vdw/nZxGoMY6QB/ap5R1TbYMOZXAScDa2Gxjc7Lm7jWop5icYQL3mcKaxRHZ
+rHr4Sg2AbcNGwaBd69Cl1Ma4rv4WWxLS5sZ3CHXoAcym8+cSysBYoYlclwhMX39
9TAHtpPVz2kIo8xfQVu50j/Z811aUWnm23TKlqwY0/2+vP8BzAOKEtk4q0i6+Iam
gZ8hA+9oAhfrb1gkxTaoBMsjhn0uhHFxtTkXaqg09+F1UYN987fJ3NeNG1pIe+nB
30EIrMGEm5Bk4BaN9ITgID755zzsM8bhozuAcpepe5kJVuhCD8h4OB6JAepB5AvB
8o3vSIdOPakash2TmUMZlRqpAGTog1+fBzofAy8TxayK8YhRw1w3m2lcHU7LuWqz
XW8L13vlRgn0ehbTw7i+OJr3/vig989olY8LgUCs5i8ZmDzSkXP1Ke5U/zYcXooR
pm5R3cPem5XntPbtaGlS0s2Wm1CiS872Nku1C96ew2tM+Z0Ml2GRnlIXPHnW3q+J
Ufc5EIFmJwfCbZSvmI3Fch8RnVhZRpvp1hfFCHtxxA7dsx22ZdecvYCIxGmAJElu
m9f41FRkoVYZDbpVHcalGred6q0QFX2xxz4viP6WIjYvvLxIvCAcSxAvJgJce5dW
zK4nYG3k0Njde5gfYLos
=LstY
-----END PGP SIGNATURE-----
_______________________________________________
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit
