Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 21.11.2014 23:57, Samuel Iglesias Gonsálvez wrote: On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote: On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. gpu.py takes much more time on quad core i7-3610QM laptop with gallium llvmpipe... After 6 hours waiting for piglit to finish, I started killing the processes that took too much time. I just tried it again with today's snapshots of LLVM SVN trunk and Mesa Git master, it took about 22.5 minutes. Did you really test gpu.py (which is a subset of quick.py), not all.py? -- Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 25/11/14 09:59, Michel Dänzer wrote: On 21.11.2014 23:57, Samuel Iglesias Gonsálvez wrote: On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote: On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. gpu.py takes much more time on quad core i7-3610QM laptop with gallium llvmpipe... After 6 hours waiting for piglit to finish, I started killing the processes that took too much time. I just tried it again with today's snapshots of LLVM SVN trunk and Mesa Git master, it took about 22.5 minutes. Did you really test gpu.py (which is a subset of quick.py), not all.py? Yes, I tried with gpu.py but with an old llvm version. I will test again with LLVM SVN trunk and last Mesa Git master. Sam ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 25/11/14 09:43, Samuel Iglesias Gonsálvez wrote: On 25/11/14 09:59, Michel Dänzer wrote: On 21.11.2014 23:57, Samuel Iglesias Gonsálvez wrote: On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote: On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. gpu.py takes much more time on quad core i7-3610QM laptop with gallium llvmpipe... After 6 hours waiting for piglit to finish, I started killing the processes that took too much time. I just tried it again with today's snapshots of LLVM SVN trunk and Mesa Git master, it took about 22.5 minutes. Did you really test gpu.py (which is a subset of quick.py), not all.py? Yes, I tried with gpu.py but with an old llvm version. I will test again with LLVM SVN trunk and last Mesa Git master. Try the new llvmpipe.py test-list instead of gpu.py. It should skip the problematic tests. Jose ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote: On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. gpu.py takes much more time on quad core i7-3610QM laptop with gallium llvmpipe... After 6 hours waiting for piglit to finish, I started killing the processes that took too much time. If I filter them from the beginning, it takes about 10 minutes too. Thanks! Sam signature.asc Description: This is a digitally signed message part ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. -- Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On Tue, 2014-11-18 at 13:41 -0800, Jason Ekstrand wrote: Iago, Most of this looks pretty good to me. The one primary concern I have is in the handling of integer formats. I made the comment in a couple of patches, but I'll make it in general here. In a lot of the code, when you convert from integer formats to float, you treat them as if they are normalized. Can you explain why you are doing this? It seems very wrong to me. Right, I have been discussing this with Samuel and it does look wrong. He will change the code and run a new piglit run to verify the changes. One other issue is that I couldn't actually get it to compile. This is probably due to the fact that I always build out-of-tree, so sourcedir and builddir are not the same. Not really sure what's going on there. Mmm... that's weird. I think I remember seeing a patch that added a new file and could be the source of that issue. We will look into it. Other than that, It's looking pretty good. I'll try and get to reviewing your second patch series tomorrow. Since my R-B obviously doesn't mean much on the code I wrote I'll try and dig up a second reviewer as well. Yes, that makes sense. Thanks for looking into these patches so fast! Iago --Jason On Tue, Nov 18, 2014 at 12:43 AM, Iago Toral Quiroga ito...@igalia.com wrote: This is the fist of two series of patches to address: https://bugs.freedesktop.org/show_bug.cgi?id=84566 The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new mesa formats and their pack/unpack functions. mesa: use format conversion functions in swrast mesa/pack: use autogenerated format_pack functions mesa/main/pack_tmp.h: Add float conversion support mesa/pack: refactor _mesa_pack_rgba_span_float() src/mesa/Makefile.am | 18 + src/mesa/Makefile.sources |4 +- src/mesa/main/colormac.h |3 -
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote: On 18.11.2014 17:43, Iago Toral Quiroga wrote: For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Any particular reason for not testing at least piglit gpu.py with llvmpipe? Last time I tried that a few months ago, it didn't take much more than ten minutes on a quad-core A10-7850K. Not really, we tried to run the full suite but many tests would take forever and we thought we should just cut it down to the tests that seemed more related to the kind of stuff we were working on. Also, since we would ran piglit very often to verify our changes, specially towards the end of the development, we needed something manageable. We will give gpu.py a try too. Thanks! Iago ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 19/11/14 09:25, Iago Toral wrote: On Tue, 2014-11-18 at 13:41 -0800, Jason Ekstrand wrote: Iago, Most of this looks pretty good to me. The one primary concern I have is in the handling of integer formats. I made the comment in a couple of patches, but I'll make it in general here. In a lot of the code, when you convert from integer formats to float, you treat them as if they are normalized. Can you explain why you are doing this? It seems very wrong to me. Right, I have been discussing this with Samuel and it does look wrong. He will change the code and run a new piglit run to verify the changes. As Iago said, I'm going to do some changes and test them. After that I will reply to the mailing list. One other issue is that I couldn't actually get it to compile. This is probably due to the fact that I always build out-of-tree, so sourcedir and builddir are not the same. Not really sure what's going on there. Mmm... that's weird. I think I remember seeing a patch that added a new file and could be the source of that issue. We will look into it. I confirmed that the out-of-tree compilation error is because of the autogeneration of format_pack.c and format_unpack.c files. I'm going to fix it. Thanks! Sam Other than that, It's looking pretty good. I'll try and get to reviewing your second patch series tomorrow. Since my R-B obviously doesn't mean much on the code I wrote I'll try and dig up a second reviewer as well. Yes, that makes sense. Thanks for looking into these patches so fast! Iago --Jason On Tue, Nov 18, 2014 at 12:43 AM, Iago Toral Quiroga ito...@igalia.com wrote: This is the fist of two series of patches to address: https://bugs.freedesktop.org/show_bug.cgi?id=84566 The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new mesa formats and their pack/unpack functions. mesa: use format conversion functions in swrast mesa/pack: use
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 19/11/14 08:29, Iago Toral wrote:
On Wed, 2014-11-19 at 17:09 +0900, Michel Dänzer wrote:
On 18.11.2014 17:43, Iago Toral Quiroga wrote:
For software drivers we worked with a trimmed set of piglit tests (related to
format conversion), ~5700 tests selected with the following filter:
-t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix
-t fbo -t frame
Any particular reason for not testing at least piglit gpu.py with
llvmpipe? Last time I tried that a few months ago, it didn't take much
more than ten minutes on a quad-core A10-7850K.
Not really, we tried to run the full suite but many tests would take
forever and we thought we should just cut it down to the tests that
seemed more related to the kind of stuff we were working on. Also, since
we would ran piglit very often to verify our changes, specially towards
the end of the development, we needed something manageable.
We will give gpu.py a try too. Thanks!
Iago
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_mesa-2Ddevd=AAIGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzEm=AedzeFofUxa1m8ruA88aWS9AZRrJAbQeOAET3aeJXMUs=s-L51NQv5NFJUygqYpmGzRjc-tlQYulkLBoFjBrMKFse=
I have this on my piglit test list for llvmpipe:
# These take too long or too much memory
profile.tests['glean'].pop('pointAtten', None)
profile.tests['glean'].pop('texCombine', None)
profile.tests['shaders'].pop('glsl-fs-inline-explosion', None)
profile.tests['shaders'].pop('glsl-fs-unroll-explosion', None)
profile.tests['shaders'].pop('glsl-vs-inline-explosion', None)
profile.tests['shaders'].pop('glsl-vs-unroll-explosion', None)
profile.tests['spec']['ARB_fragment_program'].pop('fp-indirections',
None)
profile.tests['spec']['ARB_fragment_program'].pop('fp-indirections2',
None)
profile.tests['spec']['!OpenGL 1.1'].pop('streaming-texture-leak', None)
profile.tests['spec']['!OpenGL 1.1'].pop('max-texture-size', None)
but I think the gpu.py seems quite close.
Jose
___
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On Wed, 2014-11-19 at 01:15 +, Emil Velikov wrote: Hi Iago, On 18/11/14 08:43, Iago Toral Quiroga wrote: [snip] Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. Have you noticed if any of those fixes resolves a piglit and/or a real world application ? If so it might be worth tagging them for the stable branches :) OK, we will review these fixes and tag them for the stable branch accordingly. I've noticed with this series we require mako in order to build mesa. Please update the documentation to reflect the new dependency. How gracefully do we handle the cases when it's missing ? Printing a message similar to mako vXX or later is required would be very helpful. Yeah, good idea. I am not an expert in autoconf tools but let's see if I can add some check like that. Thanks, Sam Thanks Emil ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev signature.asc Description: This is a digitally signed message part ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
This is the fist of two series of patches to address: https://bugs.freedesktop.org/show_bug.cgi?id=84566 The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new mesa formats and their pack/unpack functions. mesa: use format conversion functions in swrast mesa/pack: use autogenerated format_pack functions mesa/main/pack_tmp.h: Add float conversion support mesa/pack: refactor _mesa_pack_rgba_span_float() src/mesa/Makefile.am | 18 + src/mesa/Makefile.sources |4 +- src/mesa/main/colormac.h |3 - src/mesa/main/format_convert.py| 71 + src/mesa/main/format_info.py | 41 + src/mesa/main/format_pack.c| 2994 src/mesa/main/format_pack.c.mako | 1147 ++ src/mesa/main/format_pack.h|6 + src/mesa/main/format_unpack.c | 4400 src/mesa/main/format_unpack.c.mako | 914 src/mesa/main/format_utils.c | 248 +- src/mesa/main/format_utils.h | 105 + src/mesa/main/formats.c| 193 +- src/mesa/main/formats.csv | 13 + src/mesa/main/formats.h| 73 + src/mesa/main/pack.c | 2111 +++-- src/mesa/main/pack_tmp.h | 76 +- src/mesa/main/run_mako.py |7 + src/mesa/main/teximage.c |4 +- src/mesa/main/texstore.c |2 +- src/mesa/swrast/s_drawpix.c|3 - src/mesa/swrast/s_texfetch.c | 13 + src/mesa/swrast/s_texfetch_tmp.h | 1359 +-- 23 files changed, 3222 insertions(+), 10583 deletions(-) create mode 100644 src/mesa/main/format_convert.py delete mode 100644 src/mesa/main/format_pack.c create mode 100644 src/mesa/main/format_pack.c.mako delete mode 100644 src/mesa/main/format_unpack.c create mode 100644 src/mesa/main/format_unpack.c.mako create mode 100644 src/mesa/main/run_mako.py -- 1.9.1 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
I forgot to say that the series is available for testing here: https://github.com/Igalia/mesa/tree/itoral-autogen-packing-review Also, one of the patches was held for review by the list owner due to its size (patch 12, which handles auto-generation of format_unpack.c), so reviewers can fetch it from that link too if it does not reach the list in the end. Iago On Tue, 2014-11-18 at 09:43 +0100, Iago Toral Quiroga wrote: This is the fist of two series of patches to address: https://bugs.freedesktop.org/show_bug.cgi?id=84566 The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new mesa formats and their pack/unpack functions. mesa: use format conversion functions in swrast mesa/pack: use autogenerated format_pack functions mesa/main/pack_tmp.h: Add float conversion support mesa/pack: refactor _mesa_pack_rgba_span_float() src/mesa/Makefile.am | 18 + src/mesa/Makefile.sources |4 +- src/mesa/main/colormac.h |3 - src/mesa/main/format_convert.py| 71 + src/mesa/main/format_info.py | 41 + src/mesa/main/format_pack.c| 2994 src/mesa/main/format_pack.c.mako | 1147 ++ src/mesa/main/format_pack.h|6 + src/mesa/main/format_unpack.c | 4400 src/mesa/main/format_unpack.c.mako | 914 src/mesa/main/format_utils.c | 248 +- src/mesa/main/format_utils.h | 105 + src/mesa/main/formats.c| 193 +- src/mesa/main/formats.csv | 13 + src/mesa/main/formats.h| 73 + src/mesa/main/pack.c | 2111 +++-- src/mesa/main/pack_tmp.h | 76 +- src/mesa/main/run_mako.py |7 + src/mesa/main/teximage.c |4 +- src/mesa/main/texstore.c |2 +- src/mesa/swrast/s_drawpix.c|3 - src/mesa/swrast/s_texfetch.c | 13 + src/mesa/swrast/s_texfetch_tmp.h | 1359 +-- 23 files changed, 3222 insertions(+), 10583 deletions(-) create mode 100644 src/mesa/main/format_convert.py delete mode 100644 src/mesa/main/format_pack.c create mode 100644 src/mesa/main/format_pack.c.mako delete mode 100644 src/mesa/main/format_unpack.c create mode 100644 src/mesa/main/format_unpack.c.mako create mode 100644 src/mesa/main/run_mako.py ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. First, I think this is a great goal. And while I haven't reviewed them in detail, just from skimming through them, these patch series seem to be a great cleanup and a lot of work went into it. I certainly don't object to any of this. But I have to say I find a bit unfortunate that so much effort is being put on implementing something specific to Mesa formats instead of taking this opportunity to devise a solution that would work both for gallium and Mesa formats. Furthermore I see there is some interest speeding mesa using SSE2 intrinsics, and of course format conversion is precisely one of the code paths that can great benefit from SIMD, but I have no doubt: the most efficient and sane way of leveraging SIMD with all these formats conversion is to JIT compile format conversion code tailored to the CPU in runtime. There are just too many CPU variations to statically generate C code for every one of them. And lot of the code to do this already exists in src/gallium/auxiliary/gallivm. We should consider using it from src/mesa/*. This gallium helper code was written outside mesa to avoid the GL dependency (there was one point some of this stuff had to run in XP kernel mode! thankfully not anymore), and because when gallium was written the idea was that all Mesa drivers would eventually migrate into it. Alas, that didn't happen, and might never happen. That's OK. But in that case, I do believe we should find a way of sharing more of the stuff in src/gallium/auxiliary with all Mesa drivers, non-gallium classic drivers included. In particular, we really should have a format conversion module that is capable of handling a superset of all Mesa and Gallium formats somwhere in src/util/ . One might even leverage the auto-generated pack/unpack functions in src/gallium/auxiliary/u_format* though I wouldn't care if not, as long as the functionality is the same. In short, we can't change the past, but I wish we could have more synergy in the future. Jose On 18/11/14 08:43, Iago Toral Quiroga wrote: This is the fist of two series of patches to address: https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.freedesktop.org_show-5Fbug.cgi-3Fid-3D84566d=AAIGaQc=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEsr=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzEm=9H2UxyGUngIpuFNlQEzL9VtIOmYVKVs2o4nyL_We6o0s=xYhgcKEBxp3-d_ddjpFiNGRkqLo2BMUedLQ67ck2ce8e= The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
Jose, I haven't had time to fully review Iago and Samuel's code, so I can't 100% comment on it right now. However, let me make a few comments on the overarching plan as it were. On Tue, Nov 18, 2014 at 2:36 AM, Jose Fonseca jfons...@vmware.com wrote: The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. First, I think this is a great goal. And while I haven't reviewed them in detail, just from skimming through them, these patch series seem to be a great cleanup and a lot of work went into it. I certainly don't object to any of this. But I have to say I find a bit unfortunate that so much effort is being put on implementing something specific to Mesa formats instead of taking this opportunity to devise a solution that would work both for gallium and Mesa formats. That is the end goal. Unfortunately, getting there requires a lot of work. Probably more work on the mesa side than on the gallium side. A big part of the problem was that there was a lot of code for format conversion and it was scattered all over mesa/main directory. A lot of stuff needs to be cleaned up an unified inside mesa/main before things can be unified with gallium. Much of that work is now done. One of the things that I would like to see happen after this stuff lands is to convert the mesa pack/unpack functions to take a width, height, and stride. Then they would have exactly the same function signature as the gallium conversion functions and merging will be much easier. Then we can work on moving the format handling code into a helper library which, for the moment, I'll call libmesaformat. Then both gallium and mesa classic can pull from libmesaformat and we can kill all of the redundant code. Whose autogenerator framework we end up keeping is kind of immaterial, they're not that hard to write. One of the decisions that has to be made there (probably a topic for another thread) is how we would want to structure the format metadata. Mesa and gallium both have completely different ways of structuring it and we need to unify that if we're going to unify the conversion code. Personally, I think gallium's is cleaner and more expressive, but it lacks the GL information that core mesa needs. But, like I said, that's a topic for another thread. Furthermore I see there is some interest speeding mesa using SSE2 intrinsics, and of course format conversion is precisely one of the code paths that can great benefit from SIMD, but I have no doubt: the most efficient and sane way of leveraging SIMD with all these formats conversion is to JIT compile format conversion code tailored to the CPU in runtime. There are just too many CPU variations to statically generate C code for every one of them. And lot of the code to do this already exists in src/gallium/auxiliary/gallivm. We should consider using it from src/mesa/*. Yes, there were some patches. However, given my experiments in the past, I'm skeptical as to how much of a real benefit it would be to SSE-accelerate all the format conversion. When I did my first major rework a couple of months ago, I experimented with using the SSSE3 shuffle operation for doing swizzling of the C implementation. The net result of that experiment is that using SSSE3 had a very marginal benefit over a good C implementation such as the one we now have. The real problem we had was that the current format conversion stuff was doing the pessimal thing in a lot of cases; a lot of that stupid is now gone. So, if someone wants to work on that, I'm curious to see their results, but I'm not holding out for it. As far as doing a JIT compile, I've thought of it. Daniel Stone recently mentioned the idea of using ORC for doing things like this and it might be a good fit. However, be fore we can do that, we need a framework for these things (which we now have, thanks to this series). How is that done in gallivm? Is it done using LLVM? If so, it might be a non-starter. I don't want to rekindle any debates, but you're going to have trouble convincing some people that format conversion is a good enough reason for a hard dependency on LLVM. Another idea that has been put forward would be to, whenever possible, push the unconverted data to the GPU as a linear texture and use the GPU to do the format conversion. This would quite possibly be better than either a straight CPU path or an optimized JIT'ed path. This gallium helper code was written outside mesa to avoid the GL dependency (there was one point some of this stuff had to run in XP kernel mode! thankfully not anymore), and because when gallium was written the idea was that all Mesa drivers would eventually migrate into it. Alas, that didn't happen, and might never happen. That's OK. But in that case, I do believe we should find a way of sharing more of the stuff in
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
Iago, Most of this looks pretty good to me. The one primary concern I have is in the handling of integer formats. I made the comment in a couple of patches, but I'll make it in general here. In a lot of the code, when you convert from integer formats to float, you treat them as if they are normalized. Can you explain why you are doing this? It seems very wrong to me. One other issue is that I couldn't actually get it to compile. This is probably due to the fact that I always build out-of-tree, so sourcedir and builddir are not the same. Not really sure what's going on there. Other than that, It's looking pretty good. I'll try and get to reviewing your second patch series tomorrow. Since my R-B obviously doesn't mean much on the code I wrote I'll try and dig up a second reviewer as well. --Jason On Tue, Nov 18, 2014 at 12:43 AM, Iago Toral Quiroga ito...@igalia.com wrote: This is the fist of two series of patches to address: https://bugs.freedesktop.org/show_bug.cgi?id=84566 The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. The goal of this first series is to address auto-generation of our pack/unpack functions (format_pack.c and format_unpack.c). Currently, we have a ton of hand-coded pack/unpack functions for lots of formats, but we can auto-generate most of that code instead, so this series handles this. This is based on initial work by Jason Ekstrand. Tested on i965, classic swrast and gallium (radeon, nouveau, llvmpipe) without regressions. For software drivers we worked with a trimmed set of piglit tests (related to format conversion), ~5700 tests selected with the following filter: -t format -t color -t tex -t image -t swizzle -t clamp -t rgb -t lum -t pix -t fbo -t frame Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. * Patches 8-16 implement auto-generation of pack/unpack functions. * Patches 17-20 make use of the auto-generated pack/unpack functions in various places to simplify the current code. Notice that some of the fixes in patches 1-7 will become obsolete as soon as we auto-generate the pack/unpack functions, but we thought it would make sense to keep them in the patch set anyway since we started from that base and they should be correct fixes to the currently existing code. Iago Toral Quiroga (1): swrast: Remove unused variable. Jason Ekstrand (9): mesa/format_utils: Fix a bug in one of the format helper functions mesa: Fix packing/unpacking of MESA_FORMAT_R5G6B5_UNORM mesa/colormac: Remove an unused macro mesa: Fix A1R5G5B5 packing/unpacking mesa/format_utils: Prefix and expose the conversion helper functions mesa: Add a concept of an array format mesa: Add a _mesa_is_format_color_format helper mesa: Autogenerate most of format_pack.c mesa: Autogenerate format_unpack.c Samuel Iglesias Gonsalvez (10): mesa: Fix get_texbuffer_format(). mesa: Fix _mesa_swizzle_and_convert integer conversions to clamp properly mesa: Add _mesa_pack_uint_rgba_row() format conversion function mesa: Add non-normalized formats support for ubyte packing functions mesa/format_pack: Add _mesa_pack_int_rgba_row() mesa/formats: add new mesa formats and their pack/unpack functions. mesa: use format conversion functions in swrast mesa/pack: use autogenerated format_pack functions mesa/main/pack_tmp.h: Add float conversion support mesa/pack: refactor _mesa_pack_rgba_span_float() src/mesa/Makefile.am | 18 + src/mesa/Makefile.sources |4 +- src/mesa/main/colormac.h |3 - src/mesa/main/format_convert.py| 71 + src/mesa/main/format_info.py | 41 + src/mesa/main/format_pack.c| 2994 src/mesa/main/format_pack.c.mako | 1147 ++ src/mesa/main/format_pack.h|6 + src/mesa/main/format_unpack.c | 4400 src/mesa/main/format_unpack.c.mako | 914 src/mesa/main/format_utils.c | 248 +- src/mesa/main/format_utils.h | 105 + src/mesa/main/formats.c| 193 +- src/mesa/main/formats.csv | 13 + src/mesa/main/formats.h| 73 + src/mesa/main/pack.c | 2111 +++-- src/mesa/main/pack_tmp.h | 76 +- src/mesa/main/run_mako.py |7 + src/mesa/main/teximage.c |4 +- src/mesa/main/texstore.c |2 +- src/mesa/swrast/s_drawpix.c|3 - src/mesa/swrast/s_texfetch.c | 13 + src/mesa/swrast/s_texfetch_tmp.h | 1359 +-- 23 files changed, 3222 insertions(+), 10583 deletions(-) create mode 100644 src/mesa/main/format_convert.py delete mode 100644 src/mesa/main/format_pack.c
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On 18/11/14 17:10, Jason Ekstrand wrote: Jose, I haven't had time to fully review Iago and Samuel's code, so I can't 100% comment on it right now. However, let me make a few comments on the overarching plan as it were. On Tue, Nov 18, 2014 at 2:36 AM, Jose Fonseca jfons...@vmware.com mailto:jfons...@vmware.com wrote: The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. First, I think this is a great goal. And while I haven't reviewed them in detail, just from skimming through them, these patch series seem to be a great cleanup and a lot of work went into it. I certainly don't object to any of this. But I have to say I find a bit unfortunate that so much effort is being put on implementing something specific to Mesa formats instead of taking this opportunity to devise a solution that would work both for gallium and Mesa formats. That is the end goal. Unfortunately, getting there requires a lot of work. Probably more work on the mesa side than on the gallium side. A big part of the problem was that there was a lot of code for format conversion and it was scattered all over mesa/main directory. A lot of stuff needs to be cleaned up an unified inside mesa/main before things can be unified with gallium. Much of that work is now done. One of the things that I would like to see happen after this stuff lands is to convert the mesa pack/unpack functions to take a width, height, and stride. Then they would have exactly the same function signature as the gallium conversion functions and merging will be much easier. Then we can work on moving the format handling code into a helper library which, for the moment, I'll call libmesaformat. Then both gallium and mesa classic can pull from libmesaformat and we can kill all of the redundant code. Whose autogenerator framework we end up keeping is kind of immaterial, they're not that hard to write. Oh, got it now. Sounds great then. One of the decisions that has to be made there (probably a topic for another thread) is how we would want to structure the format metadata. Mesa and gallium both have completely different ways of structuring it and we need to unify that if we're going to unify the conversion code. Personally, I think gallium's is cleaner and more expressive, but it lacks the GL information that core mesa needs. But, like I said, that's a topic for another thread. Furthermore I see there is some interest speeding mesa using SSE2 intrinsics, and of course format conversion is precisely one of the code paths that can great benefit from SIMD, but I have no doubt: the most efficient and sane way of leveraging SIMD with all these formats conversion is to JIT compile format conversion code tailored to the CPU in runtime. There are just too many CPU variations to statically generate C code for every one of them. And lot of the code to do this already exists in src/gallium/auxiliary/gallivm. We should consider using it from src/mesa/*. Yes, there were some patches. However, given my experiments in the past, I'm skeptical as to how much of a real benefit it would be to SSE-accelerate all the format conversion. When I did my first major rework a couple of months ago, I experimented with using the SSSE3 shuffle operation for doing swizzling of the C implementation. The net result of that experiment is that using SSSE3 had a very marginal benefit over a good C implementation such as the one we now have. The real problem we had was that the current format conversion stuff was doing the pessimal thing in a lot of cases; a lot of that stupid is now gone. So, if someone wants to work on that, I'm curious to see their results, but I'm not holding out for it. As far as doing a JIT compile, I've thought of it. Daniel Stone recently mentioned the idea of using ORC for doing things like this and it might be a good fit. I never heard of ORC, and I didn't get any relevant hits from google. What is it? However, be fore we can do that, we need a framework for these things (which we now have, thanks to this series). How is that done in gallivm? Is it done using LLVM? If so, it might be a non-starter. I don't want to rekindle any debates, but you're going to have trouble convincing some people that format conversion is a good enough reason for a hard dependency on LLVM. Yes, it's done with LLVM. But note that when using LLVM as a JIT for CPU the we practically only use the LLVM C API, whose ABI is stable across any LLVM version. (We use a bit of C++ in gallivm/llvmpipe just for a few minor omissions of the LLVM C API but that's something we can and should fix. Just a matter of upstreaming that functionality.) In other words, I'm not aware of any technical/logistic argument against using LLVM C API as a CPU
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
On Tue, Nov 18, 2014 at 1:42 PM, Jose Fonseca jfons...@vmware.com wrote: On 18/11/14 17:10, Jason Ekstrand wrote: Jose, I haven't had time to fully review Iago and Samuel's code, so I can't 100% comment on it right now. However, let me make a few comments on the overarching plan as it were. On Tue, Nov 18, 2014 at 2:36 AM, Jose Fonseca jfons...@vmware.com mailto:jfons...@vmware.com wrote: The idea is that we have a lot of format conversion code scattered through different files in the repository, a lot of that is redundant / duplicated, so this intends to address that issue. First, I think this is a great goal. And while I haven't reviewed them in detail, just from skimming through them, these patch series seem to be a great cleanup and a lot of work went into it. I certainly don't object to any of this. But I have to say I find a bit unfortunate that so much effort is being put on implementing something specific to Mesa formats instead of taking this opportunity to devise a solution that would work both for gallium and Mesa formats. That is the end goal. Unfortunately, getting there requires a lot of work. Probably more work on the mesa side than on the gallium side. A big part of the problem was that there was a lot of code for format conversion and it was scattered all over mesa/main directory. A lot of stuff needs to be cleaned up an unified inside mesa/main before things can be unified with gallium. Much of that work is now done. One of the things that I would like to see happen after this stuff lands is to convert the mesa pack/unpack functions to take a width, height, and stride. Then they would have exactly the same function signature as the gallium conversion functions and merging will be much easier. Then we can work on moving the format handling code into a helper library which, for the moment, I'll call libmesaformat. Then both gallium and mesa classic can pull from libmesaformat and we can kill all of the redundant code. Whose autogenerator framework we end up keeping is kind of immaterial, they're not that hard to write. Oh, got it now. Sounds great then. One of the decisions that has to be made there (probably a topic for another thread) is how we would want to structure the format metadata. Mesa and gallium both have completely different ways of structuring it and we need to unify that if we're going to unify the conversion code. Personally, I think gallium's is cleaner and more expressive, but it lacks the GL information that core mesa needs. But, like I said, that's a topic for another thread. Furthermore I see there is some interest speeding mesa using SSE2 intrinsics, and of course format conversion is precisely one of the code paths that can great benefit from SIMD, but I have no doubt: the most efficient and sane way of leveraging SIMD with all these formats conversion is to JIT compile format conversion code tailored to the CPU in runtime. There are just too many CPU variations to statically generate C code for every one of them. And lot of the code to do this already exists in src/gallium/auxiliary/gallivm. We should consider using it from src/mesa/*. Yes, there were some patches. However, given my experiments in the past, I'm skeptical as to how much of a real benefit it would be to SSE-accelerate all the format conversion. When I did my first major rework a couple of months ago, I experimented with using the SSSE3 shuffle operation for doing swizzling of the C implementation. The net result of that experiment is that using SSSE3 had a very marginal benefit over a good C implementation such as the one we now have. The real problem we had was that the current format conversion stuff was doing the pessimal thing in a lot of cases; a lot of that stupid is now gone. So, if someone wants to work on that, I'm curious to see their results, but I'm not holding out for it. As far as doing a JIT compile, I've thought of it. Daniel Stone recently mentioned the idea of using ORC for doing things like this and it might be a good fit. I never heard of ORC, and I didn't get any relevant hits from google. What is it? It stands for oil runtime compiler. The homepage, which seems to be down at the moment, can be found here: http://code.entropywave.com/orc/ ORC is a JIT designed for producing highly optimized streaming code that takes full advantage of CPU features such as SSE. It's designed for fairly small kernels that need to be run fast on arrays of data. I haven't taken more than a very cursory look at it, so I can't say much more. Aparently, the gstreamer people use it for all of their software filters and video transformations. However, be fore we can do that, we need a framework for these things (which we now have, thanks to this series). How is that done in gallivm? Is it done
Re: [Mesa-dev] [PATCH 00/20] Auto-generate pack/unpack functions
Hi Iago, On 18/11/14 08:43, Iago Toral Quiroga wrote: [snip] Summary of the patches: * Patches 1-7 are general fixes to the current code that were found while working on this. Have you noticed if any of those fixes resolves a piglit and/or a real world application ? If so it might be worth tagging them for the stable branches :) I've noticed with this series we require mako in order to build mesa. Please update the documentation to reflect the new dependency. How gracefully do we handle the cases when it's missing ? Printing a message similar to mako vXX or later is required would be very helpful. Thanks Emil ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
