Re: [Mesa-dev] [wip 0/9] GL_OES_get_program_binary extension
On 2 January 2014 03:58, Tapani Pälli wrote: > Hi; > > Here's another take on the get_program_binary extension. I've rewritten big > chunks of it based on Paul's review and comments earlier. Here's a brief > list > out of my head of things changed: > > - uses mesa compilation time as verification method for cache, not git sha > - much smaller serialization code size, now as part of IR classes > - smaller binary blob size > - does not dump separate 'prototypes block' for functions but instead > deserialization iterates over the whole blob for functions first > - hashtables written faster than previously (iterated only once) > - no 'unique_id' for ir_variables but using the address as identifier > - patch set broken down to more individual patches/changes > > Mesa branch with these patches applied on top: > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=oes_get_program_binary > > I have plans for the addition of driver backend data but I was hoping this > to > be as a separate addition on top. Basically I extend gl_shader_program > struct > and offer API for drivers to dump their binary data there, some proto here: > > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=driver_interface > (in order to work more data needed like the keys but this is just test) > > I have also 'automatic' cache implementation here that can be used to > verify > the cache functionality with any application (without extension): > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=new_serialization > > Any comments appreciated; > > Tapani Pälli (9): > glsl: memory_writer helper class for data serialization > glsl: serialize methods for IR instructions > glsl: memory_map helper class for data deserialization > glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache > glsl: export populate_symbol_table function > glsl: ir_deserializer class for the binary shader cache > mesa: iterate method for string_to_uint_map > glsl: functions to serialize gl_shader and gl_shader_program > mesa: OES_get_program_binary extension functionality > I sent comments on all patches but 4, 5, and 7. Those patches are: Reviewed-by: Paul Berry I see that you've already responded to some of my comments. I'm on a business trip this week so I have limited time to answer email, but I'll try to respond to your responses as soon as I can. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [wip 0/9] GL_OES_get_program_binary extension
On 01/02/2014 03:58 AM, Tapani Pälli wrote: > Hi; > > Here's another take on the get_program_binary extension. I've rewritten big > chunks of it based on Paul's review and comments earlier. Here's a brief list > out of my head of things changed: > > - uses mesa compilation time as verification method for cache, not git sha > - much smaller serialization code size, now as part of IR classes > - smaller binary blob size > - does not dump separate 'prototypes block' for functions but instead > deserialization iterates over the whole blob for functions first > - hashtables written faster than previously (iterated only once) > - no 'unique_id' for ir_variables but using the address as identifier > - patch set broken down to more individual patches/changes > > Mesa branch with these patches applied on top: > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=oes_get_program_binary > > I have plans for the addition of driver backend data but I was hoping this to > be as a separate addition on top. Basically I extend gl_shader_program struct > and offer API for drivers to dump their binary data there, some proto here: > > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=driver_interface > (in order to work more data needed like the keys but this is just test) > > I have also 'automatic' cache implementation here that can be used to verify > the cache functionality with any application (without extension): > http://cgit.freedesktop.org/~tpalli/mesa/log/?h=new_serialization > > Any comments appreciated; > > Tapani Pälli (9): > glsl: memory_writer helper class for data serialization > glsl: serialize methods for IR instructions > glsl: memory_map helper class for data deserialization > glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache > glsl: export populate_symbol_table function > glsl: ir_deserializer class for the binary shader cache > mesa: iterate method for string_to_uint_map > glsl: functions to serialize gl_shader and gl_shader_program > mesa: OES_get_program_binary extension functionality Skimming through Paul's review comments...it sounds like there are a lot of cases where your code has not handled invalid data gracefully. That's not a critique of you or your code - there are a million cases to consider, and it's really easy to miss one. But, it sounds like there might be a lot of value in fuzz-testing this extension (i.e. providing random data, or data with some structure but invalid in specific places...). We should probably be as mean as possible in our testing up front to avoid security issues later. --Ken ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] [wip 0/9] GL_OES_get_program_binary extension
Hi; Here's another take on the get_program_binary extension. I've rewritten big chunks of it based on Paul's review and comments earlier. Here's a brief list out of my head of things changed: - uses mesa compilation time as verification method for cache, not git sha - much smaller serialization code size, now as part of IR classes - smaller binary blob size - does not dump separate 'prototypes block' for functions but instead deserialization iterates over the whole blob for functions first - hashtables written faster than previously (iterated only once) - no 'unique_id' for ir_variables but using the address as identifier - patch set broken down to more individual patches/changes Mesa branch with these patches applied on top: http://cgit.freedesktop.org/~tpalli/mesa/log/?h=oes_get_program_binary I have plans for the addition of driver backend data but I was hoping this to be as a separate addition on top. Basically I extend gl_shader_program struct and offer API for drivers to dump their binary data there, some proto here: http://cgit.freedesktop.org/~tpalli/mesa/log/?h=driver_interface (in order to work more data needed like the keys but this is just test) I have also 'automatic' cache implementation here that can be used to verify the cache functionality with any application (without extension): http://cgit.freedesktop.org/~tpalli/mesa/log/?h=new_serialization Any comments appreciated; Tapani Pälli (9): glsl: memory_writer helper class for data serialization glsl: serialize methods for IR instructions glsl: memory_map helper class for data deserialization glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache glsl: export populate_symbol_table function glsl: ir_deserializer class for the binary shader cache mesa: iterate method for string_to_uint_map glsl: functions to serialize gl_shader and gl_shader_program mesa: OES_get_program_binary extension functionality src/glsl/Makefile.sources | 3 + src/glsl/ir.h | 45 ++ src/glsl/ir_deserializer.cpp | 979 ++ src/glsl/ir_deserializer.h| 142 ++ src/glsl/ir_serialize.cpp | 392 + src/glsl/ir_serialize.h | 35 ++ src/glsl/linker.cpp | 2 +- src/glsl/linker.h | 3 + src/glsl/memory_map.h | 174 src/glsl/memory_writer.h | 147 +++ src/glsl/shader_cache.cpp | 489 + src/glsl/shader_cache.h | 58 +++ src/glsl/shader_cache_magic.h | 36 ++ src/mesa/main/shaderapi.c | 44 +- src/mesa/program/hash_table.h | 8 + 15 files changed, 2550 insertions(+), 7 deletions(-) create mode 100644 src/glsl/ir_deserializer.cpp create mode 100644 src/glsl/ir_deserializer.h create mode 100644 src/glsl/ir_serialize.cpp create mode 100644 src/glsl/ir_serialize.h create mode 100644 src/glsl/memory_map.h create mode 100644 src/glsl/memory_writer.h create mode 100644 src/glsl/shader_cache.cpp create mode 100644 src/glsl/shader_cache.h create mode 100644 src/glsl/shader_cache_magic.h -- 1.8.3.1 ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev