On 2018/11/21, Kyle Brenneman wrote:
> On 11/21/2018 05:04 AM, Emil Velikov wrote:
> > From: Emil Velikov <emil.veli...@collabora.com>
> >
> > Currently we have over 20 scripts that generate the libGL* dispatch and
> > various other functionality. More importantly we're using local XML
> > files instead of the Khronos provides one(s). Resulting in an
> > increasing complexity of writing, maintaining and bugfixing.
> >
> > One fairly annoying bug is handling of statically exported symbols.
> > Today, if we enable a GL extension for GLES1/2, we add a special tag to
> > the xml. Thus the ES dispatch gets generated, but also since we have no
> > separate notion of GL/ES1/ES2 static functions it also gets exported
> > statically.
> >
> > This commit adds step one towards clearing and simplifying our setup.
> > It imports the mapi generator from GLVND.
> >
> > 012fe39 ("Remove a couple of duplicate typedefs.")
> >
> > Signed-off-by: Emil Velikov <emil.veli...@collabora.com>
> > ---
> > src/mapi/Makefile.am | 2 +
> > src/mapi/new/genCommon.py | 223 ++++++++++++++++++++++++++++
> > src/mapi/new/gen_gldispatch_mapi.py | 180 ++++++++++++++++++++++
> > 3 files changed, 405 insertions(+)
> > create mode 100644 src/mapi/new/genCommon.py
> > create mode 100755 src/mapi/new/gen_gldispatch_mapi.py
> >
> > diff --git a/src/mapi/Makefile.am b/src/mapi/Makefile.am
> > index 97ebdeb1d7f..5a619bf049b 100644
> > --- a/src/mapi/Makefile.am
> > +++ b/src/mapi/Makefile.am
> > @@ -31,6 +31,8 @@ pkgconfigdir = $(libdir)/pkgconfig
> > pkgconfig_DATA =
> > EXTRA_DIST = \
> > + new/genCommon.py \
> > + new/gen_gldispatch_mapi.py \
> > es1api/ABI-check \
> > es2api/ABI-check \
> > mapi_abi.py \
> > diff --git a/src/mapi/new/genCommon.py b/src/mapi/new/genCommon.py
> > new file mode 100644
> > index 00000000000..5c721acbaca
> > --- /dev/null
> > +++ b/src/mapi/new/genCommon.py
> > @@ -0,0 +1,223 @@
> > +#!/usr/bin/env python
> > +
> > +# (C) Copyright 2015, NVIDIA CORPORATION.
> > +# All Rights Reserved.
> > +#
> > +# Permission is hereby granted, free of charge, to any person obtaining a
> > +# copy of this software and associated documentation files (the
> > "Software"),
> > +# to deal in the Software without restriction, including without limitation
> > +# on the rights to use, copy, modify, merge, publish, distribute, sub
> > +# license, and/or sell copies of the Software, and to permit persons to
> > whom
> > +# the Software is furnished to do so, subject to the following conditions:
> > +#
> > +# The above copyright notice and this permission notice (including the next
> > +# paragraph) shall be included in all copies or substantial portions of the
> > +# Software.
> > +#
> > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
> > OR
> > +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> > +# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL
> > +# IBM AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> > +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> > +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> > DEALINGS
> > +# IN THE SOFTWARE.
> > +#
> > +# Authors:
> > +# Kyle Brenneman <kbrenne...@nvidia.com>
> > +
> > +import sys
> > +import collections
> > +import re
> > +import xml.etree.cElementTree as etree
> > +
> > +MAPI_TABLE_NUM_DYNAMIC = 4096
> > +
> > +_LIBRARY_FEATURE_NAMES = {
> > + # libGL and libGLdiapatch both include every function.
> > + "gl" : None,
> > + "gldispatch" : None,
> > + "opengl" : frozenset(( "GL_VERSION_1_0", "GL_VERSION_1_1",
> > + "GL_VERSION_1_2", "GL_VERSION_1_3", "GL_VERSION_1_4",
> > "GL_VERSION_1_5",
> > + "GL_VERSION_2_0", "GL_VERSION_2_1", "GL_VERSION_3_0",
> > "GL_VERSION_3_1",
> > + "GL_VERSION_3_2", "GL_VERSION_3_3", "GL_VERSION_4_0",
> > "GL_VERSION_4_1",
> > + "GL_VERSION_4_2", "GL_VERSION_4_3", "GL_VERSION_4_4",
> > "GL_VERSION_4_5",
> > + )),
> > + "glesv1" : frozenset(("GL_VERSION_ES_CM_1_0",
> > "GL_OES_point_size_array")),
> > + "glesv2" : frozenset(("GL_ES_VERSION_2_0", "GL_ES_VERSION_3_0",
> > + "GL_ES_VERSION_3_1", "GL_ES_VERSION_3_2",
> > + )),
> > +}
> > +
> > +def getFunctions(xmlFiles):
> > + """
> > + Reads an XML file and returns all of the functions defined in it.
> > +
> > + xmlFile should be the path to Khronos's gl.xml file. The return value
> > is a
> > + sequence of FunctionDesc objects, ordered by slot number.
> > + """
> > + roots = [ etree.parse(xmlFile).getroot() for xmlFile in xmlFiles ]
> > + return getFunctionsFromRoots(roots)
> > +
> > +def getFunctionsFromRoots(roots):
> > + functions = {}
> > + for root in roots:
> > + for func in _getFunctionList(root):
> > + functions[func.name] = func
> > + functions = functions.values()
> > +
> > + # Sort the function list by name.
> > + functions = sorted(functions, key=lambda f: f.name)
> > +
> > + # Assign a slot number to each function. This isn't strictly necessary,
> > + # since you can just look at the index in the list, but it makes it
> > easier
> > + # to include the slot when formatting output.
> > + for i in range(len(functions)):
> > + functions[i] = functions[i]._replace(slot=i)
> > +
> > + return functions
> > +
> > +def getExportNamesFromRoots(target, roots):
> > + """
> > + Goes through the <feature> tags from gl.xml and returns a set of OpenGL
> > + functions that a library should export.
> > +
> > + target should be one of "gl", "gldispatch", "opengl", "glesv1", or
> > + "glesv2".
> > + """
> > + featureNames = _LIBRARY_FEATURE_NAMES[target]
> > + if (featureNames == None):
> > + return set(func.name for func in getFunctionsFromRoots(roots))
> > +
> > + names = set()
> > + for root in roots:
> > + features = []
> > + for featElem in root.findall("feature"):
> > + if (featElem.get("name") in featureNames):
> > + features.append(featElem)
> > + for featElem in root.findall("extensions/extension"):
> > + if (featElem.get("name") in featureNames):
> > + features.append(featElem)
> > + for featElem in features:
> > + for commandElem in featElem.findall("require/command"):
> > + names.add(commandElem.get("name"))
> > + return names
> > +
> > +class FunctionArg(collections.namedtuple("FunctionArg", "type name")):
> > + @property
> > + def dec(self):
> > + """
> > + Returns a "TYPE NAME" string, suitable for a function prototype.
> > + """
> > + rv = str(self.type)
> > + if(not rv.endswith("*")):
> > + rv += " "
> > + rv += self.name
> > + return rv
> > +
> > +class FunctionDesc(collections.namedtuple("FunctionDesc", "name rt args
> > slot")):
> > + def hasReturn(self):
> > + """
> > + Returns true if the function returns a value.
> > + """
> > + return (self.rt != "void")
> > +
> > + @property
> > + def decArgs(self):
> > + """
> > + Returns a string with the types and names of the arguments, as you
> > + would use in a function declaration.
> > + """
> > + if(len(self.args) == 0):
> > + return "void"
> > + else:
> > + return ", ".join(arg.dec for arg in self.args)
> > +
> > + @property
> > + def callArgs(self):
> > + """
> > + Returns a string with the names of the arguments, as you would use
> > in a
> > + function call.
> > + """
> > + return ", ".join(arg.name for arg in self.args)
> > +
> > + @property
> > + def basename(self):
> > + assert(self.name.startswith("gl"))
> > + return self.name[2:]
> > +
> > +def _getFunctionList(root):
> > + for elem in root.findall("commands/command"):
> > + yield _parseCommandElem(elem)
> > +
> > +def _parseCommandElem(elem):
> > + protoElem = elem.find("proto")
> > + (rt, name) = _parseProtoElem(protoElem)
> > +
> > + args = []
> > + for ch in elem.findall("param"):
> > + # <param> tags have the same format as a <proto> tag.
> > + args.append(FunctionArg(*_parseProtoElem(ch)))
> > + func = FunctionDesc(name, rt, tuple(args), slot=None)
> > +
> > + return func
> > +
> > +def _parseProtoElem(elem):
> > + # If I just remove the tags and string the text together, I'll get
> > valid C code.
> > + text = _flattenText(elem)
> > + text = text.strip()
> > + m = re.match(r"^(.+)\b(\w+)(?:\s*\[\s*(\d*)\s*\])?$", text, re.S)
> > + if (m):
> > + typename = _fixupTypeName(m.group(1))
> > + name = m.group(2)
> > + if (m.group(3)):
> > + # HACK: glPathGlyphIndexRangeNV defines an argument like this:
> > + # GLuint baseAndCount[2]
> > + # Convert it to a pointer and hope for the best.
> > + typename += "*"
> > + return (typename, name)
> > + else:
> > + raise ValueError("Can't parse element %r -> %r" % (elem, text))
> > +
> > +def _flattenText(elem):
> > + """
> > + Returns the text in an element and all child elements, with the tags
> > + removed.
> > + """
> > + text = ""
> > + if(elem.text != None):
> > + text = elem.text
> > + for ch in elem:
> > + text += _flattenText(ch)
> > + if(ch.tail != None):
> > + text += ch.tail
> > + return text
> > +
> > +def _fixupTypeName(typeName):
> > + """
> > + Converts a typename into a more consistent format.
> > + """
> > +
> > + rv = typeName.strip()
> > +
> > + # Replace "GLvoid" with just plain "void".
> > + rv = re.sub(r"\bGLvoid\b", "void", rv)
> > +
> > + # Remove the vendor suffixes from types that have a suffix-less
> > version.
> > + rv =
> > re.sub(r"\b(GLhalf|GLintptr|GLsizeiptr|GLint64|GLuint64)(?:ARB|EXT|NV|ATI)\b",
> > r"\1", rv)
> > +
> > + rv = re.sub(r"\bGLvoid\b", "void", rv)
> > +
> > + # Clear out any leading and trailing whitespace.
> > + rv = rv.strip()
> > +
> > + # Remove any whitespace before a '*'
> > + rv = re.sub(r"\s+\*", r"*", rv)
> > +
> > + # Change "foo*" to "foo *"
> > + rv = re.sub(r"([^\*])\*", r"\1 *", rv)
> > +
> > + # Condense all whitespace into a single space.
> > + rv = re.sub(r"\s+", " ", rv)
> > +
> > + return rv
> > +
> > diff --git a/src/mapi/new/gen_gldispatch_mapi.py
> > b/src/mapi/new/gen_gldispatch_mapi.py
> > new file mode 100755
> > index 00000000000..03fb49a7a29
> > --- /dev/null
> > +++ b/src/mapi/new/gen_gldispatch_mapi.py
> > @@ -0,0 +1,180 @@
> > +#!/usr/bin/env python
> > +
> > +# Copyright (C) 2010 LunarG Inc.
> > +# (C) Copyright 2015, NVIDIA CORPORATION.
> > +#
> > +# Permission is hereby granted, free of charge, to any person obtaining a
> > +# copy of this software and associated documentation files (the
> > "Software"),
> > +# to deal in the Software without restriction, including without limitation
> > +# the rights to use, copy, modify, merge, publish, distribute, sublicense,
> > +# and/or sell copies of the Software, and to permit persons to whom the
> > +# Software is furnished to do so, subject to the following conditions:
> > +#
> > +# The above copyright notice and this permission notice shall be included
> > +# in all copies or substantial portions of the Software.
> > +#
> > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
> > OR
> > +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> > +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> > +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
> > OTHER
> > +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> > +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> > +# DEALINGS IN THE SOFTWARE.
> > +#
> > +# Authors:
> > +# Kyle Brenneman <kbrenne...@nvidia.com>
> > +#
> > +# Based on code ogiginally by:
> > +# Chia-I Wu <o...@lunarg.com>
> > +
> > +
> > +"""
> > +Generates the glapi_mapi_tmp.h header file from Khronos's XML file.
> > +"""
> > +
> > +import sys
> > +import xml.etree.cElementTree as etree
> > +
> > +import genCommon
> > +
> > +def _main():
> > + target = sys.argv[1]
> > + xmlFiles = sys.argv[2:]
> > +
> > + roots = [ etree.parse(filename).getroot() for filename in xmlFiles ]
> > + allFunctions = genCommon.getFunctionsFromRoots(roots)
> > +
> > + names = genCommon.getExportNamesFromRoots(target, roots)
> > + functions = [f for f in allFunctions if(f.name in names)]
> > +
> > + if (target in ("gl", "gldispatch")):
> > + assert(len(functions) == len(allFunctions))
> > + assert(all(functions[i] == allFunctions[i] for i in
> > range(len(functions))))
> > + assert(all(functions[i].slot == i for i in range(len(functions))))
> > +
> > + print(r"""
> > +/* This file is automatically generated by mapi_abi.py. Do not modify. */
> > +
> > +#ifndef _GLAPI_TMP_H_
> > +#define _GLAPI_TMP_H_
> > +typedef int GLclampx;
> > +typedef void (APIENTRY *GLDEBUGPROCKHR)(GLenum source,GLenum type,GLuint
> > id,GLenum severity,GLsizei length,const GLchar *message,const void
> > *userParam);
> > +#endif /* _GLAPI_TMP_H_ */
> > +""".lstrip("\n"))
> > +
> > + print(generate_defines(functions))
> > + print(generate_table(functions, allFunctions))
> > + print(generate_noop_array(functions))
> > + print(generate_public_stubs(functions))
> > + print(generate_public_entries(functions))
> > + print(generate_stub_asm_gcc(functions))
> > +
> > +def generate_defines(functions):
> > + text = r"""
> > +#ifdef MAPI_TMP_DEFINES
> > +#define GL_GLEXT_PROTOTYPES
> > +#include "GL/gl.h"
> > +#include "GL/glext.h"
> > +
> > +""".lstrip("\n")
> > + for func in functions:
> > + text += "GLAPI {f.rt} APIENTRY
> > {f.name}({f.decArgs});\n".format(f=func)
> > + text += "#undef MAPI_TMP_DEFINES\n"
> > + text += "#endif /* MAPI_TMP_DEFINES */\n"
> > + return text
> > +
> > +def generate_table(functions, allFunctions):
> > + text = "#ifdef MAPI_TMP_TABLE\n"
> > + text += "#define MAPI_TABLE_NUM_STATIC %d\n" % (len(allFunctions))
> > + text += "#define MAPI_TABLE_NUM_DYNAMIC %d\n" %
> > (genCommon.MAPI_TABLE_NUM_DYNAMIC,)
> > + text += "#undef MAPI_TMP_TABLE\n"
> > + text += "#endif /* MAPI_TMP_TABLE */\n"
> > + return text
> > +
> > +def generate_noop_array(functions):
> > + text = "#ifdef MAPI_TMP_NOOP_ARRAY\n"
> > + text += "#ifdef DEBUG\n\n"
> > +
> > + for func in functions:
> > + text += "static {f.rt} APIENTRY
> > noop{f.basename}({f.decArgs})\n".format(f=func)
> > + text += "{\n"
> > + if (len(func.args) > 0):
> > + text += " "
> > + for arg in func.args:
> > + text += " (void) {a.name};".format(a=arg)
> > + text += "\n"
> > + text += " noop_warn(\"{f.name}\");\n".format(f=func)
> > + if (func.hasReturn()):
> > + text += " return ({f.rt}) 0;\n".format(f=func)
> > + text += "}\n\n"
> > +
> > + text += "const mapi_func table_noop_array[] = {\n"
> > + for func in functions:
> > + text += " (mapi_func) noop{f.basename},\n".format(f=func)
> > + for i in range(genCommon.MAPI_TABLE_NUM_DYNAMIC - 1):
> > + text += " (mapi_func) noop_generic,\n"
> > + text += " (mapi_func) noop_generic\n"
> > + text += "};\n\n"
> > + text += "#else /* DEBUG */\n\n"
> > + text += "const mapi_func table_noop_array[] = {\n"
> > + for i in range(len(functions) + genCommon.MAPI_TABLE_NUM_DYNAMIC - 1):
> > + text += " (mapi_func) noop_generic,\n"
> > + text += " (mapi_func) noop_generic\n"
> > +
> > + text += "};\n\n"
> > + text += "#endif /* DEBUG */\n"
> > + text += "#undef MAPI_TMP_NOOP_ARRAY\n"
> > + text += "#endif /* MAPI_TMP_NOOP_ARRAY */\n"
> > + return text
> > +
> > +def generate_public_stubs(functions):
> > + text = "#ifdef MAPI_TMP_PUBLIC_STUBS\n"
> > +
> > + text += "static const struct mapi_stub public_stubs[] = {\n"
> > + for func in functions:
> > + text += " { \"%s\", %d, NULL },\n" % (func.name, func.slot)
> > + text += "};\n"
> > + text += "#undef MAPI_TMP_PUBLIC_STUBS\n"
> > + text += "#endif /* MAPI_TMP_PUBLIC_STUBS */\n"
> > + return text
> > +
> > +def generate_public_entries(functions):
> > + text = "#ifdef MAPI_TMP_PUBLIC_ENTRIES\n"
> > +
> > + for func in functions:
> > + retStr = ("return " if func.hasReturn() else "")
> > + text += r"""
> > +GLAPI {f.rt} APIENTRY {f.name}({f.decArgs})
> > +{{
> > + const struct _glapi_table *_tbl = entry_current_get();
> > + mapi_func _func = ((const mapi_func *) _tbl)[{f.slot}];
> > + {retStr}(({f.rt} (APIENTRY *)({f.decArgs})) _func)({f.callArgs});
> > +}}
> > +
> > +""".lstrip("\n").format(f=func, retStr=retStr)
> > +
> > + text += "\n"
> > + text += "static const mapi_func public_entries[] = {\n"
> > + for func in functions:
> > + text += " (mapi_func) %s,\n" % (func.name,)
> > + text += "};\n"
> > + text += "#undef MAPI_TMP_PUBLIC_ENTRIES\n"
> > + text += "#endif /* MAPI_TMP_PUBLIC_ENTRIES */\n"
> > + return text
> > +
> > +def generate_stub_asm_gcc(functions):
> > + text = "#ifdef MAPI_TMP_STUB_ASM_GCC\n"
> > + text += "__asm__(\n"
> > +
> > + for func in functions:
> > + text += 'STUB_ASM_ENTRY("%s")"\\n"\n' % (func.name,)
> > + text += '"\\t"STUB_ASM_CODE("%d")"\\n"\n\n' % (func.slot,)
> > +
> > + text += ");\n"
> > + text += "#undef MAPI_TMP_STUB_ASM_GCC\n"
> > + text += "#endif /* MAPI_TMP_STUB_ASM_GCC */\n"
> > + return text
> > +
> > +if (__name__ == "__main__"):
> > + _main()
> > +
> Note that Mesa already has a copy of genCommon.py checked in as
> src/egl/generate/genCommon.py, so it might make sense to keep a single copy
> that both scripts can use.
>
Right, indeed. Will do with v2.
From a very quick look what is your take on the changes in 06/18, 08/18,
09/18 and 10/18? I'd imagine that all but the first one seem clean
enough to merge back?
Can you share some ideas on how to make 06/18 more appealing?
Thanks
Emil
_______________________________________________
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
