Re: Export symbol lists on Linux (was Re: RFC Marking private symbols in XFree86 shared libraries as private)
Jakub Jelinek wrote: The first is a MUST list, symbols which are exported from XFree86 shared libraries now when there is no anonymous version script, are not exported when an anonymous versions script created from stock *-def.cpp file is applied and are used by some binary or shared library (including other shared libraries in the XFree86 collection). There is IMHO no way other than adding these to *-def.cpp files (any issues with this)? For libGL.so, as anonymous version scripts accept wildcards, I think we should use gl* wildcard, as it is too error-prone to list all the gl* functions. Sorry for taking so long to reply. I was taking a few days off. :) libGL.so needs to export XF86DRI*, __glXFindDRIScreen, and a few _glapi functions on all platforms that support DRI (i.e., Linux and *BSD currently). Do a nm /usr/X11R6/lib/modules/dri/*_dri.so | grep ' U _glapi' | sort -u to see which ones. On all platforms all symbols matching gl[A-Z]* need to be exported. Other than that I don't think anything needs to be exported by libGL.so. I *believe* that the *_dri.so files only need to export __driCreateScreen. There are some other symbols that need to be exported in DRI CVS, but that code isn't in XFree86 CVS AFAIK (and won't be until after 4.4.0). Thanks for tackling this! ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: Export symbol lists on Linux (was Re: RFC Marking private symbols in XFree86 shared libraries as private)
On Tue, Oct 14, 2003 at 09:50:07PM +0200, Jakub Jelinek wrote: I'd say it would be better to reuse *-def.cpp files (didn't know something like that existed). I've preprocessed all *-def.cpp files included in XFree86/xc/lib, gathered all symbols currently exported from XFree86 shared libraries, all undefined symbols in 5800 shared libraries and binaries I found on my box which are satisfied by one of XFree86 shared libraries and attached are results. The first is a MUST list, symbols which are exported from XFree86 shared libraries now when there is no anonymous version script, are not exported when an anonymous versions script created from stock *-def.cpp file is applied and are used by some binary or shared library (including other shared libraries in the XFree86 collection). There is IMHO no way other than adding these to *-def.cpp files (any issues with this)? There is a good chance that some of these are unintentionally omitted from the -def.cpp files. That's less likely to be a problem as more platforms begin using this data. Checking the MUST list against the API specs is the best way to determine which are unintentional. Hopefully that will result in a much smaller list. That list is interesting from the point of view of reconciling the specs with actual usage. It is also possible that the -def.cpp export some symbols that shouldn't be exported, and that check against the API specs needs to be made too. For libGL.so, as anonymous version scripts accept wildcards, I think we should use gl* wildcard, as it is too error-prone to list all the gl* functions. Will the wildcard method work for the platforms that currently use the -def.cpp lists? The GL and GLX APIs should be well-defined. For libGLU.so, I think we should export everything, no version script on Linux. Second is a MAY list. These are symbols ATM exported from the shared libraries, which would be hidden by linker script but which looked to me like they are in the standard namespace of the libraries and thus might be good candidates for exports. Can anyone please review these and tell me if there are some which definitely shouldn't be exported? They need to be checked against the API specs too. I can supply a tarball with all the symbols ATM exported, exported via *-def.cpp, used etc. to interested parties if you want to do more investigations. I'll follow up with a patch which exports MUST + current *-def.cpp ones and will wait for responses about the MAY list (or candidates not even on MAY list). Jakub -- MUST list -- libGL.so XF86DRIAuthConnection XF86DRICloseConnection XF86DRICloseFullScreen XF86DRICreateContext XF86DRICreateDrawable XF86DRIDestroyContext XF86DRIDestroyDrawable XF86DRIGetClientDriverName XF86DRIGetDeviceInfo XF86DRIGetDrawableInfo XF86DRIOpenConnection XF86DRIOpenFullScreen XF86DRIQueryDirectRenderingCapable XF86DRIQueryVersion All of those are used by the DRI driver modules. I believe there are plans for having those modules use another mechanism for accessing these functions, but they'd still need to be exported for compatibility. Another alternative might be splitting them into a separate shared library that our libGL dlopen's. I don't know if any applications directly access the DRI extension. libXext.so XShmAttach XShmCreateImage XShmCreatePixmap XShmDetach XShmGetEventBase XShmGetImage XShmPixmapFormat XShmPutImage XShmQueryExtension XShmQueryVersion Those definitely get added. -- MAY list -- libX11.so KeySymToUcs4 XkbChangeKeycodeRange Xutf8DrawImageString Xutf8DrawString Xutf8DrawText Xutf8LookupString Xutf8ResetIC Xutf8SetWMProperties Xutf8TextEscapement Xutf8TextExtents Xutf8TextPerCharExtents The Xutf8 functions are part of our extensions to Xlib, so should be exported. Hopefully others can give you some more feedback about the areas they are more familiar with. David -- David Dawes X-Oz Technologies www.XFree86.org/~dawes www.x-oz.com ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Export symbol lists on Linux (was Re: RFC Marking private symbols in XFree86 shared libraries as private)
I'd say it would be better to reuse *-def.cpp files (didn't know something like that existed). I've preprocessed all *-def.cpp files included in XFree86/xc/lib, gathered all symbols currently exported from XFree86 shared libraries, all undefined symbols in 5800 shared libraries and binaries I found on my box which are satisfied by one of XFree86 shared libraries and attached are results. The first is a MUST list, symbols which are exported from XFree86 shared libraries now when there is no anonymous version script, are not exported when an anonymous versions script created from stock *-def.cpp file is applied and are used by some binary or shared library (including other shared libraries in the XFree86 collection). There is IMHO no way other than adding these to *-def.cpp files (any issues with this)? For libGL.so, as anonymous version scripts accept wildcards, I think we should use gl* wildcard, as it is too error-prone to list all the gl* functions. For libGLU.so, I think we should export everything, no version script on Linux. Second is a MAY list. These are symbols ATM exported from the shared libraries, which would be hidden by linker script but which looked to me like they are in the standard namespace of the libraries and thus might be good candidates for exports. Can anyone please review these and tell me if there are some which definitely shouldn't be exported? I can supply a tarball with all the symbols ATM exported, exported via *-def.cpp, used etc. to interested parties if you want to do more investigations. I'll follow up with a patch which exports MUST + current *-def.cpp ones and will wait for responses about the MAY list (or candidates not even on MAY list). Jakub libGL.so XF86DRIAuthConnection XF86DRICloseConnection XF86DRICloseFullScreen XF86DRICreateContext XF86DRICreateDrawable XF86DRIDestroyContext XF86DRIDestroyDrawable XF86DRIGetClientDriverName XF86DRIGetDeviceInfo XF86DRIGetDrawableInfo XF86DRIOpenConnection XF86DRIOpenFullScreen XF86DRIQueryDirectRenderingCapable XF86DRIQueryVersion __glXFindDRIScreen __glXInitialize _gl_tls_Context _glapi_Context _glapi_add_entrypoint _glapi_check_multithread _glapi_get_context _glapi_get_dispatch_table_size _glapi_noop_enable_warnings _glapi_set_context _glapi_set_dispatch _glthread_GetID glColorSubTable glColorTable glColorTableEXT glConvolutionFilter1D glConvolutionFilter2D glFogCoorddvEXT glFogCoordfEXT glFogCoordfvEXT glMultiTexCoord1dvARB glMultiTexCoord1fARB glMultiTexCoord1fvARB glMultiTexCoord1ivARB glMultiTexCoord1svARB glMultiTexCoord2dvARB glMultiTexCoord2fARB glMultiTexCoord2fvARB glMultiTexCoord2ivARB glMultiTexCoord2svARB glMultiTexCoord3dvARB glMultiTexCoord3fARB glMultiTexCoord3fvARB glMultiTexCoord3ivARB glMultiTexCoord3svARB glMultiTexCoord4dvARB glMultiTexCoord4fARB glMultiTexCoord4fvARB glMultiTexCoord4ivARB glMultiTexCoord4svARB glSecondaryColor3bvEXT glSecondaryColor3dvEXT glSecondaryColor3fEXT glSecondaryColor3fvEXT glSecondaryColor3ivEXT glSecondaryColor3svEXT glSecondaryColor3ubEXT glSecondaryColor3ubvEXT glSecondaryColor3uivEXT glSecondaryColor3usvEXT libGLU.so export all? libICE.so _IceTransNoListen libXaw.so.7 _XawTextGetSTRING libXaw.so.6 _XawTextGetSTRING libXcursor.so XcursorImageHash libXext.so XShmAttach XShmCreateImage XShmCreatePixmap XShmDetach XShmGetEventBase XShmGetImage XShmPixmapFormat XShmPutImage XShmQueryExtension XShmQueryVersion libXrandr.so XRRConfigCurrentConfiguration XRRConfigCurrentRate XRRConfigRates XRRConfigRotations XRRConfigSizes XRRFreeScreenConfigInfo XRRRates XRRSelectInput XRRSetScreenConfigAndRate XRRUpdateConfiguration libXrender.so XRenderCreateAnimCursor libXt.so _XtCountVaList _XtDefaultWarningMsg _XtVaToArgList colorConvertArgs libGL.so DRI_glXUseXFont XF86DRIQueryExtension __glFreeAttributeState __glXClientInfo __glXCloseDisplay __glXDebug __glXExtensionInfo __glXFlushRenderBuffer __glXFreeContext __glXGetCurrentContext __glXInitVertexArrayState __glXNewIndirectAPI __glXRegisterExtensions __glXRegisterGLXExtensionString __glXRegisterGLXFunction __glXSendLargeCommand __glXSetCurrentContext
Re: RFC Marking private symbols in XFree86 shared libraries as private
On Thu, Oct 09, 2003 at 09:01:15PM -0400, David Dawes wrote: Well, both version script and __attribute__((visibility (hidden))) can be used at the same time. Anonymous version script works in 2001-12-18 and later binutils and AFAIK in Solaris linker. No idea about other arches. If you think we should list all exported symbols, then maybe we could introduce .sym (or .api) files for each library which would list exported symbols one per line and Imakefile hacks to generate version scripts/whatever else on the fly and use it during linking. That sounds like a good approach. There are two ways that this is already done for some platforms. OS/2 and Cygwin use files like X11-def.cpp to list the exported symbols. Those lists might be a good starting point, but I suspect that more symbols are exported than are documented in the API specs. Also, X11R6.3 added export description files (like libX11.elist), together with scripts for a few platforms like Solaris, HPUX and AIX to process them. The .elist files are only provided for libX11 and libXt, and they simply export all globals. I don't know if the basic mechanism is worth reusing -- you should look at it and see what you think. They don't provide any useful information about what symbols should be exported for each library though. I'd say it would be better to reuse *-def.cpp files (didn't know something like that existed). They are preprocessed, so it is easy to add __linux__, whatever else, also arch specific symbols, whatever necessary. Transforming *-def.cpp files to anonymous version scripts should be trivial, whether done in lnxLib.rules, sunLib.rules or some special script. Given that there are just a handful of -shared links in whole XFree86 build, perhaps the rule can (in lnxLib.rules) first check whether ld actually supports anynymous version scripts and only use them if it does. Second thing is how to define Imakefile macros which will enable __attribute__((visibility (hidden))) and/or version scripts. Should they be just user settable, defaulting to no, or should imake do the autodetection (for that it needs an autoconf like test, since e.g. even when you have GCC 3.3 and later, still visibility attribute doesn't have to be supported if that GCC was compiled against old binutils, etc.)? What happens if you use the visibility attribute with a gcc 3.3 that doesn't have it enabled? If it's simply ignored, then it shouldn't be an issue. Otherwise, is there no pre-defined macro so that you can test for it in the source? There is no predefined macro, the attribute is ignored if not supported, but with a warning: If it is GCC which has visibility attribute support but that attribute is not supported in its particular configuration (linked against old binutils, not using binutils or architectire which doesn't support it), you get: visibility attribute not supported in this configuration; ignored warning, if it is GCC without visibility attribute support (e.g. older releases), you get: `visibility' attribute directive ignored warning. I don't know how much XFree86 relies on being warning free. GCC 3.3+ on Linux will typically be configured such that this attribute is supported and give no warning (dunno about *BSD etc.), so it would issue warnings just in a few configurations. Jakub ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: RFC Marking private symbols in XFree86 shared libraries as private
Jakub, I just noticed this thread today. If this will have problems, then they will definitely be visible on Cygwin. So, I ask that I please be included in the testing process before this is committed, if it ever is. I would gladly do test builds under Cygwin to confirm that the new scheme works. Harold Jakub Jelinek wrote: On Thu, Oct 09, 2003 at 09:01:15PM -0400, David Dawes wrote: Well, both version script and __attribute__((visibility (hidden))) can be used at the same time. Anonymous version script works in 2001-12-18 and later binutils and AFAIK in Solaris linker. No idea about other arches. If you think we should list all exported symbols, then maybe we could introduce .sym (or .api) files for each library which would list exported symbols one per line and Imakefile hacks to generate version scripts/whatever else on the fly and use it during linking. That sounds like a good approach. There are two ways that this is already done for some platforms. OS/2 and Cygwin use files like X11-def.cpp to list the exported symbols. Those lists might be a good starting point, but I suspect that more symbols are exported than are documented in the API specs. Also, X11R6.3 added export description files (like libX11.elist), together with scripts for a few platforms like Solaris, HPUX and AIX to process them. The .elist files are only provided for libX11 and libXt, and they simply export all globals. I don't know if the basic mechanism is worth reusing -- you should look at it and see what you think. They don't provide any useful information about what symbols should be exported for each library though. I'd say it would be better to reuse *-def.cpp files (didn't know something like that existed). They are preprocessed, so it is easy to add __linux__, whatever else, also arch specific symbols, whatever necessary. Transforming *-def.cpp files to anonymous version scripts should be trivial, whether done in lnxLib.rules, sunLib.rules or some special script. Given that there are just a handful of -shared links in whole XFree86 build, perhaps the rule can (in lnxLib.rules) first check whether ld actually supports anynymous version scripts and only use them if it does. Second thing is how to define Imakefile macros which will enable __attribute__((visibility (hidden))) and/or version scripts. Should they be just user settable, defaulting to no, or should imake do the autodetection (for that it needs an autoconf like test, since e.g. even when you have GCC 3.3 and later, still visibility attribute doesn't have to be supported if that GCC was compiled against old binutils, etc.)? What happens if you use the visibility attribute with a gcc 3.3 that doesn't have it enabled? If it's simply ignored, then it shouldn't be an issue. Otherwise, is there no pre-defined macro so that you can test for it in the source? There is no predefined macro, the attribute is ignored if not supported, but with a warning: If it is GCC which has visibility attribute support but that attribute is not supported in its particular configuration (linked against old binutils, not using binutils or architectire which doesn't support it), you get: visibility attribute not supported in this configuration; ignored warning, if it is GCC without visibility attribute support (e.g. older releases), you get: `visibility' attribute directive ignored warning. I don't know how much XFree86 relies on being warning free. GCC 3.3+ on Linux will typically be configured such that this attribute is supported and give no warning (dunno about *BSD etc.), so it would issue warnings just in a few configurations. Jakub ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: RFC Marking private symbols in XFree86 shared libraries as private
Jakub Jelinek wrote: 1) could be done by some header which everything uses, doing #if defined HAVE_VISIBILITY_ATTRIBUTE defined __PIC__ #define hidden __attribute__((visibility (hidden))) #else #define hidden /**/ #endif and write prototypes like: void hidden someshlibprivateroutine (void); extern int someshlibprivatevar hidden; etc. I sent you a message about this before (in reference to libGL.so), but I never heard back from you. I think this is a very good idea! I would prefer it if __HIDDEN__ or HIDDEN or something similar were used. That makes it stand out more. Also, is there any reason to not have the symbols be hidden in non-PIC mode? ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: RFC Marking private symbols in XFree86 shared libraries as private
On Thu, Oct 09, 2003 at 02:05:47PM +0200, Jakub Jelinek wrote: Looking at various /usr/X11R6/lib/lib*.so* shared libraries, I'm seeing lots of exported symbols which look like they are exported just because they cannot be static (as they are used by some other .o files in the same shared library, but never by anything outside). That is correct. If these could be classified (e.g. some _X* symbols in libX11.so.6, some _Ice* symbols in libICE.so.6 etc. might fall into this category), XFree86 could have smaller and faster shared libraries with less dynamic relocations, which would not need to set up PIC pointer so often and use less instructions to access shared library local variables (by bypassing GOT on some arches). ... and it would discourage applications from using symbols that aren't part of the published interfaces. I'd suggest starting with the library interface specs, exporting only those symbols that are documented as part of the interfaces, and seeing what applications break. If a signficant number of applications do break, that might suggest adjusting the specs to agree with common usage. I had plans of doing this with a linker version script, but it's still fairly low down on my todo list. From an API point of view, I think it would be better to specify the symbols that should be public, rather than identifying all non-statics that shouldn't be. It'd also be good if this could be implemented for as many of the platforms we support as possible. David -- David Dawes Founder/committer/developer The XFree86 Project www.XFree86.org/~dawes ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: RFC Marking private symbols in XFree86 shared libraries as private
On Thu, Oct 09, 2003 at 12:08:55PM -0400, David Dawes wrote: On Thu, Oct 09, 2003 at 02:05:47PM +0200, Jakub Jelinek wrote: Looking at various /usr/X11R6/lib/lib*.so* shared libraries, I'm seeing lots of exported symbols which look like they are exported just because they cannot be static (as they are used by some other .o files in the same shared library, but never by anything outside). That is correct. If these could be classified (e.g. some _X* symbols in libX11.so.6, some _Ice* symbols in libICE.so.6 etc. might fall into this category), XFree86 could have smaller and faster shared libraries with less dynamic relocations, which would not need to set up PIC pointer so often and use less instructions to access shared library local variables (by bypassing GOT on some arches). ... and it would discourage applications from using symbols that aren't part of the published interfaces. I'd suggest starting with the library interface specs, exporting only those symbols that are documented as part of the interfaces, and seeing what applications break. If a signficant number of applications do break, that might suggest adjusting the specs to agree with common usage. I had plans of doing this with a linker version script, but it's still fairly low down on my todo list. From an API point of view, I think it would be better to specify the symbols that should be public, rather than identifying all non-statics that shouldn't be. It'd also be good if this could be implemented for as many of the platforms we support as possible. Well, both version script and __attribute__((visibility (hidden))) can be used at the same time. Anonymous version script works in 2001-12-18 and later binutils and AFAIK in Solaris linker. No idea about other arches. If you think we should list all exported symbols, then maybe we could introduce .sym (or .api) files for each library which would list exported symbols one per line and Imakefile hacks to generate version scripts/whatever else on the fly and use it during linking. In addition to that, at least non-static but not exported variables should be marked with X11_LIBRARY_LOCAL (or whatever other macro you prefer; __HIDDEN__ is IMHO bad since it is OS implementation namespace), for functions unless their address is taken it is much less important. Now, is there some easy way how can the initial .api/.sym files be created from the specs (exported headers or doc/specs or something else)? When that's done, I can surely first see what other symbols other XFree86 libraries need and then harvest some Everything install distribution for other symbols. Second thing is how to define Imakefile macros which will enable __attribute__((visibility (hidden))) and/or version scripts. Should they be just user settable, defaulting to no, or should imake do the autodetection (for that it needs an autoconf like test, since e.g. even when you have GCC 3.3 and later, still visibility attribute doesn't have to be supported if that GCC was compiled against old binutils, etc.)? Jakub ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
Re: RFC Marking private symbols in XFree86 shared libraries as private
On Thu, Oct 09, 2003 at 04:55:25PM +0200, Jakub Jelinek wrote: On Thu, Oct 09, 2003 at 12:08:55PM -0400, David Dawes wrote: On Thu, Oct 09, 2003 at 02:05:47PM +0200, Jakub Jelinek wrote: Looking at various /usr/X11R6/lib/lib*.so* shared libraries, I'm seeing lots of exported symbols which look like they are exported just because they cannot be static (as they are used by some other .o files in the same shared library, but never by anything outside). That is correct. If these could be classified (e.g. some _X* symbols in libX11.so.6, some _Ice* symbols in libICE.so.6 etc. might fall into this category), XFree86 could have smaller and faster shared libraries with less dynamic relocations, which would not need to set up PIC pointer so often and use less instructions to access shared library local variables (by bypassing GOT on some arches). ... and it would discourage applications from using symbols that aren't part of the published interfaces. I'd suggest starting with the library interface specs, exporting only those symbols that are documented as part of the interfaces, and seeing what applications break. If a signficant number of applications do break, that might suggest adjusting the specs to agree with common usage. I had plans of doing this with a linker version script, but it's still fairly low down on my todo list. From an API point of view, I think it would be better to specify the symbols that should be public, rather than identifying all non-statics that shouldn't be. It'd also be good if this could be implemented for as many of the platforms we support as possible. Well, both version script and __attribute__((visibility (hidden))) can be used at the same time. Anonymous version script works in 2001-12-18 and later binutils and AFAIK in Solaris linker. No idea about other arches. If you think we should list all exported symbols, then maybe we could introduce .sym (or .api) files for each library which would list exported symbols one per line and Imakefile hacks to generate version scripts/whatever else on the fly and use it during linking. That sounds like a good approach. There are two ways that this is already done for some platforms. OS/2 and Cygwin use files like X11-def.cpp to list the exported symbols. Those lists might be a good starting point, but I suspect that more symbols are exported than are documented in the API specs. Also, X11R6.3 added export description files (like libX11.elist), together with scripts for a few platforms like Solaris, HPUX and AIX to process them. The .elist files are only provided for libX11 and libXt, and they simply export all globals. I don't know if the basic mechanism is worth reusing -- you should look at it and see what you think. They don't provide any useful information about what symbols should be exported for each library though. In addition to that, at least non-static but not exported variables should be marked with X11_LIBRARY_LOCAL (or whatever other macro you prefer; __HIDDEN__ is IMHO bad since it is OS implementation namespace), for functions unless their address is taken it is much less important. Now, is there some easy way how can the initial .api/.sym files be created from the specs (exported headers or doc/specs or something else)? When that's done, I can surely first see what other symbols other XFree86 libraries need and then harvest some Everything install distribution for other symbols. The *-def.cpp files are probably as good a starting place as any, since these are actually used for the OS/2 and Cygwin ports. Second thing is how to define Imakefile macros which will enable __attribute__((visibility (hidden))) and/or version scripts. Should they be just user settable, defaulting to no, or should imake do the autodetection (for that it needs an autoconf like test, since e.g. even when you have GCC 3.3 and later, still visibility attribute doesn't have to be supported if that GCC was compiled against old binutils, etc.)? What happens if you use the visibility attribute with a gcc 3.3 that doesn't have it enabled? If it's simply ignored, then it shouldn't be an issue. Otherwise, is there no pre-defined macro so that you can test for it in the source? If a gcc version check plus gcc feature macro check isn't an option, then I'd suggest adding an imake switch for now, and default it to off. Autodetection could be added to imake too. It'd be nicer if we had a mechanism for caching that information so that imake could do these tests once rather than hundreds of times per build. But nobody seems interested in the incremental approaches for achieving auto-config'd XFree86 builds :-(. But I digress. David -- David Dawes X-Oz Technologies www.XFree86.org/~dawes www.x-oz.com ___ Devel mailing list [EMAIL PROTECTED] http://XFree86.Org/mailman/listinfo/devel
