Re: How to determine the filename for dlopen()

2017-11-29 Thread Christian Seiler

Am 2017-11-26 15:26, schrieb wf...@niif.hu:

At least I can't see any other
way to express alternative groups of library dependencies like ((libnss
and libnspr) or libssl), which would be needed for crypto plugins.


Well, if a software wants to support alternatives, then the following
would work quite well:

 - Software has an internal abstraction layer for these libraries.
   (It will need that anyway.)
 - Any integration with any of these libraries is done in plugins
   for that specific software (which are dlopen()d). The plugins
   themselves expose only the abstraction layer, but are in turn
   linked against the actual libraries.

Since the internal plugin interface between the software and the
various plugins for different libraries is something that the
authors of the software themselves control, there's never going
to be an issue there, you upgrade them in lock-step and everything
just works.

And since the plugin libraries themselves are directly linked
against the actual libraries, automatic dependency generation will
just work, as well as symbol versioning.

In Debian packaging you'd ideally want to separate out each
alternative into their own package, so that the main package doesn't
need to depend on all alternatives.

Regards,
Christian



Re: How to determine the filename for dlopen()

2017-11-26 Thread Ferenc Wágner
Guillem Jover  writes:

> On Mon, 2017-11-13 at 13:23:01 +0100, Ferenc Wágner wrote:
>
>> I'm packaging a program which wants to dlopen() some library.  It finds
>> this library via pkg-config (PKG_CHECK_MODULES).  How to best determine
>> the filename to use in the dlopen() call?  It should work cross-distro,
>> for cross-compilation and whatnot.  Is it always safe to use the SONAME
>> as the filename?  I'm currently considering something like
>> 
>> ld -shared -o dummy.so $(my_LIBS)
>> objdump -p dummy.so | fgrep NEEDED
>> 
>> coded up properly for Automake.  I'd be grateful for any insight.
>
> IMO dlopen()ing an external library that is not part of the same
> project is a practice that should be very strongly discouraged, if
> not completely abolished.
>
> Please see this very nice mail from Simon McVittie [S], my reply [G],
> and Florian Weimer's [F], for several of the reasons why.
>
>  [S] https://lists.debian.org/debian-devel/2017/03/msg00164.html
>  [G] https://lists.debian.org/debian-devel/2017/03/msg00343.html
>  [F] https://lists.debian.org/debian-devel/2017/03/msg00346.html

Thanks for pointing out this discussion, Guillem.  Fortunately, most of
those points are already taken care of in upstream: they took my patches
exposing the full set of used symbols in throwaway binaries, which I can
process with dpkg-shlibdeps.  SONAMEs are also determined automatically
(though this seems somewhat fragile).  Portability beyond BSDs isn't a
concern either.  Still, the "pedantically correct" way has some appeal,
I'll probably look into implementing it.  At least I can't see any other
way to express alternative groups of library dependencies like ((libnss
and libnspr) or libssl), which would be needed for crypto plugins.
Symbol versioning is neglected as well, as pointed out by Florian.  The
downside is the proliferation of small plugin binary packages.

BTW what are "shared libraries that define themselves to the ABI and
SONAME level" you mention?  Do you mean libraries which expose their
SONAMEs like the various libc compnents do in lib-names.h?
-- 
Thanks,
Feri



Re: How to determine the filename for dlopen()

2017-11-25 Thread Guillem Jover
Hi!

On Mon, 2017-11-13 at 13:23:01 +0100, Ferenc Wágner wrote:
> I'm packaging a program which wants to dlopen() some library.  It finds
> this library via pkg-config (PKG_CHECK_MODULES).  How to best determine
> the filename to use in the dlopen() call?  It should work cross-distro,
> for cross-compilation and whatnot.  Is it always safe to use the SONAME
> as the filename?  I'm currently considering something like
> 
> ld -shared -o dummy.so $(my_LIBS)
> objdump -p dummy.so | fgrep NEEDED
> 
> coded up properly for Automake.  I'd be grateful for any insight.

IMO dlopen()ing an external library that is not part of the same
project is a practice that should be very strongly discouraged, if
not completely abolished.

Please see this very nice mail from Simon McVittie [S], my reply [G],
and Florian Weimer's [F], for several of the reasons why.

 [S] https://lists.debian.org/debian-devel/2017/03/msg00164.html
 [G] https://lists.debian.org/debian-devel/2017/03/msg00343.html
 [F] https://lists.debian.org/debian-devel/2017/03/msg00346.html

Thanks,
Guillem



Re: How to determine the filename for dlopen()

2017-11-14 Thread Ferenc Wágner
Christian Seiler  writes:

> Am 2017-11-13 13:23, schrieb wf...@niif.hu:
>
>> I'm packaging a program which wants to dlopen() some library.  It finds
>> this library via pkg-config (PKG_CHECK_MODULES).  How to best determine
>> the filename to use in the dlopen() call?  It should work cross-distro,
>> for cross-compilation and whatnot.  Is it always safe to use the SONAME
>> as the filename?
>
> The SONAME is the right thing to do here, as that is what's encoded in
> the DT_NEEDED field by the linker.
>
>> I'm currently considering something like
>>
>> ld -shared -o dummy.so $(my_LIBS)
>> objdump -p dummy.so | fgrep NEEDED
>
> That might work, but I'm not sure that's very stable.

Hi Christian,

The main problem with this is that my_LIBS as set by PKG_CHECK_MODULES
may contain multiple libraries, and nothing guarantees that I need the
first one.  But there's probably no cure for that.  And it failed badly
on FreeBSD for some reason...  So now I need a BSD machine to log on.

> find_library(PNG_LIBRARY_FILE NAMES png)

Yes, finding the library is what I wanted to offload to ld.  It's
somewhat hairy to scan the library path, or at least I haven't found the
appropriate tool.

> Important:

[Lots of interesting stuff, thanks!]

>  - If you do manage to write some relatively generic code, I would
>urge you to contribute that to CMake as a macro, so that other
>people could also profit from it.

That probably won't happen in the near future, as I've never used CMake,
just tried to parse your code based on English and common sense. :)
-- 
Regards,
Feri



Re: How to determine the filename for dlopen()

2017-11-13 Thread Christian Seiler

Hi,

Am 2017-11-13 13:23, schrieb wf...@niif.hu:

I'm packaging a program which wants to dlopen() some library.  It finds
this library via pkg-config (PKG_CHECK_MODULES).  How to best determine
the filename to use in the dlopen() call?  It should work cross-distro,
for cross-compilation and whatnot.  Is it always safe to use the SONAME
as the filename?


The SONAME is the right thing to do here, as that is what's encoded in
the DT_NEEDED field by the linker.


I'm currently considering something like

ld -shared -o dummy.so $(my_LIBS)
objdump -p dummy.so | fgrep NEEDED


That might work, but I'm not sure that's very stable.

I've created the following example code that works for me with libpng:

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example)
enable_language(C)

find_library(PNG_LIBRARY_FILE NAMES png)
if(PNG_LIBRARY_FILE)
  execute_process(COMMAND objdump -p "${PNG_LIBRARY_FILE}"
  OUTPUT_VARIABLE PNG_CONTENTS)
  if (PNG_CONTENTS)
string(REGEX MATCH "\n[ \t]*SONAME[ \t]+([^ \t\r\n]*)"
   DUMMY "${PNG_CONTENTS}")
if (DUMMY)
  set(PNG_SONAME "${CMAKE_MATCH_1}" CACHE STRING "The SONAME of the 
PNG library")

  message(STATUS "Got libpng soname: ${PNG_SONAME}")
else()
  message(FATAL_ERROR "Could not extract SONAME from 
${PNG_LIBRARY_FILE}")

endif()
  else()
  message(FATAL_ERROR "Could not run objdump -p on 
${PNG_LIBRARY_FILE}")

  endif()
else()
  message(FATAL_ERROR "Could not find -lpng")
endif()

Important:

 - This assumes that objdump -p actually works. This is basically only
   true if you use the GNU toolchain on ELF systems. If you have another
   platform then you need to call different things:

 - On Mac OS X you need to do otool -D $LIBRARY and then parse that
   (it will give you back something like (notice the newline!)

   $filename:\n$library_id_abspath

   The base name of the second line of that output is what you're
   looking for for dlopen().

 - On other UNIX platforms I don't really know.

 - On Windows with the MinGW toolchain, when you have an import
   library (*.dll.a MinGW style or *.lib Microsoft style) then you
   may use dlltool -I $IMPORTLIB to extract the DLL name from
   that. However, MinGW does support linking directly against DLLs
   in some cases (when they were also compiled with MinGW, for
   example, and fulfill some additional criteria), and it may be
   the case that your linker finds the DLL directly if no import
   library is found, in which case dlltool -I will fail, but you
   can just use the basename of the DLL.

   Note that objdump -p does work on MinGW on Windows, but doesn't
   give you a SONAME. (It does mention the DLL name multiple times,
   but I'm not sure that's easy to parse.)

 - On Windows with MSVC I have no idea how to get the DLL name
   from an import library (*.lib), but there's definitely going
   to be a tool you'll be able to use.

 - No idea on yet other operating systems.

 - I'm hacked this together and am not sure this is the sanest way of
   parsing this in CMake... YMMV.

 - CMake might only find a static library depending on how your search
   path is set up (*.a on UNIX systems including Mac OS X, as well as
   on MinGW, but *.lib on Windows system with MSVC). On Windows the
   fact that import libraries and static libraries share the same
   extension actually makes it quite difficult to handle this case
   properly.

 - If you do manage to write some relatively generic code, I would
   urge you to contribute that to CMake as a macro, so that other
   people could also profit from it.

Regards,
Christian