Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Russ Allbery]

Simon McVittie  writes:

> For some libraries, the only maintainer-supported way to consume the
> library is via pkg-config. If that's the case, then a dependency on
> pkg-config can be appropriate - although we don't add a dependency on
> cc or binutils, which is equally necessary.

Well, cc and binutils are in build-essential, so this isn't entirely
equivalent.

> For other libraries, either there are other supported ways to consume
> the library (CMake metadata or sdl2-config or similar), or the library
> is in the compiler's default search paths (not parallel-installable)
> like libjpeg - so you *can* use pkg-config, but you don't *have* to.

Yeah, I think this is the key point: it's entirely possible to use most
libraries without pkg-config because they're installed into the default
search paths, so you can just... use them.

If they *require* special flags (non-standard paths, non-standard compiler
flags, etc.) such that pkg-config is the only supported interface, then I
think one could make a good argument that pkg-config should be a
dependency of the -dev package.

> That's why I think it is best that the information about which libraries
> GLib depends on is shipped with GLib and included by reference (via
> either Requires.private: glib-2.0, or the Requires.internal: glib-2.0
> proposed on https://bugs.freedesktop.org/show_bug.cgi?id=105572),
> instead of copying the information about GLib's dependencies into
> libfoo-dev where it will become outdated when GLib changes.

Yes.

-- 
Russ Allbery (r...@debian.org)  

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Simon McVittie]

On Thu, 09 Dec 2021 at 15:24:27 +0100, Alexander Traud wrote:
> if the header included another header, 
> and that header included further headers but was not in the root but in 
> a subfolder, an -I flag *might* be required. For example, the package 
> 'libopusfile-dev' has its header file in '/usr/include/opus/opusfile.h' 
> with #include  which is part of the package 
> 'libopus-dev' and is not within '/usr/include/' but within 
> '/usr/include/opus/'. Therefore,
> $ pkg-config --cflags opusfile
> -I/usr/include/opus

This is deliberate, and is a very large part of the purpose of pkg-config.

Users of libraries should #include the library's headers in whatever
way the library maintainer has documented them to be used. The correct
#include pattern is part of the library's API. For example, for GTK,
the documentation says to always #include , and using
 would be wrong. For SDL2, it's correctly ,
not .

GTK has multiple major versions, which are incompatible and
parallel-installable. If your program uses GTK 3, it needs to see GTK 3
headers: using GTK 2 or 4 headers would not work. This is implemented
by keeping the GTK 2, 3 and 4 headers in different non-default
directories, so that you only get the version of  you asked
for, and not the others. SDL 1.2 and 2 are similar, and there are many
other libraries with a similar design. The canonical text on this is
https://ometer.com/parallel.html (and you can tell how old it is because
it talks about GNOME 2 as though that was a new thing - but it's still
relevant).

Not doing that would give us the situation we have for things like libjpeg,
libcurl and libopenssl, where you can only have one version of the
development files installed system-wide at a time: if you have a program
that needs OpenSSL 1.1, and another program that needs OpenSSL 1.0, you
have to choose which one you can compile (by installing libssl-dev or
libssl1.0-dev), and you cannot compile the other one.

Conversely, if the old and new versions of library are *mostly* compatible
(like the way some SDL programs such as src:quakespasm can compile against
either SDL 1.2 or 2 with a few #ifdefs), we don't want to have to change
every source file to do something like this:

#ifdef USING_MYLIB_3
#include 
#else
#include 
#endif

which explains why GTK and SDL have chosen to document 
and  respectively, instead of a versioned name appearing in the
source code.

For libraries with many smaller headers (popular in C++, e.g. Qt,
because parsing C++ is more expensive than parsing C), instead of one
big meta-header like the one in GTK, you certainly wouldn't want to do:

#ifdef USING_MYLIB_3
#include 
#include 
#include 
...
#else
#include 
#include 
#include 
...
#endif

The other major part of the purpose of pkg-config is that it lets you
use dependencies that are *not* installed system-wide, without having to
make source code changes: if I set the PKG_CONFIG_PATH correctly, it will
find a GTK installation in /opt/gnome-42/{include,lib} and use that,
with my source code still using #include  as usual.

smcv

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Simon McVittie]

On Fri, 17 Dec 2021 at 10:05:39 +0100, Alexander Traud wrote:
> I do not understand why:
> a) pkg-config itself is *not* a dependency as well

Sometimes it is appropriate for it to be, and sometimes it is not.

For some libraries, the only maintainer-supported way to consume the
library is via pkg-config. If that's the case, then a dependency on
pkg-config can be appropriate - although we don't add a dependency on
cc or binutils, which is equally necessary. Libraries with multiple
parallel-installable versions, like GTK, often behave like this: you have
to explicitly ask (via pkg-config) to add the version you want to use to
the compiler's search paths, so that you will not get the 
from GTK 2 or 4 where GTK 3 was expected.

For other libraries, either there are other supported ways to consume
the library (CMake metadata or sdl2-config or similar), or the library
is in the compiler's default search paths (not parallel-installable)
like libjpeg - so you *can* use pkg-config, but you don't *have* to.

> b) Lintian was not upgraded in 15 years to check the content of the .pc 
>file and aid the maintainer in not missing any required dependency.

Lintian cannot know whether the required dependencies are present: that
is outside its scope. Lintian analyzes each package in isolation, without
referring to other packages. If your library depends on foo-bar, Lintian
cannot know whether foo-bar.pc is in libfoo-bar-dev, libfoo-bar2-dev,
libfoo-dev, libfoo2-dev or something else.

> Simon McVittie wrote:
> > this doesn't scale well
> 
> Then, I do not understand static linking. I thought, please correct me 
> here, static linking is about the '.a' file in the package, the current 
> one. I have to suffice its links as it was built, not the links when I 
> build.

Imagine you are packaging a library "libfoo" that makes use of GLib
(glib-2.0.pc, from src:glib2.0) but does not expose GLib in its header
files. This is an example of the class of dependency that you want
to move from Requires.private to Libs.private.

glib-2.0 currently depends on libpcre, which it uses to implement GRegex.
It does not include libpcre headers in its header files (the GRegex API
completely wraps libpcre), so libpcre is another example of the class of
dependency that you want to move from Requires.private to Libs.private.
Imagine that GLib had done this.

When you statically link your library libfoo into a program, you want to
end up with something like this:

gcc -static ... -lfoo -lglib-2.0 -lpcre

libfoo.a provides symbols used by the program, libglib-2.0.a provides
symbols used by libfoo, and libpcre.a provides symbols used by libglib-2.0.a.
So far, everything is working.

If we were using Libs.private as you suggest, then glib-2.0.pc and libfoo.pc
would have something like:

# glib-2.0.pc
...
Libs: -L/usr/lib/MULTIARCH -lglib-2.0
# generated from: Libs.private: $(pkg-config --static --libs libpcre)
Libs.private: -L/usr/lib/MULTIARCH -lpcre

# libpcre.pc
...
Libs: -L/usr/lib/MULTIARCH -lfoo
# generated from: Libs.private: $(pkg-config --static --libs glib-2.0)
Libs.private: -L/usr/lib/MULTIARCH -lglib-2.0 -lpcre

Now imagine that GLib switches to libpcre2 (libpcre2-8.pc, -lpcre2-8)
as requested in #182. This will change the contents of glib-2.0.pc to:

# glib-2.0.pc
...
Libs: -L/usr/lib/MULTIARCH -lglib-2.0
# generated from: Libs.private: $(pkg-config --static --libs libpcre2-8)
Libs.private: -L/usr/lib/MULTIARCH -lpcre2-8

Without changing anything in libfoo, you'll find that libfoo.pc is now
wrong (until it is rebuilt and has the opportunity to update its idea of
what GLib depends on). Programs that depend on libfoo will fail to
link, because the new libglib-2.0.a requires symbols from libpcre2-8.a
instead of the old libpcre.a, but libfoo.pc still says
"Libs.private: ... -lpcre" because that's what it captured at libfoo's
build-time.

That's why I think it is best that the information about which libraries
GLib depends on is shipped with GLib and included by reference (via
either Requires.private: glib-2.0, or the Requires.internal: glib-2.0
proposed on https://bugs.freedesktop.org/show_bug.cgi?id=105572), instead
of copying the information about GLib's dependencies into libfoo-dev where
it will become outdated when GLib changes.

smcv

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Alexander Traud]

The problem of "Requires.private", for C/C++ libraries, it (might) 
contain two different things: Libraries used for static linking *and* 
Cflags to preprocess the header files.

If the position of Debian is that each reference in "Requires.private" 
translates into a required dependency in Debian/control for the -dev 
package, then we get dependencies for static linking as well. That is OK 
but a new policy, at least for me after going through the mailing list 
of the last two decades. Furthermore, then I do not understand why:
a) pkg-config itself is *not* a dependency as well
b) Lintian was not upgraded in 15 years to check the content of the .pc 
   file and aid the maintainer in not missing any required dependency.

Simon McVittie wrote: 
> converting Requires.private into Libs.private would require copying 
> all the recursive dependencies of the required library

I know. You have to use something like
$ pkg-config --libs --static libcurrent
to convert "Requires.private" to "Libs.private". Furthermore, the 
maintainer must double-check the header files and their includes to keep 
such packages/libraries/headers in "Requires.private" = their Cflags.

Simon McVittie wrote:
> this doesn't scale well

Then, I do not understand static linking. I thought, please correct me 
here, static linking is about the '.a' file in the package, the current 
one. I have to suffice its links as it was built, not the links when I 
build. With that understanding, "Libs.private" is even more accurate in 
Debian because it does not diverge and break over time.

Russ Allbery wrote:
> The dependencies for -dev packages should reflect the requirements for 
> dynamic linking.

That was my understanding, exactly.

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Russ Allbery]

Simon McVittie  writes:

> If you're linking statically, you need to be able to satisfy the
> recursive dependencies of libunbound (regardless of whether you are
> using pkg-config or not), so, no, you will need nettle-dev and
> libevent-dev either way.

And, specifically, I think we should say as a matter of policy that Debian
does not try to declare the dependencies required for static linking and
therefore static linking may require tracking down what additional -dev
packages you need and manually installing them.  The dependencies for -dev
packages should reflect the requirements for dynamic linking.

Static linking of C libraries is rare enough at this point, and (possibly
more relevantly) sufficiently untested by Debian that I don't think we
should try to support it directly in the dependency structure.  It would
pull in a whole bunch of -dev packages that wouldn't be used 95% of the
time.

This is what I already do for libremctl-dev, for example.  (Well, I list
the libraries required for static linking in Suggests currently, but I'm
not sure that list is complete any more and should probably just drop it
completely.)

-- 
Russ Allbery (r...@debian.org)  

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Simon McVittie]

On Mon, 13 Dec 2021 at 22:46:43 +0100, Alexander Traud wrote:
> Let us assume 'bar.pc' would create a dependency in Debian, then the 
> question arises: Why does the Debian maintainer not transform it to 
> 'Libs.private', not upstream but via a Debian patch? That would avoid 
> such .pc-file*only* dependencies.

Libs.private does not follow interdependencies between .pc files, so
converting Requires.private into Libs.private would require copying all
the recursive dependencies of the required library, not just the required
library itself.

For example, if you depend on glib-2.0.pc (-lglib-2.0), converting
Requires.private: glib-2.0 into Libs.private requires you to know (or more
likely, find out at configure time) that it depends on libpcre.

This doesn't scale well, and a previously-correct .pc file can become
retroactively incorrect as a result of a lower-level library changing
an implementatation detail. For example, if GLib moves from libpcre to
libpcre2, as has been requested, then a GLib-dependent library having
-lglib-2.0 -lpcre in Libs.private will no longer be sufficient for
static linking.

> Just to repeat: libunbound-dev works great without nettle-dev and 
> libevent-dev

With the change you are asking for, libunbound-dev would work *for dynamic
linking* without those.

If you're linking statically, you need to be able to satisfy the recursive
dependencies of libunbound (regardless of whether you are using pkg-config
or not), so, no, you will need nettle-dev and libevent-dev either way.

smcv

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Alexander Traud]

> If foo.pc in libfoo-dev references bar.pc [...]

That is the problem: If 'bar.pc' is referenced just because for static
libraries, why does it create a dependency for me as a user, who is 
a) not using pkg-config at all 
and/or 
b) linking dynamically?

Let us assume 'bar.pc' creates a dependency in Debian, then the question 
arises: Why is the package 'pkg-config' itself not a dependency, too?

Let us assume 'bar.pc' would create a dependency in Debian, then the 
question arises: Why does the Debian maintainer not transform it to 
'Libs.private', not upstream but via a Debian patch? That would avoid 
such .pc-file*only* dependencies.

Just to repeat: libunbound-dev works great without nettle-dev and 
libevent-dev. It is just the .pc file itself which introduced those 
dependencies. A little tweak to the .pc file, and it works for both 
dynamic and static linking, and does not add any such dependencies. My 
conclusion: The .pc file does not suite Debian (and its -dev package 
concept).

> Requires.private [is] quite clear cut.

I had a hard time to understand [1]. For platforms which have no 
separate '-dev' packages, yes, easy. For platforms which do have 
separate '-dev' packages, I see the problem one abstraction level 
higher: Those libs, which are in 'Requires.private' just for static 
linking, why not moving them to 'Libs.private'?

Let us assume it is a clear cut: What exactly should I do now in case of
- libunbound-dev: Report as that two dependencies are still missing or 
  creating a Debian patch to adjust the .pc for Debian?
- libsrtp2-dev: The package which all started this for me. It has 
  pkg-config as dep. Should I report that over depends on pkg-config?
- gnome-calculator: The default calculator in every Ubuntu Desktop 
  installation. It misses three -dev deps because of the two .pc files 
  in the package. Should I report those deps as missing? Or should I 
  report that the Gnome Calculator needs two -dev packages (and those 
  then dep on those -dev)?
When is it a wish, when is it a bug report?

Re: Deps of -dev packages with pkg-config .pc file: Policy Change?

[Julien Cristau]

This is all pretty straightforward.  If foo.pc in libfoo-dev references
bar.pc that lives in libbar-dev, then libfoo-dev needs a dependency on
libbar-dev, and the missing dependency is a serious bug.  That has been
the case for as long as I remember, and doesn't require more long
discussions or policy changes IMO.

Libs.private is a bit different because static linking is not typically
used for Debian packages, but Requires and Requires.private are quite
clear cut.

Cheers,
Julien

On Thu, Dec 09, 2021 at 03:24:27PM +0100, Alexander Traud wrote:
> Linux distributions, which have separate packages for developers, like 
> Debian, are not really supported [1] by the developer tool pkg-config 
> [2]. The problems are C/C++ libraries which depend on other libraries at 
> runtime. Let us pick one, a security library for utilizing DNSSEC:
> 
> $ sudo apt install libunbound-dev 
> $ pkg-config --print-errors --short-errors --cflags libunbound
> $ pkg-config --print-errors --short-errors --exists libunbound
> Package 'hogweed', required by 'libunbound', not found
> 
> 1) The error message is misleading because pkg-config talks about the 
> *runtime* package 'hogweed', although (in Debian) a *-dev* package is 
> required to get the .pc file. Should I report that, where?
> 
> 2) In this case, the package name is not simply an added '-dev' like 
> 'hogweed-dev'. Instead, the developer has to search the contents of all 
> packages [3] for the file 'hogweed.pc'. That is part of the package 
> 'nettle-dev'.
> 
> 3) Are 'nettle-dev' (and 'libevent-dev', by the way) missing 
> dependencies of 'libunbound-dev'?
> 
> The latter seems to be an ongoing question on debian-devel [4] if the 
> package includes a static library '.a' as libunbound-dev still does. At 
> least I found no conclusive answer. Position No: It is not a missing 
> dependency because I can use that package perfectly (as a shared 
> library) without pkg-config. Position Yes: pkg-config has to be 
> required, recommended, suggested - for each level of dependency, you 
> find an argument on debian-devel, see [5][6][7][8].
> 
> 16 years. This topic has come up for 16 years now, again and again on 
> the mailing list debian-devel, documenting the uncertainty of package 
> maintainers. Furthermore, bug reports come in, again and again for that 
> topic. Again, libunbound-dev as example for the Debian bug report [9]: 
> Upstream, in the pkg-config .pc file, the maintainer moved the libraries 
> from 'Requires' to 'Requires.private'. That was the correct approach. 
> However, the maintainer closed the Debian bug report because he falsely 
> believed to have fixed the issue that way.
> 
> Requires -> -I/subfolder -lfuu, avoid if possible, see [10]
> Requires.private -> -I/subfolder  , a header includes that header
> Libs ->  -lfoo, lib itself
> Libs.private ->  -lfuu, exclusively for static linking
> Cflags   -> -I/subfolder  , lib headers itself and/or
> header includes that header
> 
> Any idea how to approach this?
> a) Leave as is. Do not depend on 'Require.private' automatically because 
> the package can be used without pkg-config. Only depend on other -devs 
> if one of those headers is included in a header.
> b) Fix/convert. Upstream, move everything into 'Requires.private'. 
> Downstream, in the Debian package, remove 'Requires.private' and convert 
> to 'Libs.private'; again, except if one of those headers is included in 
> a header.
> 
> These uncertainties create repeated frustration, even for core Debian 
> members. Proposal: Why not decide, or at least give a recommendation on 
> these questions:
> 
> i) pkg-config tool itself: When is it a depend (which dep level?), and 
> when not? My suggestions: Because I can use a '-dev' package without any 
> tool, because of the FHS, just -lfoo should be needed [11].
> 
> ii) .pc file and its 'Require': Leave it or change it? My suggestion: 
> Report upstream that those libs have to be moved to 'Requires.private'.
> 
> iii) .pc file and its 'Require[.private]': Depend on each lib? My 
> suggestion: Only if the headers include it. For Debian, convert all 
> others to 'Libs.private' because the built '.a' file (for static 
> linking) has a known dependency tree. This gets a Debian patch in that 
> source package.
> 
> iv) Cflags: If I got it correctly, back in the year 2005, the cause of 
> this drama were -I flags [12]; if the header included another header, 
> and that header included further headers but was not in the root but in 
> a subfolder, an -I flag *might* be required. For example, the package 
> 'libopusfile-dev' has its header file in '/usr/include/opus/opusfile.h' 
> with #include  which is part of the package 
> 'libopus-dev' and is not within '/usr/include/' but within 
> '/usr/include/opus/'. Therefore,
> $ pkg-config --cflags opusfile
> -I/usr/include/opus
> 
> After reading, understanding, 

Deps of -dev packages with pkg-config .pc file: Policy Change?

[Alexander Traud]

Linux distributions, which have separate packages for developers, like 
Debian, are not really supported [1] by the developer tool pkg-config 
[2]. The problems are C/C++ libraries which depend on other libraries at 
runtime. Let us pick one, a security library for utilizing DNSSEC:

$ sudo apt install libunbound-dev 
$ pkg-config --print-errors --short-errors --cflags libunbound
$ pkg-config --print-errors --short-errors --exists libunbound
Package 'hogweed', required by 'libunbound', not found

1) The error message is misleading because pkg-config talks about the 
*runtime* package 'hogweed', although (in Debian) a *-dev* package is 
required to get the .pc file. Should I report that, where?

2) In this case, the package name is not simply an added '-dev' like 
'hogweed-dev'. Instead, the developer has to search the contents of all 
packages [3] for the file 'hogweed.pc'. That is part of the package 
'nettle-dev'.

3) Are 'nettle-dev' (and 'libevent-dev', by the way) missing 
dependencies of 'libunbound-dev'?

The latter seems to be an ongoing question on debian-devel [4] if the 
package includes a static library '.a' as libunbound-dev still does. At 
least I found no conclusive answer. Position No: It is not a missing 
dependency because I can use that package perfectly (as a shared 
library) without pkg-config. Position Yes: pkg-config has to be 
required, recommended, suggested - for each level of dependency, you 
find an argument on debian-devel, see [5][6][7][8].

16 years. This topic has come up for 16 years now, again and again on 
the mailing list debian-devel, documenting the uncertainty of package 
maintainers. Furthermore, bug reports come in, again and again for that 
topic. Again, libunbound-dev as example for the Debian bug report [9]: 
Upstream, in the pkg-config .pc file, the maintainer moved the libraries 
from 'Requires' to 'Requires.private'. That was the correct approach. 
However, the maintainer closed the Debian bug report because he falsely 
believed to have fixed the issue that way.

Requires -> -I/subfolder -lfuu, avoid if possible, see [10]
Requires.private -> -I/subfolder  , a header includes that header
Libs ->  -lfoo, lib itself
Libs.private ->  -lfuu, exclusively for static linking
Cflags   -> -I/subfolder  , lib headers itself and/or
header includes that header

Any idea how to approach this?
a) Leave as is. Do not depend on 'Require.private' automatically because 
the package can be used without pkg-config. Only depend on other -devs 
if one of those headers is included in a header.
b) Fix/convert. Upstream, move everything into 'Requires.private'. 
Downstream, in the Debian package, remove 'Requires.private' and convert 
to 'Libs.private'; again, except if one of those headers is included in 
a header.

These uncertainties create repeated frustration, even for core Debian 
members. Proposal: Why not decide, or at least give a recommendation on 
these questions:

i) pkg-config tool itself: When is it a depend (which dep level?), and 
when not? My suggestions: Because I can use a '-dev' package without any 
tool, because of the FHS, just -lfoo should be needed [11].

ii) .pc file and its 'Require': Leave it or change it? My suggestion: 
Report upstream that those libs have to be moved to 'Requires.private'.

iii) .pc file and its 'Require[.private]': Depend on each lib? My 
suggestion: Only if the headers include it. For Debian, convert all 
others to 'Libs.private' because the built '.a' file (for static 
linking) has a known dependency tree. This gets a Debian patch in that 
source package.

iv) Cflags: If I got it correctly, back in the year 2005, the cause of 
this drama were -I flags [12]; if the header included another header, 
and that header included further headers but was not in the root but in 
a subfolder, an -I flag *might* be required. For example, the package 
'libopusfile-dev' has its header file in '/usr/include/opus/opusfile.h' 
with #include  which is part of the package 
'libopus-dev' and is not within '/usr/include/' but within 
'/usr/include/opus/'. Therefore,
$ pkg-config --cflags opusfile
-I/usr/include/opus

After reading, understanding, and researching so much, I really wonder 
if this was correct. For example, if the header did
#include 
no include directory would be needed to be figured out via pkg-config. 
Actually, if the world was like that, I could shelf pkg-config and go 
just for linking libraries.

Long story short:
After 16 years of back and forth, several hundred bug reports, still new 
ones coming in, and existing reports not fixed completely, I wonder if 
someone in the Debian world is able to decide this finally and provide 
that decision either as policy change, or recommendation, or at least as 
a hint for future -dev package maintainers. And please, tell me, what I 
should do about libunbound-dev; re-open that bug report [9]?