Re: [Openembedded-architecture] Empty packages and bogus dependencies - what to do?

2023-02-25 Thread Mark Hatle



On 2/25/23 4:24 AM, Richard Purdie wrote:

Hi All,

There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.

I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.

The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:

bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b


Switching from recommends to depends, I think the biggest concern here is the 
symlinks .so -> implementation.  However, if the code (dependency) is already 
capturing this and including them, I see no problem here.


What we want to avoid is installing a -dev package, and not getting the 
associated libraries at the same time, because a user doesn't have recommends 
enabled.  (On-target development..  it's been pretty common for people I've 
worked with to turn off recommends on their systems.)



I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81


I like the way this one is structured.  Will there be any sort of helper to 
detect the older use of the RDEPENDS:${PN}-dev for conversion?  I forsee a lot 
of people not understanding this and still doing it the old way.



which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.

As I see it there are a few options:

a) Do nothing and continue with the above   
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else

I think I'm in favour of c), removing PN from PACKAGES and not even
pretending it exists. We could add an RPROVIDES:${PN}-dev += "${PN} so
that assumptions about naming work, not sure about that.


In my many cases install of the (empty) PN was treated like a package group, 
install the PN, and you get the 'collection' of runtime packages that may have 
different names.  I'm not sure this is still true though, but it's really the 
only concern I have with C.




That handles one specific corner case but there are others as bug #8222
talks about for example
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8222

Here, the proposal is to change the globally inserted RDEPENDS (now a
RRECOMMEND) from PN-dev to PN to become conditional upon (roughly hand
waving), there being libraries in other packages which PN-dev therefore
needs to depend upon. I think that idea is worthy of exploration. Are
there any cases where the -dev package depends on ${PN} for things
other than shared libraries?


Ya, I think this is more what is needed.  If there are symlink .so in the -dev, 
we need to find out what is providing the targets and those should be the 
dependencies.  In some case it may be ${PN}, but not all.



Thinking out loud, our symlink dependency handling code in do_package
might even be enough to detect and handle injecting that?


I think it is?  I know that general symlink targeting was intended to do 
something like this, so there may not be any reason to special case .so 
symlinks.  Any symlink really indicates a dependency on the provider of some kind.




So, turning this into a set of actions, my personal view is that we
should probably:

a) Add warnings for dependencies (RDEPENDS or RRECOMMENDS) on packages
which don't get created for some reason (either not in PACKAGES or not
ALLOW_EMPTY). We should be able to properly check this using
packagedata which never used to exist as it does now when these bugs
were first created! We can leave the warnings disabled at first until
we clean things up for core and our key layers

b) Work through those warnings to work out if we can silence them at
source. I think the dependency creation code for -dbg and -dev packages
will need to be taught to check if a target dependency exists to fix
the bulk of them.

c) Explore whether we should drop PN from PACKAGES in some of the above
cases.

d) Explore whether we can replace the current generic RRECOMMENDS from
PN-dev -> PN with something autogenerated based on the presence of
shared libs


I think a-d will provide us the largest 

[Openembedded-architecture] Empty packages and bogus dependencies - what to do?

2023-02-25 Thread Richard Purdie
Hi All,

There has been some interest in some older bugs about empty packages
and bogus dependencies and what we should do about them. Yohan and
Fawzi were kind enough to post some patches to prompt some discussion
and this did indeed get me thinking.

I personally think that we shouldn't be generating an RDEPENDS or
RRECOMMENDS if we know that the package in question is never going to
exist. We have "auto dependency" code which does go a bit crazy with
these.

The easiest problem case to understand is an empty PN package. Based
upon their patches I can see we likely have this situation in a subset
of:

bind, bsd-headers, libssp-nonshared, newlib, libgcc, python3, libglu
mesa, libpthread-stubs, xtrans, xcb-proto, xorgproto, util-macros,
linux-libc-headers, make-mod-scripts, argp-standalone

Reasons vary but the easy to understand case is where the code is just
development headers and all ends up in ${PN}-dev.

We have https://bugzilla.yoctoproject.org/show_bug.cgi?id=6839

We did used to set an RDEPENDS between ${PN}-dev and ${PN} but that
turned out to be a bad idea and we relaxed it to a RRECOMENDS:

https://git.yoctoproject.org/poky/commit/?id=ddbd90a37c7e9089aaeccb42dd008cf47594099b

I did at least mark up the recipes with this issue:

https://git.yoctoproject.org/poky/commit/?id=73cf55cdde1f3d2773b42f52d2fcf57665102c81

which meant the worst of these dependencies were removed where the main
package didn't exist or didn't make sense.

As I see it there are a few options:

a) Do nothing and continue with the above   
b) Generate an empty PN
c) Delete PN from PACKAGES
d) Move the contents of PN-dev to PN
e) Something else

I think I'm in favour of c), removing PN from PACKAGES and not even
pretending it exists. We could add an RPROVIDES:${PN}-dev += "${PN} so
that assumptions about naming work, not sure about that.


That handles one specific corner case but there are others as bug #8222
talks about for example
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8222

Here, the proposal is to change the globally inserted RDEPENDS (now a
RRECOMMEND) from PN-dev to PN to become conditional upon (roughly hand
waving), there being libraries in other packages which PN-dev therefore
needs to depend upon. I think that idea is worthy of exploration. Are
there any cases where the -dev package depends on ${PN} for things
other than shared libraries?

Thinking out loud, our symlink dependency handling code in do_package
might even be enough to detect and handle injecting that?


So, turning this into a set of actions, my personal view is that we
should probably:

a) Add warnings for dependencies (RDEPENDS or RRECOMMENDS) on packages
which don't get created for some reason (either not in PACKAGES or not
ALLOW_EMPTY). We should be able to properly check this using
packagedata which never used to exist as it does now when these bugs
were first created! We can leave the warnings disabled at first until
we clean things up for core and our key layers

b) Work through those warnings to work out if we can silence them at
source. I think the dependency creation code for -dbg and -dev packages
will need to be taught to check if a target dependency exists to fix
the bulk of them.

c) Explore whether we should drop PN from PACKAGES in some of the above
cases.

d) Explore whether we can replace the current generic RRECOMMENDS from
PN-dev -> PN with something autogenerated based on the presence of
shared libs

e) For a long time I've wondered if we should split the -dbg packages
from one per recipe to one per package with binaries in it. This would
solve some of the problems that we'll likely encounter in b) but might 













new ones.

I've given this some thought but not thought through the ideas in full.

I appreciate some of this was set as a newcommer bug in bugzilla and in
hindsight this perhaps shouldn't have been the case. I suspect taking
d) in isolation was the idea and that should still be possible.

I'd note that we're at feature freeze so more invasive changes like
this would likely need to be post release unless any become bug fixes.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1701): 
https://lists.openembedded.org/g/openembedded-architecture/message/1701
Mute This Topic: https://lists.openembedded.org/mt/97224039/21656
Group Owner: openembedded-architecture+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-