Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> "NG" == Neal Gompawrites: NG> That said, it's required to make debuginfo packages get generated, NG> as I found out when working on OpenMandriva's NG> rpm-openmandriva-setup: NG> https://github.com/OpenMandrivaSoftware/rpm-openmandriva-setup/commit/df94fda422f9bf3d9e32b6f8f59493d2d478daef Well, it seems pretty obvious that actually expanding the %debug_package macro (which includes the extra %package, %description and %files sections) _somewhere_ in the spec is indeed necessary to generate debuginfo. But that commit doesn't say anything about why the conditional on %buildsubdir needs to be there. I guess it would be fun to see what breaks (and how it breaks) when you use just: %install %{?_enable_debug_packages:%{debug_package}} \ %%install \ %{nil} instead. But I'm not going to have time to do much of that today. - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
Hi, On Fri, 2018-04-27 at 11:34 +0300, Panu Matilainen wrote: > On 04/27/2018 06:47 AM, Jason L Tibbitts III wrote: > > Here's a paste from the macros that Fedora uses > > (/usr/lib/rpm/redhat/macros): > > > > %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ > > %%install\ > > %{nil} > > Ew. Now there's a blast from the past, that ugly %install override is > from 2002 when debuginfo was a fairly new concept. If that's actually > needed for debuginfo then it's a bug (in rpm). I admit to not precisely know the macro expansion rules or when, what defines buildsubdir. (I always assumed just having a %setup macro was enough to get it defined). But I do know why the buildsubdir macro is necessary for autogenerating debuginfo/sources packages. The idea is that the debuginfo comes from the (installed) binaries, it gets stripped out and put into separate (.debug) files which get packaged together with some auxiliary information generated by the debuginfo scripts (like .dwz files). This debug information might reference source code files. For some profiling/tracing/debugging applications having access to the (generated) source files might be helpful. But you only want to package up the actual source files used to generate the binaries. So those that are actually referenced in the debuginfo, not other files that might be in the srpm but never used, nor source/include files from other (devel) package that happen to be installed. To know which source files to package the source named in the .debug files needs to be matched against the actual source files (produced) by the package. To do that you will need to know the buildsubdir. Only files inside the buildsubdir should be packaged as debugsources. e.g. the debuginfo might also reference /usr/include/stdio.h, but you don't want that packaged. Also in the future packages might be build with -gsplit-dwarf which places (most) debug information in separate .dwo files, which will never be installed, but will be kept in the buildsubdir from which it will have to be fetched and packaged. Cheers, Mark ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> On Apr 27, 2018, at 6:35 AM, Neal Gompawrote: ... > > That said, it's required to make debuginfo packages get generated, as I > found out when working on OpenMandriva's rpm-openmandriva-setup: > https://github.com/OpenMandrivaSoftware/rpm-openmandriva-setup/commit/df94fda422f9bf3d9e32b6f8f59493d2d478daef > If the test on %buildsubdir is actually still needed ... A test on %buildsubdir existence delays the substitution until after %setup has actually been run. Without seeing details with/without the %install overloading, I cannot guess why the overloading is/was necessary. But if the test is needed, I surely diagnosed the problem and devised the solution way back when. 73 de Jeff ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
On Fri, Apr 27, 2018 at 5:57 AM Jeff Johnsonwrote: > > On Apr 27, 2018, at 4:34 AM, Panu Matilainen wrote: > > > > On 04/27/2018 06:47 AM, Jason L Tibbitts III wrote > >> > >> Seems... similar. > >> JJ> Talk to whomever decided that there was a need to overload %install > >> JJ> with a test on %buildsubdir. > >> OK, so, again, I'm trying to talk to this mailing list, which I thought > >> provided the best chance to reach the people who know. > > > > Well, git blame brings this up (redhat-rpm-config): > > > > commit 1640cd5cac6e1308de8f303e95089053e7f5c9b5 (tag: REDHAT_RPM_CONFIG_8_0_14) > > Author: Tim Powers > > Date: Sat Dec 14 20:37:32 2002 + > > > >more debug macro tweaks > > > > ...and that's all he wrote. Gotta love the descriptive commit messages of that era. > > > Well, Tim Powers was the QA guy assigned the daunting task of ensuring that all packages in RHEL3 rebuilt and created -debuginfo packages without any change to spec files whatsoever on 3 architectures based on RawHide. > That was a huge amount of effort, scripting around rpmbuild, no chained builds or Koji with a new set of packages in RawHide every day, no mock, no depsolver, no git, no code reviews, and on a delivery deadline. > And you should certainly know how twisted the rpmbuild parser is/was, and how very hacky -debuginfo production is. > If the terseness of the checkin bothers you, well, consider how many years it's taken to even detect the bogusness. And it's taken nearly that many years to get rid of the insanely mysterious hack involved with "Canonicalization shrank by 1 byte". Well, now I feel really bad for the poor guy. That said, it's required to make debuginfo packages get generated, as I found out when working on OpenMandriva's rpm-openmandriva-setup: https://github.com/OpenMandrivaSoftware/rpm-openmandriva-setup/commit/df94fda422f9bf3d9e32b6f8f59493d2d478daef -- 真実はいつも一つ!/ Always, there's only one truth! ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
On 04/27/2018 06:47 AM, Jason L Tibbitts III wrote: "JJ" == Jeff Johnsonwrites: JJ> Note that the macro definitions you are trying to understand are not JJ> from rpm itself (so you perhaps should be consulting with other JJ> SME's than me). Well, I was consulting a mailing list. Here's a paste from the macros.debug file in the rpm 4.14.1 source: %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ %%install\ %{nil} Note that macros.debug is only used inside the rpm testsuite, none of it is active in normal installation. Here's a paste from the macros that Fedora uses (/usr/lib/rpm/redhat/macros): %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ %%install\ %{nil} Ew. Now there's a blast from the past, that ugly %install override is from 2002 when debuginfo was a fairly new concept. If that's actually needed for debuginfo then it's a bug (in rpm). Seems... similar. JJ> Talk to whomever decided that there was a need to overload %install JJ> with a test on %buildsubdir. OK, so, again, I'm trying to talk to this mailing list, which I thought provided the best chance to reach the people who know. Well, git blame brings this up (redhat-rpm-config): commit 1640cd5cac6e1308de8f303e95089053e7f5c9b5 (tag: REDHAT_RPM_CONFIG_8_0_14) Author: Tim Powers Date: Sat Dec 14 20:37:32 2002 + more debug macro tweaks ...and that's all he wrote. Gotta love the descriptive commit messages of that era. - Panu - ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
On Apr 26, 2018, at 11:47 PM, Jason L Tibbitts IIIwrote: >> "JJ" == Jeff Johnson writes: > > JJ> Note that the macro definitions you are trying to understand are not > JJ> from rpm itself (so you perhaps should be consulting with other > JJ> SME's than me). > > Well, I was consulting a mailing list. > > Here's a paste from the macros.debug file in the rpm 4.14.1 source: > > %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ > %%install\ > %{nil} > > Here's a paste from the macros that Fedora uses > (/usr/lib/rpm/redhat/macros): > > %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ > %%install\ > %{nil} > > Seems... similar. > > JJ> Talk to whomever decided that there was a need to overload %install > JJ> with a test on %buildsubdir. > > OK, so, again, I'm trying to talk to this mailing list, which I thought > provided the best chance to reach the people who know. > I recuse myself, largely because the current implementation of -debuginfo has convolved the (known!) flaws of the rpm spec file parser with automagic -debuginfo generation (splitting sources into multiple packages forces handling of painful *.spec constructs like %exclude within the side effect driven spec file parser and worse). Good luck! 73 de Jeff > - J< > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> "JJ" == Jeff Johnsonwrites: JJ> Note that the macro definitions you are trying to understand are not JJ> from rpm itself (so you perhaps should be consulting with other JJ> SME's than me). Well, I was consulting a mailing list. Here's a paste from the macros.debug file in the rpm 4.14.1 source: %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ %%install\ %{nil} Here's a paste from the macros that Fedora uses (/usr/lib/rpm/redhat/macros): %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ %%install\ %{nil} Seems... similar. JJ> Talk to whomever decided that there was a need to overload %install JJ> with a test on %buildsubdir. OK, so, again, I'm trying to talk to this mailing list, which I thought provided the best chance to reach the people who know. - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> On Apr 26, 2018, at 9:53 PM, Jason L Tibbitts IIIwrote: > > And to continue the spam, I see that in 2016 rpm gained support for > section end markers ("%end") which also conveniently solves the problem. > I know I had asked about that years ago but I had no idea that it had > been implemented. As far as I can tell, though, you need rpm 4.14. > The addition of %end in 2016 (and Fedora RPM4) as a solution fixes you problem: Be happy! Meanwhile, the rpm spec file parser is hugely flawed (and always has been). There are hacks upon hacks upon hacks with almost no design, only bug reports, for the last 20y. Disclaimer: Feel free to blame me for any/all implementations in rpmbuild even though I haven't touched any of RPM4 (RPM5 has most of the same flaws) since 2005. hth 73 de Jeff > - J< > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> On Apr 26, 2018, at 8:45 PM, Jason L Tibbitts IIIwrote: > > To sort of answer my own question... > > It isn't %build that's important. There simply must be some other > section between %prep and %install. > > What I think happens is this: > > The parser is running through %prep, doing whatever it does (expanding > macros and building the script that will be executed, I guess). It sees > %install, which is defined as a macro, so it goes to expand it. At that > moment, %buildsubdir hasn't actually been defined as a macro because rpm > is still parsing and hasn't executed anything (specifically the magic > %setup macro). So because %buildsubdir is not defined, %install just > expands to... %install. > Ok. There are means using %{expand: ...} to trigger an immediate expansion, and there are means to delay the expansion (if expanded multiple times) by escaping with a doubled %%. Again, you need to find the person who chose to add that macro definition, and ask them what is/was the intent. There was no need for %install overloading when -debuginfo was first implemented. Nor do I have sufficient Fedora Packaging Commmitte context wrto R packaging to begin to understand and suggest an alternative implementation to whatever you are attempting ... > An easy way to test this is to just move the %check section up above > %install. And... it works. Even just adding an empty %clean section > works fine. > Yes, but ... The original implementation of -debuginfo was based on a heuristic that is/was based on an expansion insertion point to drop-in the automagic definition of a -debuginfo subpackage. That isn't you, with someone else's decision to overload %install with a test in %buildsubdir. > So I guess this is down to two things: > * RPM parsing is deep magic and will hurt your brain. Oh definitely: spec file parsing is deeply flawed, largely for 2 reasons: 1) ctype(3) parsing 2) there is no proper parsing pass, everything is instead done as a side effect. I get burned every time I attempt rpmbuild changes all this century. > * Redefining the section "macros" is a recipe for "fun". No idea what this means. Again note that your %install overloading has little to do with either the original implementation, Fedora R packaging, or the rpm macro implementation. If you would like to back up and restate your problem goal differently, there is likely a solution. > > I have no idea if this weirdness would be considered a bug. I'll file a > ticket if someone thinks it's worth it, but outside of R packages, I > would expect there to be relatively few archful packages that would > legitimately not have a %build section. > You can always write Fedora R packaging policy mandating the existence of an empty %build section. WORKSFORME Meanwhile if you supply a R packaging goal, there is likely a solution. Your call. 73 de Jeff > - J< > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
On Apr 26, 2018, at 7:27 PM, Jason L Tibbitts IIIwrote: >> "JJ" == Jeff Johnson writes: > > JJ> You are unlikely to achieve any joy trying to set or change its > JJ> value. > > I'm trying to comprehend how you came to the conclusion that I wanted to > change its value. > I did not say you wished to change he value. You did ask about %buildsubdir, and since this is not just a convo between thee and me, I provided the usual caveat emptor boilerplate. > JJ> I know of no reason why a -debuginfo package needs a %build > JJ> section. > > Well, what I'm seeing is that the generation of debuginfo sections > requires a %build section to be present > Note that the macro definitions you are trying to understand are not from rpm itself (so you perhaps should be consulting with other SME's than me). > JJ> What is much more likely is that you have some odd ordering > JJ> of scriptlet sections, > > Well, it goes %prep, %build (empty), %install, %check, %files and > %changelog. As far as I know, that's the normal ordering for a Fedora > specfile. > Good. I guessed at unusual behavior causes, nothing more. > JJ> [...] because -debuginfo packages are just an invisible subpackage > JJ> where existing section markers are overloaded to insert -debuginfo > JJ> sub package info (which -- iirc from 15yo memory -- happens with > JJ> %prep, not %build, but ICBW) > > Yes, I know. It happens in %install, not %prep. I even included the > macro definition where it happens in my message. > Talk to whomever decided that there was a need to overload %install with a test on %buildsubdir. I -- as the person responsible for the original rpm -debuginfo subpackage generation implementation -- can not think of any reason why there is a need to overload %install with a test on %buildsubdir. But perhaps I'm in the early stages of senility. Oh well ... God luck! hth 73 de Jeff > - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
And to continue the spam, I see that in 2016 rpm gained support for section end markers ("%end") which also conveniently solves the problem. I know I had asked about that years ago but I had no idea that it had been implemented. As far as I can tell, though, you need rpm 4.14. - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
To sort of answer my own question... It isn't %build that's important. There simply must be some other section between %prep and %install. What I think happens is this: The parser is running through %prep, doing whatever it does (expanding macros and building the script that will be executed, I guess). It sees %install, which is defined as a macro, so it goes to expand it. At that moment, %buildsubdir hasn't actually been defined as a macro because rpm is still parsing and hasn't executed anything (specifically the magic %setup macro). So because %buildsubdir is not defined, %install just expands to... %install. An easy way to test this is to just move the %check section up above %install. And... it works. Even just adding an empty %clean section works fine. So I guess this is down to two things: * RPM parsing is deep magic and will hurt your brain. * Redefining the section "macros" is a recipe for "fun". I have no idea if this weirdness would be considered a bug. I'll file a ticket if someone thinks it's worth it, but outside of R packages, I would expect there to be relatively few archful packages that would legitimately not have a %build section. - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> "JJ" == Jeff Johnsonwrites: JJ> You are unlikely to achieve any joy trying to set or change its JJ> value. I'm trying to comprehend how you came to the conclusion that I wanted to change its value. JJ> I know of no reason why a -debuginfo package needs a %build JJ> section. Well, what I'm seeing is that the generation of debuginfo sections requires a %build section to be present JJ> What is much more likely is that you have some odd ordering JJ> of scriptlet sections, Well, it goes %prep, %build (empty), %install, %check, %files and %changelog. As far as I know, that's the normal ordering for a Fedora specfile. JJ> [...] because -debuginfo packages are just an invisible subpackage JJ> where existing section markers are overloaded to insert -debuginfo JJ> sub package info (which -- iirc from 15yo memory -- happens with JJ> %prep, not %build, but ICBW) Yes, I know. It happens in %install, not %prep. I even included the macro definition where it happens in my message. - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
> On Apr 26, 2018, at 2:48 PM, Jason L Tibbitts IIIwrote: > > A question came up in the Fedora Packaging Committee: Why is a %build > section required for debuginfo packages to be generated. > > We were looking into the R packaging guidelines. R modules are built > and installed in one step which cannot be split, so in Fedora everything > is just done in the %install step and the %build section is empty. > %prep is present as usual and has a normal call to %setup. But for some > reason %build must be present, or debuginfo packages aren't generated. > > Looking at the macros, everything related to debuginfo packages is hung > off of a redefinition of the %install macro: > > %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ > %%install\ > %{nil} > (from the redhat-rpm-config macros file) > > Obviously %_enable_debug_packages and %buildsubdir must be defined for > this to happen, and %_enable_debug_packages is set to 1 in the same > macros file, so it must be that %buildsubdir is not getting defined. > And that implies that somehow %buildsubdir gets set when %build is > parsed or processed. > > But looking in the rpm source, I see only one place where that macro is > set: in the doSetupMacro function (in build/parsePrep.c): > >rpmPushMacro(spec->macros, "buildsubdir", NULL, spec->buildSubdir, > rpmPushMacro(spec->RMIL_SPEC); > > The stock macros file seems to agree that this is defined by %setup: > > # The sub-directory (relative to %{_builddir}) where sources are > compiled. > # This macro is set after processing %setup, either explicitly from the > # value given to -n or the default name-version. > # > #%buildsubdir > > But the package does have %prep and it does call %setup. So obviously > I'm missing something. Most likely I'm misunderstanding how a internal > macro like that gets set in the rpm source. I do see in the doScript > function from build/build.c how spec->buildSubdir gets used in %build, > but I don't see anything there spec->that sets it or pushes that value > into a macro definition. > %buildsubdir is set from %setup arguments. You are unlikely to achieve any joy trying to set or change its value. I know of no reason why a -debuginfo package needs a %build section. What is much more likely is that you have some odd ordering of scriptlet sections, because -debuginfo packages are just an invisible subpackage where existing section markers are overloaded to insert -debuginfo sub package info (which -- iirc from 15yo memory -- happens with %prep, not %build, but ICBW) 73 de Jeff > Could someone please enlighten me? Thanks, > > - J< > ___ > Rpm-ecosystem mailing list > Rpm-ecosystem@lists.rpm.org > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
[Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation
A question came up in the Fedora Packaging Committee: Why is a %build section required for debuginfo packages to be generated. We were looking into the R packaging guidelines. R modules are built and installed in one step which cannot be split, so in Fedora everything is just done in the %install step and the %build section is empty. %prep is present as usual and has a normal call to %setup. But for some reason %build must be present, or debuginfo packages aren't generated. Looking at the macros, everything related to debuginfo packages is hung off of a redefinition of the %install macro: %install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\ %%install\ %{nil} (from the redhat-rpm-config macros file) Obviously %_enable_debug_packages and %buildsubdir must be defined for this to happen, and %_enable_debug_packages is set to 1 in the same macros file, so it must be that %buildsubdir is not getting defined. And that implies that somehow %buildsubdir gets set when %build is parsed or processed. But looking in the rpm source, I see only one place where that macro is set: in the doSetupMacro function (in build/parsePrep.c): rpmPushMacro(spec->macros, "buildsubdir", NULL, spec->buildSubdir, rpmPushMacro(spec->RMIL_SPEC); The stock macros file seems to agree that this is defined by %setup: # The sub-directory (relative to %{_builddir}) where sources are compiled. # This macro is set after processing %setup, either explicitly from the # value given to -n or the default name-version. # #%buildsubdir But the package does have %prep and it does call %setup. So obviously I'm missing something. Most likely I'm misunderstanding how a internal macro like that gets set in the rpm source. I do see in the doScript function from build/build.c how spec->buildSubdir gets used in %build, but I don't see anything there spec->that sets it or pushes that value into a macro definition. Could someone please enlighten me? Thanks, - J< ___ Rpm-ecosystem mailing list Rpm-ecosystem@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-ecosystem