Re: [Rpm-maint] Proposal for rpmbuild patch
Hi,
FYI: I released version 1.4 of the Python utility that helps interfacing
rpmbuild to GNU make at:
https://github.com/f18m/rpm-make-rules-dependency-lister
ChangeLog:
- changed the way the dependency file is generated: query sha256sums
directly using rpm program instead of decompressing the rpm with rpm2cpio!
this speeds up the process and makes it more robust
- allow searching packaged files in multiple search folders
- add --strip-dirname option
- improve docs
HTH,
Francesco
2018-05-18 23:25 GMT+02:00 Francesco <francesco.monto...@gmail.com>:
> Hi Jeff,
>
>
> 2018-05-18 18:54 GMT+02:00 Jeff Johnson <n3...@me.com>:
>>
>> You do realize that rpm already computes a digest (usually SHA256, but
>> MD5 can be configured) for all files in a *.rpm package file? That
>> information can be retrieved using --queryformat easily (if make can use
>> digests other than MD5).
>>
>
> No, I really did not know that. I explored --queryformat and found out on
> my Centos7 system that indeed the command
> rpm -q --qf '[%{filenames} %{FILEMD5S}\n]' coreutils | grep date
> provides:
> /usr/bin/date 0955c235d9f069b32d22830f661071a33d99425dc5eca69b8da99654c67e
> 0cc7
>
> that really looks exactly what I needed!
> Actually as you say that's not a md5 checksum (as the tag name FILEMD5S
> would suggest) but rather a sha256 checksum, but for my purpose there's no
> difference.
>
>
>
>> Meanwhile -- for general adoption/use in Makefiles -- the glue to
>> generate make dependencies should be as simple as possible. The conversions
>> involved with compression, archive format, and digest algorithm are all
>> obvious, but there really isn't good support to do the operations on a pipe
>> with an archive.
>>
>>
> Sure. I wrote a Python utility just for speed but perhaps a small C
> utility would be better (to reduce dependencies) to do the glue operations.
>
>
>> Rationale: with some simple post-processing such new option would allow
>> to e.g., find in a way transparent to the build system employed by the user
>> (automake, cmake, etc) to understand if a .rpm needs to be regenerated
>> (because some of its packaged files have been changed) or not.
>>
>>
>> The biggest issue(s) I see are:
>>
>> 1) generating another artifact (other than *.rpm) from rpmbuild would be
>> quite awkward.
>> rpmbuild (unlike gcc) is 2 layers (shell and %buildroot) from the build
>> tree, and packaging occurs long after the compilation, unlike gcc.
>>
>> 2) rpm has way too many obscure CLI options already. So instead of a CLI
>> option, the approach should be through a configuration macro, using a
>> rule-based value/pattern to generate the output location. A CLI option can
>> always be retrofitted using popt(3) aliases.
>>
>> But a separate executable, not directly integrated into rpmbuild, might
>> simplify the decompression/archive/digest handling with stdout output,
>> making it easier to automate make dependencies on *.rpm contents, since rpm
>> already has all the necessary implementations.
>>
>
> Yeah, at this point the idea of doing an rpmbuild patch is not really
> useful anymore. I was not aware that rpm was able to do all what is needed
> to achieve what I proposed (smarter rpm regeneration).
>
>
>
>>
>> hth
>>
>>
> Yes, that helped, thanks!
>
> Francesco
>
>
>
>
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] Proposal for rpmbuild patch
Hi Jeff,
2018-05-18 18:54 GMT+02:00 Jeff Johnson <n3...@me.com>:
>
> You do realize that rpm already computes a digest (usually SHA256, but MD5
> can be configured) for all files in a *.rpm package file? That information
> can be retrieved using --queryformat easily (if make can use digests other
> than MD5).
>
No, I really did not know that. I explored --queryformat and found out on
my Centos7 system that indeed the command
rpm -q --qf '[%{filenames} %{FILEMD5S}\n]' coreutils | grep date
provides:
/usr/bin/date
0955c235d9f069b32d22830f661071a33d99425dc5eca69b8da99654c67e0cc7
that really looks exactly what I needed!
Actually as you say that's not a md5 checksum (as the tag name FILEMD5S
would suggest) but rather a sha256 checksum, but for my purpose there's no
difference.
> Meanwhile -- for general adoption/use in Makefiles -- the glue to generate
> make dependencies should be as simple as possible. The conversions involved
> with compression, archive format, and digest algorithm are all obvious, but
> there really isn't good support to do the operations on a pipe with an
> archive.
>
>
Sure. I wrote a Python utility just for speed but perhaps a small C utility
would be better (to reduce dependencies) to do the glue operations.
> Rationale: with some simple post-processing such new option would allow to
> e.g., find in a way transparent to the build system employed by the user
> (automake, cmake, etc) to understand if a .rpm needs to be regenerated
> (because some of its packaged files have been changed) or not.
>
>
> The biggest issue(s) I see are:
>
> 1) generating another artifact (other than *.rpm) from rpmbuild would be
> quite awkward.
> rpmbuild (unlike gcc) is 2 layers (shell and %buildroot) from the build
> tree, and packaging occurs long after the compilation, unlike gcc.
>
> 2) rpm has way too many obscure CLI options already. So instead of a CLI
> option, the approach should be through a configuration macro, using a
> rule-based value/pattern to generate the output location. A CLI option can
> always be retrofitted using popt(3) aliases.
>
> But a separate executable, not directly integrated into rpmbuild, might
> simplify the decompression/archive/digest handling with stdout output,
> making it easier to automate make dependencies on *.rpm contents, since rpm
> already has all the necessary implementations.
>
Yeah, at this point the idea of doing an rpmbuild patch is not really
useful anymore. I was not aware that rpm was able to do all what is needed
to achieve what I proposed (smarter rpm regeneration).
>
> hth
>
>
Yes, that helped, thanks!
Francesco
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] Proposal for rpmbuild patch
Hi all, [I posted a similar proposal under the thread "How to generate GNU make dependency file for a SPEC file automatically?" but I'm reposting here with a better subject] Proposal: add a new option to rpmbuild, something like --generate-md5sum= that writes the md5sum (or whatever other hash, sha1sum, etc) of all files placed inside the RPM package. Of course the hash would be that associated to the file prior to cpio compression, after all other rpm processing (e.g., stripping of debug info), just before it gets compressed. Rationale: with some simple post-processing such new option would allow to e.g., find in a way transparent to the build system employed by the user (automake, cmake, etc) to understand if a .rpm needs to be regenerated (because some of its packaged files have been changed) or not. Would RPM maintainers accept such a patch? Thanks for comments! Francesco ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?
Hi Jeff, I was taking a look at the "tar --to-command" option but unfortunately it seems that even latest TAR versions do not support "cpio" archive type. I think that hacking directly the rpmbuild utility would be indeed the most elegant solution to the problem, but I'm not sure how open/difficult is to provide a patch to a basic utility like rpmbuild (since it's used by million of people around)!! Francesco 2018-05-14 9:38 GMT+02:00 Francesco <francesco.monto...@gmail.com>: > Hi Jeff, > > thanks for the useful pointers. I definitively think that my current > Python proof-of-concept can be improved, mostly in 2 areas: > 1) the need of RPM unpacking, as you pointed out; > 2) I found out it fails miserably to do its job when the RPM contains a > compiled ELF: the md5sum of the original file copied into the build root > and the md5sum of the file extracted from RPM CPIO archive are different! I > guess that rpmbuild is doing some operation on the file before it gets > archived. Do you have any suggestions on how to disable such behavior from > rpmbuild? > > Anyway I think that an elegant solution to both these 2 problems would be > to have this (path,md5sum) generation available inside "rpmbuild" directly: > I think it would be the best place where to put the md5sum computation of a > packaged file, just before it gets compressed. > Do you think it would be acceptable a patch that adds a new flag > --generate-md5sum= to rpmbuild? > > Thanks a lot! > Francesco > > > > 2018-05-14 0:34 GMT+02:00 Jeff Johnson <n3...@me.com>: > >> >> >> On May 13, 2018, at 2:31 PM, Jeff Johnson <n3...@me.com> wrote: >> >> >> >> On May 11, 2018, at 7:31 AM, Francesco <francesco.monto...@gmail.com> >> wrote: >> >> Hi Jeff, Hi all, >> I'm resuming this (old) topic in case somebody is interested, I created a >> very small and simple Python utility >> >> https://github.com/f18m/rpm-make-rules-dependency-lister >> >> that does what I was describing in this topic: it allows to connect in a >> slightly smarter way GNU make and rpmbuild, >> and to avoid unnecessary RPM re-packaging operations >> >> HTH, >> Francesco >> >> >> Nice! >> >> Hmmm ... it looks like you want (path,md5) pairs in GNU md5sum format. Is >> that correct? >> >> If so, You might be able to simplify the code, using a utility like >> >> https://www.guyrutenberg.com/2008/10/24/tarsum-calculate-che >> cksum-for-files-inside-tar-archive/ >> >> An rpm payload is just a compressed cpio ball. >> >> Feed that to something like the above utility to spew the Makefile >> dependencies. >> >> Disclaimer: I haven't looked at the code, nor am I Python programmer. I'm >> just trying to remove the need to unpack the *.rpm payload. >> >> hth >> >> 73 de Jeff >> >> >> Here is another approach, using common system utilities: >> >> https://stackoverflow.com/questions/16423484/get-checksum- >> of-files-inside-tar-archive-on-remote-host >> >> There is an example there that doesn't need ssh, but you will need to >> use rpm2cpio into cpio (modern GNU tar handles cpio?), as well as create >> the (path,md5) pair using awk/sed/etc. >> >> Again: nothing wrong with your script, I'm just trying to avoid the rpm >> unpacking, as well as to simplify the need for Python as a build >> prerequisite in the Makefile rules. >> >> hth >> >> 73 de Jeff >> >> >> >> >> 2018-03-27 3:31 GMT+02:00 Jeff Johnson <n3...@me.com>: >> >>> >>> >>> On Mar 26, 2018, at 6:15 PM, Francesco <francesco.monto...@gmail.com> >>> wrote: >>> >>> Hi Jeff, >>> thanks for your reply. >>> >>> >>> >>> 2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3...@me.com>: >>> >>>> >>>> There isn't an explicit tool to convert rpmbuild dependencies to >>>> Makefile dependencies afaik. >>>> >>>> OTOH, it's not impossibly hard to script a couple of missing pieces >>>> that are needed: >>>> >>>> 1) Automate by rule generating a *.src.rpm from a (possibly modified) >>>> *.spec. >>>> >>>> 2) Automate by rule installing a *.src.rpm into a local build >>>> directory. This requires configuring some rpm macros to map rpmbuild >>>> inputs/outputs into a single directory. >>>> >>>> 3) Automate by rule rebuilding of bi
Re: [Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?
Hi Jeff, thanks for the useful pointers. I definitively think that my current Python proof-of-concept can be improved, mostly in 2 areas: 1) the need of RPM unpacking, as you pointed out; 2) I found out it fails miserably to do its job when the RPM contains a compiled ELF: the md5sum of the original file copied into the build root and the md5sum of the file extracted from RPM CPIO archive are different! I guess that rpmbuild is doing some operation on the file before it gets archived. Do you have any suggestions on how to disable such behavior from rpmbuild? Anyway I think that an elegant solution to both these 2 problems would be to have this (path,md5sum) generation available inside "rpmbuild" directly: I think it would be the best place where to put the md5sum computation of a packaged file, just before it gets compressed. Do you think it would be acceptable a patch that adds a new flag --generate-md5sum= to rpmbuild? Thanks a lot! Francesco 2018-05-14 0:34 GMT+02:00 Jeff Johnson <n3...@me.com>: > > > On May 13, 2018, at 2:31 PM, Jeff Johnson <n3...@me.com> wrote: > > > > On May 11, 2018, at 7:31 AM, Francesco <francesco.monto...@gmail.com> > wrote: > > Hi Jeff, Hi all, > I'm resuming this (old) topic in case somebody is interested, I created a > very small and simple Python utility > > https://github.com/f18m/rpm-make-rules-dependency-lister > > that does what I was describing in this topic: it allows to connect in a > slightly smarter way GNU make and rpmbuild, > and to avoid unnecessary RPM re-packaging operations > > HTH, > Francesco > > > Nice! > > Hmmm ... it looks like you want (path,md5) pairs in GNU md5sum format. Is > that correct? > > If so, You might be able to simplify the code, using a utility like > > https://www.guyrutenberg.com/2008/10/24/tarsum-calculate- > checksum-for-files-inside-tar-archive/ > > An rpm payload is just a compressed cpio ball. > > Feed that to something like the above utility to spew the Makefile > dependencies. > > Disclaimer: I haven't looked at the code, nor am I Python programmer. I'm > just trying to remove the need to unpack the *.rpm payload. > > hth > > 73 de Jeff > > > Here is another approach, using common system utilities: > > https://stackoverflow.com/questions/16423484/get- > checksum-of-files-inside-tar-archive-on-remote-host > > There is an example there that doesn't need ssh, but you will need to use > rpm2cpio into cpio (modern GNU tar handles cpio?), as well as create the > (path,md5) pair using awk/sed/etc. > > Again: nothing wrong with your script, I'm just trying to avoid the rpm > unpacking, as well as to simplify the need for Python as a build > prerequisite in the Makefile rules. > > hth > > 73 de Jeff > > > > > 2018-03-27 3:31 GMT+02:00 Jeff Johnson <n3...@me.com>: > >> >> >> On Mar 26, 2018, at 6:15 PM, Francesco <francesco.monto...@gmail.com> >> wrote: >> >> Hi Jeff, >> thanks for your reply. >> >> >> >> 2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3...@me.com>: >> >>> >>> There isn't an explicit tool to convert rpmbuild dependencies to >>> Makefile dependencies afaik. >>> >>> OTOH, it's not impossibly hard to script a couple of missing pieces that >>> are needed: >>> >>> 1) Automate by rule generating a *.src.rpm from a (possibly modified) >>> *.spec. >>> >>> 2) Automate by rule installing a *.src.rpm into a local build directory. >>> This requires configuring some rpm macros to map rpmbuild inputs/outputs >>> into a single directory. >>> >>> 3) Automate by rule rebuilding of binary *.rpm from a newer *.spec in >>> the local build directory >>> >>> You can find some useful rules using wild cards and % template rules in >>> rpm5 sources: see the >>> tests/Makefile.am file for the pattern rules. >>> >> >> Honestly I'm not sure to understand what you mean... when you say >> "Automate by rule" you mean writing a GNU make rule, right? >> >> >> Yes. >> >> Then it's clear to me step 1 and 2 but is not clear why should I get a >> newer .spec file in the local build directory only if some of my sources >> have changed: if step 1 and 2 are always executed by GNU make (which btw >> would be against my goal of running unneeded packaging operations) then I >> guess that the mtime of the .spec file in the local build directory will >> always change from run to run... what am I missing? >> >> >> Sorry to be obscure. The pattern rules I wrot
[Rpm-maint] Utility to generate a DOT graph of the inter-dependencies among a set of SPEC files
Hi all, maybe it can be useful for your RPM-related projects the following small Python utility https://github.com/f18m/rpm-spec-dependency-analyzer It's a very small util that parses a list of SPEC files and generates a nice graph of the dependencies among them using DOT. This proved very useful in my company where we build a large number of RPMs and have to deal with inter-dependencies among them. Bye, Francesco ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?
Hi Jeff, Hi all,
I'm resuming this (old) topic in case somebody is interested, I created a
very small and simple Python utility
https://github.com/f18m/rpm-make-rules-dependency-lister
that does what I was describing in this topic: it allows to connect in a
slightly smarter way GNU make and rpmbuild,
and to avoid unnecessary RPM re-packaging operations
HTH,
Francesco
2018-03-27 3:31 GMT+02:00 Jeff Johnson <n3...@me.com>:
>
>
> On Mar 26, 2018, at 6:15 PM, Francesco <francesco.monto...@gmail.com>
> wrote:
>
> Hi Jeff,
> thanks for your reply.
>
>
>
> 2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3...@me.com>:
>
>>
>> There isn't an explicit tool to convert rpmbuild dependencies to Makefile
>> dependencies afaik.
>>
>> OTOH, it's not impossibly hard to script a couple of missing pieces that
>> are needed:
>>
>> 1) Automate by rule generating a *.src.rpm from a (possibly modified)
>> *.spec.
>>
>> 2) Automate by rule installing a *.src.rpm into a local build directory.
>> This requires configuring some rpm macros to map rpmbuild inputs/outputs
>> into a single directory.
>>
>> 3) Automate by rule rebuilding of binary *.rpm from a newer *.spec in the
>> local build directory
>>
>> You can find some useful rules using wild cards and % template rules in
>> rpm5 sources: see the
>> tests/Makefile.am file for the pattern rules.
>>
>
> Honestly I'm not sure to understand what you mean... when you say
> "Automate by rule" you mean writing a GNU make rule, right?
>
>
> Yes.
>
> Then it's clear to me step 1 and 2 but is not clear why should I get a
> newer .spec file in the local build directory only if some of my sources
> have changed: if step 1 and 2 are always executed by GNU make (which btw
> would be against my goal of running unneeded packaging operations) then I
> guess that the mtime of the .spec file in the local build directory will
> always change from run to run... what am I missing?
>
>
> Sorry to be obscure. The pattern rules I wrote had 2 *.spec files: the top
> level *.spec determined when a *.src.rpm should be made, which triggered an
> install not a subdirectories, with a 2nd *.spec (copy) that triggers a
> rpmbuild by rule.
>
> Does that make sense? worksforme, for the purposes I needed, QA testing of
> a just built version of rpm within the buil tree ...
>
> Also I'm particularly interested in binary-only RPMs (I need such a tool
> for a commercial software)... to give you an idea, most of my RPM spec
> install sections look like:
>
> %install
> make -C ../mysources mytarget_install DESTDIR=%{buildroot}
>
>
> That's pretty generic, yes. Meanwhile the real problem is that rpmbuild
> within Makefiles is kinda awkward.
>
>
>
>> Alternatively, one could attempt generating a Makefile include rule.
>>
>> Extract (and filter) rpmbuild dependencies from a spec (or *.src.rpm)
>> file. RPM can query a spec file (or *.src.rpm), convert those (filtered)
>> dependencies to package names, and convert the package names to file names
>> that have a time stamp that can be included into a Makefile.
>>
>>
> Sorry, I'm not sure I get this either... I know RPM allows you to query
> dependencies but you can only query the package names listed under
> "Requires:" isn't it?
>
>
> Basically yes. But to automate within a Makefile (like dependencies on
> *.h)' the build requirements need to be mapped onto a file name that has a
> time stamp.
>
>
> Btw I have sketched out a possible "solution" for my problem: first time
> "make" is run, I call rpmbuild and build my binary RPM.
> Then (automatically via GNU make rules) I unzip that RPM into some
> temporary folder, I see what's inside and go search for MD5-sum matching
> files in the source build folder. Every time I find inside the source build
> folder a file that is inside the RPM, I write that into a .d file (same
> name of the RPM spec).
> The GNU makefile has an "include $(MY_SPEC_FILE_LIST:.spec=.d)" statement,
> so that it's aware of the auto-discovered dependencies of the spec file.
> Next time I run "make", it will be able to understand if the RPM is up to
> date or needs to be regenerated because some of the file it packages has
> been updated (perhaps as a result of the build process).
>
>
> Post a sample Makefile please. Lots of people have tried to use rpmbuild
> in Makefiles, and most of the solutions I have seen are rather awkward.
>
> Of course this process is not 100% accurate: in the %install section some
> temporary file may be generated and copied inside the RPM b
Re: [Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?
Hi Jeff,
thanks for your reply.
2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3...@me.com>:
>
> There isn't an explicit tool to convert rpmbuild dependencies to Makefile
> dependencies afaik.
>
> OTOH, it's not impossibly hard to script a couple of missing pieces that
> are needed:
>
> 1) Automate by rule generating a *.src.rpm from a (possibly modified)
> *.spec.
>
> 2) Automate by rule installing a *.src.rpm into a local build directory.
> This requires configuring some rpm macros to map rpmbuild inputs/outputs
> into a single directory.
>
> 3) Automate by rule rebuilding of binary *.rpm from a newer *.spec in the
> local build directory
>
> You can find some useful rules using wild cards and % template rules in
> rpm5 sources: see the
> tests/Makefile.am file for the pattern rules.
>
Honestly I'm not sure to understand what you mean... when you say "Automate
by rule" you mean writing a GNU make rule, right?
Then it's clear to me step 1 and 2 but is not clear why should I get a
newer .spec file in the local build directory only if some of my sources
have changed: if step 1 and 2 are always executed by GNU make (which btw
would be against my goal of running unneeded packaging operations) then I
guess that the mtime of the .spec file in the local build directory will
always change from run to run... what am I missing?
Also I'm particularly interested in binary-only RPMs (I need such a tool
for a commercial software)... to give you an idea, most of my RPM spec
install sections look like:
%install
make -C ../mysources mytarget_install DESTDIR=%{buildroot}
> Alternatively, one could attempt generating a Makefile include rule.
>
> Extract (and filter) rpmbuild dependencies from a spec (or *.src.rpm)
> file. RPM can query a spec file (or *.src.rpm), convert those (filtered)
> dependencies to package names, and convert the package names to file names
> that have a time stamp that can be included into a Makefile.
>
>
Sorry, I'm not sure I get this either... I know RPM allows you to query
dependencies but you can only query the package names listed under
"Requires:" isn't it?
Btw I have sketched out a possible "solution" for my problem: first time
"make" is run, I call rpmbuild and build my binary RPM.
Then (automatically via GNU make rules) I unzip that RPM into some
temporary folder, I see what's inside and go search for MD5-sum matching
files in the source build folder. Every time I find inside the source build
folder a file that is inside the RPM, I write that into a .d file (same
name of the RPM spec).
The GNU makefile has an "include $(MY_SPEC_FILE_LIST:.spec=.d)" statement,
so that it's aware of the auto-discovered dependencies of the spec file.
Next time I run "make", it will be able to understand if the RPM is up to
date or needs to be regenerated because some of the file it packages has
been updated (perhaps as a result of the build process).
Of course this process is not 100% accurate: in the %install section some
temporary file may be generated and copied inside the RPM build root. Or
files (e.g. config files) could be renamed when they get copied inside the
build root. And maybe there are other cases as well.
However that approach might cover a wide range of use cases... what do you
think?
It would be nice to have some mechanism like that inside rpmbuild so that I
can do
rpmbuild -MD myspec.spec --output=myspec.d
and it generates such dependency file for GNU make...
Thanks!
Francesco
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?
Hi all, I have a build system using GNU make that generates at some point a (large) set of RPMs packaging some C++ software programs built from my sources. After that step I do further processing using the freshly-built RPMs (like I regenerate RPM repositories, install these RPMs on containers to test deployment etc) Currently every time I invoke "make" I ALWAYS rebuild all RPMs, because I don't have an easy way to detect which stuff goes into which RPM. Of course I could manually build a list of such dependencies (e.g., spec A depends on the binary X, spec B depends on the binary utility Y) but that would become a mess very easily. So here's my question: Is there a way to ask rpm/rpmbuild to generate a list of a .spec file dependencies (for its %install stanza)? Something like -MT -MMD -MP -MF flags for GCC? Thanks, Francesco ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
