Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
> So all debugedit-stuff is frozen in rpm and pending removal, and patches > should be submitted to https://sourceware.org/debugedit/ instead. Thanks. Makes full sense. Btw. the request was merged to the new upstream project here: https://sourceware.org/git/?p=debugedit.git;a=commit;h=1bcf472a7140220395a8eb22b30faae811de7f51 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-823065951___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
@mlschroe : May I please ping this? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-802919085___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
> Why are we doing this? We do it in order to reduce dependencies in-between `debuginfo` sub-packages of a package that uses baselibs mechanism: https://en.opensuse.org/openSUSE:Build_Service_baselibs.conf#Quickstart -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-800543997___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
That sounds good. I guess we can fix possible issues incrementally, right? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-800375139___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
> Hmm, I don't know. Are the extra requires gone if there is nore than one > debuginfo package? What do you mean by "extra requires"? Please provide an example you have concerns about. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-800372177___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
Ok, second part of my attempt is here: https://build.opensuse.org/package/rdiff/home:marxin:branches:Base:System/rpm?opackage=rpm=Base%3ASystem=2 Do I miss something else? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-800248729___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
In the single file mode, no new files are created so any new dependency cannot be created by `dwz`. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579#issuecomment-800159184___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Add --dwz-single-file-mode argument for find-debuginfo.sh. (#1579)
Sometimes its handy to disable multi-file mode and the patch adds option for that: `--dwz-single-file-mode`. It will be used in openSUSE for packages that use baselibs.conf mechanism. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1579 -- Commit Summary -- * Add --dwz-single-file-mode argument for find-debuginfo.sh. -- File Changes -- M scripts/find-debuginfo.sh (9) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/1579.patch https://github.com/rpm-software-management/rpm/pull/1579.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1579 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd: use thread pool (#1466)
I see your concerns and the pull request is hack a bit. On the other hand, `zstd` provides a nice thread pool functionality and package build really benefits from it. Right now, packages like Firefox also mostly compressed in one thread as compression of the biggest `.rpm` takes minutes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1466#issuecomment-769711886___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd: use thread pool (#1466)
> Sorry but I can't see us accepting this. It's a layering violation, the build > code cannot have any special knowledge of what goes on inside librpmio and > vice versa. Hm, that's a pity. > > Rpm really needs a resource manager (see #804) ... which hopefully can > somehow integrate with these library-specific things as well. Are you planning to work on the feature in near future? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1466#issuecomment-769075139___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd: use thread pool (#1466)
May I please ping this pull request, adding people who helped me in previous zstd-related PR: @ffesti @ignatenkobrain -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1466#issuecomment-757340252___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] zstd: use thread pool (#1466)
Starting from `zstd` version `1.4.7`, the library provides an experimental usage of thread pools (thats why `-DZSTD_STATIC_LINKING_ONLY=1` is needed). Using them can fully utilize CPU on one side and maintain memory utilization at the same time. Such an approach fixes #1324. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1466 -- Commit Summary -- * zstd: use thread pool -- File Changes -- M build/Makefile.am (1) M build/pack.c (19) M rpmio/Makefile.am (2) M rpmio/rpmio.c (11) M rpmio/rpmio.h (2) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/1466.patch https://github.com/rpm-software-management/rpm/pull/1466.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1466 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] [WIP] Use semaphore for compression in ZSTD. (#1349)
Closed #1349. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1349#event-4139307712___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] [WIP] Use semaphore for compression in ZSTD. (#1349)
@marxin pushed 1 commit. 10aeb351460a764806c558703db3c21d335ccd72 Use semaphore for compression in ZSTD. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1349/files/143d455761c53aad68045cada816b9d15eb13ee5..10aeb351460a764806c558703db3c21d335ccd72 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] [WIP] Use semaphore for compression in ZSTD. (#1349)
This proof-of-concept patch can handle when parallel compression happens in the context of parallel `.rpm` file creation. Its mentioned here: #1324 and compression semaphore can handle that. `zstd` counterpart is here: facebook/zstd#2290. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1349 -- Commit Summary -- * Use semaphore for compression in ZSTD. -- File Changes -- M build/pack.c (4) M rpmio/rpmio.c (4) M rpmio/rpmio.h (3) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/1349.patch https://github.com/rpm-software-management/rpm/pull/1349.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1349 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
> We'll need some sort of resource manager (no matter how crude) in rpm sooner > than later to handle this sort of stuff, but in the One simple possible solution can be using a semaphore that will be respected by a compression library. `zstd` issue about it lives here: https://github.com/facebook/zstd/issues/1097 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-680826776___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
@marxin commented on this pull request. > @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url, > (unsigned)flags, (unsigned)mo return fd; } +/* Return number of threads ought to be used for compression based + on a parsed value threads (e.g. from w7T0.xzdio or w7T16.xzdio). + Value -1 means automatic detection. */ + +static int +get_compression_threads (int threads) Sure. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#discussion_r472862960___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
@marxin pushed 2 commits. 2c659f7ec7dea6fb1a4a25d13a91592c0acb7d97 Add get_compression_threads and refactor code. c771b3b43d05d76d0b19df9a2c25eba3c9f1e6a1 Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324/files/b490f59832a056a897cd1835d7faed4a41585327..c771b3b43d05d76d0b19df9a2c25eba3c9f1e6a1 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
@marxin pushed 2 commits. 854bf2775fbaaa7b0621de46f3e42a165a5fd60c Add get_compression_threads and refactor code. b490f59832a056a897cd1835d7faed4a41585327 Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324/files/8480318844c3d88c1752ae0e951ba06cf0fe1b50..b490f59832a056a897cd1835d7faed4a41585327 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
@pmatilai I hope I addressed all your comments. Feel free to look at it now.. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-672804296___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
> I think it should at least follow the same convention as XZ: T with no > numbers mean autodetection (its okay if 0 means that too). This seems to > silently change XZ behavior too which is not okay. I'm going to update the documentation of my change. Yes, it was intentional to sync behavior of `T0` and `T`, as `zstd` and `xz` use `-T0` in order to autodetect number of threads. > We'll need some sort of resource manager (no matter how crude) in rpm sooner > than later to handle this sort of stuff, but in the meanwhile I think a first > step towards sanity could be axing the crazily complicated logic from XZ and > simply limit threading to 64bit platforms. Works for me. > And with that, it should be possible to refactor this to use common > thread-number autodetection code between xzstd and the XZ case. I would like to. Currently, `xz` uses the memory limit. That functionality is missing for `zstd`, but as said, it does not work for concurrent creation of `rpm` files. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-670848352___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
> Right, this isn't any worse than what we already have for XZ, but then again > what is there for XZ is simply wrong in the face of concurrent package > generation as we have now. The lowly IO stream has no way of knowing how many > threads it can legitimately consume. That said, would it be possible to merge my pull request? I'm planning to experiment with `T0` when building `openSUSE:Factory`. Or do you see a way how to communicate the top-level parallelism to the compressing API? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-668488681___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
@marxin pushed 1 commit. 34a78ee61a67a9ffe11142c045fd1fa01e44160e Enable thread autodetection for a parallel compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324/files/cd31048911a1d3c6094e4530f5e7e4975849be4a..34a78ee61a67a9ffe11142c045fd1fa01e44160e ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
It's a follow up of #1303, so adding @ignatenkobrain as a reviewer ;) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1324#issuecomment-666359866___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Enable thread autodetection for a parallel compression. (#1324)
When using xz or zstd, it can be handy to automatically detect
number of threads. We can use `%{getncpus}` for that.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/1324
-- Commit Summary --
* Enable thread autodetection for a parallel compression.
-- File Changes --
M macros.in (1)
M rpmio/rpmio.c (14)
-- Patch Links --
https://github.com/rpm-software-management/rpm/pull/1324.patch
https://github.com/rpm-software-management/rpm/pull/1324.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
CI seems happy now, can please anybody merge it? I've got another patch based on this one that I would like to propose.. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-666188501___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> Looks like it fails to detect the right commit to checkout and build. Can you > force push a new version, please, to test if that get's it unstuck? Just do > some white space changes to the commit message. Done, good point! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-665651388___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. 790bd81bc215b7aea17cbc5eb2e7a53e15b99ded Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/11358ce35d0c98d716d68bb4824e86c44f78491b..790bd81bc215b7aea17cbc5eb2e7a53e15b99ded ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
Can please anybody unblock it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-665012491___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> Looks like there's a hick-up in the CI. Re-running the semaphore tests. > > Well, trying... Thanks, apparently it's still stuck.. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-664292711___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
Thank you for the review. Having 2 approvals can please anybody merge it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-664149460___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. 11358ce35d0c98d716d68bb4824e86c44f78491b Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/0281c07a8ba82cb73abdb5488a1d8a784db91f8e..11358ce35d0c98d716d68bb4824e86c44f78491b ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin commented on this pull request. > @@ -350,7 +350,7 @@ package or when debugging this package.\ # "w9.gzdio" gzip level 9 (default). # "w9.bzdio" bzip2 level 9. # "w6.xzdio" xz level 6, xz's default. -# "w7T16.xzdio" xz level 7 using 16 thread (xz only) +# "w7T16.xzdio" xz level 7 using 16 thread (xz and zstd only) I understand it that the description in parenthesis is related to the threading mode. And `zstd` newly support it. Or am I wrong? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#discussion_r460485286___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> I'll implement support in deltarpm and you can port it to drpm, ok? Sure, appreciate that! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-663451176___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> Looks good to me, thanks. (Of course the deltarpm and drpm need to be updated > now so that they support a ZSTD_THREADED compression type as well.) Thanks. I can start working on the support for the mentioned tools. Can you please guide me a bit what would be needed? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-663448525___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> I've just asked about this explicitly here: > [facebook/zstd#2238 > (comment)](https://github.com/facebook/zstd/issues/2238#issuecomment-662906821) And `zstd` folks confirmed that a number of threads does not influence the stability of the "threaded" compression mode. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-663370519___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> From my POV it is good to go and we can follow up with improvements like > auto-detection of cores. Thank you. Yes, I agree with that. I can imagine doing what `zstd` or `xz` do for auto-detection of cores: `xz -T0 `. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-663160218___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> I'm also not so sure about the "the multi-threaded output produces the same > compressed data no matter how many threads you use" statement, because the > block size seems to depend on the number of workers (see > ZSTDMT_compress_advanced_internal). I've just asked about this explicitly here: https://github.com/facebook/zstd/issues/2238#issuecomment-662906821 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-662907291___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
All right, I changed the logic so that threaded mode is used only when a user set a `T` value in the compression algorithm. And I've just removed the automatic deduction of number of cores for now. @mlschroe Hope it's useful now? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-662906324___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. 0281c07a8ba82cb73abdb5488a1d8a784db91f8e Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/14af110e70cc1f63e462eb11dab1230028fb8ce7..0281c07a8ba82cb73abdb5488a1d8a784db91f8e ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. 14af110e70cc1f63e462eb11dab1230028fb8ce7 Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/d2465252ae96f05b874d453e7a245c2a47a4bd50..14af110e70cc1f63e462eb11dab1230028fb8ce7 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> With my suggestions, and a proper build of libzstd in Fedora it works as > expected, it fully utilizes my system. Great to hear! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-657103190___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin commented on this pull request. > goto err; } + + if (threads > 0) Good point! I changed that to `> 1`, same what lzma does. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#discussion_r453218566___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin commented on this pull request. > + if (c >= (int)'0' && c <= (int)'9') + threads = strtol(s-1, (char **), 10); Well, it's just a small piece of code. I would not put it to a separate function. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#discussion_r453218506___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. d2465252ae96f05b874d453e7a245c2a47a4bd50 Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/749bf9ecbadbbde5076bf722e1b9883e4df345ba..d2465252ae96f05b874d453e7a245c2a47a4bd50 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin commented on this pull request. >AS_IF([test "$enable_zstd" = "yes"], [ if test "$have_zstd" = "no"; then AC_MSG_ERROR([--enable-zstd specified, but not available]) fi ]) + PKG_CHECK_MODULES([ZSTD], [libzstd], [have_zstd=yes], [have_zstd=no]) Oh yeah, it is new to me! Thanks. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#discussion_r453176326___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 2 commits. 077e20cb996777e8b3e918c4e520005c73df37bc zstd compression: port to the new API. 749bf9ecbadbbde5076bf722e1b9883e4df345ba Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/2d42b78f50fb7b4f3848f0ceae916595214a5b89..749bf9ecbadbbde5076bf722e1b9883e4df345ba ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
> This change looks good and straight forward. Thanks! > > I am missing some information on what version of zstd supports the new API - > and if necessary a check in configure.ac. Updated `configure.ac` (one needs at least release `1.3.8`). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656696346___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Support threading for zstd compression. (#1303)
@marxin pushed 1 commit. 1d965759205bf5f072724c397004ba1a433f7c7f Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/fff307dd41294192560a33cca09e3675a0af925a..1d965759205bf5f072724c397004ba1a433f7c7f ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
@marxin pushed 1 commit. fff307dd41294192560a33cca09e3675a0af925a Support threading for zstd compression. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303/files/fec86dfc33a67ce75c0113b64d0ec4fede975eab..fff307dd41294192560a33cca09e3675a0af925a ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
So the answer is that it's stable and reproducible. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656525918___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
> I haven't checked the zstd implementation, I admit, I just know that this was > the problem with parallel bzip2/xz compression. I've made some experiments for a 100MB big file and the compressed output is still the same. For being sure, I asked the question in https://github.com/facebook/zstd/issues/2238. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-656046924___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
No. How does it break it? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655620102___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
Demo of the used parallel compression for `cvise` package: https://gist.github.com/marxin/5560256dbf0af4c749bd4d56c0e83626 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655495389___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] zstd compression: port to the new API. (#1303)
Port zstd compression to the new API (ZSTD_compressStream2). My motivation is to eventually enable multi-threaded compression which can help packages like Firefox where theres a single huge rpm that is being compressed. Once the change is accepted, the following one-liner will enable parallel compression: ```c ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4); ``` You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/1303 -- Commit Summary -- * zstd compression: port to the new API. -- File Changes -- M rpmio/rpmio.c (68) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/1303.patch https://github.com/rpm-software-management/rpm/pull/1303.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/1303 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Question: multi-threaded zstd compression (#1300)
The answer is that threads are not supported right now. Adding the author of the port (@n3npq), he may be interested in? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1300#issuecomment-654641377___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Question: multi-threaded zstd compression (#1300)
Related to #256. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1300#issuecomment-654130790___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Question: multi-threaded zstd compression (#1300)
I'm curious if `zstd` compression can utilize more threads for a single `.rpm` file? Is there any control of the threading level? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1300___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with replace_optflags (#814). (#815)
Closed #815. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/815#event-2556286671___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with replace_optflags (#814). (#815)
> I'm really reluctant to add one-trick-pony utility macros like this because > over time they tend to become liabilities one way or the other, and have > other issues. Good. > > Manipulating compiler flags is a very common need indeed and could use some > help from rpm, but that need is not limited to %optflags, there are various > other similar things as well and I certainly don't want separate macros for > updating %build_cflags, %build_cxxflags, %build_ldflags and so on. To be honest, I'm not much familiar with `m4` language and how `rpm` works. Can you please help me how should such macro look like? Should it take the variable name as a first argument? > > I'd be more inclined to consider a generic utility or macro primitive that > helps in these kind of tasks. Also, it's worth considering whether the flags > be stored in a more manipulation friendly manner in the first place. I'm open for a better-defined API. Can you help me with that as well? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/815#issuecomment-521206621___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Come up with replace_optflags (#814). (#815)
The macro function is intended for %{optflags} manipulation
(replacement, conditional addition).
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/815
-- Commit Summary --
* Come up with replace_optflags (#814).
-- File Changes --
M macros.in (7)
-- Patch Links --
https://github.com/rpm-software-management/rpm/pull/815.patch
https://github.com/rpm-software-management/rpm/pull/815.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/815
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Provide function for $optflags manipulation (#814)
> `%remove_optflags 'undesirable'`
> `%replace_optflags 'oldval' 'newval'`
Then we can have just the simple one. Following works for me:
```
$ rpm --define "%optflags -flto=auto -O2" --define "%replace_optflags() %global
optflags %(echo %optflags | sed s/%{?1}/%{?2}/)" --eval '%replace_optflags
"-flto=auto"' --eval 'optflags=%optflags'
optflags= -O2
rpm --define "%optflags -flto=auto -O2" --define "%replace_optflags() %global
optflags %(echo %optflags | sed s/%{?1}/%{?2}/)" --eval '%replace_optflags
"-O2" "-O3"' --eval 'optflags=%optflags'
optflags=-flto=auto -O3
```
Thoughts?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/814#issuecomment-520858624___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Provide function for $optflags manipulation (#814)
and @DimStar77 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/814#issuecomment-520847014___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Provide function for $optflags manipulation (#814)
Adding @scarabeusiv -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/814#issuecomment-520846697___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Provide function for $optflags manipulation (#814)
As the follow up of https://github.com/rpm-software-management/rpm/pull/813
I would like to have rpm macros that will do operations on `%optflags`:
Example:
- `%filter-out-optflags '-flto=auto'`
- `%append-if-to-optflags '-flto=auto' -ffat-lto-objects`
I found the example how currently we do it e.g. in MozillaFirefox:
```
%{expand:%%global optflags %(echo "%optflags"|sed -e s/i586/i686/) -march=i686
-mtune=generic}
```
Which is quite ugly and I bet multiple packages suffer from the same issue.
Gentoo has a reasonable API for that:
https://devmanual.gentoo.org/eclass-reference/flag-o-matic.eclass/index.html
@pmatilai Is it realistic to come up with the functionality?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/814___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> I'd think so. Good. I've just done that in: https://github.com/rpm-software-management/rpm/pull/812 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520833825___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
Closed #809. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#event-2553371170___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Remove %_lto_cflags. (#812)
You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/812 -- Commit Summary -- * Remove %_lto_cflags. -- File Changes -- M platform.in (3) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/812.patch https://github.com/rpm-software-management/rpm/pull/812.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/812 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> It's also a fine example how basing anything on unreleased development work > comes back to bite you. Sure. So is the conclusion that we want to rip it off? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520811301___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> In the meanwhile I learned that flto=auto is not supported by any released > gcc. We can't really go with such a value in rpm. Yep, it will be supported first in GCC 10.1 release. For openSUSE, we're using a patched GCC compiler with the support. So I'm going to push the package to our `rpm` package. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520806185___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> Sure. It just needs to grow an existence outside GH to be relevant :) That said I would like to keep the macro in `rpm` as suggested in the patch for this pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520802750___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> The remaining question I have is whether it actually is worth it to have a
> macro for this at all when we don't default to it at all and even the default
> we ship is static so it's not something rpm would need to (or want to) be
> aware of. Ie, how does this differ from the average other compiler flag that
> can cause some packages to fail build, that distros set in their default
> %optflags?
I must agree here with you. It's probably a patch that we want to have locally
in our `rpm` package as we use a lot the following construct:
`%define _lto_cflags %{nil}`. That said, will you accept a patch that will
remove it?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520746476___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> In that case, would not it make sense to rename `%_lto_cflags` to
> `%_lto_flags`?
Well, we already have quite some usage of the name in our distribution. E.g. we
use `%define _lto_cflags %{nil}` to disable LTO for a package.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520741821___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> Ah... now I get it. I somehow got the impression that it was the linking that > produced different results with different number of cpus (as often happens > with parallel compression etc) That's good we understand each other. No, our parallel linking in LTO is stable. Right now we divide the job to `128` parts and then these parts are processed in parallel by multiple processes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520399131___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> Maybe I'm missing something fundamental here, but I don't understand how is
> -flto=auto supposed to help with making builds more reproducable.
Because if you are given a builder with 8 cores:
```
[ 40s] + export 'CFLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2
-fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables
-fstack-clash-protection -Werror=return-type -flto=8 -g'
```
Then `rpm -qp --qf "%{OPTFLAGS}" $rpm` will show you the `-flto=8` and that's
the problem for reproducibility.
> In rpm context, the number of cpus make uses is typically set by rpm (whether
> configuration or "all available"), and if you let it fall back to "as many as
> possible" then it's just as system dependent as the current rpm set method,
> no?
Yes, except we can communicate with `make` job server and run tasks
dynamically. And if not, then we'll do the same. Except we'll not have issues
with reproducibility.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520397317___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
> I can see -flto=auto being useful for the average upstream makefile defaults
> but I'm not convinced this is the right thing to do in rpm context: we'd want
> our parallel activities controlled via the same central tunables
> ($RPM_BUILD_NCPUS / %{_smp_ncpus_max}).
Which is way you want to controll a build system (make). Note that `-flto=N` is
used for parallel linking phase and with the new option `-flto=auto` we can
respect make's jobserver parallelism level
and communication with it. And as a fallback we want to make linking as
parallel as possible. That's what we do in openSUSE right now.
> If LTO falls back to detecting number of cores on its own, that goal was
> missed.
>
> Is there a summary of the gcc community reasoning someplace or can you
> provide one?
The main problem is that we see a lot of differences due to package builds with
different `-flto=N` values. That's because some packages embed options (for
`--help`) option. And mainly we want to have a bitwise identical `rpm` for
reproducible builds. That's why we want to leave `-flto=N` option.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/809#issuecomment-520392154___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Use -flto=auto as a default for _lto_cflags. (#809)
After a long discussion, we as GCC community newly introduced -flto=auto option. Doing that, LTO can automatically detect make's jobserver or fall back to # of cores. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/809 -- Commit Summary -- * Use -flto=auto as a default for _lto_cflags. -- File Changes -- M platform.in (6) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/809.patch https://github.com/rpm-software-management/rpm/pull/809.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/809 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Adopt compiler flags related enhancements from Fedora (#692)
Any estimation when this will be merged? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/692#issuecomment-491786233___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
Closed #660. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#event-2320800907___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
Ok, I'm closing the pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#issuecomment-489600068___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
May I please ping this.. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#issuecomment-488596568___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
OK, I've updated the pull request based on the discussion that we've got so far. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#issuecomment-486695556___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
> I think switching to `%{build_…}` convention makes much more sense. In the
> old times where `optflags` meant everything, nowadays you want to have
> different CFLAGS, CXXFLAGS. And having `%{build_cflags}` but `%{ldflags}`
> sounds inconsistent.
I'm fine with the `%build_ldflags` name which is aligned with the
`redhat-rpm-config` name conventions.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/660#issuecomment-486668259___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
> No objection to adding LDFLAGS to %configure, it's actually long overdue. > Fedora has had its own version of this for a quite some time now, but nobody > got around to upstreaming it. This seems like as good time as any to take a > look at that and consider which parts would be upstreamable. Would you be fine with this pull request as is? > > There are a whole bunch of related things done, mostly done in > redhat-rpm-config. For details, see > https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/macros#_27 > but in summary: > > * %optflags is only a building block for language specific %build_fooflags > macros (except for ldflags which is not related) > * the %build_fooflags macros are used for setting the environment > * the environment setting is separated from %configure so it's usable for > other make systems too > * there are further %extension_fooflags macros which are meant for building > things like perl and python C-extensions > > In addition, %build_ldflags is passed as RPM_LD_FLAGS to all the build > scripts via %___build_pre, but that's an older addition than the > redhat-rpm-config stuff above so not sure how relevant/useful it is in > reality. I'm CCing SUSE related people how are much more skilled with that: @scarabeusiv @DimStar77 @mlschroe -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#issuecomment-486575591___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
Motivation for the change: I'm planning to enable LTO in `openSUSE:Factory` and I need to add `-flto=N` to LDFLAGS in order to make LTO linking phase parallel. That speeds up package build significantly. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660#issuecomment-485789563___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
marxin commented on this pull request.
> @@ -1021,6 +1021,7 @@ package or when debugging this package.\
CFLAGS="${CFLAGS:-%optflags}" ; export CFLAGS ; \
CXXFLAGS="${CXXFLAGS:-%optflags}" ; export CXXFLAGS ; \
FFLAGS="${FFLAGS:-%optflags}" ; export FFLAGS ; \
+ LDFLAGS="${LDFLAGS:-%?ldflags}" ; export LDFLAGS; \
Well, I would expect the `autotools` should be fine with empty `LDFLAGS`.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/660#discussion_r275231576___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Use $ldflags in ./configure macro in order to set LDFLAGS. (#660)
You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/660 -- Commit Summary -- * Use $ldflags in ./configure macro in order to set LDFLAGS. -- File Changes -- M macros.in (1) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/660.patch https://github.com/rpm-software-management/rpm/pull/660.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/660 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> I edited the %_lto_cflags commit message a bit and merged this manually. > > Thanks for the patch! Thank you very much for help. Btw. when is planned another rpm release? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-454311253___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> I guess we're happy with the macro itself now, but the commit message doesn't > match the action even remotely. Ideally this would be two commits, one for > the _smp_mflags refactoring and a second one to add the new macro, and a > proper rationale + explanations. For example, _smp_mflags doesn't behave > identically now because previously we didn't supply -j at all in the case of > single cpu. Done. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-454016006___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> The question is whether that's sufficient or if there should be some sort of > more user-friendly "disable lto" macro. OTOH those "user-friendly" disablers > tend to suffer from various other side-effects... That works fine for me! Then I won't need the openSUSE/rpm-config-SUSE#5 Can the SR be merged now? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-454010479___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> Right. But the place in %optflags points out that this a not _make_ flag like
> -j is but compiler flag. So maybe it really should be _lto_cflags, or
> something like that, instead of _mflags.
Done in updated version of the patch.
>
> The other thing I'm wondering if this is something that needs a special
> disabler (no matter what you do there'll be somebody who needs a way to
> disable it ) or if just redefining it to %{nil} is sufficient (probably it
> is).
I was thinking about something like:
https://github.com/openSUSE/rpm-config-SUSE/pull/5
Do you believe it should land in rpm package and not in the SUSE-specific one?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/606#issuecomment-453999500___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> That's indeed much better, thanks. There's still the naming mismatch with the > concept though: %_smp_flto enables LTO regardless of number of cpus, and flto > is the name of the compiler switch, the feature is simply LTO. Maybe it > should be %_lto_mflags instead? Yep, sounds better to me as well! > > How you envision this gets used in practise - is it something that eg distros > might want to enable by default some day, or is it strictly per-spec thing? Yes, I'm preparing openSUSE Factory for that. > Just thinking whether there should be some integration with %make_build for > example. We're planning to enable that via ```$optflags``` where we'll add the new ```%_lto_mflags```. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-453988419___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
> NAK as it is now. Copy-paste approach is not going to cut it. We need a > separate macro that just figures out the number of cpu's we're allowed to > use, which can be then used by any macros needing that info. > > Also this is a bit different from -j as AFAICS this is used for enabling LTO > regardless of the number of cpus, so the name is misleading / non-descriptive. You are right, refactoring was needed. Should be updated now. Ready to be merged? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-453927663___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Make %make_build to provide verbose output of make command. (#617)
> NAK. If someone actually wants all the output, they can append it themselves, > especially since different Makefiles handle verbosity differently. I see. What about a new ```%make_build_verbose``` that will do that? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/617#issuecomment-452687487___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Make %make_build to provide verbose output of make command. (#617)
I consider it handy to print commands executed by make command. You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/617 -- Commit Summary -- * Make %make_build to provide verbose output of make command. -- File Changes -- M macros.in (2) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/617.patch https://github.com/rpm-software-management/rpm/pull/617.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/617 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
Hello. May I please remind this patch review? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-452615568___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
Sorry. Currently -flto=N option can make LTO linking phase parallel. It's desired to expand the RPM macro when a package is built in order to shorted build time of a package. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606#issuecomment-442398628___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] [rpm-software-management/rpm] Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. (#606)
You can view, comment on, or merge this pull request online at: https://github.com/rpm-software-management/rpm/pull/606 -- Commit Summary -- * Come up with %_smp_flto macro that expands in a similar way to _smp_mflags. -- File Changes -- M platform.in (6) -- Patch Links -- https://github.com/rpm-software-management/rpm/pull/606.patch https://github.com/rpm-software-management/rpm/pull/606.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/606 ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Print debug info size in bytes. (#554)
Thanks for the patch. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/554#issuecomment-425878152___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] Print debug info size in bytes. (#554)
I bet it's non-standard due to _--apparent-size_ option which _-b_ enables. That said, what about using _-B1_ (a.k.a. _--block-size=1_)? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/pull/554#issuecomment-424683384___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
