Example output:
...
original debug info size: 120600, size after compression: 77604
...
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/535
-- Commit Summary --
* Info how beneficial is dwz compression.
-- File Changes --
marxin commented on this pull request.
> @@ -489,6 +489,7 @@ if $run_dwz \
&& [ -d "${RPM_BUILD_ROOT}/usr/lib/debug" ]; then
readarray dwz_files < <(cd "${RPM_BUILD_ROOT}/usr/lib/debug"; find -type f
-name \*.debug | LC_ALL=C sort)
if [ ${#dwz_fil
Thanks for the patch.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/554#issuecomment-425878152___
Rpm-maint mailing list
I bet it's non-standard due to _--apparent-size_ option which _-b_ enables.
That said, what about using _-B1_ (a.k.a. _--block-size=1_)?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
Closed #553.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/553#event-1865152370___
Rpm-maint mailing list
du default to 1024 bytes as default unit, that's not desired
as one can overwrite that with BLOCK_SIZE env variable.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/553
-- Commit Summary --
* Info how beneficial is dwz
du default to 1024 bytes as default unit, that's not desired
as one can overwrite that with BLOCK_SIZE env variable.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/554
-- Commit Summary --
* Print debug info size in bytes.
> I edited the %_lto_cflags commit message a bit and merged this manually.
>
> Thanks for the patch!
Thank you very much for help.
Btw. when is planned another rpm release?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/606
-- Commit Summary --
* Come up with %_smp_flto macro that expands in a similar way to _smp_mflags.
-- File Changes --
M platform.in (6)
-- Patch Links --
Sorry. Currently -flto=N option can make LTO linking phase parallel. It's
desired to expand the RPM macro when a package is built in order to shorted
build time of a package.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> NAK as it is now. Copy-paste approach is not going to cut it. We need a
> separate macro that just figures out the number of cpu's we're allowed to
> use, which can be then used by any macros needing that info.
>
> Also this is a bit different from -j as AFAICS this is used for enabling LTO
> NAK. If someone actually wants all the output, they can append it themselves,
> especially since different Makefiles handle verbosity differently.
I see. What about a new ```%make_build_verbose``` that will do that?
--
You are receiving this because you are subscribed to this thread.
Reply
I consider it handy to print commands executed by make command.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/617
-- Commit Summary --
* Make %make_build to provide verbose output of make command.
-- File Changes --
Hello. May I please remind this patch review?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/606#issuecomment-452615568___
> Right. But the place in %optflags points out that this a not _make_ flag like
> -j is but compiler flag. So maybe it really should be _lto_cflags, or
> something like that, instead of _mflags.
Done in updated version of the patch.
>
> The other thing I'm wondering if this is something that
> I guess we're happy with the macro itself now, but the commit message doesn't
> match the action even remotely. Ideally this would be two commits, one for
> the _smp_mflags refactoring and a second one to add the new macro, and a
> proper rationale + explanations. For example, _smp_mflags
> That's indeed much better, thanks. There's still the naming mismatch with the
> concept though: %_smp_flto enables LTO regardless of number of cpus, and flto
> is the name of the compiler switch, the feature is simply LTO. Maybe it
> should be %_lto_mflags instead?
Yep, sounds better to me
> The question is whether that's sufficient or if there should be some sort of
> more user-friendly "disable lto" macro. OTOH those "user-friendly" disablers
> tend to suffer from various other side-effects...
That works fine for me! Then I won't need the
openSUSE/rpm-config-SUSE#5
Can the SR
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/660
-- Commit Summary --
* Use $ldflags in ./configure macro in order to set LDFLAGS.
-- File Changes --
M macros.in (1)
-- Patch Links --
marxin commented on this pull request.
> @@ -1021,6 +1021,7 @@ package or when debugging this package.\
CFLAGS="${CFLAGS:-%optflags}" ; export CFLAGS ; \
CXXFLAGS="${CXXFLAGS:-%optflags}" ; export CXXFLAGS ; \
FFLAGS="${FFLAGS:-%optflags}" ; export
Any estimation when this will be merged?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/692#issuecomment-491786233___
Rpm-maint
May I please ping this..
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/660#issuecomment-488596568___
Rpm-maint mailing list
Closed #660.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/660#event-2320800907___
Rpm-maint mailing list
Ok, I'm closing the pull request.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/660#issuecomment-489600068___
Rpm-maint
Motivation for the change:
I'm planning to enable LTO in `openSUSE:Factory` and I need to add `-flto=N` to
LDFLAGS in order to make LTO linking phase parallel. That speeds up package
build significantly.
--
You are receiving this because you are subscribed to this thread.
Reply to this email
> No objection to adding LDFLAGS to %configure, it's actually long overdue.
> Fedora has had its own version of this for a quite some time now, but nobody
> got around to upstreaming it. This seems like as good time as any to take a
> look at that and consider which parts would be upstreamable.
OK, I've updated the pull request based on the discussion that we've got so far.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> I think switching to `%{build_…}` convention makes much more sense. In the
> old times where `optflags` meant everything, nowadays you want to have
> different CFLAGS, CXXFLAGS. And having `%{build_cflags}` but `%{ldflags}`
> sounds inconsistent.
I'm fine with the `%build_ldflags` name which
> I can see -flto=auto being useful for the average upstream makefile defaults
> but I'm not convinced this is the right thing to do in rpm context: we'd want
> our parallel activities controlled via the same central tunables
> ($RPM_BUILD_NCPUS / %{_smp_ncpus_max}).
Which is way you want to
> Ah... now I get it. I somehow got the impression that it was the linking that
> produced different results with different number of cpus (as often happens
> with parallel compression etc)
That's good we understand each other.
No, our parallel linking in LTO is stable. Right now we divide the
> Maybe I'm missing something fundamental here, but I don't understand how is
> -flto=auto supposed to help with making builds more reproducable.
Because if you are given a builder with 8 cores:
```
[ 40s] + export 'CFLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2
-fstack-protector-strong -funwind-tables
> `%remove_optflags 'undesirable'`
> `%replace_optflags 'oldval' 'newval'`
Then we can have just the simple one. Following works for me:
```
$ rpm --define "%optflags -flto=auto -O2" --define "%replace_optflags() %global
optflags %(echo %optflags | sed s/%{?1}/%{?2}/)" --eval '%replace_optflags
The macro function is intended for %{optflags} manipulation
(replacement, conditional addition).
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/815
-- Commit Summary --
* Come up with replace_optflags (#814).
-- File
Closed #815.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/815#event-2556286671___
Rpm-maint mailing list
> I'm really reluctant to add one-trick-pony utility macros like this because
> over time they tend to become liabilities one way or the other, and have
> other issues.
Good.
>
> Manipulating compiler flags is a very common need indeed and could use some
> help from rpm, but that need is not
> In that case, would not it make sense to rename `%_lto_cflags` to
> `%_lto_flags`?
Well, we already have quite some usage of the name in our distribution. E.g. we
use `%define _lto_cflags %{nil}` to disable LTO for a package.
--
You are receiving this because you are subscribed to this
> The remaining question I have is whether it actually is worth it to have a
> macro for this at all when we don't default to it at all and even the default
> we ship is static so it's not something rpm would need to (or want to) be
> aware of. Ie, how does this differ from the average other
Adding @scarabeusiv
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/814#issuecomment-520846697___
Rpm-maint mailing list
As the follow up of https://github.com/rpm-software-management/rpm/pull/813
I would like to have rpm macros that will do operations on `%optflags`:
Example:
- `%filter-out-optflags '-flto=auto'`
- `%append-if-to-optflags '-flto=auto' -ffat-lto-objects`
I found the example how currently we do it
and @DimStar77
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/814#issuecomment-520847014___
Rpm-maint mailing list
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/812
-- Commit Summary --
* Remove %_lto_cflags.
-- File Changes --
M platform.in (3)
-- Patch Links --
https://github.com/rpm-software-management/rpm/pull/812.patch
Closed #809.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/809#event-2553371170___
Rpm-maint mailing list
> I'd think so.
Good. I've just done that in:
https://github.com/rpm-software-management/rpm/pull/812
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> In the meanwhile I learned that flto=auto is not supported by any released
> gcc. We can't really go with such a value in rpm.
Yep, it will be supported first in GCC 10.1 release. For openSUSE, we're using
a patched GCC compiler with the support. So I'm going to push the package to
our `rpm`
> Sure. It just needs to grow an existence outside GH to be relevant :)
That said I would like to keep the macro in `rpm` as suggested in the patch for
this pull request.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> It's also a fine example how basing anything on unreleased development work
> comes back to bite you.
Sure. So is the conclusion that we want to rip it off?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
After a long discussion, we as GCC community newly introduced
-flto=auto option. Doing that, LTO can automatically detect
make's jobserver or fall back to # of cores.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/809
--
> We'll need some sort of resource manager (no matter how crude) in rpm sooner
> than later to handle this sort of stuff, but in the
One simple possible solution can be using a semaphore that will be respected by
a compression library.
`zstd` issue about it lives here:
@marxin pushed 2 commits.
2c659f7ec7dea6fb1a4a25d13a91592c0acb7d97 Add get_compression_threads and
refactor code.
c771b3b43d05d76d0b19df9a2c25eba3c9f1e6a1 Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View
@marxin pushed 2 commits.
854bf2775fbaaa7b0621de46f3e42a165a5fd60c Add get_compression_threads and
refactor code.
b490f59832a056a897cd1835d7faed4a41585327 Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View
@marxin commented on this pull request.
> @@ -523,6 +523,27 @@ fprintf(stderr, "*** ufdOpen(%s,0x%x,0%o)\n", url,
> (unsigned)flags, (unsigned)mo
return fd;
}
+/* Return number of threads ought to be used for compression based
+ on a parsed value threads (e.g.
This proof-of-concept patch can handle when parallel compression happens in the
context of parallel `.rpm` file creation.
Its mentioned here: #1324 and compression semaphore can handle that.
`zstd` counterpart is here: facebook/zstd#2290.
You can view, comment on, or merge this pull request
@marxin pushed 1 commit.
10aeb351460a764806c558703db3c21d335ccd72 Use semaphore for compression in ZSTD.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1349/files
> Right, this isn't any worse than what we already have for XZ, but then again
> what is there for XZ is simply wrong in the face of concurrent package
> generation as we have now. The lowly IO stream has no way of knowing how many
> threads it can legitimately consume.
That said, would it be
@pmatilai I hope I addressed all your comments. Feel free to look at it now..
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> I think it should at least follow the same convention as XZ: T with no
> numbers mean autodetection (its okay if 0 means that too). This seems to
> silently change XZ behavior too which is not okay.
I'm going to update the documentation of my change. Yes, it was intentional to
sync behavior
> I haven't checked the zstd implementation, I admit, I just know that this was
> the problem with parallel bzip2/xz compression.
I've made some experiments for a 100MB big file and the compressed output is
still the same.
For being sure, I asked the question in
Demo of the used parallel compression for `cvise` package:
https://gist.github.com/marxin/5560256dbf0af4c749bd4d56c0e83626
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull
Port zstd compression to the new API (ZSTD_compressStream2).
My motivation is to eventually enable multi-threaded compression which can help
packages like Firefox where theres a single huge rpm that is being
compressed.
Once the change is accepted, the following one-liner will enable parallel
No. How does it break it?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-655620102___
Rpm-maint mailing list
Related to #256.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/1300#issuecomment-654130790___
Rpm-maint mailing list
I'm curious if `zstd` compression can utilize more threads for a single `.rpm`
file?
Is there any control of the threading level?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
The answer is that threads are not supported right now.
Adding the author of the port (@n3npq), he may be interested in?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
So the answer is that it's stable and reproducible.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
@marxin pushed 1 commit.
1d965759205bf5f072724c397004ba1a433f7c7f Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
@marxin pushed 1 commit.
fff307dd41294192560a33cca09e3675a0af925a Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
> This change looks good and straight forward.
Thanks!
>
> I am missing some information on what version of zstd supports the new API -
> and if necessary a check in configure.ac.
Updated `configure.ac` (one needs at least release `1.3.8`).
--
You are receiving this because you are
@marxin commented on this pull request.
>AS_IF([test "$enable_zstd" = "yes"], [
if test "$have_zstd" = "no"; then
AC_MSG_ERROR([--enable-zstd specified, but not available])
fi
])
+ PKG_CHECK_MODULES([ZSTD], [libzstd], [have_zst
@marxin pushed 2 commits.
077e20cb996777e8b3e918c4e520005c73df37bc zstd compression: port to the new API.
749bf9ecbadbbde5076bf722e1b9883e4df345ba Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https
@marxin pushed 1 commit.
d2465252ae96f05b874d453e7a245c2a47a4bd50 Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
@marxin commented on this pull request.
> + if (c >= (int)'0' && c <= (int)'9')
+ threads = strtol(s-1, (char **), 10);
Well, it's just a small piece of code. I would not put it to a separate
function.
--
You are receiving this because you are subscrib
@marxin commented on this pull request.
> goto err;
}
+
+ if (threads > 0)
Good point! I changed that to `> 1`, same what lzma does.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitH
> With my suggestions, and a proper build of libzstd in Fedora it works as
> expected, it fully utilizes my system.
Great to hear!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
@marxin pushed 1 commit.
14af110e70cc1f63e462eb11dab1230028fb8ce7 Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
> Looks good to me, thanks. (Of course the deltarpm and drpm need to be updated
> now so that they support a ZSTD_THREADED compression type as well.)
Thanks.
I can start working on the support for the mentioned tools. Can you please
guide me a bit what would be needed?
--
You are receiving
> I'll implement support in deltarpm and you can port it to drpm, ok?
Sure, appreciate that!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> I'm also not so sure about the "the multi-threaded output produces the same
> compressed data no matter how many threads you use" statement, because the
> block size seems to depend on the number of workers (see
> ZSTDMT_compress_advanced_internal).
I've just asked about this explicitly
All right, I changed the logic so that threaded mode is used only when a user
set a `T` value in the compression algorithm.
And I've just removed the automatic deduction of number of cores for now.
@mlschroe Hope it's useful now?
--
You are receiving this because you are subscribed to this
@marxin pushed 1 commit.
0281c07a8ba82cb73abdb5488a1d8a784db91f8e Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
@marxin pushed 1 commit.
11358ce35d0c98d716d68bb4824e86c44f78491b Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
Can please anybody unblock it?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303#issuecomment-665012491___
Rpm-maint mailing
@marxin pushed 1 commit.
790bd81bc215b7aea17cbc5eb2e7a53e15b99ded Support threading for zstd
compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1303/files
> Looks like it fails to detect the right commit to checkout and build. Can you
> force push a new version, please, to test if that get's it unstuck? Just do
> some white space changes to the commit message.
Done, good point!
--
You are receiving this because you are subscribed to this
> Looks like there's a hick-up in the CI. Re-running the semaphore tests.
>
> Well, trying...
Thanks, apparently it's still stuck..
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
Thank you for the review. Having 2 approvals can please anybody merge it?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
@marxin commented on this pull request.
> @@ -350,7 +350,7 @@ package or when debugging this package.\
# "w9.gzdio" gzip level 9 (default).
# "w9.bzdio" bzip2 level 9.
# "w6.xzdio" xz level 6, xz's defa
> From my POV it is good to go and we can follow up with improvements like
> auto-detection of cores.
Thank you.
Yes, I agree with that. I can imagine doing what `zstd` or `xz` do for
auto-detection of cores: `xz -T0 `.
--
You are receiving this because you are subscribed to this thread.
> I've just asked about this explicitly here:
> [facebook/zstd#2238
> (comment)](https://github.com/facebook/zstd/issues/2238#issuecomment-662906821)
And `zstd` folks confirmed that a number of threads does not influence the
stability of the "threaded" compression mode.
--
You are receiving
@marxin pushed 1 commit.
34a78ee61a67a9ffe11142c045fd1fa01e44160e Enable thread autodetection for a
parallel compression.
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1324/files
When using xz or zstd, it can be handy to automatically detect
number of threads. We can use `%{getncpus}` for that.
You can view, comment on, or merge this pull request online at:
https://github.com/rpm-software-management/rpm/pull/1324
-- Commit Summary --
* Enable thread autodetection
It's a follow up of #1303, so adding @ignatenkobrain as a reviewer ;)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
CI seems happy now, can please anybody merge it? I've got another patch based
on this one that I would like to propose..
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
Closed #1349.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/1349#event-4139307712___
Rpm-maint mailing list
Starting from `zstd` version `1.4.7`, the library provides an experimental
usage of thread pools (thats why `-DZSTD_STATIC_LINKING_ONLY=1` is
needed). Using them can fully utilize CPU on one side and maintain memory
utilization at the same time. Such an approach fixes #1324.
You can view,
May I please ping this pull request, adding people who helped me in previous
zstd-related PR: @ffesti @ignatenkobrain
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> Sorry but I can't see us accepting this. It's a layering violation, the build
> code cannot have any special knowledge of what goes on inside librpmio and
> vice versa.
Hm, that's a pity.
>
> Rpm really needs a resource manager (see #804) ... which hopefully can
> somehow integrate with
I see your concerns and the pull request is hack a bit. On the other hand,
`zstd` provides a nice thread pool functionality and package build really
benefits from it. Right now, packages like Firefox also mostly compressed in
one thread as compression of the biggest `.rpm` takes minutes.
--
In the single file mode, no new files are created so any new dependency cannot
be created by `dwz`.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
That sounds good. I guess we can fix possible issues incrementally, right?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
> Hmm, I don't know. Are the extra requires gone if there is nore than one
> debuginfo package?
What do you mean by "extra requires"? Please provide an example you have
concerns about.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it
1 - 100 of 105 matches
Mail list logo
