Re: Switching XZ for ZSTD?
>[1] https://github.com/facebook/zstd/issues/395#issuecomment-535875379 > . >If that was part of zstd or even actively being looked at, then yes. I mean, per your own comment on that thread, the API *is* available and it's in zstd, but no frontend supports it yet. And per the maintainer's comment (https://github.com/facebook/zstd/issues/395#issuecomment-492741194) the only thing preventing it from becoming more official, is being "battle-tested", which means shipping a frontend that does use it (perhaps a third-party one until it's stabilized completely) and getting people to use it. They directly compare it to a chicken-and-egg problem (https://github.com/facebook/zstd/issues/395#issuecomment-492808642) and say that there's nothing more to do upstream until that happens. And per the last comment in that thread (https://github.com/facebook/zstd/issues/395#issuecomment-974796390), there is a suggestion that the frontends already exist, it's just that nobody distributes them yet, and that making that happen could expedite the process. So I guess I'm confused about what the blocker is? If Fedora wants seekable zstd, then Fedora can make it happen by being the head on the spear. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Switching XZ for ZSTD?
It's not possible to simply substitute one for another universally, there's no "Fedora default", it's something that would need to be handled on a package-by-package basis. As long as there are existing xz-compressed files in the wild, Fedora will need to support consuming them - as long as there is software that expects xz compression, Fedora will need to support creating them. It's not going to disappear any time soon, and until then we're stuck with xz All that being said, there are plenty of bits of software that could *start* using zstd by default and it would probably make sense to do so. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Three steps we could take to make supply chain attacks a bit harder
It was unreasonable before zlib-ng too [0], but zlib-ng does make it a slightly bigger issue. [0] https://github.com/rpm-software-management/createrepo_c/pull/336 -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Three steps we could take to make supply chain attacks a bit harder
It's not how free software works, but there are some interesting projects working on (distributed, not centrally managed) code review systems that are kind of similar in spirit to what OP describes. https://github.com/crev-dev/crev https://github.com/crev-dev/cargo-crev https://mozilla.github.io/cargo-vet/ That is, individuals and organizations can publish the results of their code audits publicly in a standardized format, tied to a package artifact, and a downstream consumer could denote which individuals and organizations they trust to perform said audits. It's technically possible and not an entirely ridiculous idea, it's just economically challenging. How do you find enough people willing and able to audit a package (including e.g. autotools build scripts), in multiple, to the level of scrutiny that would be required to catch something like this. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: xz backdoor
It appears that libsystemd links to libraries for lzma/xz, bzip2, gzip and also zstd, because some systemd utilities provide them as options in various different contexts (but not consistently, zstd for instance is seemingly supported by some utilities and not by others, and I see some code such as [0] that doesn't account for it) I'm sure having all of those different options available is nice in some context or another, but how unrealistic would it be to pare that back to a few slightly more opinionated and consistent choices? Bzip2 for instance isn't particularly good on *any* metric, are there legacy / ecosystem reasons that are sufficiently important for libsystemd to be dragging it around? libsystemd linking 4 different compression libraries does seem a bit excessive (if it can be helped). [0] https://github.com/systemd/systemd/blob/3799fa803efb04cdd1f1b239c6c64803fe85d13a/src/import/importctl.c#L493 -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: xz backdoor
This might be a good place to start https://gitlab.gnome.org/GNOME/nautilus/-/issues/1936 -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Change Compose Settings (system-wide)
>ry xz -9, it should be better than zstd. It will take longer to compress, but >should actually be FASTER (!) to decompress, which is what really matters. Please provide data - any data - to support this claim, because it flies completely in the face of every benchmark the internet has to offer, including the one Sirius performed below. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Change Compose Settings (system-wide)
One more point: createrepo_c uses zstd compression level 10, but the range goes all the way up to level 22. I would oppose making the default much computationally heavier than it is currently, but if spending 20x longer to compress the repo 10% more is desirable to the fedora project, then createrepo_c could perhaps add a the ability to select a compression level. zstd at high compression levels is very nearly as good at compressing as xz and sometimes better, while remaining much faster to decompress. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Change Compose Settings (system-wide)
Also, to use that squash benchmark you will probably want to run it yourself with modern libraries and modern hardware, as the data on their website (assuming it's the same as the data in their github repo) is 8+ years old. zstd has improved a fair bit during that timeframe. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F41 Change Proposal: Change Compose Settings (system-wide)
But we're not compressing text, we're compressing XML. Anyway, I ran an experiment on a local copy of the Fedora 38 release repo and the differences (while they do exist) aren't very significant. Less than 10% createrepo_c --update --skip-stat --recycle-pkglist --general-compress-type gz . du -bh repodata/* | grep .gz 18M repodata/f6dee453a7f86804214e402ad2e444b989f044f0b16fa7ba74e5a27a8a49cd07-primary.xml.gz 52M repodata/131fa4fcd206fd3a718e4765983c8b7b276e7e634e45c226d9c465145f8e69e9-filelists.xml.gz 7.1M repodata/1c4bf077a2bdf4743a7cded3e2f72282dec5f8e4910692d193e371508552322a-other.xml.gz createrepo_c --update --skip-stat --recycle-pkglist --general-compress-type zstd . = 15M repodata/5c05e888c6da5a13dc2a73fc6fc6e6b2f4ec9120a9544fa26c20cff14a8ace27-primary.xml.zst 41M repodata/289503c7ec867863ee67188b8d9981f7e291158a9821ae813124eb480b41cc94-filelists.xml.zst 5.5M repodata/f78b3010d62173a4a81951c24bad28deb7cb91ab678fdd515a56ff9a72574953-other.xml.zst createrepo_c --update --skip-stat --recycle-pkglist --general-compress-type xz . 14M repodata/a4a4b9c7da02d0cbc7bb3aa39f7f919c7ca033e685ef44e42478a6daf841b32a-primary.xml.xz 41M repodata/244e49e5b8c95280bb67a9695e4177fc9e7358f4482df2b489126c02673a48ad-filelists.xml.xz 4.9M repodata/1f492b3f77a2f9d8a0c1f646200db2575f4a37f5df4c955c8f39e622324eb3ec-other.xml.xz -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Obsoleting zlib in Fedora Rawhide
> Except that it's not 100% compatible, since all those packages aren't > building/working with zlib-ng-compat. At a minimum, you should be able > to show that everything zlib-dependent successfully rebuilds with this, > and since you've already identified some that don't, IMO they should be > fixed *first*. He didn't say the packages weren't working, he said the tests failed, typically because they hardcoded assumptions about the exact behavior of the compression library. That's a different issue. As a concrete example, a while back I wanted to add zlib-ng as a build time option to the createrepo_c package, and one such test failed because the zlib-ng gzip output was not identical to zlib gzip output, despite the fact that it *was* correct and decompresses correctly: https://github.com/rpm-software-management/createrepo_c/pull/339/commits/1f3549bfe6ea45e6fd099c0582893766061198c8 It so happened that within days there was another reporter who needed the same fix but for different reasons - because that test was broken on z390 where gzip compression is hardware-accelerated: https://github.com/rpm-software-management/createrepo_c/pull/336 So any tests that are breaking because they expected some exact gzipped output as opposed, were likely fragile anyway, and worked only because zlib has changed very little for a very long time - no new compression strategies, no adjustment of which strategies are used by which levels, etc. -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
>There are arguments > about whether it's appstream-builder specific (given that Debian's > archive is bigger, their package format is more complex to "pluck" > from, etc and appstream-generator does fine there), but the point is > that for Fedora with appstream-builder, it's too slow. Note that the > context of this is back in the era where Fedora composes took half a > day or longer. Adding even more hours to that process was not > appealing (it would have to be done for every repo being composed for > every architecture...). So, if it's theoretically possible to be much faster than it is, has anyone sat down with a profiler and tried to improve appstream-builder? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
Will this be posted soon? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
Yeah, the Zstd default of 3 is tuned to be roughly on par with alternatives but much faster. To get better compression than gzip you generally need to go to higher levels. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
> The background to this is I've spent far too long trying to optimize > the conversion of qcow2 files to raw files. Most existing qcow2 files > that you can find online are zlib compressed, including the qcow2 > images provided by Fedora. Each cluster in the file is separately > compressed as a zlib stream, and qemu uses zlib library functions to > decompress them. When downloading and decompressing these files, I > measured 40%+ of the total CPU time is doing zlib decompression. > > [You don't need to tell me how great Zstd is, qcow2 supports this for > compression also, but it is not widely used by existing content.] Independent from the decision to use zlib-ng as a distro-wide zlib replacement, which is a good idea regardless. Are there reasons why Fedora's qcow2 images cannot switch to Zstd compression? Zstd support appears to have been added by QEMU 5.1 in August 2020, and both EL8 and EL9 appear to have newer versions QEMU available (therefore, they ought to be able to support those images). Sure, it doesn't help for existing content, but it would help for future content. And I'm pretty sure zstd remains faster than zlib-ng despite the speed improvements in the latter. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
As far as I can tell it's mostly legacy removal and adoption of (now) standardized types, such as "z_size_t" being replaced by "size_t" There is a document here that tries to describe the different considerations: https://github.com/zlib-ng/zlib-ng/blob/develop/PORTING.md "The zlib-ng native has implemented some modernization and simplifications in its API, intended to make life easier for application developers." "In certain places zlib-ng native uses more appropriate data types, removing the need for some workarounds in the API compared to zlib." In another discussion the maintainer said that compat mode still includes all of the performance improvements. There is one compatibility hiccup mentioned on the porting document which also affects compat mode, which is that if you provide your own buffer, zlib-ng needs a larger buffer than zlib regardless of which way it is compiled. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
The zlib-ng 2.1 beta, apparently, has some significant further enhancements coming down the pipe. So the potential is there for users to see improvements much greater than 40% eventually. https://www.phoronix.com/news/Zlib-ng-2.1-Beta "With zlib-ng 2.1 beta there is upwards of 56% faster decompression performance when using an AVX2-capable x86_64 CPU. In general the decompression performance should be a "lot faster" and headlines this new beta release." "Zlib-ng 2.1 has also been working on compression improvements from levels 3 to 9 while the speed-ups are more focused on the decompression side. The zlib-ng 2.1 beta update has also been enhancing its CMake build system, improved support for the Apple M1, enhanced the EmScripten support for compiling to JavaScript, and many other changes." ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
Also to that point, the compatibility issues aren't always compatibility issues, but rather poorly written tests. For example, some tests might hardcode an expected checksum [0] for the compressed output, which could be broken by any number of changes even if the compressed output is entirely valid and decompresses correctly. [0] https://github.com/rpm-software-management/createrepo_c/pull/336 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: zlib-ng as a compat replacement for zlib
>In my test zlib-ng is about 40% faster. I did some testing with zlib-ng and createrepo-c a few months ago [0], and I also found that the compression portion of the workload was about 40% faster. So this matches my experience, too. (BTW, if that github comment [0] sounds negative, it isn't meant to, it's just that zlib ended up not being the primary culprit of the performance issue I was investigating at the time, so I was not as immediately interested in replacing it.) I support this proposal. A tricky detail is that one of the big upsides of zlib-ng is that it has a lot of optimized codepaths for ARM, POWER, Z, RISC-V, AVX-256, AVX-512 and so forth which madler/zlib does not have. And that's fantastic, but I expect it could make the testing process a bit more painful. Since each of the arch-specific acceleration codepaths is behind a separate build flag [1] though, (I assume) we could easily do a more conservative rollout with most arch-specific optimizations disabled at first, and then enabled gradually over time. [0] https://github.com/rpm-software-management/createrepo_c/pull/335#issuecomment-1381362220 [1] https://github.com/zlib-ng/zlib-ng#advanced-build-options ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: It’s time to transform the Fedora devel list into something new
> If one uses OpenPGP and if people verify it As you mention, that's a big "if" ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 DNF/RPM install errors due to header signatures
>and in 1-2 years, SHA256 I've not seen any speculation much less evidence about sha256 being insecure. Is this a post-quantum-crypto thing? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F39 proposal: Changes of defaults in createrepo_c-1.0.0 (System-Wide Change proposal)
> - When adding groups.xml to repodata createrepo_c currently adds two > variants to repomd.xml. The specified file as is, uncompressed, with > the type "group" and also a compressed variant with type "group_XX", > where XX is compression suffix. This is atypical and unexpected. We > propose to include just one variant of groups.xml using specified > compression and repomd.xml type "group". This is not compatible with > yum in RHEL 7. If required users will still be able to create > repositories with the old layout using modifyrepo_c. > Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318 If we're planning to drop by-default compatibility with EL7 anyway and bump to 1.0, we could go a little bit further. * "createrepo_c" doesn't need to support sqlite metadata generation at all, so long as the repository producer can run "sqliterepo_c". If I recall correctly, yum on EL7 doesn't actually need the sqlite metadata to be generated in any case, it can produce that metadata itself so long as it has the XML. Making it available only avoids a local processing step. But if "modifyrepo_c" will be necessary to work with most EL7 repos, then making "sqliterepo_c" necessary for a minor optimization isn't a big leap. * createrepo_c currently has two different switches for compression types, "compress-type" and "general-compress-type", where the latter applies to everything and the former only applies to files that are *not* "primary", "filelists", and "other". there should be one compression type option and it should be applied uniformly to all metadata produced by the "createrepo_c" tool. Yum would have choked on this because of the special handling around "comps", but I believe it should no longer be a concern? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Proposal: drop delta rpms (for real this time)
I am not sure whether by "all historical updates" you are only referring to all updates being listed in updateinfo.xml, or all history generally (including old packages). But in the latter case, note that keeping all updates massively inflates the storage requirements for maintaining a copy of the repo, which many (or even most) corporate users do. This is not a huge problem, generally, but it's also not ideal, and probably isn't the right tradeoff for Fedora. Here[0] for example is RHEL 8 baseos and appstream, for which the difference between storing "only the latest package" and "all the packages listed" is 20x and 10x, respectively. Metadata size would likewise be larger, meaning DNF clients have more to download. [0] [dalley@thinkpad repos]$ rpmrepo details el8-baseos ... Number of packages: 12910 Number of unique packages (latest version): 1798 Number of packages (latest 3 versions): 4459 Packages total size: 23.82 GB Packages total size (latest version):1.4 GB Packages total size (latest 3 versions): 4.03 GB [dalley@thinkpad repos]$ rpmrepo details el8-appstream ... Number of packages: 29103 Number of unique packages (latest version): 5902 Number of packages (latest 3 versions): 12988 Packages total size: 92.91 GB Packages total size (latest version):9.12 GB Packages total size (latest 3 versions): 23.91 GB ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Proposal: drop delta rpms (for real this time)
Are you saying that DNF does an exact version match instead of making the assumption that packages with version >= X contain a fix for a security bug which the updateinfo declares to be fixed in X? Or that the updateinfo itself gets purged of advisories that don't apply to the latest versions available. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Proposal: drop delta rpms (for real this time)
Well, regarding the "based on something", you can hand off a list of packages to createrepo_c with --pkglist, and avoid the need to download files with --update + --skip-stat. Unfortunately that doesn't help you with the package file management. In a vacuum --baseurl would help here because you could have one root directory, however in reality it breaks repository mirroring because any mirror be telling clients to fetch the packages from the source-of-truth. I'm not 100% sure how --basedir works, the description is a bit vague. Another option is to use something like Pulp which stores all the information required for metadata generation inside Postgresql and thus can do so without ever touching the packages / headers again. That approach isn't necessarily free of downsides either, but it does abstract the whole file management problem. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Quick thanks for -fno-omit-frame-pointer
A neat thing you can play around with is the color profiles, you can have kernel stack frames in one color, C / C++ userspace stack frames in another color, Java / Node / Python stack frames in another color, etc. https://www.brendangregg.com/blog/2017-07-30/coloring-flamegraphs-code-type.html It would be a nice visual addition to the demo. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Quick thanks for -fno-omit-frame-pointer
Perhaps this would be amenable to a blog post demonstrating the benefits? e.g. * What does the system performance analysis process look like, how to get started * What impacts does the lack of frame pointer information have on the outputs (incoherent / invisible / inaccurate data), and how much work is required to work around it that (recompiling packages) * Compared to Fedora-with-frame-pointers I think it would be quite useful both as a "starter guide" of sorts and as a call to action to get developers to start taking advantage of it. Maybe even to switch to Fedora. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: -fno-omit-frame-pointer does not work as advertised
> Daniel Alley wrote: > > But it can be in the function CALLING such a function, and said function > will be completely missing from the backtrace. > > Quoting your link > [https://developer.arm.com/documentation/dui0774/k/Compiler-Command-line-O...]: The quotation says the opposite of what you are saying. "leaves the frame pointer pointing to the caller's frame" => the caller set the frame pointer, the leaf did not, therefore the leaf function will not show up but the caller still will. > And that limitation makes -fno-omit-frame-pointer useless for the purpose > for which it is being forced onto Fedora users. > > The other limitation that I have mentioned is also documented there: > > That, too, means frame pointers cannot by design solve the problem which the > change intends to solve with them. > > Kevin Kofler You seem to be arguing that because the solution is not 100% perfect (and it was already acknowledged that it is not), it is useless. Is shipping debug symbols for an optimized binary completely useless just because there are situations where optimizations like inlining and TCO make the executed code radically different from the source code, and therefore make it impossible to trace the execution line by line? I don't work at Facebook and I have no idea how Facebook do things, but I _very strongly doubt_ that they are disabling all those optimizations in their binaries for the sake of perfect unwinding behavior, and yet despite this they claim to have realized a lot of value from this change. If you want to claim that this is "useless", then you are also claiming that any benefit which has already been realized by using it is retroactively invalid. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: -fno-omit-frame-pointer does not work as advertised
> Hi, > > to those who are pushing the -fno-omit-frame-pointer change: Are you aware > that neither that flag nor even -mno-omit-leaf-frame-pointer actually > guarantee that every leaf function is going to carry a frame pointer, as > required for your backtraces? This feels slightly too aggressive relative to the significance of your example. What has happened is that because -O2 optimized away all of the stack access for the function, so it uses no space on the stack, so there is no stack frame separate from the caller's. It is unlikely that the critical bottleneck of any applications will be on such a function. And also, nobody including the proponents of this change are going to argue for disabling ALL of the optimizations which may possibly make the frame pointer not-perfect, including all inlining and tail call optimization. see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97902 https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=8e941ae950ddce17 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98018 https://developer.arm.com/documentation/dui0774/k/Compiler-Command-line-Options/-fomit-frame-pointer---fno-omit-frame-pointer "When there is no stack access, there is no need to use frame pointer even if -fno-omit-frame-pointer is used and caller's frame pointer is unchanged." "A function that does not use any stack space does not need to create a frame record, and leaves the frame pointer pointing to the caller's frame." > Frame pointers sound like a simple solution to unwinding, but they are not. > They are no complete replacement for unwinding information. > > Kevin Kofler I don't think anyone ever argued that frame pointers were a complete replacement for unwinding information. Maybe I missed that part of the discussion? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: [Modularity] XML format for in-repository modules
I will always applaud any attempt at standardizing & documenting the metadata format, and I was never thrilled with glib, so this sounds great to me - I only wish that it had been this way from the beginning :) In practice I am not certain that Satellite (and similar tools) can prefer the XML metadata precisely because it is cut down, so in repos which contain both Yaml and XML metadata it will not be possible to recreate the original Yaml from the metadata in the XML without losing the "packager" specific bits, should they exist. Perhaps that is actually fine, but it makes me uncomfortable, and we have to support the Yaml parsing anyway due to the distros which will only ever support Yaml, which unfortunately makes it is the greatest common denominator. I presume there are no plans to remove the Yaml metadata from repos entirely? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
I'm quite not sure how one would go about empirically measuring something like that - at least in the general case. It might be an interesting research topic. So no, unfortunately I don't really have hard evidence for this. I just know that of all the C libraries I've looked at, in my personal experience it seems to be a very common phenomenon to copy or reimplement code that in Rust you would just import and re-use. It's just a pattern that one notices frequently when it comes to C libraries, especially crossplatform ones that can't rely exclusively on the existence of a Linux-like package manager. If you want specific examples, the ones that pop to mind are: * zchunk and deltarpm both reimplement / "bundle" multiple different hashing algorithms * libcomps implements about 4 different relatively common data structures * GTK appears to contain a bundled, forked copy of the CRoaring library ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
> I think almost all of these qualify as "Core system libraries that > pretty much everything depends on.". > Building their C dependencies from vendored copies (if that is even > supported) and statically linking them seems like a pretty bad idea in > almost all cases here, especially for things where the version of a > program on the "host" and the accompanying shared library should > match. Yes, we're in complete agreement. I'm not suggesting anything like that. Vendoring libraries like openssl is a bad idea. What I'm saying is that it's not very logically justified to say that just because core system libraries like openssl shouldn't be vendored, all vendoring is disallowed regardless of how small and focused they are or how few dependents they have. Because - most C libraries have "dark dependencies" that are effectively the same but worse, in some ways. Given the choice between 100 Rust packages vendoring 10 different copies of the sha256 crate and 100 C packages with 100 separate copies of sha256.c sitting in their source trees (which seems like an entirely realistic comparison), why would the latter be completely A-OK while the latter is completely disallowed? > But ... none of these "tiny" Rust crates are dynamically linked in > Fedora anyway - because Rust doesn't really support that? > So I fail to see your point there, unless you meant to say "C projects > don't 'bundle', they just often 'copy' some code into their > projects"? > > Fabio Yes, that's essentially what I'm saying. I feel like the "no bundling" policy draws distinctions that don't entirely make sense, especially when it comes to the small, focused leaf-node dependencies that people often complain about. Clearly the "left-pad" scenario is bad and should be avoided, but on the other hand is having 800 different linked list implementations, 500 different hash table implementations, 25 different half-baked XML parsers, really so much "better", or is it just what we're used to? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
> Do I really need to explain this point? I think linking against system > OpenSSL is *way better* than statically linking to a random vendored > copy of it. There are maybe about 100-120 libraries for which this is obviously the case. openssl, glibc, glib2, zlib, libxml2, libcurl, kde libraries, etc. Core system libraries that pretty much everything depends on. Dynamically linking such libraries has real benefits. For everything else though? No, not so much. I feel like there is insufficient recognition of the extent to which C libraries do "bundling". Not "bundling" in the sense of vendoring a whole library, but in the sense of including one-off implementations of basic data structures, configuration parsers, hashing algorithms, etc. I would love to hear anyone argue that 100 different variations of "sha256.c" across 100 different packages better follows the spirit of the "no bundling" guidelines than a vendored crate named "sha256" with 100x as many eyes on it, and a higher likelihood to actually be updated if a problem is found. Many of the tiny, "sprawling" Rust dependencies are like this - not all of them of course, but many. Torvalds has similar feelings: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=rcus...@mail.gmail.com/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: metadata of updateinfo.xml
> Is there any guide that explains fields of the updateinfo.xml files > which are generated in repositories? I very much wish there was, but alas, no. Pretty much everything to do with rpm metadata is effectively implementation-defined, and the implementations are all different, especially with respect to distribution build tooling : / > > I'm trying to figure out > https://github.com/fedora-infra/bodhi/issues/2487 and I think we > currently have weird data in updates. For example, looking to the > 'issued' and 'modified' dates of few updates I see that the issued date > is greater than the modified date... that's because Bodhi modifies the > issued date with the timestamp of when the update is pushed to the > repository, rather than using the submission date. RHEL has a similar issue on occasion, even though the build tooling is completely different (not Bodhi). The issued datetimestamp is often 1-2 seconds after the updated date. The specific reason is likely entirely different, though. In practice I have found that having any kind of expectations about updateinfo metadata making coherent sense beyond "it technically meets the expected schema" leads to a lot of pain. Fedora, SUSE, RHEL, Alma and Rocky all mess it up in different ways, 3rd party vendors that attempt to provide it do even worse, etc. > I don't think that field is meant to be modified and I suppose it can > lead yum/dnf to screw up. Is there any expert that can point me in the > right direction here? > > Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F37 proposal: Add -fno-omit-frame-pointer to default compilation flags (System-Wide Change proposal)
> On 11/1/22 07:38, Dominique Martinet wrote: > > Has adding kernel support for DWARF unwinding been considered instead? The references of the proposal has some discussion on this subject. Basically, this was already considered and rejected by the kernel developers including Torvalds directly on the basis that DWARF unwinding support is simply too complex and nonperformant to be allowed in the kernel. https://www.kernel.org/doc/html/v5.3/x86/orc-unwinder.html#orc-vs-dwarf https://lwn.net/Articles/728339 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Heads-up / for discussion: dnf not working with 1G of RAM or less
> The reason this bug isn't hitting RHEL right now is simply just because the > default RHEL repositories are much smaller - also crucially things like many -devel packages are in a separate repository. RHEL actually is hitting this issue in different contexts for an entirely different reason (a bug that caused changelog metadata to inflate over time). https://bugzilla.redhat.com/show_bug.cgi?id=2040170 It is in the progress of being fixed. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F37 proposal: Add -fno-omit-frame-pointer to default compilation flags (System-Wide Change proposal)
> Unfortunately, no, there's no in-kernel DWARF unwinder due to the complexity > involved. > Instead, the kernel uses ORC and has an unwinder for that. Adding ORC support > to all of > Linux userspace so that we can unwind it in the kernel isn't likely to > happen, since > all tooling would have to be changed to support ORC. A huge task, most definitely. Setting that aside though, which cases (if any) would that solution *not* handle? Is the level of effort required the primary blocker to such a solution, or are there technical ones as well? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: deltarpm usefulness?
That's a fair point, I was actually not aware that https://github.com/rpm-software-management/drpm contained a completely separate implementation of applydeltarpm. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: deltarpm usefulness?
> On Wed, Aug 11, 2021 at 10:03:50PM +0200, Marek Marczykowski-Górecki wrote: > I do think we should drop drpms or make them more useful, but I don't > think there's any security angle here. (see below) > > drpms work by downloading the delta, then using it + the version you > have installed to recreate the signed rpm (just like you downloaded the > full signed update) and then the gpg signature is checked of that full rpm, > just like one you downloaded. If the drpm is tampered with it won't > reassemble and it will fall back to the full signed rpm. Sorry to resurrect this thread. Another issue - which is not per-se a security issue but it's still a problem - is that deltarpm uses md5 checksums pervasively. They're everywhere. And it uses its own implementation of md5 which doesn't respect FIPS, so even when the user has *explicitly* configured their system to not use md5 for anything security-relevant, libdeltarpm won't know or care. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
