Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Hi Simo, On Fri, 14 Oct 2022 18:28:01 +0200, Simo Sorce wrote: > At this time, as far as I know, there is no OpenPGP work of any kind on > supporting PQC algorithms. Furthermore the way we use signatures in RPM > really has no resemblance to the scenarios OpenPGP was built for. > > So we should consider whether moving to PQC will also mean moving off > OpenPGP as our signature format and into something simpler. A draft for PQC in OpenPGP was published today: https://mailarchive.ietf.org/arch/msg/openpgp/zjbIqesTdiYfA7vkyMZ6EM9KWrA/ https://datatracker.ietf.org/doc/draft-wussler-openpgp-pqc/ Neal ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, 2022-12-01 at 00:41 +, Daniel Alley wrote: > > * zchunk and deltarpm both reimplement / "bundle" multiple different > hashing algorithms zchunk does have bundled versions of various hashing algorithms, but, if it's compiled against OpenSSL (as it is in Fedora), it uses the OpenSSL hashing algorithms rather than the bundled versions. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, 2022-12-01 at 00:41 +, Daniel Alley wrote: > I'm quite not sure how one would go about empirically measuring > something like that - at least in the general case. It might be an > interesting research topic. So no, unfortunately I don't really have > hard evidence for this. We did run discovery for this in RHEL when we started the SHA-1 deprecation process. Some embedded implementations were found but the vast majority of programs used one of the available libraries for SHA-1. I do not expect SHA-256 to be any different. so empirically I can tell you there isn't anywhere as much "vendoring" in C as you claim. Using dynamically linked libraries is well establish and the reason why sometimes the dependency chain is ... monstrous. > I just know that of all the C libraries I've looked at, in my > personal experience it seems to be a very common phenomenon to copy > or reimplement code that in Rust you would just import and re-use. Perhaps this is true in the niche you are interested in? > It's just a pattern that one notices frequently when it comes to C > libraries, especially crossplatform ones that can't rely exclusively > on the existence of a Linux-like package manager. Yes, those kind of libraries tend to be quite bad in this regard, OTOH it can be done right. For example the NSS library generally carries copies of external dependencies, but the configure script looks for a system version and links to the one if available on build. So if you looked at NSS you may think it vendors, and it does, but smartly and in a way that is compatible with systems integration. > If you want specific examples, the ones that pop to mind are: > > * zchunk and deltarpm both reimplement / "bundle" multiple different > hashing algorithms > * libcomps implements about 4 different relatively common data > structures I am not sure this qualify as vendoring/bundling, kind of borderline but I see why you added it as a case. > * GTK appears to contain a bundled, forked copy of the CRoaring > library > > > > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue -- Simo Sorce RHEL Crypto Team Red Hat, Inc ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
I'm quite not sure how one would go about empirically measuring something like that - at least in the general case. It might be an interesting research topic. So no, unfortunately I don't really have hard evidence for this. I just know that of all the C libraries I've looked at, in my personal experience it seems to be a very common phenomenon to copy or reimplement code that in Rust you would just import and re-use. It's just a pattern that one notices frequently when it comes to C libraries, especially crossplatform ones that can't rely exclusively on the existence of a Linux-like package manager. If you want specific examples, the ones that pop to mind are: * zchunk and deltarpm both reimplement / "bundle" multiple different hashing algorithms * libcomps implements about 4 different relatively common data structures * GTK appears to contain a bundled, forked copy of the CRoaring library ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Once upon a time, Daniel Alley said: > 100 C packages with 100 separate copies of sha256.c sitting in their source > trees (which seems like an entirely realistic comparison) You keep saying this - do you have any evidence that this is the case? -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
> I think almost all of these qualify as "Core system libraries that > pretty much everything depends on.". > Building their C dependencies from vendored copies (if that is even > supported) and statically linking them seems like a pretty bad idea in > almost all cases here, especially for things where the version of a > program on the "host" and the accompanying shared library should > match. Yes, we're in complete agreement. I'm not suggesting anything like that. Vendoring libraries like openssl is a bad idea. What I'm saying is that it's not very logically justified to say that just because core system libraries like openssl shouldn't be vendored, all vendoring is disallowed regardless of how small and focused they are or how few dependents they have. Because - most C libraries have "dark dependencies" that are effectively the same but worse, in some ways. Given the choice between 100 Rust packages vendoring 10 different copies of the sha256 crate and 100 C packages with 100 separate copies of sha256.c sitting in their source trees (which seems like an entirely realistic comparison), why would the latter be completely A-OK while the latter is completely disallowed? > But ... none of these "tiny" Rust crates are dynamically linked in > Fedora anyway - because Rust doesn't really support that? > So I fail to see your point there, unless you meant to say "C projects > don't 'bundle', they just often 'copy' some code into their > projects"? > > Fabio Yes, that's essentially what I'm saying. I feel like the "no bundling" policy draws distinctions that don't entirely make sense, especially when it comes to the small, focused leaf-node dependencies that people often complain about. Clearly the "left-pad" scenario is bad and should be avoided, but on the other hand is having 800 different linked list implementations, 500 different hash table implementations, 25 different half-baked XML parsers, really so much "better", or is it just what we're used to? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, 2022-11-30 at 18:26 +, Simon Farnsworth via devel wrote: > On Wednesday, 30 November 2022 17:47:16 GMT Fabio Valentini wrote: > > On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley wrote: > > > I feel like there is insufficient recognition of the extent to which C > > > libraries do "bundling". Not "bundling" in the sense of vendoring a > > > whole library, but in the sense of including one-off implementations of > > > basic data structures, configuration parsers, hashing algorithms, etc. I > > > would love to hear anyone argue that 100 different variations of > > > "sha256.c" across 100 different packages better follows the spirit of the > > > "no bundling" guidelines than a vendored crate named "sha256" with 100x > > > as many eyes on it, and a higher likelihood to actually be updated if a > > > problem is found. > > > > > > > > > > > Many of the tiny, "sprawling" Rust dependencies are like this - not all of > > > them of course, but many. > > > > But ... none of these "tiny" Rust crates are dynamically linked in > > Fedora anyway - because Rust doesn't really support that? > > So I fail to see your point there, unless you meant to say "C projects > > don't 'bundle', they just often 'copy' some code into their projects"? > > > I think the point he's making is that developers don't write common > functionality from scratch, in general. We reuse code from elsewhere. > > It's just that in C, I'll copy-and-paste code from the web into my library or > application, not necessarily even bothering with a full "vendoring", whereas > in Rust, I'll use the crc crate (say), or the base64 crate, or other simple > utility crate. > > The result is that I have N implementations of common functionality, each > with > its own unique quirks and security risks, in my C binaries; but my C binaries > have only a small number of dependencies. > > In Rust, however, I'm directly reusing the small utility crate, and while I > may use `cargo vendor` to import the crate's source into my tree, I'm > unlikely > to edit it. The result is that a Rust binary has a larger number of > dependencies than the equivalent C program, because I'm depending on a crate > instead of copy-and-pasting the code and "hiding" it from the packager. > > This is a challenge for Fedora: how do we cope with a world where instead of > having a few tens of dependencies, and a lot of copy-and-paste code, we have > hundreds of dependencies, but no copy-and-paste code? > > One answer is to say that Rust is bad for encouraging developers to depend on > small crates instead of copying-and-pasting "small" utilities around. > Another > (which you're doing a great job of) is to package up all the dependencies, > so > that we represent the true dependency tree in RPM. Yet another would be to > manually decide which dependencies get bundled, and which don't - doing the > same thing as the C world does to keep its dependency count down. This is a bit of a misleading characterization. - the amount of copying in C programs is overblown in my opinion (no data provided to back this up though) - dependency minimization for C programs/libraries is assumed but no data provided to back it up. - the dilemma is how to manage rust programs not to decide what to bundle or not, that's basically decided upstream Although there are certainly instances of copying/paste in C code, the vast majority of reuse comes through linking to utility libraries dynamically, which makes distro-level maintenance much easier because you have only once place to fix/rebuild/distribute when there are serious issues. The problems with Rust crates (or go modules, or any other "vendoring"), is that not only you have to go and find each place where a problematic crate was vendored; you also have to figure out, often under pressure, if the crate can simply be summarily updated or if you need a backport because the vendoring application can't cope with the semantic changes that happened in the problematic crate's new version. Multiply this by N packages using M different versions of the problematic crate. Although vendored crates can be tracked (this i much better than copy/pasting), with additional tooling, the distribution remains on the hook for solving the same problem in N packages, without easy coordination. Some upstream may be quick and do the work for you, some may not care, disappear etc... or are simply too slow for the urgency the distribution has leading potentially to diverging solutions ... Simo. -- Simo Sorce RHEL Crypto Team Red Hat, Inc ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wednesday, 30 November 2022 17:47:16 GMT Fabio Valentini wrote: > On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley wrote: > > I feel like there is insufficient recognition of the extent to which C > > libraries do "bundling". Not "bundling" in the sense of vendoring a > > whole library, but in the sense of including one-off implementations of > > basic data structures, configuration parsers, hashing algorithms, etc. I > > would love to hear anyone argue that 100 different variations of > > "sha256.c" across 100 different packages better follows the spirit of the > > "no bundling" guidelines than a vendored crate named "sha256" with 100x > > as many eyes on it, and a higher likelihood to actually be updated if a > > problem is found. > > > > > > > Many of the tiny, "sprawling" Rust dependencies are like this - not all of > > them of course, but many. > > But ... none of these "tiny" Rust crates are dynamically linked in > Fedora anyway - because Rust doesn't really support that? > So I fail to see your point there, unless you meant to say "C projects > don't 'bundle', they just often 'copy' some code into their projects"? > I think the point he's making is that developers don't write common functionality from scratch, in general. We reuse code from elsewhere. It's just that in C, I'll copy-and-paste code from the web into my library or application, not necessarily even bothering with a full "vendoring", whereas in Rust, I'll use the crc crate (say), or the base64 crate, or other simple utility crate. The result is that I have N implementations of common functionality, each with its own unique quirks and security risks, in my C binaries; but my C binaries have only a small number of dependencies. In Rust, however, I'm directly reusing the small utility crate, and while I may use `cargo vendor` to import the crate's source into my tree, I'm unlikely to edit it. The result is that a Rust binary has a larger number of dependencies than the equivalent C program, because I'm depending on a crate instead of copy-and-pasting the code and "hiding" it from the packager. This is a challenge for Fedora: how do we cope with a world where instead of having a few tens of dependencies, and a lot of copy-and-paste code, we have hundreds of dependencies, but no copy-and-paste code? One answer is to say that Rust is bad for encouraging developers to depend on small crates instead of copying-and-pasting "small" utilities around. Another (which you're doing a great job of) is to package up all the dependencies, so that we represent the true dependency tree in RPM. Yet another would be to manually decide which dependencies get bundled, and which don't - doing the same thing as the C world does to keep its dependency count down. -- Simon Farnsworth ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley wrote: > > > Do I really need to explain this point? I think linking against system > > OpenSSL is *way better* than statically linking to a random vendored > > copy of it. > > There are maybe about 100-120 libraries for which this is obviously the case. > openssl, glibc, glib2, zlib, libxml2, libcurl, kde libraries, etc. Core > system libraries that pretty much everything depends on. Dynamically linking > such libraries has real benefits. > > For everything else though? No, not so much. These are actually all good examples of stuff that we dynamically link to. This is the (not 100% exhaustive) list of system libraries we always dynamically link to: - all GTK / GNOME libraries (dbus, freetype, fontconfig, atk, gdk-pixbuf, gdk, gtk3, gtk4, gio, glib2, graphene, gsk4, pango, cairo, gstreamer, etc.) - multimedia codecs / libraries (aom, dav1d, pipewire, pulseaudio, etc.) - crypto libraries (openssl, libsodium, nettle, curl) - compression libraries (bzip2, flate2, libz, lzma, zstd) - database connectors (libsqlite, libpq) - low-level device / storage libraries (devicemapper, libblkid) I think almost all of these qualify as "Core system libraries that pretty much everything depends on.". Building their C dependencies from vendored copies (if that is even supported) and statically linking them seems like a pretty bad idea in almost all cases here, especially for things where the version of a program on the "host" and the accompanying shared library should match. The only exception (that I know of) for "crate dependency built from vendored sources and statically linked" in Fedora right now is libgit2, because the version in Fedora is chronically outdated, and dealing with that has become very painful - and started to block some necessary updates to support more recent versions of Rust / cargo. > I feel like there is insufficient recognition of the extent to which C > libraries do "bundling". Not "bundling" in the sense of vendoring a whole > library, but in the sense of including one-off implementations of basic data > structures, configuration parsers, hashing algorithms, etc. I would love to > hear anyone argue that 100 different variations of "sha256.c" across 100 > different packages better follows the spirit of the "no bundling" guidelines > than a vendored crate named "sha256" with 100x as many eyes on it, and a > higher likelihood to actually be updated if a problem is found. > > Many of the tiny, "sprawling" Rust dependencies are like this - not all of > them of course, but many. But ... none of these "tiny" Rust crates are dynamically linked in Fedora anyway - because Rust doesn't really support that? So I fail to see your point there, unless you meant to say "C projects don't 'bundle', they just often 'copy' some code into their projects"? Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
> Do I really need to explain this point? I think linking against system > OpenSSL is *way better* than statically linking to a random vendored > copy of it. There are maybe about 100-120 libraries for which this is obviously the case. openssl, glibc, glib2, zlib, libxml2, libcurl, kde libraries, etc. Core system libraries that pretty much everything depends on. Dynamically linking such libraries has real benefits. For everything else though? No, not so much. I feel like there is insufficient recognition of the extent to which C libraries do "bundling". Not "bundling" in the sense of vendoring a whole library, but in the sense of including one-off implementations of basic data structures, configuration parsers, hashing algorithms, etc. I would love to hear anyone argue that 100 different variations of "sha256.c" across 100 different packages better follows the spirit of the "no bundling" guidelines than a vendored crate named "sha256" with 100x as many eyes on it, and a higher likelihood to actually be updated if a problem is found. Many of the tiny, "sprawling" Rust dependencies are like this - not all of them of course, but many. Torvalds has similar feelings: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=rcus...@mail.gmail.com/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, 30 Nov 2022 at 07:54, Daniel P. Berrangé wrote: > On Wed, Nov 30, 2022 at 11:54:10AM +, Peter Robinson wrote: > > Hi Fabio, > > > > Been meaning to reply to this, but it got lost in the mail pile. > > > > > > > But running `cargo fetch` with a clean cache pulls down *390* > crates. Of > > > > these, it looks like 199 (!) are already packaged as > rust-[crate]-devel, > > > > which is *amazing*. But... that still is hundreds that I'd have to > add. And > > > > mostly they are things I don't know _anything_ about. > > > > > > You must realize that this is an extreme case. For many Rust > > > applications that people want to package for Fedora, the number of > > > dependencies that are missing is rather small, *because* most popular > > > libraries are already packaged. > > > > In my experience, and I'm not packaging any form of gaming engine, > > it's not an extreme case, for a few small things and one slightly > > larger thing I *have* packaged I now maintain over 60 more packages. I > > have another one I want to package and AFAICT I get to package another > > 50-100 but frankly it's hard to tell, and this is something I have > > control over and have been actively trying to do upstream reduction of > > deps. > > The magnitude of the number of deps is the real killer for not > only Rust, but also Go, and arguably anything that isn't a C > based application. > > Packaging up all of a project's deps individually is viable when > there are a relatively small number of them, especially if most > of the deps are shared by other apps, and the libraries have good > API + ABI stability promises. This is common in the case of most > C applications/libraries, since shared libraries are relatively > speaking fairly broad in functional scope, and often long lived, > because that is the mindset of the C ecosystem in general. > > Modern languages & their ecosystems though have brought about > a world where the granularity of deps is way smaller, where > the focus is on being able to evolve rapidly, with less focus > on long term stable API/ABI, as apps are expected to just fix > against specific versions they've tested on. > > I wonder if we should look at the standard libraries in the late 1960/1970 languages as being the same as 'opinionated' Operating Systems. Most of them started out with a period where every mainframe might have a 'language' and a set of 'libraries' of routines but every site pretty much mangled them to fit their own 'local' needs. Software vendors which existed usually ended up having consultants go out to 'fix' their code to match whatever routines and tooling existed on each mainframe. This got to be unworkable and various 'libraries' were put forward which were to standardize things. However, going from the 'old war tales' my mentors and professors from that era, the same arguments that the current languages (Rust, etc) have against standards were existent. The issue was the scale of the problem was much more on the side of the hardware/OS vendors putting out a standard chosen library (and then fighting over getting those to match up for the software vendors who still needed to spend a lot of consulting time). Most of those arguments seem to have 'died' out as people got tired of fighting the differences versus the delight of new challenges that many programmers have when dealing with a 'new' language. [Same with C++ back in the late 1980's where every vendor had a slightly different set of libraries which for its proponents was the best thing over all the others.. and then by the late 1990's was 'Oh no, not this again'. Java went through this up until the late 00's. ] I don't think we are at the state where enough of the new people who are getting into Rust have gotten tired of fighting 'oh, I have to f'ing change my code again to make it work with these 4 things?' Basically when enough people HAVE to code in the language versus just 'WANT' to, but hate the grind.. ability to standardize is going to happen. > So functionality that lives in single library in C, often ends > up being spread across 20-200 modules in any of the modern > non-C languages. We've been experiancing this problem in Fedora > for a very long time and while we have spec auto-generators, > that still leaves masses of busy work, and also still leaves > the problem that what we ship doesn't match what upstream has > tested, so the result is often less reliable. > > IMHO the only thing that can make packaging such fine grained > modules sustainable long term, is if the process could be 100% > automated. I don't mean just having a tool to auto-generate > a spec file. It needs to be fully automated to the extent that > you just tell a robot "i need deps for this package" and it > does everything automatically, except for human approval of > the results of a code license audit. > > That of course would be a non-trivial service to build, and > it still begs the question of whether its really beneficial > in
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 8:16 AM Fabio Valentini wrote: > > On Wed, Nov 30, 2022 at 12:54 PM Peter Robinson wrote: > > > > > This is true, and probably also not "fixable". We need to make some > > > amount of non-upstreamable patches to some crates (most notably, > > > removing Windows- or mac OS-specific dependencies, because we don't > > > want to package those), but in some cases, these are "incompatible" > > > changes, and Rust *developers* should not be targeting our downstream > > > sources that have these differences with actual upstream sources. > > > > Yet you say above "We *do* provide value to both users *and* > > developers" yet you say developers shouldn't be targeting that work? > > We provide value to developers by basically running a huge free CI > across a wide range of architectures. > > That doesn't mean that they should develop against crate sources we > have in Fedora, similarly to how upstream Python developers probably > should use pip/venv instead of "whatver is in Fedora". The same > applies to basically every language ecosystem except C/C++, where the > package manager story is just so bad that system libraries are > actually the only reliable development environment. > > > > This is due to a limitation of how cargo handles target-specific > > > dependencies - all dependencies that are *mentioned in any way* need > > > to be *present* for it to resolve dependencies / enabled optional > > > features / update its lockfile etc. But since we don't want to package > > > bindings for Windows and mac OS system APIs, we need to actually patch > > > them out, otherwise builds will fail. > > > > And that ends up being quite a bit of work from my point of view. Also > > the way the packaging works with options things like devel or optional > > features ends up being very painful. I will often drop out optional > > features just so I can do less packaging. > > Recent versions of rust2rpm automatically generate a patch to remove > non-linux dependencies, so that work has effectively been reduced to > zero. > Disabling unused optional dependencies can also be done with either a > rust2rpm config option or with a simple patch, so that should not be a > problem, either. I also don't understand why disabling some optional > features is such a bad thing? I don't think we have a policy in Fedora > that says we *must* enable *all possible functionality and optional > features*. > > > > > We're doing okay with #1, but... I think #3 _even_ with all of the work > > > > in > > > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > > > > engine and will probably have a few things it would be nice to package > > > > to > > > > make available in Fedora Linux. I might not even mind maintaining Bevy > > > > itself. > > > > > > Somebody actually already started packaging Bevy components - some > > > packages are already approved and some are still pending review. Not > > > sure what the progress has been there, but it's not *impossible*. > > > > Well nothing is *impossible* if you have enough stamina, resources or > > whatever else. I don't find saying something isn't *impossible* > > necessarily makes it compelling. > > I agree. And I think we can do better, as I said in my original post. > > > > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > > > which is *amazing*. But... that still is hundreds that I'd have to add. > > > > And > > > > mostly they are things I don't know _anything_ about. > > > > > > You must realize that this is an extreme case. For many Rust > > > applications that people want to package for Fedora, the number of > > > dependencies that are missing is rather small, *because* most popular > > > libraries are already packaged. > > > > In my experience, and I'm not packaging any form of gaming engine, > > it's not an extreme case, for a few small things and one slightly > > larger thing I *have* packaged I now maintain over 60 more packages. I > > have another one I want to package and AFAICT I get to package another > > 50-100 but frankly it's hard to tell, and this is something I have > > control over and have been actively trying to do upstream reduction of > > deps. > > As far as I know, salimma's intern worked on a tool to determine > missing dependencies for Rust projects in Fedora recursively, which > should make it easier to determine the exact set of things that you > would need to do here. > > > And requests to make things like this easier have been open for over a year: > > https://pagure.io/fedora-rust/rust2rpm/issue/140 > > Right. I'm sorry about the issue reports that have been open for a long time. > > rust2rpm / rust-packaging is an entirely community-run project. We > don't get any support from Red Hat (or other places) for anything > except the Rust compiler itself. And with the original developer of > rust2rpm (Igor) being gone, nobody
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 12:54 PM Peter Robinson wrote: > > > This is true, and probably also not "fixable". We need to make some > > amount of non-upstreamable patches to some crates (most notably, > > removing Windows- or mac OS-specific dependencies, because we don't > > want to package those), but in some cases, these are "incompatible" > > changes, and Rust *developers* should not be targeting our downstream > > sources that have these differences with actual upstream sources. > > Yet you say above "We *do* provide value to both users *and* > developers" yet you say developers shouldn't be targeting that work? We provide value to developers by basically running a huge free CI across a wide range of architectures. That doesn't mean that they should develop against crate sources we have in Fedora, similarly to how upstream Python developers probably should use pip/venv instead of "whatver is in Fedora". The same applies to basically every language ecosystem except C/C++, where the package manager story is just so bad that system libraries are actually the only reliable development environment. > > This is due to a limitation of how cargo handles target-specific > > dependencies - all dependencies that are *mentioned in any way* need > > to be *present* for it to resolve dependencies / enabled optional > > features / update its lockfile etc. But since we don't want to package > > bindings for Windows and mac OS system APIs, we need to actually patch > > them out, otherwise builds will fail. > > And that ends up being quite a bit of work from my point of view. Also > the way the packaging works with options things like devel or optional > features ends up being very painful. I will often drop out optional > features just so I can do less packaging. Recent versions of rust2rpm automatically generate a patch to remove non-linux dependencies, so that work has effectively been reduced to zero. Disabling unused optional dependencies can also be done with either a rust2rpm config option or with a simple patch, so that should not be a problem, either. I also don't understand why disabling some optional features is such a bad thing? I don't think we have a policy in Fedora that says we *must* enable *all possible functionality and optional features*. > > > We're doing okay with #1, but... I think #3 _even_ with all of the work in > > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > > > engine and will probably have a few things it would be nice to package to > > > make available in Fedora Linux. I might not even mind maintaining Bevy > > > itself. > > > > Somebody actually already started packaging Bevy components - some > > packages are already approved and some are still pending review. Not > > sure what the progress has been there, but it's not *impossible*. > > Well nothing is *impossible* if you have enough stamina, resources or > whatever else. I don't find saying something isn't *impossible* > necessarily makes it compelling. I agree. And I think we can do better, as I said in my original post. > > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > > which is *amazing*. But... that still is hundreds that I'd have to add. > > > And > > > mostly they are things I don't know _anything_ about. > > > > You must realize that this is an extreme case. For many Rust > > applications that people want to package for Fedora, the number of > > dependencies that are missing is rather small, *because* most popular > > libraries are already packaged. > > In my experience, and I'm not packaging any form of gaming engine, > it's not an extreme case, for a few small things and one slightly > larger thing I *have* packaged I now maintain over 60 more packages. I > have another one I want to package and AFAICT I get to package another > 50-100 but frankly it's hard to tell, and this is something I have > control over and have been actively trying to do upstream reduction of > deps. As far as I know, salimma's intern worked on a tool to determine missing dependencies for Rust projects in Fedora recursively, which should make it easier to determine the exact set of things that you would need to do here. > And requests to make things like this easier have been open for over a year: > https://pagure.io/fedora-rust/rust2rpm/issue/140 Right. I'm sorry about the issue reports that have been open for a long time. rust2rpm / rust-packaging is an entirely community-run project. We don't get any support from Red Hat (or other places) for anything except the Rust compiler itself. And with the original developer of rust2rpm (Igor) being gone, nobody who understands the low-level stuff in there is still around. zbyszek and I are trying our best to keep things running, but at this point, we'll need basically a rewrite of both rust2rpm and the RPM dependency / provides generators to support new
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 11:54:10AM +, Peter Robinson wrote: > Hi Fabio, > > Been meaning to reply to this, but it got lost in the mail pile. > > > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > > which is *amazing*. But... that still is hundreds that I'd have to add. > > > And > > > mostly they are things I don't know _anything_ about. > > > > You must realize that this is an extreme case. For many Rust > > applications that people want to package for Fedora, the number of > > dependencies that are missing is rather small, *because* most popular > > libraries are already packaged. > > In my experience, and I'm not packaging any form of gaming engine, > it's not an extreme case, for a few small things and one slightly > larger thing I *have* packaged I now maintain over 60 more packages. I > have another one I want to package and AFAICT I get to package another > 50-100 but frankly it's hard to tell, and this is something I have > control over and have been actively trying to do upstream reduction of > deps. The magnitude of the number of deps is the real killer for not only Rust, but also Go, and arguably anything that isn't a C based application. Packaging up all of a project's deps individually is viable when there are a relatively small number of them, especially if most of the deps are shared by other apps, and the libraries have good API + ABI stability promises. This is common in the case of most C applications/libraries, since shared libraries are relatively speaking fairly broad in functional scope, and often long lived, because that is the mindset of the C ecosystem in general. Modern languages & their ecosystems though have brought about a world where the granularity of deps is way smaller, where the focus is on being able to evolve rapidly, with less focus on long term stable API/ABI, as apps are expected to just fix against specific versions they've tested on. So functionality that lives in single library in C, often ends up being spread across 20-200 modules in any of the modern non-C languages. We've been experiancing this problem in Fedora for a very long time and while we have spec auto-generators, that still leaves masses of busy work, and also still leaves the problem that what we ship doesn't match what upstream has tested, so the result is often less reliable. IMHO the only thing that can make packaging such fine grained modules sustainable long term, is if the process could be 100% automated. I don't mean just having a tool to auto-generate a spec file. It needs to be fully automated to the extent that you just tell a robot "i need deps for this package" and it does everything automatically, except for human approval of the results of a code license audit. That of course would be a non-trivial service to build, and it still begs the question of whether its really beneficial in the long term. > > Sure, but isn't that the case for most projects that a newcomer wants > > to package, regardless of programming language? Say, somebody wants to > > package some cool new Python project for machine learning, then > > there's probably also some linear algebra package or SIMD math library > > in the dependency tree that's missing from Fedora. How is that > > different? > > I think the big difference here from my experience, and I've packaged > right across the Fedora spectrum, is that most of the python style > projects are able to use a pretty comprehensive standard library and > crypto library which helps minimise a lot of the extras I've seen in > the rust ecosystem. Python isn't that different from the Rust world to be honest. For a short while we had OpenStack in Fedora, but it wasn't sustainable to maintain its large set of python module deps, fighting between the versions Fedora already had, vs the versions that OpenStack actually wanted (and was tested against by upstream. It was never ending busy-work for the people trying to keep it working in Fedora, taking time away from more beneficial work, and delivering a system that was often broken because module version combinations we used were not tested. > > - we port projects to new versions of dependencies > > That's valuable for other projects but not Fedora, and it's not > something I am going to do personally as a rust packager. It isn't sustainable in the general case, as it requires a level of expertize / knowledge that most packagers can't be assumed to have. If they have that knowledge and the time to invest in this, they can do it directly in upstream, regardless of what Fedora packaging approach is used. > I don't think it's as bad as other ecosystems but I don't agree with > your assessment of "kind of a *good* situation", I feel there may be a > little bit of Stockholm syndrome coming into play here to be honest. I feel like Fedora is successfull in packaging huge numbers of deps
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Hi Fabio, Been meaning to reply to this, but it got lost in the mail pile. > > I _very much_ appreciate all the work you and the other Rust SIG folks > > (Igor and Zbyszek in particular but I'm sure others as well!) have put into > > packaging rust apps and crates and all of the systems around that. > > I'll respond inline. Same. > > I fundamentally disagree with Kevin on a deep level about "entirely > > useless", but ... find myself kind of agreeing about the "unpackagable" > > part. I mean: clearly we've found a way, but I'm really not sure we're > > providing a lot of _value_ in this approach, and I'm also not sure it's as > > successful as it could be. > > We *do* provide value to both users *and* developers by doing things > the way we do, but the benefits might not be obvious to people who > don't know how (Rust) packaging works, and what we as package > maintainers do. As a rust packager I initially agreed but I am now having doubts here. > > There are three ways having things packaged in Fedora repos _can_ be > > helpful: > > > > 1. End-user applications and tools > > 2. Useful development environment > > 3. As convenience for ourselves for building packages for #1 or #2 > > > > I am not discounting the value of #3 -- making a shared thing that we all > > work on together is kind of the whole point, and the nicer we can make that > > the better we can bring in more people, and those of us already here have a > > lighter load and can work on the things we're most interested in. But > > ultimately, we're doing it so we make a useful system for users. That means > > the first two. > > This I can agree with :) > > > I'll start with the second: our system for Rust doesn't really do that. > > Developers are going to use cargo and crates.io and we're not going to > > convince them that they should do otherwise. (I don't expect anyone > > disagrees with this.) > > This is true, and probably also not "fixable". We need to make some > amount of non-upstreamable patches to some crates (most notably, > removing Windows- or mac OS-specific dependencies, because we don't > want to package those), but in some cases, these are "incompatible" > changes, and Rust *developers* should not be targeting our downstream > sources that have these differences with actual upstream sources. Yet you say above "We *do* provide value to both users *and* developers" yet you say developers shouldn't be targeting that work? > This is due to a limitation of how cargo handles target-specific > dependencies - all dependencies that are *mentioned in any way* need > to be *present* for it to resolve dependencies / enabled optional > features / update its lockfile etc. But since we don't want to package > bindings for Windows and mac OS system APIs, we need to actually patch > them out, otherwise builds will fail. And that ends up being quite a bit of work from my point of view. Also the way the packaging works with options things like devel or optional features ends up being very painful. I will often drop out optional features just so I can do less packaging. > > We're doing okay with #1, but... I think #3 _even_ with all of the work in > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > > engine and will probably have a few things it would be nice to package to > > make available in Fedora Linux. I might not even mind maintaining Bevy > > itself. > > Somebody actually already started packaging Bevy components - some > packages are already approved and some are still pending review. Not > sure what the progress has been there, but it's not *impossible*. Well nothing is *impossible* if you have enough stamina, resources or whatever else. I don't find saying something isn't *impossible* necessarily makes it compelling. > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > which is *amazing*. But... that still is hundreds that I'd have to add. And > > mostly they are things I don't know _anything_ about. > > You must realize that this is an extreme case. For many Rust > applications that people want to package for Fedora, the number of > dependencies that are missing is rather small, *because* most popular > libraries are already packaged. In my experience, and I'm not packaging any form of gaming engine, it's not an extreme case, for a few small things and one slightly larger thing I *have* packaged I now maintain over 60 more packages. I have another one I want to package and AFAICT I get to package another 50-100 but frankly it's hard to tell, and this is something I have control over and have been actively trying to do upstream reduction of deps. And requests to make things like this easier have been open for over a year: https://pagure.io/fedora-rust/rust2rpm/issue/140 > Bevy is a bit special, because it (presumably) pulls in lots of GPU / > OpenGL / Vulkan related libraries, which we didn't
Re: [Rust] Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Tue, Nov 01, 2022 at 01:30:01PM -0500, Michel Alexandre Salim wrote: > I've finally gotten round to doing some polishing and getting it > packaged: > - updates for Fedora 36, 37, and Rawhide: > https://bodhi.fedoraproject.org/updates/?search=rust-update-set-0.0.1=python-rust-update-set > - Pagure repo: https://pagure.io/fedora-rust/rust-update-set > > There are some fixes for corner cases I encountered while trying to update our > `below` packages with this, and some changes needed to get this packageable, > but those are minor. It mostly works really well, and is at a stage where > hopefully people can take a look and make suggestions for improvements. Cool -- glad to see this! -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Hi Simo, On Fri, 14 Oct 2022 22:36:09 +0200, Neal H. Walfield wrote: > On Fri, 14 Oct 2022 18:28:01 +0200, > Simo Sorce wrote: > > At this time, as far as I know, there is no OpenPGP work of any kind on > > supporting PQC algorithms. > > The German BSI contracted MTG AG to design and implement PQC for > OpenPGP. They presented their work at IETF 113, and at the OpenPGP > email summit this past May. > > https://datatracker.ietf.org/meeting/113/materials/agenda-113-openpgp-01 The OpenPGP session at IETF 115 (Tuesday Nov 8th, 1300-1430 UTC) will include a discussion on PQC in OpenPGP: https://datatracker.ietf.org/doc/agenda-115-openpgp/ That will be followed by a side meeting dedicated to PQC in OpenPGP: https://mailarchive.ietf.org/arch/msg/openpgp/ZTPCQQ13OookbjmSNeyptR4ItcM/ I hope you'll be able to attend and raise your concerns. Neal ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Matthew Miller wrote: > Rust tends to be more fine-grained. I don't think this is necessarily > rust-specific _really_ — I think it's a trend as people get more used to > this way of doing things. And this is inherently a PITA to package, unfortunately. It is indeed not Rust-specific, other new fancy languages are a similar mess, see Node.js (with NPM), Go, etc. Another reason why GNU/Linux developers should stick to C and/or C++. That said, both KDE and GNOME have also gone through a phase of splitting craze whose consequences we are still suffering from: The software from the KDE project used to be a dozen packages that could be updated and built by one person by hand in a day. Now we have hundreds of packages released on 3 different release cycles (4 if you include the new Plasma Mobile Gear) that need scripts to update, take days to build even with scripts, and lead to updates on whose sheer number of packages Bodhi and OpenQA frequently choke. And this goes also and especially for the libraries: kdelibs used to be one package, now there are dozens of kf5-* packages, likewise Qt. And it is no better in GNOME land, they started splitting stuff even before KDE did. But the situation is worse in all those "modern" languages like Rust or Go. (And also that language that calls itself "modern" even though it has been designed decades ago to allow for pointlessly jumping text on websites. NPM is infamous for containing "packages" with a single one-line function.) > I have not tried this with any Rust package. My experience in the past is > that many upstreams find this the kind of thing that makes them go on long > blog rants about distro packaging -- they picked a version, it works fine, > they don't need the distraction of being told they must update it. That unhelpfulness (and complete lack of understanding of and for how distributions work) by upstreams is a big issue in those parallel ecosystems. > But even when this doesn't happen, it gets into the matter of expertise. > If I need to update a dependency for a newer-version of the > sub-dependency, and I don't know enough about either code base to do > anything other than file a "please update" bug, then everything is blocked > on that. Normally, it should just be a matter of changing the version number. If that fails to build, IMHO, the dependency (the library) is broken. Library upstream "maintainers" with a complete disregard for backwards compatibility are a PITA, no matter in what language. >> We only maintain compat packages where porting to the new version (and >> submitting the changes upstream) is not feasible. Again, isn't that >> how Fedora is supposed to work? > > I guess it depends on how broadly one reads "feasible". :) We normally try pretty aggressively to port packages to new library versions where the incompatibilities are not too bad. I do not see why it should be any different when the library happens to be written in Rust. >> Examples of that might be: >> - wasmtime: I ultimately abandoned the attempt to package it "because >> Fedora Legal", but the packages themselves worked fine > > An aside, but: did I miss something with this on the Legal list? The only > thing I'm finding is a question about how to phrase `Apache-2.0 WITH > LLVM-exception`. See: https://bugzilla.redhat.com/show_bug.cgi?id=2130953 >> We have talked about this multiple times, but it won't work. >> I think this was tried with first-class maven artifact support in >> koji, but we all know how the Java packaging fiasco ended. > > I would rather see it as: we learned some lessons from that approach and > can do it better. Without a concrete proposal on how you want to "do it better", there is really nothing to discuss, because the only thing that we can talk about as is is the approach that we know did not work. So, suggest a new approach and we can analyze whether it has any chance of working any better or not. My guess is that any working approach to allow foreign artifact types in Koji, and also reliably deliver them to users (including ones that want to build or rebuild software), would ultimately be more work than just using RPMs. >> - we change build flags to default to dynamically linking to system >> libraries instead of statically linking against vendored copies > > This too. > > Mostly, at least. Assuming this isn't _prebuilt binaries_ or similar, > upstream may or may not have a good reason or strong opinion. Why would we care about a "strong opinion"? Either there is a good reason or there is not. Irrational demands by unreasonable, uncooperative upstreams ought to be just ignored. Free Software means we can adapt the software to our needs. If upstream will not allow that, it is not Free Software. > I really hope we can look at these and learn how to do it better, instead > of deciding that better isn't possible. And — while I'm not really up on > node — I have pretty good hindsight on what went wrong
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Tue, Nov 1, 2022 at 7:07 PM Demi Marie Obenour wrote: > > On 11/1/22 10:40, Matthew Miller wrote: > > On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote: > >> For intra-project dependencies (i.e. bevy components depending on > >> exact versions of bevy components), this is kind of expected, and we > >> have tools to deal with this kind of situation (though bevy is on a > >> different scale). For dependencies on third-party libraries, this is > >> kind of unexpected, and I wonder why they do things like that? Locking > >> some dependencies to exact versions is usually handled by relying on > >> the lockfile, instead. > > > > I was wrong about this. I actually didn't realize that the ^ was optional. I > > was, um, cargo-culting that around. Ah well. Anyway, that's less of a > > problem than I worried. > > Will the bevy components ever be used outside of bevy? If not, > then they should be bundled. if so, they should not be bundled. Yeah, this is something that I hope to be able to improve soon. > >>> The packaging guidelines say that I SHOULD create patches to update to > >>> latest versions of dependencies, and that I should further convince the > >>> upstream to take them. Candidly, that seems like a waste of everone's > >>> time. > >> This is *not* a waste of time. If we don't invest time to do that, many > >> project's dependencies grow stale, and actually *increase* the need for us > >> to maintain compat packages. > > > > I have not tried this with any Rust package. My experience in the past is > > that many upstreams find this the kind of thing that makes them go on long > > blog rants about distro packaging -- they picked a version, it works fine, > > they don't need the distraction of being told they must update it. > > I heavily doubt this will be an issue with Rust. There is a reason > that dependabot is so popular. Exactly. I forgot to reply to this in my previous post, but many Rust projects on GitHub use dependabot (or similar bots) to automatically bump their dependencies to the latest version. The projects that *don'*t do that are the odd ones out, but still, PRs submitted to these projects are usually received positively. (snip) > > I don't dispute that helping projects keep up to the latest is valuable > > work. It even seems like it might be in-scope work for Fedora. But couldn't > > we do that as something _separate_ from blocking ourselves (either literally > > or through the extra overhead of compat packages) from packaging the > > dependent app? > > I don’t think it should be a blocker unless e.g. the out-of-date > dependency has a serious security vulnerability. Yeah, I tend to be rather pragmatic in this regard. Patch and submit to upstream if it's easy, keep a compat for the older version around if it's not. The only exception to this rule I recently made was to retire some outdated versions of HTTP libraries, which were only used by obsolete packages anyway. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Hi all, Just a note that over the summer, our intern did a project to try and address some of these issues (namely, that while it's trivial to convert a single crate to an RPM, trying to automate packaging all the dependencies and making sure that you don't break anything else while doing so is tedious and error-prone). Matt and Fabio might recall me getting their inputs on this several months ago. The tool: - recursively checking out packages - updating existing packages while carrying over manual changes - creating compatibility packages when upgrading packages with dependents - automate parallel COPR test builds of sets of packages in dependency order - chain-build all packages in Koji with requested side tag - merging and chain-building all packages across release branches - help review a rust update in Bodhi and verify no compatibility issues are introduced It does not automatically commit anything, of course, and right now is a bit opinionated in favor of picking the lowest satisfactory version possible, which is probably not what we want long term -- but it hopefully provides a nice foundation on which to build on I've finally gotten round to doing some polishing and getting it packaged: - updates for Fedora 36, 37, and Rawhide: https://bodhi.fedoraproject.org/updates/?search=rust-update-set-0.0.1=python-rust-update-set - Pagure repo: https://pagure.io/fedora-rust/rust-update-set There are some fixes for corner cases I encountered while trying to update our `below` packages with this, and some changes needed to get this packageable, but those are minor. It mostly works really well, and is at a stage where hopefully people can take a look and make suggestions for improvements. Best regards, -- Michel Alexandre Salim identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2 signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Tue, Nov 1, 2022 at 3:40 PM Matthew Miller wrote: > > On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote: > > I'll respond inline. > > Me too -- and apologies for the delay. > > > > > I fundamentally disagree with Kevin on a deep level about "entirely > > > useless", but ... find myself kind of agreeing about the "unpackagable" > > > part. I mean: clearly we've found a way, but I'm really not sure we're > > > providing a lot of _value_ in this approach, and I'm also not sure it's > > > as successful as it could be. > > We *do* provide value to both users *and* developers by doing things > > the way we do, but the benefits might not be obvious to people who > > don't know how (Rust) packaging works, and what we as package > > maintainers do. > > Let me rephrase: I absolutely think you've provided value and are providing > value (and I appreciate it). I am not convinced that the value is in the > RPM-izing part, though. > > > [...] > > This is due to a limitation of how cargo handles target-specific > > dependencies - all dependencies that are *mentioned in any way* need > > to be *present* for it to resolve dependencies / enabled optional > > features / update its lockfile etc. But since we don't want to package > > bindings for Windows and mac OS system APIs, we need to actually patch > > them out, otherwise builds will fail. > > Theoretically, if we had our own crate repository, we could either make > those changes there (possibly using something like packit to carry the > patches) -- or, just, not make the changes and not worry because we know > those won't end up used anyway? That won't do. It would mean that we end up having to review and keep track of *more* crates instead of fewer. > > You must realize that this is an extreme case. For many Rust > > applications that people want to package for Fedora, the number of > > dependencies that are missing is rather small, *because* most popular > > libraries are already packaged. > > It may be that I just hear about the difficult cases. Of course. People who manager to package their favourite Rust app without problems don't come to complain, do they? > > We might need to reconsider how to package projects like this. I'm > > pretty sure we could find a way to package them in a way that's > > compatible with how we're currently doing things but would be much > > less busywork. > > Okay, I'm open to that. I'll have more time to spend on this soon, I hope, so I'll try to come back with a prototype ASAP. > > Sure, but isn't that the case for most projects that a newcomer wants > > to package, regardless of programming language? Say, somebody wants to > > package some cool new Python project for machine learning, then > > there's probably also some linear algebra package or SIMD math library > > in the dependency tree that's missing from Fedora. How is that > > different? > > Rust tends to be more fine-grained. I don't think this is necessarily > rust-specific _really_ — I think it's a trend as people get more used to > this way of doing things. With Python, there are some big packages > (including "batteries included" standard Python itself) which tend to group > big related sets of functionality. (notably: numpy, scipy, pandas...) It's probably a misconception that the Rust standard library is small. It's not small, it's just not not very *broad*. And there's some libraries that are also quite "batteries included", for example, hyper and tokio, which could be considered "standard" at this point. (snip) > I have not tried this with any Rust package. My experience in the past is > that many upstreams find this the kind of thing that makes them go on long > blog rants about distro packaging -- they picked a version, it works fine, > they don't need the distraction of being told they must update it. > > But even when this doesn't happen, it gets into the matter of expertise. If > I need to update a dependency for a newer-version of the sub-dependency, and > I don't know enough about either code base to do anything other than file a > "please update" bug, then everything is blocked on that. It's not, though. This is what SIGs are for. I explicitly tell people who are new to Rust packaging that I'll help them get their dependencies reviewed and keep them up-to-date so they can focus on their own stuff. That some people don't accept help when it is offered is a different problem ... > I don't dispute that helping projects keep up to the latest is valuable > work. It even seems like it might be in-scope work for Fedora. But couldn't > we do that as something _separate_ from blocking ourselves (either literally > or through the extra overhead of compat packages) from packaging the > dependent app? > > > > The guidelines provide for creating compat packages, but that means 1) the > > > existing shared work is less useful, 2) requires even more extra steps, > > > and > > > 3) even without reviews for compat has extra administrative overhead.
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On 11/1/22 10:40, Matthew Miller wrote: > On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote: >> I'll respond inline. > > Me too -- and apologies for the delay. > > >>> I fundamentally disagree with Kevin on a deep level about "entirely >>> useless", but ... find myself kind of agreeing about the "unpackagable" >>> part. I mean: clearly we've found a way, but I'm really not sure we're >>> providing a lot of _value_ in this approach, and I'm also not sure it's >>> as successful as it could be. >> We *do* provide value to both users *and* developers by doing things >> the way we do, but the benefits might not be obvious to people who >> don't know how (Rust) packaging works, and what we as package >> maintainers do. > > Let me rephrase: I absolutely think you've provided value and are providing > value (and I appreciate it). I am not convinced that the value is in the > RPM-izing part, though. Please do not stop packaging Rust crates. Qubes OS at least will be relying on these packages to build its own Rust code in the not-too-distant future. > [...] >> This is due to a limitation of how cargo handles target-specific >> dependencies - all dependencies that are *mentioned in any way* need >> to be *present* for it to resolve dependencies / enabled optional >> features / update its lockfile etc. But since we don't want to package >> bindings for Windows and mac OS system APIs, we need to actually patch >> them out, otherwise builds will fail. > > Theoretically, if we had our own crate repository, we could either make > those changes there (possibly using something like packit to carry the > patches) -- or, just, not make the changes and not worry because we know > those won't end up used anyway? > > >> You must realize that this is an extreme case. For many Rust >> applications that people want to package for Fedora, the number of >> dependencies that are missing is rather small, *because* most popular >> libraries are already packaged. > > It may be that I just hear about the difficult cases. Probably so. >> Sure, but isn't that the case for most projects that a newcomer wants >> to package, regardless of programming language? Say, somebody wants to >> package some cool new Python project for machine learning, then >> there's probably also some linear algebra package or SIMD math library >> in the dependency tree that's missing from Fedora. How is that >> different? > > Rust tends to be more fine-grained. I don't think this is necessarily > rust-specific _really_ — I think it's a trend as people get more used to > this way of doing things. With Python, there are some big packages > (including "batteries included" standard Python itself) which tend to group > big related sets of functionality. (notably: numpy, scipy, pandas...) There is another factor that I think needs to be mentioned here: In Rust, a crate is not only the unit of packaging, but also of compilation. A project may well have several crates purely for internal code organization or to reduce build time. Some of these crates may never be intended for use outside of the larger project. In this case, bundling them is absolutely the correct decision, as nobody else should ever be depending on them. However, bundling is *not* the right decision if a crate *is* intended to have outside users. >> For intra-project dependencies (i.e. bevy components depending on >> exact versions of bevy components), this is kind of expected, and we >> have tools to deal with this kind of situation (though bevy is on a >> different scale). For dependencies on third-party libraries, this is >> kind of unexpected, and I wonder why they do things like that? Locking >> some dependencies to exact versions is usually handled by relying on >> the lockfile, instead. > > I was wrong about this. I actually didn't realize that the ^ was optional. I > was, um, cargo-culting that around. Ah well. Anyway, that's less of a > problem than I worried. Will the bevy components ever be used outside of bevy? If not, then they should be bundled. if so, they should not be bundled. >>> The packaging guidelines say that I SHOULD create patches to update to >>> latest versions of dependencies, and that I should further convince the >>> upstream to take them. Candidly, that seems like a waste of everone's >>> time. >> This is *not* a waste of time. If we don't invest time to do that, many >> project's dependencies grow stale, and actually *increase* the need for us >> to maintain compat packages. > > I have not tried this with any Rust package. My experience in the past is > that many upstreams find this the kind of thing that makes them go on long > blog rants about distro packaging -- they picked a version, it works fine, > they don't need the distraction of being told they must update it. I heavily doubt this will be an issue with Rust. There is a reason that dependabot is so popular. > But even when this doesn't happen, it gets into the matter of
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote: > I'll respond inline. Me too -- and apologies for the delay. > > I fundamentally disagree with Kevin on a deep level about "entirely > > useless", but ... find myself kind of agreeing about the "unpackagable" > > part. I mean: clearly we've found a way, but I'm really not sure we're > > providing a lot of _value_ in this approach, and I'm also not sure it's > > as successful as it could be. > We *do* provide value to both users *and* developers by doing things > the way we do, but the benefits might not be obvious to people who > don't know how (Rust) packaging works, and what we as package > maintainers do. Let me rephrase: I absolutely think you've provided value and are providing value (and I appreciate it). I am not convinced that the value is in the RPM-izing part, though. [...] > This is due to a limitation of how cargo handles target-specific > dependencies - all dependencies that are *mentioned in any way* need > to be *present* for it to resolve dependencies / enabled optional > features / update its lockfile etc. But since we don't want to package > bindings for Windows and mac OS system APIs, we need to actually patch > them out, otherwise builds will fail. Theoretically, if we had our own crate repository, we could either make those changes there (possibly using something like packit to carry the patches) -- or, just, not make the changes and not worry because we know those won't end up used anyway? > You must realize that this is an extreme case. For many Rust > applications that people want to package for Fedora, the number of > dependencies that are missing is rather small, *because* most popular > libraries are already packaged. It may be that I just hear about the difficult cases. > We might need to reconsider how to package projects like this. I'm > pretty sure we could find a way to package them in a way that's > compatible with how we're currently doing things but would be much > less busywork. Okay, I'm open to that. > Sure, but isn't that the case for most projects that a newcomer wants > to package, regardless of programming language? Say, somebody wants to > package some cool new Python project for machine learning, then > there's probably also some linear algebra package or SIMD math library > in the dependency tree that's missing from Fedora. How is that > different? Rust tends to be more fine-grained. I don't think this is necessarily rust-specific _really_ — I think it's a trend as people get more used to this way of doing things. With Python, there are some big packages (including "batteries included" standard Python itself) which tend to group big related sets of functionality. (notably: numpy, scipy, pandas...) > For intra-project dependencies (i.e. bevy components depending on > exact versions of bevy components), this is kind of expected, and we > have tools to deal with this kind of situation (though bevy is on a > different scale). For dependencies on third-party libraries, this is > kind of unexpected, and I wonder why they do things like that? Locking > some dependencies to exact versions is usually handled by relying on > the lockfile, instead. I was wrong about this. I actually didn't realize that the ^ was optional. I was, um, cargo-culting that around. Ah well. Anyway, that's less of a problem than I worried. > > The packaging guidelines say that I SHOULD create patches to update to > > latest versions of dependencies, and that I should further convince the > > upstream to take them. Candidly, that seems like a waste of everone's > > time. > This is *not* a waste of time. If we don't invest time to do that, many > project's dependencies grow stale, and actually *increase* the need for us > to maintain compat packages. I have not tried this with any Rust package. My experience in the past is that many upstreams find this the kind of thing that makes them go on long blog rants about distro packaging -- they picked a version, it works fine, they don't need the distraction of being told they must update it. But even when this doesn't happen, it gets into the matter of expertise. If I need to update a dependency for a newer-version of the sub-dependency, and I don't know enough about either code base to do anything other than file a "please update" bug, then everything is blocked on that. I don't dispute that helping projects keep up to the latest is valuable work. It even seems like it might be in-scope work for Fedora. But couldn't we do that as something _separate_ from blocking ourselves (either literally or through the extra overhead of compat packages) from packaging the dependent app? > > The guidelines provide for creating compat packages, but that means 1) the > > existing shared work is less useful, 2) requires even more extra steps, and > > 3) even without reviews for compat has extra administrative overhead. > > We only maintain compat packages where porting to the new
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/24/22 23:23, Petr Menšík wrote: Hi, maybe it was already answered. Not long ago Thunderbird switched from using installed GPG to its own implementation inside. I think I have found the library part and it seems to be in C++, which should be much more easier to integrate than rust libraries. I think the project link is: https://github.com/rnpgp/rnp Wouldn't it solve the problems cause in more compatible way? Is has relatively recent release, so it does not seem abandoned. Is there a specific reason, why is a Rust implementation chosen instead? Yes it was already answered, see https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/WSKLHCVFABW442MWDHEIBBE4ZJMLACB2/ and https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/YOAI3MQYD3DX7KTM3M6ENFJ5ULHYO3I3/ We would've, *of course*, gone for something C-nativeish if that had been an option at all. As I said in some other post in this thread, I've been on the lookout for a viable C-native option for 15+ years. Yet here we are. And as I've also said elsewhere in this thread, the plan is to keep the options open for the future. I don't like the shotgun marriage to Rust any more than the next person out there. I like Rust language, but its integration into a core system component does not seem easy. Except for the matter of bootstrap dependencies (which has also been discussed here already), I don't know what the difficulty in this case is supposed to be. - Panu ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Hi, maybe it was already answered.
Not long ago Thunderbird switched from using installed GPG to its own
implementation inside. I think I have found the library part and it
seems to be in C++, which should be much more easier to integrate than
rust libraries.
I think the project link is:
https://github.com/rnpgp/rnp
Wouldn't it solve the problems cause in more compatible way? Is has
relatively recent release, so it does not seem abandoned. Is there a
specific reason, why is a Rust implementation chosen instead? I like
Rust language, but its integration into a core system component does not
seem easy.
Cheers,
Petr
On 20. 10. 22 11:03, Miro Hrončok wrote:
On 10. 10. 22 16:32, Ben Cotton wrote:
For the last 20 years or so, RPM has used a home-grown OpenPGP parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100%
available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't
be able to build rpms. 3) seems somewhat tedious for no good reason.
--
Petr Menšík
Software Engineer, RHEL
Red Hat, http://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
OpenPGP_0x4931CA5B6C9FC5CB.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote: > On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller > wrote: > > > > I _very much_ appreciate all the work you and the other Rust SIG folks > > (Igor and Zbyszek in particular but I'm sure others as well!) have put into [… and Fabio in particular] > > I'll start with the second: our system for Rust doesn't really do that. > > Developers are going to use cargo and crates.io and we're not going to > > convince them that they should do otherwise. (I don't expect anyone > > disagrees with this.) > > This is true, and probably also not "fixable". Also, arguably it's not something we'd like to "fix" at all. I think not-using-the-rpm-packaged-software-during-development is something that is equally true for Rust, C, Python, Java, and pretty much any language. As long as Rust is not exclusive to Fedora/CentOS/RHEL family, rpms are just going to be one of the possible ways to get rust code. For me, the goal is to make the automatic conversion from rust crates to rpms as smooth as possible. I want the rpm package database to be the CDN we use to get verified Rust code. > > *This is what open source winning looks like.* [Snip various details here. I think Fabio's points are very good.] > > We could also attach other metadata to the packages in the cache. Maybe some > > popularity, update frequency from Cargo.io, but also package review flags: > > checked license against source, and whatever other auditing we think should > > be done. This moves the focus from specfile-correctness to the package > > itself, and the effort from packaging to reviewing. (I'd suggest that for > > the experiment, we not make any deep auditing manditory, but instead > > encouraged.) And these flags should be able to be added by anyone in the > > Rust SIG, not necessarily just at import. > > This is already the case, though? > Writing a spec file for a new crate is already automated to the point > where "standard" crates can be 100% automatically generated and need > zero manual edits. > If manual changes *are* required, then these changes would also be > required in the "first-class crate artifact" scenario, so you don't > gain anything. > And if there's other problems that are caught during package review, > the distribution mechanism doesn't matter, either. > > In my experience, changing the distribution mechanism or packaging > paradigm will often make things *worse* instead of better. For > example, the implosion of the NodeJS package ecosystem in Fedora was > not only caused by the horrid state NPM, but also because the new > packaging guidelines which prefer bundling essentially made it > impossible for packagers to verify that objectionable content is > present in vendored dependencies. For Java, Modularity was seen as a > "solution", but the result was that basically everybody - except for > the Red Hat maintainers who maintained the modules - just stopped > doing Java packaging because of the hostile environment. > > > Fedora _needs_ to adapt to stay relevant in the world where every language > > stack has developed a packaging ecosystem which effectively ignores us. Some > > of them are missing lessons they could have learned, ah well — but they also > > have a lot of nice new ideas we're missing. And, no matter what we think, > > we're clearly not going to stop them. > > > > Rust packaging seems like a great place to lead the way — and then we can > > maybe expand to Go, which has similar issues, and then Java (where, you > > know, things have already collapsed despite heroic effort.) > > Oh, actually, I don't think Rust packaging is a good place to start > here at all. :) > > The way cargo works already maps very neatly onto how RPM packages > work, which is definitely *not* the case for other language > ecosystems. I also think we could even massively improve handling of > "large" projects with many sub-components (like bevy, zola, wasmtime, > deno, etc.) - which are currently the only projects that are "painful" > to package - *without* completely changing the underlying packaging > paradigm or distribution mechanism. (I've been wanting to actually > write better tooling for this use case, but alas, Bachelor thesis is > more important for now.) > > Given that, I think we're actually in kind of a *good* situation with > Rust packaging, especially compared to other language ecosystems - not > only right now, but also looking at the future. And looking at the > alternatives, all attempts at trying different approaches (maven > artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.) > have *failed* and ultimately made things worse instead of improving > the situation - the only thing that has proven to be sustainable (for > now) is ... maybe surprisingly, plain RPM packages. Yep. I expect some power-law distribution of package popularity. It seems like there's an endless supply of crates, but most just don't matter. If we get some
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On 10/20/22 10:01, Neal Gompa wrote: > On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel > wrote: >> Rust needs to adapt to become relevant in GNU/Linux distributions. >> > > There is nobody pushing for Rust to improve anymore. When Igor and I > were building out Fedora Rust, we did some engagement with Rust > upstream about stabilizing Rust's ABI so we could ship dynamic > libraries. While one or two members of the Rust core team were > sympathetic, most of the Rust community attacked me for "trying to > make Rust like C" and I got crap from people on the Rust community > channels, Twitter, and other places. Eventually, I flamed out because > there's only so much punishment someone can take over it. Personal attacks are not okay. > A couple of years ago, there was a revival of the topic[1], but it > went nowhere again. > > Until the situation changes, I'm very firmly anti-Rust. Unfortunately, > sometimes I have no choice but to deal with it. > > [1]: https://github.com/slightknack/rust-abi-wiki There are a couple major constraints that apply to Rust: 1. Rust implements generics via monomorphisation. This means that e.g. Vec *has no ABI at all*: code will only be generated for Vec when it is instantiated. 2. Rust relies heavily on cross-crate inlining to get good performance. If e.g. Option::map is not inlined into the caller, performance will be terrible. Neither of those constraints are unique to Rust. C++ template libraries have exactly the same problems. It’s just that we think of C++ headers as being the interface to a library, instead of the implementation. Trying to ship the implementation of types like Vec and Option in a shared library makes no more sense than doing so with C++’s std::vector and std::optional, which is to say, none at all. Another factor is maintaining a stable ABI severely limits library evolution. In C++, ABI stability has resulted in std::regex being *far* slower than third-party regular expression libraries. I am not surprised that Rust wants to avoid similar problems. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
To Neal's point, I had the audacity to suggest some improvements along the lines of docutils and the response was underwhelming. https://users.rust-lang.org/t/rust-analog-to-the-python-compilers-docutils/82813?u=blaisep On Thu, Oct 20, 2022 at 10:02 AM Neal Gompa wrote: > On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel > wrote: > > > > Matthew Miller wrote: > > > I fundamentally disagree with Kevin on a deep level about "entirely > > > useless", but ... find myself kind of agreeing about the "unpackagable" > > > part. I mean: clearly we've found a way, but I'm really not sure we're > > > providing a lot of _value_ in this approach, and I'm also not sure > it's as > > > successful as it could be. > > > > We cannot ship what we cannot package. (Just repackaging upstream blobs > is a > > no-go, it is explicitly banned by Fedora Packaging Guidelines for good > > reasons.) > > > > > I'll start with the second: our system for Rust doesn't really do that. > > > Developers are going to use cargo and crates.io and we're not going to > > > convince them that they should do otherwise. (I don't expect anyone > > > disagrees with this.) > > > > That is one of the issues (along with lack of proper shared library > support) > > that makes Rust such a painful language: it makes it way too easy for > > developers to just add a dependency on some random unpackaged (by > > distributions) library with some random license and with random > transitive > > dependencies. > > > > In a reasonable programming language, you have to think twice before you > add > > yet another dependency to your project. Tools like cargo make it way too > > easy, just add a line and rebuild. So then you end up with a mess like > this: > > > > > But running `cargo fetch` with a clean cache pulls down *390* crates. > > > > 390 dependencies is absolutely insane! No project should ever depend on > so > > many libraries. This is completely unmaintainable. > > > > > *This is what open source winning looks like.* > > > > > > I remember a Byte magazine article from the 1990 (I just checked!) with > > > the title "There Is a Silver Bullet: The birth of interchangeable, > > > reusable software components will bring software into the information > > > age". [1] This was about the newly-hot idea of Object Oriented > > > Programming. It was very exciting. But, of course, that vision of the > > > world did not happen. It turns out proprietary software *can't* do > this. > > > > > > But now we have it! I don't have to reinvent every basic wheel — but > even > > > more than that, I do not have to be an expert in the intricacies of > safe > > > concurrency to write an app that uses that under the hood. That's > amazing! > > > I can do such powerful things from high-level interfaces and trust the > > > expertise of those who really understand the deep computer science > some of > > > this requires. > > > > Code reuse is great when it makes sense, but this "NPM culture" where > > developers happily depend on tons of packages containing trivial one-line > > functions is completely insane. > > > > If this is now the essence of "Open Source", then it will ultimately > lose. > > Proprietary software companies will just be waiting for the next paradigm > > shift to lock us into their proprietary ecosystem again. For a large > part, > > this has already happened with smartphones: you get to choose only > between > > the Android walled garden (with an open core and lots of proprietary > Google > > services and (Google and third-party) apps on top) and the iOS walled > garden > > (which is completely proprietary and closed, even strictly enforcing an > App > > Store monopoly). We are struggling to escape from that with devices like > the > > PinePhone. But with development practices relying on an unmaintainable > > dependency hell, it will never happen. > > > > > I am competent enough to write a silly toy game using Bevy. It might be > > > good enough that others will enjoy it. *I am not competent to maintain > > > many of these dependencies.* I don't even know what most of them DO. > > > "anyhow"? "bytemuck"? > > > > Welcome to dependency hell! > > > > > Worse, many of the Bevy deps are specified with exact versions. Maybe I > > > could make the package work with the packaged versions, but ... that > > > requires deep expertise and even then might lead to unexpected behavior > > > and has a high chance of putting me at odds with both the engine > upstream > > > and any other games which use it. The packaging guidelines say that I > > > SHOULD create patches to update to latest versions of dependencies, and > > > that I should further convince the upstream to take them. Candidly, > that > > > seems like a waste of everone's time. > > > > Hardcoding exact versions of dependencies is one of the worst > misfeatures of > > those language-specific build tools. It effectively prevents you from > fixing > > security issues without patching every single application. And of
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 20. 10. 22 14:26, Panu Matilainen wrote:
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100% available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
to build rpms. 3) seems somewhat tedious for no good reason.
I was under the impression Rust was available for all architectures (for Fedora
anyway), no? There's no Rust code in rpm now either this didn't even cross my
mind really :D
Technically, I guess the right thing to do is 1) when Sequoia is enabled. Ie:
%if %{with sequoia}
%global crypto sequoia
BuildRequires: rpm-sequoia-devel >= 1.0.0
ExclusiveArch: %{rust_archves}
%else
%global crypto openssl
BuildRequires: openssl-devel
%endif
This is already in rawhide, except for the ExclusiveArch thing.
That said, the non-sequoia options should be considered only a bootstrap aid,
we're not going to provide security support for the internal parser for some
fringe architectures only.
I'm not sure that answered your questions though.
Partially.
My main point is: If we don't support rust-free (stainless?) RPM, what is the
point in mandating the use of ExclusiveArch: %{rust_arches} in our packages,
when it is by definition always contain the architecture this is build on?
Hence, I proposed to the FPC that we should no longer make the use of
ExclusiveArch: %{rust_arches} mandatory:
https://pagure.io/packaging-committee/issue/1220
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 20. 10. 22 12:11, Fabio Valentini wrote:
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100% available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
to build rpms. 3) seems somewhat tedious for no good reason.
I think 2) would be the easiest solution in the long term. rustc and
LLVM now already support all targets that we*might* want to add in
the near-to-mid-future (riscv64gc-unknown-linux-gnu comes to mind -
its support is at the same level as other targets we already have).
I've opened https://pagure.io/packaging-committee/issue/1220
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel wrote: > > Matthew Miller wrote: > > I fundamentally disagree with Kevin on a deep level about "entirely > > useless", but ... find myself kind of agreeing about the "unpackagable" > > part. I mean: clearly we've found a way, but I'm really not sure we're > > providing a lot of _value_ in this approach, and I'm also not sure it's as > > successful as it could be. > > We cannot ship what we cannot package. (Just repackaging upstream blobs is a > no-go, it is explicitly banned by Fedora Packaging Guidelines for good > reasons.) > > > I'll start with the second: our system for Rust doesn't really do that. > > Developers are going to use cargo and crates.io and we're not going to > > convince them that they should do otherwise. (I don't expect anyone > > disagrees with this.) > > That is one of the issues (along with lack of proper shared library support) > that makes Rust such a painful language: it makes it way too easy for > developers to just add a dependency on some random unpackaged (by > distributions) library with some random license and with random transitive > dependencies. > > In a reasonable programming language, you have to think twice before you add > yet another dependency to your project. Tools like cargo make it way too > easy, just add a line and rebuild. So then you end up with a mess like this: > > > But running `cargo fetch` with a clean cache pulls down *390* crates. > > 390 dependencies is absolutely insane! No project should ever depend on so > many libraries. This is completely unmaintainable. > > > *This is what open source winning looks like.* > > > > I remember a Byte magazine article from the 1990 (I just checked!) with > > the title "There Is a Silver Bullet: The birth of interchangeable, > > reusable software components will bring software into the information > > age". [1] This was about the newly-hot idea of Object Oriented > > Programming. It was very exciting. But, of course, that vision of the > > world did not happen. It turns out proprietary software *can't* do this. > > > > But now we have it! I don't have to reinvent every basic wheel — but even > > more than that, I do not have to be an expert in the intricacies of safe > > concurrency to write an app that uses that under the hood. That's amazing! > > I can do such powerful things from high-level interfaces and trust the > > expertise of those who really understand the deep computer science some of > > this requires. > > Code reuse is great when it makes sense, but this "NPM culture" where > developers happily depend on tons of packages containing trivial one-line > functions is completely insane. > > If this is now the essence of "Open Source", then it will ultimately lose. > Proprietary software companies will just be waiting for the next paradigm > shift to lock us into their proprietary ecosystem again. For a large part, > this has already happened with smartphones: you get to choose only between > the Android walled garden (with an open core and lots of proprietary Google > services and (Google and third-party) apps on top) and the iOS walled garden > (which is completely proprietary and closed, even strictly enforcing an App > Store monopoly). We are struggling to escape from that with devices like the > PinePhone. But with development practices relying on an unmaintainable > dependency hell, it will never happen. > > > I am competent enough to write a silly toy game using Bevy. It might be > > good enough that others will enjoy it. *I am not competent to maintain > > many of these dependencies.* I don't even know what most of them DO. > > "anyhow"? "bytemuck"? > > Welcome to dependency hell! > > > Worse, many of the Bevy deps are specified with exact versions. Maybe I > > could make the package work with the packaged versions, but ... that > > requires deep expertise and even then might lead to unexpected behavior > > and has a high chance of putting me at odds with both the engine upstream > > and any other games which use it. The packaging guidelines say that I > > SHOULD create patches to update to latest versions of dependencies, and > > that I should further convince the upstream to take them. Candidly, that > > seems like a waste of everone's time. > > Hardcoding exact versions of dependencies is one of the worst misfeatures of > those language-specific build tools. It effectively prevents you from fixing > security issues without patching every single application. And of course it > can cause version conflicts between different applications or even between > transitive dependencies within an application. (And, while the former ones > can typically be resolved by shipping both versions of the library, the > latter ones cannot, because you cannot usually link two versions of the same > library into the same program.) > > > So, going back to Kevin's point: it _does_ feel like this is unpackagable. > > But that's because the barrier to participation
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Matthew Miller wrote: > I fundamentally disagree with Kevin on a deep level about "entirely > useless", but ... find myself kind of agreeing about the "unpackagable" > part. I mean: clearly we've found a way, but I'm really not sure we're > providing a lot of _value_ in this approach, and I'm also not sure it's as > successful as it could be. We cannot ship what we cannot package. (Just repackaging upstream blobs is a no-go, it is explicitly banned by Fedora Packaging Guidelines for good reasons.) > I'll start with the second: our system for Rust doesn't really do that. > Developers are going to use cargo and crates.io and we're not going to > convince them that they should do otherwise. (I don't expect anyone > disagrees with this.) That is one of the issues (along with lack of proper shared library support) that makes Rust such a painful language: it makes it way too easy for developers to just add a dependency on some random unpackaged (by distributions) library with some random license and with random transitive dependencies. In a reasonable programming language, you have to think twice before you add yet another dependency to your project. Tools like cargo make it way too easy, just add a line and rebuild. So then you end up with a mess like this: > But running `cargo fetch` with a clean cache pulls down *390* crates. 390 dependencies is absolutely insane! No project should ever depend on so many libraries. This is completely unmaintainable. > *This is what open source winning looks like.* > > I remember a Byte magazine article from the 1990 (I just checked!) with > the title "There Is a Silver Bullet: The birth of interchangeable, > reusable software components will bring software into the information > age". [1] This was about the newly-hot idea of Object Oriented > Programming. It was very exciting. But, of course, that vision of the > world did not happen. It turns out proprietary software *can't* do this. > > But now we have it! I don't have to reinvent every basic wheel — but even > more than that, I do not have to be an expert in the intricacies of safe > concurrency to write an app that uses that under the hood. That's amazing! > I can do such powerful things from high-level interfaces and trust the > expertise of those who really understand the deep computer science some of > this requires. Code reuse is great when it makes sense, but this "NPM culture" where developers happily depend on tons of packages containing trivial one-line functions is completely insane. If this is now the essence of "Open Source", then it will ultimately lose. Proprietary software companies will just be waiting for the next paradigm shift to lock us into their proprietary ecosystem again. For a large part, this has already happened with smartphones: you get to choose only between the Android walled garden (with an open core and lots of proprietary Google services and (Google and third-party) apps on top) and the iOS walled garden (which is completely proprietary and closed, even strictly enforcing an App Store monopoly). We are struggling to escape from that with devices like the PinePhone. But with development practices relying on an unmaintainable dependency hell, it will never happen. > I am competent enough to write a silly toy game using Bevy. It might be > good enough that others will enjoy it. *I am not competent to maintain > many of these dependencies.* I don't even know what most of them DO. > "anyhow"? "bytemuck"? Welcome to dependency hell! > Worse, many of the Bevy deps are specified with exact versions. Maybe I > could make the package work with the packaged versions, but ... that > requires deep expertise and even then might lead to unexpected behavior > and has a high chance of putting me at odds with both the engine upstream > and any other games which use it. The packaging guidelines say that I > SHOULD create patches to update to latest versions of dependencies, and > that I should further convince the upstream to take them. Candidly, that > seems like a waste of everone's time. Hardcoding exact versions of dependencies is one of the worst misfeatures of those language-specific build tools. It effectively prevents you from fixing security issues without patching every single application. And of course it can cause version conflicts between different applications or even between transitive dependencies within an application. (And, while the former ones can typically be resolved by shipping both versions of the library, the latter ones cannot, because you cannot usually link two versions of the same library into the same program.) > So, going back to Kevin's point: it _does_ feel like this is unpackagable. > But that's because the barrier to participation seems too high. No, the answer to upstream shipping an unpackageable mess cannot possibly be to make it easier to smuggle unpackageable messes into Fedora! > It's not because it's statically-linked
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Peter Robinson wrote: > Why are they insecure? This is public open data, not banking data, > where the data being downloaded is verifiable by the rpm signatures > and signing keys. The metadata is not, at least not currently. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/20/22 12:03, Miro Hrončok wrote:
On 10. 10. 22 16:32, Ben Cotton wrote:
For the last 20 years or so, RPM has used a home-grown OpenPGP parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100%
available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be
able to build rpms. 3) seems somewhat tedious for no good reason.
I was under the impression Rust was available for all architectures (for
Fedora anyway), no? There's no Rust code in rpm now either this didn't
even cross my mind really :D
Technically, I guess the right thing to do is 1) when Sequoia is
enabled. Ie:
%if %{with sequoia}
%global crypto sequoia
BuildRequires: rpm-sequoia-devel >= 1.0.0
ExclusiveArch: %{rust_archves}
%else
%global crypto openssl
BuildRequires: openssl-devel
%endif
This is already in rawhide, except for the ExclusiveArch thing.
That said, the non-sequoia options should be considered only a bootstrap
aid, we're not going to provide security support for the internal parser
for some fringe architectures only.
I'm not sure that answered your questions though.
- Panu -
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Lots of good wisdom here, thank you. IMHO Rust will benefit from whatever "adult supervision" Fedora can provide. -Blaise (currently undergoing treatment for injuries sustained supporting npm in production) On Wed, Oct 19, 2022 at 7:05 AM Fabio Valentini wrote: > On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller > wrote: > > > > I _very much_ appreciate all the work you and the other Rust SIG folks > > (Igor and Zbyszek in particular but I'm sure others as well!) have put > into > > packaging rust apps and crates and all of the systems around that. > > I'll respond inline. > > > I fundamentally disagree with Kevin on a deep level about "entirely > > useless", but ... find myself kind of agreeing about the "unpackagable" > > part. I mean: clearly we've found a way, but I'm really not sure we're > > providing a lot of _value_ in this approach, and I'm also not sure it's > as > > successful as it could be. > > We *do* provide value to both users *and* developers by doing things > the way we do, but the benefits might not be obvious to people who > don't know how (Rust) packaging works, and what we as package > maintainers do. > > > There are three ways having things packaged in Fedora repos _can_ be > > helpful: > > > > 1. End-user applications and tools > > 2. Useful development environment > > 3. As convenience for ourselves for building packages for #1 or #2 > > > > I am not discounting the value of #3 -- making a shared thing that we all > > work on together is kind of the whole point, and the nicer we can make > that > > the better we can bring in more people, and those of us already here > have a > > lighter load and can work on the things we're most interested in. But > > ultimately, we're doing it so we make a useful system for users. That > means > > the first two. > > This I can agree with :) > > > I'll start with the second: our system for Rust doesn't really do that. > > Developers are going to use cargo and crates.io and we're not going to > > convince them that they should do otherwise. (I don't expect anyone > > disagrees with this.) > > This is true, and probably also not "fixable". We need to make some > amount of non-upstreamable patches to some crates (most notably, > removing Windows- or mac OS-specific dependencies, because we don't > want to package those), but in some cases, these are "incompatible" > changes, and Rust *developers* should not be targeting our downstream > sources that have these differences with actual upstream sources. > > This is due to a limitation of how cargo handles target-specific > dependencies - all dependencies that are *mentioned in any way* need > to be *present* for it to resolve dependencies / enabled optional > features / update its lockfile etc. But since we don't want to package > bindings for Windows and mac OS system APIs, we need to actually patch > them out, otherwise builds will fail. > > > We're doing okay with #1, but... I think #3 _even_ with all of the work > in > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > > engine and will probably have a few things it would be nice to package to > > make available in Fedora Linux. I might not even mind maintaining Bevy > > itself. > > Somebody actually already started packaging Bevy components - some > packages are already approved and some are still pending review. Not > sure what the progress has been there, but it's not *impossible*. > > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > which is *amazing*. But... that still is hundreds that I'd have to add. > And > > mostly they are things I don't know _anything_ about. > > You must realize that this is an extreme case. For many Rust > applications that people want to package for Fedora, the number of > dependencies that are missing is rather small, *because* most popular > libraries are already packaged. > > Bevy is a bit special, because it (presumably) pulls in lots of GPU / > OpenGL / Vulkan related libraries, which we didn't need to package for > anything else yet, and it's also split into dozens of small libraries > itself, which can be painful to package, that is true. > > We might need to reconsider how to package projects like this. I'm > pretty sure we could find a way to package them in a way that's > compatible with how we're currently doing things but would be much > less busywork. > > > *This is what open source winning looks like.* > > > > I remember a Byte magazine article from the 1990 (I just checked!) with > the > > title "There Is a Silver Bullet: The birth of interchangeable, reusable > > software components will bring software into the information age". [1] > > This was about the newly-hot idea of Object Oriented Programming. It was > > very exciting. But, of course, that vision of the world did not happen. > It > > turns out proprietary software *can't* do this. > > > > But now we have it! I
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 20, 2022 at 11:03 AM Miro Hrončok wrote:
>
> On 10. 10. 22 16:32, Ben Cotton wrote:
> > For the last 20 years or so, RPM has used a home-grown OpenPGP parser
> > for dealing with keys and signatures. That parser is rather infamous
> > for its limitations and flaws, and especially in recent years has
> > proven a significant burden to RPM development. In order to improve
> > security and free developer resources for dealing with RPM's "core
> > business" instead, RPM upstream is in the process of deprecating the
> > internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
> > based solution written in Rust.
> > At this point the change is mostly invisible in normal daily use.
>
> Which of the following will happen:
>
> 1) rpm will gain ExclusiveArch: %{rust_arches}
> 2) we will stop requiring the above in Rust packages, as Rust is 100%
> available
> 3) rpm will %ifarch %{rust_arches} this change
> 4) something else (what?)
>
> IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
> to build rpms. 3) seems somewhat tedious for no good reason.
I think 2) would be the easiest solution in the long term. rustc and
LLVM now already support all targets that we *might* want to add in
the near-to-mid-future (riscv64gc-unknown-linux-gnu comes to mind -
its support is at the same level as other targets we already have).
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10. 10. 22 16:32, Ben Cotton wrote:
For the last 20 years or so, RPM has used a home-grown OpenPGP parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100% available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
to build rpms. 3) seems somewhat tedious for no good reason.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19/10/2022 13:56, Vitaly Zaitsev via devel wrote: On 19/10/2022 10:31, Neal Gompa wrote: HTTPS does not help with that. It's just a transport protocol. It will. All requests will be encrypted. ISP will only see server's IP-address and its hostname (only if SNI is enabled). HTTPS does not automatically add privacy for users of mirrors. See for example the excellent Debian related website and OpenBSD presentation about how you can still identify downloaded packages even when using HTTPS. [1] https://whydoesaptnotusehttps.com/ [2] https://www.openbsd.org/papers/eurobsdcon_2018_https.pdf ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19-10-2022 14:51, Stephen Smoogen wrote: On Wed, 19 Oct 2022 at 05:09, Sandro wrote: On 19-10-2022 10:31, Neal Gompa wrote: On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel We don't have a "main mirror" for that to work. dl.fedoraproject.org would be what this would aim at but we would probably have to change the disk layout for what the CDN works best for. So, this has been looked into already? It definitely sounds like it could help in sparsely served parts of the world at a reasonable cost. Please do not use the word 'reasonable' cost without actually knowing what the costs are :). 1. Amazon is currently donating bandwidth and systems for our work but it is in a non-written format. If Amazon tomorrow decided that they needed to concentrate on other things or look at what we are costing and say 'well it was time to stop', we would have to turn it all off then. If we require various flags in the CDN we have to go through their change management and accounting to get them done. 2. Akamai and other CDN's are expensive. You pay to put content into them, you pay to clear out caches, you pay differently for the areas where the CDN is covering, and you pay for the amount of data come out. You also pay for them to block bad actors whose job is to try and make you go out of business by downloading constantly. For even a 'niche' market like Fedora, we would be looking at least 1.5 million per year in costs. 3. CDN's work well for software which does not change a lot but if you have updates daily you end up having to force cache cleanouts and new upload costs. Even then you end up with needing to deal with 'broken' parts of the CDN and people getting old data. This happens regularly with COPR and CentOS parts which have been using the Amazon CDN at times. All of these are possible to get past, but they will take time, effort and may be more than what anyone wants to be 'reasonable'. Point taken. ;) Thanks for the very detailed answer. I only glanced at the pricing and just now noticed the *most Important* part: '_per GB_'. So, yeah, this will add up for sure. I should have been more careful. -- Sandro ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 1:47 PM Vitaly Zaitsev via devel wrote: > > On 19/10/2022 14:14, Daniel P. Berrangé wrote: > > IOW, the impact of AES on server peformance will vary depending > > on CPU models, NIC models / network switches and whether other > > workloads are competing for CPU time. Admins need to decide > > what tradeoffs are important to them. > > In future, modern web browsers will start rejecting unencrypted HTTP > traffic (except localhost and intranet, ofc). All server owners should > switch to HTTPS as soon as possible. We don't use a browser for general updates, we use an application which can configure libcurl for it's desired requirements. The use of AES is not just required for client side, but on server side too, it takes non inconsiderable amount of compute to saturate a 10G+ link using https, that compute needs to come from somewhere and against most of the mirrors are being provided as a community donation and likely have requirements where if it hits a threshold it likely makes it unviable for that company to provide where they don't get a return (advertising, or something pro-rata) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 9:33 AM Neal Gompa wrote: > > On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel > wrote: > > > > On 19/10/2022 09:48, Peter Robinson wrote: > > > Sure but as mentioned it's public data, and the modification, and I > > > covered that in my reply, would be picked up by the other mechanisms. > > > > They can collect a lot of sensitive information: your IP, Fedora > > version, packages version, etc. This can help with recon for attackers. > > > > HTTPS does not help with that. It's just a transport protocol. > > > > There isn't actually that many mirrors left do we > > > really want to reduce the number more for end users for no actual > > > improvement in security? > > > > It will improve privacy at least. > > > > Not in any meaningful way, and in most cases HTTPS makes mirrors slower too. > > > > Ultimately bandwidth is expensive in a lot of > > > parts of the world for commercial entities to provide, that's why > > > there's mirrors. > > > > Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can > > switch to a CDN too? > > > > We don't have a "main mirror" for that to work. We do, it's called download.fedoraproject.org (or dl.fedoraproject.org for the non redirect to mirror*) but who is going to pay for the AWS traffic, the mirror already is plugged into AWS for AWS traffic so if you're running instances in the cloud you actually get it directly from AWS as that's in their best interest but it's not covered for everyone on the general internet. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, 19 Oct 2022 at 05:09, Sandro wrote: > On 19-10-2022 10:31, Neal Gompa wrote: > > On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel > > > We don't have a "main mirror" for that to work. > > dl.fedoraproject.org would be what this would aim at but we would probably have to change the disk layout for what the CDN works best for. > So, this has been looked into already? It definitely sounds like it > could help in sparsely served parts of the world at a reasonable cost. > Please do not use the word 'reasonable' cost without actually knowing what the costs are :). 1. Amazon is currently donating bandwidth and systems for our work but it is in a non-written format. If Amazon tomorrow decided that they needed to concentrate on other things or look at what we are costing and say 'well it was time to stop', we would have to turn it all off then. If we require various flags in the CDN we have to go through their change management and accounting to get them done. 2. Akamai and other CDN's are expensive. You pay to put content into them, you pay to clear out caches, you pay differently for the areas where the CDN is covering, and you pay for the amount of data come out. You also pay for them to block bad actors whose job is to try and make you go out of business by downloading constantly. For even a 'niche' market like Fedora, we would be looking at least 1.5 million per year in costs. 3. CDN's work well for software which does not change a lot but if you have updates daily you end up having to force cache cleanouts and new upload costs. Even then you end up with needing to deal with 'broken' parts of the CDN and people getting old data. This happens regularly with COPR and CentOS parts which have been using the Amazon CDN at times. All of these are possible to get past, but they will take time, effort and may be more than what anyone wants to be 'reasonable'. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Once upon a time, Vitaly Zaitsev said: > On 19/10/2022 09:33, Peter Robinson wrote: > >Why are they insecure? This is public open data, not banking data, > >where the data being downloaded is verifiable by the rpm signatures > >and signing keys. > > ISPs or anyone on the the same network can view, intercept or even > modify HTTP/rsync traffic. And so? The mirror network is essentially already an untrusted man in the middle between the Fedora build servers and you. Most mirror servers are hosted by large organizations (companies and schools with extra bandwidth) - in many cases, they ARE the ISPs. At a previous ISP job, I ran a mirror and marked it as "preferred" for my networks, which meant all my customers used my mirror by default. Anybody can set that up, with pretty minimal verification. You are trying to insist on a trust layer for an already-untrusted system. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19/10/2022 14:14, Daniel P. Berrangé wrote: IOW, the impact of AES on server peformance will vary depending on CPU models, NIC models / network switches and whether other workloads are competing for CPU time. Admins need to decide what tradeoffs are important to them. In future, modern web browsers will start rejecting unencrypted HTTP traffic (except localhost and intranet, ofc). All server owners should switch to HTTPS as soon as possible. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 01:56:47PM +0200, Vitaly Zaitsev via devel wrote: > On 19/10/2022 10:31, Neal Gompa wrote: > > HTTPS does not help with that. It's just a transport protocol. > > It will. All requests will be encrypted. ISP will only see server's > IP-address and its hostname (only if SNI is enabled). > > > Not in any meaningful way, and in most cases HTTPS makes mirrors slower too. > > No. All modern servers support AES-NI, so encryption doesn't slow down > servers. AES-NI does not provide infinite speed. Each CPU has a limit to how much data it can shuffle through AES-NI in a given timeframe. AES-NI may well be x10 faster than doing AES in software with generic instructions, but it still has a performance upper bound. In my previous testing of AES-NI for QEMU live migration, I was unable to saturate the max available NIC bandwidth available. It was massively better than not using AES-NI, but not encrypting at all was still faster by a significant degree. IOW, the impact of AES on server peformance will vary depending on CPU models, NIC models / network switches and whether other workloads are competing for CPU time. Admins need to decide what tradeoffs are important to them. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19/10/2022 10:31, Neal Gompa wrote: HTTPS does not help with that. It's just a transport protocol. It will. All requests will be encrypted. ISP will only see server's IP-address and its hostname (only if SNI is enabled). Not in any meaningful way, and in most cases HTTPS makes mirrors slower too. No. All modern servers support AES-NI, so encryption doesn't slow down servers. We don't have a "main mirror" for that to work. Then where do mirrors get information about updates? -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller wrote: > > I _very much_ appreciate all the work you and the other Rust SIG folks > (Igor and Zbyszek in particular but I'm sure others as well!) have put into > packaging rust apps and crates and all of the systems around that. I'll respond inline. > I fundamentally disagree with Kevin on a deep level about "entirely > useless", but ... find myself kind of agreeing about the "unpackagable" > part. I mean: clearly we've found a way, but I'm really not sure we're > providing a lot of _value_ in this approach, and I'm also not sure it's as > successful as it could be. We *do* provide value to both users *and* developers by doing things the way we do, but the benefits might not be obvious to people who don't know how (Rust) packaging works, and what we as package maintainers do. > There are three ways having things packaged in Fedora repos _can_ be > helpful: > > 1. End-user applications and tools > 2. Useful development environment > 3. As convenience for ourselves for building packages for #1 or #2 > > I am not discounting the value of #3 -- making a shared thing that we all > work on together is kind of the whole point, and the nicer we can make that > the better we can bring in more people, and those of us already here have a > lighter load and can work on the things we're most interested in. But > ultimately, we're doing it so we make a useful system for users. That means > the first two. This I can agree with :) > I'll start with the second: our system for Rust doesn't really do that. > Developers are going to use cargo and crates.io and we're not going to > convince them that they should do otherwise. (I don't expect anyone > disagrees with this.) This is true, and probably also not "fixable". We need to make some amount of non-upstreamable patches to some crates (most notably, removing Windows- or mac OS-specific dependencies, because we don't want to package those), but in some cases, these are "incompatible" changes, and Rust *developers* should not be targeting our downstream sources that have these differences with actual upstream sources. This is due to a limitation of how cargo handles target-specific dependencies - all dependencies that are *mentioned in any way* need to be *present* for it to resolve dependencies / enabled optional features / update its lockfile etc. But since we don't want to package bindings for Windows and mac OS system APIs, we need to actually patch them out, otherwise builds will fail. > We're doing okay with #1, but... I think #3 _even_ with all of the work in > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > engine and will probably have a few things it would be nice to package to > make available in Fedora Linux. I might not even mind maintaining Bevy > itself. Somebody actually already started packaging Bevy components - some packages are already approved and some are still pending review. Not sure what the progress has been there, but it's not *impossible*. > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > which is *amazing*. But... that still is hundreds that I'd have to add. And > mostly they are things I don't know _anything_ about. You must realize that this is an extreme case. For many Rust applications that people want to package for Fedora, the number of dependencies that are missing is rather small, *because* most popular libraries are already packaged. Bevy is a bit special, because it (presumably) pulls in lots of GPU / OpenGL / Vulkan related libraries, which we didn't need to package for anything else yet, and it's also split into dozens of small libraries itself, which can be painful to package, that is true. We might need to reconsider how to package projects like this. I'm pretty sure we could find a way to package them in a way that's compatible with how we're currently doing things but would be much less busywork. > *This is what open source winning looks like.* > > I remember a Byte magazine article from the 1990 (I just checked!) with the > title "There Is a Silver Bullet: The birth of interchangeable, reusable > software components will bring software into the information age". [1] > This was about the newly-hot idea of Object Oriented Programming. It was > very exciting. But, of course, that vision of the world did not happen. It > turns out proprietary software *can't* do this. > > But now we have it! I don't have to reinvent every basic wheel — but even > more than that, I do not have to be an expert in the intricacies of safe > concurrency to write an app that uses that under the hood. That's amazing! I > can do such powerful things from high-level interfaces and trust the > expertise of those who really understand the deep computer science some of > this requires. > > I am competent enough to write a silly toy game using Bevy. It might be good > enough
Amazon mirrors [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 10:50:19AM +0200, Sandro wrote: > >We don't have a "main mirror" for that to work. > > So, this has been looked into already? It definitely sounds like it > could help in sparsely served parts of the world at a reasonable > cost. Yes. I think this is basically just a matter of _doing_. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, Oct 13, 2022 at 03:46:49PM +0200, Fabio Valentini wrote: > > The dependency on LLVM is not even the worst issue in my eyes. LLVM is also > > used by other core projects, e.g., mesa, these days. > > > > The worst issue I see with Rust is the way libraries are "packaged", which > > just implies installing source code and recompiling that source code for > > every single application. (And as a result, the output obviously gets > > statically linked into the application, with all the drawbacks of static > > linking.) I consider a language with no usable shared library support to be > > entirely unpackageable and hence entirely useless. > > Do you have suggestions for improving this situation? I think we're > pretty close to doing the best we can with packaging Rust projects, > given the current limitations of the language (i.e. the support for > building "true shared Rust libraries" is still very limited and > unstable; but that might improve in the future). I _very much_ appreciate all the work you and the other Rust SIG folks (Igor and Zbyszek in particular but I'm sure others as well!) have put into packaging rust apps and crates and all of the systems around that. I fundamentally disagree with Kevin on a deep level about "entirely useless", but ... find myself kind of agreeing about the "unpackagable" part. I mean: clearly we've found a way, but I'm really not sure we're providing a lot of _value_ in this approach, and I'm also not sure it's as successful as it could be. There are three ways having things packaged in Fedora repos _can_ be helpful: 1. End-user applications and tools 2. Useful development environment 3. As convenience for ourselves for building packages for #1 or #2 I am not discounting the value of #3 -- making a shared thing that we all work on together is kind of the whole point, and the nicer we can make that the better we can bring in more people, and those of us already here have a lighter load and can work on the things we're most interested in. But ultimately, we're doing it so we make a useful system for users. That means the first two. I'll start with the second: our system for Rust doesn't really do that. Developers are going to use cargo and crates.io and we're not going to convince them that they should do otherwise. (I don't expect anyone disagrees with this.) We're doing okay with #1, but... I think #3 _even_ with all of the work in Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game engine and will probably have a few things it would be nice to package to make available in Fedora Linux. I might not even mind maintaining Bevy itself. But running `cargo fetch` with a clean cache pulls down *390* crates. Of these, it looks like 199 (!) are already packaged as rust-[crate]-devel, which is *amazing*. But... that still is hundreds that I'd have to add. And mostly they are things I don't know _anything_ about. *This is what open source winning looks like.* I remember a Byte magazine article from the 1990 (I just checked!) with the title "There Is a Silver Bullet: The birth of interchangeable, reusable software components will bring software into the information age". [1] This was about the newly-hot idea of Object Oriented Programming. It was very exciting. But, of course, that vision of the world did not happen. It turns out proprietary software *can't* do this. But now we have it! I don't have to reinvent every basic wheel — but even more than that, I do not have to be an expert in the intricacies of safe concurrency to write an app that uses that under the hood. That's amazing! I can do such powerful things from high-level interfaces and trust the expertise of those who really understand the deep computer science some of this requires. I am competent enough to write a silly toy game using Bevy. It might be good enough that others will enjoy it. *I am not competent to maintain many of these dependencies.* I don't even know what most of them DO. "anyhow"? "bytemuck"? Worse, many of the Bevy deps are specified with exact versions. Maybe I could make the package work with the packaged versions, but ... that requires deep expertise and even then might lead to unexpected behavior and has a high chance of putting me at odds with both the engine upstream and any other games which use it. The packaging guidelines say that I SHOULD create patches to update to latest versions of dependencies, and that I should further convince the upstream to take them. Candidly, that seems like a waste of everone's time. The guidelines provide for creating compat packages, but that means 1) the existing shared work is less useful, 2) requires even more extra steps, and 3) even without reviews for compat has extra administrative overhead. So, going back to Kevin's point: it _does_ feel like this is unpackagable. But that's because the barrier to participation seems too high. It's not because it's statically-linked binaries [2] can't or shouldn't exist in Fedora
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19-10-2022 10:31, Neal Gompa wrote: On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel wrote: On 19/10/2022 09:48, Peter Robinson wrote: Sure but as mentioned it's public data, and the modification, and I covered that in my reply, would be picked up by the other mechanisms. They can collect a lot of sensitive information: your IP, Fedora version, packages version, etc. This can help with recon for attackers. HTTPS does not help with that. It's just a transport protocol. Security is covered, as probinson already mentioned, by internal verification. There isn't actually that many mirrors left do we really want to reduce the number more for end users for no actual improvement in security? It will improve privacy at least. Not in any meaningful way, and in most cases HTTPS makes mirrors slower too. If privacy is your concern, which is something the user has to decide, there's the torproxy plugin for DNF. I think that would protect your privacy and allow use of plain http - win/win. Ultimately bandwidth is expensive in a lot of parts of the world for commercial entities to provide, that's why there's mirrors. Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can switch to a CDN too? We don't have a "main mirror" for that to work. So, this has been looked into already? It definitely sounds like it could help in sparsely served parts of the world at a reasonable cost. -- Sandro (just throwing in my $0.02) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel wrote: > > On 19/10/2022 09:48, Peter Robinson wrote: > > Sure but as mentioned it's public data, and the modification, and I > > covered that in my reply, would be picked up by the other mechanisms. > > They can collect a lot of sensitive information: your IP, Fedora > version, packages version, etc. This can help with recon for attackers. > HTTPS does not help with that. It's just a transport protocol. > > There isn't actually that many mirrors left do we > > really want to reduce the number more for end users for no actual > > improvement in security? > > It will improve privacy at least. > Not in any meaningful way, and in most cases HTTPS makes mirrors slower too. > > Ultimately bandwidth is expensive in a lot of > > parts of the world for commercial entities to provide, that's why > > there's mirrors. > > Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can > switch to a CDN too? > We don't have a "main mirror" for that to work. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19/10/2022 09:48, Peter Robinson wrote: Sure but as mentioned it's public data, and the modification, and I covered that in my reply, would be picked up by the other mechanisms. They can collect a lot of sensitive information: your IP, Fedora version, packages version, etc. This can help with recon for attackers. There isn't actually that many mirrors left do we really want to reduce the number more for end users for no actual improvement in security? It will improve privacy at least. Ultimately bandwidth is expensive in a lot of parts of the world for commercial entities to provide, that's why there's mirrors. Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can switch to a CDN too? -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 8:40 AM Vitaly Zaitsev via devel wrote: > > On 19/10/2022 09:33, Peter Robinson wrote: > > Why are they insecure? This is public open data, not banking data, > > where the data being downloaded is verifiable by the rpm signatures > > and signing keys. > > ISPs or anyone on the the same network can view, intercept or even > modify HTTP/rsync traffic. Sure but as mentioned it's public data, and the modification, and I covered that in my reply, would be picked up by the other mechanisms. > > The flip side is we remove the non https mirrors and the mirror system > > slows down considerably, or in some cases is even unavailable, for > > users which in IMO is more of a problem because people then really do > > have insecure systems. > > We can give mirror owners some time (for example, a month) to migrate > their mirrors to HTTPS. The ones that actually care have already migrated, they've been actively encouraged to do this previously and haven't, see point above about a lack of mirrors. Ultimately bandwidth is expensive in a lot of parts of the world for commercial entities to provide, that's why there's mirrors. There isn't actually that many mirrors left do we really want to reduce the number more for end users for no actual improvement in security? While where you are there may be lots of mirrors there are places where there are very few and having less will likely make Fedora unusable, I had a report on the arm channel just this week of problems with lack of available mirrors just this week. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 19/10/2022 09:33, Peter Robinson wrote: Why are they insecure? This is public open data, not banking data, where the data being downloaded is verifiable by the rpm signatures and signing keys. ISPs or anyone on the the same network can view, intercept or even modify HTTP/rsync traffic. The flip side is we remove the non https mirrors and the mirror system slows down considerably, or in some cases is even unavailable, for users which in IMO is more of a problem because people then really do have insecure systems. We can give mirror owners some time (for example, a month) to migrate their mirrors to HTTPS. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Wed, Oct 19, 2022 at 8:17 AM Vitaly Zaitsev via devel wrote: > > On 13/10/2022 15:46, Neal Gompa wrote: > > Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. > > I think such insecure mirrors should be removed from metalink. Why are they insecure? This is public open data, not banking data, where the data being downloaded is verifiable by the rpm signatures and signing keys. There is of course a chance an MitM attacker could work out you're installing sendmail or tuxracer but there's other mechanisms in place to ensure it's not compromisable mid stream. The flip side is we remove the non https mirrors and the mirror system slows down considerably, or in some cases is even unavailable, for users which in IMO is more of a problem because people then really do have insecure systems. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 13/10/2022 15:46, Neal Gompa wrote: Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. I think such insecure mirrors should be removed from metalink. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
http[s] mirrors [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, Oct 13, 2022 at 03:57:41PM +0200, Kevin Kofler via devel wrote: > > Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. > I would say that those mirrors ought to be kicked out of the mirror list > immediately. There are also a lot of rsync mirrors. I don't think any of them are using rsync-ssl. I think "kicked out" is a bit harsh -- but we should definitely suggest it. And I think we should also do the metadata signing as soon as practical... defense in depth and all that. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Fri, 14 Oct 2022 18:28:01 +0200, Simo Sorce wrote: > At this time, as far as I know, there is no OpenPGP work of any kind on > supporting PQC algorithms. The German BSI contracted MTG AG to design and implement PQC for OpenPGP. They presented their work at IETF 113, and at the OpenPGP email summit this past May. https://datatracker.ietf.org/meeting/113/materials/agenda-113-openpgp-01 > Furthermore the way we use signatures in RPM > really has no resemblance to the scenarios OpenPGP was built for. Could you elaborate on this, please. Thanks, Neal ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Fri, 2022-10-14 at 10:14 +0300, Panu Matilainen wrote: > On 10/13/22 15:14, Neal Gompa wrote: > > On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen wrote: > > > > > > On 10/13/22 10:53, Neal Gompa wrote: > > > > On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen > > > > wrote: > > > > > > > > > > On 10/13/22 07:18, Kevin Kofler via devel wrote: > > > > > > > For the last 20 years or so, RPM has used a home-grown OpenPGP > > > > > > > parser > > > > > > > for dealing with keys and signatures. That parser is rather > > > > > > > infamous > > > > > > > for its limitations and flaws, and especially in recent years has > > > > > > > proven a significant burden to RPM development. In order to > > > > > > > improve > > > > > > > security and free developer resources for dealing with RPM's "core > > > > > > > business" instead, RPM upstream is in the process of deprecating > > > > > > > the > > > > > > > internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > > > > > > > based solution written in Rust. > > > > > > > > > > > > Why are you using a new library written in Rust? Can you not use > > > > > > one of the > > > > > > existing mature C implementations of OpenPGP? gpgme maybe? > > > > > > > > > > Had there been such an option, we would've switched over years ago. > > > > > Gpgme is based around calling and communicating with an external gpg > > > > > process, which is a setup you do NOT want in the rpm context where > > > > > chroots come and go etc. Also, rpm requires access to the "low-level" > > > > > digests-in-progress because it calculates multiple things on a single > > > > > read. > > > > > > > > > > > > > The real problem is that all other OpenPGP implementations died out > > > > because of GnuPG. And then GnuPG made the choice to force an > > > > inter-process model. > > > > > > > > At work, I deal with this on Debian systems, which do indeed use GnuPG > > > > for this. It creates a lot of problems, especially for building images > > > > and dealing with chroots, which is why in the context of RPM PGP > > > > upstream, I never pushed to consider it. > > > > > > > > The most serious problems with PackageKit memory leaks and hangs are > > > > actually caused by GnuPG, since DNF uses it for some GPG stuff because > > > > there's no API for using RPM's PGP capabilities. There's no fix unless > > > > the RPM and DNF teams agree on an API and build it out so that libdnf > > > > and librepo no longer need to use GnuPG through gpgme anymore. > > > > > > > > This is also the underlying reason why Red Hat has resisted > > > > implementing signed repository metadata and enforcing it by default. > > > > Of course this is a bit of a catch-22 as well, as there's no > > > > motivation to find a solution because neither Fedora nor RHEL offer > > > > signed repository metadata despite repeated calls for it over the past > > > > decade. > > > > > > > > Now, don't get me wrong: I'm personally extremely unhappy about having > > > > to depend on the Sequoia stack for RPM PGP. I have a strong distaste > > > > for the Rust community ecosystem these days, and I don't love the idea > > > > of having to have LLVM in the core bootstrap chain (hopefully gcc-rs > > > > will be in place soon enough!). But we literally don't have any other > > > > options. GnuPG/GPGME is out of the question for the reasons Panu and I > > > > stated, and NeoPG died several years ago. There are no C/C++ libraries > > > > for OpenPGP verification. > > > > > > There's RNP (in C++, used by Thunderbird at least), but alas that > > > doesn't expose the digest-in-progress either. So at least in it's > > > current form, it's not an option for rpm. Also, the API appears to have > > > all manner of quirks and gotchas that aren't welcome in a > > > security-critical piece. > > > > > > > Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend > > and at least the verification API (which is what RPM would use) seems > > to be getting love lately. Insofar as quirks and gotchas, I'm not a > > great judge of that at the moment, but I don't think it could be worse > > than what we have with GnuPG. > > > > The missing "digest-in-progress" thing is an issue, I guess. Have we > > raised the issue with them about it? > > No, because beneath the surface it didn't seem all that appetizing > afterall. See > https://sequoia-pgp.org/blog/2021/05/06/202105-thunderbird-rnp-and-the-importance-of-a-good-api/ > > for an example (even bearing in mind that a blog post from a competitor > may be necessarily a little opionated), but its reputation/security > record doesn't seem that great. > > It's something to keep an eye on of course, it would be good to have > alternatives available. > > > > > > As for bootstrap, there will always (have to) be a way to build rpm > > > without depending on Rust. Even if that meant no signature verification > > > support in such a configuration. > > > > > > > Eck. What about the x509
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 15:14, Neal Gompa wrote: On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen wrote: On 10/13/22 10:53, Neal Gompa wrote: On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: On 10/13/22 07:18, Kevin Kofler via devel wrote: For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? Had there been such an option, we would've switched over years ago. Gpgme is based around calling and communicating with an external gpg process, which is a setup you do NOT want in the rpm context where chroots come and go etc. Also, rpm requires access to the "low-level" digests-in-progress because it calculates multiple things on a single read. The real problem is that all other OpenPGP implementations died out because of GnuPG. And then GnuPG made the choice to force an inter-process model. At work, I deal with this on Debian systems, which do indeed use GnuPG for this. It creates a lot of problems, especially for building images and dealing with chroots, which is why in the context of RPM PGP upstream, I never pushed to consider it. The most serious problems with PackageKit memory leaks and hangs are actually caused by GnuPG, since DNF uses it for some GPG stuff because there's no API for using RPM's PGP capabilities. There's no fix unless the RPM and DNF teams agree on an API and build it out so that libdnf and librepo no longer need to use GnuPG through gpgme anymore. This is also the underlying reason why Red Hat has resisted implementing signed repository metadata and enforcing it by default. Of course this is a bit of a catch-22 as well, as there's no motivation to find a solution because neither Fedora nor RHEL offer signed repository metadata despite repeated calls for it over the past decade. Now, don't get me wrong: I'm personally extremely unhappy about having to depend on the Sequoia stack for RPM PGP. I have a strong distaste for the Rust community ecosystem these days, and I don't love the idea of having to have LLVM in the core bootstrap chain (hopefully gcc-rs will be in place soon enough!). But we literally don't have any other options. GnuPG/GPGME is out of the question for the reasons Panu and I stated, and NeoPG died several years ago. There are no C/C++ libraries for OpenPGP verification. There's RNP (in C++, used by Thunderbird at least), but alas that doesn't expose the digest-in-progress either. So at least in it's current form, it's not an option for rpm. Also, the API appears to have all manner of quirks and gotchas that aren't welcome in a security-critical piece. Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend and at least the verification API (which is what RPM would use) seems to be getting love lately. Insofar as quirks and gotchas, I'm not a great judge of that at the moment, but I don't think it could be worse than what we have with GnuPG. The missing "digest-in-progress" thing is an issue, I guess. Have we raised the issue with them about it? No, because beneath the surface it didn't seem all that appetizing afterall. See https://sequoia-pgp.org/blog/2021/05/06/202105-thunderbird-rnp-and-the-importance-of-a-good-api/ for an example (even bearing in mind that a blog post from a competitor may be necessarily a little opionated), but its reputation/security record doesn't seem that great. It's something to keep an eye on of course, it would be good to have alternatives available. As for bootstrap, there will always (have to) be a way to build rpm without depending on Rust. Even if that meant no signature verification support in such a configuration. Eck. What about the x509 based stuff we were talking about last year? All the crypto backends RPM supports now support that stuff out of the box. Embedded x509 signatures (certs) to replace GPG signatures could work as an alternative. x509 seems to be loathed even more than PGP, so it doesn't seem that appetizing either. If someone with known crypto-clue would send patches they would be looked at, *I* have no prejudice about x509 because I also have no clue about it. Ditto for Signify, which often gets brought up in these discussions. And yet, that all is largely irrelevant for the subject at hand: no matter what, rpm will need OpenPGP support for years to come because existing content will need to remain usable for a long, long time. - Panu -
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 17:31, Neal Gompa wrote: On Thu, Oct 13, 2022 at 9:58 AM Kevin Kofler via devel wrote: Neal Gompa wrote: No, because when you do things like mirror repositories (especially for private mirrors), that signature is the only way to verify the integrity. HTTPS is only transport encryption from a particular connection. HTTPS protects against a MITM on the connection introducing invalid repository contents, which I would assume to be the biggest threat here. But sure, it by design does not guarantee that the data on the remote end is valid to begin with. Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. I would say that those mirrors ought to be kicked out of the mirror list immediately. With Let's Encrypt having been available for years, there is really no excuse for not offering HTTPS. Assuming you own a domain name (which I assume to already be the case for all mirrors), setting up HTTPS with Let's Encrypt does not cost you a dime. Even if you are a commercial entity. I'm not going to get into this too much, but suffice to say, it's not universally accessible as a CA. And using Let's Encrypt for private mirrors is sufficiently painful that I wouldn't recommend it. Well, it might still be worthwhile to split out RPM's OpenPGP implementation into its own project and allow people to contribute to it. The worst that can happen is that nothing changes. If that implementation is really as awfully broken as Panu is saying, I do not think that that would be of much use, unfortunately. There have been attempts to fix things, but Panu doesn't feel qualified to review the changes. That doesn't mean someone else who would be willing to do so couldn't. But because of... reasons, as long as it's in the RPM codebase, it's unlikely someone else will be trusted enough to do those reviews. This misses the point by a mile. Parsing OpenPGP is not rpm's "core business", by any stretch of imagination. As such, I refuse to let it take up any significant portion of rpm development resources. The parser has been largely dormant in the codebase for like 20 years, except for a couple of cleanup rounds when people have pointed out random flaws and vulnerabilities. Last year when people started to really look into it, it's become obvious that the effort to fix & review it all is just nowhere near justifiable. Consider this the RPM burning platform memo if you like ;) - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 19:35, Demi Marie Obenour wrote: On 10/13/22 04:23, Panu Matilainen wrote: On 10/13/22 10:53, Neal Gompa wrote: On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: On 10/13/22 07:18, Kevin Kofler via devel wrote: For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? Had there been such an option, we would've switched over years ago. Gpgme is based around calling and communicating with an external gpg process, which is a setup you do NOT want in the rpm context where chroots come and go etc. Also, rpm requires access to the "low-level" digests-in-progress because it calculates multiple things on a single read. The real problem is that all other OpenPGP implementations died out because of GnuPG. And then GnuPG made the choice to force an inter-process model. At work, I deal with this on Debian systems, which do indeed use GnuPG for this. It creates a lot of problems, especially for building images and dealing with chroots, which is why in the context of RPM PGP upstream, I never pushed to consider it. The most serious problems with PackageKit memory leaks and hangs are actually caused by GnuPG, since DNF uses it for some GPG stuff because there's no API for using RPM's PGP capabilities. There's no fix unless the RPM and DNF teams agree on an API and build it out so that libdnf and librepo no longer need to use GnuPG through gpgme anymore. This is also the underlying reason why Red Hat has resisted implementing signed repository metadata and enforcing it by default. Of course this is a bit of a catch-22 as well, as there's no motivation to find a solution because neither Fedora nor RHEL offer signed repository metadata despite repeated calls for it over the past decade. Now, don't get me wrong: I'm personally extremely unhappy about having to depend on the Sequoia stack for RPM PGP. I have a strong distaste for the Rust community ecosystem these days, and I don't love the idea of having to have LLVM in the core bootstrap chain (hopefully gcc-rs will be in place soon enough!). But we literally don't have any other options. GnuPG/GPGME is out of the question for the reasons Panu and I stated, and NeoPG died several years ago. There are no C/C++ libraries for OpenPGP verification. There's RNP (in C++, used by Thunderbird at least), but alas that doesn't expose the digest-in-progress either. So at least in it's current form, it's not an option for rpm. Also, the API appears to have all manner of quirks and gotchas that aren't welcome in a security-critical piece. As for bootstrap, there will always (have to) be a way to build rpm without depending on Rust. Even if that meant no signature verification support in such a configuration. RPM could keep its own internal parser for bootstrap only. Bootstrap does not need support for revocation, etc. I very intentionally left the possibilities open here. Rust-free bootstrap is a hard requirement for rpm, how that is achieved is a whole other topic. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Let's Encrypt also supports the dns-01 challenge[1] that doesn't require any publicly available IPs. Using dns verification is required to obtain a Let's Encrypt wildcard certificate. While I tend to prefer using the dns-01 challenge approach when possible, not all DNS providers have made it easy to accomplish (the certbot folk have implementations for a number of the major DNS providers, and one can sometimes find other 3rd party code for others, but it can still be hard to setup and use, which means just enough additional impedance that sometimes people will choose not to use it; I can't blame them, as sometimes free has a higher cost than having someone else order the cert from one of the non-free CAs). fwiw, IME, one of the lowest-friction dns-challenge tools I've recommended, and see actually getting used by clients, is acme.sh, https://github.com/acmesh-official/acme.sh#user-content-8-automatic-dns-api-integration which supports 'most' of the big dns apis, https://github.com/acmesh-official/acme.sh/wiki/dnsapi and, when not an option, is fairly trivial to use manually https://github.com/acmesh-official/acme.sh#user-content-9-use-dns-manual-mode https://github.com/acmesh-official/acme.sh/wiki/dns-manual-mode all of this with no cumbersome python, go, webserver, etc deps. just bash shell. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 4:57 PM Maxwell G via devel wrote: > Let's Encrypt also supports the dns-01 challenge[1] that doesn't require > any publicly available IPs. Using dns verification is required to obtain a > Let's Encrypt wildcard certificate. While I tend to prefer using the dns-01 challenge approach when possible, not all DNS providers have made it easy to accomplish (the certbot folk have implementations for a number of the major DNS providers, and one can sometimes find other 3rd party code for others, but it can still be hard to setup and use, which means just enough additional impedance that sometimes people will choose not to use it; I can't blame them, as sometimes free has a higher cost than having someone else order the cert from one of the non-free CAs). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 11:28, Demi Marie Obenour wrote: systemd (yes, systemd) is considering using Rust, though it has not yet started including it, and there is already Rust code in Mesa IIRC. Don't forget the Python 'cryptography' package... also a Rust user. It's here to stay, at least for the foreseeable future. -- Kevin P. Fleming He/Him/His Principal Program Manager, RHEL Red Hat US/Eastern Time Zone ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu Oct 13, 2022 at 17:12 +0200, Kevin Kofler via devel wrote: > > And using Let's Encrypt for private mirrors is sufficiently painful that I > > wouldn't recommend it. > > Set up a subdomain like vpn.example.com, point it to the public IP, then > configure the VPN's internal DNS to resolve vpn.example.com to the VPN- > internal address instead, the /etc/hosts on the VPN server itself to resolve > it to 127.0.0.1, and the mirror server on port 443 (whereas port 80 is > reserved for certbot's builtin temporary (and world-readable) webserver with > the http-01 challenge) to accept connections only from the VPN and from > localhost and to use the Let's Encrypt certificate. Been there, done that > (not for a repository mirror though, my employer is small enough for that > not to be worthwhile). I assume that this approach should also work for a > physical LAN in lieu of the VPN. Let's Encrypt also supports the dns-01 challenge[1] that doesn't require any publicly available IPs. Using dns verification is required to obtain a Let's Encrypt wildcard certificate. [1]: https://letsencrypt.org/docs/challenge-types/#dns-01-challenge -- Maxwell G (@gotmax23) Pronouns: He/Him/His signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 04:23, Panu Matilainen wrote: > On 10/13/22 10:53, Neal Gompa wrote: >> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: >>> >>> On 10/13/22 07:18, Kevin Kofler via devel wrote: > For the last 20 years or so, RPM has used a home-grown OpenPGP parser > for dealing with keys and signatures. That parser is rather infamous > for its limitations and flaws, and especially in recent years has > proven a significant burden to RPM development. In order to improve > security and free developer resources for dealing with RPM's "core > business" instead, RPM upstream is in the process of deprecating the > internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? >>> >>> Had there been such an option, we would've switched over years ago. >>> Gpgme is based around calling and communicating with an external gpg >>> process, which is a setup you do NOT want in the rpm context where >>> chroots come and go etc. Also, rpm requires access to the "low-level" >>> digests-in-progress because it calculates multiple things on a single read. >>> >> >> The real problem is that all other OpenPGP implementations died out >> because of GnuPG. And then GnuPG made the choice to force an >> inter-process model. >> >> At work, I deal with this on Debian systems, which do indeed use GnuPG >> for this. It creates a lot of problems, especially for building images >> and dealing with chroots, which is why in the context of RPM PGP >> upstream, I never pushed to consider it. >> >> The most serious problems with PackageKit memory leaks and hangs are >> actually caused by GnuPG, since DNF uses it for some GPG stuff because >> there's no API for using RPM's PGP capabilities. There's no fix unless >> the RPM and DNF teams agree on an API and build it out so that libdnf >> and librepo no longer need to use GnuPG through gpgme anymore. >> >> This is also the underlying reason why Red Hat has resisted >> implementing signed repository metadata and enforcing it by default. >> Of course this is a bit of a catch-22 as well, as there's no >> motivation to find a solution because neither Fedora nor RHEL offer >> signed repository metadata despite repeated calls for it over the past >> decade. >> >> Now, don't get me wrong: I'm personally extremely unhappy about having >> to depend on the Sequoia stack for RPM PGP. I have a strong distaste >> for the Rust community ecosystem these days, and I don't love the idea >> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs >> will be in place soon enough!). But we literally don't have any other >> options. GnuPG/GPGME is out of the question for the reasons Panu and I >> stated, and NeoPG died several years ago. There are no C/C++ libraries >> for OpenPGP verification. > > There's RNP (in C++, used by Thunderbird at least), but alas that > doesn't expose the digest-in-progress either. So at least in it's > current form, it's not an option for rpm. Also, the API appears to have > all manner of quirks and gotchas that aren't welcome in a > security-critical piece. > > As for bootstrap, there will always (have to) be a way to build rpm > without depending on Rust. Even if that meant no signature verification > support in such a configuration. RPM could keep its own internal parser for bootstrap only. Bootstrap does not need support for revocation, etc. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 11:18, Kevin Kofler via devel wrote: > Fabio Valentini wrote: >> Do you have suggestions for improving this situation? I think we're >> pretty close to doing the best we can with packaging Rust projects, >> given the current limitations of the language (i.e. the support for >> building "true shared Rust libraries" is still very limited and >> unstable; but that might improve in the future). > > Building true shared libraries is pretty much a prerequisite for any sane > form of packaging. I strongly recommend that you bring this up on the rust-lang Zulip (https://rust-lang.zulipchat.com) and see if you can find people who can improve this. >> Additionally, the way the Sequoia GPG backend for RPM is implemented, >> it's actually shipped as a standard shared library with a C ABI and >> accompanying C headers (which are binary compatible with the existing >> in-house GPG backend code in RPM). No Rust code will be statically >> linked into /usr/bin/rpm. > > That at least makes sense. (Though I assume that that C-style .so still > internally depends on a whole bunch of Rust crates that are statically > compiled and linked into the .so, does it not?) It does. >> I'm sorry to disappoint you, but Rust is here to stay, whether you >> like it or not. >> For example, it's been voted the "most loved" and "most wanted" >> language for a few years in a row in Stack Overflow's surveys: >> https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-wanted > > And according to the same statistics, the majority of developers runs > Windows, yet hopefully you do not propose that Fedora ship Windows… > > We need to ship what we can reasonably ship, not what the (relative) > majority of developers in the world (most of whom do not even run Fedora) > happens to prefer. There is now Rust code in the Linux kernel. It currently is not needed by any shipping driver, but that is not a matter of *if*, but of *when*. The Asahi GPU driver is being written in Rust, for example, so anyone building Linux for Apple Silicon will need CONFIG_RUST=y in the not-too-distant future. systemd (yes, systemd) is considering using Rust, though it has not yet started including it, and there is already Rust code in Mesa IIRC. C and C++ are being moved away from because they have unfixable security problems. Memory unsafety, which is predominantly (though not exclusively) found in C and C++ codebases, accounts for well over half of all security vulnerabilities. Google tried to retrofit memory safety into C++ and failed, and the existing methods for making C memory-safe either have massive overheads (4x slowdown for a single-threaded application, worse for multi-threaded, IIRC) or require a completely unreasonable amount of developer effort for a volunteer project (formal methods). -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 11:38 AM Demi Marie Obenour wrote: > > On 10/13/22 08:14, Neal Gompa wrote: > > On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen wrote: > >> > >> On 10/13/22 10:53, Neal Gompa wrote: > >>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen > >>> wrote: > > On 10/13/22 07:18, Kevin Kofler via devel wrote: > >> For the last 20 years or so, RPM has used a home-grown OpenPGP parser > >> for dealing with keys and signatures. That parser is rather infamous > >> for its limitations and flaws, and especially in recent years has > >> proven a significant burden to RPM development. In order to improve > >> security and free developer resources for dealing with RPM's "core > >> business" instead, RPM upstream is in the process of deprecating the > >> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > >> based solution written in Rust. > > > > Why are you using a new library written in Rust? Can you not use one of > > the > > existing mature C implementations of OpenPGP? gpgme maybe? > > Had there been such an option, we would've switched over years ago. > Gpgme is based around calling and communicating with an external gpg > process, which is a setup you do NOT want in the rpm context where > chroots come and go etc. Also, rpm requires access to the "low-level" > digests-in-progress because it calculates multiple things on a single > read. > > >>> > >>> The real problem is that all other OpenPGP implementations died out > >>> because of GnuPG. And then GnuPG made the choice to force an > >>> inter-process model. > >>> > >>> At work, I deal with this on Debian systems, which do indeed use GnuPG > >>> for this. It creates a lot of problems, especially for building images > >>> and dealing with chroots, which is why in the context of RPM PGP > >>> upstream, I never pushed to consider it. > >>> > >>> The most serious problems with PackageKit memory leaks and hangs are > >>> actually caused by GnuPG, since DNF uses it for some GPG stuff because > >>> there's no API for using RPM's PGP capabilities. There's no fix unless > >>> the RPM and DNF teams agree on an API and build it out so that libdnf > >>> and librepo no longer need to use GnuPG through gpgme anymore. > >>> > >>> This is also the underlying reason why Red Hat has resisted > >>> implementing signed repository metadata and enforcing it by default. > >>> Of course this is a bit of a catch-22 as well, as there's no > >>> motivation to find a solution because neither Fedora nor RHEL offer > >>> signed repository metadata despite repeated calls for it over the past > >>> decade. > >>> > >>> Now, don't get me wrong: I'm personally extremely unhappy about having > >>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste > >>> for the Rust community ecosystem these days, and I don't love the idea > >>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs > >>> will be in place soon enough!). But we literally don't have any other > >>> options. GnuPG/GPGME is out of the question for the reasons Panu and I > >>> stated, and NeoPG died several years ago. There are no C/C++ libraries > >>> for OpenPGP verification. > >> > >> There's RNP (in C++, used by Thunderbird at least), but alas that > >> doesn't expose the digest-in-progress either. So at least in it's > >> current form, it's not an option for rpm. Also, the API appears to have > >> all manner of quirks and gotchas that aren't welcome in a > >> security-critical piece. > >> > > > > Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend > > and at least the verification API (which is what RPM would use) seems > > to be getting love lately. Insofar as quirks and gotchas, I'm not a > > great judge of that at the moment, but I don't think it could be worse > > than what we have with GnuPG. > > > > The missing "digest-in-progress" thing is an issue, I guess. Have we > > raised the issue with them about it? > > > >> As for bootstrap, there will always (have to) be a way to build rpm > >> without depending on Rust. Even if that meant no signature verification > >> support in such a configuration. > >> > > > > Eck. What about the x509 based stuff we were talking about last year? > > All the crypto backends RPM supports now support that stuff out of the > > box. > > > > Embedded x509 signatures (certs) to replace GPG signatures could work > > as an alternative. > > OpenPGP is not great, but X.509 is an absolute disaster in comparison. But there are more implementations of the latter. I'll take that. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 08:14, Neal Gompa wrote: > On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen wrote: >> >> On 10/13/22 10:53, Neal Gompa wrote: >>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: On 10/13/22 07:18, Kevin Kofler via devel wrote: >> For the last 20 years or so, RPM has used a home-grown OpenPGP parser >> for dealing with keys and signatures. That parser is rather infamous >> for its limitations and flaws, and especially in recent years has >> proven a significant burden to RPM development. In order to improve >> security and free developer resources for dealing with RPM's "core >> business" instead, RPM upstream is in the process of deprecating the >> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] >> based solution written in Rust. > > Why are you using a new library written in Rust? Can you not use one of > the > existing mature C implementations of OpenPGP? gpgme maybe? Had there been such an option, we would've switched over years ago. Gpgme is based around calling and communicating with an external gpg process, which is a setup you do NOT want in the rpm context where chroots come and go etc. Also, rpm requires access to the "low-level" digests-in-progress because it calculates multiple things on a single read. >>> >>> The real problem is that all other OpenPGP implementations died out >>> because of GnuPG. And then GnuPG made the choice to force an >>> inter-process model. >>> >>> At work, I deal with this on Debian systems, which do indeed use GnuPG >>> for this. It creates a lot of problems, especially for building images >>> and dealing with chroots, which is why in the context of RPM PGP >>> upstream, I never pushed to consider it. >>> >>> The most serious problems with PackageKit memory leaks and hangs are >>> actually caused by GnuPG, since DNF uses it for some GPG stuff because >>> there's no API for using RPM's PGP capabilities. There's no fix unless >>> the RPM and DNF teams agree on an API and build it out so that libdnf >>> and librepo no longer need to use GnuPG through gpgme anymore. >>> >>> This is also the underlying reason why Red Hat has resisted >>> implementing signed repository metadata and enforcing it by default. >>> Of course this is a bit of a catch-22 as well, as there's no >>> motivation to find a solution because neither Fedora nor RHEL offer >>> signed repository metadata despite repeated calls for it over the past >>> decade. >>> >>> Now, don't get me wrong: I'm personally extremely unhappy about having >>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste >>> for the Rust community ecosystem these days, and I don't love the idea >>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs >>> will be in place soon enough!). But we literally don't have any other >>> options. GnuPG/GPGME is out of the question for the reasons Panu and I >>> stated, and NeoPG died several years ago. There are no C/C++ libraries >>> for OpenPGP verification. >> >> There's RNP (in C++, used by Thunderbird at least), but alas that >> doesn't expose the digest-in-progress either. So at least in it's >> current form, it's not an option for rpm. Also, the API appears to have >> all manner of quirks and gotchas that aren't welcome in a >> security-critical piece. >> > > Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend > and at least the verification API (which is what RPM would use) seems > to be getting love lately. Insofar as quirks and gotchas, I'm not a > great judge of that at the moment, but I don't think it could be worse > than what we have with GnuPG. > > The missing "digest-in-progress" thing is an issue, I guess. Have we > raised the issue with them about it? > >> As for bootstrap, there will always (have to) be a way to build rpm >> without depending on Rust. Even if that meant no signature verification >> support in such a configuration. >> > > Eck. What about the x509 based stuff we were talking about last year? > All the crypto backends RPM supports now support that stuff out of the > box. > > Embedded x509 signatures (certs) to replace GPG signatures could work > as an alternative. OpenPGP is not great, but X.509 is an absolute disaster in comparison. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 3:13 PM Kevin Kofler via devel wrote: > > Neal Gompa wrote: > > I'm not going to get into this too much, but suffice to say, it's not > > universally accessible as a CA. > > I would very much be interested in those details though. As I recall, the current LE root certificate is not (and in some cases may not be able to be) installed in all clients. When the cross signing cert (from a well known/trusted CA) expired a year ago a number of clients suddenly found themselves getting errors. It is easy to say "install new root certs", but that is not actually easy to do in all cases. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Fabio Valentini wrote: > Do you have suggestions for improving this situation? I think we're > pretty close to doing the best we can with packaging Rust projects, > given the current limitations of the language (i.e. the support for > building "true shared Rust libraries" is still very limited and > unstable; but that might improve in the future). Building true shared libraries is pretty much a prerequisite for any sane form of packaging. > Additionally, the way the Sequoia GPG backend for RPM is implemented, > it's actually shipped as a standard shared library with a C ABI and > accompanying C headers (which are binary compatible with the existing > in-house GPG backend code in RPM). No Rust code will be statically > linked into /usr/bin/rpm. That at least makes sense. (Though I assume that that C-style .so still internally depends on a whole bunch of Rust crates that are statically compiled and linked into the .so, does it not?) > I'm sorry to disappoint you, but Rust is here to stay, whether you > like it or not. > For example, it's been voted the "most loved" and "most wanted" > language for a few years in a row in Stack Overflow's surveys: > https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-wanted And according to the same statistics, the majority of developers runs Windows, yet hopefully you do not propose that Fedora ship Windows… We need to ship what we can reasonably ship, not what the (relative) majority of developers in the world (most of whom do not even run Fedora) happens to prefer. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Neal Gompa wrote: > I'm not going to get into this too much, but suffice to say, it's not > universally accessible as a CA. I would very much be interested in those details though. I do not see anybody being excluded from Let's Encrypt, not even countries under US embargo (e.g., over 30 sites in Iran are apparently using it successfully). > And using Let's Encrypt for private mirrors is sufficiently painful that I > wouldn't recommend it. Set up a subdomain like vpn.example.com, point it to the public IP, then configure the VPN's internal DNS to resolve vpn.example.com to the VPN- internal address instead, the /etc/hosts on the VPN server itself to resolve it to 127.0.0.1, and the mirror server on port 443 (whereas port 80 is reserved for certbot's builtin temporary (and world-readable) webserver with the http-01 challenge) to accept connections only from the VPN and from localhost and to use the Let's Encrypt certificate. Been there, done that (not for a repository mirror though, my employer is small enough for that not to be worthwhile). I assume that this approach should also work for a physical LAN in lieu of the VPN. > There have been attempts to fix things, but Panu doesn't feel > qualified to review the changes. That doesn't mean someone else who > would be willing to do so couldn't. But because of... reasons, as long > as it's in the RPM codebase, it's unlikely someone else will be > trusted enough to do those reviews. I see. So splitting might be worthwhile then. Assuming someone will care enough to actually maintain the code. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 9:58 AM Kevin Kofler via devel wrote: > > Neal Gompa wrote: > > No, because when you do things like mirror repositories (especially > > for private mirrors), that signature is the only way to verify the > > integrity. HTTPS is only transport encryption from a particular > > connection. > > HTTPS protects against a MITM on the connection introducing invalid > repository contents, which I would assume to be the biggest threat here. But > sure, it by design does not guarantee that the data on the remote end is > valid to begin with. > > > Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. > > I would say that those mirrors ought to be kicked out of the mirror list > immediately. > > With Let's Encrypt having been available for years, there is really no > excuse for not offering HTTPS. Assuming you own a domain name (which I > assume to already be the case for all mirrors), setting up HTTPS with Let's > Encrypt does not cost you a dime. Even if you are a commercial entity. > I'm not going to get into this too much, but suffice to say, it's not universally accessible as a CA. And using Let's Encrypt for private mirrors is sufficiently painful that I wouldn't recommend it. > > Well, it might still be worthwhile to split out RPM's OpenPGP > > implementation into its own project and allow people to contribute to > > it. The worst that can happen is that nothing changes. > > If that implementation is really as awfully broken as Panu is saying, I do > not think that that would be of much use, unfortunately. > There have been attempts to fix things, but Panu doesn't feel qualified to review the changes. That doesn't mean someone else who would be willing to do so couldn't. But because of... reasons, as long as it's in the RPM codebase, it's unlikely someone else will be trusted enough to do those reviews. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Neal Gompa wrote: > No, because when you do things like mirror repositories (especially > for private mirrors), that signature is the only way to verify the > integrity. HTTPS is only transport encryption from a particular > connection. HTTPS protects against a MITM on the connection introducing invalid repository contents, which I would assume to be the biggest threat here. But sure, it by design does not guarantee that the data on the remote end is valid to begin with. > Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. I would say that those mirrors ought to be kicked out of the mirror list immediately. With Let's Encrypt having been available for years, there is really no excuse for not offering HTTPS. Assuming you own a domain name (which I assume to already be the case for all mirrors), setting up HTTPS with Let's Encrypt does not cost you a dime. Even if you are a commercial entity. > Well, it might still be worthwhile to split out RPM's OpenPGP > implementation into its own project and allow people to contribute to > it. The worst that can happen is that nothing changes. If that implementation is really as awfully broken as Panu is saying, I do not think that that would be of much use, unfortunately. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 3:31 PM Kevin Kofler via devel wrote: > > The dependency on LLVM is not even the worst issue in my eyes. LLVM is also > used by other core projects, e.g., mesa, these days. > > The worst issue I see with Rust is the way libraries are "packaged", which > just implies installing source code and recompiling that source code for > every single application. (And as a result, the output obviously gets > statically linked into the application, with all the drawbacks of static > linking.) I consider a language with no usable shared library support to be > entirely unpackageable and hence entirely useless. Do you have suggestions for improving this situation? I think we're pretty close to doing the best we can with packaging Rust projects, given the current limitations of the language (i.e. the support for building "true shared Rust libraries" is still very limited and unstable; but that might improve in the future). Additionally, the way the Sequoia GPG backend for RPM is implemented, it's actually shipped as a standard shared library with a C ABI and accompanying C headers (which are binary compatible with the existing in-house GPG backend code in RPM). No Rust code will be statically linked into /usr/bin/rpm. > And then of course there is the issue that it is yet another language with > yet another syntax (and an only partially C-like one, so the learning curve > is unnecessarily high), yet another library ecosystem, etc. C has been the > de facto lingua franca all this time, now we are back into a tower-of-babel > scenario with tons of programming languages, which will necessarily bloat > the core system over time. I'm sorry to disappoint you, but Rust is here to stay, whether you like it or not. For example, it's been voted the "most loved" and "most wanted" language for a few years in a row in Stack Overflow's surveys: https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-wanted So, do you have any actionable / constructive criticism for how we do Rust packaging in Fedora, or did you just want to post that "I hate Rust because it's not C"? Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 9:31 AM Kevin Kofler via devel wrote: > > Neal Gompa wrote: > > This is also the underlying reason why Red Hat has resisted > > implementing signed repository metadata and enforcing it by default. > > Of course this is a bit of a catch-22 as well, as there's no > > motivation to find a solution because neither Fedora nor RHEL offer > > signed repository metadata despite repeated calls for it over the past > > decade. > > Is signed repository metadata not basically moot now that pretty much all > the world has moved on from unencrypted HTTP to secure HTTPS? > No, because when you do things like mirror repositories (especially for private mirrors), that signature is the only way to verify the integrity. HTTPS is only transport encryption from a particular connection. Also, a ton of Fedora mirrors still don't use HTTPS for various reasons. > > Now, don't get me wrong: I'm personally extremely unhappy about having > > to depend on the Sequoia stack for RPM PGP. I have a strong distaste > > for the Rust community ecosystem these days, and I don't love the idea > > of having to have LLVM in the core bootstrap chain (hopefully gcc-rs > > will be in place soon enough!). > > The dependency on LLVM is not even the worst issue in my eyes. LLVM is also > used by other core projects, e.g., mesa, these days. > > The worst issue I see with Rust is the way libraries are "packaged", which > just implies installing source code and recompiling that source code for > every single application. (And as a result, the output obviously gets > statically linked into the application, with all the drawbacks of static > linking.) I consider a language with no usable shared library support to be > entirely unpackageable and hence entirely useless. > > And then of course there is the issue that it is yet another language with > yet another syntax (and an only partially C-like one, so the learning curve > is unnecessarily high), yet another library ecosystem, etc. C has been the > de facto lingua franca all this time, now we are back into a tower-of-babel > scenario with tons of programming languages, which will necessarily bloat > the core system over time. > > > So here we are, in a subpar situation created by bad tools because > > nobody cares enough about security anyway. > > Sounds like a mess indeed. > Well, it might still be worthwhile to split out RPM's OpenPGP implementation into its own project and allow people to contribute to it. The worst that can happen is that nothing changes. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Neal Gompa wrote: > This is also the underlying reason why Red Hat has resisted > implementing signed repository metadata and enforcing it by default. > Of course this is a bit of a catch-22 as well, as there's no > motivation to find a solution because neither Fedora nor RHEL offer > signed repository metadata despite repeated calls for it over the past > decade. Is signed repository metadata not basically moot now that pretty much all the world has moved on from unencrypted HTTP to secure HTTPS? > Now, don't get me wrong: I'm personally extremely unhappy about having > to depend on the Sequoia stack for RPM PGP. I have a strong distaste > for the Rust community ecosystem these days, and I don't love the idea > of having to have LLVM in the core bootstrap chain (hopefully gcc-rs > will be in place soon enough!). The dependency on LLVM is not even the worst issue in my eyes. LLVM is also used by other core projects, e.g., mesa, these days. The worst issue I see with Rust is the way libraries are "packaged", which just implies installing source code and recompiling that source code for every single application. (And as a result, the output obviously gets statically linked into the application, with all the drawbacks of static linking.) I consider a language with no usable shared library support to be entirely unpackageable and hence entirely useless. And then of course there is the issue that it is yet another language with yet another syntax (and an only partially C-like one, so the learning curve is unnecessarily high), yet another library ecosystem, etc. C has been the de facto lingua franca all this time, now we are back into a tower-of-babel scenario with tons of programming languages, which will necessarily bloat the core system over time. > So here we are, in a subpar situation created by bad tools because > nobody cares enough about security anyway. Sounds like a mess indeed. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen wrote: > > On 10/13/22 10:53, Neal Gompa wrote: > > On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: > >> > >> On 10/13/22 07:18, Kevin Kofler via devel wrote: > For the last 20 years or so, RPM has used a home-grown OpenPGP parser > for dealing with keys and signatures. That parser is rather infamous > for its limitations and flaws, and especially in recent years has > proven a significant burden to RPM development. In order to improve > security and free developer resources for dealing with RPM's "core > business" instead, RPM upstream is in the process of deprecating the > internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > based solution written in Rust. > >>> > >>> Why are you using a new library written in Rust? Can you not use one of > >>> the > >>> existing mature C implementations of OpenPGP? gpgme maybe? > >> > >> Had there been such an option, we would've switched over years ago. > >> Gpgme is based around calling and communicating with an external gpg > >> process, which is a setup you do NOT want in the rpm context where > >> chroots come and go etc. Also, rpm requires access to the "low-level" > >> digests-in-progress because it calculates multiple things on a single read. > >> > > > > The real problem is that all other OpenPGP implementations died out > > because of GnuPG. And then GnuPG made the choice to force an > > inter-process model. > > > > At work, I deal with this on Debian systems, which do indeed use GnuPG > > for this. It creates a lot of problems, especially for building images > > and dealing with chroots, which is why in the context of RPM PGP > > upstream, I never pushed to consider it. > > > > The most serious problems with PackageKit memory leaks and hangs are > > actually caused by GnuPG, since DNF uses it for some GPG stuff because > > there's no API for using RPM's PGP capabilities. There's no fix unless > > the RPM and DNF teams agree on an API and build it out so that libdnf > > and librepo no longer need to use GnuPG through gpgme anymore. > > > > This is also the underlying reason why Red Hat has resisted > > implementing signed repository metadata and enforcing it by default. > > Of course this is a bit of a catch-22 as well, as there's no > > motivation to find a solution because neither Fedora nor RHEL offer > > signed repository metadata despite repeated calls for it over the past > > decade. > > > > Now, don't get me wrong: I'm personally extremely unhappy about having > > to depend on the Sequoia stack for RPM PGP. I have a strong distaste > > for the Rust community ecosystem these days, and I don't love the idea > > of having to have LLVM in the core bootstrap chain (hopefully gcc-rs > > will be in place soon enough!). But we literally don't have any other > > options. GnuPG/GPGME is out of the question for the reasons Panu and I > > stated, and NeoPG died several years ago. There are no C/C++ libraries > > for OpenPGP verification. > > There's RNP (in C++, used by Thunderbird at least), but alas that > doesn't expose the digest-in-progress either. So at least in it's > current form, it's not an option for rpm. Also, the API appears to have > all manner of quirks and gotchas that aren't welcome in a > security-critical piece. > Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend and at least the verification API (which is what RPM would use) seems to be getting love lately. Insofar as quirks and gotchas, I'm not a great judge of that at the moment, but I don't think it could be worse than what we have with GnuPG. The missing "digest-in-progress" thing is an issue, I guess. Have we raised the issue with them about it? > As for bootstrap, there will always (have to) be a way to build rpm > without depending on Rust. Even if that meant no signature verification > support in such a configuration. > Eck. What about the x509 based stuff we were talking about last year? All the crypto backends RPM supports now support that stuff out of the box. Embedded x509 signatures (certs) to replace GPG signatures could work as an alternative. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 10:53, Neal Gompa wrote: On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: On 10/13/22 07:18, Kevin Kofler via devel wrote: For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? Had there been such an option, we would've switched over years ago. Gpgme is based around calling and communicating with an external gpg process, which is a setup you do NOT want in the rpm context where chroots come and go etc. Also, rpm requires access to the "low-level" digests-in-progress because it calculates multiple things on a single read. The real problem is that all other OpenPGP implementations died out because of GnuPG. And then GnuPG made the choice to force an inter-process model. At work, I deal with this on Debian systems, which do indeed use GnuPG for this. It creates a lot of problems, especially for building images and dealing with chroots, which is why in the context of RPM PGP upstream, I never pushed to consider it. The most serious problems with PackageKit memory leaks and hangs are actually caused by GnuPG, since DNF uses it for some GPG stuff because there's no API for using RPM's PGP capabilities. There's no fix unless the RPM and DNF teams agree on an API and build it out so that libdnf and librepo no longer need to use GnuPG through gpgme anymore. This is also the underlying reason why Red Hat has resisted implementing signed repository metadata and enforcing it by default. Of course this is a bit of a catch-22 as well, as there's no motivation to find a solution because neither Fedora nor RHEL offer signed repository metadata despite repeated calls for it over the past decade. Now, don't get me wrong: I'm personally extremely unhappy about having to depend on the Sequoia stack for RPM PGP. I have a strong distaste for the Rust community ecosystem these days, and I don't love the idea of having to have LLVM in the core bootstrap chain (hopefully gcc-rs will be in place soon enough!). But we literally don't have any other options. GnuPG/GPGME is out of the question for the reasons Panu and I stated, and NeoPG died several years ago. There are no C/C++ libraries for OpenPGP verification. There's RNP (in C++, used by Thunderbird at least), but alas that doesn't expose the digest-in-progress either. So at least in it's current form, it's not an option for rpm. Also, the API appears to have all manner of quirks and gotchas that aren't welcome in a security-critical piece. As for bootstrap, there will always (have to) be a way to build rpm without depending on Rust. Even if that meant no signature verification support in such a configuration. If there was *any other choice*, we would have taken it. Even splitting out RPM's internal OpenPGP code into its own project for someone to rework and upgrade would be an option if someone actually wanted to. The reality is that nobody wants to work on that. Yep. In case there's any doubt, I too would've much, MUCH, VERY MUCH preferred a C (or even C++) library but in the ~15 years I've been on the lookout, no viable and credible candidates have appeared. Rpm will remain open to alternative backends, should such a thing emerge, but I'm not holding my breath. - Panu - So here we are, in a subpar situation created by bad tools because nobody cares enough about security anyway. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen wrote: > > On 10/13/22 07:18, Kevin Kofler via devel wrote: > >> For the last 20 years or so, RPM has used a home-grown OpenPGP parser > >> for dealing with keys and signatures. That parser is rather infamous > >> for its limitations and flaws, and especially in recent years has > >> proven a significant burden to RPM development. In order to improve > >> security and free developer resources for dealing with RPM's "core > >> business" instead, RPM upstream is in the process of deprecating the > >> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > >> based solution written in Rust. > > > > Why are you using a new library written in Rust? Can you not use one of the > > existing mature C implementations of OpenPGP? gpgme maybe? > > Had there been such an option, we would've switched over years ago. > Gpgme is based around calling and communicating with an external gpg > process, which is a setup you do NOT want in the rpm context where > chroots come and go etc. Also, rpm requires access to the "low-level" > digests-in-progress because it calculates multiple things on a single read. > The real problem is that all other OpenPGP implementations died out because of GnuPG. And then GnuPG made the choice to force an inter-process model. At work, I deal with this on Debian systems, which do indeed use GnuPG for this. It creates a lot of problems, especially for building images and dealing with chroots, which is why in the context of RPM PGP upstream, I never pushed to consider it. The most serious problems with PackageKit memory leaks and hangs are actually caused by GnuPG, since DNF uses it for some GPG stuff because there's no API for using RPM's PGP capabilities. There's no fix unless the RPM and DNF teams agree on an API and build it out so that libdnf and librepo no longer need to use GnuPG through gpgme anymore. This is also the underlying reason why Red Hat has resisted implementing signed repository metadata and enforcing it by default. Of course this is a bit of a catch-22 as well, as there's no motivation to find a solution because neither Fedora nor RHEL offer signed repository metadata despite repeated calls for it over the past decade. Now, don't get me wrong: I'm personally extremely unhappy about having to depend on the Sequoia stack for RPM PGP. I have a strong distaste for the Rust community ecosystem these days, and I don't love the idea of having to have LLVM in the core bootstrap chain (hopefully gcc-rs will be in place soon enough!). But we literally don't have any other options. GnuPG/GPGME is out of the question for the reasons Panu and I stated, and NeoPG died several years ago. There are no C/C++ libraries for OpenPGP verification. If there was *any other choice*, we would have taken it. Even splitting out RPM's internal OpenPGP code into its own project for someone to rework and upgrade would be an option if someone actually wanted to. The reality is that nobody wants to work on that. So here we are, in a subpar situation created by bad tools because nobody cares enough about security anyway. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
On 10/13/22 07:18, Kevin Kofler via devel wrote: For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? Had there been such an option, we would've switched over years ago. Gpgme is based around calling and communicating with an external gpg process, which is a setup you do NOT want in the rpm context where chroots come and go etc. Also, rpm requires access to the "low-level" digests-in-progress because it calculates multiple things on a single read. At this point the change is mostly invisible in normal daily use. Not really, because it makes some packages uninstallable. Not uninstallable. You can use --nosignature, or resign such packages with something stronger. Which one is the better course depends on the situation of course. - Some old, insecure (MD5/SHA1 based) signatures are rejected (this is in line with the stronger crypto settings proposed elsewhere for F38) Such a hardcoded restriction, without a way for the local administrator to allow the legacy signatures, is not acceptable. Mind you, I don't exactly agree with this style of explicit disabling either (see https://lists.rpm.org/pipermail/rpm-maint/2021-October/018344.html and onwards). But. I doubt many people realize just how thin the ice is (and has always been) with the existing parser. I consider this step a matter of survival, and ultimately some legacy content becoming harder to use is an acceptable tradeoff for *that*. I don't know how deep this all is wired inside Sequoia, but I totally agree (as you see in the thread linked above) that this should be based on the system crypto policy. As explained in the change, nettle (which doesn't support the system crypto policies AIUI) should be seen as a temporary stepstone in Fedora, with a plan to switch to openssl (which does) in the nearish future. So technically this is a matter of "Sequoia should honor system crypto policy", rpm is just a dumb API user here that sometimes get told "nope" by the underlying libraries, whether due to crypto policy, FIPS or whatever. - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
> For the last 20 years or so, RPM has used a home-grown OpenPGP parser > for dealing with keys and signatures. That parser is rather infamous > for its limitations and flaws, and especially in recent years has > proven a significant burden to RPM development. In order to improve > security and free developer resources for dealing with RPM's "core > business" instead, RPM upstream is in the process of deprecating the > internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] > based solution written in Rust. Why are you using a new library written in Rust? Can you not use one of the existing mature C implementations of OpenPGP? gpgme maybe? > At this point the change is mostly invisible in normal daily use. Not really, because it makes some packages uninstallable. > - Some old, insecure (MD5/SHA1 based) signatures are rejected (this is > in line with the stronger crypto settings proposed elsewhere for F38) Such a hardcoded restriction, without a way for the local administrator to allow the legacy signatures, is not acceptable. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)
Hi Ben, Ben Cotton wrote: Within Fedora package set, this has no impact as everything is already using sufficiently strong crypto. Third party repositories / packages could be signed with insecure crypto, and those may require working around with --nosignature. However this incidentally overlaps with https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2 which has effectively the same effect on rpm. Note that the StrongCryptoSettings3Forewarning2 proposal recently failed to gather enough votes to be accepted, so it will likely not be happening (or not in this form) for Fedora 38. Additionally, crypto-policies would have supported switching to LEGACY to allow installation of non-conforming RPMs, so you should at least provide a method to also install such old RPMs, ideally while still verifying the old SHA-1 signature rather than ignoring it completely. HTH, Clemens -- Clemens Lang RHEL Crypto Team Red Hat ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
F38 proposal: RPM Sequoia (System-Wide Change proposal)
https://fedoraproject.org/wiki/Changes/RpmSequoia This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee. == Summary == Change RPM to use [https://sequoia-pgp.org/ Sequoia] based OpenPGP parser instead of it's own, flawed and limited implementation. == Owner == * Name: [[User:pmatilai| Panu Matilainen]] * Email: pmati...@redhat.com == Detailed Description == For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. At this point the change is mostly invisible in normal daily use. == Feedback == == Benefit to Fedora == The main, direct benefit to Fedora is improved security and standards-compliance (RFC-4880) in one of the corner-stones of the whole distribution. Longer term, we can expect better error messages and other functional improvements regarding key and signature handling. == Scope == * Proposal owners: ** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499 package/review rpm-sequoia] ** Build rpm with --with-crypto=sequoia ** Watch out for the unexpected * Other developers: ** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499 package/review rpm-sequoia] * Release engineering: [https://pagure.io/releng/issue/11077 #11077] * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A == Upgrade/compatibility impact == Within Fedora package set, this has no impact as everything is already using sufficiently strong crypto. Third party repositories / packages could be signed with insecure crypto, and those may require working around with --nosignature. However this incidentally overlaps with https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2 which has effectively the same effect on rpm. == How To Test == In general, normal rpm/dnf use provides sufficient test coverage. For more advanced testers: try signing and verifying with different keys and their subkeys, using different algorithms etc. == User Experience == For normal usage, the change is quite invisible. The notable exceptions are - Some old, insecure (MD5/SHA1 based) signatures are rejected (this is in line with the stronger crypto settings proposed elsewhere for F38) - Key import may accept some previously rejected keys, in part due to limitations of old parser etc but in particular, the old implementation verifies self-signatures at import time whereas Sequoia verifies them at time of use. - Key import may reject some previously accepted keys due to better validation. == Dependencies == The change introduces one new direct dependency: [https://github.com/rpm-software-management/rpm-sequoia/ rpm-sequoia]. The rpm-sequoia package also takes over other crypto besides OpenPGP, currently Sequoia uses nettle as its low-level crypto provider, but work is underway to [https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1361 support openssl in Sequoia], and the plan is to have Sequoia in Fedora use that once it becomes available. This plan [https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/message/EY5VVR2VPKSISHRANZTK2HYA6RP6345L/ has support of the crypto team]. == Contingency Plan == * Contingency mechanism: Revert back to the internal PGP parser * Contingency deadline: Beta release * Blocks release? No == Documentation == There's not much in the way of documentation as there's not much to document, except for the deprecation of the internal parser: https://github.com/rpm-software-management/rpm/issues/1935 rpm-sequoia build instructions can be found in https://github.com/rpm-software-management/rpm-sequoia/ == Release Notes == -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis ___ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
F38 proposal: RPM Sequoia (System-Wide Change proposal)
https://fedoraproject.org/wiki/Changes/RpmSequoia This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee. == Summary == Change RPM to use [https://sequoia-pgp.org/ Sequoia] based OpenPGP parser instead of it's own, flawed and limited implementation. == Owner == * Name: [[User:pmatilai| Panu Matilainen]] * Email: pmati...@redhat.com == Detailed Description == For the last 20 years or so, RPM has used a home-grown OpenPGP parser for dealing with keys and signatures. That parser is rather infamous for its limitations and flaws, and especially in recent years has proven a significant burden to RPM development. In order to improve security and free developer resources for dealing with RPM's "core business" instead, RPM upstream is in the process of deprecating the internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP] based solution written in Rust. At this point the change is mostly invisible in normal daily use. == Feedback == == Benefit to Fedora == The main, direct benefit to Fedora is improved security and standards-compliance (RFC-4880) in one of the corner-stones of the whole distribution. Longer term, we can expect better error messages and other functional improvements regarding key and signature handling. == Scope == * Proposal owners: ** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499 package/review rpm-sequoia] ** Build rpm with --with-crypto=sequoia ** Watch out for the unexpected * Other developers: ** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499 package/review rpm-sequoia] * Release engineering: [https://pagure.io/releng/issue/11077 #11077] * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A == Upgrade/compatibility impact == Within Fedora package set, this has no impact as everything is already using sufficiently strong crypto. Third party repositories / packages could be signed with insecure crypto, and those may require working around with --nosignature. However this incidentally overlaps with https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2 which has effectively the same effect on rpm. == How To Test == In general, normal rpm/dnf use provides sufficient test coverage. For more advanced testers: try signing and verifying with different keys and their subkeys, using different algorithms etc. == User Experience == For normal usage, the change is quite invisible. The notable exceptions are - Some old, insecure (MD5/SHA1 based) signatures are rejected (this is in line with the stronger crypto settings proposed elsewhere for F38) - Key import may accept some previously rejected keys, in part due to limitations of old parser etc but in particular, the old implementation verifies self-signatures at import time whereas Sequoia verifies them at time of use. - Key import may reject some previously accepted keys due to better validation. == Dependencies == The change introduces one new direct dependency: [https://github.com/rpm-software-management/rpm-sequoia/ rpm-sequoia]. The rpm-sequoia package also takes over other crypto besides OpenPGP, currently Sequoia uses nettle as its low-level crypto provider, but work is underway to [https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1361 support openssl in Sequoia], and the plan is to have Sequoia in Fedora use that once it becomes available. This plan [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/EY5VVR2VPKSISHRANZTK2HYA6RP6345L/ has support of the crypto team]. == Contingency Plan == * Contingency mechanism: Revert back to the internal PGP parser * Contingency deadline: Beta release * Blocks release? No == Documentation == There's not much in the way of documentation as there's not much to document, except for the deprecation of the internal parser: https://github.com/rpm-software-management/rpm/issues/1935 rpm-sequoia build instructions can be found in https://github.com/rpm-software-management/rpm-sequoia/ == Release Notes == -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
