Re: Preventing supply chain attacks via rekor
On Sat, Jun 12, 2021 at 10:29:50AM +0200, Marius Schwarz wrote: > Am 12.06.21 um 02:51 schrieb Kevin Fenzi: > > > > > Also, not having it available has made it *very* hard to prioritize > > > getting the issues fixed in DNF. So being able to improve this is > > > predicated on the existence of signed metadata. > > This seems odd to me. I mean, it can't be hard to setup a test repo, is > > it? I suspect we could even ask QE folks to do some testing and map out > > the issues they find. I don't think it's nice/ethical to break users > > just as a means to make bugs we want to have fixed higher priority. > > > > Anyhow, we are pretty off topic for this thread, so I'll try and stop... > > > > kevin > > > > Does it really hurt someone, if the repos get signed and clients just do not > check this by default? Nope, but it's still not technically possible. There needs to be work in bodhi and robosignatory at least. > > Of course, the signing should not break the unchecked repo in any form, > which needs a small testcase ;) Sure, but if all thats needed is a test repo, I can setup one right now on my fedorapeople space... :) kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
Am 12.06.21 um 02:51 schrieb Kevin Fenzi: Also, not having it available has made it *very* hard to prioritize getting the issues fixed in DNF. So being able to improve this is predicated on the existence of signed metadata. This seems odd to me. I mean, it can't be hard to setup a test repo, is it? I suspect we could even ask QE folks to do some testing and map out the issues they find. I don't think it's nice/ethical to break users just as a means to make bugs we want to have fixed higher priority. Anyhow, we are pretty off topic for this thread, so I'll try and stop... kevin Does it really hurt someone, if the repos get signed and clients just do not check this by default? Of course, the signing should not break the unchecked repo in any form, which needs a small testcase ;) Best regards, Marius ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 11:46:42AM -0400, Neal Gompa wrote: > > I would like repos signed even if we don't enable it in the repo > definitions by default for now. That would make it possible for my Open > Build Service instance to validate Fedora content for package builds > (it can't use metalinks or mirrorlists, but it can check and validate > signed repodata). I asked CentOS years ago to do this for the same > reason, and they did it[1]. Sure, and when we can we can... but I don't think it should be prioritzed over work that actually has wider benifits. > > Also, not having it available has made it *very* hard to prioritize > getting the issues fixed in DNF. So being able to improve this is > predicated on the existence of signed metadata. This seems odd to me. I mean, it can't be hard to setup a test repo, is it? I suspect we could even ask QE folks to do some testing and map out the issues they find. I don't think it's nice/ethical to break users just as a means to make bugs we want to have fixed higher priority. Anyhow, we are pretty off topic for this thread, so I'll try and stop... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 04:11:24PM -0700, Stewart Smith via devel wrote: > Björn Persson writes: > > I believe Yum has a feature to verify signed repository metadata. I > > don't know why it's not used. If that verification would be turned on, > > are there any attacks that would still be possible then, that Rekor > > could prevent? > > There's still the classic downgrade attack: point to an older version of > the repositories. Enforcing https helps mitigate it by having the client > put trust in the certificate owner to run a secure mirror which is kept > up to date. In the default configuration you get a metalink from mirrors.fedoraproject.org. That metalink has checksums for the last 2 repomd.xml files in it. If a mirror you go to then offers you a incorrect repomd.xml or any files (which are in turn checksumed in repomd.xml) dnf will treat the mirror as broken and skip to the next one. So you are trusting mirrors.fedoraproject.org to send you a correct metalink. You can also use/enable dnssec to make sure you get the correct ip address(es) as well. But all of this is not related to rekor... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
RE: Preventing supply chain attacks via rekor
Björn Persson writes: > I believe Yum has a feature to verify signed repository metadata. I > don't know why it's not used. If that verification would be turned on, > are there any attacks that would still be possible then, that Rekor > could prevent? There's still the classic downgrade attack: point to an older version of the repositories. Enforcing https helps mitigate it by having the client put trust in the certificate owner to run a secure mirror which is kept up to date. You get some protection from *some* downgrade attacks since there's timestamps on repo metadata and if you see older metadata than what you saw last time (yum at least, I haven't double-checked DNF) will complain at you. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 8:09 PM Miloslav Trmac wrote:
> Hello,
> pá 11. 6. 2021 v 20:23 odesílatel Luke Hinds napsal:
>
>> On Fri, Jun 11, 2021 at 7:01 PM Miloslav Trmac wrote:
>>
>>> pá 11. 6. 2021 v 18:54 odesílatel Luke Hinds napsal:
>>>
Why is this useful? You get a timestamped / tamper resistance record of
all signing events. This is very useful for understanding the exact blast
radius of a key compromise and monitoring for suspicious events. Most of
the time you will not interact with the tlog, it would sit off to side
hashing in entries, so it would make no changes to a maintainers working
flow or not being in any build path where it could cause an outage.
>>>
>>> That seems to suggest that ordinary clients pulling updates would not
>>> depend on finding a Rekor entry (and the “not being in any build path”
>>> wording even suggests that builds (composes?) would not send anything to
>>> Rekor ??!), and in that case…
>>>
>>
>> I may not have articulated this well. My main point was this is not an
>> inline / gating system, whereby if rekor has a system outage its not going
>> to stop the entire build chain. I saw this as folks always get nervous
>> about adding new actors to a build pipeline. "would not send anything to
>> Rekor ??!), and in that case…" no, that is very far from the case :)
>>
>
> I don’t understand at all. Would the build usually send things to Rekor,
> or not? If not, what‘s the value of Rekor? If yes, and that process failed,
> would the build fail or not? Or are you differentiating between a build
> (which would not create consumable signatures) and a compose/publish step
> (which could fail if Rekor is unavailable)?
>
Yes it would. Forget my earlier wording, it was not helpful. I was just
trying to state if it's in collection only mode (and not a queryable system
for entry by clients), then it would not cause other systems to backup with
quotes etc.
> It's the sort of system most would not be aware of, unless something awful
happens, and then it has a lot of value, as you have an immutable record of
events.
>>>
>>> … creating the immutable record is entirely optional, in particular the
>>> attacker doesn’t have to do this to attack a consumer successfully. The
>>> record would only exist if the attacker could trigger an existing
>>> build/signing system to build/sign a malicious artifact, without having
>>> full control over the build/signing system — and in that case any other
>>> logging solution from that build/signing system (just send it to a
>>> write-only log host) would have very similar security properties, it seems
>>> to me.
>>>
>>
>> I don't understand where you're getting entirely optional from?
>>
>
> My attack scenario is that an attacker $somehow obtains a signature of a
> malicious RPM (by building it in Koji directly, attacking Koji to get it
> built, stealing the private key and signing it itself), and then publishes
> that malicious RPM on an attacker-controller mirror/server for a victim to
> update to. At least in the “stolen private key” scenario, and maybe in the
> others (see questions above), the attacker can just *not upload any entry*
> to Rekor. What forces the attacker to upload an entry? If the attacker
> doesn’t have to enter anything into Rekor, what is the value? The only way
> this makes sense to me is to make proof of Rekor inclusion a required
> component of signature verification, for everyone. Is that the proposal?
>
Sorry, I understand now (long week), so what you describe is the ideal
implementation, which is where clients verify for inclusion ("Am I seeing
what everyone else is seeing") , or discoverability as the
certificate transparency folks call it. This is exactly what browsers do
with certificate transparency. They have a header 'Expect-CT' [0] , if no
entry / cert chain is in the log and they are in enforcement mode, they
reject a visit to the site . This way it negates the 'just don't put it in
the log' concern. This is what we do in sigstore, if no software signing
materials are in the transparency log, it's not a valid signing and it's
rejected - as there is no transparency watcher of the event.
[0] https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-expect-ct-02
> Mirek
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora
Re: Preventing supply chain attacks via rekor
Hello, pá 11. 6. 2021 v 20:23 odesílatel Luke Hinds napsal: > On Fri, Jun 11, 2021 at 7:01 PM Miloslav Trmac wrote: > >> pá 11. 6. 2021 v 18:54 odesílatel Luke Hinds napsal: >> >>> Why is this useful? You get a timestamped / tamper resistance record of >>> all signing events. This is very useful for understanding the exact blast >>> radius of a key compromise and monitoring for suspicious events. Most of >>> the time you will not interact with the tlog, it would sit off to side >>> hashing in entries, so it would make no changes to a maintainers working >>> flow or not being in any build path where it could cause an outage. >>> >> >> That seems to suggest that ordinary clients pulling updates would not >> depend on finding a Rekor entry (and the “not being in any build path” >> wording even suggests that builds (composes?) would not send anything to >> Rekor ??!), and in that case… >> > > I may not have articulated this well. My main point was this is not an > inline / gating system, whereby if rekor has a system outage its not going > to stop the entire build chain. I saw this as folks always get nervous > about adding new actors to a build pipeline. "would not send anything to > Rekor ??!), and in that case…" no, that is very far from the case :) > I don’t understand at all. Would the build usually send things to Rekor, or not? If not, what‘s the value of Rekor? If yes, and that process failed, would the build fail or not? Or are you differentiating between a build (which would not create consumable signatures) and a compose/publish step (which could fail if Rekor is unavailable)? It's the sort of system most would not be aware of, unless something awful >>> happens, and then it has a lot of value, as you have an immutable record of >>> events. >>> >> >> … creating the immutable record is entirely optional, in particular the >> attacker doesn’t have to do this to attack a consumer successfully. The >> record would only exist if the attacker could trigger an existing >> build/signing system to build/sign a malicious artifact, without having >> full control over the build/signing system — and in that case any other >> logging solution from that build/signing system (just send it to a >> write-only log host) would have very similar security properties, it seems >> to me. >> > > I don't understand where you're getting entirely optional from? > My attack scenario is that an attacker $somehow obtains a signature of a malicious RPM (by building it in Koji directly, attacking Koji to get it built, stealing the private key and signing it itself), and then publishes that malicious RPM on an attacker-controller mirror/server for a victim to update to. At least in the “stolen private key” scenario, and maybe in the others (see questions above), the attacker can just *not upload any entry* to Rekor. What forces the attacker to upload an entry? If the attacker doesn’t have to enter anything into Rekor, what is the value? The only way this makes sense to me is to make proof of Rekor inclusion a required component of signature verification, for everyone. Is that the proposal? Mirek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 7:01 PM Miloslav Trmac wrote: > Hello, > pá 11. 6. 2021 v 18:54 odesílatel Luke Hinds napsal: > >> Why is this useful? You get a timestamped / tamper resistance record of >> all signing events. This is very useful for understanding the exact blast >> radius of a key compromise and monitoring for suspicious events. Most of >> the time you will not interact with the tlog, it would sit off to side >> hashing in entries, so it would make no changes to a maintainers working >> flow or not being in any build path where it could cause an outage. >> > > That seems to suggest that ordinary clients pulling updates would not > depend on finding a Rekor entry (and the “not being in any build path” > wording even suggests that builds (composes?) would not send anything to > Rekor ??!), and in that case… > I may not have articulated this well. My main point was this is not an inline / gating system, whereby if rekor has a system outage its not going to stop the entire build chain. I saw this as folks always get nervous about adding new actors to a build pipeline. "would not send anything to Rekor ??!), and in that case…" no, that is very far from the case :) > >> It's the sort of system most would not be aware of, unless something >> awful happens, and then it has a lot of value, as you have an immutable >> record of events. >> > > … creating the immutable record is entirely optional, in particular the > attacker doesn’t have to do this to attack a consumer successfully. The > record would only exist if the attacker could trigger an existing > build/signing system to build/sign a malicious artifact, without having > full control over the build/signing system — and in that case any other > logging solution from that build/signing system (just send it to a > write-only log host) would have very similar security properties, it seems > to me. > > I don't understand where you're getting entirely optional from? > *So what’s the new security property? *It’s not that we have a full > record, if I understand the proposal correctly (maybe I don’t); it’s not > that anyone can on-line query whether an artifact is the latest currently > signed version (for that, clients can already use HTTPS to get the metalink > data IIRC). > > This would only make sense if a proof of presence in the Rekor log were a > required part of signature verification by all clients — and that would > still *only* give us an audit log, it would not actually prevent clients > from installing the maliciously-signed RPM (the current metalinks sort-of > do that already, sure). Given the above discussion about signing metadata, > presumably a much smaller change, what makes such a Rekor integration more > likely to get done? > Mirek > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
Hello, pá 11. 6. 2021 v 18:54 odesílatel Luke Hinds napsal: > Why is this useful? You get a timestamped / tamper resistance record of > all signing events. This is very useful for understanding the exact blast > radius of a key compromise and monitoring for suspicious events. Most of > the time you will not interact with the tlog, it would sit off to side > hashing in entries, so it would make no changes to a maintainers working > flow or not being in any build path where it could cause an outage. > That seems to suggest that ordinary clients pulling updates would not depend on finding a Rekor entry (and the “not being in any build path” wording even suggests that builds (composes?) would not send anything to Rekor ??!), and in that case… > It's the sort of system most would not be aware of, unless something awful > happens, and then it has a lot of value, as you have an immutable record of > events. > … creating the immutable record is entirely optional, in particular the attacker doesn’t have to do this to attack a consumer successfully. The record would only exist if the attacker could trigger an existing build/signing system to build/sign a malicious artifact, without having full control over the build/signing system — and in that case any other logging solution from that build/signing system (just send it to a write-only log host) would have very similar security properties, it seems to me. *So what’s the new security property? *It’s not that we have a full record, if I understand the proposal correctly (maybe I don’t); it’s not that anyone can on-line query whether an artifact is the latest currently signed version (for that, clients can already use HTTPS to get the metalink data IIRC). This would only make sense if a proof of presence in the Rekor log were a required part of signature verification by all clients — and that would still *only* give us an audit log, it would not actually prevent clients from installing the maliciously-signed RPM (the current metalinks sort-of do that already, sure). Given the above discussion about signing metadata, presumably a much smaller change, what makes such a Rekor integration more likely to get done? Mirek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 4:48 PM Neal Gompa wrote: > On Fri, Jun 11, 2021 at 11:17 AM Kevin Fenzi wrote: > > > > On Fri, Jun 11, 2021 at 06:27:18AM -0400, Neal Gompa wrote: > > > > > > We do not, however, have GPG signatures on repository metadata. Which > > > > True. > > > > > means that we can't guarantee the repositories aren't tampered with. > > > > False. > > > > > This is especially problematic for people who use local mirrors or do > > > net installs. > > > > We could do more for local mirrors indeed. > > > > On Fri, Jun 11, 2021 at 01:48:50PM +0200, Björn Persson wrote: > > ...snip... > > > > > > As it is now, mirrors can't modify RPM packages without a key that the > > > clients have installed. Mirrors could however withhold security updates > > > so that clients remain vulnerable. Is that a thing that Rekor could > > > prevent? > > > > They cannot in the default, normal case where users use metalinks. > > (well, they could for 3 days or less, but then they cannot) > > > > > I believe Yum has a feature to verify signed repository metadata. I > > > don't know why it's not used. If that verification would be turned on, > > > are there any attacks that would still be possible then, that Rekor > > > could prevent? > > > > You mean dnf? :) > > > > It does, but it has... a number of issues. > > > > ...snip... > > > > On Fri, Jun 11, 2021 at 07:55:49AM -0400, Neal Gompa wrote: > > > > > > Fedora Infrastructure has stonewalled on signing our RPM repositories > > > for almost a decade. Reasons have ranged from "I don't like it" to > > > "It's useless because metalinks" to "RPM repository format should be > > > modified first". However, openSUSE signs their repositories and it > > > works fine there. > > > > I would like to strongly object to this being called 'stonewalling' > > > > You make it sound like there's a simple config switch we just need to > > turn on that we refuse to do. Thats completely 100% not the case. > > > > The reasons this is not enabled have in fact changed over the years, not > > because we 'made up a new reason', but because progress has been made > > and you can't instantly enable something before everything to enable > > that is in place. > > > > First the issue was that we were manually signing things. This meant in > > order to sign repodata we would need to have a releng person sitting > > there and signing away at the _end_ of pushes. > > > > Then, that was fixed by sigul, but then there were issues with other > > tools in the chain, like bodhi needs to be able to inject security > > update info into the repodata at the end of a compose, so you have to > > make sure it's signed after that, etc, etc. Those issues have all slowly > > moved forward over the years. > > > > Then, we actually did enable signed repodata a number of years ago for > > the cisco h264 repo. This exposed all kinds of dnf issues with signed > > repodata: The keystore it has is seperate from the rpmdb one, so it has > > to import things and get a 'ok' from the user again, it doesn't error > > correctly on non interactive cases, and in fact this blocked a fedora > > release a while back. So, we changed it to not use signed repodata. > > > > IN summary: I'm ok with enabling signed repodata when it is possible to > > do so and don't cause pain and suffering for our users. This has in fact > > always been my answer to this. It's not a priority for me personally > > because metalinks handle almost all the cases. The only things that > > would be improved by signed repodata are local mirrors that sync without > > checking for the mirror they sync from being up to date in > > mirrormanager. If you know of more cases, please share them. > > > > I would like repos signed even if we don't enable it in the repo > definitions by default for now. That would make it possible for my Open > Build Service instance to validate Fedora content for package builds > (it can't use metalinks or mirrorlists, but it can check and validate > signed repodata). I asked CentOS years ago to do this for the same > reason, and they did it[1]. > > Also, not having it available has made it *very* hard to prioritize > getting the issues fixed in DNF. So being able to improve this is > predicated on the existence of signed metadata. > > [1]: > https://blog.centos.org/2018/07/improving-centos-package-delivery-security-with-signed-repository-metadata/ > > > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure Hello, There may
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 11:17 AM Kevin Fenzi wrote: > > On Fri, Jun 11, 2021 at 06:27:18AM -0400, Neal Gompa wrote: > > > > We do not, however, have GPG signatures on repository metadata. Which > > True. > > > means that we can't guarantee the repositories aren't tampered with. > > False. > > > This is especially problematic for people who use local mirrors or do > > net installs. > > We could do more for local mirrors indeed. > > On Fri, Jun 11, 2021 at 01:48:50PM +0200, Björn Persson wrote: > ...snip... > > > > As it is now, mirrors can't modify RPM packages without a key that the > > clients have installed. Mirrors could however withhold security updates > > so that clients remain vulnerable. Is that a thing that Rekor could > > prevent? > > They cannot in the default, normal case where users use metalinks. > (well, they could for 3 days or less, but then they cannot) > > > I believe Yum has a feature to verify signed repository metadata. I > > don't know why it's not used. If that verification would be turned on, > > are there any attacks that would still be possible then, that Rekor > > could prevent? > > You mean dnf? :) > > It does, but it has... a number of issues. > > ...snip... > > On Fri, Jun 11, 2021 at 07:55:49AM -0400, Neal Gompa wrote: > > > > Fedora Infrastructure has stonewalled on signing our RPM repositories > > for almost a decade. Reasons have ranged from "I don't like it" to > > "It's useless because metalinks" to "RPM repository format should be > > modified first". However, openSUSE signs their repositories and it > > works fine there. > > I would like to strongly object to this being called 'stonewalling' > > You make it sound like there's a simple config switch we just need to > turn on that we refuse to do. Thats completely 100% not the case. > > The reasons this is not enabled have in fact changed over the years, not > because we 'made up a new reason', but because progress has been made > and you can't instantly enable something before everything to enable > that is in place. > > First the issue was that we were manually signing things. This meant in > order to sign repodata we would need to have a releng person sitting > there and signing away at the _end_ of pushes. > > Then, that was fixed by sigul, but then there were issues with other > tools in the chain, like bodhi needs to be able to inject security > update info into the repodata at the end of a compose, so you have to > make sure it's signed after that, etc, etc. Those issues have all slowly > moved forward over the years. > > Then, we actually did enable signed repodata a number of years ago for > the cisco h264 repo. This exposed all kinds of dnf issues with signed > repodata: The keystore it has is seperate from the rpmdb one, so it has > to import things and get a 'ok' from the user again, it doesn't error > correctly on non interactive cases, and in fact this blocked a fedora > release a while back. So, we changed it to not use signed repodata. > > IN summary: I'm ok with enabling signed repodata when it is possible to > do so and don't cause pain and suffering for our users. This has in fact > always been my answer to this. It's not a priority for me personally > because metalinks handle almost all the cases. The only things that > would be improved by signed repodata are local mirrors that sync without > checking for the mirror they sync from being up to date in > mirrormanager. If you know of more cases, please share them. > I would like repos signed even if we don't enable it in the repo definitions by default for now. That would make it possible for my Open Build Service instance to validate Fedora content for package builds (it can't use metalinks or mirrorlists, but it can check and validate signed repodata). I asked CentOS years ago to do this for the same reason, and they did it[1]. Also, not having it available has made it *very* hard to prioritize getting the issues fixed in DNF. So being able to improve this is predicated on the existence of signed metadata. [1]: https://blog.centos.org/2018/07/improving-centos-package-delivery-security-with-signed-repository-metadata/ -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 06:27:18AM -0400, Neal Gompa wrote: > > We do not, however, have GPG signatures on repository metadata. Which True. > means that we can't guarantee the repositories aren't tampered with. False. > This is especially problematic for people who use local mirrors or do > net installs. We could do more for local mirrors indeed. On Fri, Jun 11, 2021 at 01:48:50PM +0200, Björn Persson wrote: ...snip... > > As it is now, mirrors can't modify RPM packages without a key that the > clients have installed. Mirrors could however withhold security updates > so that clients remain vulnerable. Is that a thing that Rekor could > prevent? They cannot in the default, normal case where users use metalinks. (well, they could for 3 days or less, but then they cannot) > I believe Yum has a feature to verify signed repository metadata. I > don't know why it's not used. If that verification would be turned on, > are there any attacks that would still be possible then, that Rekor > could prevent? You mean dnf? :) It does, but it has... a number of issues. ...snip... On Fri, Jun 11, 2021 at 07:55:49AM -0400, Neal Gompa wrote: > > Fedora Infrastructure has stonewalled on signing our RPM repositories > for almost a decade. Reasons have ranged from "I don't like it" to > "It's useless because metalinks" to "RPM repository format should be > modified first". However, openSUSE signs their repositories and it > works fine there. I would like to strongly object to this being called 'stonewalling' You make it sound like there's a simple config switch we just need to turn on that we refuse to do. Thats completely 100% not the case. The reasons this is not enabled have in fact changed over the years, not because we 'made up a new reason', but because progress has been made and you can't instantly enable something before everything to enable that is in place. First the issue was that we were manually signing things. This meant in order to sign repodata we would need to have a releng person sitting there and signing away at the _end_ of pushes. Then, that was fixed by sigul, but then there were issues with other tools in the chain, like bodhi needs to be able to inject security update info into the repodata at the end of a compose, so you have to make sure it's signed after that, etc, etc. Those issues have all slowly moved forward over the years. Then, we actually did enable signed repodata a number of years ago for the cisco h264 repo. This exposed all kinds of dnf issues with signed repodata: The keystore it has is seperate from the rpmdb one, so it has to import things and get a 'ok' from the user again, it doesn't error correctly on non interactive cases, and in fact this blocked a fedora release a while back. So, we changed it to not use signed repodata. IN summary: I'm ok with enabling signed repodata when it is possible to do so and don't cause pain and suffering for our users. This has in fact always been my answer to this. It's not a priority for me personally because metalinks handle almost all the cases. The only things that would be improved by signed repodata are local mirrors that sync without checking for the mirror they sync from being up to date in mirrormanager. If you know of more cases, please share them. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 7:49 AM Björn Persson wrote: > > Huzaifa Sidhpurwala wrote: > > I am sure everyone has heard about the recent Solarwinds software supply > > chain attacks. This attack has made all software vendors think about > > securing their supply chain, and it is even more applicable to linux > > distributions which are made of thousands of components built from > > sources they dont even have control over. > > Yes, there is much that could be improved in this area. > > > One possible step in this direction is the ability to ensure that there > > is no distribution point tampering of binaries shipped in Fedora. > > What would "distribution point" mean here? Repository mirrors? The > master repository? mirrors.fedoraproject.org? > > > this could be a post-build thing, in which ones the rpms reach > > stable and are signed, rekor would run on it and store the binary > > metadata in the transparency logs. > > As it is now, mirrors can't modify RPM packages without a key that the > clients have installed. Mirrors could however withhold security updates > so that clients remain vulnerable. Is that a thing that Rekor could > prevent? > > I believe Yum has a feature to verify signed repository metadata. I > don't know why it's not used. If that verification would be turned on, > are there any attacks that would still be possible then, that Rekor > could prevent? > Fedora Infrastructure has stonewalled on signing our RPM repositories for almost a decade. Reasons have ranged from "I don't like it" to "It's useless because metalinks" to "RPM repository format should be modified first". However, openSUSE signs their repositories and it works fine there. > > More information at: > > > > https://sigstore.dev/what_is_sigstore/ > > On that page I can't see anything but a page header and a pointless > animation. Even the text that is right there in the HTML code is hidden. > Instead it wants me to execute a bunch of Javascript from at least three > different domains. When a website expects me to execute some unknown > program before they'll even tell me who they are or what they do, then > I'm much more likely to just ignore that website. > It's a Linux Foundation project. It mostly has produced some Go code for supporting detached signatures for containers, since container images are not designed to have attestation and integrity signing. It's not very useful since no runtimes validate these things. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
Huzaifa Sidhpurwala wrote: > I am sure everyone has heard about the recent Solarwinds software supply > chain attacks. This attack has made all software vendors think about > securing their supply chain, and it is even more applicable to linux > distributions which are made of thousands of components built from > sources they dont even have control over. Yes, there is much that could be improved in this area. > One possible step in this direction is the ability to ensure that there > is no distribution point tampering of binaries shipped in Fedora. What would "distribution point" mean here? Repository mirrors? The master repository? mirrors.fedoraproject.org? > this could be a post-build thing, in which ones the rpms reach > stable and are signed, rekor would run on it and store the binary > metadata in the transparency logs. As it is now, mirrors can't modify RPM packages without a key that the clients have installed. Mirrors could however withhold security updates so that clients remain vulnerable. Is that a thing that Rekor could prevent? I believe Yum has a feature to verify signed repository metadata. I don't know why it's not used. If that verification would be turned on, are there any attacks that would still be possible then, that Rekor could prevent? > More information at: > > https://sigstore.dev/what_is_sigstore/ On that page I can't see anything but a page header and a pointless animation. Even the text that is right there in the HTML code is hidden. Instead it wants me to execute a bunch of Javascript from at least three different domains. When a website expects me to execute some unknown program before they'll even tell me who they are or what they do, then I'm much more likely to just ignore that website. Björn Persson pgpkB9LKZss3D.pgp Description: OpenPGP digital signatur ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On Fri, Jun 11, 2021 at 5:09 AM Vitaly Zaitsev via devel wrote: > > On 11.06.2021 09:42, Huzaifa Sidhpurwala wrote: > > One possible step in this direction is the ability to ensure that there > > is no distribution point tampering of binaries shipped in Fedora. > > All RPM packages are already digitally signed by Fedora GPG keys. No > further actions is required. > > If someone MITM replaces the RPM package, it will fail the gpg signature > check and will not be installed. > We do not, however, have GPG signatures on repository metadata. Which means that we can't guarantee the repositories aren't tampered with. This is especially problematic for people who use local mirrors or do net installs. Also, Anaconda still doesn't do GPG checks[1]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=998 (yes, really!) -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Preventing supply chain attacks via rekor
On 11.06.2021 09:42, Huzaifa Sidhpurwala wrote: One possible step in this direction is the ability to ensure that there is no distribution point tampering of binaries shipped in Fedora. All RPM packages are already digitally signed by Fedora GPG keys. No further actions is required. If someone MITM replaces the RPM package, it will fail the gpg signature check and will not be installed. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Preventing supply chain attacks via rekor
Hi All, I am sure everyone has heard about the recent Solarwinds software supply chain attacks. This attack has made all software vendors think about securing their supply chain, and it is even more applicable to linux distributions which are made of thousands of components built from sources they dont even have control over. One possible step in this direction is the ability to ensure that there is no distribution point tampering of binaries shipped in Fedora. rekor is a an immutable tamper resistant ledger of metadata generated within a software projects supply chain. Rekor will enable software maintainers and build systems to record signed metadata to an immutable record. How would it work, would package maintainers need to be involved? Not really, this could be a post-build thing, in which ones the rpms reach stable and are signed, rekor would run on it and store the binary metadata in the transparency logs. I just wanted to send out a email to fedora folks to get a feel of this and what you think, before really going into implementation or any other details. More information at: https://sigstore.dev/what_is_sigstore/ https://github.com/sigstore/rekor -- Regards, Huzaifa Sidhpurwala / Red Hat Product Security ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
