Re: [gentoo-dev] Trustless Infrastructure
> On Mon, 2 Jul 2018, Jason A Donenfeld wrote: > Proposal: > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. Not possible, because there are some directories in profiles that must not contain any files other than those explicitly specified by PMS. And we cannot change that retroactively, because it would break existing implementations. (Also, is adding another 10 files a good idea? That's 10 inodes, and 400 MB on file systems with 4k blocks.) Ulrich pgpoFppXHtTxT.pgp Description: PGP signature
Re: [gentoo-dev] Trustless Infrastructure
On 07/02/2018 10:16 PM, Michał Górny wrote: > W dniu pon, 02.07.2018 o godzinie 17∶36 +0200, użytkownik Jason A. > Donenfeld napisał: >> Hey guys, >> >> While our infrastructure team has some nice technical competence, the >> recent disaster and ongoing embarrassing aftermath has made ever more >> urgent the need to have end-to-end signatures between developers and >> users. While the infrastructure team seems fairly impressive at >> deploying services and keeping the house running smoothly, I'd rather >> we don't place additional burden on them to do everything they're >> doing securely. Specifically, I'd like to ensure that 100% of Gentoo's >> infrastructure can be hacked, yet not backdoor a single witting user >> of the portage tree. Right now, as it stands, rsync distributes >> signatures to users that are derived from some >> infrastructure-controlled keys, not from the developers themselves. >> >> Proposal: >> - Sign every file in the portage tree so that it has a corresponding >> .asc. Repoman will need support for this. >> - Ensure the naming scheme of portage files is sufficiently strict, so >> that renaming or re-parenting signed files doesn't result in RCE. [*] >> - Distribute said .asc files with rsync per usual. >> > > Another problem: how do you prevent attacks based on removing files? > For example, let's say a MITM that removes new version of some packages > and related GLSAs in order to force the user to stay at vulnerable > version. > right, just to point out, this is already covered in the metamanifest signing scheme, but wouldn't be in a separate file signing mechanism. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Trustless Infrastructure
On 07/02/2018 07:47 PM, Hanno Böck wrote: > I don't want to say this is unworkable. But these are challenges and > imho fixing them all is really, I'll say it, it is unworkable, you need a trusted party doing verification of developers at point in time, then sign it with release engineering keys for distribution to end-users. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Trustless Infrastructure
On 07/02/2018 08:08 PM, Jason A. Donenfeld wrote: > On Mon, Jul 2, 2018 at 7:57 PM Rich Freeman wrote: >> This only helps you if a dev you don't trust is compromised. If a dev >> you trust is compromised, they can modify anything in the tree and >> you're hosed. > Yes indeed. This is more or less what we're aiming for. Putting the > trust in developers. The goal is for infra not to be the weak link in > this, as it currently is. > >> Sure, I'd prefer to not extract git signatures and just distribute via >> git purely without any rsync. > Yea, I personally don't really care much for rsync either. I've just > kind of been assuming this is a requirement of any gentoo solution. > But maybe this whole thing should take another dimension, and we > should instead talk about sunsetting rsync, and moving to a model of: > 1) git fetch, 2) git verify, 3) git checkout? There still might be > problems with "untrusting" devs, as I wrote above, but perhaps there's > room to grow within the git framework, by manually filtering commits > during checkout, or even by imposing ebuild directory signature-based > ACLs that I think you were hinting at before. So, sure, if you want to > call for an abolition of rsync, maybe I'd follow you in that direction > instead of the one here I'm proposing. > > picking a semi-random post to respond to, but the key management you're introducing with such a proposal is just silly. -- Kristian Fiskerstrand OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Trustless Infrastructure
Il 02/07/2018 17:36, Jason A. Donenfeld ha scritto: > Proposal: > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. > implementation detail: Adding an .asc file for every file would bring the total files from 130k to 260k. Would be possible to embed the signature _inside_ ebuilds and other common files to spare at least 80k from those 260k?
Re: [gentoo-dev] Trustless Infrastructure
W dniu pon, 02.07.2018 o godzinie 17∶36 +0200, użytkownik Jason A. Donenfeld napisał: > Hey guys, > > While our infrastructure team has some nice technical competence, the > recent disaster and ongoing embarrassing aftermath has made ever more > urgent the need to have end-to-end signatures between developers and > users. While the infrastructure team seems fairly impressive at > deploying services and keeping the house running smoothly, I'd rather > we don't place additional burden on them to do everything they're > doing securely. Specifically, I'd like to ensure that 100% of Gentoo's > infrastructure can be hacked, yet not backdoor a single witting user > of the portage tree. Right now, as it stands, rsync distributes > signatures to users that are derived from some > infrastructure-controlled keys, not from the developers themselves. > > Proposal: > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. > - Ensure the naming scheme of portage files is sufficiently strict, so > that renaming or re-parenting signed files doesn't result in RCE. [*] > - Distribute said .asc files with rsync per usual. > Another problem: how do you prevent attacks based on removing files? For example, let's say a MITM that removes new version of some packages and related GLSAs in order to force the user to stay at vulnerable version. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Trustless Infrastructure
W dniu pon, 02.07.2018 o godzinie 20∶01 +0200, użytkownik Jason A. Donenfeld napisał: > On Mon, Jul 2, 2018 at 7:21 PM Michał Górny wrote: > > > > W dniu pon, 02.07.2018 o godzinie 19∶01 +0200, użytkownik Jason A. > > Donenfeld napisał: > > > On Mon, Jul 2, 2018 at 6:58 PM Michał Górny wrote: > > > > - Have verification use a keyring of all Gentoo developers, with a > > > > > manual prompt to add new Gentoo developers to it. > > > > > > > > How are you going to distribute this keyring, and how are you going to > > > > protect attacker from injecting malicious key into it? > > > > > > Same model as Arch. > > > > > > > Please write it down here instead of expecting us to figure it out. > > It's your proposal, and it should be complete. > > I believe Arch's system relies on some core developers having master > keys and the revocation certificates being distributed amongst them: > > https://www.archlinux.org/master-keys/ > > Then all other developers are signed from there in one way or another. > It's kind of a modified web of trust. > > I don't know whether or not this is necessarily the best model to > emulate -- perhaps we could do better, for example -- but it does seem > like a good starting point. Instead we might prefer a single hardware > device somewhere. > > The idea would be -- portage fetches an updated "key list" from > somewhere. This new list of keys is considered if it is: a) signed by > the master keys and b) internally fulfills some WoT topological > requirements. Then, if these pass, it is up to the user to then > manually [y/N] the addition of new keys to the key ring. If they > suspect a particular developer has bad security practices, for > example, they could trivially [N] it, and then not have tree files he > touched copied from the shadow location to the portage directory. I'm afraid that in order to convince me you need to have a clear, well- defined model that improves security over the current solution. In other words, I see no purpose in adding a lot of complexity in order to shift the weakest link from one Infra machine handling the signing to another single point of failure in distributing the keys. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 11:47 AM, Jason A. Donenfeld wrote: > On Mon, Jul 2, 2018 at 6:02 PM R0b0t1 wrote: >> Signed hashes should be faster, no? Each directory with files could >> have a manifest. > > Signatures work over hashes of data, anyway. I think what you're > wondering, though, is the granularity of each signature? I'd recommend > this be done on the per-file level, since we wouldn't want gentoo devs > signing files in a directory they haven't actually inspected. For > example, eclasses. > Ah, okay then - I think at one time in the past GPG did something strange with file contents directly, or perhaps the implementation was just inefficient. It was maybe related to Debian where I first read about this? They were signing an .iso directly and found it was faster to hash it and then sign the hash. >> >> > - Ensure the naming scheme of portage files is sufficiently strict, so >> > that renaming or re-parenting signed files doesn't result in RCE. [*] >> > - Distribute said .asc files with rsync per usual. >> >> Rsync would work with this setup, but there is also webrsync-gpg in >> Portage right now. This covers the vast majority of usecases right >> now. > > Not sure whether you've missed the point or if you're responding to > something slightly different, but it's worth noting that both rsync > and webrsync-gpg right now check against infra signatures, rather than > developer signatures, and this is a big problem. Right, I lost track of that. The infrastructure or release developers are implicitly trusting all developers, so I suppose cutting out the middleman will reduce work.
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 2:23 PM Alec Warner wrote: > > We might reduce the complexity here by saying things like: > > "We have N trust levels" > I snipped most of your text because I didn't really see a natural way to include only some of it, though this really pertains to most of it, and I didn't want to quote the whole thing. I think that getting to a point where you can trust dev A and not trust dev B is probably something that should be viewed as a future goal, because the reality is that there are few devs you aren't depending on in some way. I do think that moving from trusting infra to trusting all the devs is an improvement though. Today you have to trust infra, and infra trusts all the devs, so you're implicitly trusting them anyway. That means that you can't distribute the tree without going through infra, and it means that infra keys are a weak point, and those keys are necessarily accessible by automated processes on machines that are network-reachable. Sure, they could be in HSMs/etc, but those HSMs still will sign whatever they're given by the automated process because there is no human in the loop. If you just go straight to verifying dev keys then infra is less of a bottleneck (it opens new distribution options without compromising security), and now all the keys are potentially in hardware modules that don't have to just blindly sign whatever they're given, and which are not online most of the time. That isn't perfect because a rooted box could probably still MITM the module, but it is an improvement. -- Rich
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 1:50 PM, Jason A. Donenfeld wrote: > On Mon, Jul 2, 2018 at 7:41 PM Rich Freeman wrote: > > To be fair, this is relying quite a bit on the last dev doing a commit > > to be checking his own tree, though that would certainly be helped if > > repoman/portage/etc were checking git sigs so that the starting tree > > is more likely to be clean. Also, if the last dev's key is > > compromised you're hosed no matter what as they can introduce anything > > to the tree and everybody will trust it. > > Actually, no. By keeping a local keychain of trusted developer keys -- > Arch Linux style -- you can just remove developers you don't trust, > and since the .asc files they signed will no longer verify, files from > them will no longer be copied into the real portage tree. It's a nice > way to make the entire process granular, yet still flexible. > Mostly re-posting my comments from irc (including typos!) 18:03 < antarus> zx2c4: my other problem with these scheme is simply a QA / functionality one 18:04 < antarus> zx2c4: we currently have tools that operate on the entire repo 18:04 < antarus> I'm not convinced we can really make any gaurantees about the subtree 18:04 < antarus> e.g. if my commits are not trusted for example; it might result in a tree that can't install any packages 18:05 < antarus> Which I mean, from some PoV is great (because antarus is a terrible human) 18:05 < antarus> but if at the end of the day you need to trust antarus to use Gentoo; I guess this leads you back to "why am I using gentoo", questoin 18:06 < antarus> (its nto a problem in practice because antarus doesn't have commit access to ::gentoo; but as a hypothetical, obviously 18:06 < antarus> It also means that support for anyone with this 'filtering' is a nightmare 18:06 < antarus> because who knows what subset of the tree their keyring filters In short I'm looking for a support and QA story here (how do we tell if a compilation failure is real, or just due to an incomplete tree?) We have similar efforts in QA (e.g. continuous integration projects) that verifies the dependency trees and those too require a consistent repo snapshot of the tree. We might reduce the complexity here by saying things like: "We have N trust levels" 0) We believe this person is using state of the art processes. ...) Some other level descriptor. N) This persons security stance is unknown. Then we can do stuff like run CI against each level (because we can compute the tree users will see on every commit) and it clarifies the support story somewhat. -A > > > The original point zx2c4 is making is reasonably valid. IMO it is > > much cleaner to just verify directly against the git history. If you > > wanted to verify the whole git history via rsync you're going to have > > to extract a lot more metadata from the git repo to reconstruct all of > > that down to the blobs, because otherwise the commit content hashes > > are referring to blobs that are no longer in the rsync tree. > > This kind of git-signature extraction also just sounds difficult and > fiddly to do, even if you do manage to figure out which partial bits > of metadata you need for reconstructing signed hashes. Whereas simply > adding .asc files is a single invocation of GPG -- a pretty > well-understood non fancy procedure. > > > I think a big question here is whether we just need to test the last > > dev's commit for normal user use. If so, then that simplifies things > > a lot, either for rsync or git syncing. If you want to test all the > > prior history then the rsync solution gets more complex if you want to > > leverage git sigs. However, I'd argue that with our current trust > > model the entire history is only as good as the last dev's commit > > anyway. Maybe if devs had a more limited scope of authority that a > > verification tool could take into account you'd get more from looking > > backwards in time (it would reject a commit to openrc by an X11 dev, > > as an example). > > I think you assessment of the situation here is incorrect. It's not > necessary to trust a developer for the entire state of the tree. You > only need to trust (or distrust) a developer for the files he's > changed. > > >From a user-story prospective, I'm curious how end users (e.g. people who are not you) are supposed to go about vetting developer keys. Is there a story there? -A
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:57 PM Rich Freeman wrote: > This only helps you if a dev you don't trust is compromised. If a dev > you trust is compromised, they can modify anything in the tree and > you're hosed. Yes indeed. This is more or less what we're aiming for. Putting the trust in developers. The goal is for infra not to be the weak link in this, as it currently is. > Sure, I'd prefer to not extract git signatures and just distribute via > git purely without any rsync. Yea, I personally don't really care much for rsync either. I've just kind of been assuming this is a requirement of any gentoo solution. But maybe this whole thing should take another dimension, and we should instead talk about sunsetting rsync, and moving to a model of: 1) git fetch, 2) git verify, 3) git checkout? There still might be problems with "untrusting" devs, as I wrote above, but perhaps there's room to grow within the git framework, by manually filtering commits during checkout, or even by imposing ebuild directory signature-based ACLs that I think you were hinting at before. So, sure, if you want to call for an abolition of rsync, maybe I'd follow you in that direction instead of the one here I'm proposing. > > Sure, but since any developer can change any file, that is basically > the same thing. If somebody steals a single dev's key they can just > rootkit any file of their choosing, sign that one change, and it will > slip through any of these methods. The only protection against that > is tracking who is allowed to touch what files, and then you're still > hosed if a dev for a widely-used file gets their key stolen. As stated above, yes, this is the intended model. And I think it's considerably better than the current state of affairs. The threats get even less scary as we convince Nitrokey or Yubikey to send all gentoo developers free devices, and then mandate keys remain on those devices and never get copied, etc etc.
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:21 PM Michał Górny wrote: > > W dniu pon, 02.07.2018 o godzinie 19∶01 +0200, użytkownik Jason A. > Donenfeld napisał: > > On Mon, Jul 2, 2018 at 6:58 PM Michał Górny wrote: > > > - Have verification use a keyring of all Gentoo developers, with a > > > > manual prompt to add new Gentoo developers to it. > > > > > > How are you going to distribute this keyring, and how are you going to > > > protect attacker from injecting malicious key into it? > > > > Same model as Arch. > > > > Please write it down here instead of expecting us to figure it out. > It's your proposal, and it should be complete. I believe Arch's system relies on some core developers having master keys and the revocation certificates being distributed amongst them: https://www.archlinux.org/master-keys/ Then all other developers are signed from there in one way or another. It's kind of a modified web of trust. I don't know whether or not this is necessarily the best model to emulate -- perhaps we could do better, for example -- but it does seem like a good starting point. Instead we might prefer a single hardware device somewhere. The idea would be -- portage fetches an updated "key list" from somewhere. This new list of keys is considered if it is: a) signed by the master keys and b) internally fulfills some WoT topological requirements. Then, if these pass, it is up to the user to then manually [y/N] the addition of new keys to the key ring. If they suspect a particular developer has bad security practices, for example, they could trivially [N] it, and then not have tree files he touched copied from the shadow location to the portage directory.
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 1:50 PM Jason A. Donenfeld wrote: > > On Mon, Jul 2, 2018 at 7:41 PM Rich Freeman wrote: > > To be fair, this is relying quite a bit on the last dev doing a commit > > to be checking his own tree, though that would certainly be helped if > > repoman/portage/etc were checking git sigs so that the starting tree > > is more likely to be clean. Also, if the last dev's key is > > compromised you're hosed no matter what as they can introduce anything > > to the tree and everybody will trust it. > > Actually, no. By keeping a local keychain of trusted developer keys -- > Arch Linux style -- you can just remove developers you don't trust, > and since the .asc files they signed will no longer verify, files from > them will no longer be copied into the real portage tree. It's a nice > way to make the entire process granular, yet still flexible. This only helps you if a dev you don't trust is compromised. If a dev you trust is compromised, they can modify anything in the tree and you're hosed. > > This kind of git-signature extraction also just sounds difficult and > fiddly to do, even if you do manage to figure out which partial bits > of metadata you need for reconstructing signed hashes. Whereas simply > adding .asc files is a single invocation of GPG -- a pretty > well-understood non fancy procedure. Sure, I'd prefer to not extract git signatures and just distribute via git purely without any rsync. > I think you assessment of the situation here is incorrect. It's not > necessary to trust a developer for the entire state of the tree. You > only need to trust (or distrust) a developer for the files he's > changed. Sure, but since any developer can change any file, that is basically the same thing. If somebody steals a single dev's key they can just rootkit any file of their choosing, sign that one change, and it will slip through any of these methods. The only protection against that is tracking who is allowed to touch what files, and then you're still hosed if a dev for a widely-used file gets their key stolen. I still prefer it to trusting infra though, because: 1. The same attacks still work even with a final infra signoff. 2. The dev signoffs could use hardware tokens where a random server process doesn't have the ability to trigger the signoffs (think hardware button on a yubikey/etc). 3. This enables more mirroring options - like sticking the whole repo on github/etc for direct syncing that is still completely secure due to signoffs. -- Rich
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:48 PM Hanno Böck wrote: > Hi, Oh, thank you for arriving! I've been trying to ping you the last few days hoping you'd pop in here. :) > Something like this was I believe the original idea behind signed > manifests. Not sure how long ago this was, but we used to sign Manifest > files at some point, though it never was part of any consistent concept > as far as I know, and they weren't checked regularly. Right, but I believe the manifest signatures had some limitations, with eclasses and with overly broad granularity. Whereas adding a .asc for each individual file as it's edited is really quite simple. > Anything like this comes with some obvious problems that you need to > answer if you want to have such a system: > * How are you keeping the keys up to date? Which keys are included > there? All currently active developers? All active and former > developers? > * What happens if a key expires? Do you accept expired signatures if > the package has been committed before the expiration date? Or is > there some kind of resign process if that happens? Does the developer > have to do this himself or can other developers do this? If it's up > on the developer what happens if he's inactive / on long holiday / > not reachable when his key expires? > * What happens if a key is revoked, because a developer decides to > create a new key? Same question as with expired keys: Do all > signatures need to be recreated? How's that going to happen? > * What happens if a developer leaves Gentoo? We'll still want to have > his packages. Again a resign procedure? > > I don't want to say this is unworkable. But these are challenges and > imho fixing them all is really, really tricky. Either you break stuff > regularly or you have procedures that someone has to do regularly in > order to avoid breakage (more work for gentoo devs) or you expand the > scope of accepted signatures very excessively. I think the answer to these questions is to start out fairly restrictive and see how it goes and what sorts of easing is needed in practice. For example, it might work fine to let keys expire, to let developers retire and have their keys removed, and so forth, and simply have some nice monitoring infrastructure and alert tools (repoman full, for example) so that some developer somewhere re-ups the signature after taking a glance. Jason
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:41 PM Rich Freeman wrote: > To be fair, this is relying quite a bit on the last dev doing a commit > to be checking his own tree, though that would certainly be helped if > repoman/portage/etc were checking git sigs so that the starting tree > is more likely to be clean. Also, if the last dev's key is > compromised you're hosed no matter what as they can introduce anything > to the tree and everybody will trust it. Actually, no. By keeping a local keychain of trusted developer keys -- Arch Linux style -- you can just remove developers you don't trust, and since the .asc files they signed will no longer verify, files from them will no longer be copied into the real portage tree. It's a nice way to make the entire process granular, yet still flexible. > The original point zx2c4 is making is reasonably valid. IMO it is > much cleaner to just verify directly against the git history. If you > wanted to verify the whole git history via rsync you're going to have > to extract a lot more metadata from the git repo to reconstruct all of > that down to the blobs, because otherwise the commit content hashes > are referring to blobs that are no longer in the rsync tree. This kind of git-signature extraction also just sounds difficult and fiddly to do, even if you do manage to figure out which partial bits of metadata you need for reconstructing signed hashes. Whereas simply adding .asc files is a single invocation of GPG -- a pretty well-understood non fancy procedure. > I think a big question here is whether we just need to test the last > dev's commit for normal user use. If so, then that simplifies things > a lot, either for rsync or git syncing. If you want to test all the > prior history then the rsync solution gets more complex if you want to > leverage git sigs. However, I'd argue that with our current trust > model the entire history is only as good as the last dev's commit > anyway. Maybe if devs had a more limited scope of authority that a > verification tool could take into account you'd get more from looking > backwards in time (it would reject a commit to openrc by an X11 dev, > as an example). I think you assessment of the situation here is incorrect. It's not necessary to trust a developer for the entire state of the tree. You only need to trust (or distrust) a developer for the files he's changed.
Re: [gentoo-dev] Trustless Infrastructure
Hi, Something like this was I believe the original idea behind signed manifests. Not sure how long ago this was, but we used to sign Manifest files at some point, though it never was part of any consistent concept as far as I know, and they weren't checked regularly. Anything like this comes with some obvious problems that you need to answer if you want to have such a system: * How are you keeping the keys up to date? Which keys are included there? All currently active developers? All active and former developers? * What happens if a key expires? Do you accept expired signatures if the package has been committed before the expiration date? Or is there some kind of resign process if that happens? Does the developer have to do this himself or can other developers do this? If it's up on the developer what happens if he's inactive / on long holiday / not reachable when his key expires? * What happens if a key is revoked, because a developer decides to create a new key? Same question as with expired keys: Do all signatures need to be recreated? How's that going to happen? * What happens if a developer leaves Gentoo? We'll still want to have his packages. Again a resign procedure? I don't want to say this is unworkable. But these are challenges and imho fixing them all is really, really tricky. Either you break stuff regularly or you have procedures that someone has to do regularly in order to avoid breakage (more work for gentoo devs) or you expand the scope of accepted signatures very excessively. And I believe these challenges are one of the reasons the old attempts to have a signed Gentoo never went anywhere. I'm glad we have some form of signed Gentoo now, even if it relies on some centralized infrastructure. -- Hanno Böck https://hboeck.de/ mail/jabber: ha...@hboeck.de GPG: FE73757FA60E4E21B937579FA5880072BBB51E42
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:44 PM Mariusz Ceier wrote: > I wonder how would the local changes to ebuilds be handled ? > Currently it's possible to change ebuild, do "ebuild package.ebuild > manifest" and emerge package - will this still be supported ? My proposal is to fit in at the layer of sync'ing -- so that only files that verify get merged into the real portage tree. This, therefore, wouldn't interfere with anything else you felt inclined to stick in the portage tree yourself.
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 7:23 PM Matthias Maier wrote: > stores tree snapshots (and not differences). So all you need is exactly > one signed commit to verify that > > - this is the full repository tree the developer saw at the time of the >commit > - this is the full history the developer saw at the time of the commit I'm not sure this is as good, though. I don't think all developers verify the whole tree before adding a signature on top. And this leaves out file-level granularity, so I can't choose to distrust a certain set of developers I know to have poor security practices and have .asc files from those developers simply not verify. With the extracted-git model, it winds up being "the most recent developer signs everything for everybody". This is a bit weaker than what I've proposed.
Re: [gentoo-dev] Trustless Infrastructure
Hi, I wonder how would the local changes to ebuilds be handled ? Currently it's possible to change ebuild, do "ebuild package.ebuild manifest" and emerge package - will this still be supported ? If not, would there be any plans for a tool for creating trusted local overlays ? This feature is useful when fixing ebuilds or adding new versions by copying existing ones locally. Best regards, Mariusz Ceier On 2 July 2018 at 17:36, Jason A. Donenfeld wrote: > Hey guys, > > While our infrastructure team has some nice technical competence, the > recent disaster and ongoing embarrassing aftermath has made ever more > urgent the need to have end-to-end signatures between developers and > users. While the infrastructure team seems fairly impressive at > deploying services and keeping the house running smoothly, I'd rather > we don't place additional burden on them to do everything they're > doing securely. Specifically, I'd like to ensure that 100% of Gentoo's > infrastructure can be hacked, yet not backdoor a single witting user > of the portage tree. Right now, as it stands, rsync distributes > signatures to users that are derived from some > infrastructure-controlled keys, not from the developers themselves. > > Proposal: > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. > - Ensure the naming scheme of portage files is sufficiently strict, so > that renaming or re-parenting signed files doesn't result in RCE. [*] > - Distribute said .asc files with rsync per usual. > - Never rsync into the /usr/portage directory, but rather into an > unused shadow directory, and only copy files from the shadow directory > into /usr/portage after verification succeeds. (The fact that those > files are visible to portage prior to verification and following a > failed verification is a shameful oversight of the current system.) > - Have verification use a keyring of all Gentoo developers, with a > manual prompt to add new Gentoo developers to it. > - Eventually acquire (sponsors?) nitrokeys/yubikeys for all > developers, and mandate everyone stores their Gentoo key material in > there exclusively. > > This is very similar to what Arch Linux is doing, and AFAIK, it works > well there. I'm sure this list will want to bikeshed over the > particulars of the implementation, but the two design goals from this > are: > > - Signatures are made by developers, not by infra. > - Portage doesn't see any files that haven't yet been verified. > > Regarding the [*] comment above about the directory tree. There might > be more clever ways of handling this, like having the last developer > who modified the tree compute some kind of holistic signature, in > addition to each of the individual ones. Or, perhaps, this is the one > place where we would consider relying on infrakeys, provided portage > isn't victim to clever RCE by rearrangement. Other attacks include, of > course, downgrades and DoS. > > Hopefully we can move Gentoo's portage tree security up to modern > security expectations, and no longer be forced to trust vulnerable > sitting-duck infra machines for our signatures. > > Jason >
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 1:23 PM Matthias Maier wrote: > > > On Mon, Jul 2, 2018, at 12:01 CDT, "Jason A. Donenfeld" > wrote: > > > Aren't git signatures done over the full commit objects? Meaning you'd > > need the entire tree of metadata and thus all commits in order to > > verify? Or do you see some clever opportunity for extracting just > > enough metadata that you could actually have a file-based, rather than > > commit-based, verification? > > > Git signatures are over the full commit object - and the commit contains > a hash of the root of the full repository tree. Git internally only > stores tree snapshots (and not differences). So all you need is exactly > one signed commit to verify that > > - this is the full repository tree the developer saw at the time of the >commit > > - this is the full history the developer saw at the time of the commit > To be fair, this is relying quite a bit on the last dev doing a commit to be checking his own tree, though that would certainly be helped if repoman/portage/etc were checking git sigs so that the starting tree is more likely to be clean. Also, if the last dev's key is compromised you're hosed no matter what as they can introduce anything to the tree and everybody will trust it. The original point zx2c4 is making is reasonably valid. IMO it is much cleaner to just verify directly against the git history. If you wanted to verify the whole git history via rsync you're going to have to extract a lot more metadata from the git repo to reconstruct all of that down to the blobs, because otherwise the commit content hashes are referring to blobs that are no longer in the rsync tree. I think a big question here is whether we just need to test the last dev's commit for normal user use. If so, then that simplifies things a lot, either for rsync or git syncing. If you want to test all the prior history then the rsync solution gets more complex if you want to leverage git sigs. However, I'd argue that with our current trust model the entire history is only as good as the last dev's commit anyway. Maybe if devs had a more limited scope of authority that a verification tool could take into account you'd get more from looking backwards in time (it would reject a commit to openrc by an X11 dev, as an example). I'm just trying to avoid going back to the old manifest solution when git seems a lot cleaner otherwise. -- Rich
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018, at 12:01 CDT, "Jason A. Donenfeld" wrote: > Aren't git signatures done over the full commit objects? Meaning you'd > need the entire tree of metadata and thus all commits in order to > verify? Or do you see some clever opportunity for extracting just > enough metadata that you could actually have a file-based, rather than > commit-based, verification? Git signatures are over the full commit object - and the commit contains a hash of the root of the full repository tree. Git internally only stores tree snapshots (and not differences). So all you need is exactly one signed commit to verify that - this is the full repository tree the developer saw at the time of the commit - this is the full history the developer saw at the time of the commit Meaning, our current tree signing practice already ensures that - history cannot be tampered with - allows for a complete audit log (in buzzspeak, we're doing blockchain verification *SCNR*) Best, Matthias
Re: [gentoo-dev] Trustless Infrastructure
W dniu pon, 02.07.2018 o godzinie 19∶01 +0200, użytkownik Jason A. Donenfeld napisał: > On Mon, Jul 2, 2018 at 6:58 PM Michał Górny wrote: > > - Have verification use a keyring of all Gentoo developers, with a > > > manual prompt to add new Gentoo developers to it. > > > > How are you going to distribute this keyring, and how are you going to > > protect attacker from injecting malicious key into it? > > Same model as Arch. > Please write it down here instead of expecting us to figure it out. It's your proposal, and it should be complete. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 6:58 PM Michał Górny wrote: > - Have verification use a keyring of all Gentoo developers, with a > > manual prompt to add new Gentoo developers to it. > > How are you going to distribute this keyring, and how are you going to > protect attacker from injecting malicious key into it? Same model as Arch.
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 6:55 PM Rich Freeman wrote: > You might want to read what I wrote then, because I proposed options > for using the git signatures over rsync, as well as for with git > syncing > having a tool that extracts the git > signatures and stores the metadata in the repo (ideally done by infra > before mirroring, but it could be done after the fact as well) Aren't git signatures done over the full commit objects? Meaning you'd need the entire tree of metadata and thus all commits in order to verify? Or do you see some clever opportunity for extracting just enough metadata that you could actually have a file-based, rather than commit-based, verification?
Re: [gentoo-dev] Trustless Infrastructure
W dniu pon, 02.07.2018 o godzinie 17∶36 +0200, użytkownik Jason A. Donenfeld napisał: > - Have verification use a keyring of all Gentoo developers, with a > manual prompt to add new Gentoo developers to it. How are you going to distribute this keyring, and how are you going to protect attacker from injecting malicious key into it? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 12:43 PM Jason A. Donenfeld wrote: > > There's a lot of text there, and rather than trying to parse all of > that, I'll just reiterate a primary important design goal that might > be overlooked: > > - End to end signatures from the developer to the user. > > This means that no matter the operation infra does before shipping it > out to the user, the user still needs to verify that the packages came > from the developers. In other words, whatever complicated mechanism > you propose, it needs to not rely on trusting infra to hold onto any > secrets. For example, I don't know whether this is attainable with the > the git signatures alone, without requiring users to sync the entire > git repository, which might not be acceptable for some. > You might want to read what I wrote then, because I proposed options for using the git signatures over rsync, as well as for with git syncing (which IMO is an under-rated option, as there is no need to preserve all the past commits in the repository once it is known-good, which I also mention). Everything I wrote is possible with dev signatures only, and would work fine over a completely compromised infrastructure, or non-Gentoo infrastructure. Granted, if you want to do git sigs over rsync you do need an untrusted program to do the extractions and if that fails it would be a DOS. If you think that one of my alternatives fails to meet your goals I'd be interested in hearing about it. -- Rich
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 6:02 PM R0b0t1 wrote: > Signed hashes should be faster, no? Each directory with files could > have a manifest. Signatures work over hashes of data, anyway. I think what you're wondering, though, is the granularity of each signature? I'd recommend this be done on the per-file level, since we wouldn't want gentoo devs signing files in a directory they haven't actually inspected. For example, eclasses. > > > - Ensure the naming scheme of portage files is sufficiently strict, so > > that renaming or re-parenting signed files doesn't result in RCE. [*] > > - Distribute said .asc files with rsync per usual. > > Rsync would work with this setup, but there is also webrsync-gpg in > Portage right now. This covers the vast majority of usecases right > now. Not sure whether you've missed the point or if you're responding to something slightly different, but it's worth noting that both rsync and webrsync-gpg right now check against infra signatures, rather than developer signatures, and this is a big problem.
Re: [gentoo-dev] Trustless Infrastructure
Hello Rich, There's a lot of text there, and rather than trying to parse all of that, I'll just reiterate a primary important design goal that might be overlooked: - End to end signatures from the developer to the user. This means that no matter the operation infra does before shipping it out to the user, the user still needs to verify that the packages came from the developers. In other words, whatever complicated mechanism you propose, it needs to not rely on trusting infra to hold onto any secrets. For example, I don't know whether this is attainable with the the git signatures alone, without requiring users to sync the entire git repository, which might not be acceptable for some. Jason
Re: [gentoo-dev] Trustless Infrastructure
On Mon, 2 Jul 2018 11:01:58 -0500 R0b0t1 wrote: > On Mon, Jul 2, 2018 at 10:36 AM, Jason A. Donenfeld > wrote: > > Hey guys, > > > > While our infrastructure team has some nice technical competence, > > the recent disaster and ongoing embarrassing aftermath has made > > ever more urgent the need to have end-to-end signatures between > > developers and users. While the infrastructure team seems fairly > > impressive at deploying services and keeping the house running > > smoothly, I'd rather we don't place additional burden on them to do > > everything they're doing securely. Specifically, I'd like to ensure > > that 100% of Gentoo's infrastructure can be hacked, yet not > > backdoor a single witting user of the portage tree. Right now, as > > it stands, rsync distributes signatures to users that are derived > > from some infrastructure-controlled keys, not from the developers > > themselves. > > > > Proposal: > > - Sign every file in the portage tree so that it has a corresponding > > .asc. Repoman will need support for this. > > Signed hashes should be faster, no? Each directory with files could > have a manifest. > > > - Ensure the naming scheme of portage files is sufficiently strict, > > so that renaming or re-parenting signed files doesn't result in > > RCE. [*] > > - Distribute said .asc files with rsync per usual. > > Rsync would work with this setup, but there is also webrsync-gpg in > Portage right now. This covers the vast majority of usecases right > now. There is often no need to sync more than once per day. > > Speaking of, the keys for that have lapsed. Will they be updated? > > Cheers, > R0b0t1 > app-crypt/gentoo-keys has been updated yesterday. with renewed interest in gpg signatures, I will endeavor to keep it updated until is fully automated. -- Brian Dolbec
Re: [gentoo-dev] Trustless Infrastructure
Overall I very much like the concept, but I might propose a few tweaks (just quoting the stuff that might benefit from adjustment): On Mon, Jul 2, 2018 at 11:36 AM Jason A. Donenfeld wrote: > > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. > - Ensure the naming scheme of portage files is sufficiently strict, so > that renaming or re-parenting signed files doesn't result in RCE. [*] > - Distribute said .asc files with rsync per usual. This has two issues: 1. It requires changes to the repos/infra/etc to work, which means it is painful to just pilot/etc and grow organically. 2. It is 99% redundant with the git signatures we already have. Why not build this off of git signatures? This could be done directly by syncing via git, or by having a tool that extracts the git signatures and stores the metadata in the repo (ideally done by infra before mirroring, but it could be done after the fact as well). Git is just content-hashed, so as long as the files aren't modified you should be able to verify the git content hash against a repo synced outside of git, assuming no modifications (obviously this means accounting for stuff like metadata that infra adds after the fact). You still need a solution for metadata in your original proposal anyway. The only downside I see to git signatures is how far back to go to check history. With the .asc solution you'd remove the signatures when you remove the files they pertain to. With git there is no trivial way to know when to stop going back with the signature verfiication since every signature applies to a mix of both current and subsequently-removed files, with the percentage of each slowly shifting as you go back further in time. That said if you just track the last known-good sync you could just check the subsequent signatures, which would be very efficient (probably more efficient than checking all of rsync unless you sync very infrequently). > - Never rsync into the /usr/portage directory, but rather into an > unused shadow directory, and only copy files from the shadow directory > into /usr/portage after verification succeeds. (The fact that those > files are visible to portage prior to verification and following a > failed verification is a shameful oversight of the current system.) I certainly agree that /usr/portage being usable if it fails verification is a major weakness right now. Other alternatives to your proposal include: 1. Store state somewhere that portage checks. It is invalidated before starting a sync, and set back to "secure" after verification. 2. Store a last-known-good hash if using git signature checking. Portage would check the current tree state against this in all operations. 3. Have portage check signatures on all files it access at the time of access. This would make portage safe to use even in a compromised tree. Especially options 1/2 are going to be more efficient than copying files at the filesystem level from a scratch location. Also, all three options would be compatible with git syncing, while trying to copy a git repo after the sync would probably be messier (though still possible). But, I have no objection to your original proposal either - I'd prefer it to what we have today at least for rsync. In general I do advocate giving serious consideration to the benefits of syncing via git. If you sync frequently (which most Gentoo users probably do, and which we generally advocate), then it tends to be a lot more efficient than rsync. It naturally tracks changes over time as well, so it fits in very well with merging untrusted changes into a known-good tree, as only the changes need to be verified. The main downside to git signature checking is sha1. It baffles me that nobody has bothered to fix this, especially since I'd think it would be pretty simple to do. Just designate new tree/blob/parent record types that use the new hash - like tree256/blob256/parent256. Git would use the appropriate hash when following references, so you could have continuity in a repository with older sha1 commits and newer sha256 ones. Obviously the newer repos would be incompatible with older versions of git, but anything like this would be phased in, and updating git isn't particularly painful. Projects that care about the security could consider rebasing the entire thing, but that would of course discard history. Presumably you could even do a merge where one branch of the merge is the original untouched sha1 commits, and the other branch is the rebased sha-256 commits, and the merge ties them together into a forward-going sha256 history. -- Rich
Re: [gentoo-dev] Trustless Infrastructure
On Mon, Jul 2, 2018 at 10:36 AM, Jason A. Donenfeld wrote: > Hey guys, > > While our infrastructure team has some nice technical competence, the > recent disaster and ongoing embarrassing aftermath has made ever more > urgent the need to have end-to-end signatures between developers and > users. While the infrastructure team seems fairly impressive at > deploying services and keeping the house running smoothly, I'd rather > we don't place additional burden on them to do everything they're > doing securely. Specifically, I'd like to ensure that 100% of Gentoo's > infrastructure can be hacked, yet not backdoor a single witting user > of the portage tree. Right now, as it stands, rsync distributes > signatures to users that are derived from some > infrastructure-controlled keys, not from the developers themselves. > > Proposal: > - Sign every file in the portage tree so that it has a corresponding > .asc. Repoman will need support for this. Signed hashes should be faster, no? Each directory with files could have a manifest. > - Ensure the naming scheme of portage files is sufficiently strict, so > that renaming or re-parenting signed files doesn't result in RCE. [*] > - Distribute said .asc files with rsync per usual. Rsync would work with this setup, but there is also webrsync-gpg in Portage right now. This covers the vast majority of usecases right now. There is often no need to sync more than once per day. Speaking of, the keys for that have lapsed. Will they be updated? Cheers, R0b0t1
[gentoo-dev] Trustless Infrastructure
Hey guys, While our infrastructure team has some nice technical competence, the recent disaster and ongoing embarrassing aftermath has made ever more urgent the need to have end-to-end signatures between developers and users. While the infrastructure team seems fairly impressive at deploying services and keeping the house running smoothly, I'd rather we don't place additional burden on them to do everything they're doing securely. Specifically, I'd like to ensure that 100% of Gentoo's infrastructure can be hacked, yet not backdoor a single witting user of the portage tree. Right now, as it stands, rsync distributes signatures to users that are derived from some infrastructure-controlled keys, not from the developers themselves. Proposal: - Sign every file in the portage tree so that it has a corresponding .asc. Repoman will need support for this. - Ensure the naming scheme of portage files is sufficiently strict, so that renaming or re-parenting signed files doesn't result in RCE. [*] - Distribute said .asc files with rsync per usual. - Never rsync into the /usr/portage directory, but rather into an unused shadow directory, and only copy files from the shadow directory into /usr/portage after verification succeeds. (The fact that those files are visible to portage prior to verification and following a failed verification is a shameful oversight of the current system.) - Have verification use a keyring of all Gentoo developers, with a manual prompt to add new Gentoo developers to it. - Eventually acquire (sponsors?) nitrokeys/yubikeys for all developers, and mandate everyone stores their Gentoo key material in there exclusively. This is very similar to what Arch Linux is doing, and AFAIK, it works well there. I'm sure this list will want to bikeshed over the particulars of the implementation, but the two design goals from this are: - Signatures are made by developers, not by infra. - Portage doesn't see any files that haven't yet been verified. Regarding the [*] comment above about the directory tree. There might be more clever ways of handling this, like having the last developer who modified the tree compute some kind of holistic signature, in addition to each of the individual ones. Or, perhaps, this is the one place where we would consider relying on infrakeys, provided portage isn't victim to clever RCE by rearrangement. Other attacks include, of course, downgrades and DoS. Hopefully we can move Gentoo's portage tree security up to modern security expectations, and no longer be forced to trust vulnerable sitting-duck infra machines for our signatures. Jason
