Re: Linux 5.3-rc8
On 9/10/19 12:21 AM, Ahmed S. Darwish wrote: > Can this even be considered a user-space breakage? I'm honestly not > sure. On my modern RDRAND-capable x86, just running rng-tools rngd(8) > early-on fixes the problem. I'm not sure about the status of older > CPUs though. Tangent: I asked aloud on Twitter last night if anyone had exploited Rowhammer-like effects to generate entropy...and sure enough, the usual suspects have: https://arxiv.org/pdf/1808.04286.pdf While this requires low level access to a memory controller, it's perhaps an example of something a platform designer could look at as a source to introduce boot-time entropy for e.g. EFI_RNG_PROTOCOL even on an existing platform without dedicated hardware for the purpose. Just a thought. Jon.
Re: Linux 5.3-rc8
On 9/10/19 12:21 AM, Ahmed S. Darwish wrote: > Can this even be considered a user-space breakage? I'm honestly not > sure. On my modern RDRAND-capable x86, just running rng-tools rngd(8) > early-on fixes the problem. I'm not sure about the status of older > CPUs though. Tangent: I asked aloud on Twitter last night if anyone had exploited Rowhammer-like effects to generate entropy...and sure enough, the usual suspects have: https://arxiv.org/pdf/1808.04286.pdf While this requires low level access to a memory controller, it's perhaps an example of something a platform designer could look at as a source to introduce boot-time entropy for e.g. EFI_RNG_PROTOCOL even on an existing platform without dedicated hardware for the purpose. Just a thought. Jon.
Re: Linux 5.3-rc8
On Fr, 27.09.19 08:58, Linus Torvalds (torva...@linux-foundation.org) wrote: > On Fri, Sep 27, 2019 at 6:57 AM Lennart Poettering > wrote: > > > > Doing the random seed in the boot loader is nice for two reasons: > > > > 1. It runs very very early, so that the OS can come up with fully > >initialized entropy right from the beginning. > > Oh, that part I love. > > But I don't believe in your second case: > > > 2. The boot loader generally has found some disk to read the kernel from, > >i.e. has a place where stuff can be stored and which can be updated > >(most modern boot loaders can write to disk these days, and so can > >EFI). Thus, it can derive a new random seed from a stored seed on disk > >and pass it to the OS *AND* update it right away on disk ensuring that > >it is never reused again. > > No. This is absolutely no different at all from user space doing it > early with a file. > > All the same "golden image" issues exist, and in general the less the > boot loader writes to disk, the better. > > Plus it doesn't actually work anyway in the one situation where people > _really_ want it - embedded devices, where the kernel image is quite > possibly in read-only flash that needs major setup for updates. > > PLUS. > > Your "it can update it right away on disk" is just crazy talk. With > WHAT? It has no randomness to play with, and it doesn't have time to > do jitter entropy stuff. So these two issues are addressed by the logic implemented in sd-boot (systemd's boot loader) like this: The old seed is read off the ESP seed file. We then calculate two hash sums in counter mode from it (SHA256), one we pass to the OS as seed to initialize the random pool from. The other we use to update the ESP seed file with. Unless you are capable of breaking SHA256 this means the seed passed to the OS and the new seed stored on disk are derived from the same seed but in a way you cannot determine one if you managed to learn the other. Moreover, on each boot you are guaranteed to get two new seeds, each time, and you cannot derive the sums used on previous boots from those. This means we are robust towards potential seed reuse when turning the system forcibly off during boot. Now, what's still missing in the above is protection against "golden image" issues, as you correctly pointed out. To deal with that the SHA256 sums are not just hashed from the old seed and the counter, but also include a system specific "system token" (you may also call it "salt") which is stored in an EFI variable, persistently, which was created once, during system installation. This hence gives you the behaviour your are looking for, using the NVRAM like you suggested, but we don't need to write the EFI vars all the time, as instead we update the seed file stored in the ESP each time, and updating the ESP should be safer and less problematic (i.e. if everything is done right it's a single sector write). To make this safer, on EFI firmwares that support the RNG protocol we also include some data derived from that in the hash, just for good measure. To sumarize: NEWDISKSEED = SHA256(OLDDISKSEED || SYSTEMTOKEN || EFIRNGVAL || "1") SEEDFORLINUX = SHA256(OLDDISKSEED || SYSTEMTOKEN || EFIRNGVAL || "2") (and no, this is not a crypto scheme I designed, but something Dr. Bertram Poettering (my brother, a cryptographer) suggested) > So all it can do is a really bad job at taking the previous random > seed, doing some transformation on it, and add a little bit of > whatever system randomness it can find. None of which is any better > than what the kernel can do. Well, the kernel cannot hash and rewrite the old seed file early enough, it's that simple. It can do that only when /var becomes writable, i.e. very late during boot, much later than when we need entropy for. The boot loader on the hand, can hash and rewrite the old seed file even before the kernel initializes, and that's the big benefit! > End result: you'd need to have the kernel update whatever bootloader > data later on, and I'm not seeing that happening. Afaik the current > bootloader interface has no way to specify how to update it when you > actually have better randomness. So, you could, but don't have to update the ESP random seed file from the OS too, every now and then, but the security of the model dos not rely on that. (And yes, the above doesn't help if you have a fully R/O medium, but those tend to be embedded devices, and I am much less concerned about those, the designers really can deal with the RNG seed issues themselves, and maybe provide some hw to do it; it's the generic user PCs that we should be concerned about, and for those the above should generally work) Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Fri, Sep 27, 2019 at 6:57 AM Lennart Poettering wrote: > > Doing the random seed in the boot loader is nice for two reasons: > > 1. It runs very very early, so that the OS can come up with fully >initialized entropy right from the beginning. Oh, that part I love. But I don't believe in your second case: > 2. The boot loader generally has found some disk to read the kernel from, >i.e. has a place where stuff can be stored and which can be updated >(most modern boot loaders can write to disk these days, and so can >EFI). Thus, it can derive a new random seed from a stored seed on disk >and pass it to the OS *AND* update it right away on disk ensuring that >it is never reused again. No. This is absolutely no different at all from user space doing it early with a file. All the same "golden image" issues exist, and in general the less the boot loader writes to disk, the better. Plus it doesn't actually work anyway in the one situation where people _really_ want it - embedded devices, where the kernel image is quite possibly in read-only flash that needs major setup for updates. PLUS. Your "it can update it right away on disk" is just crazy talk. With WHAT? It has no randomness to play with, and it doesn't have time to do jitter entropy stuff. So all it can do is a really bad job at taking the previous random seed, doing some transformation on it, and add a little bit of whatever system randomness it can find. None of which is any better than what the kernel can do. End result: you'd need to have the kernel update whatever bootloader data later on, and I'm not seeing that happening. Afaik the current bootloader interface has no way to specify how to update it when you actually have better randomness. > NVRAM backing EFI vars sucks. Nothing you want to update on every > cycle. It's OK to update during OS installation, but during every > single boot? I'd rather not. I do agree that EFI nvram isn't wonderful, but hopefully nonvolatile storage is improving, and it's conceptually the right thing. Linus
Re: Linux 5.3-rc8
On Mi, 18.09.19 13:26, Linus Torvalds (torva...@linux-foundation.org) wrote: > On Wed, Sep 18, 2019 at 1:15 PM Alexander E. Patrakov > wrote: > > > > No, this is not the solution, if we take seriously not only getrandom > > hangs, but also urandom warnings. In some setups (root on LUKS is one of > > them) they happen early in the initramfs. Therefore "restoring" entropy > > from the previous boot by a script that runs from the main system is too > > late. That's why it is suggested to load at least a part of the random > > seed in the boot loader, and that has not been commonly implemented. > > Honestly, I think the bootloader suggestion is naive and silly too. > > Yes, we now support it. And no, I don't think people will trust that > either. And I suspect for good reason: there's really very little > reason to believe that bootloaders would be any better than any other > part of the system. > > So right now some people trust bootloaders exactly _because_ there > basically is just one or two that do this, and the people who use them > are usually the people who wrote them or are at least closely > associated with them. That will change, and then people will say "why > would I trust that, when we know of bug Xyz". Doing the random seed in the boot loader is nice for two reasons: 1. It runs very very early, so that the OS can come up with fully initialized entropy right from the beginning. 2. The boot loader generally has found some disk to read the kernel from, i.e. has a place where stuff can be stored and which can be updated (most modern boot loaders can write to disk these days, and so can EFI). Thus, it can derive a new random seed from a stored seed on disk and pass it to the OS *AND* update it right away on disk ensuring that it is never reused again. The point where the OS kernel comes to an equivalent point where it can write to disk is much much later, i.e. after the initrd, after the transition to the actual OS, ony after /var has been remounted writable. So to me this is not about trust, but about "first place we can read *AND* write a seed on disk". i.e. the key to grok here: it's not OK to use a stored seed unless you can at the same time update the it on disk, as only that protects you from reusing the key if the system's startup is aborted due to power failure or such. > Adding an EFI variable (or other platform nonvolatile thing), and > reading (and writing to it) purely from the kernel ends up being one > of those things where you can then say "ok, if we trust the platform > AT ALL, we can trust that". Since you can't reasonably do things like > add EFI variables to your distro image by mistake. NVRAM backing EFI vars sucks. Nothing you want to update on every cycle. It's OK to update during OS installation, but during every single boot? I'd rather not. Lennart -- Lennart Poettering, Berlin
chaos generating driver was Re: Linux 5.3-rc8
Hi!
> > => src/random-seed/random-seed.c:
> > /*
> > * Let's make this whole job asynchronous, i.e. let's make
> > * ourselves a barrier for proper initialization of the
> > * random pool.
> > */
...
> > k = getrandom(buf, buf_size, GRND_NONBLOCK);
> > if (k < 0 && errno == EAGAIN && synchronous) {
> > log_notice("Kernel entropy pool is not initialized yet, "
> > "waiting until it is.");
> >
> > k = getrandom(buf, buf_size, 0); /* retry synchronously */
> > }
>
> Yeah, the above is yet another example of completely broken garbage.
>
> You can't just wait and block at boot. That is simply 100%
> unacceptable, and always has been, exactly because that may
> potentially mean waiting forever since you didn't do anything that
> actually is likely to add any entropy.
Hmm. This actually points to a solution, and I believe solution is in the
kernel. Userspace is not the best place to decide what is the best way to
generate entropy.
> As mentioned, this has already historically been a huge issue on
> embedded devices, and with disks turnign not just to NVMe but to
> actual polling nvdimm/xpoint/flash, the amount of true "entropy"
> randomness we can give at boot is very questionable.
>
> We can (and will) continue to do a best-effort thing (including very
> much using rdread and friends), but the whole "wait for entropy"
> simply *must* stop.
And we can stop it... from kernel, and without hacks. Simply by generating some
entropy. We do not need to sit quietly while userspace waits for entropy to
appear.
We can for example do some reads from the disk. (find / should be good for
generating
entropy on many systems). For systems with rtc but not timestamp counter, we can
actually just increase register, then read it from interrupt...
...to get precise timings. We know system is blocked waiting for entropy, we can
do expensive things we would not "normally" do.
Yes, it would probably mean new kind of "driver" whose purpose is to generate
some
kind of activity so that interrupts happen and entropy is generated... But that
is
still better solution than fixing all of the userspace.
(With some proposals here, userspace _could_ do
while (getrandom() == -EINVAL) {
system("find / &");
sleep(1);
}
...but I believe we really want to do it once, in kernel, and less hacky than
this)
Best regards,
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: Linux 5.3-rc8
Dear Lennart. Lennart Poettering - 18.09.19, 15:53:25 CEST: > On Mi, 18.09.19 00:10, Martin Steigerwald (mar...@lichtvoll.de) wrote: > > > getrandom() will never "consume entropy" in a way that will block > > > any > > > users of getrandom(). If you don't have enough collected entropy > > > to > > > seed the rng, getrandom() will block. If you do, getrandom() will > > > generate as many numbers as you ask it to, even if no more entropy > > > is > > > ever collected by the system. So it doesn't matter how many > > > clients > > > you have calling getrandom() in the boot process - either there'll > > > be > > > enough entropy available to satisfy all of them, or there'll be > > > too > > > little to satisfy any of them. > > > > Right, but then Systemd would not use getrandom() for initial > > hashmap/ UUID stuff since it > > Actually things are more complex. In systemd there are four classes of > random values we need: > > 1. High "cryptographic" quality. There are very few needs for this in […] > 2. High "non-cryptographic" quality. This is used for example for […] > 3. Medium quality. This is used for seeding hash tables. These may be […] > 4. Crap quality. There are only a few uses of this, where rand_r() is >is OK. > > Of these four case, the first two might block boot. Because the first > case is not common you won't see blocking that often though for > them. The second case is very common, but since we use RDRAND you > won't see it on any recent Intel machines. > > Or to say this all differently: the hash table seeding and the uuid > case are two distinct cases in systemd, and I am sure they should be. Thank you very much for your summary of uses of random numbers in Systemd and also for your other mail that "neither RDRAND nor /dev/ urandom know a concept of of "depleting entropy"". I thought they would deplete entropy needed to the initial seeding of crng. Thank you also for taking part in this discussion, even if someone put your mail address on carbon copy without asking with. I do not claim I understand enough of this random number stuff. But I feel its important that kernel and userspace developers actually talk with each other about a sane approach for it. And I believe that the complexity involved is part of the issue. I feel an API for attaining random number with different quality levels needs to be much, much, much more simple to use *properly*. I felt a bit overwhelmed by the discussion (and by what else is happening in my life, just having come back from holding a Linux performance workshop in front of about two dozen people), so I intend to step back from it. If one of my mails actually helped to encourage or facilitate kernel space and user space developers talking with each other about a sane approach to random numbers, then I may have used my soft skills in a way that brings some benefit. For the technical aspects certainly people are taking part in this discussion who are much much deeper into the intricacies of entropy in Linux and computers in general, so I just hope for a good outcome. Best, -- Martin
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 01:26:39PM -0700, Linus Torvalds wrote: > Of course, even then people will say "I don't trust the platform". But > at some point you just say "you have trust issues" and move on. It's where our extreme configurability can hurt. Sometimes we'd rather avoid providing some of these "I don't trust this or that" options and impose some choices to users: "you need entropy to boot, stop being childish and collect the small entropy where it is, period". I'm not certain the other operating systems not experiencing entropy issues leave as many choices as we do. I can understand how some choices may be problematic in virtual environments but there are so many other attack vectors there that randomness is probably a detail. Willy
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 1:15 PM Alexander E. Patrakov wrote: > > No, this is not the solution, if we take seriously not only getrandom > hangs, but also urandom warnings. In some setups (root on LUKS is one of > them) they happen early in the initramfs. Therefore "restoring" entropy > from the previous boot by a script that runs from the main system is too > late. That's why it is suggested to load at least a part of the random > seed in the boot loader, and that has not been commonly implemented. Honestly, I think the bootloader suggestion is naive and silly too. Yes, we now support it. And no, I don't think people will trust that either. And I suspect for good reason: there's really very little reason to believe that bootloaders would be any better than any other part of the system. So right now some people trust bootloaders exactly _because_ there basically is just one or two that do this, and the people who use them are usually the people who wrote them or are at least closely associated with them. That will change, and then people will say "why would I trust that, when we know of bug Xyz". And I guarantee that those bugs _will_ happen, and people will quite reasonably then say "yeah, I don't trust the bootloader". Bootloaders do some questionable things. The most likely thing to actually be somewhat useful is I feel things like the kernel just saving the seed by itself in nvram. There's already an example of this for the EFI random seed thing, but that's used purely for kexec, I think. Adding an EFI variable (or other platform nonvolatile thing), and reading (and writing to it) purely from the kernel ends up being one of those things where you can then say "ok, if we trust the platform AT ALL, we can trust that". Since you can't reasonably do things like add EFI variables to your distro image by mistake. Of course, even then people will say "I don't trust the platform". But at some point you just say "you have trust issues" and move on. Linus
Re: Linux 5.3-rc8
19.09.2019 00:56, Eric W. Biederman пишет: The cheap solution appears to be copying a random seed from a previous boot, and I think that will take care of many many cases, and has already been implemented. Which reduces this to a system first boot issue. No, this is not the solution, if we take seriously not only getrandom hangs, but also urandom warnings. In some setups (root on LUKS is one of them) they happen early in the initramfs. Therefore "restoring" entropy from the previous boot by a script that runs from the main system is too late. That's why it is suggested to load at least a part of the random seed in the boot loader, and that has not been commonly implemented. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 12:56 PM Eric W. Biederman
wrote:
>
> The cheap solution appears to be copying a random seed from a previous
> boot, and I think that will take care of many many cases, and has
> already been implemented. Which reduces this to a system first
> boot issue.
Not really.
Part of the problem is that many people don't _trust_ that "previous
boot entropy".
The lack of trust is sometimes fundamental mistrust ("Who knows where
it came from"), which also tends to cover things like not trusting
rdrand or not trusting the boot loader claimed randomness data.
But the lack of trust has been realistic - if you generated your disk
image by cloning a pre-existing one, you may well have two (or more -
up to any infinite number) of subsequent boots that use the same
"random" data for initialization.
And doing that "boot a pre-existing image" is not as unusual as you'd
think. Some people do it to make bootup faster - there have been
people who work on pre-populating bootup all the way to user mode by
basically making boot be a "resume from disk" kind of event.
So a large part of the problem is that we don't actually trust things
that _should_ be trust-worthy, because we've seen (over and over
again) people mis-using it. So then we do mix in the data into the
randomness pool (because there's no downside to _that_), but we don't
treat it as entropy (because while it _probably_ is, we don't actually
trust it sufficiently).
A _lot_ of the problems with randomness come from these trust issues.
Our entropy counting is very very conservative indeed.
Linus
Re: Linux 5.3-rc8
Lennart Poettering writes: > On Di, 17.09.19 09:23, Linus Torvalds (torva...@linux-foundation.org) wrote: > >> On Tue, Sep 17, 2019 at 9:08 AM Lennart Poettering >> wrote: >> > >> > Here's what I'd propose: >> >> So I think this is ok, but I have another proposal. Before I post that >> one, though, I just wanted to point out: >> >> > 1) Add GRND_INSECURE to get those users of getrandom() who do not need >> >high quality entropy off its use (systemd has uses for this, for >> >seeding hash tables for example), thus reducing the places where >> >things might block. >> >> I really think that trhe logic should be the other way around. >> >> The getrandom() users that don't need high quality entropy are the >> ones that don't really think about this, and so _they_ shouldn't be >> the ones that have to explicitly state anything. To those users, >> "random is random". By definition they don't much care, and quite >> possibly they don't even know what "entropy" really means in that >> context. > > So I think people nowadays prefer getrandom() over /dev/urandom > primarily because of the noisy logging the kernel does when you use > the latter on a non-initialized pool. If that'd be dropped then I am > pretty sure that the porting from /dev/urandom to getrandom() you see > in various projects (such as gdm/x11) would probably not take place. > > In fact, speaking for systemd: the noisy logging in the kernel is the > primary (actually: only) reason that we prefer using RDRAND (if > available) over /dev/urandom if we need "medium quality" random > numbers, for example to seed hash tables and such. If the log message > wasn't there we wouldn't be tempted to bother with RDRAND and would > just use /dev/urandom like we used to for that. > >> > 2) Add a kernel log message if a getrandom(0) client hung for 15s or >> >more, explaining the situation briefly, but not otherwise changing >> >behaviour. >> >> The problem is that when you have some graphical boot, you'll not even >> see the kernel messages ;( > > Well, but as mentioned, there's infrastructure for this, that's why I > suggested changing systemd-random-seed.service. > > We can make boot hang in "sane", discoverable way. > > The reason why I think this should also be logged by the kernel since > people use netconsole and pstore and whatnot and they should see this > there. If systemd with its infrastructure brings this to screen via > plymouth then this wouldn't help people who debug much more low-level. > > (I mean, there have been requests to add a logic to systemd that > refuses booting — or delays it — if the system has a battery and it is > nearly empty. I am pretty sure adding a cleanm discoverable concept of > "uh, i can't boot for a good reason which is this" wouldn't be the > worst of ideas) As I understand it the deep problem is that sometimes we have not observed enough random activity early in boot. The cheap solution appears to be copying a random seed from a previous boot, and I think that will take care of many many cases, and has already been implemented. Which reduces this to a system first boot issue. So for first system boot can we take some special actions to make it possible to see randomness sooner. An unconditional filesystem check of the filesystem perhaps. Something that will initiate disk activity or other hardware activity that will generate interrupts and allow us to capture randomness. For many systems we could even have the installer capture some random data as a final stage of the installation, and use that to seed randomness on the first boot. Somewhere in installing the random seed we need to be careful about people just copying disk images from one system to another, and a replicated seed probably can not be considered very random. My sense is that by copying a random seed from one boot to the next and by initiating system activity to hurry along the process of having enough randomness we can have systems where we can almost always have good random numbers available. And if we almost always have good random numbers available we won't have to worry about people getting this wrong. Am I wrong or can we just solve random number availablity is practically all cases? Eric
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 2:33 AM Rasmus Villemoes wrote: > > And unrelated to the non-use of the RTC (which I agree seems weird), but > because there's no better place in this thread: How "random" is the > contents of RAM after boot? It varies all over the place. Some machines will most definitely clear it at each boot. Others will clear it on cold boots but not warm boots. Yet other environments never clear it at all, or leave it with odd patterns. So it _could_ be useful as added input to the initial random state, but it equally well might be totally pointless. It's really hard to even guess. There would be nothing wrong by trying to do add_device_randomness() from some unused-at-boot memory area, but it's unclear what memory area you should even attempt to use. Certainly not beginning of RAM or end of RAM, which are both special and more likely to have been used by the boot sequence even if it is then marked as unused in the memory maps. And if you do it, it's not clear it will add any noise at all. It _might_. But it might equally well not. Linus
Re: Linux 5.3-rc8
18.09.2019 18:59, Alexander E. Patrakov пишет: 18.09.2019 18:38, Lennart Poettering пишет: On Di, 17.09.19 19:29, Willy Tarreau (w...@1wt.eu) wrote: What do you expect these systems to do though? I mean, think about general purpose distros: they put together live images that are supposed to work on a myriad of similar (as in: same arch) but otherwise very different systems (i.e. VMs that might lack any form of RNG source the same as beefy servers with muliple sources the same as older netbooks with few and crappy sources, ...). They can't know what the specific hw will provide or won't. It's not their incompetence that they build the image like that. It's a common, very common usecase to install a system via SSH, and it's also very common to have very generic images for a large number varied systems to run on. I'm totally file with installing the system via SSH, using a temporary SSH key. I do make a strong distinction between the installation phase and the final deployment. The SSH key used *for installation* doesn't need to the be same as the final one. And very often at the end of the installation we'll have produced enough entropy to produce a correct key. That's not how systems are built today though. And I am not sure they should be. I mean, the majority of systems at this point probably have some form of hardware (or virtualized) RNG available (even raspi has one these days!), so generating these keys once at boot is totally OK. Probably a number of others need just a few seconds to get the entropy needed, where things are totally OK too. The only problem is systems that lack any reasonable source of entropy and where initialization of the pool will take overly long. I figure we can reduce the number of systems where entropy is scarce quite a bit if we'd start crediting entropy by default from various hw rngs we currently don't credit entropy for. For example, the TPM and older intel/amd chipsets. You currently have to specify rng_core.default_quality=1000 on the kernel cmdline to make them credit entropy. I am pretty sure this should be the default now, in a world where CONFIG_RANDOM_TRUST_CPU=y is set anyway. i.e. why say RDRAND is fine but those chipsets are not? That makes no sense to me. I am very sure that crediting entropy to chipset hwrngs is a much better way to solve the issue on those systems than to just hand out rubbish randomness. Very well said. However, 1000 is more than the hard-coded quality of some existing rngs, and so would send a misleading message that they are somehow worse. I would suggest case-by-case reevaluation of all existing hwrng drivers by their maintainers, and then setting the default to something like 899, so that evaluated drivers have priority. Well, I have to provide another data point. On Arch Linux and MSI Z87I desktop board: $ lsmod | grep rng $ modinfo rng_core So this particular board has no sources of randomness except interrupts (which are scarce), RDRAND (which is not trusted in Arch Linux by default) and jitter entropy (which is not collected by the kernel and needs haveged or equivalent). -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
18.09.2019 18:38, Lennart Poettering пишет: On Di, 17.09.19 19:29, Willy Tarreau (w...@1wt.eu) wrote: What do you expect these systems to do though? I mean, think about general purpose distros: they put together live images that are supposed to work on a myriad of similar (as in: same arch) but otherwise very different systems (i.e. VMs that might lack any form of RNG source the same as beefy servers with muliple sources the same as older netbooks with few and crappy sources, ...). They can't know what the specific hw will provide or won't. It's not their incompetence that they build the image like that. It's a common, very common usecase to install a system via SSH, and it's also very common to have very generic images for a large number varied systems to run on. I'm totally file with installing the system via SSH, using a temporary SSH key. I do make a strong distinction between the installation phase and the final deployment. The SSH key used *for installation* doesn't need to the be same as the final one. And very often at the end of the installation we'll have produced enough entropy to produce a correct key. That's not how systems are built today though. And I am not sure they should be. I mean, the majority of systems at this point probably have some form of hardware (or virtualized) RNG available (even raspi has one these days!), so generating these keys once at boot is totally OK. Probably a number of others need just a few seconds to get the entropy needed, where things are totally OK too. The only problem is systems that lack any reasonable source of entropy and where initialization of the pool will take overly long. I figure we can reduce the number of systems where entropy is scarce quite a bit if we'd start crediting entropy by default from various hw rngs we currently don't credit entropy for. For example, the TPM and older intel/amd chipsets. You currently have to specify rng_core.default_quality=1000 on the kernel cmdline to make them credit entropy. I am pretty sure this should be the default now, in a world where CONFIG_RANDOM_TRUST_CPU=y is set anyway. i.e. why say RDRAND is fine but those chipsets are not? That makes no sense to me. I am very sure that crediting entropy to chipset hwrngs is a much better way to solve the issue on those systems than to just hand out rubbish randomness. Very well said. However, 1000 is more than the hard-coded quality of some existing rngs, and so would send a misleading message that they are somehow worse. I would suggest case-by-case reevaluation of all existing hwrng drivers by their maintainers, and then setting the default to something like 899, so that evaluated drivers have priority. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
On Mi, 18.09.19 00:10, Martin Steigerwald (mar...@lichtvoll.de) wrote: > > getrandom() will never "consume entropy" in a way that will block any > > users of getrandom(). If you don't have enough collected entropy to > > seed the rng, getrandom() will block. If you do, getrandom() will > > generate as many numbers as you ask it to, even if no more entropy is > > ever collected by the system. So it doesn't matter how many clients > > you have calling getrandom() in the boot process - either there'll be > > enough entropy available to satisfy all of them, or there'll be too > > little to satisfy any of them. > > Right, but then Systemd would not use getrandom() for initial hashmap/ > UUID stuff since it Actually things are more complex. In systemd there are four classes of random values we need: 1. High "cryptographic" quality. There are very few needs for this in systemd, as we do very little in this area. It's basically only used for generating salt values for hashed passwords, in the systemd-firstboot component, which can be used to set the root pw. systemd uses synchronous getrandom() for this. It does not use RDRAND for this. 2. High "non-cryptographic" quality. This is used for example for generating type 4 uuids, i.e uuids that are supposed to be globally unique, but aren't key material. We use RDRAND for this if available, falling back to synchronous getrandom(). Type 3 UUIDs are frequently needed by systemd, as we assign a uuid to each service invocation implicitly, so that people can match logging data and such to a specific instance and runtime of a service. 3. Medium quality. This is used for seeding hash tables. These may be crap initially, but should not be guessable in the long run. /dev/urandom would be perfect for this, but the mentioned log message sucks, hence we use RDRAND for this if available, and fall back to /dev/urandom if that isn't available, accepting the log message. 4. Crap quality. There are only a few uses of this, where rand_r() is is OK. Of these four case, the first two might block boot. Because the first case is not common you won't see blocking that often though for them. The second case is very common, but since we use RDRAND you won't see it on any recent Intel machines. Or to say this all differently: the hash table seeding and the uuid case are two distinct cases in systemd, and I am sure they should be. Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Di, 17.09.19 23:38, Martin Steigerwald (mar...@lichtvoll.de) wrote: > (I know that it still with /dev/urandom, so if it is using RDRAND now, > this may indeed be different, but would it then deplete entropy the CPU > has available and that by default is fed into the Linux crng as well > (even without trusting it completely)?) Neither RDRAND nor /dev/urandom know a concept of "depleting entropy". That concept does not exist for them. It does exist for /dev/random, but only crazy people use that. systemd does not. Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Di, 17.09.19 19:29, Willy Tarreau (w...@1wt.eu) wrote: > > What do you expect these systems to do though? > > > > I mean, think about general purpose distros: they put together live > > images that are supposed to work on a myriad of similar (as in: same > > arch) but otherwise very different systems (i.e. VMs that might lack > > any form of RNG source the same as beefy servers with muliple sources > > the same as older netbooks with few and crappy sources, ...). They can't > > know what the specific hw will provide or won't. It's not their > > incompetence that they build the image like that. It's a common, very > > common usecase to install a system via SSH, and it's also very common > > to have very generic images for a large number varied systems to run > > on. > > I'm totally file with installing the system via SSH, using a temporary > SSH key. I do make a strong distinction between the installation phase > and the final deployment. The SSH key used *for installation* doesn't > need to the be same as the final one. And very often at the end of the > installation we'll have produced enough entropy to produce a correct > key. That's not how systems are built today though. And I am not sure they should be. I mean, the majority of systems at this point probably have some form of hardware (or virtualized) RNG available (even raspi has one these days!), so generating these keys once at boot is totally OK. Probably a number of others need just a few seconds to get the entropy needed, where things are totally OK too. The only problem is systems that lack any reasonable source of entropy and where initialization of the pool will take overly long. I figure we can reduce the number of systems where entropy is scarce quite a bit if we'd start crediting entropy by default from various hw rngs we currently don't credit entropy for. For example, the TPM and older intel/amd chipsets. You currently have to specify rng_core.default_quality=1000 on the kernel cmdline to make them credit entropy. I am pretty sure this should be the default now, in a world where CONFIG_RANDOM_TRUST_CPU=y is set anyway. i.e. why say RDRAND is fine but those chipsets are not? That makes no sense to me. I am very sure that crediting entropy to chipset hwrngs is a much better way to solve the issue on those systems than to just hand out rubbish randomness. Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 03:25:51PM +0500, Alexander E. Patrakov wrote: > The results so far are: > > 1. Desktop with MSI Z87I board: works. > 2. Lenovo Yoga 2 Pro laptop: works. > 3. Server based on the Intel Corporation S1200SPL board (available from OVH > as EG-32): does not work, memory is cleared. > 4. Cheap server based on Gooxi G1SCN-B board (the cheapes thing with IPMI > available on bacloud.com): works. > > So that's 75% of success stories (found at least one page that is preserved > after the "reboot" command) based on my samples. That's pretty good! I didn't have this luck each time I tried this in the past :-/ I remember noticing that video RAM from graphics card was often usable however, which I figured I could use after seeing a ghost image from a previous boot when switching to graphics mode. Willy
Re: Linux 5.3-rc8
18.09.2019 15:16, Willy Tarreau пишет: We've already discussed that point a few times. The issue is that bootloaders and/or BIOSes tend to wipe everything. Ideally we should let the boot loader collect entropy from the DDR training phase since it's a period where noise is observed. It's also the right moment to collect some random contents that may lie in the RAM cells. Similarly asynchronous clocks driving external components can be used as well if you can measure their phase with the CPU's clock. This does not correspond to my own observations. I have a setup where a secondary key is saved into RAM for unlocking a LUKS container after a reboot. It is documented by me (sorry, in Russian only) here: https://habr.com/ru/post/457396/ , will publish an English translation in my blog if I get at least one request (in private email, please). The results so far are: 1. Desktop with MSI Z87I board: works. 2. Lenovo Yoga 2 Pro laptop: works. 3. Server based on the Intel Corporation S1200SPL board (available from OVH as EG-32): does not work, memory is cleared. 4. Cheap server based on Gooxi G1SCN-B board (the cheapes thing with IPMI available on bacloud.com): works. So that's 75% of success stories (found at least one page that is preserved after the "reboot" command) based on my samples. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
On Wed, Sep 18, 2019 at 11:33:39AM +0200, Rasmus Villemoes wrote: > On 17/09/2019 22.58, Linus Torvalds wrote: > > Side note, and entirely unrelated to this particular problem, but > > _because_ I was looking at the entropy init and sources of randomness > > we have, I notice that we still don't use the ToD clock as a source. > > And unrelated to the non-use of the RTC (which I agree seems weird), but > because there's no better place in this thread: How "random" is the > contents of RAM after boot? Sure, for virtualized environments one > probably always gets zeroed pages from the host (otherwise the host has > a problem...), and on PCs maybe the BIOS interferes. > > But for cheap embedded devices with non-ECC RAM and not a lot of > value-add firmware between power-on and start_kernel(), would it make > sense to read a few MB of memory outside of where the kernel was loaded > and feed those to add_device_randomness() (of course, doing it as early > as possible, maybe first thing in start_kernel())? Or do the reading in > the bootloader and pass on the sha256() in the DT/rng-seed property? > > A quick "kitchen-table" experiment with the board I have on my desk > shows that there are at least some randomness to be had after a cold boot. > > Maybe this has already been suggested and rejected? We've already discussed that point a few times. The issue is that bootloaders and/or BIOSes tend to wipe everything. Ideally we should let the boot loader collect entropy from the DDR training phase since it's a period where noise is observed. It's also the right moment to collect some random contents that may lie in the RAM cells. Similarly asynchronous clocks driving external components can be used as well if you can measure their phase with the CPU's clock. Regards, Willy
Re: Linux 5.3-rc8
On 17/09/2019 22.58, Linus Torvalds wrote: > Side note, and entirely unrelated to this particular problem, but > _because_ I was looking at the entropy init and sources of randomness > we have, I notice that we still don't use the ToD clock as a source. And unrelated to the non-use of the RTC (which I agree seems weird), but because there's no better place in this thread: How "random" is the contents of RAM after boot? Sure, for virtualized environments one probably always gets zeroed pages from the host (otherwise the host has a problem...), and on PCs maybe the BIOS interferes. But for cheap embedded devices with non-ECC RAM and not a lot of value-add firmware between power-on and start_kernel(), would it make sense to read a few MB of memory outside of where the kernel was loaded and feed those to add_device_randomness() (of course, doing it as early as possible, maybe first thing in start_kernel())? Or do the reading in the bootloader and pass on the sha256() in the DT/rng-seed property? A quick "kitchen-table" experiment with the board I have on my desk shows that there are at least some randomness to be had after a cold boot. Maybe this has already been suggested and rejected? Rasmus
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 2:52 PM Matthew Garrett wrote: > > getrandom() will never "consume entropy" in a way that will block any > users of getrandom(). Yes, this is true for any common and sane use. And by that I just mean that we do have GRND_RANDOM, which currently does exactly that entropy consumption. But it only consumes it for other GRND_RANDOM users - of which there are approximately zero, because nobody wants that rats nest. Linus
Re: Linux 5.3-rc8
Matthew Garrett - 17.09.19, 23:52:00 CEST: > On Tue, Sep 17, 2019 at 11:38:33PM +0200, Martin Steigerwald wrote: > > My understanding of entropy always has been that only a certain > > amount of it can be produced in a certain amount of time. If that > > is wrong… please by all means, please teach me, how it would be. > > getrandom() will never "consume entropy" in a way that will block any > users of getrandom(). If you don't have enough collected entropy to > seed the rng, getrandom() will block. If you do, getrandom() will > generate as many numbers as you ask it to, even if no more entropy is > ever collected by the system. So it doesn't matter how many clients > you have calling getrandom() in the boot process - either there'll be > enough entropy available to satisfy all of them, or there'll be too > little to satisfy any of them. Right, but then Systemd would not use getrandom() for initial hashmap/ UUID stuff since it 1) would block boot very early then, which is not desirable and 2) it does not need strong random numbers anyway. At least that is how I understood Lennart's comments on the Systemd bug report I referenced. AFAIK hashmap/UUID stuff uses *some* entropy *before* crng has been seeded with entropy and all I wondered was whether this using *some* entropy *before* crng has been seeded – by /dev/urandom initially, but now as far as I got with RDRAND if available – will delay the process of gathering the entropy necessary to seed crng… if that is the case then anything that uses crng during or soon after boot, like gdm, sddm, OpenSSH ssh-keygen will be blocked for a longer time will the initial seeding of crng has been done. Of course if hashmap/UUID stuff does not use any entropy that would be required for the *initial* seeding or crng, then… that would not be the case. But from what I understood, it does. And yes, for "systemd-random-seed" it is true that it does not drain entropy for getrandom, cause it writes the seed to disk *after* crng has been initialized, i.e. at a time where getrandom would never block again as long as the system is running. If I am still completely misunderstanding something there, then it may be better to go to sleep. Which I will do now anyway. Or I may just not be very good at explaining what I mean. -- Martin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 11:38:33PM +0200, Martin Steigerwald wrote: > My understanding of entropy always has been that only a certain amount > of it can be produced in a certain amount of time. If that is wrong… > please by all means, please teach me, how it would be. getrandom() will never "consume entropy" in a way that will block any users of getrandom(). If you don't have enough collected entropy to seed the rng, getrandom() will block. If you do, getrandom() will generate as many numbers as you ask it to, even if no more entropy is ever collected by the system. So it doesn't matter how many clients you have calling getrandom() in the boot process - either there'll be enough entropy available to satisfy all of them, or there'll be too little to satisfy any of them. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
Ahmed S. Darwish - 17.09.19, 22:52:34 CEST: > On Tue, Sep 17, 2019 at 10:28:47PM +0200, Martin Steigerwald wrote: > [...] > > > I don't have any kernel logs old enough to see whether whether crng > > init times have been different with Systemd due to asking for > > randomness for UUID/hashmaps. > > Please stop claiming this. It has been pointed out to you, __multiple > times__, that this makes no difference. For example: > > https://lkml.kernel.org/r/20190916024904.ga22...@mit.edu > > No. getrandom(2) uses the new CRNG, which is either initialized, > or it's not ... So to the extent that systemd has made systems > boot faster, you could call that systemd's "fault". > > You've claimed this like 3 times before in this thread already, and > multiple people replied with the same response. If you don't get the > paragraph above, then please don't continue replying further on this > thread. First off, this mail you referenced has not been an answer to a mail of mine. It does not have my mail address in Cc. So no, it has not been pointed out directly to me in that mail. Secondly: Pardon me, but I do not see how asking for entropy early at boot times or not doing so has *no effect* on the available entropy¹. And I do not see the above mail actually saying this. To my knowledge Sysvinit does not need entropy for itself². The above mail merely talks about the blocking on boot. And whether systemd-random-seed would drain entropy, not whether hashmaps/UUID do. And also not the effect that asking for entropy early has on the available entropy and on the *initial* initialization time of the new CRNG. However I did not claim that Systemd would block booting. *Not at all*. Thirdly: I disagree with the tone you use in your mail. And for that alone I feel it may be better for me to let go of this discussion. My understanding of entropy always has been that only a certain amount of it can be produced in a certain amount of time. If that is wrong… please by all means, please teach me, how it would be. However I am not even claiming anything. All I wrote above is that I do not have any measurements. But I'd expect that the more entropy is asked for early during boot, the longer the initial initialization of the new CRNG will take. And if someone else relies on this initialization, that something else would block for a longer time. I got that it the new crng won't block after that anymore. [1] https://github.com/systemd/systemd/issues/4167 (I know that it still with /dev/urandom, so if it is using RDRAND now, this may indeed be different, but would it then deplete entropy the CPU has available and that by default is fed into the Linux crng as well (even without trusting it completely)?) [2] According to https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html sysvinit does not contain a single line of code about entropy or random numbers. Daniel even updated his blog post with a hint to this discussion. Thanks, -- Martin
Re: Linux 5.3-rc8
Side note, and entirely unrelated to this particular problem, but _because_ I was looking at the entropy init and sources of randomness we have, I notice that we still don't use the ToD clock as a source. There's not a whole lot of bits there, but at least one of the attacks against entirely missing boot-time randomness was to look at the output of get_random_bytes(), and just compare it across machines. We sanitize things by going through a cryptographic hash function, but that helps hide the internal entropy buffers from direct viewing, but it still leaves the "are those internal entropy buffers the _same_ across machines" for the nasty embedded hardware case with identical hardware. Of course, some of those machines didn't even have a a time-of-day clock either. But the fact that some didn't doesn't mean we shouldn't take it into account. So adding a "add_device_randomness()" to do_settimeofday64() (which catches them all) wouldn't be a bad idea. Not perhaps "entropy", but helping against detecting the case of basically very limited entropy at all at early boot. I'm pretty sure we discussed that case when we did those things originally, but I don't actually see us doing it anywhere right now. So we definitely have some sources of differences for different systems that we could/should use, even if we might not be able to really account them as "entropy". The whole "people generated a number of the same keys" is just horrendously bad, even if they were to use /dev/urandom that doesn't have any strict entropy guarantees. Linus
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 10:28:47PM +0200, Martin Steigerwald wrote: [...] > > I don't have any kernel logs old enough to see whether whether crng init > times have been different with Systemd due to asking for randomness for > UUID/hashmaps. > Please stop claiming this. It has been pointed out to you, __multiple times__, that this makes no difference. For example: https://lkml.kernel.org/r/20190916024904.ga22...@mit.edu No. getrandom(2) uses the new CRNG, which is either initialized, or it's not ... So to the extent that systemd has made systems boot faster, you could call that systemd's "fault". You've claimed this like 3 times before in this thread already, and multiple people replied with the same response. If you don't get the paragraph above, then please don't continue replying further on this thread. thanks, -- Ahmed Darwish http://darwish.chasingpointers.com
Re: Linux 5.3-rc8
Willy Tarreau - 17.09.19, 19:29:29 CEST: > On Tue, Sep 17, 2019 at 07:13:28PM +0200, Lennart Poettering wrote: > > On Di, 17.09.19 18:21, Willy Tarreau (w...@1wt.eu) wrote: > > > On Tue, Sep 17, 2019 at 05:57:43PM +0200, Lennart Poettering > > > wrote: > > > > Note that calling getrandom(0) "too early" is not something > > > > people do > > > > on purpose. It happens by accident, i.e. because we live in a > > > > world > > > > where SSH or HTTPS or so is run in the initrd already, and in a > > > > world > > > > where booting sometimes can be very very fast. > > > > > > It's not an accident, it's a lack of understanding of the impacts > > > from the people who package the systems. Generating an SSH key > > > from > > > an initramfs without thinking where the randomness used for this > > > could come from is not accidental, it's a lack of experience that > > > will be fixed once they start to collect such reports. And those > > > who absolutely need their SSH daemon or HTTPS server for a > > > recovery image in initramfs can very well feed fake entropy by > > > dumping whatever they want into /dev/random to make it possible > > > to build temporary keys for use within this single session. At > > > least all supposedly incorrect use will be made *on purpose* and > > > will still be possible to match what users need.> > > What do you expect these systems to do though? > > > > I mean, think about general purpose distros: they put together live > > images that are supposed to work on a myriad of similar (as in: same > > arch) but otherwise very different systems (i.e. VMs that might lack > > any form of RNG source the same as beefy servers with muliple > > sources > > the same as older netbooks with few and crappy sources, ...). They > > can't know what the specific hw will provide or won't. It's not > > their incompetence that they build the image like that. It's a > > common, very common usecase to install a system via SSH, and it's > > also very common to have very generic images for a large number > > varied systems to run on. > > I'm totally file with installing the system via SSH, using a temporary > SSH key. I do make a strong distinction between the installation > phase and the final deployment. The SSH key used *for installation* > doesn't need to the be same as the final one. And very often at the > end of the installation we'll have produced enough entropy to produce > a correct key. Well… systems cloud-init adapts may come from the same template. Cloud Init thus replaces the key that has been there before on their first boot. There is no "installation". Cloud Init could replace the key in the background… and restart SSH then… but that will give those big fat man in the middle warnings and all systems would use the same SSH host key initially. I just don't see a good way at the moment how to handle this. Introducing an SSH mode for this is still a temporary not so random key with proper warnings might be challenging to get right from both a security and usability point of view. And it would add complexity. That said with Proxmox VE on Fujitsu S8 or Intel NUCs I have never seen this issue even when starting 50 VMs in a row, however, with large cloud providers starting 50 VMs in a row does not sound like all that much. And I bet with Proxmox VE virtio rng is easily available cause it uses KVM. -- Martin
Re: Linux 5.3-rc8
Willy Tarreau - 17.09.19, 18:21:37 CEST: > On Tue, Sep 17, 2019 at 05:57:43PM +0200, Lennart Poettering wrote: > > Note that calling getrandom(0) "too early" is not something people > > do > > on purpose. It happens by accident, i.e. because we live in a world > > where SSH or HTTPS or so is run in the initrd already, and in a > > world > > where booting sometimes can be very very fast. > > It's not an accident, it's a lack of understanding of the impacts > from the people who package the systems. Generating an SSH key from > an initramfs without thinking where the randomness used for this could > come from is not accidental, it's a lack of experience that will be > fixed once they start to collect such reports. And those who > absolutely need their SSH daemon or HTTPS server for a recovery image > in initramfs can very well feed fake entropy by dumping whatever they > want into /dev/random to make it possible to build temporary keys for > use within this single session. At least all supposedly incorrect use > will be made *on purpose* and will still be possible to match what > users need. Well I wondered before whether SSH key generation for cloud init or other automatically individualized systems could happen in the background. Replacing a key that would be there before it would be replaced. So SSH would be available *before* the key is regenerated. But then there are those big fast man in the middle warnings… and I have no clear idea to handle this in a way that would both be secure and not scare users off too much. Well probably systems at some point better have good entropy very quickly… and that is it. (And then quantum computers may crack those good keys anyway in the future.) -- Martin
Re: Linux 5.3-rc8
Linus Torvalds - 17.09.19, 20:01:23 CEST: > > We can make boot hang in "sane", discoverable way. > > That is certainly a huge advantage, yes. Right now I suspect that what > has happened is that this has probably been going on as some > low-level background noise for a while, and people either figured it > out and switched away from gdm (example: Christoph), or more likely > some unexplained boot problems that people just didn't chase down. So > it took basically a random happenstance to make this a kernel issue. > > But "easily discoverable" would be good. Well I meanwhile remembered how it was with sddm: Without CPU assistance (RDRAND) or haveged or any other source of entropy, sddm would simply not appear and I'd see the tty1 login. Then I start to type something and after a while sddm popped up. If I would not type anything it took easily at least have a minute till it appeared. Actually I used my system like this quite a while, cause I did not feel comfortable with haveged and RDRAND. AFAIR this was as this Debian still ran with Systemd. What Debian maintainer for sddm did was this: sddm (0.18.0-1) unstable; urgency=medium […] [ Maximiliano Curia ] * Workaround entropy starvation by recommending haveged * Release to unstable -- Maximiliano Curia […] Sun, 22 Jul 2018 13:26:44 +0200 With Sysvinit I still have neither haveged nor RDRAND enabled, but behavior changed a bit. crng init still takes a while % zgrep -h "crng init" /var/log/kern.log* Sep 16 09:06:23 merkaba kernel: [ 16.910096][C3] random: crng init done Sep 8 14:08:39 merkaba kernel: [ 16.682014][C2] random: crng init done Sep 9 09:16:43 merkaba kernel: [ 46.084188][C2] random: crng init done Sep 11 10:52:37 merkaba kernel: [ 47.209825][C3] random: crng init done Sep 12 08:32:08 merkaba kernel: [ 76.624375][C3] random: crng init done Sep 12 20:07:29 merkaba kernel: [ 10.726349][C2] random: crng init done Sep 8 10:02:42 merkaba kernel: [ 37.391577][C2] random: crng init done Aug 26 09:23:51 merkaba kernel: [ 40.555337][C3] random: crng init done Aug 28 09:45:28 merkaba kernel: [ 39.446847][C1] random: crng init done Aug 20 10:14:59 merkaba kernel: [ 12.242467][C1] random: crng init done and there might be a slight delay before sddm appears, before tty has been initialized. I am not completely sure whether it is related to sddm or something else. But AFAIR delays have been in the range of a maximum of 5-10 seconds, so I did not bother to check more closely. Note this is on a ThinkPad T520 which is a PC. And if I read above kernel log excerpts right, it can still take up to 76 second for crng to be initialized with entropy. Would be interesting to see other people's numbers there. There might be a different ordering with Sysvinit and it may still be sddm. But I never have seen a delay of 76 seconds AFAIR… so something else might be different or I just did not notice the delay. Sometimes I switch on the laptop and do something else to come back in a minute or so. I don't have any kernel logs old enough to see whether whether crng init times have been different with Systemd due to asking for randomness for UUID/hashmaps. Thanks, -- Martin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 10:42 AM Lennart Poettering wrote: > > So I think people nowadays prefer getrandom() over /dev/urandom > primarily because of the noisy logging the kernel does when you use > the latter on a non-initialized pool. If that'd be dropped then I am > pretty sure that the porting from /dev/urandom to getrandom() you see > in various projects (such as gdm/x11) would probably not take place. Sad. So people were actually are perfectly happy with urandom, but you don't want the warning, so you use getrandom() and as a result your boot blocks. What a sad sad reason for a bug. Btw, having a "I really don't care deeply about some long-term secure key" flag would soilve that problem too. We'd happily and silently give you data. The only reason we _do_ that silly printout for /dev/urandom is exactly because /dev/random wasn't useful even for the people who _really_ wanted secure randomness, so they started using /dev/urandom despite the fact that it didn't necessarily have any entropy at all. So this all actually fundamentally goes back to the absolutely horrid and entirely wrong semantics of /dev/random that made it entirely useless for the only thing it was actually designed for. This is also an example of how hard-line "security" people that don't see the shades of gray in between black and white are very much part of the problem. If you have some unreasonable hard requirements, you're going to do the wrong thing in the end. At some point even security people need to realize that reality isn't black-and-white. It's not also keeping us from making any sane progress, I feel, because of that bogus "entropy is sacred", despite the fact that our entropy calculations are actually just random made-up stuff (but "hey, reasonable") to begin with and aren't really black-and-white themselves. > In fact, speaking for systemd: the noisy logging in the kernel is the > primary (actually: only) reason that we prefer using RDRAND (if > available) over /dev/urandom if we need "medium quality" random > numbers, for example to seed hash tables and such. If the log message > wasn't there we wouldn't be tempted to bother with RDRAND and would > just use /dev/urandom like we used to for that. That's also very sad. If we have rdrand, we'll actually mix it into /dev/urandom regardless, so it's again just the whole "people started using urandom for keys because random was broken" that is the cause of this all. I really really detest the whole inflexible "security" mindset. > We can make boot hang in "sane", discoverable way. That is certainly a huge advantage, yes. Right now I suspect that what has happened is that this has probably been going on as some low-level background noise for a while, and people either figured it out and switched away from gdm (example: Christoph), or more likely some unexplained boot problems that people just didn't chase down. So it took basically a random happenstance to make this a kernel issue. But "easily discoverable" would be good. > The reason why I think this should also be logged by the kernel since > people use netconsole and pstore and whatnot and they should see this > there. If systemd with its infrastructure brings this to screen via > plymouth then this wouldn't help people who debug much more low-level. Well, I certainly agree with a kernel message (including a big WARN_ON_ONCE), but you also point out that the last time we added helpful messages to let people know, it had some seriously unintended consequences ;) Linus
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 06:20:02PM +0100, Matthew Garrett wrote: > AES keys are used for a variety of long-lived purposes (eg, disk > encryption). True, good point. Willy
Re: Linux 5.3-rc8
On Di, 17.09.19 09:23, Linus Torvalds (torva...@linux-foundation.org) wrote: > On Tue, Sep 17, 2019 at 9:08 AM Lennart Poettering > wrote: > > > > Here's what I'd propose: > > So I think this is ok, but I have another proposal. Before I post that > one, though, I just wanted to point out: > > > 1) Add GRND_INSECURE to get those users of getrandom() who do not need > >high quality entropy off its use (systemd has uses for this, for > >seeding hash tables for example), thus reducing the places where > >things might block. > > I really think that trhe logic should be the other way around. > > The getrandom() users that don't need high quality entropy are the > ones that don't really think about this, and so _they_ shouldn't be > the ones that have to explicitly state anything. To those users, > "random is random". By definition they don't much care, and quite > possibly they don't even know what "entropy" really means in that > context. So I think people nowadays prefer getrandom() over /dev/urandom primarily because of the noisy logging the kernel does when you use the latter on a non-initialized pool. If that'd be dropped then I am pretty sure that the porting from /dev/urandom to getrandom() you see in various projects (such as gdm/x11) would probably not take place. In fact, speaking for systemd: the noisy logging in the kernel is the primary (actually: only) reason that we prefer using RDRAND (if available) over /dev/urandom if we need "medium quality" random numbers, for example to seed hash tables and such. If the log message wasn't there we wouldn't be tempted to bother with RDRAND and would just use /dev/urandom like we used to for that. > > 2) Add a kernel log message if a getrandom(0) client hung for 15s or > >more, explaining the situation briefly, but not otherwise changing > >behaviour. > > The problem is that when you have some graphical boot, you'll not even > see the kernel messages ;( Well, but as mentioned, there's infrastructure for this, that's why I suggested changing systemd-random-seed.service. We can make boot hang in "sane", discoverable way. The reason why I think this should also be logged by the kernel since people use netconsole and pstore and whatnot and they should see this there. If systemd with its infrastructure brings this to screen via plymouth then this wouldn't help people who debug much more low-level. (I mean, there have been requests to add a logic to systemd that refuses booting — or delays it — if the system has a battery and it is nearly empty. I am pretty sure adding a cleanm discoverable concept of "uh, i can't boot for a good reason which is this" wouldn't be the worst of ideas) Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
17.09.2019 22:32, Willy Tarreau пишет: On Tue, Sep 17, 2019 at 07:30:36PM +0200, Lennart Poettering wrote: On Di, 17.09.19 21:58, Alexander E. Patrakov (patra...@gmail.com) wrote: I am worried that the getrandom delays will be serialized, because processes sometimes run one after another. If there are enough chained/dependent processes that ask for randomness before it is ready, the end result is still a too-big delay, essentially a failed boot. In other words: your approach of adding delays only makes sense for heavily parallelized boot, which may not be the case, especially for embedded systems that don't like systemd. As mentioned elsewhere: once the pool is initialized it's initialized. This means any pending getrandom() on the whole system will unblock at the same time, and from the on all getrandom()s will be non-blocking. He means that all process will experience this delay until there's enough entropy. Willy Indeed, my wording was not clear enough. Linus' patch has a 5-second timeout for small entropy requests, after which they get converted to the equivalent of urandom. However, in the following shell script: #!/bin/sh p1 p2 if both p1 and p2 ask for a small amount of entropy before crng is fully initialized, and do nothing that produces more entropy, the total delay will be 10 seconds. -- Alexander E. Patrakov
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 07:30:36PM +0200, Lennart Poettering wrote: > On Di, 17.09.19 21:58, Alexander E. Patrakov (patra...@gmail.com) wrote: > > > I am worried that the getrandom delays will be serialized, because processes > > sometimes run one after another. If there are enough chained/dependent > > processes that ask for randomness before it is ready, the end result is > > still a too-big delay, essentially a failed boot. > > > > In other words: your approach of adding delays only makes sense for heavily > > parallelized boot, which may not be the case, especially for embedded > > systems that don't like systemd. > > As mentioned elsewhere: once the pool is initialized it's > initialized. This means any pending getrandom() on the whole system > will unblock at the same time, and from the on all getrandom()s will > be non-blocking. He means that all process will experience this delay until there's enough entropy. Willy
Re: Linux 5.3-rc8
On Di, 17.09.19 21:58, Alexander E. Patrakov (patra...@gmail.com) wrote: > I am worried that the getrandom delays will be serialized, because processes > sometimes run one after another. If there are enough chained/dependent > processes that ask for randomness before it is ready, the end result is > still a too-big delay, essentially a failed boot. > > In other words: your approach of adding delays only makes sense for heavily > parallelized boot, which may not be the case, especially for embedded > systems that don't like systemd. As mentioned elsewhere: once the pool is initialized it's initialized. This means any pending getrandom() on the whole system will unblock at the same time, and from the on all getrandom()s will be non-blocking. systemd-random-seed.service is nowadays a synchronization point for exactly the moment where the pool is considered full. Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 07:13:28PM +0200, Lennart Poettering wrote: > On Di, 17.09.19 18:21, Willy Tarreau (w...@1wt.eu) wrote: > > > On Tue, Sep 17, 2019 at 05:57:43PM +0200, Lennart Poettering wrote: > > > Note that calling getrandom(0) "too early" is not something people do > > > on purpose. It happens by accident, i.e. because we live in a world > > > where SSH or HTTPS or so is run in the initrd already, and in a world > > > where booting sometimes can be very very fast. > > > > It's not an accident, it's a lack of understanding of the impacts > > from the people who package the systems. Generating an SSH key from > > an initramfs without thinking where the randomness used for this could > > come from is not accidental, it's a lack of experience that will be > > fixed once they start to collect such reports. And those who absolutely > > need their SSH daemon or HTTPS server for a recovery image in initramfs > > can very well feed fake entropy by dumping whatever they want into > > /dev/random to make it possible to build temporary keys for use within > > this single session. At least all supposedly incorrect use will be made > > *on purpose* and will still be possible to match what users need. > > What do you expect these systems to do though? > > I mean, think about general purpose distros: they put together live > images that are supposed to work on a myriad of similar (as in: same > arch) but otherwise very different systems (i.e. VMs that might lack > any form of RNG source the same as beefy servers with muliple sources > the same as older netbooks with few and crappy sources, ...). They can't > know what the specific hw will provide or won't. It's not their > incompetence that they build the image like that. It's a common, very > common usecase to install a system via SSH, and it's also very common > to have very generic images for a large number varied systems to run > on. I'm totally file with installing the system via SSH, using a temporary SSH key. I do make a strong distinction between the installation phase and the final deployment. The SSH key used *for installation* doesn't need to the be same as the final one. And very often at the end of the installation we'll have produced enough entropy to produce a correct key. It's not because people got used to doing things the wrong way by ignorance of how randomness works and raised this to an industrial level that they must not adapt a little bit. If they insist on producing an SSH key immediately at boot, you can be sure that many of those that never fail are probably bad because they probably used some of the tricks mentioned in this thread (like the fairly common mknod trick that can make sense in a temporary system installation image) :-/ I maintain that we don't need the same amount of entropy to run a regular system and to create a new key, and that as such it is not a reasonable thing to do to create such a key as the first action. I'm not saying that doing things correctly is as easy, but it's not impossible at all: many of us have already used systems which use something like dropbear with temporary key on the install image but run off openssh in the final system image. And even when booting off a pre-configured final image we could easily imagine that the ssh service detects lack of entropy and runs with a temporary key that is not saved, and in the background starts a process trying to produce a final key for later use. Willy
Re: Linux 5.3-rc8
On Di, 17.09.19 09:27, Linus Torvalds (torva...@linux-foundation.org) wrote: > But look at what gnome-shell and gnome-session-b does: > > https://lore.kernel.org/linux-ext4/20190912034421.GA2085@darwi-home-pc/ > > and most of them already set GRND_NONBLOCK, but look at the > problematic one that actually causes the boot problem: > > gnome-session-b-327 4.400620: getrandom(16 bytes, flags = 0) > > and here the big clue is: "Hey, it only asks for 128 bits of > randomness". I don't think this is a good check to make. In fact most cryptography folks say taking out more than 256bit is never going to make sense, that's why BSD getentropy() even returns an error if you ask for more than 256bit. (and glibc's getentropy() wrapper around getrandom() enforces the same size limit btw) On the BSDs the kernel's getentropy() call is primarily used to seed their libc's arc4random() every now and then, and userspace is supposed to use only arc4random(). I am pretty sure we should do the same on Linux in the long run. i.e. the idea that everyone uses the kernel syscall directly sounds wrong to me, and designing the syscall so that everyone calls it is hence wrong too. On the BSDs getentropy() is hence unconditionally blocking, without any flags or so, which makes sense since it's not supposed to be user-facing really so much, but more a basic primitive for low-level userspace infrastructure only, that is supposed to be wrapped non-trivially to be useful. (that's at least how I understood their APIs) > Does anybody believe that 128 bits of randomness is a good basis for a > long-term secure key? Even if the key itself contains than that, if > you are generating a long-term secure key in this day and age, you had > better be asking for more than 128 bits of actual unpredictable base > data. So just based on the size of the request we can determine that > this is not hugely important. aes128 is very common today. It's what baseline security is. I have the suspicion crypto folks would argue that 128…256 is the only sane range for cryptographic keys... Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 06:20:02PM +0100, Matthew Garrett wrote: > AES keys are used for a variety of long-lived purposes (eg, disk > encryption). And as an example of when we'd want to do that during early boot - swap is frequently encrypted with a random key generated on each boot, but it's still important for that key to be strong in order to avoid someone being able to recover the contents of swap. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 07:16:41PM +0200, Willy Tarreau wrote: > On Tue, Sep 17, 2019 at 05:34:56PM +0100, Matthew Garrett wrote: > > On Tue, Sep 17, 2019 at 09:27:44AM -0700, Linus Torvalds wrote: > > > > > Does anybody believe that 128 bits of randomness is a good basis for a > > > long-term secure key? > > > > Yes, it's exactly what you'd expect for an AES 128 key, which is still > > considered to be secure. > > AES keys are for symmetrical encryption and thus as such are short-lived. > We're back to what Linus was saying about the fact that our urandom is > already very good for such use cases, it should just not be used to > produce long-lived keys (i.e. asymmetrical). AES keys are used for a variety of long-lived purposes (eg, disk encryption). -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 05:34:56PM +0100, Matthew Garrett wrote: > On Tue, Sep 17, 2019 at 09:27:44AM -0700, Linus Torvalds wrote: > > > Does anybody believe that 128 bits of randomness is a good basis for a > > long-term secure key? > > Yes, it's exactly what you'd expect for an AES 128 key, which is still > considered to be secure. AES keys are for symmetrical encryption and thus as such are short-lived. We're back to what Linus was saying about the fact that our urandom is already very good for such use cases, it should just not be used to produce long-lived keys (i.e. asymmetrical). However I'm worried regarding this precise patch about the fact that delays will add up. I think that once we've failed to wait for a first process, we've broken any hypothetical trust in terms of random quality so there's no point continuing to wait for future requests. Willy
Re: Linux 5.3-rc8
On Di, 17.09.19 18:21, Willy Tarreau (w...@1wt.eu) wrote: > On Tue, Sep 17, 2019 at 05:57:43PM +0200, Lennart Poettering wrote: > > Note that calling getrandom(0) "too early" is not something people do > > on purpose. It happens by accident, i.e. because we live in a world > > where SSH or HTTPS or so is run in the initrd already, and in a world > > where booting sometimes can be very very fast. > > It's not an accident, it's a lack of understanding of the impacts > from the people who package the systems. Generating an SSH key from > an initramfs without thinking where the randomness used for this could > come from is not accidental, it's a lack of experience that will be > fixed once they start to collect such reports. And those who absolutely > need their SSH daemon or HTTPS server for a recovery image in initramfs > can very well feed fake entropy by dumping whatever they want into > /dev/random to make it possible to build temporary keys for use within > this single session. At least all supposedly incorrect use will be made > *on purpose* and will still be possible to match what users need. What do you expect these systems to do though? I mean, think about general purpose distros: they put together live images that are supposed to work on a myriad of similar (as in: same arch) but otherwise very different systems (i.e. VMs that might lack any form of RNG source the same as beefy servers with muliple sources the same as older netbooks with few and crappy sources, …). They can't know what the specific hw will provide or won't. It's not their incompetence that they build the image like that. It's a common, very common usecase to install a system via SSH, and it's also very common to have very generic images for a large number varied systems to run on. Lennart -- Lennart Poettering, Berlin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 09:27:44AM -0700, Linus Torvalds wrote: > Does anybody believe that 128 bits of randomness is a good basis for a > long-term secure key? Yes, it's exactly what you'd expect for an AES 128 key, which is still considered to be secure. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
17.09.2019 21:27, Linus Torvalds пишет: On Tue, Sep 17, 2019 at 12:33 AM Martin Steigerwald wrote: So yes, that would it make it harder to abuse the API, but not impossible. Which may still be good, I don't know. So the real problem is not people abusing the ABI per se. Yes, I was a bit worried about that too, but it's not the cause of the immediate issue. The real problem is that "getrandom(0)" is really _convenient_ for people who just want random numbers - and not at all the "secure" kind. And it's convenient, and during development and testing, it always "just works", because it doesn't ever block in any normal situation. And then you deploy it, and on some poor users machine it *does* block, because the program now encounters the "oops, no entropy" situation that it never ever encountered on the development machine, because the testing there was mainly done not during booting, but the developer also probably had a much more modern machine that had rdrand, and that quite possibly also had more services enabled at bootup etc so even without rdrand it got tons of entropy. That's why (a) killing the process is _completely_ silly. It misses the whole point of the problem in the first place and only makes things much worse. (b) we should just change getrandom() and add that GRND_SECURE flag instead. Because the current API is fundamentally confusing. If you want secure random numbers, you should really deeply _know_ about it, and think about it, rather than have it be the "oh, don't even bother passing any flags, it's secure by default". (c) the timeout approach isn't wonderful, but it at least helps with the "this was never tested under those circumstances" kind of problem. Note that the people who actually *thought* about getrandom() and use it correctly should already handle error returns (even for the blocking version), because getrandom() can already return EINTR. So the argument that we should cater primarily to the secure key people is not all that strong. We should be able to return EINTR, and the people who *thought* about blocking and about entropy should be fine. And gdm and other silly random users that never wanted entropy in the first place, just "random" random numbers, wouldn't be in the situation they are now. That said - looking at some of the problematic traces that Ahmed posted for his bootup problem, I actually think we can use *another* heuristic to solve the problem. Namely just looking at how much randomness the caller wants. The processes that ask for randomness for an actual secure key have a very fundamental constraint: they need enough randomness for the key to be secure in the first place. But look at what gnome-shell and gnome-session-b does: https://lore.kernel.org/linux-ext4/20190912034421.GA2085@darwi-home-pc/ and most of them already set GRND_NONBLOCK, but look at the problematic one that actually causes the boot problem: gnome-session-b-327 4.400620: getrandom(16 bytes, flags = 0) and here the big clue is: "Hey, it only asks for 128 bits of randomness". Does anybody believe that 128 bits of randomness is a good basis for a long-term secure key? Even if the key itself contains than that, if you are generating a long-term secure key in this day and age, you had better be asking for more than 128 bits of actual unpredictable base data. So just based on the size of the request we can determine that this is not hugely important. Compare that to the case later on for something that seems to ask for actual interesting randomness. and - just judging by the name - probably even has a reason for it: gsd-smartcard-388 51.433924: getrandom(110 bytes, flags = 0) gsd-smartcard-388 51.433936: getrandom(256 bytes, flags = 0) big difference. End result: I would propose the attached patch. Ahmed, can you just verify that it works for you (obviously with the ext4 plugging reinstated)? It looks like it should "obviously" fix things, but still... I have looked at the patch, but have not tested it. I am worried that the getrandom delays will be serialized, because processes sometimes run one after another. If there are enough chained/dependent processes that ask for randomness before it is ready, the end result is still a too-big delay, essentially a failed boot. In other words: your approach of adding delays only makes sense for heavily parallelized boot, which may not be the case, especially for embedded systems that don't like systemd. -- Alexander E. Patrakov
Re: Linux 5.3-rc8
Am 17.09.19 um 18:23 schrieb Linus Torvalds: > I do agree that a message is a good idea regardless, but I don't think > it necessarily solves the problems except for developers sadly in our current world dvelopers and maintainers don't read any logs and as long it compiles and boots it works and can be pushed :-( they even argue instead fix a dmaned line in a textfile which could have been fixed 8 years in advance and i have written a ton of such reports for F30 not talking about 15 others where software spits warnings with the source file and line into the syslog and nobody out there gives a damn about it one example of many https://bugzilla.redhat.com/show_bug.cgi?id=1748322 the only way you can get developers to clean up their mess these days is to spit it straight into their face in modal window everytime they login but how to exclude innocent endusers. half of my "rsyslog.conf" is to filter out stuff i can't fix anyways to have my peace when call the script below every time i reboot whatever linux machine the 'usb_serial_init - returning with error' is BTW Linux when you boot with 'nousb usbcore.nousb' -- [root@srv-rhsoft:~]$ cat /scripts/system-errors.sh #!/usr/bin/dash dmesg -T | grep --color -i warn | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' grep --color -i warn /var/log/messages | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' dmesg -T | grep --color -i fail | grep -v 'BAR 13' | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' grep --color -i fail /var/log/messages | grep -v 'BAR 13' | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' dmesg -T | grep --color -i error | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' grep --color -i error /var/log/messages | grep -v 'Perf event create on CPU' | grep -v 'Hardware RNG Device' | grep -v 'TPM RNG Device' | grep -v 'Correctable Errors collector initialized' | grep -v 'error=format-security' | grep -v 'MHD_USE_THREAD_PER_CONNECTION' | grep -v 'usb_serial_init - returning with error' | grep -v 'systemd-journald.service' | grep -v 'usb_serial_init - registering generic driver failed' grep --color -i "scheduling restart" /var/log/messages | grep -v 'systemd-journald.service' [root@srv-rhsoft:~]$
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 12:33 AM Martin Steigerwald wrote:
>
> So yes, that would it make it harder to abuse the API, but not
> impossible. Which may still be good, I don't know.
So the real problem is not people abusing the ABI per se. Yes, I was a
bit worried about that too, but it's not the cause of the immediate
issue.
The real problem is that "getrandom(0)" is really _convenient_ for
people who just want random numbers - and not at all the "secure"
kind.
And it's convenient, and during development and testing, it always
"just works", because it doesn't ever block in any normal situation.
And then you deploy it, and on some poor users machine it *does*
block, because the program now encounters the "oops, no entropy"
situation that it never ever encountered on the development machine,
because the testing there was mainly done not during booting, but the
developer also probably had a much more modern machine that had
rdrand, and that quite possibly also had more services enabled at
bootup etc so even without rdrand it got tons of entropy.
That's why
(a) killing the process is _completely_ silly. It misses the whole
point of the problem in the first place and only makes things much
worse.
(b) we should just change getrandom() and add that GRND_SECURE flag
instead. Because the current API is fundamentally confusing. If you
want secure random numbers, you should really deeply _know_ about it,
and think about it, rather than have it be the "oh, don't even bother
passing any flags, it's secure by default".
(c) the timeout approach isn't wonderful, but it at least helps with
the "this was never tested under those circumstances" kind of problem.
Note that the people who actually *thought* about getrandom() and use
it correctly should already handle error returns (even for the
blocking version), because getrandom() can already return EINTR. So
the argument that we should cater primarily to the secure key people
is not all that strong. We should be able to return EINTR, and the
people who *thought* about blocking and about entropy should be fine.
And gdm and other silly random users that never wanted entropy in the
first place, just "random" random numbers, wouldn't be in the
situation they are now.
That said - looking at some of the problematic traces that Ahmed
posted for his bootup problem, I actually think we can use *another*
heuristic to solve the problem. Namely just looking at how much
randomness the caller wants.
The processes that ask for randomness for an actual secure key have a
very fundamental constraint: they need enough randomness for the key
to be secure in the first place.
But look at what gnome-shell and gnome-session-b does:
https://lore.kernel.org/linux-ext4/20190912034421.GA2085@darwi-home-pc/
and most of them already set GRND_NONBLOCK, but look at the
problematic one that actually causes the boot problem:
gnome-session-b-327 4.400620: getrandom(16 bytes, flags = 0)
and here the big clue is: "Hey, it only asks for 128 bits of randomness".
Does anybody believe that 128 bits of randomness is a good basis for a
long-term secure key? Even if the key itself contains than that, if
you are generating a long-term secure key in this day and age, you had
better be asking for more than 128 bits of actual unpredictable base
data. So just based on the size of the request we can determine that
this is not hugely important.
Compare that to the case later on for something that seems to ask for
actual interesting randomness. and - just judging by the name -
probably even has a reason for it:
gsd-smartcard-388 51.433924: getrandom(110 bytes, flags = 0)
gsd-smartcard-388 51.433936: getrandom(256 bytes, flags = 0)
big difference.
End result: I would propose the attached patch.
Ahmed, can you just verify that it works for you (obviously with the
ext4 plugging reinstated)? It looks like it should "obviously" fix
things, but still...
Linus
drivers/char/random.c | 33 -
1 file changed, 32 insertions(+), 1 deletion(-)
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 566922df4b7b..7be771eac969 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -2118,6 +2118,37 @@ const struct file_operations urandom_fops = {
.llseek = noop_llseek,
};
+/*
+ * Hacky workaround for the fact that some processes
+ * ask for truly secure random numbers and absolutely want
+ * to wait for the entropy pool to fill, and others just
+ * do "getrandom(0)" to get some ad-hoc random numbers.
+ *
+ * If you're generating a secure key, you'd better ask for
+ * more than 128 bits of randomness. Otherwise it's not
+ * really all that secure by definition.
+ *
+ * We should add a GRND_SECURE flag so that people can state
+ * this "I want secure random numbers" explicitly.
+ */
+static int wait_for_getrandom(size_t count)
+{
+ unsigned long timeout = MAX_SCHEDULE_TIMEOUT;
+ int ret;
+
+ /* We'll give even
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 9:08 AM Lennart Poettering wrote: > > Here's what I'd propose: So I think this is ok, but I have another proposal. Before I post that one, though, I just wanted to point out: > 1) Add GRND_INSECURE to get those users of getrandom() who do not need >high quality entropy off its use (systemd has uses for this, for >seeding hash tables for example), thus reducing the places where >things might block. I really think that trhe logic should be the other way around. The getrandom() users that don't need high quality entropy are the ones that don't really think about this, and so _they_ shouldn't be the ones that have to explicitly state anything. To those users, "random is random". By definition they don't much care, and quite possibly they don't even know what "entropy" really means in that context. The ones that *do* want high security randomness should be the ones that know that "random" means different things to different people, and that randomness is hard. So the onus should be on them to say that "yes, I'm one of those people willing to wait". That's why I'd like to see GRND_SECURE instead. That's kind of what GRND_RANDOM is right now, but it went overboard and it's not useful even to the people who do want secure random numners. Besides, the GRND_RANDOM naming doesn't really help the people who don't know anyway, so it's just bad in so many ways. We should probably just get rid of that flag entirely and make it imply GRND_SECURE without the overdone entropy accounting, but that's a separate issue. When we do add GRND_SECURE, we should also add the GRND_INSECURE just to allow people to mark their use, and to avoid the whole existing confusion about "0". > 2) Add a kernel log message if a getrandom(0) client hung for 15s or >more, explaining the situation briefly, but not otherwise changing >behaviour. The problem is that when you have some graphical boot, you'll not even see the kernel messages ;( I do agree that a message is a good idea regardless, but I don't think it necessarily solves the problems except for developers. > 3) Change systemd-random-seed.service to log to console in the same >case, blocking boot cleanly and discoverably. So I think systemd-random-seed might as well just use a new GRND_SECURE, and then not even have to worry about it. That said, I think I have a suggestion that everybody can live with - even if they might not be _happy_ about it. See next email. > I am not a fan of randomly killing userspace processes that just > happened to be the unlucky ones, to call this first... I see no > benefit in killing stuff over letting boot hang in a discoverable way. Absolutely agreed. The point was to not break things. Linus
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 05:57:43PM +0200, Lennart Poettering wrote: > Note that calling getrandom(0) "too early" is not something people do > on purpose. It happens by accident, i.e. because we live in a world > where SSH or HTTPS or so is run in the initrd already, and in a world > where booting sometimes can be very very fast. It's not an accident, it's a lack of understanding of the impacts from the people who package the systems. Generating an SSH key from an initramfs without thinking where the randomness used for this could come from is not accidental, it's a lack of experience that will be fixed once they start to collect such reports. And those who absolutely need their SSH daemon or HTTPS server for a recovery image in initramfs can very well feed fake entropy by dumping whatever they want into /dev/random to make it possible to build temporary keys for use within this single session. At least all supposedly incorrect use will be made *on purpose* and will still be possible to match what users need. > So even if you write a > program and you think "this stuff should run late I'll just > getrandom(0)" it might not actually be that case IRL because people > deploy it a slightly bit differently than you initially thought in a > slightly differently equipped system with other runtime behaviour... I agree with this, it's precisely because I think we should not restrict userspace capabilities that I want the issue addressed in a way that lets users do what they need instead of relying on dangerous workarounds. Just googling for "mknod /dev/random c 1 9" returns tens, maybe hundreds of pages all explaining how to fix the problem of non-booting systems. It simply proves that the kernel is not the place to decide what users are allowed to do. Let's give them the tools to work correctly and be responsible for their choices. They just need to be hit by bad choices to get some feedback from the field other than a new list of well-known SSH keys. Willy
Re: Linux 5.3-rc8
On Di, 17.09.19 12:30, Ahmed S. Darwish (darwish...@gmail.com) wrote: > Ideally, systems would be configured with hardware random > number generators, and/or configured to trust the CPU-provided > RNG's (CONFIG_RANDOM_TRUST_CPU) or boot-loader provided ones > (CONFIG_RANDOM_TRUST_BOOTLOADER). In addition, userspace > should generate cryptographic keys only as late as possible, > when they are needed, instead of during early boot. (For > non-cryptographic use cases, such as dictionary seeds or MIT > Magic Cookies, other mechanisms such as /dev/urandom or > random(3) may be more appropropriate.) > > Sounds good? This sounds mean. You make apps pay for something they aren't really at fault for. I mean, in the cloud people typically put together images that are replicated to many systems, and as first thing generate an SSH key, on the individual system. In fact, most big distros tend to ship SSH that is precisely set up this way: on first boot the SSH key is generated. They tend to call getrandom(0) for this right now, and rightfully so. Now suddenly you kill them because they are doing everything correctly? Those systems aren't going to be more useful if they have no SSH key at all than they would be if they would hang at boot: either way you can't log in. Here's what I'd propose: 1) Add GRND_INSECURE to get those users of getrandom() who do not need high quality entropy off its use (systemd has uses for this, for seeding hash tables for example), thus reducing the places where things might block. 2) Add a kernel log message if a getrandom(0) client hung for 15s or more, explaining the situation briefly, but not otherwise changing behaviour. 3) Change systemd-random-seed.service to log to console in the same case, blocking boot cleanly and discoverably. I am not a fan of randomly killing userspace processes that just happened to be the unlucky ones, to call this first... I see no benefit in killing stuff over letting boot hang in a discoverable way. Lennart
Re: Linux 5.3-rc8
On Di, 17.09.19 08:11, Theodore Y. Ts'o (ty...@mit.edu) wrote: > On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: > > Willy Tarreau - 17.09.19, 07:24:38 CEST: > > > On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: > > > > >Well, the patch actually made getrandom() return en error too, but > > > > >you seem more interested in the hypotheticals than in arguing > > > > >actualities.> > > > > If you want to be safe, terminate the process. > > > > > > This is an interesting approach. At least it will cause bug reports in > > > application using getrandom() in an unreliable way and they will > > > check for other options. Because one of the issues with systems that > > > do not finish to boot is that usually the user doesn't know what > > > process is hanging. > > > > I would be happy with a change which changes getrandom(0) to send a > kill -9 to the process if it is called too early, with a new flag, > getrandom(GRND_BLOCK) which blocks until entropy is available. That > leaves it up to the application developer to decide what behavior they > want. Note that calling getrandom(0) "too early" is not something people do on purpose. It happens by accident, i.e. because we live in a world where SSH or HTTPS or so is run in the initrd already, and in a world where booting sometimes can be very very fast. So even if you write a program and you think "this stuff should run late I'll just getrandom(0)" it might not actually be that case IRL because people deploy it a slightly bit differently than you initially thought in a slightly differently equipped system with other runtime behaviour... Lennart
Re: Linux 5.3-rc8
On Mo, 16.09.19 13:21, Theodore Y. Ts'o (ty...@mit.edu) wrote: > We could create a new flag, GRND_INSECURE, which never blocks. And > that that allows us to solve the problem for silly applications that > are using getrandom(2) for non-cryptographic use cases. Use cases > might include Python dictionary seeds, gdm for MIT Magic Cookie, UUID > generation where best efforts probably is good enough, etc. The > answer today is they should just use /dev/urandom, since that exists > today, and we have to support it for backwards compatibility anyway. > It sounds like gdm recently switched to getrandom(2), and I suspect > that it's going to get caught on some hardware configs anyway, even > without the ext4 optimization patch. So I suspect gdm will switch > back to /dev/urandom, and this particular pain point will probably go > away. The problem is that reading from /dev/urandom at a point where it's not initialized yet results in noisy kernel logging on current kernels. If you want people to use /dev/urandom then the logging needs to go away, because it scares people, makes them file bug reports and so on, even though there isn't actually any problem for these specific purposes. For that reason I'd prefer GRND_INSECURE I must say, because it indicates people grokked "I know I might get questionnable entropy". Lennart
Re: Linux 5.3-rc8
17.09.2019 18:11, Alexander E. Patrakov пишет: 17.09.2019 17:11, Theodore Y. Ts'o пишет: There are only two ways out of this mess. The first option is we take functionality away from a userspace author who Really Wants A Secure Random Number Generator. And there are an awful lot of programs who really want secure crypto, becuase this is not a hypothetical. The result in "Mining your P's and Q's" did happen before. If we forget the history, we are doomed to repeat it. You cannot take away functionality that does not really exist. Every time somebody tries to use it, there is a huge news, "the boot process is blocked on application FOO", followed by an insecure fallback to /dev/urandom in the said application or library. Regarding the "Mining your P's and Q's" paper: I would say it is a combination of TWO faults, only one of which (poor, or, as explained below, "marginally poor" entropy) is discussed and the other one (not really sound crypto when deriving the RSA key from the presumedly-available entropy) is ignored. The authors of the paper factored the weak keys by applying the generalized GCD algorithm, thus looking for common factors in the RSA public keys. For two RSA public keys to be detected as faulty, they must share exactly one of their prime factors. In other words: repeated keys were specifically excluded from the study by the paper authors. Sharing only one of the two primes means that that the systems in question behaved identically when they generated the first prime, but diverged (possibly due to the extra entropy becoming available) when they generated the second one. And asking the randomness for p and for q separately is what I would call the application bug here that nobody wants to talk about: both p and q should have been derived from a CSPRNG seeded by a single read from a random source. If that practice were followed, then it would either result in a duplicate key (which is not as bad as a factorable one), or in completely unrelated keys. I take this back. Of course, completely duplicate keys are weak keys, and they are even more dangerous because they are not distinguishable from intentionally copied good keys by the method in the paper. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
17.09.2019 17:11, Theodore Y. Ts'o пишет: There are only two ways out of this mess. The first option is we take functionality away from a userspace author who Really Wants A Secure Random Number Generator. And there are an awful lot of programs who really want secure crypto, becuase this is not a hypothetical. The result in "Mining your P's and Q's" did happen before. If we forget the history, we are doomed to repeat it. You cannot take away functionality that does not really exist. Every time somebody tries to use it, there is a huge news, "the boot process is blocked on application FOO", followed by an insecure fallback to /dev/urandom in the said application or library. Regarding the "Mining your P's and Q's" paper: I would say it is a combination of TWO faults, only one of which (poor, or, as explained below, "marginally poor" entropy) is discussed and the other one (not really sound crypto when deriving the RSA key from the presumedly-available entropy) is ignored. The authors of the paper factored the weak keys by applying the generalized GCD algorithm, thus looking for common factors in the RSA public keys. For two RSA public keys to be detected as faulty, they must share exactly one of their prime factors. In other words: repeated keys were specifically excluded from the study by the paper authors. Sharing only one of the two primes means that that the systems in question behaved identically when they generated the first prime, but diverged (possibly due to the extra entropy becoming available) when they generated the second one. And asking the randomness for p and for q separately is what I would call the application bug here that nobody wants to talk about: both p and q should have been derived from a CSPRNG seeded by a single read from a random source. If that practice were followed, then it would either result in a duplicate key (which is not as bad as a factorable one), or in completely unrelated keys. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 12:30:15PM +, Ahmed S. Darwish wrote:
> Sounds good?
Sounds good to me except that I'd like to have the option to get
poor randoms. getrandom() is used when /dev/urandom is not accessible
or painful to use. Until we provide applications with a solution to
this fairly common need, the problem will continue to regularly pop
up, in a different way ("my application randomly crashes at boot").
Let's get GRND_INSECURE in addition to your change and I think all
needs will be properly covered.
Thanks,
Willy
Re: Linux 5.3-rc8
17.09.2019 17:30, Ahmed S. Darwish пишет: On Tue, Sep 17, 2019 at 08:11:56AM -0400, Theodore Y. Ts'o wrote: On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: Willy Tarreau - 17.09.19, 07:24:38 CEST: On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: Well, the patch actually made getrandom() return en error too, but you seem more interested in the hypotheticals than in arguing actualities.> If you want to be safe, terminate the process. This is an interesting approach. At least it will cause bug reports in application using getrandom() in an unreliable way and they will check for other options. Because one of the issues with systems that do not finish to boot is that usually the user doesn't know what process is hanging. I would be happy with a change which changes getrandom(0) to send a kill -9 to the process if it is called too early, with a new flag, getrandom(GRND_BLOCK) which blocks until entropy is available. That leaves it up to the application developer to decide what behavior they want. Yup, I'm convinced that's the sanest option too. I'll send a final RFC patch tonight implementing the following: config GETRANDOM_CRNG_ENTROPY_MAX_WAIT_MS int default 3000 help Default max wait in milliseconds, for the getrandom(2) system call when asking for entropy from the urandom source, until the Cryptographic Random Number Generator (CRNG) gets initialized. Any process exceeding this duration for entropy wait will get killed by kernel. The maximum wait can be overriden through the "random.getrandom_max_wait_ms" kernel boot parameter. Rationale follows. When the getrandom(2) system call was created, it came with the clear warning: "Any userspace program which uses this new functionality must take care to assure that if it is used during the boot process, that it will not cause the init scripts or other portions of the system startup to hang indefinitely. Unfortunately, due to multiple factors, including not having this warning written in a scary enough language in the manpages, and due to glibc since v2.25 implementing a BSD-like getentropy(3) in terms of getrandom(2), modern user-space is calling getrandom(2) in the boot path everywhere. Embedded Linux systems were first hit by this, and reports of embedded system "getting stuck at boot" began to be common. Over time, the issue began to even creep into consumer level x86 laptops: mainstream distributions, like Debian Buster, began to recommend installing haveged as a workaround, just to let the system boot. Filesystem optimizations in EXT4 and XFS exagerated the problem, due to aggressive batching of IO requests, and thus minimizing sources of entropy at boot. This led to large delays until the kernel's Cryptographic Random Number Generator (CRNG) got initialized, and thus having reports of getrandom(2) inidifinitely stuck at boot. Solve this problem by setting a conservative upper bound for getrandom(2) wait. Kill the process, instead of returning an error code, because otherwise crypto-sensitive applications may revert to less secure mechanisms (e.g. /dev/urandom). We __deeply encourage__ system integrators and distribution builders not to considerably increase this value: during system boot, you either have entropy, or you don't. And if you didn't have entropy, it will stay like this forever, because if you had, you wouldn't have blocked in the first place. It's an atomic "either/or" situation, with no middle ground. Please think twice. Ideally, systems would be configured with hardware random number generators, and/or configured to trust the CPU-provided RNG's (CONFIG_RANDOM_TRUST_CPU) or boot-loader provided ones (CONFIG_RANDOM_TRUST_BOOTLOADER). In addition, userspace should generate cryptographic keys only as late as possible, when they are needed, instead of during early boot. (For non-cryptographic use cases, such as dictionary seeds or MIT Magic Cookies, other mechanisms such as /dev/urandom or random(3) may be more appropropriate.) Sounds good? thanks, -- Ahmed Darwish http://darwish.chasingpointers.com This would fail the litmus test that started this thread, re-explained below. 0. Linus applies your patch. 1. A kernel release happens, and it boots fine. 2. Ted Ts'o invents yet another brilliant ext4 optimization, and it gets merged. 3. Somebody discovers that the new kernel kills all his processes, up to and including gnome-session, and that's obviously a
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 08:11:56AM -0400, Theodore Y. Ts'o wrote: > On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: > > Willy Tarreau - 17.09.19, 07:24:38 CEST: > > > On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: > > > > >Well, the patch actually made getrandom() return en error too, but > > > > >you seem more interested in the hypotheticals than in arguing > > > > >actualities.> > > > > If you want to be safe, terminate the process. > > > > > > This is an interesting approach. At least it will cause bug reports in > > > application using getrandom() in an unreliable way and they will > > > check for other options. Because one of the issues with systems that > > > do not finish to boot is that usually the user doesn't know what > > > process is hanging. > > > > I would be happy with a change which changes getrandom(0) to send a > kill -9 to the process if it is called too early, with a new flag, > getrandom(GRND_BLOCK) which blocks until entropy is available. That > leaves it up to the application developer to decide what behavior they > want. > Yup, I'm convinced that's the sanest option too. I'll send a final RFC patch tonight implementing the following: config GETRANDOM_CRNG_ENTROPY_MAX_WAIT_MS int default 3000 help Default max wait in milliseconds, for the getrandom(2) system call when asking for entropy from the urandom source, until the Cryptographic Random Number Generator (CRNG) gets initialized. Any process exceeding this duration for entropy wait will get killed by kernel. The maximum wait can be overriden through the "random.getrandom_max_wait_ms" kernel boot parameter. Rationale follows. When the getrandom(2) system call was created, it came with the clear warning: "Any userspace program which uses this new functionality must take care to assure that if it is used during the boot process, that it will not cause the init scripts or other portions of the system startup to hang indefinitely. Unfortunately, due to multiple factors, including not having this warning written in a scary enough language in the manpages, and due to glibc since v2.25 implementing a BSD-like getentropy(3) in terms of getrandom(2), modern user-space is calling getrandom(2) in the boot path everywhere. Embedded Linux systems were first hit by this, and reports of embedded system "getting stuck at boot" began to be common. Over time, the issue began to even creep into consumer level x86 laptops: mainstream distributions, like Debian Buster, began to recommend installing haveged as a workaround, just to let the system boot. Filesystem optimizations in EXT4 and XFS exagerated the problem, due to aggressive batching of IO requests, and thus minimizing sources of entropy at boot. This led to large delays until the kernel's Cryptographic Random Number Generator (CRNG) got initialized, and thus having reports of getrandom(2) inidifinitely stuck at boot. Solve this problem by setting a conservative upper bound for getrandom(2) wait. Kill the process, instead of returning an error code, because otherwise crypto-sensitive applications may revert to less secure mechanisms (e.g. /dev/urandom). We __deeply encourage__ system integrators and distribution builders not to considerably increase this value: during system boot, you either have entropy, or you don't. And if you didn't have entropy, it will stay like this forever, because if you had, you wouldn't have blocked in the first place. It's an atomic "either/or" situation, with no middle ground. Please think twice. Ideally, systems would be configured with hardware random number generators, and/or configured to trust the CPU-provided RNG's (CONFIG_RANDOM_TRUST_CPU) or boot-loader provided ones (CONFIG_RANDOM_TRUST_BOOTLOADER). In addition, userspace should generate cryptographic keys only as late as possible, when they are needed, instead of during early boot. (For non-cryptographic use cases, such as dictionary seeds or MIT Magic Cookies, other mechanisms such as /dev/urandom or random(3) may be more appropropriate.) Sounds good? thanks, -- Ahmed Darwish http://darwish.chasingpointers.com
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: > Willy Tarreau - 17.09.19, 07:24:38 CEST: > > On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: > > > >Well, the patch actually made getrandom() return en error too, but > > > >you seem more interested in the hypotheticals than in arguing > > > >actualities.> > > > If you want to be safe, terminate the process. > > > > This is an interesting approach. At least it will cause bug reports in > > application using getrandom() in an unreliable way and they will > > check for other options. Because one of the issues with systems that > > do not finish to boot is that usually the user doesn't know what > > process is hanging. > I would be happy with a change which changes getrandom(0) to send a kill -9 to the process if it is called too early, with a new flag, getrandom(GRND_BLOCK) which blocks until entropy is available. That leaves it up to the application developer to decide what behavior they want. Userspace applications which want to do something more sophisticated could set a timer which will cause getrandom(GRND_BLOCK) to return with EINTR (or the signal handler could use longjmp; whatever) to abort and do something else, like calling random_r if it's for some pathetic use of random numbers like MIT-MAGIC-COOKIE. > A userspace process could just poll on the kernel by forking a process > to use getrandom() and waiting until it does not get terminated anymore. > And then it would still hang. So I'm not too worried about that, because if a process is determined to do something stupid, they can always do something stupid. This could potentially be a problem, as would GRND_BLOCK, in that if an application author decides to use to do something to wait for real randomness, because in the good judgement of the application author, it d*mned needs real security because otherwise an attacker could, say, force a launch of nuclear weapons and cause world war III, and then some small 3rd-tier distro decides to repurpose that application for some other use, and puts it in early boot, it's possible that a user will report it as a "regression", and we'll be back to the question of whether we revert a performance optimization patch. There are only two ways out of this mess. The first option is we take functionality away from a userspace author who Really Wants A Secure Random Number Generator. And there are an awful lot of programs who really want secure crypto, becuase this is not a hypothetical. The result in "Mining your P's and Q's" did happen before. If we forget the history, we are doomed to repeat it. The only other way is that we need to try to get the CRNG initialized securely in early boot, before we let userspace start. If we do it early enough, we can also make the kernel facilities like KASLR and Stack Canaries more secure. And this is *doable*, at least for most common platforms. We can leverage UEFI; we cn try to use the TPM's random number generator, etc. It won't help so much for certain brain-dead architectures, like MIPS and ARM, but if they are used for embedded use cases, it will be caught before the product is released for consumer use. And this is where blocking is *way* better than a big fat warning, or sleeping for 15 seconds, both of which can easily get missed in the embedded case. If we can fix this for traditional servers/desktops/laptops, then users won't be complaining to Linus, and I think we can all be happy. Regards, - Ted
Re: Linux 5.3-rc8
Willy Tarreau - 17.09.19, 10:35:16 CEST: > On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: > > However this again would be burdening users with an issue they > > should > > not have to care about. Unless userspace developers care enough and > > manage to take time to fix the issue before updated kernels come to > > their systems. Cause again it would be users systems that would not > > be working. Just cause kernel and userspace developers did not > > agree and chose to fight with each other instead of talking *with* > > each other. > It has nothing to do with fighting at all, it has to do with offering > what applications *need* without breaking existing assumptions that > make most applications work. And more importantly it involves not […] Well I got the impression or interpretation that it would be about fighting… if it is not, all the better! > > At least with killing gdm Systemd may restart it if configured to do > > so. But if it doesn't, the user is again stuck with a non working > > system until restarting gdm themselves. > > > > It may still make sense to make the API harder to use, > > No. What is hard to use is often misused. It must be harder to misuse > it, which means it should be easier to correctly use it. The choice of > flag names and the emission of warnings definitely helps during the > development stage. Sorry, this was a typo of mine. I actually meant harder to abuse. Anything else would not make sense in the context of what I have written. Make it easier to use properly and harder to abuse. > > but it does not > > replace talking with userspace developers and it would need some > > time to allow for adapting userspace applications and services. > > Which is how adding new flags can definitely help even if adoption > takes time. By the way in this discussion I am a userspace developer > and have been hit several times by libraries switching to getrandom() > that silently failed to respond in field. As a userspace developer, I > really want to see a solution to this problem. And I'm fine if the > kernel decides to kill haproxy for using getrandom() with the old > settings, at least users will notice, will complain to me and will > update. Good to see that you are also engaging as a userspace developer in the discussion. Thanks, -- Martin
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 09:33:40AM +0200, Martin Steigerwald wrote: > However this again would be burdening users with an issue they should > not have to care about. Unless userspace developers care enough and > manage to take time to fix the issue before updated kernels come to their > systems. Cause again it would be users systems that would not be > working. Just cause kernel and userspace developers did not agree and > chose to fight with each other instead of talking *with* each other. It has nothing to do with fighting at all, it has to do with offering what applications *need* without breaking existing assumptions that make most applications work. And more importantly it involves not silently breaking applications which need good randomness for long lived keys because the breakage will not be visible initially and can hit them hard later. Right now most applications which block in the early stages are only victim of the current situation and their developers possibly didn't understand the possible impacts of lack of entropy (or how real an issue it was). These applications do need to be able to get low-quality random without blocking forever, provided these are not accidently used by those who need security. At some point, just like for any syscall, the doc makes the difference. > At least with killing gdm Systemd may restart it if configured to do so. > But if it doesn't, the user is again stuck with a non working system > until restarting gdm themselves. > > It may still make sense to make the API harder to use, No. What is hard to use is often misused. It must be harder to misuse it, which means it should be easier to correctly use it. The choice of flag names and the emission of warnings definitely helps during the development stage. > but it does not > replace talking with userspace developers and it would need some time to > allow for adapting userspace applications and services. Which is how adding new flags can definitely help even if adoption takes time. By the way in this discussion I am a userspace developer and have been hit several times by libraries switching to getrandom() that silently failed to respond in field. As a userspace developer, I really want to see a solution to this problem. And I'm fine if the kernel decides to kill haproxy for using getrandom() with the old settings, at least users will notice, will complain to me and will update. Willy
Re: Linux 5.3-rc8
Willy Tarreau - 17.09.19, 07:24:38 CEST: > On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: > > >Well, the patch actually made getrandom() return en error too, but > > >you seem more interested in the hypotheticals than in arguing > > >actualities.> > > If you want to be safe, terminate the process. > > This is an interesting approach. At least it will cause bug reports in > application using getrandom() in an unreliable way and they will > check for other options. Because one of the issues with systems that > do not finish to boot is that usually the user doesn't know what > process is hanging. A userspace process could just poll on the kernel by forking a process to use getrandom() and waiting until it does not get terminated anymore. And then it would still hang. So yes, that would it make it harder to abuse the API, but not impossible. Which may still be good, I don't know. Either the kernel does not reveal at all whether it has seeded CRNG and leaves GnuPG, OpenSSH and others in the dark, or it does and risk that userspace does stupid things whether it prints a big fat warning or not. Of course the warning could be worded like: process blocking on entropy too early on boot without giving the kernel much chance to gather entropy. this is not a kernel issue, report to userspace developers And probably then kill the process, so at least users will know. However this again would be burdening users with an issue they should not have to care about. Unless userspace developers care enough and manage to take time to fix the issue before updated kernels come to their systems. Cause again it would be users systems that would not be working. Just cause kernel and userspace developers did not agree and chose to fight with each other instead of talking *with* each other. At least with killing gdm Systemd may restart it if configured to do so. But if it doesn't, the user is again stuck with a non working system until restarting gdm themselves. It may still make sense to make the API harder to use, but it does not replace talking with userspace developers and it would need some time to allow for adapting userspace applications and services. -- Martin
a sane approach to random numbers (was: Re: Linux 5.3-rc8)
As this is not about Linux 5.3-rc8 anymore I took the liberty to change the subject. Linus Torvalds - 17.09.19, 01:05:47 CEST: > On Mon, Sep 16, 2019 at 4:02 PM Matthew Garrett > wrote: > > The semantics many people want for secure key generation is urandom, > > but with a guarantee that it's seeded. > > And that is exactly what I'd suggest GRND_SECURE should do. > > The problem with: > > getrandom()'s default behaviour at present provides that > > is that exactly because it's the "default" (ie when you don't pass any > flags at all), that behavior is what all the random people get who do > *not* really intentionally want it, they just don't think about it. > > Changing the default (even with kernel warnings) seems like > > it risks people generating keys from an unseeded prng, and that > > seems > > like a bad thing? > > I agree that it's a horrible thing, but the fact that the default 0 > behavior had that "wait for entropy" is what now causes boot problems > for people. Seeing all the discussion, I just got the impression that it may be best to start from scratch. To stop trying to fix something that was broken to begin with – at least that was what I got from the discussion here. Do a sane API with new function names, new flag names and over time deprecate the old one completely so that one day it hopefully could be gradually disabled until it can be removed. Similar like with statx() replacing stat() someday hopefully. And do some documentation about how it is to be used by userspace developers. I.e. like: If the kernel says it is not random, do not block and poll on it, but do something to generate entropy. But maybe that is naive, too. However, in the end, what ever you kernel developers will come up with, I bet there will be no way to make the kernel control userspace developers. However I have the impression that that is what you attempt to do here. As long as you have an API to obtain guaranteed random numbers or at least somewhat guaranteed random numbers that is not directly available at boot time, userspace could poll on its availability. At least as long as the kernel would be honest about its unavailability and tell about it. And if it doesn't applications that *require* random numbers can never know whether they got some from the kernel. Maybe you can make an API that is hard to abuse, yes. And that is good. But impossible? I wonder: How could the Linux experience look like if kernel developers and userspace developers actually work together instead of finding ways to fight each other? I mean, for the most common userspace applications in the free software continuum, there would not be all that many people to talk with, or would there? It is basically gdm, sddm, some other display managers probably, SSH, GnuPG and probably a few more. For example for gdm someone could open a bug report about its use of the current API and ask it to use something that is non blocking? And does Systemd really need to deplete the random pool early at boot in order to generate UUIDs? Even tough I do not use GNOME I'd be willing to help with doing a few bug reports there and there. AFAIR there has been something similar with sddm which I used, but I believe there it has been fixed already with sddm. Sometimes I wonder what would happen if kernel and userspace developers actually *talk* to each other, or better *with* each other. But instead for example with Lennart appears to be afraid to interact with the kernel community and some kernel developers just talked about personalities that they find difficult to interact it, judging them to be like this and like that. There is a social, soft skill issue here that no amount of technical excellence will resolve. That is at least how I observe this. Does it make it easier? Probably not. I fully appreciate that some people may have a difficult time to talk with each other, I experienced this myself often enough. I did not report a bug report with Systemd I found recently just cause I do not like to repeat the experience I had when I reported bugs about it before and I do not use it anymore personally anyway. So I totally get that. However… not talking with each other is not going to resolve those userspace uses kernel API in a way kernel developers do not agree with and that causes issues like stalled boots. Cause basically userspace can abuse any kernel API and in the end the kernel can do nothing about it. Of course feel free to ignore this, if you think it is not useful. Thanks, -- Martin
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 06:46:07PM -0700, Matthew Garrett wrote: > >Well, the patch actually made getrandom() return en error too, but you > >seem more interested in the hypotheticals than in arguing actualities. > > If you want to be safe, terminate the process. This is an interesting approach. At least it will cause bug reports in application using getrandom() in an unreliable way and they will check for other options. Because one of the issues with systems that do not finish to boot is that usually the user doesn't know what process is hanging. Anyway regarding the impact on applications relying on getrandom() for security, I'm in favor of not *silently* changing their behavior and provide a new flag to help others get insecure randoms without waiting. With your option above we could then have this way to go: - GRND_SECURE: the application wants secure randoms, i.e. like the current getrandom(0), waiting for entropy. - GRND_INSECURE: the application never wants to wait, it just wants a replacement for /dev/urandom. - GRND_RANDOM: unchanged, or subject to CAP_xxx, or maybe just emit a "deprecated" warning if called without a certain capability, to spot potentially harmful applications. - by default (0), the application continues to wait but when the timeout strikes (30 seconds ?), it gets terminated with a message in the logs for users to report the issue. After some time all relevant applications which accidently misuse getrandom() will be fixed to either use GRND_INSECURE or GRND_SECURE and be able to wait longer if they want (likely SECURE|NONBLOCK). Willy
Re: Linux 5.3-rc8
On 16 September 2019 18:41:36 GMT-07:00, Linus Torvalds wrote: >On Mon, Sep 16, 2019 at 6:24 PM Matthew Garrett >wrote: >> >> Exactly the scenario where you want getrandom() to block, yes. > >It *would* block. Just not forever. It's already not forever - there's enough running in the background of that system that it'll unblock eventually. >And btw, the whole "generate key at boot when nothing else is going >on" is already broken, so presumably nobody actually does it. If nothing ever did this, why was getrandom() designed in a way to protect against this situation? >See why I'm saying "hypothetical"? You're doing it again. > >> >Then you have to ignore the big warning too. >> >> The big warning that's only printed in dmesg? > >Well, the patch actually made getrandom() return en error too, but you >seem more interested in the hypotheticals than in arguing actualities. If you want to be safe, terminate the process. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 6:24 PM Matthew Garrett wrote: > > Exactly the scenario where you want getrandom() to block, yes. It *would* block. Just not forever. And btw, the whole "generate key at boot when nothing else is going on" is already broken, so presumably nobody actually does it. See why I'm saying "hypothetical"? You're doing it again. > >Then you have to ignore the big warning too. > > The big warning that's only printed in dmesg? Well, the patch actually made getrandom() return en error too, but you seem more interested in the hypotheticals than in arguing actualities. Linus
Re: Linux 5.3-rc8
On 16 September 2019 18:05:57 GMT-07:00, Linus Torvalds wrote: >On Mon, Sep 16, 2019 at 4:29 PM Ahmed S. Darwish >wrote: >> >> Linus, in all honesty, the other case is _not_ a hypothetical . > >Oh yes it is. > >You're confusing "use" with "breakage". > >The _use_ of getrandom(0) for key generation isn't hypothetical. > >But the _breakage_ from the suggested patch that makes it time out is. > >See the difference? > >The thing is, to break, you have to > > (a) do that key generation at boot time > > (b) do it on an idle machine that doesn't have entropy Exactly the scenario where you want getrandom() to block, yes. >in order to basically reproduce the current boot-time hang situation >with the broken gdm, except with an actual "generate key". > >Then you have to ignore the big warning too. The big warning that's only printed in dmesg? -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 4:29 PM Ahmed S. Darwish wrote: > > Linus, in all honesty, the other case is _not_ a hypothetical . Oh yes it is. You're confusing "use" with "breakage". The _use_ of getrandom(0) for key generation isn't hypothetical. But the _breakage_ from the suggested patch that makes it time out is. See the difference? The thing is, to break, you have to (a) do that key generation at boot time (b) do it on an idle machine that doesn't have entropy in order to basically reproduce the current boot-time hang situation with the broken gdm, except with an actual "generate key". Then you have to ignore the big warning too. Linus
Re: Linux 5.3-rc8
On 16 September 2019 16:18:00 GMT-07:00, Linus Torvalds wrote: >On Mon, Sep 16, 2019 at 4:11 PM Matthew Garrett >wrote: >> >> In one case we have "Systems don't boot, but you can downgrade your >> kernel" and in the other case we have "Your cryptographic keys are >weak >> and you have no way of knowing unless you read dmesg", and I think >> causing boot problems is the better outcome here. > >Or: In one case you have a real and present problem. In the other >case, people are talking hypotheticals. (resending because accidental HTML, sorry about that) We've been recommending that people use the default getrandom() behaviour for key generation since it was merged. Github shows users, and it's likely there's cases in internal code as well. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On 16 September 2019 16:18:00 GMT-07:00, Linus Torvalds wrote: >On Mon, Sep 16, 2019 at 4:11 PM Matthew Garrett >wrote: >> >> In one case we have "Systems don't boot, but you can downgrade your >> kernel" and in the other case we have "Your cryptographic keys are >weak >> and you have no way of knowing unless you read dmesg", and I think >> causing boot problems is the better outcome here. > >Or: In one case you have a real and present problem. In the other >case, people are talking hypotheticals. We've been recommending that people use getrandom() for key generation since it was first added to the kernel. Github suggests there are users in the wild - there's almost certainly more cases where internal code depends on the existing semantics. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 10:44:31AM -0700, Linus Torvalds wrote: > - admit that the current situation actually causes problems, and has > _existing_ bugs. > > - throw it out the window, with the timeout and big BIG warning when > the problem cases trigger The semantics many people want for secure key generation is urandom, but with a guarantee that it's seeded. getrandom()'s default behaviour at present provides that, and as a result it's used for a bunch of key generation. Changing the default (even with kernel warnings) seems like it risks people generating keys from an unseeded prng, and that seems like a bad thing? It's definitely unfortunate that getrandom() doesn't have a GRND_URANDOM flag that would make it useful for the "I want some vaguely random numbers but I don't care that much and I don't necessarily have access to /dev/urandom" case, but at the moment we have no way of distinguishing between applications that are making this call because they want the semantics of urandom but need it to be seeded (which is one of the usecases getrandom() was introduced for in the first place) and applications that are making this call because it was convenient and the kernel usually ended up generating enough entropy in the first place. Given the ambiguity, I don't see an easy way to solve for the latter without breaking the former - and that could have some *very* bad outcomes. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 04:18:00PM -0700, Linus Torvalds wrote:
> On Mon, Sep 16, 2019 at 4:11 PM Matthew Garrett wrote:
> >
> > In one case we have "Systems don't boot, but you can downgrade your
> > kernel" and in the other case we have "Your cryptographic keys are weak
> > and you have no way of knowing unless you read dmesg", and I think
> > causing boot problems is the better outcome here.
>
> Or: In one case you have a real and present problem. In the other
> case, people are talking hypotheticals.
>
Linus, in all honesty, the other case is _not_ a hypothetical . For
example, here is a fresh comment on LWN from gnupg developers:
https://lwn.net/Articles/799352
It's about this libgnupg code:
=> https://dev.gnupg.org/source/libgcrypt.git
=> random/rdlinux.c:
/* If we have a modern operating system, we first try to use the new
* getentropy function. That call guarantees that the kernel's
* RNG has been properly seeded before returning any data. This
* is different from /dev/urandom which may, due to its
* non-blocking semantics, return data even if the kernel has
* not been properly seeded. And it differs from /dev/random by never
* blocking once the kernel is seeded. */
#if defined(HAVE_GETENTROPY) || defined(__NR_getrandom)
do {
...
ret = getentropy (buffer, nbytes);
...
} while (ret == -1 && errno == EINTR);
thanks,
--
Ahmed Darwish
http://darwish.chasingpointers.com
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 4:11 PM Matthew Garrett wrote: > > In one case we have "Systems don't boot, but you can downgrade your > kernel" and in the other case we have "Your cryptographic keys are weak > and you have no way of knowing unless you read dmesg", and I think > causing boot problems is the better outcome here. Or: In one case you have a real and present problem. In the other case, people are talking hypotheticals. Linus
Re: Linux 5.3-rc8
On Tue, Sep 17, 2019 at 04:13:36AM +0500, Alexander E. Patrakov wrote: > 17.09.2019 04:11, Matthew Garrett пишет: > > In one case we have "Systems don't boot, but you can downgrade your > > kernel" > > You can't. There are way too many dedicated server providers where there is > no IPMI or any equivalent, and the only help that the staff can do is to > reinstall, wiping your data. In which case you're presumably running a distro kernel that's had a decent amount of testing before you upgrade to it? -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
17.09.2019 04:11, Matthew Garrett пишет: In one case we have "Systems don't boot, but you can downgrade your kernel" You can't. There are way too many dedicated server providers where there is no IPMI or any equivalent, and the only help that the staff can do is to reinstall, wiping your data. -- Alexander E. Patrakov smime.p7s Description: Криптографическая подпись S/MIME
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 04:05:47PM -0700, Linus Torvalds wrote: > On Mon, Sep 16, 2019 at 4:02 PM Matthew Garrett wrote: > > Changing the default (even with kernel warnings) seems like > > it risks people generating keys from an unseeded prng, and that seems > > like a bad thing? > > I agree that it's a horrible thing, but the fact that the default 0 > behavior had that "wait for entropy" is what now causes boot problems > for people. In one case we have "Systems don't boot, but you can downgrade your kernel" and in the other case we have "Your cryptographic keys are weak and you have no way of knowing unless you read dmesg", and I think causing boot problems is the better outcome here. -- Matthew Garrett | mj...@srcf.ucam.org
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 4:02 PM Matthew Garrett wrote: > > The semantics many people want for secure key generation is urandom, but > with a guarantee that it's seeded. And that is exactly what I'd suggest GRND_SECURE should do. The problem with: > getrandom()'s default behaviour at present provides that is that exactly because it's the "default" (ie when you don't pass any flags at all), that behavior is what all the random people get who do *not* really intentionally want it, they just don't think about it. > Changing the default (even with kernel warnings) seems like > it risks people generating keys from an unseeded prng, and that seems > like a bad thing? I agree that it's a horrible thing, but the fact that the default 0 behavior had that "wait for entropy" is what now causes boot problems for people. Linus
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 01:21:17PM -0400, Theodore Y. Ts'o wrote: > On Mon, Sep 16, 2019 at 09:17:10AM -0700, Linus Torvalds wrote: > > So the semantics that getrandom() should have had are: > > > > getrandom(0) - just give me reasonable random numbers for any of a > > million non-strict-long-term-security use (ie the old urandom) > > > > - the nonblocking flag makes no sense here and would be a no-op > > That change is what I consider highly problematic. There are a *huge* > number of applications which use cryptography which assumes that > getrandom(0) means, "I'm guaranteed to get something safe > cryptographic use". Changing his now would expose a very large number > of applications to be insecure. Part of the problem here is that > there are many different actors. There is the application or > cryptographic library developer, who may want to be sure they have > cryptographically secure random numbers. They are the ones who will > select getrandom(0). > > Then you have the distribution or consumer-grade electronics > developers who may choose to run them too early in some init script or > systemd unit files. And some of these people may do something stupid, > like run things too early, or omit the a hardware random number > generator in their design, even though it's for a security critical > purpose (say, a digital wallet for bitcoin). Ted, you're really the expert here. My apologies though, every time I see the words "too early" I get a cramp... Please check my earlier reply: https://lkml.kernel.org/r/20190912034421.GA2085@darwi-home-pc Specifically the trace_printk log of all the getrandom(2) calls during an standard Archlinux boot... where is the "too early" boundary there? It's undefinable. You either have entropy, or you don't. And if you don't, it will stay like this forever, because if you had, you wouldn't have blocked in the first place... Thanks, -- Ahmed Darwish http://darwish.chasingpointers.com
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 10:44:31AM -0700, Linus Torvalds wrote: > - add new GRND_SECURE and GRND_INSECURE flags that have the actual > useful behaviors that we currently pretty much lack > > - consider the old 0-3 flag values legacy, deprecated, and unsafe > because they _will_ time out to fix the existing problem we have right > now because of their bad behavior. I think we can keep a flag to work like the current /dev/random and deplete entropy for the very rare cases where it's really desired to run this way (maybe even just for research), but it should require special permissions as it impacts the whole system. I think that your GRND_SECURE above means the current 0 situation, where we wait for initial entropy then not wait anymore, right ? If so it could remain the default setting, because at least it will not betray applications which rely on this reliability. And GRND_INSECURE will be decided on a case by case basis by applications that are caught waiting like sfdisk in initramfs or a MAC address generator for example. In this case it could even be called GRND_PREDICTABLE maybe to enforce its property compared to others. My guess is that we can fix the situation because nobody likes the problems that sporadically hit users. getrandom() was adopted quite quickly to solve issues related to using /dev/*random in chroots, I think the new flags will be adopted by those experiencing issues. Just my two cents, Willy
Re: Linux 5.3-rc8
16.09.2019 22:21, Theodore Y. Ts'o пишет: On Mon, Sep 16, 2019 at 09:17:10AM -0700, Linus Torvalds wrote: So the semantics that getrandom() should have had are: getrandom(0) - just give me reasonable random numbers for any of a million non-strict-long-term-security use (ie the old urandom) - the nonblocking flag makes no sense here and would be a no-op That change is what I consider highly problematic. There are a *huge* number of applications which use cryptography which assumes that getrandom(0) means, "I'm guaranteed to get something safe cryptographic use". Changing his now would expose a very large number of applications to be insecure. Part of the problem here is that there are many different actors. There is the application or cryptographic library developer, who may want to be sure they have cryptographically secure random numbers. They are the ones who will select getrandom(0). Then you have the distribution or consumer-grade electronics developers who may choose to run them too early in some init script or systemd unit files. And some of these people may do something stupid, like run things too early, or omit the a hardware random number generator in their design, even though it's for a security critical purpose (say, a digital wallet for bitcoin). Because some of these people might do something stupid, one argument (not mine) is that we must therefore not let getrandom() block. But doing this penalizes the security of all the users of the application, not just the stupid ones. On Linux, there is no such thing as "too early", that's the problem. First, we already had one lesson about this, regarding applications that require libraries from /usr. There, it was due to various programs that run from udev rules, and dynamic/unpredictable dependencies. See https://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/, almost all arguments from there apply 1:1 here. Second, people/distributions put unexpected stuff into their initramfs images, and we cannot say that they have no right to do so. E.g., on my system that's "cryptsetup" that unlocks the root partition, but manages to read a few bytes of uninitialized urandom before that. A warning here is almost unavoidable, and thus will be treated as SPAM. No such considerations apply to OpenBSD (initramfs does not exist, and there is no equivalent of udev that reacts to cold-plug events by running programs), that's why the getentropy() design works there. If we were to fix it, we should focus on making true entropy available unconditionally, even before /init in the initramfs starts, and warn not on the first access to urandom, but on the exec of /init. Look - distributions are already running "haveged" which harvests entropy from clock jitter. And they still manage to do it wrong (regardless whether the "haveged" idea is wrong by itself), by running it too late (at least I don't know any kind of stock initramfs with either it or rngd included). So it's too complex, and needs to be simplified. The kernel already has jitterentropy-rng, it uses the same idea as "haveged", but, alas, it is exposed as a crypto rng algorithm, not a hwrng. And I think it is a bug: cryptoapi rng algorithms are for things that get a seed and generate random numbers by rehashing it over and over, while jitterentropy-rng requires no seed. Would a patch be accepted to convert it to hwrng? (this is essentially the reverse of what commit c46ea13 did for exynos-rng) getrandom(GRND_RANDOM) - get me actual _secure_ random numbers with blocking until entropy pool fills (but not the completely invalid entropy decrease accounting) - the nonblocking flag is useful for bootup and for "I will actually try to generate entropy". and both of those are very very sensible actions. That would actually have _fixed_ the problems we had with /dev/[u]random, both from a performance standpoint and for a filesystem access standpoint. But that is sadly not what we have right now. And I suspect we can't fix it, since people have grown to depend on the old behavior, and already know to avoid GRND_RANDOM because it's useless with old kernels even if we fixed it with new ones. I don't think we can fix it, because it's the changing of getrandom(0)'s behavior which is the problem, not GRND_RANDOM. People *expect* getrandom(0) to always return secure results. I don't think we can make it sometimes return not-necessarily secure results depending on when the systems integrator or distribution decides to run the application, and depending on the hardware platform (yes, traditional x86 systems are probably fine, and fortunately x86 embedded CPU are too expensive and have lousy power management, so no one really uses x86 for embedded yet, despite Intel's best efforts). That would just be a purely irresponsible thing to do, IMO. Does anybody really seriously debate the above? Ted? Are you seriously trying to claim that the
Re: Linux 5.3-rc8
> - add new GRND_SECURE and GRND_INSECURE flags that have the actual > useful behaviors that we currently pretty much lack > > - consider the old 0-3 flag values legacy, deprecated, and unsafe > because they _will_ time out to fix the existing problem we have right > now because of their bad behavior. Just for the record because I did not see it mentioned in this thread, this patch by Andy Lutomirski, posted two weeks ago, adds GRND_INSECURE and makes GRND_RANDOM a no-op: https://lore.kernel.org/lkml/cover.1567126741.git.l...@kernel.org/
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 10:21 AM Theodore Y. Ts'o wrote: > > We could create a new flag, GRND_INSECURE, which never blocks. And > that that allows us to solve the problem for silly applications that > are using getrandom(2) for non-cryptographic use cases. Note that getting "reasonably good random numbers" is definitely not silly. If you are doing things like just shuffling a deck of cards for playing solitaire on your computer, getting a good enough source of randomness is nontrivial. Using getrandom() for that is a _very_ valid use. But it obviously does not need _secure_ random numbers. It is, in fact, _so_ random that we give that AT_RANDOM thing to every new process because people want things like that. Sadly, people often aren't aware of it, and don't use that as much as they could. (Btw, we should probably also mix in other per-process state, because right now people have actually attacked the boot-time AT_RANDOM to find canary data etc). So I think you are completely out to lunch by calling these "insecure" things "silly". They are very very common. *WAY* more common than making a long-term secure key will ever be. There's just a lot of use of reasonable randomness. You also are ignoring that we have an existing problem with existing applications. That happened exactly because those things are so common. So here's my suggestion: - admit that the current situation actually causes problems, and has _existing_ bugs. - throw it out the window, with the timeout and big BIG warning when the problem cases trigger - add new GRND_SECURE and GRND_INSECURE flags that have the actual useful behaviors that we currently pretty much lack - consider the old 0-3 flag values legacy, deprecated, and unsafe because they _will_ time out to fix the existing problem we have right now because of their bad behavior. And stop with the "insecure is silly". Insecure is not silly, and in fact should have been the default, because (a) insecure is and basically always will be the common case by far (b) insecure is the "not thinking about it" case and should thus be default and that (b) is also very much why 0 should have been that insecure case. Part of the problem is exactly the whole "_normally_ it just works, so using 0 without thinking about it tests out well". Which is why getrandom(0) is the main problem we face. Because defaults matter. Linus
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 09:17:10AM -0700, Linus Torvalds wrote: > So the semantics that getrandom() should have had are: > > getrandom(0) - just give me reasonable random numbers for any of a > million non-strict-long-term-security use (ie the old urandom) > > - the nonblocking flag makes no sense here and would be a no-op That change is what I consider highly problematic. There are a *huge* number of applications which use cryptography which assumes that getrandom(0) means, "I'm guaranteed to get something safe cryptographic use". Changing his now would expose a very large number of applications to be insecure. Part of the problem here is that there are many different actors. There is the application or cryptographic library developer, who may want to be sure they have cryptographically secure random numbers. They are the ones who will select getrandom(0). Then you have the distribution or consumer-grade electronics developers who may choose to run them too early in some init script or systemd unit files. And some of these people may do something stupid, like run things too early, or omit the a hardware random number generator in their design, even though it's for a security critical purpose (say, a digital wallet for bitcoin). Because some of these people might do something stupid, one argument (not mine) is that we must therefore not let getrandom() block. But doing this penalizes the security of all the users of the application, not just the stupid ones. > getrandom(GRND_RANDOM) - get me actual _secure_ random numbers with > blocking until entropy pool fills (but not the completely invalid > entropy decrease accounting) > > - the nonblocking flag is useful for bootup and for "I will > actually try to generate entropy". > > and both of those are very very sensible actions. That would actually > have _fixed_ the problems we had with /dev/[u]random, both from a > performance standpoint and for a filesystem access standpoint. > > But that is sadly not what we have right now. > > And I suspect we can't fix it, since people have grown to depend on > the old behavior, and already know to avoid GRND_RANDOM because it's > useless with old kernels even if we fixed it with new ones. I don't think we can fix it, because it's the changing of getrandom(0)'s behavior which is the problem, not GRND_RANDOM. People *expect* getrandom(0) to always return secure results. I don't think we can make it sometimes return not-necessarily secure results depending on when the systems integrator or distribution decides to run the application, and depending on the hardware platform (yes, traditional x86 systems are probably fine, and fortunately x86 embedded CPU are too expensive and have lousy power management, so no one really uses x86 for embedded yet, despite Intel's best efforts). That would just be a purely irresponsible thing to do, IMO. > Does anybody really seriously debate the above? Ted? Are you seriously > trying to claim that the existing GRND_RANDOM has any sensible use? > Are you seriously trying to claim that the fact that we don't have a > sane urandom source is a "feature"? There are people who can debate that GRND_RANDOM has any sensible use cases. GPG uses /dev/random, and that was a fully informed choice. I'm not convinced, because I think that at least for now the CRNG is perfectly fine for 99.999% of the use cases. Yes, in a post-quantum cryptography world, the CRNG might be screwed --- but so will most of the other cryptographic algorithms in the kernel. So if anyone ever gets post-quantum cryptoanalytic attacks working, the use of the CRNG is going to be least of our problems. As I mentioned to you in Lisbon, I've been going back and forth about whether or not to rip out the entire /dev/random infrastructure, mainly for code maintainability reasons. The only reason why I've been holding back is because there are (very few) non-insane people who do want to use it. There are also a much larger of rational people who use it because they want some insane PCI compliance labs to go away. What I suspect most of them are actually doing in practice is they use /dev/random, but they also use a hardware random number generator so /dev/random never actually blocks in practice. The use of /dev/random is enough to make the PCI compliance lab go away, and the hardware random number generator (or virtio-rng on a VM) makes /dev/random useable. But I don't think we can reuse GRND_RANDOM for that reason. We could create a new flag, GRND_INSECURE, which never blocks. And that that allows us to solve the problem for silly applications that are using getrandom(2) for non-cryptographic use cases. Use cases might include Python dictionary seeds, gdm for MIT Magic Cookie, UUID generation where best efforts probably is good enough, etc. The answer today is they should just use /dev/urandom, since that exists today, and we have to support it for backwards compatibility anyway. It sounds
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 10:00 AM Theodore Y. Ts'o wrote: > > How /dev/random blocks is very different from how getrandom(2) blocks. > Getrandom(2) blocks until the CRNG, and then it never blocks again. Yes and no. getrandom() very much blocks exactly like /dev/random, when you give it the GRND_RANDOM flag. Which is completely broken, and was already known to be broken. So that flag is just plain stupid. And getrandom() does *not* block like /dev/urandom does (ie not at all), which was actually useful, and very widely used. So you really have the worst of both worlds. Yes, getrandom(0) does what /dev/random _should_ have done, and what getrandom(GRND_RANDOM) should be but isn't. But by making the choice it did, we now have three useless flag combinations, and we lack one people _want_ and need. And this design mistake very much caused the particular bug we are now hitting. Linus
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 08:40:30PM -0700, Linus Torvalds wrote: > On Sun, Sep 15, 2019 at 8:23 PM Theodore Y. Ts'o wrote: > > > > But not blocking is *precisely* what lead us to weak keys in network > > devices that were sold by the millions to users in their printers, > > wifi routers, etc. > > Ted, just admit that you are wrong on this, instead of writing the > above kind of bad fantasy. > > We have *always* supported blocking. It's called "/dev/random". And > guess what? Not blocking wasn't what lead to weak keys like you try to > imply. > > What led to weak keys is that /dev/random is useless and nobody sane > uses it, exactly because it always blocks. How /dev/random blocks is very different from how getrandom(2) blocks. Getrandom(2) blocks until the CRNG, and then it never blocks again. /dev/random tries to do entropy accounting, and it blocks randomly all the time. *That* is why it is useless. I agree that /dev/random is bad, but I think you're taking the wrong message from it. It's not that blocking is always bad; it's that insisting on entropy accounting and "true randomness" is bad. The getrandom(2) system call is modelled after *BSD's getentropy(2) call, and the fact that everyone is using is because for most use cases, it really is the right way to go. I think that's the core of my disagreement with you. I agree that what /dev/random does is wrong, and to date, we've kept it for backwards compatibility reasons. Some of these reasons could be rational, or at least debated. For example, GPG wants to use /dev/random because it thinks it's more secure, and if they are generating 4096 bit RSA keys, or something else which might be "post-quantuum cryptography", it's possible that /dev/random is going to be better than the CRNG for the hyper-paranoid. Other use cases, such as some PCI compliance labs who think that getrandom(2) is not sufficiently secure, are just purely insane --- but that's assuming today's getrandom(2) is guaranteed to return cryptographically strong results, or nothing at all. If we change the existing behavior of getrandom(2) with the default flags to mean, "we return whatever we feel like, and this includes something which looks random, but might be trivially reverse engineered by a research engineer", that is in my mind, a Really Bad Thing To Do. And no, a big fat warning isn't sufficient, because there will be some systems integrators and application programmers who will ignore the kernel warning message. They might not even look at dmesg, and a system console might not exist. - Ted
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 11:13 PM Willy Tarreau wrote: > > > > > So three out of four flag combinations end up being mostly "don't > > use", and the fourth one isn't what you'd normally want (which is just > > plain /dev/urandom semantics). > > I'm seeing it from a different angle. I now understand better why > getrandom() absolutely wants to have an initialized pool, it's to > encourage private key producers to use a secure, infinite source of > randomness. Right. There is absolutely no question that that is a useful thing to have. And that's what GRND_RANDOM _should_ have meant. But didn't. So the semantics that getrandom() should have had are: getrandom(0) - just give me reasonable random numbers for any of a million non-strict-long-term-security use (ie the old urandom) - the nonblocking flag makes no sense here and would be a no-op getrandom(GRND_RANDOM) - get me actual _secure_ random numbers with blocking until entropy pool fills (but not the completely invalid entropy decrease accounting) - the nonblocking flag is useful for bootup and for "I will actually try to generate entropy". and both of those are very very sensible actions. That would actually have _fixed_ the problems we had with /dev/[u]random, both from a performance standpoint and for a filesystem access standpoint. But that is sadly not what we have right now. And I suspect we can't fix it, since people have grown to depend on the old behavior, and already know to avoid GRND_RANDOM because it's useless with old kernels even if we fixed it with new ones. Does anybody really seriously debate the above? Ted? Are you seriously trying to claim that the existing GRND_RANDOM has any sensible use? Are you seriously trying to claim that the fact that we don't have a sane urandom source is a "feature"? Linus
Re: Linux 5.3-rc8
On Tue, Sep 10, 2019 at 07:56:35AM -0400, Theodore Y. Ts'o wrote: > Hmm, I'm not seeing this on a Dell XPS 13 (model 9380) using a Debian > Bullseye (Testing) running a rc4+ kernel. > > This could be because Debian is simply doing more I/O; or it could be > because I don't have some package installed which is trying to reading > from /dev/random or calling getrandom(2). Previously, Fedora ran into > blocking issues because of some FIPS compliance patches to some > userspace daemons. So it's going to be very user space dependent and > package dependent. Btw, I've been seeing this issue on debian testing with an XFS root file system ever since the blocking random changes went in. There are a few reports (not from me) in the BTS since. I ended up just giving up on gdm and using lightdm instead as it was clearly related to that.
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 10:02:02PM -0700, Linus Torvalds wrote: > On Sun, Sep 15, 2019 at 9:30 PM Willy Tarreau wrote: > > > > I'd be in favor of adding in the man page something like "this > > random source is only suitable for applications which will not be > > harmed by getting a predictable value on output, and as such it is > > not suitable for generation of system keys or passwords, please > > use GRND_RANDOM for this". > > The problem with GRND_RANDOM is that it also ends up extracting > entropy, and has absolutely horrendous performance behavior. It's why > hardly anybody uses /dev/random. > > Which nobody should really ever do. I don't understand why people want > that thing, considering that the second law of thermodynamics really > pretty much applies. If you can crack the cryptographic hashes well > enough to break them despite reseeding etc, people will have much more > serious issues than the entropy accounting. That's exactly what I was thinking about a few minutes ago and which drove me back to mutt :-) > So the problem with getrandom() is that it only offered two flags, and > to make things worse they were the wrong ones. (...) > - GRND_RANDOM | GRND_NONBLOCK - don't use > >This won't block, but it will decrease the blocking pool entropy. > >It might be an acceptable "get me a truly secure ring with reliable > performance", but when it fails, you're going to be unhappy, and there > is no obvious fallback. > > So three out of four flag combinations end up being mostly "don't > use", and the fourth one isn't what you'd normally want (which is just > plain /dev/urandom semantics). I'm seeing it from a different angle. I now understand better why getrandom() absolutely wants to have an initialized pool, it's to encourage private key producers to use a secure, infinite source of randomness. Something that neither /dev/random nor /dev/urandom reliably provide. Unfortunately it does it by changing how urandom works while it ought to have done it as the replacement of /dev/random. The 3 random generation behaviors we currently support are : - /dev/random: only returns safe random (blocks), AND depletes entropy. getrandom(GRND_RANDOM) does the same. - /dev/urandom: returns whatever (never blocks), inexhaustible - getrandom(0): returns safe random (blocks), inexhaustible Historically we used to want to rely on /dev/random for SSH keys and certificates. It's arguable that with the massive increase of crypto usage, what used to be done only once in a system's life happens a bit more often and using /dev/random here can sometimes become a problem because it harms the whole system (thus why I said I think that we could almost require CAP_something to access it). Applications falling back to /dev/urandom obviously resulted in the massive mess we've seen years ago, even if it apparently solved the problem for their users. Thus getrandom(0) does make sense, but not as an alternative to urandom but to random, since it returns randoms safe for use for long lived keys. Couldn't we simply change the way things work ? Make GRND_RANDOM *not* deplate entropy, and document it as the only safe source, and make the default call return the same as /dev/urandom ? We can then use your timeout mechanism for the first one (which is not supposed to be called often and would be more accepted with a moderately long delay). Applications need to evolve as well. It's fine to use libraries to do whatever you need for you but ultimately the lib exports a function for a generic use case and doesn't know how to best adapt to the use case. Typically I would expect an SSH/HTTP daemon running in a recovery initramfs to produce unsafe randoms so that I can connect there without having to dance around it. However the self-signed cert produced there must not be saved, just like the SSH host key. But this means that the application (here the ssh-keygen or openssl) also need to be taught to purposely produce insecure keys when explicitly instructed to do so. Otherwise we know what will happen in the long term, since history repeats itself as long as the conditions are not changed :-/ Willy
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 9:30 PM Willy Tarreau wrote: > > I'd be in favor of adding in the man page something like "this > random source is only suitable for applications which will not be > harmed by getting a predictable value on output, and as such it is > not suitable for generation of system keys or passwords, please > use GRND_RANDOM for this". The problem with GRND_RANDOM is that it also ends up extracting entropy, and has absolutely horrendous performance behavior. It's why hardly anybody uses /dev/random. Which nobody should really ever do. I don't understand why people want that thing, considering that the second law of thermodynamics really pretty much applies. If you can crack the cryptographic hashes well enough to break them despite reseeding etc, people will have much more serious issues than the entropy accounting. So the problem with getrandom() is that it only offered two flags, and to make things worse they were the wrong ones. Nobody should basically _ever_ use the silly "entropy can go away" model, yet that is exactly what GRND_RANDOM does. End result: GRND_RANDOM is almost entirely useless, and is actively dangerous, because it can actually block not just during boot, it can block (and cause others to block) during random running of the system because it does that entropy accounting(). Nobody can use GRND_RANDOM if they have _any_ performance requirements what-so-ever. It's possibly useful for one-time ssh host keys etc. So GRND_RANDOM is just bad - with or without GRND_NONBLOCK, because even in the nonblocking form it will account for entropy in the blocking pool (until it's all gone, and it will return -EAGAIN). And the non-GRND_RANDOM case avoids that problem, but requires the initial entropy with no way to opt out of it. Yes, GRND_NONBLOCK makes it work. So we have four flag combinations: - 0 - don't use if it could possibly run at boot Possibly useful for the systemd-random-seed case, and if you *know* you're way past boot, but clearly overused. This is the one that bit us this time. - GRND_NONBLOCK - fine, but you now don't get even untrusted random numbers, and you have to come up with a way to fill the entropy pool This one is most useful as a quick "get me urandom", but needs a fallback to _actual_ /dev/urandom when it fails. This is the best choice by far, and has no inherent downsides apart from needing that fallback code. - GRND_RANDOM - don't use This will block and it will decrease the blocking pool entropy so that others will block too, and has horrible performance. Just don't use it outside of very occasional non-serious work. Yes, it will give you secure numbers, but because of performance issues it's not viable for any serious code, and obviously not for bootup. It can be useful as a seed for future serious use that just does all random handling in user space. Just not during boot. - GRND_RANDOM | GRND_NONBLOCK - don't use This won't block, but it will decrease the blocking pool entropy. It might be an acceptable "get me a truly secure ring with reliable performance", but when it fails, you're going to be unhappy, and there is no obvious fallback. So three out of four flag combinations end up being mostly "don't use", and the fourth one isn't what you'd normally want (which is just plain /dev/urandom semantics). Linus
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 09:21:06PM -0700, Linus Torvalds wrote:
> The timer interrupt could be somewhat interesting if you are also
> CPU-bound on a non-trivial load, because then "what program counter
> got interrupted" ends up being possibly unpredictable - even with a
> very stable timer interrupt source - and effectively stand in for a
> cycle counter even on hardware that doesn't have a native TSC. Lots of
> possible low-level jitter there to use for entropy. But especially if
> you're just idly _waiting_ for entropy, you won't be "CPU-bound on an
> interesting load" - you'll just hit the CPU idle loop all the time so
> even that wouldn't work.
In the old DOS era, I used to produce randoms by measuring the time it
took for some devices to reset themselves (typically 8250 UARTs could
take in the order of milliseconds). And reading their status registers
during the reset phase used to show various sequences of flags at
approximate timings.
I suspect this method is still usable, even with SoCs full of peripherals,
in part because not all clocks are synchronous, so we can retrieve a
little bit of entropy by measuring edge transitions. I don't know how
we can assess the number of bits provided by such method (probably
log2(card(discrete values))) but maybe this is something we should
progressively encourage drivers authors to do in the various device
probing functions once we figure the best way to do it.
The idea is around this. Instead of :
probe(dev)
{
(...)
while (timeout && !(status_reg & STATUS_RDY))
timeout--;
(...)
}
We could do something like this (assuming 1 bit of randomness here) :
probe(dev)
{
(...)
prev_timeout = timeout;
prev_reg = status_reg;
while (timeout && !(status_reg & STATUS_RDY)) {
if (status_reg != prev_reg) {
add_device_randomness_bits(timeout - prev_timeout, 1);
prev_timeout = timeout;
prev_reg = status_reg;
}
timeout--;
}
(...)
}
It's also interesting to note that on many motherboards there are still
multiple crystal oscillators (typically one per ethernet port) and that
such types of independent, free-running clocks do present unpredictable
edges compared to the CPU's clock, so when they affect the device's
setup time, this does help quite a bit.
Willy
Re: Linux 5.3-rc8
Hi Ted, On Sun, Sep 15, 2019 at 10:49:04PM -0400, Theodore Y. Ts'o wrote: > No matter *what* we do, it's going to either (a) make some > systems insecure, or (b) make some systems more likely hang while > booting. I continue to strongly disagree with opposing these two. (b) is caused precisely because of this conflation. Life long keys are produced around once per system's life (at least this order of magnitude). Boot happens way more often. Users would not complain that systems fail to start if the two types of random are properly distinguished so that we don't fail to boot just for the sake of secure randoms that will never be consumed as such. Before systems had HWRNGs it was pretty common for some tools to ask the user to type hundreds of characters on the keyboard and use that (content+timings) to feed entropy while generating a key. This is acceptable once in a system's life. And on some systems with no entropy like VMs, it's commonly generated from a central place and never from the VM itself, so it's not a problem either. In my opinion the problem recently happened because getrandom() was perceived as a good replacement for /dev/urandom and is way more convenient to use, so applications progressively started to use it without realizing that contrary to its ancestor it can block. And each time a system fails to boot confirms that entropy still remains a problem even on PCs in 2019. This is one more reason for clearly keeping two interfaces depending on what type of random is needed. I'd be in favor of adding in the man page something like "this random source is only suitable for applications which will not be harmed by getting a predictable value on output, and as such it is not suitable for generation of system keys or passwords, please use GRND_RANDOM for this". This distinction currently is not clear enough for people who don't know this subtle difference, and can increase the interface's misuse. Regards, Willy
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 8:52 PM Herbert Xu wrote: > > Can we perhaps artifically increase the interrupt rate while the > CRNG is not initialised? Long term (or even medium term in some areas), the problem really is that device interrupts during boot really are going away, rather than becoming more common. That just happened to be the case now because of improved plugging, but it's fundamentally the direction any storage is moving with faster flash interfaces. The only interrupt we could easily increase the rate of in the kernel is the timer interrupt, but that's also the interrupt that is the least useful for randomness. The timer interrupt could be somewhat interesting if you are also CPU-bound on a non-trivial load, because then "what program counter got interrupted" ends up being possibly unpredictable - even with a very stable timer interrupt source - and effectively stand in for a cycle counter even on hardware that doesn't have a native TSC. Lots of possible low-level jitter there to use for entropy. But especially if you're just idly _waiting_ for entropy, you won't be "CPU-bound on an interesting load" - you'll just hit the CPU idle loop all the time so even that wouldn't work. But practically speaking timers really are not really much of an option. And if we are idle, even having a high-frequency TSC isn't all that useful with the timer interrupt, because the two tend to be very intimately related. Of course, if you're generating a host key for SSH or something like that, you could try to at least cause some network traffic while generating the key. That's not much of an option for the _kernel_, but for a program like ssh-keygen it certainly could be. Blocking is fine if you simply don't care about time at all (the "five hours later is fine" situation), or if you have some a-priori knowledge that the machine is doing real interesting work that will generate entropy. But I don't see how the kernel can generate entropy on its own, particularly during boot (which is when the problem happens), when most devices aren't even necessarily meaningfully set up yet. Hopefully hw random number generators will make this issue effectively moot before we really end up having the "nvdimms and their ilk are common enough that you really have no early boot irq-driven disk IO at all". Linus
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 8:40 PM Linus Torvalds wrote: > > If you want secure keys, you can't rely on a blocking model, because > it ends up not working. Blocking leads to problems. Side note: I'd argue that (despite my earlier mis-understanding) the only really valid use of "block until there is entropy" is the systemd-random-seed model that blocks not because it wants a secure key, but blocks because it wants to save the (now properly) random seed for later. So apologies to Lennart - he was very much right, and I mis-understood Ahmed's bug report. Systemd was blameless, and blocked correctly. While blocking for actual random keys was the usual bug, just for that silly and pointless MIT cookie that doesn't even need the secure randomness. But because the getrandom() interface was mis-designed (and only _looks_ like a more convenient interface for /dev/urandom, without being one), the MIT cookie code got the blocking whether it wanted to or not. Just say no to blocking for key data. Linus
Re: Linux 5.3-rc8
Theodore Y. Ts'o wrote: > > Ultimately, though, we need to find *some* way to fix userspace's > assumptions that they can always get high quality entropy in early > boot, or we need to get over people's distrust of Intel and RDRAND. > Otherwise, future performance improvements in any part of the system > which reduces the number of interrupts is always going to potentially > result in somebody's misconfigured system or badly written > applications to fail to boot. :-( Can we perhaps artifically increase the interrupt rate while the CRNG is not initialised? Cheers, -- Email: Herbert Xu Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 8:23 PM Theodore Y. Ts'o wrote: > > But not blocking is *precisely* what lead us to weak keys in network > devices that were sold by the millions to users in their printers, > wifi routers, etc. Ted, just admit that you are wrong on this, instead of writing the above kind of bad fantasy. We have *always* supported blocking. It's called "/dev/random". And guess what? Not blocking wasn't what lead to weak keys like you try to imply. What led to weak keys is that /dev/random is useless and nobody sane uses it, exactly because it always blocks. So you claim that it is lack of blocking that is the problem, but you're ignoring reality. You are ignoring the very real fact that blocking is what led to people not using the blocking interface in the first place, because IT IS THE WRONG MODEL. It really is fundamentally wrong. Blocking by definition will never work, because it doesn't add any entropy. So people then don't use the blocking interface, because it doesn't _work_. End result: they then use another interface that does work, but isn't secure. I have told you that in this thread, and HISTORY should have told you that. You're not listening. If you want secure keys, you can't rely on a blocking model, because it ends up not working. Blocking leads to problems. If you want secure keys, you should do the exact opposite of blocking: you should encourage people to explicitly use a non-blocking "I want secure random numbers", and then if that fails, they should do things that cause entropy. So getrandom() just repeated a known broken model. And you're parroting that same old known broken stuff. It didn't work with /dev/random, why do you think it magically works with getrandom()? Stop fighting reality. The fact is, either you have sufficient entropy or you don't. - if you have sufficient entropy, blocking is stupid and pointless - if you don't have sufficient entropy, blocking is exactly the wrong thing to do. Seriously. Don't make excuses for bad interfaces. We should have learnt this long ago. Linus
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 6:41 PM Ahmed S. Darwish wrote: > > Yes, the systemd-random-seed(8) process blocks, but this is an > isolated process, and it's only there as a synchronization point and > to load/restore random seeds from disk across reboots. > > What blocked the system boot was GDM/gnome-session implicitly calling > getrandom() for the Xorg MIT cookie. Aahh. I saw that email, but then in the discussion the systemd case always ended up coming up first, and I never made the connection. What a complete crock that silly MIT random cookie is, and what a sad sad reason for blocking. Linus
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 10:02:18AM -0700, Linus Torvalds wrote: > But on a PC, we can _almost_ guarantee entropy. Even with a golden > image, we do mix in: > > - timestamp counter on every device interrupt (but "device interrupt" > doesn't include things like the local CPU timer, so it really needs > device activity) > > - random boot and BIOS memory (dmi tables, the EFI RNG entry, etc) > > - various device state (things like MAC addresses when registering > network devices, USB device numbers, etc) > > - and obviously any CPU rdrand data > > But also note the "on a PC" part. Hopefully there is no disagreement with this. I completely agree that if we only care about user desktops running on PC's, getrandom(2) should never block, and *hopefully* a big fact kernel stack dump will cause developers to wake up and pay attention. And even if they don't essentially all modern systems have RDRAND, and RDRAND will save you. We're also not using the EFI RNG yet, but we should, and once we do, that will again help for all modern PC's. However, there are exceptions here --- and we don't even need to leave the X86 architecture. If you are running in a VM, there won't be a lot of interrutps, and some hosts may disable RDRAND (or are on a system where RDRAND was buggy, and hence disabled), and the dmi tables are pretty much constant and trivial for an attacker to deduce. > But basically, you should never *ever* try to generate some long-lived > key and then just wait for it without doing anything else. The > "without doing anything else" is key here. > > But every time we've had a blocking interface, that's exactly what > somebody has done. Which is why I consider that long blocking thing to > be completely unacceptable. There is no reason to believe that the > wait will ever end, partly exactly because we don't consider timer > interrupts to add any timer randomness. So if you are just waiting, > nothing necessarily ever happen. Ultimately, the question is whether blocking is unacceptable, or compromising the user's security is unacceptable. The former is much more likely to cause users to whine on LKML and send complaints of regressions to Linus. No question about that. But not blocking is *precisely* what lead us to weak keys in network devices that were sold by the millions to users in their printers, wifi routers, etc. And with /dev/urandom, we didn't block, and we did issue a warning messages, and it didn't stop consumer electronic vendors from screwing up. And then there will be another paper published, and someone will contact secur...@kernel.org, and it will be blamed on the Linux kernel, because best practice really *is* to block until you can return cryptographic randomness, because we can take it on *faith* that there will be some (and probably many) user space programmers which rally don't know how to do system design, especially secure systems design. Many of them won't even bother to look at system logs. And even blocking for 15 seconds may not necessarily help much, since consumer grade electronics won't have a serial console, and hardware engineers might not even notice a 15 second delay. Sure, someone who is used to a laptop booting up in 3 seconds will be super annoyed by a 15 second delay --- but there are many contexts where a 15 second delay is nothing. It often takes a minute or more to start up a Cloud VM, for example, and if users aren't checking the system logs --- and most IOT application programmers won't be checking system logs, and 15 seconds to boot might not even be noticed during development for some devices. And even on a big x86 server, it can take 5+ minutes for it to boot (between BIOS and kernel probe time), so 15 seconds won't be noticed. Linus, I know you don't like the config approach, but the problem is there is not going to be any "one size fits all" solution, because Linux gets used in so many places. We can set up defaults so that for x86, we never block and just create a big fat warning, and cross our fingers and hope that's enough. But on other platforms, 15 seconds won't be the right number, and you might actually need something closer to two minutes before the delay will be noticed. And on some of these other platforms, the use of "best effort" randomness might be ***far*** more catastrophic from a security perspective than on an x86. This is why I really want the CONFIG option. I'm willing to believe that the x86 architecture will mostly be safe, so we could never ask for the option on some set of architectures (unless CONFIG_EXPERT is enabled). But there will be other architectures and use cases where "never blocking" and "return best effort randomness" is going to be unacceptable, and lead to massive security problems, that could be quite harmful. So for those architectures, I'd really like to make the CONFIG option be visible, and even default it to "block". For the embedded use case, we want it to be blatently obvious that
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 06:48:34PM -0700, Vito Caputo wrote:
> > A small note here, especially after I've just read the commit log of
> > 72dbcf721566 ('Revert ext4: "make __ext4_get_inode_loc plug"'), which
> > unfairly blames systemd there.
...
> > What blocked the system boot was GDM/gnome-session implicitly calling
> > getrandom() for the Xorg MIT cookie. This was shown in the strace log
> > below:
> >
> >https://lkml.kernel.org/r/20190910173243.GA3992@darwi-home-pc
Yes, that's correct, this isn't really systemd's fault. It's a
combination of GDM/gnome-session stupidly using MIT Magic Cookie at
*all* (it was a bad idea 30 years ago, and it's a bad idea in 2019),
GDM/gnome-session using getrandom(2) at all; it should have just stuck
with /dev/urandom, or heck just used random_r(3) since when we're
talking about MIT Magic Cookie, there's no real security *anyway*.
It's also a combination of the hardware used by this particular user,
the init scripts in use that were probably not generating enough read
requests compared to other distributions (ironically, distributions
and init systems that try the hardest to accelerate the boot make this
problem worse by reducing the entropy that can be harvested from I/O).
And then when we optimzied ext4 so it would be more efficient, that
tipped this particular user over the edge.
Linus might not have liked my proposal to disable the optimization if
the CRNG isn't optimized, but ultimately this problem *has* gotten
worse because we've optimized things more. So to the extent that
systemd has made systems boot faster, you could call that systemd's
"fault" --- just as Linus reverting ext4's performance optimization is
ssaying that it's ext4 "fault" because we had the temerity to try to
make the file system be more efficient, and hence, reduce entropy that
can be collected.
Ultimately, though, the person who touches this last is whose "fault"
it is. And the problem is because it really is a no-win situation
here. No matter *what* we do, it's going to either (a) make some
systems insecure, or (b) make some systems more likely hang while
booting. Whether you consider the risk of (a) or (b) to be worse is
ultimately going to cause you to say that people of the contrary
opinion are either "being reckless with system security", or
"incompetent at system design".
And really, it's all going to depend on how the Linux kernel is being
used. The fact that Linux is being used in IOT devices, mobile
handsets, desktops, servers running in VM's, user desktops, etc.,
means that there will be some situations where blocking is going to be
terrible, and some situations where a failure to provide system
security could result in risking someone's life, health, or mission
failure in some critical system.
That's why this discussion can easily get toxic. If you are only
focusing on one part of Linux market, then obviously *you* are the
only sane one, and everyone *else* who disagrees with you must be
incompetent. When, perhaps, they may simply be focusing on a
different part of the ecosystem where Linux is used.
> So did systemd-random-seed instead drain what little entropy there was
> before GDM started, increasing the likelihood a subsequent getrandom()
> call would block?
No. Getrandom(2) uses the new CRNG, which is either initialized, or
it's not. Once it's initialized, it won't block again ever.
- Ted
Re: Linux 5.3-rc8
On Mon, Sep 16, 2019 at 03:40:50AM +0200, Ahmed S. Darwish wrote:
> On Sun, Sep 15, 2019 at 09:29:55AM -0700, Linus Torvalds wrote:
> > On Sat, Sep 14, 2019 at 11:51 PM Lennart Poettering
> > wrote:
> > >
> > > Oh man. Just spend 5min to understand the situation, before claiming
> > > this was garbage or that was garbage. The code above does not block
> > > boot.
> >
> > Yes it does. You clearly didn't read the thread.
> >
> > > It blocks startup of services that explicit order themselves
> > > after the code above. There's only a few services that should do that,
> > > and the main system boots up just fine without waiting for this.
> >
> > That's a nice theory, but it doesn't actually match reality.
> >
> > There are clearly broken setups that use this for things that it
> > really shouldn't be used for. Asking for true randomness at boot
> > before there is any indication that randomness exists, and then just
> > blocking with no further action that could actually _generate_ said
> > randomness.
> >
> > If your description was true that the system would come up and be
> > usable while the blocked thread is waiting for that to happen, things
> > would be fine.
> >
>
> A small note here, especially after I've just read the commit log of
> 72dbcf721566 ('Revert ext4: "make __ext4_get_inode_loc plug"'), which
> unfairly blames systemd there.
>
> Yes, the systemd-random-seed(8) process blocks, but this is an
> isolated process, and it's only there as a synchronization point and
> to load/restore random seeds from disk across reboots.
>
> The wisdom of having a sysnchronization service ("before/after urandom
> CRNG is inited") can be debated. That service though, and systemd in
> general, did _not_ block the overall system boot.
>
> What blocked the system boot was GDM/gnome-session implicitly calling
> getrandom() for the Xorg MIT cookie. This was shown in the strace log
> below:
>
>https://lkml.kernel.org/r/20190910173243.GA3992@darwi-home-pc
>
So did systemd-random-seed instead drain what little entropy there was
before GDM started, increasing the likelihood a subsequent getrandom()
call would block?
Regards,
Vito Caputo
Re: Linux 5.3-rc8
On Sun, Sep 15, 2019 at 09:29:55AM -0700, Linus Torvalds wrote:
> On Sat, Sep 14, 2019 at 11:51 PM Lennart Poettering
> wrote:
> >
> > Oh man. Just spend 5min to understand the situation, before claiming
> > this was garbage or that was garbage. The code above does not block
> > boot.
>
> Yes it does. You clearly didn't read the thread.
>
> > It blocks startup of services that explicit order themselves
> > after the code above. There's only a few services that should do that,
> > and the main system boots up just fine without waiting for this.
>
> That's a nice theory, but it doesn't actually match reality.
>
> There are clearly broken setups that use this for things that it
> really shouldn't be used for. Asking for true randomness at boot
> before there is any indication that randomness exists, and then just
> blocking with no further action that could actually _generate_ said
> randomness.
>
> If your description was true that the system would come up and be
> usable while the blocked thread is waiting for that to happen, things
> would be fine.
>
A small note here, especially after I've just read the commit log of
72dbcf721566 ('Revert ext4: "make __ext4_get_inode_loc plug"'), which
unfairly blames systemd there.
Yes, the systemd-random-seed(8) process blocks, but this is an
isolated process, and it's only there as a synchronization point and
to load/restore random seeds from disk across reboots.
The wisdom of having a sysnchronization service ("before/after urandom
CRNG is inited") can be debated. That service though, and systemd in
general, did _not_ block the overall system boot.
What blocked the system boot was GDM/gnome-session implicitly calling
getrandom() for the Xorg MIT cookie. This was shown in the strace log
below:
https://lkml.kernel.org/r/20190910173243.GA3992@darwi-home-pc
thanks,
--
darwi
http://darwish.chasingpointers.com
Re: Linux 5.3-rc8
On Sat, Sep 14, 2019 at 11:56 PM Lennart Poettering wrote: > > I am not expecting the kernel to guarantee entropy. I just expecting > the kernel to not give me garbage knowingly. It's OK if it gives me > garbage unknowingly, but I have a problem if it gives me trash all the > time. So realistically, we never actually give you *garbage*. It's just that we try very hard to actually give you some entropy guarantees, and that we can't always do in a timely manner - particularly if you don't help. But on a PC, we can _almost_ guarantee entropy. Even with a golden image, we do mix in: - timestamp counter on every device interrupt (but "device interrupt" doesn't include things like the local CPU timer, so it really needs device activity) - random boot and BIOS memory (dmi tables, the EFI RNG entry, etc) - various device state (things like MAC addresses when registering network devices, USB device numbers, etc) - and obviously any CPU rdrand data and note the "mix in" part - it's all designed so that you don't trust any of this for randomness on its own, but very much hopefully it means that almost *any* differences in boot environment will add a fair amount of unpredictable behavior. But also note the "on a PC" part. Also note that as far as the kernel is concerned, none of the above counts as "entropy" for us, except to a very small degree the device interrupt timing thing. But you need hundreds of interrupts for that to be considered really sufficient. And that's why things broke. It turns out that making ext4 be more efficient at boot caused fewer disk interrupts, and now we weren't convinced we had sufficient entropy. And the systemd boot thing just *stopped* waiting for entropy to magically appear, which is never will if the machine is idle and not doing anything. So do we give you "garbage" in getrandom()? We try really really hard not to, but it's exactly the "can we _guarantee_ that it has entropy" that ends up being the problem. So if some silly early boot process comes along, and asks for "true randomness", and just blocks for it without doing anything else, that's broken from a kernel perspective. In practice, the only situation we have had really big problems with not giving "garbage" isn't actually the "golden distro image" case you talk about. It's the "embedded device golden _system_ image" case, where the image isn't just the distribution, but the full bootloader state. Some cheap embedded MIPS CPU without even a timestamp counter, with identical flash contents for millions of devices, and doing a "on first boot, generate a long-term key" without even connecting to the network first. That's the thing Ted was pointing at: https://factorable.net/weakkeys12.extended.pdf so yes, it can be "garbage", but it can be garbage only if you really really do things entirely wrong. But basically, you should never *ever* try to generate some long-lived key and then just wait for it without doing anything else. The "without doing anything else" is key here. But every time we've had a blocking interface, that's exactly what somebody has done. Which is why I consider that long blocking thing to be completely unacceptable. There is no reason to believe that the wait will ever end, partly exactly because we don't consider timer interrupts to add any timer randomness. So if you are just waiting, nothing necessarily ever happen. Linus
