Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Stephen J. Turnbull]

Donald Stufft writes:

 > I guess one question would be, what does the secrets module do if
 > it’s on a Linux that is too old to have getrandom(0), off the top
 > of my head I can think of:
 > 
 > * Silently fall back to reading os.urandom and hope that it’s been
 >   seeded.
 > * Fall back to os.urandom and hope that it’s been seeded and add a
 >   SecurityWarning or something like it to mention that it’s
 >   falling back to os.urandom and it may be getting predictable
 >   random from /dev/urandom.
 > * Hard fail because it can’t guarantee secure cryptographic
 >   random.

I'm going to hide behind the Linux manpage (which actually suggests
saving the data in a file to speed initialization at boot) in
mentioning this:

* if random_initialized_timestamp_pre_boot():
  r = open("/dev/random", "rb")
  u = open("/dev/urandom", "wb")
  u.write(r.read(enough_bytes))
  set_random_initialized_timestamp()
  # in theory, secrets can now use os.urandom

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[David Mertz]

Yes 'secrets' is one-liners. However, it might grow a few more lines around
the blocking in getrandom() on Linux. But still, not more than a few.

But the reason it should be on PyPI is so that programs can have a uniform
API across various Python versions. There's no real reason that someone
stick on Python 2.7 or 3.3 shouldn't be able to include the future-style:

import secrets
Answer = secrets.token_bytes(42)
On Jun 16, 2016 4:53 PM, "Nick Coghlan"  wrote:

> On 16 June 2016 at 13:09, Barry Warsaw  wrote:
> > On Jun 16, 2016, at 01:01 PM, David Mertz wrote:
> >
> >>It seems to me that backporting 'secrets' and putting it on Warehouse
> would
> >>be a lot more productive than complaining about 3.5.2 reverting to
> (almost)
> >>the behavior of 2.3-3.4.
> >
> > Very wise suggestion indeed.  We have all kinds of stdlib modules
> backported
> > and released as third party packages.  Why not secrets too?  If such
> were on
> > PyPI, I'd happily package it up for the Debian ecosystem.  Problem solved
> > .
>
> The secrets module is just a collection of one liners pulling together
> other stdlib components that have been around for years - the main
> problem it aims to address is one of discoverability (rather than one
> of code complexity), while also eliminating the "simulation is in the
> standard library, secrecy requires a third party module" discrepancy
> in the long term.
>
> Once you're aware the problem exists, the easiest way to use it in a
> version independent manner is to just copy the relevant snippet into
> your own project's utility library - adding an entire new dependency
> to your project just for those utility functions would be overkill.
>
> If you *do* add a dependency, you'd typically be better off with
> something more comprehensive and tailored to the particular problem
> domain you're dealing with, like passlib or cryptography or
> itsdangerous.
>
> Cheers,
> Nick.
>
> P.S. Having the secrets module available on PyPI wouldn't *hurt*, I
> just don't think it would help much.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 16 June 2016 at 13:09, Barry Warsaw  wrote:
> On Jun 16, 2016, at 01:01 PM, David Mertz wrote:
>
>>It seems to me that backporting 'secrets' and putting it on Warehouse would
>>be a lot more productive than complaining about 3.5.2 reverting to (almost)
>>the behavior of 2.3-3.4.
>
> Very wise suggestion indeed.  We have all kinds of stdlib modules backported
> and released as third party packages.  Why not secrets too?  If such were on
> PyPI, I'd happily package it up for the Debian ecosystem.  Problem solved
> .

The secrets module is just a collection of one liners pulling together
other stdlib components that have been around for years - the main
problem it aims to address is one of discoverability (rather than one
of code complexity), while also eliminating the "simulation is in the
standard library, secrecy requires a third party module" discrepancy
in the long term.

Once you're aware the problem exists, the easiest way to use it in a
version independent manner is to just copy the relevant snippet into
your own project's utility library - adding an entire new dependency
to your project just for those utility functions would be overkill.

If you *do* add a dependency, you'd typically be better off with
something more comprehensive and tailored to the particular problem
domain you're dealing with, like passlib or cryptography or
itsdangerous.

Cheers,
Nick.

P.S. Having the secrets module available on PyPI wouldn't *hurt*, I
just don't think it would help much.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 01:01 PM, David Mertz wrote:

>It seems to me that backporting 'secrets' and putting it on Warehouse would
>be a lot more productive than complaining about 3.5.2 reverting to (almost)
>the behavior of 2.3-3.4.

Very wise suggestion indeed.  We have all kinds of stdlib modules backported
and released as third party packages.  Why not secrets too?  If such were on
PyPI, I'd happily package it up for the Debian ecosystem.  Problem solved
.

But I'm *really* going to try to disengage from this discussion until Nick's
PEP is posted.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Paul Moore]

On 16 June 2016 at 18:03, Nick Coghlan  wrote:
> On 16 June 2016 at 09:39, Paul Moore  wrote:
>> I'm willing to accept the view of the security experts that there's a
>> problem here. But without a clear explanation of the problem, how can
>> a non-specialist like myself have an opinion? (And I hope the security
>> POV isn't "you don't need an opinion, just do as we say").
>
> If you're not writing Linux (and presumably *BSD) scripts and
> applications that run during system initialisation or on embedded ARM
> hardware with no good sources of randomness, then there's zero chance
> of any change made in relation to this affecting you (Windows and Mac
> OS X are completely immune, since they don't allow Python scripts to
> run early enough in the boot sequence for there to ever be a problem).

Understood. I could quite happily ignore this thread for all the
impact it will have on me. However, I've seen enough of these debates
(and witnessed the frustration of the security advocates) that I want
to try to understand the issues better - as much as anything so that I
don't end up adding uninformed opposition to these threads (in my day
job, unfortunately, security is generally the excuse for all sorts of
counter-productive rules, and never offers any practical benefits that
I am aware of, so I'm predisposed to rejecting arguments based on
security - that background isn't accurate in this environment and I'm
actively trying to counter it).

> The only question at hand is what CPython should do in the case where
> the operating system *does* let Python scripts run before the system
> random number generator is ready, and the application calls a security
> sensitive API that relies on that RNG:
>
> - throw BlockingIOError (so the script developer knows they have a
> potential problem to fix)
> - block (so the script developer has a system hang to debug)
> - return low quality random data (so the script developer doesn't even
> know they have a potential problem)
>
> The last option is the status quo, and has a remarkable number of
> vocal defenders.

Understood. It seems to me that there are two arguments here -
backward compatibility (which is always a pressure, but sometimes
applied too vigourously and not always consistently) and "we've always
done it that way" (aka "people will have to consider what happens when
they run under 3.4 anyway, so how will changing help?"). Jusging
backward compatibility is always a matter of trade-offs, hence my
interest in the actual benefits.

> The second option is what we changed the behaviour to in 3.5 as a side
> effect of switching to a syscall to save a file descriptor (and *also*
> inadvertently made a gating requirement for CPython starting at all,
> without which I'd be very surprised if anyone actually noticed the
> potentially blocking behaviour in os.urandom itself)

OK, so (given that the issue of CPython starting at all was an
accidental, and now corrected, side effect) why is this so bad? Maybe
not in a minor release, but at least for 3.6? How come this has caused
such a fuss? I genuinely don't understand why people see blocking as
such an issue (and as far as I can tell, Ted Tso seems to agree). The
one case where this had an impact was a quickly fixed bug - so as far
as I can tell, the risk of problems caused by blocking is purely
hypothetical.

> The first option is the one I'm currently writing a PEP for, since it
> makes the longstanding advice to use os.urandom() as the low level
> random data API for security sensitive operations unequivocally
> correct (as it will either do the right thing, or throw an exception
> which the developer can handle as appropriate for their particular
> application)

In my code, I typically prefer Python to make detailed decisions for
me (e.g. requests follows redirects by default, it doesn't expect me
to do so manually). Now certainly this is a low-level interface so the
rules are different, but I don't see why blocking by default isn't
"unequivocally correct" in the same way that it is on other platforms,
rather than raising an exception and requiring the developer to do the
wait manually. (What else would they do - fall back to insecure data?
I thought the point here was that that's the wrong thing to do?)
Having a blocking default with a non-blocking version seems just as
arguable, and has the advantage that naive users (I don't even know if
we're allowing for naive users here) won't get an unexpected exception
and handle it badly because they don't know what to do (a sadly common
practice in my experience).

OK. Guido has pronounced, you're writing a PEP. None of this debate is
really constructive any more. But I still don't understand the
trade-offs, which frustrates me. Surely security isn't so hard that it
can't be explained in a way that an interested layman like myself can
follow? :-(

Paul
___
Python-Dev mailing list

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[A.M. Kuchling]

On Thu, Jun 16, 2016 at 10:26:22AM -0700, Nick Coghlan wrote:
> meta-guidance. However, there are multiple levels of improvement being
> pursued here, since developer ignorance of security concerns and
> problematic defaults at the language level is a chronic problem rather
> than an acute one (and one that affects all languages, not just
> Python).

For a while Christian Heimes has speculated on Twitter about writing a
Secure Programming HOWTO.  At the last language summit in Montreal, I
told him I'd be happy to do the actual writing and editing if given a
detailed outline.  (I miss not having an ongoing writing project since
ceasing to write the "What's New", but have no ideas for anything to
write about.)

That offer is still open, if Christian or someone else wants to
produce an outline.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 16 2016, Nick Coghlan  wrote:
> On 16 June 2016 at 09:39, Paul Moore  wrote:
>> I'm willing to accept the view of the security experts that there's a
>> problem here. But without a clear explanation of the problem, how can
>> a non-specialist like myself have an opinion? (And I hope the security
>> POV isn't "you don't need an opinion, just do as we say").
>
> If you're not writing Linux (and presumably *BSD) scripts and
> applications that run during system initialisation or on embedded ARM
> hardware with no good sources of randomness, then there's zero chance
> of any change made in relation to this affecting you (Windows and Mac
> OS X are completely immune, since they don't allow Python scripts to
> run early enough in the boot sequence for there to ever be a problem).
>
> The only question at hand is what CPython should do in the case where
> the operating system *does* let Python scripts run before the system
> random number generator is ready, and the application calls a security
> sensitive API that relies on that RNG:
>
> - throw BlockingIOError (so the script developer knows they have a
> potential problem to fix)
> - block (so the script developer has a system hang to debug)
> - return low quality random data (so the script developer doesn't even
> know they have a potential problem)
>
> The last option is the status quo, and has a remarkable number of
> vocal defenders.

*applaud*


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 16 June 2016 at 10:40, Nathaniel Smith  wrote:
> On Jun 16, 2016 10:01 AM, "David Mertz"  wrote:
>> Python 3.6 is introducing a NEW MODULE, with new APIs.  The 'secrets'
>> module is the very first time that Python has ever really explicitly
>> addressed cryptography in the standard library.
>
> This is completely, objectively untrue. If you look up os.urandom in the
> official manual for the standard library, then it have always stated
> explicitly, as the very first line, that os.urandom returns "a string of n
> random bytes suitable for cryptographic use." This is *exactly* the same
> explicit guarantee that the secrets module makes. The motivation for adding
> the secrets module was to make this functionality easier to find and more
> convenient to use (e.g. by providing convenience functions for getting
> random strings of ASCII characters), not to suddenly start addressing
> cryptographic concerns for the first time.

An analogy that occurred to me that may help some folks: secrets is a
higher level API around os.urandom and some other standard library
features (like base64 and binascii.hexlify) in the same way that
shutil and pathlib are higher level APIs that aggregate other os
module functions with other parts of the standard library.

The existence of those higher level APIs doesn't make the lower level
building blocks redundant.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Jun 16, 2016 10:01 AM, "David Mertz"  wrote:
> Python 3.6 is introducing a NEW MODULE, with new APIs.  The 'secrets'
module is the very first time that Python has ever really explicitly
addressed cryptography in the standard library.

This is completely, objectively untrue. If you look up os.urandom in the
official manual for the standard library, then it have always stated
explicitly, as the very first line, that os.urandom returns "a string of n
random bytes suitable for cryptographic use." This is *exactly* the same
explicit guarantee that the secrets module makes. The motivation for adding
the secrets module was to make this functionality easier to find and more
convenient to use (e.g. by providing convenience functions for getting
random strings of ASCII characters), not to suddenly start addressing
cryptographic concerns for the first time.

(Will try to address other more nuanced points later.)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 16 June 2016 at 10:01, David Mertz  wrote:
> It seems to me that backporting 'secrets' and putting it on Warehouse would
> be a lot more productive than complaining about 3.5.2 reverting to (almost)
> the behavior of 2.3-3.4.

"Let Flask/Django/passlib/cryptography/whatever handle the problem
rather than rolling your own" is already the higher level
meta-guidance. However, there are multiple levels of improvement being
pursued here, since developer ignorance of security concerns and
problematic defaults at the language level is a chronic problem rather
than an acute one (and one that affects all languages, not just
Python).

In that context, the main benefit of the secrets module is as a
deterrent against people reaching for the reproducible simulation
focused random module to implement security sensitive operations. By
offering both secrets and random in the standard library, we help make
it clear that secrecy and simulation are *not the same problem*, even
though they both involve random numbers. Folks that learn Python 3.6
first and then later start supporting earlier versions are likely to
be more aware of the difference, and hence go looking for "What's the
equivalent of the secrets module on earlier Python versions?" (at
which point they can just copy whichever one-liner they actually need
into their particular application - just as not every 3 line function
needs to be a builtin, not every 3 line function needs to be a module
on PyPI)

The os.urandom proposal is aimed more at removing any remaining
equivocation from the longstanding "Use os.urandom() for security
sensitive operations in Python" advice - it's for the benefit of folks
that are *already* attempting to do the right thing given the tools
they have available. The sole source of that equivocation is that in
some cases, at least on Linux, and potentially on *BSD (although we
haven't seen a confirmed reproducer there), os.urandom() may return
results that are sufficiently predictable to be inappropriate for use
in security sensitive applications.

At the moment, determining whether or not you're risking exposure to
that problem requires that you know a whole lot about Linux (and *BSD,
where even we haven't been able to determine the level of exposure on
embedded systems), and also about how ``os.urandom()`` is implemented
on different platforms.

My proposal is that we do away with the requirement for all that
assumed knowledge and instead say "Are you using os.urandom(),
random.SystemRandom(), or an API in the secrets module? Are you using
Python 3.6+? Did it raise BlockingIOError? No? Then you're fine".

The vast majority of Python developers will thus be free to remain
entirely ignorant of these platform specific idiosyncracies, while
those that have a potential need to know will get an exception from
the interpreter that they can then feed into a search engine and get
pointed in the right direction.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 16 June 2016 at 09:39, Paul Moore  wrote:
> I'm willing to accept the view of the security experts that there's a
> problem here. But without a clear explanation of the problem, how can
> a non-specialist like myself have an opinion? (And I hope the security
> POV isn't "you don't need an opinion, just do as we say").

If you're not writing Linux (and presumably *BSD) scripts and
applications that run during system initialisation or on embedded ARM
hardware with no good sources of randomness, then there's zero chance
of any change made in relation to this affecting you (Windows and Mac
OS X are completely immune, since they don't allow Python scripts to
run early enough in the boot sequence for there to ever be a problem).

The only question at hand is what CPython should do in the case where
the operating system *does* let Python scripts run before the system
random number generator is ready, and the application calls a security
sensitive API that relies on that RNG:

- throw BlockingIOError (so the script developer knows they have a
potential problem to fix)
- block (so the script developer has a system hang to debug)
- return low quality random data (so the script developer doesn't even
know they have a potential problem)

The last option is the status quo, and has a remarkable number of
vocal defenders.

The second option is what we changed the behaviour to in 3.5 as a side
effect of switching to a syscall to save a file descriptor (and *also*
inadvertently made a gating requirement for CPython starting at all,
without which I'd be very surprised if anyone actually noticed the
potentially blocking behaviour in os.urandom itself)

The first option is the one I'm currently writing a PEP for, since it
makes the longstanding advice to use os.urandom() as the low level
random data API for security sensitive operations unequivocally
correct (as it will either do the right thing, or throw an exception
which the developer can handle as appropriate for their particular
application)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[David Mertz]

On Thu, Jun 16, 2016 at 11:58 AM, Nathaniel Smith  wrote:

> [...] no one else be able to predict what session cookie I sent [...] In
> python 2.3-3.5, the most correct way to write this code is to use
> os.urandom. The question in this thread is whether we should break that in
> 3.6, so that conscientious users are forced to switch existing code over to
> using the secrets module if they want to continue to get the most correct
> available behavior, or whether we should preserve that in 3.6, so that code
> like my hypothetical web app that was correct on 2.3-3.5 remains correct on
> 3.6
>
This is kinda silly.  Unless you specifically wrote your code for Python
3.5.1, and NOT for 2.3.x through 3.4.x, your code is NO WORSE in 3.5.2 than
it has been under all those prior versions.  The cases where the behavior
in everything other than 3.5.0-3.5.1 is suboptimal are *extremely limited*,
as you understand (things that run in Python very early in the boot
process, and only on recent versions of Linux, no other OS).  This does not
even remotely describe the web-server-with-cookies example that you outline.

Python 3.6 is introducing a NEW MODULE, with new APIs.  The 'secrets'
module is the very first time that Python has ever really explicitly
addressed cryptography in the standard library.  Yes, there have been
third-party modules and libraries, but any cryptographic application of
Python prior to 'secrets' is very much roll-your-own and
know-what-you-are-doing.

Yes, there has been a history of telling people to "use os.urandom()" on
StackOverflow and places like that.  That's about the best advice that was
available prior to 3.6.  Adding a new module and API is specifically
designed to allow for a better answer, otherwise there'd be no reason to
include it.  And that advice that's been on StackOverflow and wherever has
been subject to the narrow, edge-case flaw we've discussed here for at
least a decade without anyone noticing or caring.

It seems to me that backporting 'secrets' and putting it on Warehouse would
be a lot more productive than complaining about 3.5.2 reverting to (almost)
the behavior of 2.3-3.4.



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Paul Moore]

On 16 June 2016 at 16:58, Nathaniel Smith  wrote:
> The word "cryptographic" here is a bit of a red herring. The guarantee that
> a CSPRNG makes is that the output should be *unguessable by third parties*.
> There are plenty of times when this is what you need even when you aren't
> using actual cryptography. For example, when someone logs into a web app, I
> may want to send back a session cookie so that I can recognize this person
> later without making then reauthenticate all the time. For this to work
> securely, it's extremely important that no one else be able to predict what
> session cookie I sent, because if you can guess the cookie then you can
> impersonate the user.
>
> In python 2.3-3.5, the most correct way to write this code is to use
> os.urandom. The question in this thread is whether we should break that in
> 3.6, so that conscientious users are forced to switch existing code over to
> using the secrets module if they want to continue to get the most correct
> available behavior, or whether we should preserve that in 3.6, so that code
> like my hypothetical web app that was correct on 2.3-3.5 remains correct on
> 3.6 (with the secrets module being a more friendly wrapper that we recommend
> for new code, but with no urgency about porting existing code to it).

While your example is understandable and clear, it's also a bit of a
red herring as well. Nobody's setting up a web session cookie during
the first moments of Linux boot (are they?), so os.urandom is
perfectly OK in all cases here. We have a new API in 3.6 that might
better express the *intent* of generating a secret token, but
(cryptographic) correctness is the same either way for this example.

As someone who isn't experienced in crypto, I genuinely don't have the
slightest idea of what sort of program we're talking about that is
written in Python, runs in the early stages of OS startup, and needs
crypto-strength random numbers. So I can't reason about whether the
proposed solutions are sensible. Would such programs be used in a
variety of environments with different Python versions? Would the
developers be non-specialists? Which of the mistakes being made that
result in a vulnerability is the easiest to solve (move the code to
run later, modify the Python code, require a fixed version of Python)?
How severe is the security hole compared to others (for example, users
with weak passwords)? What attacks are possible, and what damage could
be done? (I know that in principle, any security hole needs to be
plugged, but I work in an environment where production services with a
password of "password" exist, and applying system security patches is
treated as a "think about it when things are quiet" activity - so
forgive me if I don't immediately understand why obscure
vulnerabilities are important).

I'm willing to accept the view of the security experts that there's a
problem here. But without a clear explanation of the problem, how can
a non-specialist like myself have an opinion? (And I hope the security
POV isn't "you don't need an opinion, just do as we say").

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Jun 16, 2016 1:23 AM, "Stefan Krah"  wrote:
>
> Nathaniel Smith  pobox.com> writes:
> > In practice, your proposal means that ~all existing code that uses
> > os.urandom becomes incorrect and should be switched to either secrets
> > or random. This is *far* more churn for end-users than Nick's
> > proposal.
>
> This should only concern code that a) was specifically written for
> 3.5.0/3.5.1 and b) implements a serious cryptographic application
> in Python.
>
> I think b) is not a good idea anyway due to timing and side channel
> attacks and the lack of secure wiping of memory. Such applications
> should be written in C, where one does not have to predict the
> behavior of multiple layers of abstractions.

This is completely unhelpful. Firstly because it's an argument that
os.urandom and the secrets module shouldn't exist, which doesn't tell is
much about what their behavior should be given that they do exist, and
secondly because it fundamentally misunderstands why they exist.

The word "cryptographic" here is a bit of a red herring. The guarantee that
a CSPRNG makes is that the output should be *unguessable by third parties*.
There are plenty of times when this is what you need even when you aren't
using actual cryptography. For example, when someone logs into a web app, I
may want to send back a session cookie so that I can recognize this person
later without making then reauthenticate all the time. For this to work
securely, it's extremely important that no one else be able to predict what
session cookie I sent, because if you can guess the cookie then you can
impersonate the user.

In python 2.3-3.5, the most correct way to write this code is to use
os.urandom. The question in this thread is whether we should break that in
3.6, so that conscientious users are forced to switch existing code over to
using the secrets module if they want to continue to get the most correct
available behavior, or whether we should preserve that in 3.6, so that code
like my hypothetical web app that was correct on 2.3-3.5 remains correct on
3.6 (with the secrets module being a more friendly wrapper that we
recommend for new code, but with no urgency about porting existing code to
it).

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ionel Cristian Mărieș via Python-Dev]

On Thu, Jun 16, 2016 at 1:04 PM, Donald Stufft  wrote:

> In my opinion, this is a usability issue as well. You have a ton of third
> party documentation and effort around “just use urandom” for Cryptographic
> random which is generally the right (and best!) answer except for this one
> little niggle on a Linux platform where /dev/urandom *may* produce
> predictable bytes (but usually doesn’t).


​Why not consider opt-out behavior with environment variables​? Eg: people
that don't care about crypto mumbojumbo and want fast interpreter startup
could just use a PYTHONWEAKURANDOM=y or PYTHONFASTURANDOM=y.

That ways there's no need to change api of os.urandom() and users have a
clear and easy path to get old behavior.




Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Thu, Jun 16, 2016, at 10:04, Barry Warsaw wrote:
> On Jun 16, 2016, at 09:51 AM, Random832 wrote:
> 
> >On Thu, Jun 16, 2016, at 04:03, Barry Warsaw wrote:
> >> *If* we can guarantee that os.urandom() will never block or raise an
> >> exception when only poor entropy is available, then it may be indeed
> >> indistinguishably backward compatible for most if not all cases.  
> >
> >Why can't we exclude cases when only poor entropy is available from
> >"most if not all cases"?
>
> Because if it blocks or raises a new exception on poor entropy it's an
> API break.

Yes, but in only very rare cases. Which as I *just said* makes it
backwards compatible for "most" cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 09:51 AM, Random832 wrote:

>On Thu, Jun 16, 2016, at 04:03, Barry Warsaw wrote:
>> *If* we can guarantee that os.urandom() will never block or raise an
>> exception when only poor entropy is available, then it may be indeed
>> indistinguishably backward compatible for most if not all cases.  
>
>Why can't we exclude cases when only poor entropy is available from
>"most if not all cases"?

Because if it blocks or raises a new exception on poor entropy it's an API
break.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Thu, Jun 16, 2016, at 04:03, Barry Warsaw wrote:
> *If* we can guarantee that os.urandom() will never block or raise an
> exception when only poor entropy is available, then it may be indeed
> indistinguishably backward compatible for most if not all cases.

Why can't we exclude cases when only poor entropy is available from
"most if not all cases"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Stefan Krah]

Cory Benfield  lukasa.co.uk> writes:
> python-dev cannot wash its hands of the security decision here. As I’ve
said many times, I’m pleased to
> see the decision makers have not done that: while I don’t agree with their
decision, I totally respect
> that it was theirs to make, and they made it with all of the facts.

I think the sysadmin's responsibility still plays a major role here.
If a Linux system crucially relies on the quality of /dev/urandom, it
should be possible to insert a small C program (call it ensure_random)
into the boot sequence that does *exactly* what Python did in the
bug report: block until entropy is available.

Well, it *was* possible with SysVinit ... :)


Python is not the only application that needs a secure /dev/urandom.



Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 16 Jun 2016, at 09:19, Stefan Krah  wrote:
> 
> This should only concern code that a) was specifically written for
> 3.5.0/3.5.1 and b) implements a serious cryptographic application
> in Python.
> 
> I think b) is not a good idea anyway due to timing and side channel
> attacks and the lack of secure wiping of memory. Such applications
> should be written in C, where one does not have to predict the
> behavior of multiple layers of abstractions.

No, it concerns code that generates its random numbers from Python. For 
example, you may want to use AES GCM to encrypt a file at rest. AES GCM 
requires the use of an nonce, and has only one rule about this nonce: you MUST 
NOT, under any circumstances, re-use an nonce/key combination. If you do, AES 
GCM fails catastrophically (I cannot emphasise this enough, re-using a 
nonce/key combination in AES GCM totally destroys all the properties the 
algorithm provides)[0].

You can use a C implementation of all of the AES logic, including offload to 
your x86 CPU with its fancy AES GCM instruction set. However, you *need* to 
provide an nonce: AES GCM can’t magically guess what it is, and it needs to be 
communicated in some way for the decryption[1]. In situations where you do not 
have an easily available nonce (you do have it for TLS, for example), you will 
need to provide one, and the logical and obvious thing to do is to use a random 
number. Your Python application needs to obtain that random number, and the 
safest way to do it is via os.urandom().

This is the problem with this argument: we cannot wave our hands and say 
“os.urandom can be as unsafe as we want because crypto code must not be written 
in Python”. Even if we never implement an algorithm in Python (and I agree with 
you that crypto primitives in general should not be implemented in Python for 
the exact reasons you suggest), most algorithms require the ability to be 
provided with good random numbers by their callers. As long as crypto 
algorithms require good nonces, Python needs access to a secure CSPRNG. Kernel 
CSPRNGs are *strongly* favoured for many reasons that I won’t go into here, so 
os.urandom is our winner.

python-dev cannot wash its hands of the security decision here. As I’ve said 
many times, I’m pleased to see the decision makers have not done that: while I 
don’t agree with their decision, I totally respect that it was theirs to make, 
and they made it with all of the facts.

Cory


[0]: Someone will *inevitably* point out that other algorithms resist nonce 
misuse somewhat better than this. While that’s true, it’s a) not relevant, 
because some standards require use of the non-NMR algorithms, and b) unhelpful, 
because even if we could switch, we’d need access to the better primitives, 
which we don’t have.

[1]: Again, to head off some questions at the pass: the reason nonces are 
usually provided by the user of the algorithm is that sometimes they’re 
generated semi-deterministically. For example, TLS generates a unique key for 
each session (again, requiring randomness, but that’s neither here nor there), 
and so TLS can use deterministic *but non-repeated* nonces, which in practice 
it derives from record numbers. Because you have two options (re-use keys with 
random nonces, or random keys with deterministic nonces), a generic algorithm 
implementation does not constrain your choice of nonce.



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 16, 2016, at 4:46 AM, Barry Warsaw  wrote:
> 
> We can educate them through documentation, but I don't think it's appropriate
> to retrofit existing APIs to different behavior based on those faulty
> assumptions, because that has other negative effects, such as breaking the
> promises we make to experienced and knowledgeable developers.

You can’t document your way out of a usability problem, in the same way that 
while it was true that urllib was *documented* to not verify certificates by 
default, that didn’t matter because a large set of users used it like it did 
anyways.

In my opinion, this is a usability issue as well. You have a ton of third party 
documentation and effort around “just use urandom” for Cryptographic random 
which is generally the right (and best!) answer except for this one little 
niggle on a Linux platform where /dev/urandom *may* produce predictable bytes 
(but usually doesn’t). That documentation typically doesn’t go into telling 
people this small niggle because prior to getrandom(0) there wasn’t much they 
could do about it except use /dev/random which is bad in every other situation 
but early boot cryptographic keys.

Regardless of what we document it as, people are going to use os.urandom for 
cryptographic purposes because for everyone who doesn’t keep up on exactly what 
modules are being added to Python who has any idea about cryptography at all is 
going to look for a Python interface to urandom. That doesn’t even begin to 
touch the thousands upon thousands of uses that already exist in the wild that 
are assuming that os.urandom will always give them cryptographic random, who 
now *need* to write this as:

try:
from secrets import token_bytes
except ImportError:
from os import urandom as token_bytes

In order to get the best cryptographic random available to them on their 
system, which assumes they’re even going to notice at all that there’s a new 
secrets model, and requires each and every use of os.urandom to change.

Honestly, I think that the first sentence in the documentation should most 
obviously be the most pertinent one, and the first sentence here is "Return a 
string of n random bytes suitable for cryptographic use.”. The bit about how 
the exact quality depends on the OS and documenting what device it uses is, to 
my eyes, obviously a hedge to say that “Hey, if this gives you bad random it’s 
your OSs fault not ours, we can’t produce good random where your OS can’t give 
us some” and to give people a suggestion of where to look to determine if 
they’re going to get good random or not.

I do not think “uses /dev/urandom” is, or should be considered a core part of 
this API, it already doesn’t use /dev/urandom on Windows where it doesn’t exist 
nor does it use /dev/urandom in 3.5+ if it can help it. Using getrandom(0) or 
using getrandom(GRND_NONBLOCK) and raising an exception on EAGAIN is still 
accessing the urandom CSPRNG with the same general runtime characteristics of 
/dev/urandom outside of cases where it’s not safe to actually use /dev/urandom.

Frankly, I think it’s a disservice to Python developers to leave in this 
footgun.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 01:40 AM, Larry Hastings wrote:

>As Robert Collins points out, this does change the behavior ever-so-slightly
>from 3.4;

Ah yes, I misunderstood Robert's point.

>if urandom is initialized, and the kernel has the getrandom system call,
>getrandom() will give us the bytes we asked for and we won't open and read
>from /dev/urandom.  In this state os.urandom() behaves ever-so-slightly
>differently:
>
>  * os.urandom() will now work in chroot environments where /dev/urandom
>doesn't exist.
>  * If Python runs in a chroot environment with a fake /dev/urandom,
>we'll ignore that and use the kernel's urandom device.
>  * If the sysadmin changed what the systemwide /dev/urandom points to,
>we'll ignore that and use the kernel's urandom device.
>
>But os.urandom() is documented as calling getrandom() when available in
>3.5... though doesn't detail how it calls it or what it uses the result for.
>Anyway, I feel these differences were minor, and covered by the documented
>change in 3.5, so I thought it was reasonable and un-broken.
>
>If this isn't backwards-compatible enough to suit you, please speak up now!

It does seem like a narrow corner case, which of course means *someone* will
be affected by it .

I'll leave it up to you, though it should at least be clearly documented.
Let's hope the googles will also help our hypothetical future head-scratcher.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 12:36 AM, Nathaniel Smith wrote:

>Basically this is a question of whether we should make an (unlikely) error
>totally invisible to the user, and "errors should never pass silently" is
>right there in the Zen of Python :-).

I'd phrase it differently though.  To me, it comes down to hand-holding our
users who for whatever reason, don't use the appropriate APIs for what they're
trying to accomplish.

We can educate them through documentation, but I don't think it's appropriate
to retrofit existing APIs to different behavior based on those faulty
assumptions, because that has other negative effects, such as breaking the
promises we make to experienced and knowledgeable developers.

To me, the better policy is to admit our mistake in 3.5.0 and 3.5.1, restore
pre-existing behavior, accurately document the trade-offs, and provide a
clear, better upgrade path for our users.  We've done this beautifully and
effectively via the secrets module in Python 3.6.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/16/2016 01:03 AM, Barry Warsaw wrote:

*If* we can guarantee that os.urandom() will never block or raise an exception
when only poor entropy is available, then it may be indeed indistinguishably
backward compatible for most if not all cases.


I stepped through the code that shipped in 3.5.2rc1.  It only ever calls 
getrandom() with the GRND_NONBLOCK flag.  If getrandom() returns -1 and 
errno is EAGAIN it falls back to /dev/urandom--I actually simulated this 
condition in gdb and watched it open /dev/urandom.  I didn't see any 
code for raising an exception or blocking when only poor entropy is 
available.


As Robert Collins points out, this does change the behavior 
ever-so-slightly from 3.4; if urandom is initialized, and the kernel has 
the getrandom system call, getrandom() will give us the bytes we asked 
for and we won't open and read from /dev/urandom.  In this state 
os.urandom() behaves ever-so-slightly differently:


 * os.urandom() will now work in chroot environments where /dev/urandom
   doesn't exist.
 * If Python runs in a chroot environment with a fake /dev/urandom,
   we'll ignore that and use the kernel's urandom device.
 * If the sysadmin changed what the systemwide /dev/urandom points to,
   we'll ignore that and use the kernel's urandom device.

But os.urandom() is documented as calling getrandom() when available in 
3.5... though doesn't detail how it calls it or what it uses the result 
for.  Anyway, I feel these differences were minor, and covered by the 
documented change in 3.5, so I thought it was reasonable and un-broken.


If this isn't backwards-compatible enough to suit you, please speak up now!


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 12:53 AM, Nathaniel Smith wrote:

>> We have introduced churn.  Predicting a future SO question such as "Can
>> os.urandom() block on Linux?" the answer is "No in Python 3.4 and earlier,
>> yes possibly in Python 3.5.0 and 3.5.1, no in Python 3.5.2 and the rest of
>> the 3.5.x series, and yes possibly in Python 3.6 and beyond".
>
>It also depends on the kernel version, since it will never block on
>old kernels that are missing getrandom(), but it might block on future
>kernels if Linux's /dev/urandom ever becomes blocking. (Ted's said
>that this is not going to happen now, but the only reason it isn't was
>that he tried to make the change and it broke some distros that are
>still in use -- so it seems entirely possible that it will happen a
>few years from now.)

Right; I noticed this and had it in my copious notes for my follow up but
forgot to mention it.  Thanks!

>This is not an accurate docstring, though. The more accurate docstring
>for your proposed behavior would be:

[...]

>You should never use this function. If you need unguessable random
>bytes, then the 'secrets' module is always a strictly better choice --
>unlike this function, it always uses the best available source of
>cryptographic randomness, even on Linux. Alternatively, if you need
>random bytes but it doesn't matter whether other people can guess
>them, then the 'random' module is always a strictly better choice --
>it will be faster, as well as providing useful features like
>deterministic seeding.

Note that I was talking about 3.5.x, where we don't have the secrets module.

I'd quibble about the admonition about never using the function.  It *can* be
useful if the trade-offs are appropriate for your application (e.g. "almost
always random enough, but maybe not though at least you won't block and you'll
get back something").  Getting the words right is useful, but I agree that we
should be strongly recommending crypto applications use the secrets module in
Python 3.6.

>In practice, your proposal means that ~all existing code that uses os.urandom
>becomes incorrect and should be switched to either secrets or random. This is
>*far* more churn for end-users than Nick's proposal.

I disagree.  We have a clear upgrade path for end-users.  If you're using
os.urandom() in pre-3.5 and understand what you're getting (or not getting as
the case may be), you will continue to get or not get exactly the same bits in
3.5.x (where x >= 2).  No changes to your code are necessary.  This is also
the case in 3.6 but there you can do much better by porting your code to the
new secrets module.  Go do that!

>...Anyway, since there's clearly going to be at least one PEP about this,
>maybe we should stop rehashing bits and pieces of the argument in these long
>threads that most people end up skipping and then rehashing again later?

Sure, I'll try. ;)

Cheers,
-Barry


pgppqMdOn150C.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 16, 2016, at 07:26 PM, Robert Collins wrote:

>Which is a contract change. Someone testing in E.g. a chroot could have a
>different device on /dev/urandom, and now they will need to intercept
>syscalls for the same effect. Personally I think this is fine, but assuming
>i see Barry's point correctly, it is indeed but the same as it was.

It's true there could be a different device on /dev/urandom, but by my reading
of the getrandom() manpage I think that *should* be transparent since

By default, getrandom() draws entropy from the /dev/urandom pool.  This
behavior can be changed via the flags argument.

and we don't pass the GRND_RANDOM flag to getrandom().

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Stefan Krah]

Nathaniel Smith  pobox.com> writes:
> In practice, your proposal means that ~all existing code that uses
> os.urandom becomes incorrect and should be switched to either secrets
> or random. This is *far* more churn for end-users than Nick's
> proposal.

This should only concern code that a) was specifically written for
3.5.0/3.5.1 and b) implements a serious cryptographic application
in Python.

I think b) is not a good idea anyway due to timing and side channel
attacks and the lack of secure wiping of memory. Such applications
should be written in C, where one does not have to predict the
behavior of multiple layers of abstractions.


Stefan Krah

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 15, 2016, at 11:52 PM, Larry Hastings wrote:

>Well, 3.5.2 hasn't happened yet.  So if you see it as still being broken,
>please speak up now.

In discussion with other Ubuntu developers, several salient points were raised.

The documentation for os.urandom() in 3.5.2rc1 doesn't make sense:

On Linux, getrandom() syscall is used if available and the urandom entropy
pool is initialized (getrandom() does not block). On a Unix-like system
this will query /dev/urandom.

Perhaps better would be:

Where available, the getrandom() syscall is used (with the GRND_NONBLOCK
flag) if available and the urandom entropy pool is initialized.  When
getrandom() returns EAGAIN because of insufficient entropy, fallback to
reading from /dev/urandom.  When the getrandom() syscall is unavailable on
other Unix-like systems, this will query /dev/urandom.

It's actually a rather twisty maze of code to verify these claims, and I'm
nearly certain we don't have any tests to guarantee this is what actually
happens in those cases, so there are many caveats.

This means that an experienced developer can no longer just `man urandom` to
understand the unique operational behavior of os.urandom() on their platform,
but instead would be forced to actually read our code to find out what's
actually happening when/if things break.

It is unacceptable if any new exceptions are raised when insufficient entropy
is available.  Python 3.4 essentially promises that "if only crap entropy is
available, you'll get crap, but at least it won't block and no exceptions are
raised".  Proper backward compatibility requires the same in 3.5 and beyond.
Are we sure that's still the case?

Using the system call *may* be faster in the we-have-good-entropy-case, but it
will definitely be slower in the we-don't-have-good-entropy-case (because of
the fallback logic).  Maybe that doesn't matter in practice but it's worth
noting.

>Why do you call it only "semi-fixed"?  As far as I understand it, the
>semantics of os.urandom() in 3.5.2rc1 are indistinguishable from reading from
>/dev/urandom directly, except it may not need to use a file handle.

Semi-fixed because os.urandom() will still not be strictly backward compatible
between Python 3.5.2 and 3.4.

*If* we can guarantee that os.urandom() will never block or raise an exception
when only poor entropy is available, then it may be indeed indistinguishably
backward compatible for most if not all cases.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Wed, Jun 15, 2016 at 11:45 PM, Barry Warsaw  wrote:
> On Jun 15, 2016, at 01:01 PM, Nick Coghlan wrote:
>
>>No, this is a bad idea. Asking novice developers to make security
>>decisions they're not yet qualified to make when it's genuinely
>>possible for us to do the right thing by default is the antithesis of
>>good security API design, and os.urandom() *is* a security API
>>(whether we like it or not - third party documentation written by the
>>cryptographic software development community has made it so, since
>>it's part of their guidelines for writing security sensitive code in
>>pure Python).
>
> Regardless of what third parties have said about os.urandom(), let's look at
> what *we* have said about it.  Going back to pre-churn 3.4 documentation:
>
> os.urandom(n)
> Return a string of n random bytes suitable for cryptographic use.
>
> This function returns random bytes from an OS-specific randomness
> source. The returned data should be unpredictable enough for cryptographic
> applications, though its exact quality depends on the OS
> implementation. On a Unix-like system this will query /dev/urandom, and on
> Windows it will use CryptGenRandom(). If a randomness source is not found,
> NotImplementedError will be raised.
>
> For an easy-to-use interface to the random number generator provided by
> your platform, please see random.SystemRandom.
>
> So we very clearly provided platform-dependent caveats on the cryptographic
> quality of os.urandom().  We also made a strong claim that there's a direct
> connection between os.urandom() and /dev/urandom on "Unix-like system(s)".
>
> We broke that particular promise in 3.5. and semi-fixed it 3.5.2.
>
>>Adding *new* APIs is also a bad idea, since "os.urandom() is the right
>>answer on every OS except Linux, and also the best currently available
>>answer on Linux" has been the standard security advice for generating
>>cryptographic secrets in pure Python code for years now, so we should
>>only change that guidance if we have extraordinarily compelling
>>reasons to do so, and we don't.
>
> Disagree.
>
> We have broken one long-term promise on os.urandom() ("On a Unix-like system
> this will query /dev/urandom") and changed another ("should be unpredictable
> enough for cryptographic applications, though its exact quality depends on OS
> implementations").
>
> We broke the experienced Linux developer's natural and long-standing link
> between the API called os.urandom() and /dev/urandom.  This breaks pre-3.5
> code that assumes read-from-/dev/urandom semantics for os.urandom().
>
> We have introduced churn.  Predicting a future SO question such as "Can
> os.urandom() block on Linux?" the answer is "No in Python 3.4 and earlier, yes
> possibly in Python 3.5.0 and 3.5.1, no in Python 3.5.2 and the rest of the
> 3.5.x series, and yes possibly in Python 3.6 and beyond".

It also depends on the kernel version, since it will never block on
old kernels that are missing getrandom(), but it might block on future
kernels if Linux's /dev/urandom ever becomes blocking. (Ted's said
that this is not going to happen now, but the only reason it isn't was
that he tried to make the change and it broke some distros that are
still in use -- so it seems entirely possible that it will happen a
few years from now.)

> We have a better answer for "cryptographically appropriate" use cases in
> Python 3.6 - the secrets module.  Trying to make os.urandom() "the right
> answer on every OS" weakens the promotion of secrets as *the* module to use
> for cryptographically appropriate use cases.
>
> IMHO it would be better to leave os.urandom() well enough alone, except for
> the documentation which should effectively say, a la 3.4:
>
> os.urandom(n)
> Return a string of n random bytes suitable for cryptographic use.
>
> This function returns random bytes from an OS-specific randomness
> source. The returned data should be unpredictable enough for cryptographic
> applications, though its exact quality depends on the OS
> implementation. On a Unix-like system this will query /dev/urandom, and on
> Windows it will use CryptGenRandom(). If a randomness source is not found,
> NotImplementedError will be raised.
>
> Cryptographic applications should use the secrets module for stronger
> guaranteed sources of randomness.
>
> For an easy-to-use interface to the random number generator provided by
> your platform, please see random.SystemRandom.

This is not an accurate docstring, though. The more accurate docstring
for your proposed behavior would be:

os.urandom(n)
Return a string of n bytes that will usually, but not always, be
suitable for cryptographic use.

This function returns random bytes from an OS-specific randomness
source. On non-Linux OSes, this uses the best available source of
randomness, e.g. CryptGenRandom() on Windows and /dev/urandom on OS X,
and thus will be strong enough 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Stefan Krah]

Nathaniel Smith  pobox.com> writes:
> On Wed, Jun 15, 2016 at 10:25 PM, Theodore Ts'o  mit.edu> wrote:
> > In practice, those Python ivocation which are exposed to hostile input
> > are those that are started while the network are up.
> 
> Not sure what you mean about the vast majority of Python invocations
> being launched by the web browser?

"Python invocations which are exposed to hostile input". ;)


Stefan Krah

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Wed, Jun 15, 2016 at 10:25 PM, Theodore Ts'o  wrote:
> On Wed, Jun 15, 2016 at 04:12:57PM -0700, Nathaniel Smith wrote:
>> - It's not exactly true that the Python interpreter doesn't need
>> cryptographic randomness to initialize SipHash -- it's more that
>> *some* Python invocations need unguessable randomness (to first
>> approximation: all those which are exposed to hostile input), and some
>> don't. And since the Python interpreter has no idea which case it's
>> in, and since it's unacceptable for it to break invocations that don't
>> need unguessable hashes, then it has to err on the side of continuing
>> without randomness. All that's fine.
>
> In practice, those Python ivocation which are exposed to hostile input
> are those that are started while the network are up.  The vast
> majority of time, they are launched by the web brwoser --- and if this
> happens after a second or so of the system getting networking
> interrupts, (a) getrandom won't block, and (b) /dev/urandom and
> getrandom will be initialized.

Not sure what you mean about the vast majority of Python invocations
being launched by the web browser? But anyway, sure, usually this
isn't an issue. This is just discussing about what to do in the
unlikely case when it actually has become an issue, and it's hard to
be certain that this will *never* happen. E.g. it's entirely plausible
that someone will write some cloud-init plugin that exposes an HTTP
server or something. People do all kinds of weird things in VMs these
days... Basically this is a question of whether we should make an
(unlikely) error totally invisible to the user, and "errors should
never pass silently" is right there in the Zen of Python :-).

> Also, I wish people would say that this is only an issue on Linux.
> Again, FreeBSD's /dev/urandom will block as well if it is
> uninitialized.  It's just that in practice, for both Linux and
> Freebsd, we try very hard to make sure /dev/urandom is fully
> initialized by the time it matters.  It's just that so far, it's only
> on Linux when there was an attempt to use Python in the early init
> scripts, and in a VM and in a system where everything is modularized
> such that the deadlock became visible.
>
>
>> (I guess the way to implement this would be for the SipHash
>> initialization code -- which runs very early -- to set some flag, and
>> then we expose that flag in sys._something, and later in the startup
>> sequence check for it after the warnings module is functional.
>> Exposing the flag at the Python level would also make it possible for
>> code like cloud-init to do its own explicit check and respond
>> appropriately.)
>
> I really don't think it's that big a of a deal in *practice*, and but
> if you really are concerned about the very remote possibility that a
> Python invocation could start in early boot, and *then* also stick
> around for the long term, and *then* be exosed to hostile input ---
> what if you set the flag, and then later on, N minutes, either
> automatically, or via some trigger such as cloud-init --- try and see
> if /dev/urandom is initialized (even a few seconds later, so long as
> the init scripts are hanging, it should be initialized) have Python
> hash all of its dicts, or maybe just the non-system dicts (since those
> are presumably the ones mos tlikely to be exposed hostile input).

I don't think this is technically doable. There's no global list of
hash tables, and Python exposes the actual hash values to user code
with some guarantee that they won't change.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Robert Collins]

On 16 Jun 2016 6:55 PM, "Larry Hastings"  wrote:
>
>

> Why do you call it only "semi-fixed"?  As far as I understand it, the
semantics of os.urandom() in 3.5.2rc1 are indistinguishable from reading
from /dev/urandom directly, except it may not need to use a file handle.

Which is a contract change. Someone testing in E.g. a chroot could have a
different device on /dev/urandom, and now they will need to intercept
syscalls for the same effect. Personally I think this is fine, but assuming
i see Barry's point correctly, it is indeed but the same as it was.

-rob
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/15/2016 11:45 PM, Barry Warsaw wrote:

So we very clearly provided platform-dependent caveats on the cryptographic
quality of os.urandom().  We also made a strong claim that there's a direct
connection between os.urandom() and /dev/urandom on "Unix-like system(s)".

We broke that particular promise in 3.5. and semi-fixed it 3.5.2.


Well, 3.5.2 hasn't happened yet.  So if you see it as still being 
broken, please speak up now.


Why do you call it only "semi-fixed"?  As far as I understand it, the 
semantics of os.urandom() in 3.5.2rc1 are indistinguishable from reading 
from /dev/urandom directly, except it may not need to use a file handle.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 15, 2016, at 01:01 PM, Nick Coghlan wrote:

>No, this is a bad idea. Asking novice developers to make security
>decisions they're not yet qualified to make when it's genuinely
>possible for us to do the right thing by default is the antithesis of
>good security API design, and os.urandom() *is* a security API
>(whether we like it or not - third party documentation written by the
>cryptographic software development community has made it so, since
>it's part of their guidelines for writing security sensitive code in
>pure Python).

Regardless of what third parties have said about os.urandom(), let's look at
what *we* have said about it.  Going back to pre-churn 3.4 documentation:

os.urandom(n)
Return a string of n random bytes suitable for cryptographic use.

This function returns random bytes from an OS-specific randomness
source. The returned data should be unpredictable enough for cryptographic
applications, though its exact quality depends on the OS
implementation. On a Unix-like system this will query /dev/urandom, and on
Windows it will use CryptGenRandom(). If a randomness source is not found,
NotImplementedError will be raised.

For an easy-to-use interface to the random number generator provided by
your platform, please see random.SystemRandom.

So we very clearly provided platform-dependent caveats on the cryptographic
quality of os.urandom().  We also made a strong claim that there's a direct
connection between os.urandom() and /dev/urandom on "Unix-like system(s)".

We broke that particular promise in 3.5. and semi-fixed it 3.5.2.

>Adding *new* APIs is also a bad idea, since "os.urandom() is the right
>answer on every OS except Linux, and also the best currently available
>answer on Linux" has been the standard security advice for generating
>cryptographic secrets in pure Python code for years now, so we should
>only change that guidance if we have extraordinarily compelling
>reasons to do so, and we don't.

Disagree.

We have broken one long-term promise on os.urandom() ("On a Unix-like system
this will query /dev/urandom") and changed another ("should be unpredictable
enough for cryptographic applications, though its exact quality depends on OS
implementations").

We broke the experienced Linux developer's natural and long-standing link
between the API called os.urandom() and /dev/urandom.  This breaks pre-3.5
code that assumes read-from-/dev/urandom semantics for os.urandom().

We have introduced churn.  Predicting a future SO question such as "Can
os.urandom() block on Linux?" the answer is "No in Python 3.4 and earlier, yes
possibly in Python 3.5.0 and 3.5.1, no in Python 3.5.2 and the rest of the
3.5.x series, and yes possibly in Python 3.6 and beyond".

We have a better answer for "cryptographically appropriate" use cases in
Python 3.6 - the secrets module.  Trying to make os.urandom() "the right
answer on every OS" weakens the promotion of secrets as *the* module to use
for cryptographically appropriate use cases.

IMHO it would be better to leave os.urandom() well enough alone, except for
the documentation which should effectively say, a la 3.4:

os.urandom(n)
Return a string of n random bytes suitable for cryptographic use.

This function returns random bytes from an OS-specific randomness
source. The returned data should be unpredictable enough for cryptographic
applications, though its exact quality depends on the OS
implementation. On a Unix-like system this will query /dev/urandom, and on
Windows it will use CryptGenRandom(). If a randomness source is not found,
NotImplementedError will be raised.

Cryptographic applications should use the secrets module for stronger
guaranteed sources of randomness.

For an easy-to-use interface to the random number generator provided by
your platform, please see random.SystemRandom.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Wed, Jun 15, 2016 at 04:12:57PM -0700, Nathaniel Smith wrote:
> - It's not exactly true that the Python interpreter doesn't need
> cryptographic randomness to initialize SipHash -- it's more that
> *some* Python invocations need unguessable randomness (to first
> approximation: all those which are exposed to hostile input), and some
> don't. And since the Python interpreter has no idea which case it's
> in, and since it's unacceptable for it to break invocations that don't
> need unguessable hashes, then it has to err on the side of continuing
> without randomness. All that's fine.

In practice, those Python ivocation which are exposed to hostile input
are those that are started while the network are up.  The vast
majority of time, they are launched by the web brwoser --- and if this
happens after a second or so of the system getting networking
interrupts, (a) getrandom won't block, and (b) /dev/urandom and
getrandom will be initialized.

Also, I wish people would say that this is only an issue on Linux.
Again, FreeBSD's /dev/urandom will block as well if it is
uninitialized.  It's just that in practice, for both Linux and
Freebsd, we try very hard to make sure /dev/urandom is fully
initialized by the time it matters.  It's just that so far, it's only
on Linux when there was an attempt to use Python in the early init
scripts, and in a VM and in a system where everything is modularized
such that the deadlock became visible.


> (I guess the way to implement this would be for the SipHash
> initialization code -- which runs very early -- to set some flag, and
> then we expose that flag in sys._something, and later in the startup
> sequence check for it after the warnings module is functional.
> Exposing the flag at the Python level would also make it possible for
> code like cloud-init to do its own explicit check and respond
> appropriately.)

I really don't think it's that big a of a deal in *practice*, and but
if you really are concerned about the very remote possibility that a
Python invocation could start in early boot, and *then* also stick
around for the long term, and *then* be exosed to hostile input ---
what if you set the flag, and then later on, N minutes, either
automatically, or via some trigger such as cloud-init --- try and see
if /dev/urandom is initialized (even a few seconds later, so long as
the init scripts are hanging, it should be initialized) have Python
hash all of its dicts, or maybe just the non-system dicts (since those
are presumably the ones mos tlikely to be exposed hostile input).

  - Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 15 June 2016 at 16:12, Nathaniel Smith  wrote:
> On Wed, Jun 15, 2016 at 1:01 PM, Nick Coghlan  wrote:
>> Victor has some additional technical details written up at
>> http://haypo-notes.readthedocs.io/pep_random.html and I'd be happy to
>> formalise this proposed approach as a PEP (the current reference is
>> http://bugs.python.org/issue27282 )
>
> I'd make two additional suggestions:
>
> - one person did chime in on the thread to say that they've used
> os.urandom for non-security-sensitive purposes, simply because it
> provided a convenient "give me a random byte-string" API that is
> missing from random. I think we should go ahead and add a .randbytes
> method to random.Random that simply returns a random bytestring using
> the regular RNG, to give these users a nice drop-in replacement for
> os.urandom.

That seems reasonable.

> - It's not exactly true that the Python interpreter doesn't need
> cryptographic randomness to initialize SipHash -- it's more that
> *some* Python invocations need unguessable randomness (to first
> approximation: all those which are exposed to hostile input), and some
> don't. And since the Python interpreter has no idea which case it's
> in, and since it's unacceptable for it to break invocations that don't
> need unguessable hashes, then it has to err on the side of continuing
> without randomness. All that's fine.
>
> But, given that the interpreter doesn't know which state it's in,
> there's also the possibility that this invocation *will* be exposed to
> hostile input, and the 3.5.2+ behavior gives absolutely no warning
> that this is what's happening. So instead of letting this potential
> error pass silently, I propose that if SipHash fails to acquire real
> randomness at startup, then it should issue a warning. In practice,
> this will almost never happen. But in the rare cases it does, it at
> least gives the user a fighting chance to realize that their system is
> in a potentially dangerous state. And by using the warnings module, we
> automatically get quite a bit of flexibility.
>
> If some particular
> invocation (e.g. systemd-cron) has audited their code and decided that
> they don't care about this issue, they can make the message go away:
>
>PYTHONWARNINGS=ignore::NoEntropyAtStartupWarning
>
> OTOH if some particular invocation knows that they do process
> potentially hostile input early on (e.g. cloud-init, maybe?), then
> they can explicitly promote the warning to an error:
>
>   PYTHONWARNINGS=error::NoEntropyAtStartupWarning
>
> (I guess the way to implement this would be for the SipHash
> initialization code -- which runs very early -- to set some flag, and
> then we expose that flag in sys._something, and later in the startup
> sequence check for it after the warnings module is functional.
> Exposing the flag at the Python level would also make it possible for
> code like cloud-init to do its own explicit check and respond
> appropriately.)

A Python level warning/flag seems overly elaborate to me, but we can
easily emit a warning on stderr when SipHash is initialised via the
fallback rather than the operating system's RNG.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Wed, Jun 15, 2016 at 1:01 PM, Nick Coghlan  wrote:
[...]
> For 3.6+, we can instead make it so that the only things that actually
> rely on cryptographic quality randomness being available are:
>
> - calling a secrets module API
> - calling a random.SystemRandom method
> - calling os.urandom directly
>
> These are all APIs that were either created specifically for use in
> security sensitive situations (secrets module), or have long been
> documented (both within our own documentation, and in third party
> documentation, books and Q sites) as being an appropriate choice for
> use in security sensitive situations (os.urandom and
> random.SystemRandom).
>
> However, we don't need to make those block waiting for randomness to
> be available - we can update them to raise BlockingIOError instead
> (which makes it trivial for people to decide for themselves how they
> want to handle that case).
>
> Along with that change, we can make it so that starting the
> interpreter will never block waiting for cryptographic randomness to
> be available (since it doesn't need it), and importing the random
> module won't block waiting for it either.

This all seems exactly right to me, to the point that I've been
dreading having to find the time to write pretty much this exact
email. So thank you :-)

> To the best of our knowledge, on all operating systems other than
> Linux, encountering the new exception will still be impossible in
> practice, as there is no known opportunity to run Python code before
> the kernel random number generator is ready.
>
> On Linux, init scripts may still run before the kernel random number
> generator is ready, but will now throw an immediate BlockingIOError if
> they access an API that relies on crytographic randomness being
> available, rather than potentially deadlocking the init process. Folks
> encountering that situation will then need to make an explicit
> decision:
>
> - loop until the exception is no longer thrown
> - switch to reading from /dev/urandom directly instead of calling os.urandom()
> - switch to using a cross-platform non-cryptographic API (probably the
> random module)
>
> Victor has some additional technical details written up at
> http://haypo-notes.readthedocs.io/pep_random.html and I'd be happy to
> formalise this proposed approach as a PEP (the current reference is
> http://bugs.python.org/issue27282 )

I'd make two additional suggestions:

- one person did chime in on the thread to say that they've used
os.urandom for non-security-sensitive purposes, simply because it
provided a convenient "give me a random byte-string" API that is
missing from random. I think we should go ahead and add a .randbytes
method to random.Random that simply returns a random bytestring using
the regular RNG, to give these users a nice drop-in replacement for
os.urandom.

Rationale: I don't think the existence of these users should block
making os.urandom appropriate for generating secrets, because (1) a
glance at github shows that this is very unusual -- if you skim
through this search you get page after page of functions with names
like "generate_secret_key"

  
https://github.com/search?l=python=2=urandom=searchresults=Code=%E2%9C%93

and (2) for the minority of people who are using os.urandom for
non-security-sensitive purposes, if they find os.urandom raising an
error, then this is just a regular bug that they will notice
immediately and fix, and anyway it's basically never going to happen.
(As far as we can tell, this has never yet happened in the wild, even
once.) OTOH if os.urandom is allowed to fail silently, then people who
are using it to generate secrets will get silent catastrophic
failures, plus those users can't assume it will never happen because
they have to worry about active attackers trying to drive systems into
unusual states. So I'd much rather ask the non-security-sensitive
users to switch to using something in random, than force the
cryptographic users to switch to using secrets. But it does seem like
it would be good to give those non-security-sensitive users something
to switch to :-).

- It's not exactly true that the Python interpreter doesn't need
cryptographic randomness to initialize SipHash -- it's more that
*some* Python invocations need unguessable randomness (to first
approximation: all those which are exposed to hostile input), and some
don't. And since the Python interpreter has no idea which case it's
in, and since it's unacceptable for it to break invocations that don't
need unguessable hashes, then it has to err on the side of continuing
without randomness. All that's fine.

But, given that the interpreter doesn't know which state it's in,
there's also the possibility that this invocation *will* be exposed to
hostile input, and the 3.5.2+ behavior gives absolutely no warning
that this is what's happening. So instead of letting this potential
error pass silently, I propose that if SipHash fails to acquire real
randomness at 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ethan Furman]

On 06/15/2016 01:01 PM, Nick Coghlan wrote:


For 3.6+, we can instead make it so that the only things that actually
rely on cryptographic quality randomness being available are:

- calling a secrets module API
- calling a random.SystemRandom method
- calling os.urandom directly




However, we don't need to make those block waiting for randomness to
be available - we can update them to raise BlockingIOError instead
(which makes it trivial for people to decide for themselves how they
want to handle that case).

Along with that change, we can make it so that starting the
interpreter will never block waiting for cryptographic randomness to
be available (since it doesn't need it), and importing the random
module won't block waiting for it either.


+1

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

[whew, actually read the whole thread]

On 11 June 2016 at 10:28, Terry Reedy  wrote:
> On 6/11/2016 11:34 AM, Guido van Rossum wrote:
>>
>> In terms of API design, I'd prefer a flag to os.urandom() indicating a
>> preference for
>> - blocking
>> - raising an exception
>> - weaker random bits
>
>
> +100 ;-)
>
> I proposed exactly this 2 days ago, 5 hours after Larry's initial post.

No, this is a bad idea. Asking novice developers to make security
decisions they're not yet qualified to make when it's genuinely
possible for us to do the right thing by default is the antithesis of
good security API design, and os.urandom() *is* a security API
(whether we like it or not - third party documentation written by the
cryptographic software development community has made it so, since
it's part of their guidelines for writing security sensitive code in
pure Python).

Adding *new* APIs is also a bad idea, since "os.urandom() is the right
answer on every OS except Linux, and also the best currently available
answer on Linux" has been the standard security advice for generating
cryptographic secrets in pure Python code for years now, so we should
only change that guidance if we have extraordinarily compelling
reasons to do so, and we don't. Instead, we have Ted T'so himself
chiming in to say: "My preference would be that os.[u]random should
block, because the odds that people would be trying to generate
long-term cryptographic secrets within seconds after boot is very
small, and if you *do* block for a second or two, it's not the end of
the world."

The *actual bug* that triggered this latest firestorm of commentary
(from experts and non-experts alike) had *nothing* to do with user
code calling os.urandom, and instead was a combination of:

- CPython startup requesting cryptographically secure randomness when
it didn't need it
- a systemd init script written in Python running before the kernel
RNG was fully initialised

That created a deadlock between CPython startup and the rest of the
Linux init process, so the latter only continued when the systemd
watchdog timed out and killed the offending script. As others have
noted, this kind of deadlock scenario is generally impossible on other
operating systems, as the operating system doesn't provide a way to
run Python code before the random number generator is ready.

The change Victor made in 3.5.2 to fall back to reading /dev/urandom
directly if the getrandom() syscall returns EAGAIN (effectively
reverting to the Python 3.4 behaviour) was the simplest possible fix
for that problem (and an approach I thoroughly endorse, both for 3.5.2
and for the life of the 3.5 series), but that doesn't make it the
right answer for 3.6+.

To repeat: the problem encountered was NOT due to user code calling
os.urandom(), but rather due to the way CPython initialises its own
internal hash algorithm at interpreter startup. However, due to the
way CPython is currently implemented, fixing the regression in that
not only changed the behaviour of CPython startup, it *also* changed
the behaviour of every call to os.urandom() in Python 3.5.2+.

For 3.6+, we can instead make it so that the only things that actually
rely on cryptographic quality randomness being available are:

- calling a secrets module API
- calling a random.SystemRandom method
- calling os.urandom directly

These are all APIs that were either created specifically for use in
security sensitive situations (secrets module), or have long been
documented (both within our own documentation, and in third party
documentation, books and Q sites) as being an appropriate choice for
use in security sensitive situations (os.urandom and
random.SystemRandom).

However, we don't need to make those block waiting for randomness to
be available - we can update them to raise BlockingIOError instead
(which makes it trivial for people to decide for themselves how they
want to handle that case).

Along with that change, we can make it so that starting the
interpreter will never block waiting for cryptographic randomness to
be available (since it doesn't need it), and importing the random
module won't block waiting for it either.

To the best of our knowledge, on all operating systems other than
Linux, encountering the new exception will still be impossible in
practice, as there is no known opportunity to run Python code before
the kernel random number generator is ready.

On Linux, init scripts may still run before the kernel random number
generator is ready, but will now throw an immediate BlockingIOError if
they access an API that relies on crytographic randomness being
available, rather than potentially deadlocking the init process. Folks
encountering that situation will then need to make an explicit
decision:

- loop until the exception is no longer thrown
- switch to reading from /dev/urandom directly instead of calling os.urandom()
- switch to using a cross-platform non-cryptographic API (probably the
random module)

Victor has 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sun, Jun 12, 2016 at 06:53:54PM -0700, Nathaniel Smith wrote:
> 
> Speaking of full-stack perspectives, would it affect your decision if
> Debian Stretch were made robust against blocking /dev/urandom on
> AWS/GCE? Because I think we could find lots of people who would be
> overjoyed to fix Stretch before the next merge window even opens
> (AFAICT the quick fix is literally a 1 line patch), if that allowed
> the blocking /dev/urandom patches to go in upstream...

Alas, it's not just Debian.  Apparently it breaks the boot on Openwrt
as well as Ubuntu Quantal:

https://lkml.org/lkml/2016/6/13/48
https://lkml.org/lkml/2016/5/31/599

(Yay for an automated test infrastructure that fires off as soon as
you push to an externally visible git repository.  :-)

I haven't investigated to see exactly *why* it's blowing up on these
userspace setups, but it's a great reminder for why changing an
established interface is something that has to be done very carefully
indeed.

   - Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Sun, Jun 12, 2016 at 4:28 PM, Theodore Ts'o  wrote:
> P.S.  BTW, I probably won't change the behaviour of /dev/urandom to
> make it be blocking.  Before I found out about Pyhton Bug #26839, I
> actually had patches that did make /dev/urandom blocking, and they
> were planned to for the next kernel merge window.  But ultimately, the
> reason why I won't is because there is a set of real users (Debian
> Stretch users on Amazon AWS and Google GCE) for which if I changed how
> /dev/urandom worked, then I would be screwing them over, even if
> Python 3.5.2 falls back to /dev/urandom.  It's not a problem for bare
> metal hardware and cloud systems with virtio-rng; I have patches that
> will take care of those scenarios.
>
> Unfortunately, both AWS and GCE don't support virtio-rng currently,
> and as much as some poeple are worried about the hypothetical problems
> of stupidly written/deployed Python scripts that try to generate
> long-term secrets during early boot, weighed against the very real
> prospect of user lossage on two of the most popular Cloud environments
> out there --- it's simply no contest.

Speaking of full-stack perspectives, would it affect your decision if
Debian Stretch were made robust against blocking /dev/urandom on
AWS/GCE? Because I think we could find lots of people who would be
overjoyed to fix Stretch before the next merge window even opens
(AFAICT the quick fix is literally a 1 line patch), if that allowed
the blocking /dev/urandom patches to go in upstream...

(It looks like Jessie isn't affected, because while Jessie does
provide a systemd-cron package for those who decide to install it,
Jessie's systemd-cron is still using python2, python2 doesn't have
hash randomization so it doesn't touch /dev/urandom at startup, and
systemd-cron doesn't have any code that would trigger access to
/dev/urandom otherwise. It looks like Xenial *is* affected, because
they ship systemd-cron with python3, but their python3 is still
unconditionally using getrandom() in blocking mode, so they need to
patch that regardless, and could just as easily make it robust against
blocking /dev/urandom at the same time. I don't understand the RPM
world as well, but I can't find any evidence that Fedora or SuSE ship
systemd-cron at all.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sun, Jun 12, 2016 at 09:01:09PM +0100, Cory Benfield wrote:
> My problem with /dev/urandom is that it’s a trap, lying in wait for
> someone who doesn’t know enough about the problem they’re solving to
> step into it.

And my answer to that question is absent backwards compatibility
concerns, use getrandom(2) on Linux, or getentropy(2) on *BSD, and be
happy.  Don't use /dev/urandom; use getrandom(2) instead.  That way
you also solve a number of other problems such as the file descriptor
DOS attack issue, etc.

The problem with Python is that you *do* have backwards compatibility
concerns.  At which point you are faced with the same issues that we
are in the kernel; except I gather than that the commitment to
backwards compatibility isn't quite as absolute (although it is
strong).  Which is why I've been trying very hard not to tell
python-dev what to do, but rather to give you folks the best
information I can, and then encouraging you to do whatever seems most
"Pythony" --- which might or might not be the same as the decisions
we've made in the kernel.

Cheers,

- Ted

P.S.  BTW, I probably won't change the behaviour of /dev/urandom to
make it be blocking.  Before I found out about Pyhton Bug #26839, I
actually had patches that did make /dev/urandom blocking, and they
were planned to for the next kernel merge window.  But ultimately, the
reason why I won't is because there is a set of real users (Debian
Stretch users on Amazon AWS and Google GCE) for which if I changed how
/dev/urandom worked, then I would be screwing them over, even if
Python 3.5.2 falls back to /dev/urandom.  It's not a problem for bare
metal hardware and cloud systems with virtio-rng; I have patches that
will take care of those scenarios.

Unfortunately, both AWS and GCE don't support virtio-rng currently,
and as much as some poeple are worried about the hypothetical problems
of stupidly written/deployed Python scripts that try to generate
long-term secrets during early boot, weighed against the very real
prospect of user lossage on two of the most popular Cloud environments
out there --- it's simply no contest.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sun, Jun 12, 2016 at 11:07:22AM -0700, Nathaniel Smith wrote:
> But for example, if a process is actively blocked waiting
> for the initial entropy, one could spawn a kernel thread that keeps the
> system from quiescing by attempting to scrounge up entropy as fast as
> possible, via whatever mechanisms are locally appropriate (e.g. doing a
> busy-loop racing two clocks against each other, or just scheduling lots of
> interrupts -- which I guess is the same thing, more or less).

There's a lot of snake oil, or at least, hand waving, that goes on
with respect to what will actually work to gather randomness.  One of
the worst possible choices is a standard, kernel-defined workload that
tries to just busy loop two clocks against each other.  For one thing,
on many embedded systems, all of your clocks are generated off of a
single master oscillator anyway.  And in early boot, it's not
realistic for the kernel to be able to measure network interrupt
timings and radio strength indicators from the WiFi, which ultimately
is going to be much more likely to be unpredictable by an outside
attacker sitting in Fort Meade than pretending that you can just
"schedule lots of interrupts".

Again, part of the problem here is that if you really want to be
secure, it needs to be a full stack perspective, where the hardware
designers, the OS developers, and the application level developers are
all working together.  If one side tries to exert a strong "somebody
else's problem field", it's very likely the end solution isn't going
to be secure.  Because in many cases this is simply not practical, we
all have to make assumptions at the OS and C-Python interpreter level,
and hope that the assumptions that we make are are conservative
enough.

> Is this an approach that you've considered?

Ultimately, the arguments made by approaches such as Jitterbug are, to
put it succiently and perhaps a little unfairly, "gee whillikers, the
Intel L1/L2 cache hierarchy is really complicated and it's a closed
hardware implementation so no one can understand it, and besides, the
statistical analysis of the output looks good".

To which I would say, "the first argument is an argument of security
through ignorance", and "AES(NSA_KEY, COUNTER++)" also has really
great statistical results, and if you don't know the NSA_KEY, it will
look very strong and as far as we know, we wouldn't be able to
distinguish it from truly secure random number generator --- but it
really isn't secure.

So yeah, I don't buy it.  In order for it to be secure, we need to be
grabbing measurements which can't be replicated or determined by a
remote attacker.  So having the kernel kick off a kernel thread is not
going to be useful unless we can mix in entropy from the user, or the
workload, or the local configuration, or from the local environment.
(Using RSSI is helpful because the remote attacker might not know
whether your mobile handset is in the knapsack under the table, or on
the desk, and that will change the RSSI numbers.)  Remember, the whole
*point* of modern CPU designs is that the huge amounts of engineering
effort is put into making the CPU be predictable, and so spawning a
kernel thread in isolation isn't going perform magic in terms of
getting guaranteed unpredictability.

> FWIW, the systemd thing is a red herring -- this was debian's configuration
> of a particular daemon that is not maintained by the systemd project, and
> the exact same thing would have happened with sysvinit if debian had tried
> using python 3.5 early in their rcS.

It's not a daemon.  It's the script in
/lib/systemd/system-generators/systemd-crontab-generator, and it's
needed because systemd subsumed the cron daemon, and developers who
wanted to not break user's existing crontab files turned to it.  I
suppose you are technically correct that it is not mainained by
systemd, but the need for it was generated out of systemd's lack of
concern of backwards compatibility.

Because FreeBSD and Mac OS are not using systemd, they are not likely
to run into this problem.  I will grant that if they decided to try to
run a python script out of their /etc/rc script, they would run into
the same problem.

- Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 12 Jun 2016, at 14:43, Theodore Ts'o  wrote:
> 
> Well, it can only happen on Linux because you insist on falling back
> to /dev/urandom --- and because other OS's have the good taste not to
> use systemd and/or Python very early in the boot process.  If someone
> tried to run a python script in early FreeBSD init scripts, it would
> block just as you were seeing on Linux --- you just haven't seen that
> yet, because arguably the FreeBSD developers have better taste in
> their choice of init scripts than Red Hat and Debian.  :-)

Heh, yes, so to be clear, I said “this can only happen on Linux” because I’m 
talking about the world that we live in: the one where I lost this debate. =D

Certainly right now the codebase as it stands could encounter the same problems 
on FreeBSD. That’s a problem for Python to deal with.

> So the question is whether I should do what FreeBSD did, which will
> statisfy those people who are freaking out and whinging about how
> Linux could allow stupidly written or deployed Python scripts get
> cryptographically insecure bytes, by removing that option from Python
> developers.  Or should I remove that one line from changes in the
> random.git patch series, and allow /dev/urandom to be used even when
> it might be insecure, so as to satisfy all of the people who are
> freaking out and whinging about the fact that a stupildly written
> and/or deployed Python script might block during early boot and hang a
> system?
> 
> Note that I've tried to do what I can to make the time that
> /dev/urandom might block as small as possible, but at the end of the
> day, there is still the question of whether I should remove the choice
> re: blocking from userspace, ala FreeBSD, or not.  And either way,
> some number of people will be whinging and freaking out.  Which is why
> I completely sympathetic to how Guido might be getting a little
> exasperated over this whole thread.  :-)

I don’t know that we need to talk about removing the choice. I understand the 
desire to commit to backwards compatibility, of course I do. My problem with 
/dev/urandom is not that it *exists*, per se: all kinds of stupid stuff exists 
for the sake of backward compatibility.

My problem with /dev/urandom is that it’s a trap, lying in wait for someone who 
doesn’t know enough about the problem they’re solving to step into it. And it’s 
the worst kind of trap: it’s one you don’t know you’ve stepped in. Nothing 
about the failure mode of /dev/urandom is obvious. Worse, well-written apps 
that try their best to do the right thing can still step into that failure mode 
if they’re run in a situation that they weren’t expecting (e.g. on an embedded 
device without hardware RNG or early in the boot process).

So my real problem with /dev/urandom is that the man page doesn’t say, in 
gigantic letters, “this device has a really nasty failure mode that you cannot 
possibly detect by just running the code in the dangerous mode”. It’s 
understandable to have insecure weak stuff available to users: Python has loads 
of it. But where possible, the documentation marks it as such. It’d be good to 
have /dev/urandom’s man page say “hey, by the way, you almost certainly don’t 
want this: try using getrandom() instead”.

Anyway, regarding changing the behaviour of /dev/urandom: as you’ve correctly 
highlighted, at this point you’re damned if you do and damned if you don’t. If 
you don’t change, you’ll forever have people like me saying that /dev/urandom 
is dangerous, and that its behaviour in the unseeded/poorly-seeded state is a 
misfeature. I trust you’ll understand when I tell you that that opinion has 
nothing to do with *you* or the Linux kernel maintainership. This is all about 
the way software security evolves: things that used to be ok start to become 
not ok over time. We learn, we improve.

Of course, if you do change the behaviour, you’ll rightly have programmers 
stumble onto this exact problem. They’ll be unhappy too. And the worst part of 
all of this is that neither side of that debate is *wrong*: they just 
prioritise different things. Guido, Larry, and friends aren’t wrong, any more 
than I am: we just rate the different concerns differently. That’s fine: after 
all, it’s probably why Guido invented and maintains an extremely popular 
programming language and I haven’t and never will! I have absolutely no problem 
with breaking “working” code if I believe that that code is exposing users to 
risks they aren’t aware of (you can check my OSS record to prove it, and I’m 
happy to provide references).

The best advice I can give anyone in this debate, on either side, is to make 
decisions that you can live with. Consider the consequences, consider the 
promises you’ve made to users, and then do what you think is right. Guido and 
Larry have decided to go with backward-compatibility: fine. They’re 
responsible, the buck stops with them, they know that. The same is true for 
you, Ted, with the 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Jun 11, 2016 11:13 PM, "Theodore Ts'o"  wrote:
>
> On Sat, Jun 11, 2016 at 05:46:29PM -0400, Donald Stufft wrote:
> >
> > It was a RaspberryPI that ran a shell script on boot that called
> > ssh-keygen.  That shell script could have just as easily been a
> > Python script that called os.urandom via
> > https://github.com/sybrenstuvel/python-rsa instead of a shell script
> > that called ssh-keygen.
>
> So I'm going to argue that the primary bug was in the how the systemd
> init scripts were configured.  In generally, creating keypairs at boot
> time is just a bad idea.  They should be created lazily, in a
> just-in-time paradigm.
>
> Consider that if you assume that os.urandom can block, this isn't
> necessarily going to do the right thing either --- if you use
> getrandom and it blocks, and it's part of a systemd unit which is
> blocking futher boot progress, then the system will hang for 90
> seconds, and while it's hanging, there won't be any interrupts, so the
> system will be dead in the water, just like the orignal bug report
> complaining that Python was hanging when it was using getrandom() to
> initialize its SipHash.

Hi Ted,

>From another perspective, I guess one could also argue that the best place
to fix this is in the kernel: if a process is blocked waiting for entropy
then the kernel probably shouldn't take that its cue to turn off all the
entropy generation mechanisms, just like how if a process is blocked
waiting for disk I/O then we probably shouldn't power down the disk
controller. Obviously this is a weird case because the kernel is
architected in a way that makes the dependency between the disk controller
and the I/O request obvious, while the dependency between the random pool
and... well... everything else, more or less, is much more subtle and goes
outside the usual channels, and we wouldn't want to rearchitect everything
just for this. But for example, if a process is actively blocked waiting
for the initial entropy, one could spawn a kernel thread that keeps the
system from quiescing by attempting to scrounge up entropy as fast as
possible, via whatever mechanisms are locally appropriate (e.g. doing a
busy-loop racing two clocks against each other, or just scheduling lots of
interrupts -- which I guess is the same thing, more or less). And the
thread would go away again as soon as userspace wasn't blocked on entropy.
That way this deadlock wouldn't be possible.

I guess someone *might* complain about the idea of the entropy pool
actually spending resources instead of being quietly parasitic, because
this is the kernel and someone will always complain about everything :-).
But complaining about this makes about much sense as complaining about the
idea of spending resources trying to service I/O when a process is blocked
on that ("maybe if we wait long enough then some other part of the system
will just kind of accidentally page in the data we need as a side effect of
whatever it's doing, and then this thread will be able to proceed").

Is this an approach that you've considered?

> At which point there will be another bug complaining about how python
> was causing systemd to hang for 90 seconds, and there will be demand
> to make os.random no longer block.  (Since by definition, systemd can
> do no wrong; it's always other programs that have to change to
> accomodate systemd.  :-)

FWIW, the systemd thing is a red herring -- this was debian's configuration
of a particular daemon that is not maintained by the systemd project, and
the exact same thing would have happened with sysvinit if debian had tried
using python 3.5 early in their rcS.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sun, Jun 12, 2016 at 11:40:58AM +0100, Cory Benfield wrote:
> 
> Of this set, only cloud-init worries me, and it worries me for the
> *opposite* reason that Guido and Larry are worried. Guido and Larry
> are worried that programs like cloud-init will be delayed by two
> minutes while they wait for entropy: that’s an understandable
> concern. I’m much more worried that programs like cloud-init may
> attempt to establish TLS connections or create keys during this two
> minute window, leaving them staring down the possibility of
> performing “secure” actions with insecure keys.

There are patches in the dev branch of:

 https://git.kernel.org/cgit/linux/kernel/git/tytso/random.git/

which will automatically use virtio-rng (if it is provided by the
cloud provider) to initialize /dev/urandom.  It also uses a much more
aggressive mechanism to initialize the /dev/urandom pool, so that
getrandom(2) will block for a much shorter period of time immediately
after boot time on real hardware.  I'm confident it's secure for x86
platforms.  I'm still thinking about whether I should fall back to
something more conservative for crappy embedded processors that don't
have a cycle counter or an CPU-provided RDRAND-like instruction.
Related to this is whether I should finally make the change so that
/dev/urandom will block until it is initialized.  (This would make
Linux work like FreeBSD, which *will* also block if its entropy pool
is not initialized.)

> This is why I advocate, like Donald does, for having *some* tool in
> Python that allows Python programs to crash if they attempt to
> generate cryptographically secure random bytes on a system that is
> incapable of providing them (which, in practice, can only happen on
> Linux systems).

Well, it can only happen on Linux because you insist on falling back
to /dev/urandom --- and because other OS's have the good taste not to
use systemd and/or Python very early in the boot process.  If someone
tried to run a python script in early FreeBSD init scripts, it would
block just as you were seeing on Linux --- you just haven't seen that
yet, because arguably the FreeBSD developers have better taste in
their choice of init scripts than Red Hat and Debian.  :-)

So the question is whether I should do what FreeBSD did, which will
statisfy those people who are freaking out and whinging about how
Linux could allow stupidly written or deployed Python scripts get
cryptographically insecure bytes, by removing that option from Python
developers.  Or should I remove that one line from changes in the
random.git patch series, and allow /dev/urandom to be used even when
it might be insecure, so as to satisfy all of the people who are
freaking out and whinging about the fact that a stupildly written
and/or deployed Python script might block during early boot and hang a
system?

Note that I've tried to do what I can to make the time that
/dev/urandom might block as small as possible, but at the end of the
day, there is still the question of whether I should remove the choice
re: blocking from userspace, ala FreeBSD, or not.  And either way,
some number of people will be whinging and freaking out.  Which is why
I completely sympathetic to how Guido might be getting a little
exasperated over this whole thread.  :-)

- Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sun, Jun 12, 2016 at 01:49:34AM -0400, Random832 wrote:
> > The intention behind getrandom() is that it is intended *only* for
> > cryptographic purposes. 
> 
> I'm somewhat confused now because if that's the case it seems to
> accomplish multiple unrelated things. Why was this implemented as a
> system call rather than a device (or an ioctl on the existing ones)? If
> there's a benefit in not going through the non-atomic (and possibly
> resource limited) procedure of acquiring a file descriptor, reading from
> it, and closing it, why is that benefit not also extended to
> non-cryptographic users of urandom via allowing the system call to be
> used in that way?

This design was taken from OpenBSD, and the goal with getentropy(2)
(which is also designed only for cryptographic use cases), was so that
a denial of service attack (fd exhaustion) could force an application
to fall back to a weaker -- in some cases, very weak or non-existent
--- source of randomness.

Non-cryptographic users don't need to use this interface at all.  They
can just use srandom(3)/random(3) and be happy.

> > Anyway, if you don't need cryptographic guarantees, you don't need
> > getrandom(2) or getentropy(2); something like this will do just fine:
> 
> Then what's /dev/urandom *for*, anyway?

/dev/urandom is a legacy interface.  It was intended originally for
cryptographic use cases, but it was intended for the days when very
few programs needed a secure cryptographic random generator, and it
was assumed that application programmers would be very careful in
checking error codes, etc.

It also dates back to a time when the NSA was still pushing very hard
for cryptographic export controls (hence the use of SHA-1 versus an
encryption algorithm) and when many people questioned whether or not
the SHA-1 algorithm, as designed by the NSA, had a backdoor in it.
(As it turns out, the NSA put a back door into DUAL-EC, so retrospect
this concern really wasn't that unreasonable.)  Because of those
concerns, the assumption is those few applications who really wanted
to get security right (e.g., PGP, which still uses /dev/random for
long-term key generation), would want to use /dev/random and deal with
entropy accounting, and asking the user to type randomness on the
keyboard and move their mouse around while generating a random key.

But times change, and these days people are much more likely to
believe that SHA-1 is in fact cryptographically secure, and future
crypto hash algorithms are designed by teams from all over the world
and NIST/NSA merely review the submissions (along with everyone else).
So for example, SHA-3 was *not* designed by the NSA, and it was
evaluated using a much more open process than SHA-1.

Also, we have a much larger set of people writing code which is
sensitive to cryptographic issues (back when I wrote /dev/random, I
probably had met, or at least electronically corresponded with a large
number of the folks who were working on network security protocols, at
least in the non-classified world), and these days, there is much less
trust that people writing code to use /dev/[u]random are in fact
careful and competent security engineers.  Whether or not this is a
fair concern or not, it is true that there has been a change in API
design ethos away from the "Unix let's make things as general as
possible, in case someone clever comes up use case we didn't think
of", to "idiots are ingenious so they will come up with ways to misuse
an idiot-proof interface, so we need to lock it down as much as
possible."  OpenBSD's getentropy(2) interface is a strong example of
this new attitude towards API design, and getrandom(2) is not quite so
doctrinaire (I added a flags field when getentropy(2) didn't even give
those options to progammers), but it is following in the same tradition.

Cheers,

- Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Paul Moore]

On 11 June 2016 at 22:46, Donald Stufft  wrote:
> I guess one question would be, what does the secrets module do if it’s on a
> Linux that is too old to have getrandom(0), off the top of my head I can
> think of:
>
> * Silently fall back to reading os.urandom and hope that it’s been seeded.
> * Fall back to os.urandom and hope that it’s been seeded and add a
> SecurityWarning or something like it to mention that it’s falling back to
> os.urandom and it may be getting predictable random from /dev/urandom.
> * Hard fail because it can’t guarantee secure cryptographic random.
>
> Of the three, I would probably suggest the second one, it doesn’t let the
> problem happen silently, but it still “works” (where it’s basically just
> hoping it’s being called late enough that /dev/urandom has been seeded), and
> people can convert it to the third case using the warnings module to turn
> the warning into an exception.

I have kept out of this discussion as I don't know enough about
security to comment, but in this instance I think the answer is clear
- there is no requirement for Python to protect the user against
security bugs in the underlying OS (sure, it's nice if it can, but
it's not necessary) so fallng back to os.urandom (with no warning) is
fine. A warning, or even worse a hard fail, that 99.99% of the time
should be ignored (because you're *not* writing a boot script) seems
like a very bad idea.

By all means document "if your OS provides no means of getting
guaranteed secure randon mumbers (e.g., older versions of Linux very
early in the boot sequence) then the secrets module cannot give you
results that are any better than the OS provides". It seems
self-evident to me that this would be the case, but I see no reason to
object if the experts feel it's worth adding.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 12 Jun 2016, at 07:11, Theodore Ts'o  wrote:
> 
> On Sat, Jun 11, 2016 at 05:46:29PM -0400, Donald Stufft wrote:
>> 
>> It was a RaspberryPI that ran a shell script on boot that called
>> ssh-keygen.  That shell script could have just as easily been a
>> Python script that called os.urandom via
>> https://github.com/sybrenstuvel/python-rsa instead of a shell script
>> that called ssh-keygen.
> 
> So I'm going to argue that the primary bug was in the how the systemd
> init scripts were configured.  In generally, creating keypairs at boot
> time is just a bad idea.  They should be created lazily, in a
> just-in-time paradigm.

Agreed. I hope that if there is only one thing every participant has learned 
from this (extremely painful for all concerned) discussion, it’s that doing 
anything that requires really good random numbers should be delayed as long as 
possible on all systems, and should absolutely not be done during the boot 
process on Linux. Don’t generate key pairs, don’t make TLS connections, just 
don’t perform any action that requires really good randomness at all.

> So some people will freak out when the keygen systemd unit hangs,
> blocking the boot --- and other people will freak out of the systemd
> unit doesn't hang, and you get predictable SSH keys --- and some wiser
> folks will be asking the question, why the *heck* is it not
> openssh/systemd's fault for trying to generate keys this early,
> instead of after the first time sshd needs host ssh keys?  If you wait
> until the first time the host ssh keys are needed, then the system is
> fully booted, so it's likely that the entropy will be collected -- and
> even if it isn't, networking will already be brought up, and the
> system will be in multi-user mode, so entropy will be collected very
> quickly.

As far as I know we still only have three programs that were encountering this 
problem: Debian’s autopkgtest (which patched with PYTHONHASHSEED=0), 
systemd-cron (which is moving from Python to Rust anyway), and cloud-init (not 
formally reported but mentioned to me by a third-party). It remains unclear to 
me why the systemd-cron service files can’t simply request to be delayed until 
the kernel CSPRNG is seeded: I guess systemd doesn’t have any way to express 
that constraint? Perhaps it should.

Of this set, only cloud-init worries me, and it worries me for the *opposite* 
reason that Guido and Larry are worried. Guido and Larry are worried that 
programs like cloud-init will be delayed by two minutes while they wait for 
entropy: that’s an understandable concern. I’m much more worried that programs 
like cloud-init may attempt to establish TLS connections or create keys during 
this two minute window, leaving them staring down the possibility of performing 
“secure” actions with insecure keys.

This is why I advocate, like Donald does, for having *some* tool in Python that 
allows Python programs to crash if they attempt to generate cryptographically 
secure random bytes on a system that is incapable of providing them (which, in 
practice, can only happen on Linux systems). I don’t care how it’s spelled, I 
just care that programs that want to use a properly-seeded CSPRNG can error out 
effectively when one is not available. That allows us to ensure that Python 
programs that want to do TLS or build key pairs correctly refuse to do so when 
used in this state, *and* that they provide a clearly debuggable reason for why 
they refused. That allows the savvy application developers that Ted talked 
about to make their own decisions about whether their rapid startup is 
sufficiently important to take the risk.

Cory


[0]: 
https://github.com/systemd-cron/systemd-cron/issues/43#issuecomment-160343989



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Sat, Jun 11, 2016 at 05:46:29PM -0400, Donald Stufft wrote:
> 
> It was a RaspberryPI that ran a shell script on boot that called
> ssh-keygen.  That shell script could have just as easily been a
> Python script that called os.urandom via
> https://github.com/sybrenstuvel/python-rsa instead of a shell script
> that called ssh-keygen.

So I'm going to argue that the primary bug was in the how the systemd
init scripts were configured.  In generally, creating keypairs at boot
time is just a bad idea.  They should be created lazily, in a
just-in-time paradigm.

Consider that if you assume that os.urandom can block, this isn't
necessarily going to do the right thing either --- if you use
getrandom and it blocks, and it's part of a systemd unit which is
blocking futher boot progress, then the system will hang for 90
seconds, and while it's hanging, there won't be any interrupts, so the
system will be dead in the water, just like the orignal bug report
complaining that Python was hanging when it was using getrandom() to
initialize its SipHash.

At which point there will be another bug complaining about how python
was causing systemd to hang for 90 seconds, and there will be demand
to make os.random no longer block.  (Since by definition, systemd can
do no wrong; it's always other programs that have to change to
accomodate systemd.  :-)

So some people will freak out when the keygen systemd unit hangs,
blocking the boot --- and other people will freak out of the systemd
unit doesn't hang, and you get predictable SSH keys --- and some wiser
folks will be asking the question, why the *heck* is it not
openssh/systemd's fault for trying to generate keys this early,
instead of after the first time sshd needs host ssh keys?  If you wait
until the first time the host ssh keys are needed, then the system is
fully booted, so it's likely that the entropy will be collected -- and
even if it isn't, networking will already be brought up, and the
system will be in multi-user mode, so entropy will be collected very
quickly.

Sometimes, we can't solve the problem at the Python level or at the
Kernel level.  It will require security-saavy userspace/application
programmers as well.

Cheers,

- Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Sat, Jun 11, 2016, at 22:37, Theodore Ts'o wrote:
> On Fri, Jun 10, 2016 at 05:14:50PM -0400, Random832 wrote:
> > So, I have a question. If this "weakness" in /dev/urandom is so
> > unimportant to 99% of situations... why isn't there a flag that can be
> > passed to getrandom() to allow the same behavior?
> 
> The intention behind getrandom() is that it is intended *only* for
> cryptographic purposes. 

I'm somewhat confused now because if that's the case it seems to
accomplish multiple unrelated things. Why was this implemented as a
system call rather than a device (or an ioctl on the existing ones)? If
there's a benefit in not going through the non-atomic (and possibly
resource limited) procedure of acquiring a file descriptor, reading from
it, and closing it, why is that benefit not also extended to
non-cryptographic users of urandom via allowing the system call to be
used in that way?

> Anyway, if you don't need cryptographic guarantees, you don't need
> getrandom(2) or getentropy(2); something like this will do just fine:

Then what's /dev/urandom *for*, anyway?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Sat, Jun 11, 2016 at 02:16:21PM -0700, Guido van Rossum wrote:

[on the real-world consequences of degraded randomness from /dev/urandom]

> Actually it's not clear to me at all that it could have happened to Python.
> (Wasn't it an embedded system?)

A Raspberry Pi. But don't people run Python on at least some embedded 
systems? The wiki thinks so:

https://wiki.python.org/moin/EmbeddedPython

And I thought that was the purpose of µPython.


> > Actually the proposal for that was the secrets module. And the secrets
> > module would be the only user of os.urandom(blocking=True).
> >
> > I’m fine if this lives in the secrets module— Steven asked for it to be an
> > os function so that secrets.py could continue to be pure python.
> 
> The main thing that I want to avoid is that people start cargo-culting
> whatever the secrets module uses rather than just using the secrets module.
> Having it redundantly available as os.getrandom() is just begging for
> people to show off how much they know about writing secure code.

That makes sense.

I'm happy for getrandom to be an implementation detail of secrets, but 
I'll need help with that part.


> >> * If you want to ensure you get cryptographically secure bytes,
> >> os.getrandom, falling back to os.urandom on non Linux platforms and
> >> erroring on Linux.
[...]

> But what is a Python script going to do with that error? IIUC this kind of
> error would only happen very early during boot time, and rarely, so the
> most likely outcome is a hard-to-debug mystery failure.

In my day job, I work for a Linux sys admin consulting company, and I 
can tell you from our experience that debugging a process that 
occasionally hangs mysteriously during boot is much harder than 
debugging a process that occasionally fails with an explicit error in 
the logs, especially if the error message is explicit about the cause:

OSError: entropy pool has not been initialized yet

At that point, you can take whatever action is appropriate for your 
script:

- fail altogether, just as it might fail if it requires a writable
  file system and can't find one;
- sleep for three seconds and try again;
- log the error and proceed with degraded randomness or functionality;
- change it so the script runs later in the boot process.




-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

On Fri, Jun 10, 2016 at 05:14:50PM -0400, Random832 wrote:
> On Fri, Jun 10, 2016, at 15:54, Theodore Ts'o wrote:
> > So even on Python pre-3.5.0, realistically speaking, the "weakness" of
> > os.random would only be an issue (a) if it is run within the first few
> > seconds of boot, and (b) os.random is used to directly generate a
> > long-term cryptographic secret.  If you are fork openssl or ssh-keygen
> > to generate a public/private keypair, then you aren't using os.random.
> 
> So, I have a question. If this "weakness" in /dev/urandom is so
> unimportant to 99% of situations... why isn't there a flag that can be
> passed to getrandom() to allow the same behavior?

The intention behind getrandom() is that it is intended *only* for
cryptographic purposes.  For that use case, there's no point having a
"return a potentially unseeded cryptographic secret" option.  This
makes this much like FreeBSD's /dev/random and getentropy system
calls.

(BTW, I've seen an assertion on this thread that FreeBSD's
getentropy(2) never blocks.  As far as I know, this is **not** true.
FreeBSD's getentropy(2) works like its /dev/random device, in that if
it is not fully seeded, it will block.  The only reason why OpenBSD's
getentropy(2) and /dev/random devices will never block is because they
only support architectures where they can make sure that entropy is
passed from a previous boot session to the next, given specialized
bootloader support.  Linux can't do this because we support a very
large number of bootloaders, and the bootloaders are not under the
kernel developers' control.  Fundamentally, you can't guarantee both
(a) that your RNG will never block, and (b) will always be of high
cryptographic quality, in a completely general sense.  You can if you
make caveats about your hardware or when the code runs, but that's
fundamentally the problem with the documentation of os.urandom(); it's
making promises which can't be true 100% of the time, for all
hardware, operating environments, etc.)

Anyway, if you don't need cryptographic guarantees, you don't need
getrandom(2) or getentropy(2); something like this will do just fine:

long getrand()
{
static int initialized = 0;
struct timeval tv;

if (!initialized) {
gettimeofday(, NULL);
srandom(tv.tv_sec ^ tv.tv_usec ^ getpid());
initialized++;
}
return random();
}

So this is why I did what I did.  If Python decides to go down this
same path, you could define a new interface ala getrandom(2), which is
specifically designed for cryptogaphic purposes, and perhaps a new,
more efficient interface for those people who don't need cryptogaphic
guarantees --- and then keep the behavior of os.urandom consistent
with Python 3.4, but update the documentation to reflect the reality.

Alternatively, you could keep the implementation of os.urandom
consistent with Python 3.5, and then document that under some
circumstances, it will block.  Both approaches have certain tradeoffs,
but it's not going to be the end of the world regardless of which way
you decide to go.   I'd suggest that you use your existing mechanisms
to decide on which approach is more Pythony, and then make sure you
communicate and over-communicate it to your user/developer base.

And then --- relax.  It may seem like a big deal today, but in a year
or so people will have gotten used to whatever interface or
documentation changes you decide to make, and it will be all fine.
As Dame Julian of Norwich once said, "All shall be well, and all shall
be well, and all manner of things shall be well."  

Cheers,

- Ted
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Tim Peters]

[Sebastian Krause]
> ...
> Ideally I would only want to use the random module for
> non-secure and (in 3.6) the secrets module (which could block) for
> secure random data and never bother with os.urandom (and knowing how
> it behaves). But then those modules should probably get new
> functions to directly return bytes.

`secrets.token_bytes()` does just that, and other token_XXX()
functions return bytes too but with different spellings (e.g., if you
want, with the byte values represented as ASCII hex digits)..

I believe everyone agrees token_bytes() will potentially block in 3.6
(along with all the other `secrets` facilities) on platforms
supporting getrandom().

You're right that `random` doesn't expose such a function, and that
the closest it gets is .getrandbits() (which returns a potentially
giant int).  So far, nobody has proposed adding new functions to
`random`.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

Fortunately, 3.6 feature freeze isn't until September, so we can all cool
off and figure out the best way forward. I'm going on vacation for a week,
and after sending this I'm going to mute the thread so I won't be pulled
into it while I'm supposed to be relaxing.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/11/2016 01:48 PM, Guido van Rossum wrote:
So I still don't see why we need os.getrandom() -- it has nothing to 
recommend it over the secrets module (since both won't happen before 3.6).


I have two reasons, neither of which I think are necessarily all that 
persuasive.  Don't consider this an argument--merely some observations.


First, simply as a practical matter: the secrets module is currently 
pure Python.  ISTM that the os module is where we put miscellaneous bits 
of os functionality; getrandom() definitely falls into that category.  
Rather than adding a new _secrets module or whatever it seemed easiest 
just to add it there.


Second, I'd put this under the "consenting adults" rule.  Clearly 
cryptography is a contentious subject with sharply differing opinions.  
There are many, many cryptography libraries available on PyPi; perhaps 
those libraries would like to use getrandom(), or /dev/urandom, or even 
getentropy(), in a way different than how secrets does it.  My thinking 
is, the os module should provide platform support, the secrets module 
should be our codified best-practices, and we encourage everyone to use 
secrets.  I'd go so far as to add that recommendation to the doc *and* 
the docstrings of os.urandom(), random.SystemRandom, and os.getrandom() 
(and os.getentropy()) if we add it.  But by providing the OS 
functionality in a neutral way we allow external cryptographers to write 
what *they* view as best-practices code without wading into 
implementation detalis of secrets, or using ctypes, or whatnot.



But like I said I don't have a strong opinion.  As long as we're not 
adding mysterious flags to os.urandom() I'll probably sit the rest of 
this one out.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 5:16 PM, Guido van Rossum  wrote:
> 
> On Sat, Jun 11, 2016 at 1:48 PM, Donald Stufft  > wrote:
> 
>> On Jun 11, 2016, at 3:40 PM, Guido van Rossum > > wrote:
>> 
>> Yeah, but we've already established that there's a lot more upset, rhetoric 
>> and worry than warranted by the situation.
> 
> Have we? There are real, documented security failures in the wild because of 
> /dev/urandom’s behavior. This isn’t just a theoretical problem, it actually 
> has had consequences in real life, and those same consequences could just 
> have easily happened to Python (in one of the cases that most recently comes 
> to mind it was a C program, but that’s not really relevant because the same 
> problem would have happened if they had written in Python using os.urandom in 
> 3.4 but not in 3.5.0 or 3.5.1.
> 
> Actually it's not clear to me at all that it could have happened to Python. 
> (Wasn't it an embedded system?)

It was a RaspberryPI that ran a shell script on boot that called ssh-keygen. 
That shell script could have just as easily been a Python script that called 
os.urandom via https://github.com/sybrenstuvel/python-rsa 
 instead of a shell script that 
called ssh-keygen.

>> Actually the proposal for that was the secrets module. And the secrets 
>> module would be the only user of os.urandom(blocking=True).
> 
> I’m fine if this lives in the secrets module— Steven asked for it to be an os 
> function so that secrets.py could continue to be pure python.
> 
> The main thing that I want to avoid is that people start cargo-culting 
> whatever the secrets module uses rather than just using the secrets module. 
> Having it redundantly available as os.getrandom() is just begging for people 
> to show off how much they know about writing secure code. 


I guess one question would be, what does the secrets module do if it’s on a 
Linux that is too old to have getrandom(0), off the top of my head I can think 
of:

* Silently fall back to reading os.urandom and hope that it’s been seeded.
* Fall back to os.urandom and hope that it’s been seeded and add a 
SecurityWarning or something like it to mention that it’s falling back to 
os.urandom and it may be getting predictable random from /dev/urandom.
* Hard fail because it can’t guarantee secure cryptographic random.

Of the three, I would probably suggest the second one, it doesn’t let the 
problem happen silently, but it still “works” (where it’s basically just hoping 
it’s being called late enough that /dev/urandom has been seeded), and people 
can convert it to the third case using the warnings module to turn the warning 
into an exception.


>>  
>> * If you want to ensure you get cryptographically secure bytes, 
>> os.getrandom, falling back to os.urandom on non Linux platforms and erroring 
>> on Linux.
>> 
>> "Erroring" doesn't sound like it satisfies the "ensure" part of the 
>> requirement. And I don't see the advantage of os.getrandom() over the 
>> secrets module. (Either way you have to fall back on os.urandom() to 
>> suppport Python 3.5 and before.)
> 
> Erroring does satisfy the ensure part, because if it’s not possible to get 
> cryptographically secure bytes then the only option is to error if you want 
> to be ensured of cryptographically secure bytes. 
> 
> It’s a bit like if you did open(“somefile.txt”), it’s reasonable to say that 
> we should ensure that open(“somefile.txt”) actually opens ./somefile.txt, and 
> doesn’t randomly open a different file if ./somefile.txt doesn’t exist— if it 
> can’t open ./somefile.txt it should error. If I *need* cryptographically 
> secure random bytes, and I’m on a platform that doesn’t provide those, then 
> erroring is often times the correct behavior. This is such an important thing 
> that OS X will flat out kernel panic and refuse to boot if it can’t ensure 
> that it can give people cryptographically secure random bytes.
> 
> But what is a Python script going to do with that error? IIUC this kind of 
> error would only happen very early during boot time, and rarely, so the most 
> likely outcome is a hard-to-debug mystery failure.

Depends on why they’re calling it, which is sort of the underlying problem I 
suspect with why there isn’t agreement about what the right default behavior 
is. The correct answer for some application might be to hard fail and wait for 
the operator to fix the environment that it’s running in. It depends on how 
important the thing that is getting this random is.

One example: If I was writing a communication platform for people who are 
fighting oppressive regimes or to securely discuss sexual orientation in more 
dangerous parts of the world, I would want to make this program hard fail if it 
couldn’t ensure that it was using an interface that ensured cryptographic 
random, because the alternative is 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Tim Peters]

[Guido]
> ...
> An alternative would be to keep the secrets module linked to SystemRandom,
> and improve the latter. Its link with os.random() is AFAIK undocumented. Its
> API is clumsy but for code that needs some form of secret-ish bytes and
> requires platform and Python version independence it might be better than
> anything else. Then the secrets module is just what we recommend new users
> on Python 3.6.

There's an issue currently open about this:

http://bugs.python.org/issue27288

The docs for SystemRandom are very brief, so people may have actually
noticed ;-) the first sentence:

Class that uses the os.urandom() function for generating random numbers ...

IOW, "uses os.urandom()" has been one of its only advertised qualities.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

On Sat, Jun 11, 2016 at 2:16 PM, Donald Stufft  wrote:

>
> On Jun 11, 2016, at 4:48 PM, Guido van Rossum  wrote:
>
> But I find an os.getrandom() that only exists on those (few?) platforms
> that support it a nuisance too -- this just encourages cargo cult code
> that's unnecessarily complicated and believed to be secure without anybody
> ever verifying.
>
>
>
> Well, new enough Linux has getrandom(0), OpenBSD has getentropy(), Solaris
> has getrandom(), Windows has CryptGenRandom which all make it possible (or
> it’s the only way to invoke it) to get cryptographically secure random
> bytes or block and no in-between. So it’d likely be possible to have
> os.getrandom() with blocking semantics and no FD on all of the most popular
> platforms we support.
>
> If we relax the no FD then FreeBSD and OS X also have /dev/random (or
> /dev/urandom it’s the same thing) which will ensure that you give
> cryptographically secure random bytes.
>

OK, so we should implement the best we can do for the secrets module, and
leave os.urandom() alone. I think the requirement that the secrets module
remain pure Python has to be dropped. I'm not sure what it should do if
even blocking can't give it sufficiently strong random bytes, but I care
much less -- it's a new API and it doesn't resemble any OS function, so as
long as it is documented it should be fine.

An alternative would be to keep the secrets module linked to SystemRandom,
and improve the latter. Its link with os.random() is AFAIK undocumented.
Its API is clumsy but for code that needs some form of secret-ish bytes and
requires platform and Python version independence it might be better than
anything else. Then the secrets module is just what we recommend new users
on Python 3.6.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

On Sat, Jun 11, 2016 at 1:48 PM, Donald Stufft  wrote:

>
> On Jun 11, 2016, at 3:40 PM, Guido van Rossum  wrote:
>
> Yeah, but we've already established that there's a lot more upset,
> rhetoric and worry than warranted by the situation.
>
>
> Have we? There are real, documented security failures in the wild because
> of /dev/urandom’s behavior. This isn’t just a theoretical problem, it
> actually has had consequences in real life, and those same consequences
> could just have easily happened to Python (in one of the cases that most
> recently comes to mind it was a C program, but that’s not really relevant
> because the same problem would have happened if they had written in Python
> using os.urandom in 3.4 but not in 3.5.0 or 3.5.1.
>

Actually it's not clear to me at all that it could have happened to Python.
(Wasn't it an embedded system?)

> Actually the proposal for that was the secrets module. And the secrets
> module would be the only user of os.urandom(blocking=True).
>
>
> I’m fine if this lives in the secrets module— Steven asked for it to be an
> os function so that secrets.py could continue to be pure python.
>

The main thing that I want to avoid is that people start cargo-culting
whatever the secrets module uses rather than just using the secrets module.
Having it redundantly available as os.getrandom() is just begging for
people to show off how much they know about writing secure code.

>
>
>> * If you want to ensure you get cryptographically secure bytes,
>> os.getrandom, falling back to os.urandom on non Linux platforms and
>> erroring on Linux.
>>
>
> "Erroring" doesn't sound like it satisfies the "ensure" part of the
> requirement. And I don't see the advantage of os.getrandom() over the
> secrets module. (Either way you have to fall back on os.urandom() to
> suppport Python 3.5 and before.)
>
>
> Erroring does satisfy the ensure part, because if it’s not possible to get
> cryptographically secure bytes then the only option is to error if you want
> to be ensured of cryptographically secure bytes.
>

> It’s a bit like if you did open(“somefile.txt”), it’s reasonable to say
> that we should ensure that open(“somefile.txt”) actually opens
> ./somefile.txt, and doesn’t randomly open a different file if
> ./somefile.txt doesn’t exist— if it can’t open ./somefile.txt it should
> error. If I *need* cryptographically secure random bytes, and I’m on a
> platform that doesn’t provide those, then erroring is often times the
> correct behavior. This is such an important thing that OS X will flat out
> kernel panic and refuse to boot if it can’t ensure that it can give people
> cryptographically secure random bytes.
>

But what is a Python script going to do with that error? IIUC this kind of
error would only happen very early during boot time, and rarely, so the
most likely outcome is a hard-to-debug mystery failure.


> It’s a fairly simple decision tree, I go “hey, give me cryptographically
> secure random bytes, and only cryptographically secure random bytes”. If it
> cannot give them to me because the APIs of the system cannot guarantee they
> are cryptographically secure then there are only two options, either A) it
> is explicit about it’s inability to do this and raises an error or B) it
> does something completely different than what I asked it to do and pretends
> that it’s what I wanted.
>

I really don't believe that there is only one kind of cryptographically
secure random bytes. There are many different applications (use cases) of
randomness and they need different behaviors. (If it was simple we wouldn't
still be arguing. :-)

>
>
>> * If you want to *ensure* that there’s no blocking, then os.urandom on
>> Linux (or os.urandom wrapped with timeout code anywhere else, as that’s the
>> only way to ensure not blocking cross platform).
>>
>
> That's fine with me.
>
>
>> * If you just don’t care, YOLO it up with either os.urandom or
>> os.getrandom or random.random.
>>
>
> Now you're just taking the mickey.
>
>
> No I’m not— random.Random is such a use case where it wants to seed with
> as secure of bytes as it can get it’s hands on, but it doesn’t care if it
> falls back to insecure bytes if it’s not possible to get secure bytes. This
> code even falls back to using time as a seed if all else fails.
>

Fair enough. The hash randomization is the other case I suppose (since not
running any Python code at all isn't an option, and neither is waiting
indefinitely before the user's code gets control).

It does show the point that there are different use cases with different
needs. But I think the stdlib should limit the choices.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 4:48 PM, Guido van Rossum  wrote:
> 
> But I find an os.getrandom() that only exists on those (few?) platforms that 
> support it a nuisance too -- this just encourages cargo cult code that's 
> unnecessarily complicated and believed to be secure without anybody ever 
> verifying.



Well, new enough Linux has getrandom(0), OpenBSD has getentropy(), Solaris has 
getrandom(), Windows has CryptGenRandom which all make it possible (or it’s the 
only way to invoke it) to get cryptographically secure random bytes or block 
and no in-between. So it’d likely be possible to have os.getrandom() with 
blocking semantics and no FD on all of the most popular platforms we support.

If we relax the no FD then FreeBSD and OS X also have /dev/random (or 
/dev/urandom it’s the same thing) which will ensure that you give 
cryptographically secure random bytes.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

On Sat, Jun 11, 2016 at 12:53 PM, Larry Hastings  wrote:

>
> On 06/11/2016 11:30 AM, Donald Stufft wrote:
>
> The problem is that someone writing software that does
> os.urandom(block=True) or os.urandom(exception=True) which gets some bytes
> doesn’t know if it got back cryptographically secure random because Python
> called getrandom() or if it got back cryptographically secure random
> because it called /dev/urandom and that gave it secure random because it’s
> on a platform that defines that as always returning secure or because it’s
> on Linux and the urandom pool is initialized or if it got back some random
> bytes that are not cryptographically secure because it fell back to reading
> /dev/urandom on Linux prior to the pool being initialized.
>
>
> Let me jump in tangentially to say: I think os.urandom(block=True) is
> simply a bad API.  On FreeBSD and OpenBSD, /dev/urandom may block, and you
> don't have a choice.  On OS X, /dev/urandom will never block, and you don't
> have a choice.  In Victor's initial patch where he proposed it, the flag
> was accepted on all platforms but only affected its behavior on Linux and
> possibly Solaris.  I think it's bad API design to have a flag that seems
> like it would be meaningful on multiple platforms, but in practice is
> useful only in very limited circumstances.  If this were old code, or
> behavior we inherited from the platform and we were making the best of a
> bad situation, that'd be one thing.  But this is a proposed new API and I
> definitely think we can do better.
>
> As I understand the proposed semantics for os.urandom(exception=True), I
> feel it falls into the same trap though not to the same degree.
>
> Of course, both flags break backwards-compatibility if they default to
> True, and I strongly disagree with .
>
> It's far better in my opinion to keep the os module as a thin shell over
> platform functionality.  That makes Python's behavior more predictable on a
> platform-by-platform basis.  So I think the best approach here is to add
> os.getrandom() as a thin shell over the local getrandom() (if any).
>

OK, the flags are unpopular, so let's forget about them.

But I find an os.getrandom() that only exists on those (few?) platforms
that support it a nuisance too -- this just encourages cargo cult code
that's unnecessarily complicated and believed to be secure without anybody
ever verifying.

I'd like to consider what people freak out about.

- You could freak out about blocking

- You could freak out about getting slightly less random bits

- You could freak out about supporting Python 3.5 and earlier

- You could freak out about supporting all platforms

You could also freak out about combinations of the above, but that gets
complicated and you should probably consider that you're over-constraining
matters. If you freak out about all at once (or both the first and the
second bullet) you should consider a career change.

If you don't freak out about any of these (meaning you're happy with Python
3.6+) you should use the secrets module.

If you freak out about support for older Python versions, try the secrets
module first and fall back to os.urandom() -- there really isn't any other
choice.

If you freak out about getting slightly less random bits you should
probably do a complete security assessment of your entire stack and fix the
OS and Python version, and use the best you can get for that combination.
You may not want to rely on the standard library at all.

If you freak out about blocking you're probably on a specific platform, and
if that platform is Linux, you're in luck: use os.urandom() and avoid
Python 3.5.0 and 3.5.1. On other platforms you're out of luck.

So I still don't see why we need os.getrandom() -- it has nothing to
recommend it over the secrets module (since both won't happen before 3.6).

So what should the secrets module use? Let's make that part an extension
module.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 4:26 PM, Brett Cannon  wrote:
> 
> Only issue I can think of that we're missing is one to track reverting 
> os.urandom() to 3.4 semantics (any doc updates required for the random 
> module?). Am I missing anything?


It’s already been reverted to 3.4 semantics (well, it will try to use 
getrandom(GRD_NONBLOCK) but falls back to /dev/urandom if that would have 
blocked).

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 3:40 PM, Guido van Rossum  wrote:
> 
> On Sat, Jun 11, 2016 at 11:30 AM, Donald Stufft  > wrote:
> 
>> On Jun 11, 2016, at 1:39 PM, Guido van Rossum > > wrote:
>> 
>> Is the feature detection desire about being able to write code that runs on 
>> older Python versions or for platforms that just don't have getrandom()?
>> 
>> My assumption was that nobody would actually use these flags except the 
>> secrets module and people writing code that generates long-lived secrets -- 
>> and the latter category should be checking platform and versions anyway 
>> since they need the whole stack to be secure (if I understand Ted Ts'o's 
>> email right).
>> 
>> My assumption is also that the flags should be hints (perhaps only relevant 
>> on Linux) -- platforms that can't perform the action desired (because their 
>> system's API doesn't support it) would just do their default action, 
>> assuming the system API does the best it can.
> 
> The problem is that someone writing software that does os.urandom(block=True) 
> or os.urandom(exception=True) which gets some bytes doesn’t know if it got 
> back cryptographically secure random because Python called getrandom() or if 
> it got back cryptographically secure random because it called /dev/urandom 
> and that gave it secure random because it’s on a platform that defines that 
> as always returning secure or because it’s on Linux and the urandom pool is 
> initialized or if it got back some random bytes that are not 
> cryptographically secure because it fell back to reading /dev/urandom on 
> Linux prior to the pool being initialized.
> 
> The “silently does the wrong thing, even though I explicitly asked for it do 
> something different” is something that I would consider to be a footgun and 
> footgun’s in security sensitive code make me really worried.
> 
> Yeah, but we've already established that there's a lot more upset, rhetoric 
> and worry than warranted by the situation.


Have we? There are real, documented security failures in the wild because of 
/dev/urandom’s behavior. This isn’t just a theoretical problem, it actually has 
had consequences in real life, and those same consequences could just have 
easily happened to Python (in one of the cases that most recently comes to mind 
it was a C program, but that’s not really relevant because the same problem 
would have happened if they had written in Python using os.urandom in 3.4 but 
not in 3.5.0 or 3.5.1.

>  
> Outside of the security side of things, if someone goes “Ok I need some 
> random bytes and I need to make sure it doesn’t block”, then doing 
> ``os.random(block=False, exception=False)`` isn’t going to make sure that it 
> doesn’t block except on Linux.
> 
> To people who "just want some random bytes" we should recommend the random 
> module.
>  
> In other words, it’s basically impossible to ensure you get the behavior you 
> want with these flags which I feel like will make everyone unhappy (both the 
> people who want to ensure non-blocking, and the people who want to ensure 
> cryptographically secure). These flags are an attractive nuisance that look 
> like they do the right thing, but silently don’t.
> 
> OK, it looks like the flags just won't make you happy, and I'm happy to give 
> up on them. By default the status quo will win, and that means neither these 
> flags nor os.getrandom(). (But of course you can roll your own using ctypes. 
> :-)
>  
> Meanwhile if we have os.urandom that reads from /dev/urandom and 
> os.getrandom() which reads from blocking random, then we make it both easier 
> to ensure you get the behavior you want, either by using the function that 
> best suits your needs:
> 
> * If you just want the best the OS has to offer, os.getrandom falling back to 
> os.urandom.
> 
> Actually the proposal for that was the secrets module. And the secrets module 
> would be the only user of os.urandom(blocking=True).

I’m fine if this lives in the secrets module— Steven asked for it to be an os 
function so that secrets.py could continue to be pure python.

>  
> * If you want to ensure you get cryptographically secure bytes, os.getrandom, 
> falling back to os.urandom on non Linux platforms and erroring on Linux.
> 
> "Erroring" doesn't sound like it satisfies the "ensure" part of the 
> requirement. And I don't see the advantage of os.getrandom() over the secrets 
> module. (Either way you have to fall back on os.urandom() to suppport Python 
> 3.5 and before.)

Erroring does satisfy the ensure part, because if it’s not possible to get 
cryptographically secure bytes then the only option is to error if you want to 
be ensured of cryptographically secure bytes.

It’s a bit like if you did open(“somefile.txt”), it’s reasonable to say that we 
should ensure that open(“somefile.txt”) actually opens ./somefile.txt, and 
doesn’t randomly open a different 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Brett Cannon]

http://bugs.python.org/issue27288 covers updating the secrets module to use
getrandom().

http://bugs.python.org/issue27292 covers documenting the drawbacks of
os.urandom()

http://bugs.python.org/issue27293 covers documenting all of the issues
pointed out in this discussion.

Only issue I can think of that we're missing is one to track reverting
os.urandom() to 3.4 semantics (any doc updates required for the random
module?). Am I missing anything?

On Sat, Jun 11, 2016, 12:41 Guido van Rossum  wrote:

> On Sat, Jun 11, 2016 at 11:30 AM, Donald Stufft  wrote:
>
>>
>> On Jun 11, 2016, at 1:39 PM, Guido van Rossum  wrote:
>>
>> Is the feature detection desire about being able to write code that runs
>> on older Python versions or for platforms that just don't have getrandom()?
>>
>> My assumption was that nobody would actually use these flags except the
>> secrets module and people writing code that generates long-lived secrets --
>> and the latter category should be checking platform and versions anyway
>> since they need the whole stack to be secure (if I understand Ted Ts'o's
>> email right).
>>
>> My assumption is also that the flags should be hints (perhaps only
>> relevant on Linux) -- platforms that can't perform the action desired
>> (because their system's API doesn't support it) would just do their default
>> action, assuming the system API does the best it can.
>>
>>
>> The problem is that someone writing software that does
>> os.urandom(block=True) or os.urandom(exception=True) which gets some bytes
>> doesn’t know if it got back cryptographically secure random because Python
>> called getrandom() or if it got back cryptographically secure random
>> because it called /dev/urandom and that gave it secure random because it’s
>> on a platform that defines that as always returning secure or because it’s
>> on Linux and the urandom pool is initialized or if it got back some random
>> bytes that are not cryptographically secure because it fell back to reading
>> /dev/urandom on Linux prior to the pool being initialized.
>>
>> The “silently does the wrong thing, even though I explicitly asked for it
>> do something different” is something that I would consider to be a footgun
>> and footgun’s in security sensitive code make me really worried.
>>
>
> Yeah, but we've already established that there's a lot more upset,
> rhetoric and worry than warranted by the situation.
>
>
>> Outside of the security side of things, if someone goes “Ok I need some
>> random bytes and I need to make sure it doesn’t block”, then doing
>> ``os.random(block=False, exception=False)`` isn’t going to make sure that
>> it doesn’t block except on Linux.
>>
>
> To people who "just want some random bytes" we should recommend the random
> module.
>
>
>> In other words, it’s basically impossible to ensure you get the behavior
>> you want with these flags which I feel like will make everyone unhappy
>> (both the people who want to ensure non-blocking, and the people who want
>> to ensure cryptographically secure). These flags are an attractive nuisance
>> that look like they do the right thing, but silently don’t.
>>
>
> OK, it looks like the flags just won't make you happy, and I'm happy to
> give up on them. By default the status quo will win, and that means neither
> these flags nor os.getrandom(). (But of course you can roll your own using
> ctypes. :-)
>
>
>> Meanwhile if we have os.urandom that reads from /dev/urandom and
>> os.getrandom() which reads from blocking random, then we make it both
>> easier to ensure you get the behavior you want, either by using the
>> function that best suits your needs:
>>
>> * If you just want the best the OS has to offer, os.getrandom falling
>> back to os.urandom.
>>
>
> Actually the proposal for that was the secrets module. And the secrets
> module would be the only user of os.urandom(blocking=True).
>
>
>> * If you want to ensure you get cryptographically secure bytes,
>> os.getrandom, falling back to os.urandom on non Linux platforms and
>> erroring on Linux.
>>
>
> "Erroring" doesn't sound like it satisfies the "ensure" part of the
> requirement. And I don't see the advantage of os.getrandom() over the
> secrets module. (Either way you have to fall back on os.urandom() to
> suppport Python 3.5 and before.)
>
>
>> * If you want to *ensure* that there’s no blocking, then os.urandom on
>> Linux (or os.urandom wrapped with timeout code anywhere else, as that’s the
>> only way to ensure not blocking cross platform).
>>
>
> That's fine with me.
>
>
>> * If you just don’t care, YOLO it up with either os.urandom or
>> os.getrandom or random.random.
>>
>
> Now you're just taking the mickey.
>
>
>>
>> I think the problem with making os.urandom() go back to always reading
>> /dev/urandom is that we've come to rely on it on all platforms, so we've
>> passed that station.
>>
>>
>> Sorry, to be more specific I meant the 3.4 behavior, which was
>> 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/11/2016 11:30 AM, Donald Stufft wrote:
The problem is that someone writing software that does 
os.urandom(block=True) or os.urandom(exception=True) which gets some 
bytes doesn’t know if it got back cryptographically secure random 
because Python called getrandom() or if it got back cryptographically 
secure random because it called /dev/urandom and that gave it secure 
random because it’s on a platform that defines that as always 
returning secure or because it’s on Linux and the urandom pool is 
initialized or if it got back some random bytes that are not 
cryptographically secure because it fell back to reading /dev/urandom 
on Linux prior to the pool being initialized.


Let me jump in tangentially to say: I think os.urandom(block=True) is 
simply a bad API.  On FreeBSD and OpenBSD, /dev/urandom may block, and 
you don't have a choice.  On OS X, /dev/urandom will never block, and 
you don't have a choice.  In Victor's initial patch where he proposed 
it, the flag was accepted on all platforms but only affected its 
behavior on Linux and possibly Solaris.  I think it's bad API design to 
have a flag that seems like it would be meaningful on multiple 
platforms, but in practice is useful only in very limited 
circumstances.  If this were old code, or behavior we inherited from the 
platform and we were making the best of a bad situation, that'd be one 
thing.  But this is a proposed new API and I definitely think we can do 
better.


As I understand the proposed semantics for os.urandom(exception=True), I 
feel it falls into the same trap though not to the same degree.


Of course, both flags break backwards-compatibility if they default to 
True, and I strongly disagree with .


It's far better in my opinion to keep the os module as a thin shell over 
platform functionality.  That makes Python's behavior more predictable 
on a platform-by-platform basis.  So I think the best approach here is 
to add os.getrandom() as a thin shell over the local getrandom() (if any).



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

On Sat, Jun 11, 2016 at 11:30 AM, Donald Stufft  wrote:

>
> On Jun 11, 2016, at 1:39 PM, Guido van Rossum  wrote:
>
> Is the feature detection desire about being able to write code that runs
> on older Python versions or for platforms that just don't have getrandom()?
>
> My assumption was that nobody would actually use these flags except the
> secrets module and people writing code that generates long-lived secrets --
> and the latter category should be checking platform and versions anyway
> since they need the whole stack to be secure (if I understand Ted Ts'o's
> email right).
>
> My assumption is also that the flags should be hints (perhaps only
> relevant on Linux) -- platforms that can't perform the action desired
> (because their system's API doesn't support it) would just do their default
> action, assuming the system API does the best it can.
>
>
> The problem is that someone writing software that does
> os.urandom(block=True) or os.urandom(exception=True) which gets some bytes
> doesn’t know if it got back cryptographically secure random because Python
> called getrandom() or if it got back cryptographically secure random
> because it called /dev/urandom and that gave it secure random because it’s
> on a platform that defines that as always returning secure or because it’s
> on Linux and the urandom pool is initialized or if it got back some random
> bytes that are not cryptographically secure because it fell back to reading
> /dev/urandom on Linux prior to the pool being initialized.
>
> The “silently does the wrong thing, even though I explicitly asked for it
> do something different” is something that I would consider to be a footgun
> and footgun’s in security sensitive code make me really worried.
>

Yeah, but we've already established that there's a lot more upset, rhetoric
and worry than warranted by the situation.


> Outside of the security side of things, if someone goes “Ok I need some
> random bytes and I need to make sure it doesn’t block”, then doing
> ``os.random(block=False, exception=False)`` isn’t going to make sure that
> it doesn’t block except on Linux.
>

To people who "just want some random bytes" we should recommend the random
module.


> In other words, it’s basically impossible to ensure you get the behavior
> you want with these flags which I feel like will make everyone unhappy
> (both the people who want to ensure non-blocking, and the people who want
> to ensure cryptographically secure). These flags are an attractive nuisance
> that look like they do the right thing, but silently don’t.
>

OK, it looks like the flags just won't make you happy, and I'm happy to
give up on them. By default the status quo will win, and that means neither
these flags nor os.getrandom(). (But of course you can roll your own using
ctypes. :-)


> Meanwhile if we have os.urandom that reads from /dev/urandom and
> os.getrandom() which reads from blocking random, then we make it both
> easier to ensure you get the behavior you want, either by using the
> function that best suits your needs:
>
> * If you just want the best the OS has to offer, os.getrandom falling back
> to os.urandom.
>

Actually the proposal for that was the secrets module. And the secrets
module would be the only user of os.urandom(blocking=True).


> * If you want to ensure you get cryptographically secure bytes,
> os.getrandom, falling back to os.urandom on non Linux platforms and
> erroring on Linux.
>

"Erroring" doesn't sound like it satisfies the "ensure" part of the
requirement. And I don't see the advantage of os.getrandom() over the
secrets module. (Either way you have to fall back on os.urandom() to
suppport Python 3.5 and before.)


> * If you want to *ensure* that there’s no blocking, then os.urandom on
> Linux (or os.urandom wrapped with timeout code anywhere else, as that’s the
> only way to ensure not blocking cross platform).
>

That's fine with me.


> * If you just don’t care, YOLO it up with either os.urandom or
> os.getrandom or random.random.
>

Now you're just taking the mickey.


>
> I think the problem with making os.urandom() go back to always reading
> /dev/urandom is that we've come to rely on it on all platforms, so we've
> passed that station.
>
>
> Sorry, to be more specific I meant the 3.4 behavior, which was
> open(“/dev/urandom”).read() on *nix and CryptGenRandom on Windows.
>

I am all for keeping it that way. The secrets module doesn't have to use
any of these, it can use an undocumented extension module for all I care.
Or it can use os.urandom() and trust Ted Ts'o.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Brett Cannon]

On Sat, 11 Jun 2016 at 11:31 Donald Stufft  wrote:

> On Jun 11, 2016, at 1:39 PM, Guido van Rossum  wrote:
>
> Is the feature detection desire about being able to write code that runs
> on older Python versions or for platforms that just don't have getrandom()?
>
> My assumption was that nobody would actually use these flags except the
> secrets module and people writing code that generates long-lived secrets --
> and the latter category should be checking platform and versions anyway
> since they need the whole stack to be secure (if I understand Ted Ts'o's
> email right).
>
> My assumption is also that the flags should be hints (perhaps only
> relevant on Linux) -- platforms that can't perform the action desired
> (because their system's API doesn't support it) would just do their default
> action, assuming the system API does the best it can.
>
>
> The problem is that someone writing software that does
> os.urandom(block=True) or os.urandom(exception=True) which gets some bytes
> doesn’t know if it got back cryptographically secure random because Python
> called getrandom() or if it got back cryptographically secure random
> because it called /dev/urandom and that gave it secure random because it’s
> on a platform that defines that as always returning secure or because it’s
> on Linux and the urandom pool is initialized or if it got back some random
> bytes that are not cryptographically secure because it fell back to reading
> /dev/urandom on Linux prior to the pool being initialized.
>
> The “silently does the wrong thing, even though I explicitly asked for it
> do something different” is something that I would consider to be a footgun
> and footgun’s in security sensitive code make me really worried.
>
> Outside of the security side of things, if someone goes “Ok I need some
> random bytes and I need to make sure it doesn’t block”, then doing
> ``os.random(block=False, exception=False)`` isn’t going to make sure that
> it doesn’t block except on Linux.
>
> In other words, it’s basically impossible to ensure you get the behavior
> you want with these flags which I feel like will make everyone unhappy
> (both the people who want to ensure non-blocking, and the people who want
> to ensure cryptographically secure). These flags are an attractive nuisance
> that look like they do the right thing, but silently don’t.
>
> Meanwhile if we have os.urandom that reads from /dev/urandom and
> os.getrandom() which reads from blocking random, then we make it both
> easier to ensure you get the behavior you want, either by using the
> function that best suits your needs:
>
> * If you just want the best the OS has to offer, os.getrandom falling back
> to os.urandom.
> * If you want to ensure you get cryptographically secure bytes,
> os.getrandom, falling back to os.urandom on non Linux platforms and
> erroring on Linux.
> * If you want to *ensure* that there’s no blocking, then os.urandom on
> Linux (or os.urandom wrapped with timeout code anywhere else, as that’s the
> only way to ensure not blocking cross platform).
> * If you just don’t care, YOLO it up with either os.urandom or
> os.getrandom or random.random.
>

I'm +1 w/ what Donald is suggesting here and below w/ proper documentation
in both the secrets and random modules to explain when to use what (i.e.
secrets for crypto-no-matter-what randomness, random for quick-and-dirty
randomness). This also includes any appropriate decoupling of the secrets
module from the random module so there's no reliance on the random module
in the docs of the secrets module beyond "this class has the same
interface", and letting the secrets module be the way people generally get
crypto randomness.

-Brett


>
>
> I think the problem with making os.urandom() go back to always reading
> /dev/urandom is that we've come to rely on it on all platforms, so we've
> passed that station.
>
>
> Sorry, to be more specific I meant the 3.4 behavior, which was
> open(“/dev/urandom”).read() on *nix and CryptGenRandom on Windows.
>
>
> —
>
> Donald Stufft
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 1:39 PM, Guido van Rossum  wrote:
> 
> Is the feature detection desire about being able to write code that runs on 
> older Python versions or for platforms that just don't have getrandom()?
> 
> My assumption was that nobody would actually use these flags except the 
> secrets module and people writing code that generates long-lived secrets -- 
> and the latter category should be checking platform and versions anyway since 
> they need the whole stack to be secure (if I understand Ted Ts'o's email 
> right).
> 
> My assumption is also that the flags should be hints (perhaps only relevant 
> on Linux) -- platforms that can't perform the action desired (because their 
> system's API doesn't support it) would just do their default action, assuming 
> the system API does the best it can.

The problem is that someone writing software that does os.urandom(block=True) 
or os.urandom(exception=True) which gets some bytes doesn’t know if it got back 
cryptographically secure random because Python called getrandom() or if it got 
back cryptographically secure random because it called /dev/urandom and that 
gave it secure random because it’s on a platform that defines that as always 
returning secure or because it’s on Linux and the urandom pool is initialized 
or if it got back some random bytes that are not cryptographically secure 
because it fell back to reading /dev/urandom on Linux prior to the pool being 
initialized.

The “silently does the wrong thing, even though I explicitly asked for it do 
something different” is something that I would consider to be a footgun and 
footgun’s in security sensitive code make me really worried.

Outside of the security side of things, if someone goes “Ok I need some random 
bytes and I need to make sure it doesn’t block”, then doing 
``os.random(block=False, exception=False)`` isn’t going to make sure that it 
doesn’t block except on Linux.

In other words, it’s basically impossible to ensure you get the behavior you 
want with these flags which I feel like will make everyone unhappy (both the 
people who want to ensure non-blocking, and the people who want to ensure 
cryptographically secure). These flags are an attractive nuisance that look 
like they do the right thing, but silently don’t.

Meanwhile if we have os.urandom that reads from /dev/urandom and os.getrandom() 
which reads from blocking random, then we make it both easier to ensure you get 
the behavior you want, either by using the function that best suits your needs:

* If you just want the best the OS has to offer, os.getrandom falling back to 
os.urandom.
* If you want to ensure you get cryptographically secure bytes, os.getrandom, 
falling back to os.urandom on non Linux platforms and erroring on Linux.
* If you want to *ensure* that there’s no blocking, then os.urandom on Linux 
(or os.urandom wrapped with timeout code anywhere else, as that’s the only way 
to ensure not blocking cross platform).
* If you just don’t care, YOLO it up with either os.urandom or os.getrandom or 
random.random.

> 
> I think the problem with making os.urandom() go back to always reading 
> /dev/urandom is that we've come to rely on it on all platforms, so we've 
> passed that station.
> 

Sorry, to be more specific I meant the 3.4 behavior, which was 
open(“/dev/urandom”).read() on *nix and CryptGenRandom on Windows.


—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

You can add me to the list of people who feel like disappearing.

On Sat, Jun 11, 2016 at 10:28 AM, Terry Reedy  wrote:

> On 6/11/2016 11:34 AM, Guido van Rossum wrote:
>
>> In terms of API design, I'd prefer a flag to os.urandom() indicating a
>> preference for
>> - blocking
>> - raising an exception
>> - weaker random bits
>>
>
> +100 ;-)
>
> I proposed exactly this 2 days ago, 5 hours after Larry's initial post.
>
> '''
> I think the 'new API' should be a parameter, not a new function. With just
> two choices, 'wait' = True/False  could work.  If 'raise an exception' were
> added, then
> 'action (when good bits are not immediately available' =
> 'return (best possible)' or
> 'wait (until have good bits)' or
> 'raise (CryptBitsNotAvailable)'
>
> In either case, there would then be the question of whether the default
> should match 3.5.0/1 or 3.4 and before.
> '''
>
> Deciding on this then might have saved some hurt feelings, to the point
> where two contributors feel like disappearing, and a release manager must
> feel the same.  In any case, Guido already picked 3.4 behavior as the
> default.  Can we agree and move on?
>
> --
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

Is the feature detection desire about being able to write code that runs on
older Python versions or for platforms that just don't have getrandom()?

My assumption was that nobody would actually use these flags except the
secrets module and people writing code that generates long-lived secrets --
and the latter category should be checking platform and versions anyway
since they need the whole stack to be secure (if I understand Ted Ts'o's
email right).

My assumption is also that the flags should be hints (perhaps only relevant
on Linux) -- platforms that can't perform the action desired (because their
system's API doesn't support it) would just do their default action,
assuming the system API does the best it can.

I think the problem with making os.urandom() go back to always reading
/dev/urandom is that we've come to rely on it on all platforms, so we've
passed that station.

On Sat, Jun 11, 2016 at 10:15 AM, Donald Stufft  wrote:

>
> On Jun 11, 2016, at 11:34 AM, Guido van Rossum  wrote:
>
> In terms of API design, I'd prefer a flag to os.urandom() indicating a
> preference for
> - blocking
> - raising an exception
> - weaker random bits
>
>
> If os.urandom can’t block on Linux, then I feel like it’d be saner to add
> os.getrandom(). I feel like these flags are going to confuse people,
> particularly when you take into account that all 3 of them are only going
> to really matter on Linux (and particularly on newer Linux) and for things
> like “blocking” it’s going to get confused with the blocking that
> /dev/random does on Linux.
>
> Right now there are two ways to access the system CSPRNG on *nix, there is
> /dev/urandom pretty much always, and then there is getrandom() (or
> arc4random, etc, depending on the specific OS you’re on).
>
> Perhaps the right answer is to go back to making os.urandom always
> open(“/dev/urandom”).read() instead of trying to save a FD by using
> getrandom() and just add os.getrandom() which will interface with
> getrandom()/arc4random()/etc and always in blocking mode. Why always in
> blocking mode? Because it’s the only way to get consistent behavior across
> different platforms, all non Linux OSs either block or they otherwise
> ensure that it is initialized prior to it even being possible to access the
> CSPRNG.
>
> Using this, code can be smarter about what to do in edge cases than we can
> reasonably be in os.urandom, for example see
> https://bpaste.net/show/41d89e520913.
>
> The reasons I think this is preferable to adding parameters to os.urandom
> are:
>
> * If we add parameters to os.urandom, you can’t feature detect their
> existence easily, you have to use version checks.
> * With flags, unless we add even more flags we can’t dictate what should
> happen if we’re on a system where the person’s desired preference can’t be
> satisfied. We either have to just silently do something that may be wrong,
> or add more flags. By adding two functions people can pick which of the
> following they want with some programming (see example):
>
> * Just try to get the strongest random, but fall back to maybe not
> random if it’s early enough in boot process.
> * Fail on old Linux rather than possibly get insecure random.
> * Actually write cross platform code to prevent blocking (since only
> Linux allows you to not block)
> * Fail hard rather than block if we can’t get secure random bytes
> without blocking.
> * Soft fail and get “probably good enough” random from os.urandom
> on Linux.
> * Hard fail on non Linux if we would block since there’s no
> non-blocking and “probably good enough” interface.
> * Soft fail and get “probably good enough” random from os.urandom
> on Linux, and use time/pid/memory offsets on non Linux.
> * Just use the best source of random available to use on the system,
> and block rather than fail.
>
> I don’t see any way to get the same wide set of options by just adding
> flags to os.urandom unless we add flags that work for every possible
> combination of what people may or may not want to.
>
> —
> Donald Stufft
>
>
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Terry Reedy]

On 6/11/2016 11:34 AM, Guido van Rossum wrote:

In terms of API design, I'd prefer a flag to os.urandom() indicating a
preference for
- blocking
- raising an exception
- weaker random bits


+100 ;-)

I proposed exactly this 2 days ago, 5 hours after Larry's initial post.

'''
I think the 'new API' should be a parameter, not a new function. With 
just two choices, 'wait' = True/False  could work.  If 'raise an 
exception' were added, then

'action (when good bits are not immediately available' =
'return (best possible)' or
'wait (until have good bits)' or
'raise (CryptBitsNotAvailable)'

In either case, there would then be the question of whether the default 
should match 3.5.0/1 or 3.4 and before.

'''

Deciding on this then might have saved some hurt feelings, to the point 
where two contributors feel like disappearing, and a release manager 
must feel the same.  In any case, Guido already picked 3.4 behavior as 
the default.  Can we agree and move on?


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 11, 2016, at 11:34 AM, Guido van Rossum  wrote:
> 
> In terms of API design, I'd prefer a flag to os.urandom() indicating a 
> preference for
> - blocking
> - raising an exception
> - weaker random bits

If os.urandom can’t block on Linux, then I feel like it’d be saner to add 
os.getrandom(). I feel like these flags are going to confuse people, 
particularly when you take into account that all 3 of them are only going to 
really matter on Linux (and particularly on newer Linux) and for things like 
“blocking” it’s going to get confused with the blocking that /dev/random does 
on Linux.

Right now there are two ways to access the system CSPRNG on *nix, there is 
/dev/urandom pretty much always, and then there is getrandom() (or arc4random, 
etc, depending on the specific OS you’re on). 

Perhaps the right answer is to go back to making os.urandom always 
open(“/dev/urandom”).read() instead of trying to save a FD by using getrandom() 
and just add os.getrandom() which will interface with 
getrandom()/arc4random()/etc and always in blocking mode. Why always in 
blocking mode? Because it’s the only way to get consistent behavior across 
different platforms, all non Linux OSs either block or they otherwise ensure 
that it is initialized prior to it even being possible to access the CSPRNG.

Using this, code can be smarter about what to do in edge cases than we can 
reasonably be in os.urandom, for example see 
https://bpaste.net/show/41d89e520913 .

The reasons I think this is preferable to adding parameters to os.urandom are:

* If we add parameters to os.urandom, you can’t feature detect their existence 
easily, you have to use version checks.
* With flags, unless we add even more flags we can’t dictate what should happen 
if we’re on a system where the person’s desired preference can’t be satisfied. 
We either have to just silently do something that may be wrong, or add more 
flags. By adding two functions people can pick which of the following they want 
with some programming (see example):

* Just try to get the strongest random, but fall back to maybe not random 
if it’s early enough in boot process.
* Fail on old Linux rather than possibly get insecure random.
* Actually write cross platform code to prevent blocking (since only Linux 
allows you to not block)
* Fail hard rather than block if we can’t get secure random bytes 
without blocking.
* Soft fail and get “probably good enough” random from os.urandom on 
Linux.
* Hard fail on non Linux if we would block since there’s no 
non-blocking and “probably good enough” interface.
* Soft fail and get “probably good enough” random from os.urandom on 
Linux, and use time/pid/memory offsets on non Linux.
* Just use the best source of random available to use on the system, and 
block rather than fail.

I don’t see any way to get the same wide set of options by just adding flags to 
os.urandom unless we add flags that work for every possible combination of what 
people may or may not want to.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

In terms of API design, I'd prefer a flag to os.urandom() indicating a
preference for
- blocking
- raising an exception
- weaker random bits

To those still upset by the decision, please read Ted Ts'o's message.

On Saturday, June 11, 2016, Steven D'Aprano  wrote:

> On Thu, Jun 09, 2016 at 07:52:31PM -0700, Nikolaus Rath wrote:
> > On Jun 09 2016, Guido van Rossum >
> wrote:
> > > I don't think we should add a new function. I think we should convince
> > > ourselves that there is not enough of a risk of an exploit even if
> > > os.urandom() falls back.
> >
> > That will be hard, because you have to consider an active, clever
> > adversary.
>
> We know that there are exploitable bugs from Linux systems due to
> urandom, e.g. the Raspberry Pi bug referenced elsewhere in this thread.
>
> https://www.raspberrypi.org/forums/viewtopic.php?f=66=126892
>
>
> > On the other hand, convincing yourself that in practice os.urandom would
> > never block unless the setup is super exotic or there is active
> > maliciousness seems much easier.
>
> Not that super exotic. In my day job, I've seen processes hang for five
> or ten minutes during boot up, waiting for the OS to collect enough
> entropy, although this was not recently and it wasn't involving Python.
> But VMs or embedded devices may take a long time to generate entropy. If
> the device doesn't have a hardware source of randomness, and isn't
> connected to an external source of noise like networking or a user who
> habitually fiddles with the mouse, it might take a very long time indeed
> to gather entropy...
>
> If I have understood the concensus, I think we're on the right track:
>
> (1) os.urandom should do whatever the OS says it should do, which on
> Linux is fall back on pseudo-random bytes when the entropy pool hasn't
> be initialised yet. It won't block and won't raise.
>
> (2) os.getrandom will be added to 3.6, and it will block, or possibly
> raise, whichever the caller specifies.
>
> (3) The secrets module in 3.6 will stop relying on os.urandom, and use
> os.getrandom. It may provide a switch to choose between blocking and
> non-blocking (raise an exception) behaviour. It WON'T fall back to
> predictable non-crypto bytes (unless the OS itself is completely
> broken).
>
> (4) random will continue to seed itself from os.urandom, because it
> doesn't care if urandom provides degraded randomness. It just needs to
> be better than using the time as seed.
>
> (5) What about random.SysRandom? I think it should use os.getrandom.
>
> (6) A bunch of stuff will happen to make the hash randomisation not
> break when systemd runs Python scripts early in the boot process, but I
> haven't been paying attention to that part :-)
>
> Is this a good summary of where we are at?
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Christian Heimes]

On 2016-06-11 16:37, Victor Stinner wrote:
>> I  repeat for like the fifth time:
> 
> So, is there a candidate to write a PEP?
> 
> I didn't read the thread. As expected, the discussion restarted for the
> 3rd time, there are almost 100 emails in this thread.

Sorry, I'm out.

I simply lack the necessary strength and mental energy to persuade the
issue any further. Donald Stufft just forwarded a quote that resonates
with my current state of mind (replace 'lists' with 'current topic'):

"I feel I no longer possess either the necessary strength or perhaps
the necessary faith to continue rolling the stone of Sisyphus against
the forces of reaction which are triumphing everywhere. I am therefore
retiring from the lists, and ask if my dear contemporaries only one
thing — oblivion."

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 11 Jun 2016, at 09:24, Larry Hastings  wrote:
> Only Linux and OS X have never-blocking /dev/urandom.  On Linux, you can 
> choose to block by calling getrandom().  On OS X you have no choice, you can 
> only use the never-blocking /dev/urandom.  (OS X also has a /dev/random but 
> it behaves identically to /dev/urandom.)  OS X's man page reassuringly claims 
> blocking is never necessary; the blogosphere disagrees.
> If I were writing the function for the secrets module, I'd write it like you 
> have above: call os.getrandom() if it's present, and os.urandom() if it 
> isn't.  I believe that achieves current-best-practice everywhere: it does the 
> right thing on Linux, it does the right thing on Solaris, it does the right 
> thing on all the other OSes where reading from /dev/urandom can block, and it 
> uses the only facility available to us on OS X.

Sorry Larry, but as far as I know this is misleading (it’s not *wrong*, but it 
suggests that OS X’s /dev/urandom is the same as Linux’s, which is emphatically 
not true).

I’ve found the discussion around OS X’s random devices to be weirdly abstract, 
given that the source code for it is public, so I went and took a look. My 
initial reading of it (and, to be clear, this is a high-level read of a 
codebase I don’t know well, so please take this with the grain of salt that is 
intended) is that the operating system literally will not boot without at least 
128 bits of entropy to read from the EFI boot loader. In the absence of 128 
bits of entropy the kernel will panic, rather than continue to boot.

Generally speaking that entropy will come from RDRAND, given the restrictions 
on where OS X can be run (Intel CPUs for real OS X, virtualised on top of OS X, 
and so on top of Intel CPUs, for VMs), which imposes a baseline on the quality 
of the entropy you can get. Assuming that OS X is being run in a manner that is 
acceptable from the perspective of its license agreement (and we can all agree 
that no-one would violate the terms of OS X’s license agreement, right?), I 
think it’s reasonable to assume that OS X, either virtualised or not, is 
getting 128 bits of somewhat sensible entropy from the boot loader/CPU before 
it boots.

That means we can say this about OS X’s /dev/urandom: the reason it never 
blocks is because the situation of “not enough entropy to generate good random 
numbers” is synonymous with “not enough entropy to boot the OS”. So maybe we 
can stop casting aspersions on OS X’s RNG now.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Victor Stinner]

> I  repeat for like the fifth time:

So, is there a candidate to write a PEP?

I didn't read the thread. As expected, the discussion restarted for the 3rd
time, there are almost 100 emails in this thread.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Christian Heimes]

On 2016-06-10 20:42, Chris Jerdonek wrote:
> On Fri, Jun 10, 2016 at 11:29 AM, David Mertz  wrote:
>> This is fairly academic, since I do not anticipate needing to do this
>> myself, but I have a specific question.  I'll assume that Python 3.5.2 will
>> go back to the 2.6-3.4 behavior in which os.urandom() never blocks on Linux.
>> Moreover, I understand that the case where the insecure bits might be
>> returned are limited to Python scripts that run on system initialization on
>> Linux.
>>
>> If I *were* someone who needed to write a Linux system initialization script
>> using Python 3.5.2, what would the code look like.  I think for this use
>> case, requiring something with a little bit of "code smell" is fine, but I
>> kinda hope it exists at all.
> 
> Good question.  And going back to Larry's original e-mail, where he said--
> 
> On Thu, Jun 9, 2016 at 4:25 AM, Larry Hastings  wrote:
>> THE PROBLEM
>> ...
>> The issue author had already identified the cause: CPython was blocking on
>> getrandom() in order to initialize hash randomization.  On this fresh
>> virtual machine the entropy pool started out uninitialized.  And since the
>> only thing running on the machine was CPython, and since CPython was blocked
>> on initialization, the entropy pool was initializing very, very slowly.

I repeat for like the fifth time:

os.urandom() and Python startup are totally unrelated. They just happen
to use the same internal function to set the hash randomization state.
The startup problem can be solved without f... up the security
properties of os.urandom().


The correct questions to ask are:

1) Does hash randomization for bytes, text and XML always require
cryptographically strong random values from a potentially blocking CPRNG?

2) Does the initial state of the Mersenne-Twister of the default
random.Random instance really need cryptographically strong values?

3) Should os.urandom() always use the best CSPRNG source available and
make sure it never returns weak, predictable values (when possible)?

The answers are:

1) No
2) No
3) HELL YES!

If you think that the answer to 3 is "No" and that a CSPRNG is permitted
to return predictable values, then you are *by definition* ineligible to
vote on security issues.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Sebastian Krause]

Steven D'Aprano  wrote:
>> it seems to me that you'd want such a solution to have code that
>> causes the initialization of the entropy pool to be sped up so that it
>> happens as quickly as possible (if that is even possible).  Is it
>> possible? (E.g. by causing the machine to start doing things other
>> than just CPython?)
>
> I don't think that's something which the Python interpreter ought to do 
> for you, but you can write to /dev/urandom or /dev/random (both keep 
> their own, separate, entropy pools):

There are projects like http://www.issihosts.com/haveged/ that use
some tiny timing fluctuations in CPUs to feed the entropy pool and
which are available in most Linux distributions. But as you said,
that is something completely outside of Python's scope.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Fri, Jun 10, 2016 at 11:42:40AM -0700, Chris Jerdonek wrote:

> And going back to Larry's original e-mail, where he said--
> 
> On Thu, Jun 9, 2016 at 4:25 AM, Larry Hastings  wrote:
> > THE PROBLEM
> > ...
> > The issue author had already identified the cause: CPython was blocking on
> > getrandom() in order to initialize hash randomization.  On this fresh
> > virtual machine the entropy pool started out uninitialized.  And since the
> > only thing running on the machine was CPython, and since CPython was blocked
> > on initialization, the entropy pool was initializing very, very slowly.
> 
> it seems to me that you'd want such a solution to have code that
> causes the initialization of the entropy pool to be sped up so that it
> happens as quickly as possible (if that is even possible).  Is it
> possible? (E.g. by causing the machine to start doing things other
> than just CPython?)

I don't think that's something which the Python interpreter ought to do 
for you, but you can write to /dev/urandom or /dev/random (both keep 
their own, separate, entropy pools):

open("/dev/urandom", "w").write("hello world")

But of course there's the question of where you're going to get a source 
of noise to write to the file. While it's (probably?) harmless to write 
a hard-coded string to it, I don't think its going to give you much 
entropy.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/11/2016 12:49 AM, Steven D'Aprano wrote:

Will there be platforms where os.getrandom doesn't exist? If not, then
secrets can just rely on it, otherwise what should it do?

if hasattr(os, 'getrandom'):
 return os.getrandom(n)
else:
 # Fail? Fall back on os.urandom?


AFAIK:

 * Only Linux and Solaris have getrandom() right now.  IIUC Solaris
   duplicated Linux's API, but I don't know that for certain, and I
   don't know in particular what GRND_RANDOM does on Solaris.  (Of
   course, you don't need GRND_RANDOM for secrets.token_bytes().)
 * Only Linux and OS X have never-blocking /dev/urandom.  On Linux, you
   can choose to block by calling getrandom().  On OS X you have no
   choice, you can only use the never-blocking /dev/urandom.  (OS X
   also has a /dev/random but it behaves identically to /dev/urandom.) 
   OS X's man page reassuringly claims blocking is never necessary; the

   blogosphere disagrees.

If I were writing the function for the secrets module, I'd write it like 
you have above: call os.getrandom() if it's present, and os.urandom() if 
it isn't.  I believe that achieves current-best-practice everywhere: it 
does the right thing on Linux, it does the right thing on Solaris, it 
does the right thing on all the other OSes where reading from 
/dev/urandom can block, and it uses the only facility available to us on 
OS X.



//arry/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Fri, Jun 10, 2016 at 01:06:45PM -0700, Larry Hastings wrote:
> 
> On 06/10/2016 01:01 PM, David Mertz wrote:
> >So yes, I think 3.5.2 should restore the 2.6-3.4 behavior of os.urandom(),
> 
> That makes... five of us I think ;-) (Larry Guido Barry Tim David)
> 
> 
> >and the NEW APIs in secrets should use the "best available randomness 
> >(even if it blocks)"
> 
> I'm not particular about how the new API is spelled.  However, I do 
> think os.getrandom() should be exposed as a thin wrapper over 
> getrandom() in 3.6.   That would permit Python programmers to take 
> maximal advantage of the features offered by their platform.  It would 
> also permit the secrets module to continue to be written in pure Python.

A big +1 for that.

Will there be platforms where os.getrandom doesn't exist? If not, then 
secrets can just rely on it, otherwise what should it do?

if hasattr(os, 'getrandom'):
return os.getrandom(n)
else:
# Fail? Fall back on os.urandom?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Thu, Jun 09, 2016 at 07:52:31PM -0700, Nikolaus Rath wrote:
> On Jun 09 2016, Guido van Rossum  wrote:
> > I don't think we should add a new function. I think we should convince
> > ourselves that there is not enough of a risk of an exploit even if
> > os.urandom() falls back.
> 
> That will be hard, because you have to consider an active, clever
> adversary.

We know that there are exploitable bugs from Linux systems due to 
urandom, e.g. the Raspberry Pi bug referenced elsewhere in this thread.

https://www.raspberrypi.org/forums/viewtopic.php?f=66=126892


> On the other hand, convincing yourself that in practice os.urandom would
> never block unless the setup is super exotic or there is active
> maliciousness seems much easier.

Not that super exotic. In my day job, I've seen processes hang for five 
or ten minutes during boot up, waiting for the OS to collect enough 
entropy, although this was not recently and it wasn't involving Python. 
But VMs or embedded devices may take a long time to generate entropy. If 
the device doesn't have a hardware source of randomness, and isn't 
connected to an external source of noise like networking or a user who 
habitually fiddles with the mouse, it might take a very long time indeed 
to gather entropy...

If I have understood the concensus, I think we're on the right track:

(1) os.urandom should do whatever the OS says it should do, which on 
Linux is fall back on pseudo-random bytes when the entropy pool hasn't 
be initialised yet. It won't block and won't raise.

(2) os.getrandom will be added to 3.6, and it will block, or possibly 
raise, whichever the caller specifies.

(3) The secrets module in 3.6 will stop relying on os.urandom, and use 
os.getrandom. It may provide a switch to choose between blocking and 
non-blocking (raise an exception) behaviour. It WON'T fall back to 
predictable non-crypto bytes (unless the OS itself is completely 
broken).

(4) random will continue to seed itself from os.urandom, because it 
doesn't care if urandom provides degraded randomness. It just needs to 
be better than using the time as seed.

(5) What about random.SysRandom? I think it should use os.getrandom.

(6) A bunch of stuff will happen to make the hash randomisation not 
break when systemd runs Python scripts early in the boot process, but I 
haven't been paying attention to that part :-)

Is this a good summary of where we are at?



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 10, 2016, at 5:21 PM, Tim Peters  wrote:
> 
> Isn't that precisely the purpose of the GRND_NONBLOCK flag?


It doesn’t behave exactly the same as /dev/urandom. If the pool hasn’t been 
initialized yet /dev/urandom will return possibly predictable data whereas 
getrandom(GRND_NONBLOCK) will EAGAIN.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Tim Peters]

[Random832]
> So, I have a question. If this "weakness" in /dev/urandom is so
> unimportant to 99% of situations... why isn't there a flag that can be
> passed to getrandom() to allow the same behavior?

Isn't that precisely the purpose of the GRND_NONBLOCK flag?

http://man7.org/linux/man-pages/man2/getrandom.2.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Random832]

On Fri, Jun 10, 2016, at 15:54, Theodore Ts'o wrote:
> So even on Python pre-3.5.0, realistically speaking, the "weakness" of
> os.random would only be an issue (a) if it is run within the first few
> seconds of boot, and (b) os.random is used to directly generate a
> long-term cryptographic secret.  If you are fork openssl or ssh-keygen
> to generate a public/private keypair, then you aren't using os.random.

So, I have a question. If this "weakness" in /dev/urandom is so
unimportant to 99% of situations... why isn't there a flag that can be
passed to getrandom() to allow the same behavior?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/10/2016 12:54 PM, Theodore Ts'o wrote:

So even on Python pre-3.5.0, realistically speaking, the "weakness" of
os.random would only be an issue (a) if it is run within the first few
seconds of boot, and (b) os.random is used to directly generate a
long-term cryptographic secret.  If you are fork openssl or ssh-keygen
to generate a public/private keypair, then you aren't using os.random.


Just a gentle correction: wherever Mr. Ts'o says "os.random", he means 
"os.urandom()".  We don't have an "os.random" in Python.


My thanks to today's celebrity guest correspondent, Mr. Theodore Ts'o!


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[M.-A. Lemburg]

On 10.06.2016 20:55, Donald Stufft wrote:
> Ok, so you’re looking for how would you replicate the blocking behavior of 
> os.urandom that exists in 3.5.0 and 3.5.1?
> 
> In that case, it’s hard. I don’t think linux provides any way to externally 
> determine if /dev/urandom has been initialized or not. Probably the easiest 
> thing to do would be to interface with the getrandom() function using a 
> c-ext, CFFI, or ctypes. If you’re looking for a way of doing this without 
> calling the getrandom() function.. I believe the answer is you can’t.

Well, you can see the effect by running Python early in the boot process.

See e.g. http://bugs.python.org/issue26839#msg267749

and if you look at the system log file, you'll find a notice
entry "random: %s pool is initialized" which gets written once the
pool is initialized:

http://lxr.free-electrons.com/source/drivers/char/random.c#L684

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jun 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Theodore Ts'o]

I will observe that feelings have gotten a little heated, so without
making any suggestions to how the python-dev community should decide
things, let me offer some observations that might perhaps shed a
little light, and perhaps dispell a little bit of the heat.

As someone who has been working in security for a long time --- before
I started getting paid to hack Linux full-time, worked on Kerberos,
was on the Security Area Directorate of the IETF, where among other
things I was one of the working group chairs for the IP Security
(ipsec) working group --- I tend to cringe a bit when people talk
about security in terms of absolutes.  For example, the phrase
"improving Python's security".  Security is something that is best
talked about given a specific threat environment, where the value of
what you are trying to protect, the capabilities and resources of the
attackers, etc., are all well known.

This gets hard for those of us who work on infrastructure which can
get used in many different arenas, and so that's something that
applies both to the Linux Kernel and to C-Python, because how people
will use the tools that we spend so much of our passion crafting is
largely out of our control, and we may not even know how they are
using it.

As far as /dev/urandom is concerned, it's true that it doesn't block
before it has been initialized.  If you are a security academic who
likes to write papers about how great you are at finding defects in
other people's work.  This is definitely a weakness.

Is it a fatal weakness?  Well, first of all, on most server and
desktop deployments, we save 1 kilobyte or so of /dev/urandom output
during the shutdown sequence, and immediately after the init scripts
are completed.  This saved entropy is then piped back into /dev/random
infrastructure and used initialized /dev/random and /dev/urandom very
early in the init scripts.  On a freshly instaled machine, this won't
help, true, but in practice, on most systems, /dev/urandom will get
initialized from interrupt timing sampling within a few seconds after
boot.  For example, on a sample Google Compute Engine VM which is
booted into Debian and then left idle, /dev/urandom was initialized
within 2.8 seconds after boot, while the root file system was
remounted read-only 1.6 seconds after boot.

So even on Python pre-3.5.0, realistically speaking, the "weakness" of
os.random would only be an issue (a) if it is run within the first few
seconds of boot, and (b) os.random is used to directly generate a
long-term cryptographic secret.  If you are fork openssl or ssh-keygen
to generate a public/private keypair, then you aren't using os.random.

Furthermore, if you are running on a modern x86 system with RDRAND,
you'll also be fine, because we mix in randomness from the CPU chip
via the RDRAND instruction.

So this whole question of whether os.random should block *is*
important in certain very specific cases, and if you are generating
long-term cryptogaphic secrets in Python, maybe you should be worrying
about that.  But to be honest, there are lots of other things you
should be worrying about as well, and I would hope that people writing
cryptographic code would be asking questions of how the random nunmber
stack is working, not just at the C-Python interpretor level, but also
at the OS level.

My preference would be that os.random should block, because the odds
that people would be trying to generate long-term cryptographic
secrets within seconds after boot is very small, and if you *do* block
for a second or two, it's not the end of the world.  The problem that
triggered this was specifically because systemd was trying to use
C-Python very early in the boot process to initialize the SIPHASH used
for the dictionary, and it's not clear that really needed to be
extremely strong because it wasn't a long-term cryptogaphic secret ---
certainly not how systemd was using that specific script!

The reason why I think blocking is better is that once you've solved
the "don't hang the VM for 90 seconds until python has started up",
someone who is using os.random will almost certainly not be on the
blocking path of the system boot sequence, and so blocking for 2
seconds before generating a long-term cryptographic secret is not the
end of the world.

And if it does block by accident, in a security critical scenario it
will hopefully force the progammer to think, and and in a non-security
critical scenario, it should be easy to switch to either a totally
non-blocking interface, or switch to a pseudo-random interface hwich
is more efficient.


*HOWEVER*, on the flip side, if os.random *doesn't* block, in 99.999%
percent of the cases, the python script that is directly generating a
long-term secret will not be started 1.2 seconds after the root file
system is remounted read/write, so it is *also* not the end of the
world.  Realistically speaking, we do know which processes are likely
to be generating long-term cryptographic secrets imnmediately after

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 10, 2016, at 12:05 PM, David Mertz wrote:

>OK.  My understanding is that Guido ruled out introducing an os.getrandom()
>API in 3.5.2.  But would you be happy if that interface is added to 3.6?

I would.

>It feels to me like the correct spelling in 3.6 should probably be
>secrets.getrandom() or something related to that.

ISTM that secrets is a somewhat higher level API while it makes sense that a
fairly simple plumbing of the underlying C call should go in os.  But I
wouldn't argue much if folks had strong opinions to the contrary.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/10/2016 01:01 PM, David Mertz wrote:

So yes, I think 3.5.2 should restore the 2.6-3.4 behavior of os.urandom(),


That makes... five of us I think ;-) (Larry Guido Barry Tim David)


and the NEW APIs in secrets should use the "best available randomness 
(even if it blocks)"


I'm not particular about how the new API is spelled.  However, I do 
think os.getrandom() should be exposed as a thin wrapper over 
getrandom() in 3.6.   That would permit Python programmers to take 
maximal advantage of the features offered by their platform.  It would 
also permit the secrets module to continue to be written in pure Python.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Tim Peters]

[Tim]
>> secrets.token_bytes() is already the way to spell "get a string of
>> messed-up bytes", and that's the dead obvious (according to me) place
>> to add the potentially blocking implementation.

[Sebastian Krause]
> I honestly didn't think that this was the dead obvious function to
> use. To me the naming kind of suggested that it would do some
> special magic that tokens needed, instead of just returning random
> bytes (even though the best token is probably just perfectly random
> data). If you want to provide a general function for secure random
> bytes I would suggest at least a better naming.

There was ample bikeshedding over the names of `secrets` functions at
the time.  If token_bytes wasn't the obvious function to you, I
suspect you have scant idea what _is_ in the `secrets` module.   The
naming is logical in context, where various "token_xxx" functions
supply random-ish bytes in different formats.  In that context,
xxx=bytes is the obvious way to get raw bytes.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/10/2016 11:55 AM, Donald Stufft wrote:
Ok, so you’re looking for how would you replicate the blocking 
behavior of os.urandom that exists in 3.5.0 and 3.5.1?


In that case, it’s hard. I don’t think linux provides any way to 
externally determine if /dev/urandom has been initialized or not. 
Probably the easiest thing to do would be to interface with the 
getrandom() function using a c-ext, CFFI, or ctypes. If you’re looking 
for a way of doing this without calling the getrandom() function.. I 
believe the answer is you can’t.


I'm certain you're correct: you can't perform any operation on 
/dev/urandom to determine whether or not the urandom device has been 
initialized.  That's one of the reasons why Mr. Ts'o added 
getrandom()--you can use it to test exactly that (getrandom(GRND_NONBLOCK)).


That's also why I proposed adding os.getrandom() in 3.5.2, to make it 
possible to block until urandom was initialized (without using ctypes 
etc as you suggest).  However, none of the cryptography guys jumped up 
and said they wanted it, and in any case it was overruled by Guido, so 
we're not adding it to 3.5.2.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[David Mertz]

On Fri, Jun 10, 2016 at 12:55 PM, Larry Hastings  wrote:

> On 06/10/2016 12:29 PM, David Mertz wrote:
>
> I believe that secrets.token_bytes() and secrets.SystemRandom() should be
> changed even for 3.5.1 to use getrandom() on Linux.
>
> Surely you meant 3.5.2?  3.5.1 shipped last December.
>

Yeah, that combines a couple thinkos even.  I had intended to write "for
3.5.2" ... but that is also an error, since the secrets module doesn't
exist until 3.6.  So yes, I think 3.5.2 should restore the 2.6-3.4 behavior
of os.urandom(), and the NEW APIs in secrets should use the "best available
randomness (even if it blocks)"

Donald is correct that we have the spelling secrets.token_bytes() available
in 3.6a1, so the spellings secrets.getrandom() or secrets.randbytes() are
not needed.  However, Sebastian's (adapted) suggestion to allow
secrets.token_bytes(k,
*, nonblock=False) as the signature makes sense to me (i.e. it's a choice
of "block or raise exception", not an option to get non-crypto bytes).

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 10, 2016, at 3:33 PM, Brett Cannon  wrote:
> 
> If that's the case then we should file a bug so we are sure this is the case 
> and we need to decouple the secrets documentation from random so that they 
> can operate independently with secrets always doing whatever is required to 
> be as secure as possible.


https://bugs.python.org/issue27288 

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Sebastian Krause]

Tim Peters  wrote:
> secrets.token_bytes() is already the way to spell "get a string of
> messed-up bytes", and that's the dead obvious (according to me) place
> to add the potentially blocking implementation.

I honestly didn't think that this was the dead obvious function to
use. To me the naming kind of suggested that it would do some
special magic that tokens needed, instead of just returning random
bytes (even though the best token is probably just perfectly random
data). If you want to provide a general function for secure random
bytes I would suggest at least a better naming.

Sebastian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/10/2016 12:29 PM, David Mertz wrote:
I believe that secrets.token_bytes() and secrets.SystemRandom() should 
be changed even for 3.5.1 to use getrandom() on Linux.


Surely you meant 3.5.2?  3.5.1 shipped last December.


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Sven R. Kunze]

On 10.06.2016 21:17, Donald Stufft wrote:


On Jun 10, 2016, at 3:05 PM, David Mertz > wrote:


OK.  My understanding is that Guido ruled out introducing an 
os.getrandom() API in 3.5.2.  But would you be happy if that 
interface is added to 3.6?


It feels to me like the correct spelling in 3.6 should probably be 
secrets.getrandom() or something related to that.




I am not a security expert but your reply makes it clear to me. So, for 
me this makes:


os -> os-dependent and because of this varies from os to os (also 
quality-wise)

random -> pseudo-random, but it works for most non-critical use-cases
secret -> that's for crypto


If don't need crypto, secret would be a waste of resources, but if you 
need crypto, then os and random are unsafe. I think that's simple 
enough. At least, I would understand it.



Just my 2 cents: if I need crypto, I would pay the price of blocking 
rather then to get an exception (what are my alternatives? I need those 
bits! ) or get unsecure bits.



Sven


Well we have 
https://docs.python.org/dev/library/secrets.html#secrets.token_bytes so adding 
a getrandom() function to secrets would largely be the same as that 
function.


The problem of course is that the secrets library in 3.6 uses 
os.urandom under the covers, so it’s security rests on the security of 
os.urandom. To ensure that the secrets library is actually safe even 
in early boot it’ll need to stop using os.urandom on Linux and use the 
getrandom() function.


That same library exposes random.SystemRandom as secrets.SystemRandom 
[1], and of course SystemRandom uses os.urandom too. So if we want 
people to treat secrets.SystemRandom as “always secure” then it would 
need to stop using os.urandom and start using the get random() 
function on Linux as well.



[1] This is actually documented as "using the highest-quality sources 
provided by the operating system” in the secrets documentation, and 
I’d argue that it is not using the highest-quality source if it’s 
reading from /dev/urandom or getrandom(GRD_NONBLOCK) on Linux systems 
where getrandom() is available. Of course, it’s just an alias for 
random.SystemRandom, and that is documented as using os.urandom.


—
Donald Stufft





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Sebastian Krause]

David Mertz  wrote:
> It feels to me like the correct spelling in 3.6 should probably be
> secrets.getrandom() or something related to that.

Since there already is a secrets.randbits(k), I would keep the
naming similar and suggest something like:

secrets.randbytes(k, *, nonblock=False)

With the argument "nonblock" you can control what happens when not
enough entropy is available: It either blocks or (if nonblock=True)
raises an exception. The third option, getting unsecure random data,
is simply not available in this function.

Then you can keep os.urandom() as it was in Python 3.4 and earlier,
but update the documentation to better warn about its behavior and
point developers to the secrets module.

Sebastian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Brett Cannon]

On Fri, 10 Jun 2016 at 12:20 Donald Stufft  wrote:

>
> On Jun 10, 2016, at 3:05 PM, David Mertz  wrote:
>
> OK.  My understanding is that Guido ruled out introducing an
> os.getrandom() API in 3.5.2.  But would you be happy if that interface is
> added to 3.6?
>
> It feels to me like the correct spelling in 3.6 should probably be
> secrets.getrandom() or something related to that.
>
>
>
> Well we have
> https://docs.python.org/dev/library/secrets.html#secrets.token_bytes so
> adding a getrandom() function to secrets would largely be the same as that
> function.
>
> The problem of course is that the secrets library in 3.6 uses os.urandom
> under the covers, so it’s security rests on the security of os.urandom. To
> ensure that the secrets library is actually safe even in early boot it’ll
> need to stop using os.urandom on Linux and use the getrandom() function.
>
> That same library exposes random.SystemRandom as secrets.SystemRandom [1],
> and of course SystemRandom uses os.urandom too. So if we want people to
> treat secrets.SystemRandom as “always secure” then it would need to stop
> using os.urandom and start using the get random() function on Linux as well.
>
>
> [1] This is actually documented as "using the highest-quality sources
> provided by the operating system” in the secrets documentation, and I’d
> argue that it is not using the highest-quality source if it’s reading from
> /dev/urandom or getrandom(GRD_NONBLOCK) on Linux systems where getrandom()
> is available. Of course, it’s just an alias for random.SystemRandom, and
> that is documented as using os.urandom.
>

If that's the case then we should file a bug so we are sure this is the
case and we need to decouple the secrets documentation from random so that
they can operate independently with secrets always doing whatever is
required to be as secure as possible.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com