Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Martin Panter]

> On 06/09/2016 08:52 AM, Guido van Rossum wrote:
> That leaves direct calls to os.urandom(). I don't think this should block
> either.

On 9 June 2016 at 22:22, Larry Hastings  wrote:
> Then it's you and me against the rest of the world ;-)
>
>
> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never block on
> a getrandom() call.  It's permissible to take advantage of
> getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from
> /dev/urandom.

So assuming this is the “final” decision, where to from here? I think
the latest change by Colm and committed by Victor already implements
this decision:

https://hg.python.org/cpython/rev/9de508dc4837

Getrandom() is still called, but if it would block, we fall back to
trying the less-secure Linux /dev/urandom, or fail if /dev/urandom is
missing. The Python hash seed is still set using this code. And
os.urandom() calls this code. Random.seed() and SystemRandom still use
os.urandom(), as documented.

So I suggest we close the original mega bug thread
 as fixed. Unless people think
they can change Larry or Guido’s mind, we should focus further
discussion on any changes for 3.6.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Ordered Class Definition Namespace

[Ethan Furman]

On 06/07/2016 11:13 AM, Eric Snow wrote:

On Tue, Jun 7, 2016 at 11:01 AM, Ethan Furman  wrote:

On 06/07/2016 10:51 AM, Eric Snow wrote:

Specification
=




* types for which `__prepare__()`` returned something other than
  ``OrderedDict`` (or a subclass) have their ``__definition_order__``
  set to ``None``



I assume this check happens in type.__new__?  If a non-OrderedDict is used
as the namespace, but a __definition_order__ key and value are supplied, is
it used or still set to None?


A __definition_order__ in the class body always takes precedence.  So
a supplied value will be honored (and not replaced with None).


Nice.  I'll add it to the Enum, enum34, and aenum as soon as it lands 
(give or take a couple months ;)


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Thu, Jun 9, 2016 at 8:11 PM, Larry Hastings  wrote:

>
> On 06/09/2016 07:58 PM, Nathaniel Smith wrote:
>
> I suspect the crypto folks would be okay with pushing this back to
> 3.6, so long as the final resolution is that os.urandom remains the
> standard interface for, as the docstring says, "Return[ing] a string
> of n random bytes suitable for cryptographic use" using the
> OS-recommended method, and they don't have to go change all their
> code.
>
>
> The Linux core devs didn't like the behavior of /dev/urandom.  But they
> couldn't change its behavior without breaking userspace code.  Linux takes
> backwards-compatibility very seriously, so they left /dev/urandom exactly
> the way it was and added new functionality (the getrandom() system call)
> that had the semantics they felt were best.
>
> I don't understand why so many people seem to think it's okay to break old
> code in new versions of Python, when Python's history has shown a similarly
> strong commitment to backwards-compatibility.  os.urandom() was added in
> Python 2.4, in 2004, and remained unchanged for about thirteen years.
> That's thirteen years of people calling it and assuming its semantics were
> identical to the local "urandom" man page, which was correct.
>

I can only speak for myself, but the the reason it doesn't bother me is
that the documentation for os.urandom has always been very clear that it is
an abstraction over multiple OS-specific sources of cryptographic
randomness -- even in the 2.4 docs [1] we read that its output "depends on
the OS implementation", and that it might be /dev/urandom, it might be
CryptGenRandom, and it might even raise an exception if "a randomness
source is not found". So as a user I've always expected that it will make a
best-effort attempt to use whatever the best source of cryptographic
randomness is in a given environment, or else make a best-effort attempt to
raise an error if it's determined that it can't give me cryptographic
randomness, and it's been doing that unchanged for thirteen years too.

But now Linux has moved forward and provided an improved OS-specific source
of cryptographic randomness, and in particular one that actually signals to
userspace when it doesn't have randomness available. So we have a choice:
either we have to break the guarantee that os.urandom is identical to
/dev/urandom, or we have to break the guarantee that os.urandom uses the
best OS-specific source of cryptographic randomness. Either way we're
breaking some guarantee we used to make. And AFAICT so far 100% of the
people who actually maintain libraries that call os.urandom are asking
python-dev to break the identical-to-/dev/urandom guarantee and preserve
the uses-the-best-OS-specific-cryptographic-randomness guarantee.
Disrupting working code is a bad thing, but in the long run, no-one is
actually asking for an os.urandom that silently falls back on the xkcd #221
PRNG [2].

All that said, on the eve of the 3.5.2 release is a terrible time to be
trying to decide this, and it makes perfect sense to me that maybe 3.5
should kick this can down the road. Your efforts as RM are appreciated and
I'm glad I'm not in your spot :-).

-n

[1] https://docs.python.org/2.4/lib/os-miscfunc.html
[2] https://xkcd.com/221/

-- 
Nathaniel J. Smith -- https://vorpus.org 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

So secrets.py needs an upgrade; it currently uses random.SysRandom.

On Thursday, June 9, 2016, Tim Peters  wrote:

> [Nikolaus Rath]
> >> Aeh, what the tin says is "return random bytes".
>
> [Larry Hastings]
> > What the tin says is "urandom", which has local man pages that dictate
> > exactly how it behaves.  On Linux the "urandom" man page says:
> >
> > A read from the /dev/urandom device will not block waiting for more
> entropy.
> > If there is not sufficient entropy, a pseudorandom number generator
> is used
> > to create the requested bytes.
> >
> > os.urandom() needs to behave like that on Linux, which is how it behaved
> in
> > Python 2.4 through 3.4.
>
> I agree (with Larry).  If the change hadn't already been made, nobody
> would get anywhere trying to make it now.  So best to pretend it was
> never made to begin with ;-)
>
> The tin that _will_ say "return random bytes" in Python will
> be`secrets.token_bytes()`.  That's self-evidently (to me) where the
> "possibly block forever" implementation belongs.
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Tim Peters]

[Nikolaus Rath]
>> Aeh, what the tin says is "return random bytes".

[Larry Hastings]
> What the tin says is "urandom", which has local man pages that dictate
> exactly how it behaves.  On Linux the "urandom" man page says:
>
> A read from the /dev/urandom device will not block waiting for more 
> entropy.
> If there is not sufficient entropy, a pseudorandom number generator is 
> used
> to create the requested bytes.
>
> os.urandom() needs to behave like that on Linux, which is how it behaved in
> Python 2.4 through 3.4.

I agree (with Larry).  If the change hadn't already been made, nobody
would get anywhere trying to make it now.  So best to pretend it was
never made to begin with ;-)

The tin that _will_ say "return random bytes" in Python will
be`secrets.token_bytes()`.  That's self-evidently (to me) where the
"possibly block forever" implementation belongs.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Larry Hastings  wrote:
> On 06/09/2016 07:38 PM, Nikolaus Rath wrote:
>> On Jun 09 2016, Larry Hastings  wrote:
>>> Nope, I want the old behavior back.  os.urandom() should read
>>> /dev/random if getrandom() would block.  As the British say, "it
>>> should do what it says on the tin".
>> Aeh, what the tin says is "return random bytes".
>
> What the tin says is "urandom", which has local man pages that dictate
> exactly how it behaves. 
[...]

I disagree. The authoritative source for the behavior of the Python
'urandom' function is the Python documentation, not the Linux manpage
for the "urandom" device.

And https://docs.python.org/3.4/library/os.html says first and foremost:

 os.urandom(n)¶

Return a string of n random bytes suitable for cryptographic use.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block foreverwaiting for high-quality random bits?

[Larry Hastings]

On 06/09/2016 05:00 PM, Steve Dower wrote:
If the pattern is really going to be the hasattr check you posted 
earlier, can we just do it for people and save them writing code that 
won't work on different OSs?


No.  That's what got us into this mess in the first place.

3.5.0 and 3.5.1 *already* changed to the new behavior, and it resulted 
in the situation where CPython blocked forever at startup in these 
certain edge cases.  os.urandom() has been around for more than a 
decade, we can't unilaterally change its semantics now. os.urandom() in 
3.5 has to go back to how it behaved on Linux in 3.4.  And if I were 
release manager for 3.6, I'd say "it has to stay that way in 3.6 too".


However, Guido's already said "don't add os.getrandom() in 3.5", so the 
debate is somewhat irrelevant.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Thu, Jun 9, 2016 at 6:53 PM, Barry Warsaw  wrote:
> On Jun 09, 2016, at 03:22 PM, Larry Hastings wrote:
>
>>On 06/09/2016 08:52 AM, Guido van Rossum wrote:
>>> That leaves direct calls to os.urandom(). I don't think this should > block 
>>> either.
>>
>>Then it's you and me against the rest of the world ;-)
>
> FWIW, I agree with you and Guido.  I'm also not opposed to adding a more
> direct exposure of getrandom(), but in Python 3.6 only.  Like it or not,
> that's the right approach for our backward compatibility policies.

I suspect the crypto folks would be okay with pushing this back to
3.6, so long as the final resolution is that os.urandom remains the
standard interface for, as the docstring says, "Return[ing] a string
of n random bytes suitable for cryptographic use" using the
OS-recommended method, and they don't have to go change all their
code. After all, 3.4 and 2.7 will still have this subtle brokenness
for some time.

But I'm a little uncertain what you think would need to happen to
satisfy the backwards compatibility policies. If we can change it in
3.6 without having a warning in 3.5, then presumably we can also
change it in 3.5 without a warning in 3.4, which is what already
happened accidentally :-).

Would it be acceptable for 3.5.2 to start raising a warning "urandom
returning non-random bytes -- in 3.6 this will be an error", and then
make it an error in 3.6?

(And it would probably be good even in the long run to issue a
prominent warning if hash seeding fails.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/09/2016 07:38 PM, Nikolaus Rath wrote:

On Jun 09 2016, Larry Hastings  wrote:

Nope, I want the old behavior back.  os.urandom() should read
/dev/random if getrandom() would block.  As the British say, "it
should do what it says on the tin".

Aeh, what the tin says is "return random bytes".


What the tin says is "urandom", which has local man pages that dictate 
exactly how it behaves.  On Linux the "urandom" man page says:


   A read from the /dev/urandom device will not block waiting for more
   entropy.  If there is not sufficient entropy, a pseudorandom number
   generator is used to create the requested bytes.

os.urandom() needs to behave like that on Linux, which is how it behaved 
in Python 2.4 through 3.4.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Guido van Rossum  wrote:
> I don't think we should add a new function. I think we should convince
> ourselves that there is not enough of a risk of an exploit even if
> os.urandom() falls back.

That will be hard, because you have to consider an active, clever
adversary.

On the other hand, convincing yourself that in practice os.urandom would
never block unless the setup is super exotic or there is active
maliciousness seems much easier.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Larry Hastings  wrote:
> On 06/09/2016 03:44 PM, Ethan Furman wrote:
>> On 06/09/2016 03:22 PM, Larry Hastings wrote:
>>> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never
>>> block on a getrandom() call.
>>
>> One way to not block is to raise an exception.  Since this is such a
>> rare occurrence anyway I don't see this being a problem, plus it
>> keeps everybody mostly happy:  normal users won't see it hang,
>> crypto-folk won't see vulnerable-from-this-cause-by-default
>> machines, and those running Python early in the boot sequence will
>> have something they can figure out, plus an existing knob to work
>> around it [hashseed, I think?].
>
> Nope, I want the old behavior back.  os.urandom() should read
> /dev/random if getrandom() would block.  As the British say, "it
> should do what it says on the tin".

Aeh, what the tin says is "return random bytes". What everyone uses it
for (including the standard library) is to provide randomness for
cryptographic purposes. What it does (in the problematic case) is return
something that's not random.

To me this sounds about as sensible as having open('/dev/zero') return
non-zero values in some rare situations. And yes, for most people "the
kernel running out of zeros" makes exactly as much sense as "the kernel
runs out of random data". 


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Barry Warsaw]

On Jun 09, 2016, at 05:35 PM, Neil Schemenauer wrote:

>Amber Brown claimed that she spent $60k of her time porting Twisted to Python
>3.  I think there is lots of room to make our porting tools better.

Amber gave a presentation at the language summit and a Pycon talk.  The latter
video is up on YouTube but the former wasn't recorded.  I'm hoping Jake will
post a summary of it though.

She's done a truly impressive amount of work in porting Twisted and has a lot
of good insight.  I've ported a fair bit, but nothing of the size and
complexity of Twisted.  FWIW, I did port the Mailman 3 core, which is now
Python 3.4 and 3.5 compatible.

In my own experience, and IIRC Amber had a similar experience, the ease of
porting to Python 3 really comes down to how bytes/unicode clean your code
base is.  Almost all the other pieces are either pretty manageable or fairly
easily automated.  But if you're code isn't bytes-clean you're in for a world
of hurt because you first have to decide how to represent those things.
Twisted's job is especially fun because it's all about wire protocols, which I
think Amber described as (paraphrasing) bytes that happen to have contents
that look like strings.

I've ported some libraries that weren't bytes-clean.  With one of them, I
actually failed twice before I hit on the correct representation.  Once I got
that right the rest went much more quickly.

There's does seem to be a wide variety of experiences in porting to Python 3.
I think is worth both accepting, acknowledging, and promoting that for a lot
of code, it's really not that hard, but that for some code it's really
painful.  It's within our job to help understand the remaining pain and
address it in some way.  But let's also not scare people away from Python 3,
because it *can* be very easy to port, and I think there's fairly widespread
agreement that once you're in the Python 3 world, you don't want to look back.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 09, 2016, at 03:22 PM, Larry Hastings wrote:

>On 06/09/2016 08:52 AM, Guido van Rossum wrote:
>> That leaves direct calls to os.urandom(). I don't think this should > block 
>> either.  
>
>Then it's you and me against the rest of the world ;-)

FWIW, I agree with you and Guido.  I'm also not opposed to adding a more
direct exposure of getrandom(), but in Python 3.6 only.  Like it or not,
that's the right approach for our backward compatibility policies.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

I don't think we should add a new function. I think we should convince
ourselves that there is not enough of a risk of an exploit even if
os.urandom() falls back.

On Thu, Jun 9, 2016 at 6:03 PM, Nathaniel Smith  wrote:

> On Thu, Jun 9, 2016 at 3:22 PM, Larry Hastings  wrote:
> >
> > On 06/09/2016 08:52 AM, Guido van Rossum wrote:
> >
> >> That leaves direct calls to os.urandom(). I don't think this should
> block
> >> either.
> >
> >
> > Then it's you and me against the rest of the world ;-)
> >
> >
> > Okay, it's decided: os.urandom() must be changed for 3.5.2 to never
> block on
> > a getrandom() call.  It's permissible to take advantage of
> > getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from
> > /dev/urandom.
> >
> > It's already well established that this will upset the cryptography
> experts.
> > As a concession to them, I propose adding a simple! predictable!
> function to
> > Python 3.5.2: os.getrandom().  This would be a simple wrapper over
> > getrandom, only available on platforms that expose it.  It would provide
> a
> > way to use both extant flags, GRND_RANDOM and  GRND_NONBLOCK, though
> > possibly not exactly mirroring the native API.
> >
> > This would enable cryptography libraries to easily do what (IIUC) they
> > regard as the "correct" thing on Linux for all supported versions of
> Python:
> >
> > if hasattr(os, "getrandom"):
> > bits = os.getrandom(n)
> > else:
> > bits = os.urandom(n)
>
> So I understand that the trade-offs between crypto users and regular
> users are tricky, but this resolution concerns me quite a bit :-(
>
> Specifically, it seems to me that:
> 1) we now have these two functions that need to be supported forever,
> and AFAICT in every case where someone is currently explicitly calling
> os.urandom and the behavior differs, they want os.getrandom instead.
> (This is based on the assumption that the only time that explicitly
> calling os.urandom is the best option is when one cares about the
> cryptographic strength of the result -- I'm explicitly distinguishing
> here between the hash seeding issue that triggered the original bug
> report and explicit calls to os.urandom.) So in practice this change
> makes it so that the only correct way of calling either of these
> functions is the if/else stanza above.
> 2) every piece of security-sensitive software is going to spend
> resources churning their code to implement the above,
> 3) every future security audit of Python software is going to spend
> resources making sure this is on their checklist of incredibly subtle
> gotchas that have to be audited for,
> 4) the crypto folks are going to have to spin up a whole evangelism
> effort to re-educate everyone that (contrary to what we've been
> telling everyone for years), os.urandom is no longer the right way to
> get cryptographic randomness.
>
> OTOH if we allow explicit calls to os.urandom to block or raise an
> exception, then AFAICT from this thread this will break exactly zero
> projects.
>
> Maybe this is just rehashing the same things that have already been
> discussed ad naseaum, in which case I apologize. But I really feel
> like this is one of those cases where the crypto folks aren't so much
> saying "oh BUT what if  oppressive regimes and ticking bombs>"; they're more saying "oh $#@
> you're going to cause me a *massive* amount of real work and churn and
> ongoing costs for no perceivable gain and I'm exhausted even thinking
> about it".
>
> > I'm not excited about adding a new function in 3.5.2, but on the other
> hand
> > we are taking away this functionality they had in 3.5.0 and 3.5.1 so only
> > seems fair.  And the implementation of os.getrandom() should be very
> > straightforward, and its semantics will mirror the native call, so I'm
> > pretty confident we can get it solid in a couple of days, though we might
> > slip 3.5.2rc1 by a day or two.
> >
> > Guido: do you see this as an acceptable compromise?
> >
> > Cryptographers: given that os.urandom() will no longer block in 3.5.2, do
> > you want this?
> >
> >
> > Pointing out an alternate approach: Marc-Andre Lemburg proposes in issue
> > #27279 ( http://bugs.python.org/issue27279 ) that we should add two
> "known
> > best-practices" functions to get pseudo-random bits; one merely for
> pseudo
> > random bits, the other for crypto-strength pseudo random bits.  While I
> > think this is a fine idea, the exact spelling, semantics, and
> per-platform
> > implementation of these functions is far from settled, and nobody is
> > proposing that we do something like that for 3.5.
>
> We already have a function for non-crypto-strength pseudo-random bits:
> random.getrandbits. os.urandom is the one for the cryptographers (I
> thought).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nathaniel Smith]

On Thu, Jun 9, 2016 at 3:22 PM, Larry Hastings  wrote:
>
> On 06/09/2016 08:52 AM, Guido van Rossum wrote:
>
>> That leaves direct calls to os.urandom(). I don't think this should block
>> either.
>
>
> Then it's you and me against the rest of the world ;-)
>
>
> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never block on
> a getrandom() call.  It's permissible to take advantage of
> getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from
> /dev/urandom.
>
> It's already well established that this will upset the cryptography experts.
> As a concession to them, I propose adding a simple! predictable! function to
> Python 3.5.2: os.getrandom().  This would be a simple wrapper over
> getrandom, only available on platforms that expose it.  It would provide a
> way to use both extant flags, GRND_RANDOM and  GRND_NONBLOCK, though
> possibly not exactly mirroring the native API.
>
> This would enable cryptography libraries to easily do what (IIUC) they
> regard as the "correct" thing on Linux for all supported versions of Python:
>
> if hasattr(os, "getrandom"):
> bits = os.getrandom(n)
> else:
> bits = os.urandom(n)

So I understand that the trade-offs between crypto users and regular
users are tricky, but this resolution concerns me quite a bit :-(

Specifically, it seems to me that:
1) we now have these two functions that need to be supported forever,
and AFAICT in every case where someone is currently explicitly calling
os.urandom and the behavior differs, they want os.getrandom instead.
(This is based on the assumption that the only time that explicitly
calling os.urandom is the best option is when one cares about the
cryptographic strength of the result -- I'm explicitly distinguishing
here between the hash seeding issue that triggered the original bug
report and explicit calls to os.urandom.) So in practice this change
makes it so that the only correct way of calling either of these
functions is the if/else stanza above.
2) every piece of security-sensitive software is going to spend
resources churning their code to implement the above,
3) every future security audit of Python software is going to spend
resources making sure this is on their checklist of incredibly subtle
gotchas that have to be audited for,
4) the crypto folks are going to have to spin up a whole evangelism
effort to re-educate everyone that (contrary to what we've been
telling everyone for years), os.urandom is no longer the right way to
get cryptographic randomness.

OTOH if we allow explicit calls to os.urandom to block or raise an
exception, then AFAICT from this thread this will break exactly zero
projects.

Maybe this is just rehashing the same things that have already been
discussed ad naseaum, in which case I apologize. But I really feel
like this is one of those cases where the crypto folks aren't so much
saying "oh BUT what if "; they're more saying "oh $#@
you're going to cause me a *massive* amount of real work and churn and
ongoing costs for no perceivable gain and I'm exhausted even thinking
about it".

> I'm not excited about adding a new function in 3.5.2, but on the other hand
> we are taking away this functionality they had in 3.5.0 and 3.5.1 so only
> seems fair.  And the implementation of os.getrandom() should be very
> straightforward, and its semantics will mirror the native call, so I'm
> pretty confident we can get it solid in a couple of days, though we might
> slip 3.5.2rc1 by a day or two.
>
> Guido: do you see this as an acceptable compromise?
>
> Cryptographers: given that os.urandom() will no longer block in 3.5.2, do
> you want this?
>
>
> Pointing out an alternate approach: Marc-Andre Lemburg proposes in issue
> #27279 ( http://bugs.python.org/issue27279 ) that we should add two "known
> best-practices" functions to get pseudo-random bits; one merely for pseudo
> random bits, the other for crypto-strength pseudo random bits.  While I
> think this is a fine idea, the exact spelling, semantics, and per-platform
> implementation of these functions is far from settled, and nobody is
> proposing that we do something like that for 3.5.

We already have a function for non-crypto-strength pseudo-random bits:
random.getrandbits. os.urandom is the one for the cryptographers (I
thought).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Greg Ewing]

Steven D'Aprano wrote:
- Linux /dev/urandom doesn't block, but it might return predictable, 
  poor-quality pseudo-random bytes (i.e. a potential exploit);


- Other OSes may block for potentially many minutes (i.e. a 
  potential DOS).


It's even possible that it could block *forever*.

There was a case here recently in the cosc dept where students were
running Clojure programs in a virtual machine environment. When
they updated to a newer version of Clojure, everyone's programs
started hanging on startup. It turned out the Clojure library was
initialising its RNG from /dev/random, and the VM didn't have any
real spinning disks or other devices to provide entropy.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block foreverwaiting for high-quality random bits?

[Steve Dower]

Can we get any new function on all platforms, deferring to urandom() if 
getrandom() isn't there?

If the pattern is really going to be the hasattr check you posted earlier

Top-posted from my Windows Phone

-Original Message-
From: "Larry Hastings" 
Sent: ‎6/‎10/‎2016 8:50
To: "python-dev@python.org" 
Subject: Re: [Python-Dev] BDFL ruling request: should we block foreverwaiting 
for high-quality random bits?


On 06/09/2016 03:44 PM, Ethan Furman wrote:

On 06/09/2016 03:22 PM, Larry Hastings wrote: 

Okay, it's decided: os.urandom() must be changed for 3.5.2 to never 
block on a getrandom() call. 


One way to not block is to raise an exception.  Since this is such a rare 
occurrence anyway I don't see this being a problem, plus it keeps everybody 
mostly happy:  normal users won't see it hang, crypto-folk won't see 
vulnerable-from-this-cause-by-default machines, and those running Python early 
in the boot sequence will have something they can figure out, plus an existing 
knob to work around it [hashseed, I think?].



Nope, I want the old behavior back.  os.urandom() should read /dev/random if 
getrandom() would block.  As the British say, "it should do what it says on the 
tin".


/arry___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Neil Schemenauer]

On 2016-06-09, Brett Cannon wrote:
> I don't think you meant for what you said to sound insulting,
> Neil, but it did feel like it upon first reading.

Sorry, I think I misunderstood what you and Nick were saying.  I've
experienced a fair amount of negative feedback on my idea so I'm
pretty cranky at this point.  Amber Brown claimed that she spent
$60k of her time porting Twisted to Python 3.  I think there is lots
of room to make our porting tools better.

Using something like modernize, 2to6, or sixer seems like a better
idea than trying to improve on 2to3.  I agree on that point.
However, those tools combined with my modified Python 3.6 makes for
a much easier migration path than going directly to Python 3.x.  My
runtime warnings catch many common problems and make it easy to see
what needs fixing.

We have a lot more freedom to put ugly, backwards compatibility
hacks into this stepping stone version, rather than changing either
Python 2.7.x or the main 3.x line.  I'm hoping to get community
contributions to add more backwards compatibility and runtime
warnings.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block foreverwaiting for high-quality random bits?

[Steve Dower]

(fat fingered the send button, picking up where I left off)

If the pattern is really going to be the hasattr check you posted earlier, can 
we just do it for people and save them writing code that won't work on 
different OSs?

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Larry Hastings" 
Sent: ‎6/‎10/‎2016 8:50
To: "python-dev@python.org" 
Subject: Re: [Python-Dev] BDFL ruling request: should we block foreverwaiting 
for high-quality random bits?


On 06/09/2016 03:44 PM, Ethan Furman wrote:

On 06/09/2016 03:22 PM, Larry Hastings wrote: 

Okay, it's decided: os.urandom() must be changed for 3.5.2 to never 
block on a getrandom() call. 


One way to not block is to raise an exception.  Since this is such a rare 
occurrence anyway I don't see this being a problem, plus it keeps everybody 
mostly happy:  normal users won't see it hang, crypto-folk won't see 
vulnerable-from-this-cause-by-default machines, and those running Python early 
in the boot sequence will have something they can figure out, plus an existing 
knob to work around it [hashseed, I think?].



Nope, I want the old behavior back.  os.urandom() should read /dev/random if 
getrandom() would block.  As the British say, "it should do what it says on the 
tin".


/arry___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Brett Cannon]

On Thu, 9 Jun 2016 at 16:08 Neil Schemenauer  wrote:

> On 2016-06-09, Brett Cannon wrote:
> > On Thu, 9 Jun 2016 at 14:56 Nick Coghlan  wrote:
> > > Once you switch to those now recommended more conservative migration
> > > tools, the tool suite you request already exists:
> > >
> > > - update your code with modernize or futurize
> > > - check it still runs on Python 2.7
> > > - check it doesn't generate warnings under 2.7's "-3" switch
> > > - check it passes "pylint --py3k"
> > > - check if it runs on Python 3.5
> > >
> >
> > `python3.5 -bb` is best to help keep Python 2.7 compatibility, otherwise
> > what Nick said. :)
>
> I have to wonder if you guys actually ported at lot of Python 2
> code.


Yes I have, including code that needed to be 2.4-3.4 compatible of all
things. Plus I'm the author of the porting HOWTO so I know the edge cases
pretty well.

I don't think you meant for what you said to sound insulting, Neil, but it
did feel like it upon first reading.


> Maybe you somehow avoided the problematic behavior. Below is
> a pretty trival set of functions.  The tools you recommend do not
> help at all.  One problem is that the str literals should be bytes
> literals.


At least for Modernize that's on purpose as it can't tell semantically what
is meant to be binary data vs. textual ASCII data (which you obviously
know, else you wouldn't be trying to add runtime warnings for this sort of
stuff).


> Comparison with None needs to be avoided.
>
> With Python 2 code runs successfully.  With Python 3 the code
> crashes with a traceback.  With my modified Python 3.6, the code
> runs successfully but generates the following warnings:
>
> test.py:13: DeprecationWarning: encoding bytes to str
>   output.write('%d:' % len(s))
> test.py:14: DeprecationWarning: encoding bytes to str
>   output.write(s)
> test.py:15: DeprecationWarning: encoding bytes to str
>   output.write(',')
> test.py:5: DeprecationWarning: encoding bytes to str
>   if c == ':':
> test.py:9: DeprecationWarning: encoding bytes to str
>   size += c
> test.py:24: DeprecationWarning: encoding bytes to str
>   data = data + s
> test.py:26: DeprecationWarning: encoding bytes to str
>   if input.read(1) != ',':
> test.py:31: DeprecationWarning: default compare is depreciated
>   if a > 0:
>
> It is very easy for me to find code written for Python 2 that will
> fail in the same way.  According to you guys, there is no problem
> and we already have good enough tooling. ;-(
>

That's not what I'm saying at all (nor what I think Nick is saying); more
tooling to ease the transition is always welcomed. The point we are trying
to make is 2to3 is not considered best practice anymore, and so targeting
its specific output might not be the best use of your time. I'm totally
happy to have your fork work out and help give warnings for situations
where runtime semantics are the only way to know there will be a problem
that static analyzing tools can't handle and have the porting HOWTO updated
so that people can run their test suite with your interpreter to help with
that final bit of porting. I personally just don't want to see you waste
time on warnings that are handled by the tools already or ignore the fact
that six, modernize, and futurize can help more than 2to3 typically can
with the easy stuff when trying to keep 2/3 compatibility. IOW some of us
have become allergic to the word "2to3" in regards to porting. :) But if
you want to target 2to3 output then by all means please do and your work
will still be appreciated.

And I should also mention in case you don't know -- and assuming I'm
remembering correctly -- that adding new Py3kWarning cases to Python 2.7 is
still allowed, so if there is a warning you want to add that makes sense to
be upstream then we can consider adding it in Python 2.7.12 (or later).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Neil Schemenauer]

On 2016-06-09, Brett Cannon wrote:
> On Thu, 9 Jun 2016 at 14:56 Nick Coghlan  wrote:
> > Once you switch to those now recommended more conservative migration
> > tools, the tool suite you request already exists:
> >
> > - update your code with modernize or futurize
> > - check it still runs on Python 2.7
> > - check it doesn't generate warnings under 2.7's "-3" switch
> > - check it passes "pylint --py3k"
> > - check if it runs on Python 3.5
> >
> 
> `python3.5 -bb` is best to help keep Python 2.7 compatibility, otherwise
> what Nick said. :)

I have to wonder if you guys actually ported at lot of Python 2
code.  Maybe you somehow avoided the problematic behavior. Below is
a pretty trival set of functions.  The tools you recommend do not
help at all.  One problem is that the str literals should be bytes
literals.  Comparison with None needs to be avoided.

With Python 2 code runs successfully.  With Python 3 the code
crashes with a traceback.  With my modified Python 3.6, the code
runs successfully but generates the following warnings:

test.py:13: DeprecationWarning: encoding bytes to str
  output.write('%d:' % len(s))
test.py:14: DeprecationWarning: encoding bytes to str
  output.write(s)
test.py:15: DeprecationWarning: encoding bytes to str
  output.write(',')
test.py:5: DeprecationWarning: encoding bytes to str
  if c == ':':
test.py:9: DeprecationWarning: encoding bytes to str
  size += c
test.py:24: DeprecationWarning: encoding bytes to str
  data = data + s
test.py:26: DeprecationWarning: encoding bytes to str
  if input.read(1) != ',':
test.py:31: DeprecationWarning: default compare is depreciated
  if a > 0:

It is very easy for me to find code written for Python 2 that will
fail in the same way.  According to you guys, there is no problem
and we already have good enough tooling. ;-(
def ns_read_size(input):
size = ''
while 1:
c = input.read(1)
if c == ':':
break
elif not c:
raise IOError('short netstring read')
size += c
return int(size)

def ns_write_string(s, output):
output.write('%d:' % len(s))
output.write(s)
output.write(',')

def ns_read_string(input):
size = ns_read_size(input)
data = ''
while size > 0:
s = input.read(size)
if not s:
raise IOError('short netstring read')
data = data + s
size -= len(s)
if input.read(1) != ',':
raise IOError('missing netstring terminator')
return data

def compare(a, b):
if a > 0:
return b + 10
return 0

def main():
import tempfile
out = tempfile.TemporaryFile()
ns_write_string('Hello world', out)
out.seek(0)
s = ns_read_string(out)
if s != 'Hello world':
print('Failed')
else:
print('Ok')
if (compare(None, 5) == 0 and
compare(1, 5) == 15):
print('Ok')
else:
print('Failed')

if __name__ == '__main__':
main()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/09/2016 03:44 PM, Ethan Furman wrote:

On 06/09/2016 03:22 PM, Larry Hastings wrote:

Okay, it's decided: os.urandom() must be changed for 3.5.2 to never
block on a getrandom() call.


One way to not block is to raise an exception.  Since this is such a 
rare occurrence anyway I don't see this being a problem, plus it keeps 
everybody mostly happy:  normal users won't see it hang, crypto-folk 
won't see vulnerable-from-this-cause-by-default machines, and those 
running Python early in the boot sequence will have something they can 
figure out, plus an existing knob to work around it [hashseed, I think?].



Nope, I want the old behavior back.  os.urandom() should read 
/dev/random if getrandom() would block.  As the British say, "it should 
do what it says on the tin".



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ethan Furman]

On 06/09/2016 03:22 PM, Larry Hastings wrote:


On 06/09/2016 08:52 AM, Guido van Rossum wrote:

That leaves direct calls to os.urandom(). I don't think this should
block either.


Then it's you and me against the rest of the world ;-)


Okay, it's decided: os.urandom() must be changed for 3.5.2 to never
block on a getrandom() call.


One way to not block is to raise an exception.  Since this is such a 
rare occurrence anyway I don't see this being a problem, plus it keeps 
everybody mostly happy:  normal users won't see it hang, crypto-folk 
won't see vulnerable-from-this-cause-by-default machines, and those 
running Python early in the boot sequence will have something they can 
figure out, plus an existing knob to work around it [hashseed, I think?].



As a concession to [the crypto experts], I propose adding a simple!
predictable! function to Python 3.5.2: os.getrandom().


This would be unnecessary if we go the exception route.


And the implementation of os.getrandom() should be
very straightforward, and its semantics will mirror the native call, so
I'm pretty confident we can get it solid in a couple of days, though we
might slip 3.5.2rc1 by a day or two.


I would think the exception route would also not take very long to make 
solid.


Okay, I'll shut up now.  ;)

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/09/2016 10:22 AM, Donald Stufft wrote:

On Jun 9, 2016, at 1:14 PM, Steven D'Aprano  wrote:

Just to be clear, this is only an option on Linux, right? All the other
major platforms block, whatever we decide to do on Linux. Including
Windows?

To my knowledge, all other major platforms block or otherwise ensure that 
/dev/urandom can never return anything but cryptographically secure random. [1]


I've done some research into this over the past couple of days.  To the 
best of my knowledge:


* Linux: /dev/urandom will never block.  If the entropy pool isn't 
initialized yet, it will return poor-quality random bits from what is 
effectively an unseeded PRNG.  (Yes: it uses a custom algorithm which 
isn't considered CPRNG-strength, it is merely a PRNG seeded with entropy.)


* OS X: AFAICT, /dev/urandom guarantees it will never block.  It uses an 
old CSPRNG, 160-bit Yarrow.  The documentation states that if the 
entropy pool is "drained", it won't block; instead it'll degrade 
("output quality will suffer over time without any explicit indication 
from the random device itself").  It isn't clear how initialization of 
the entropy pool during early startup might affect this.  
http://www.manpages.info/macosx/random.4.html


* FreeBSD: /dev/urandom may block.  It also using Yarrow (but maybe with 
more bits? and possibly switching soon to Yarrow's successor, 
Fortuna?).  Both devices guarantee high-quality random bits, and will 
block if they feel like they're running low on entropy.


* OpenBSD 5.1 is like FreeBSD, except the algorithm used is ARC4. In 
OpenBSD 5.5 they changed to using ChaCha20.


On all of those platforms *except* Linux, /dev/random and /dev/urandom 
are exactly the same.



Also, regarding Windows: Victor Stinner did some experiments with a VM, 
and even in early startup he was able to get random bits from 
os.urandom().  But it's hard to have a "fresh" Windows VM, so it's 
possible it had residual entropy from a previous boot, so this isn't 
conclusive.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

On 06/09/2016 08:52 AM, Guido van Rossum wrote:
That leaves direct calls to os.urandom(). I don't think this should 
block either.


Then it's you and me against the rest of the world ;-)


Okay, it's decided: os.urandom() must be changed for 3.5.2 to never 
block on a getrandom() call.  It's permissible to take advantage of 
getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from 
/dev/urandom.



It's already well established that this will upset the cryptography 
experts.  As a concession to them, I propose adding a simple! 
predictable! function to Python 3.5.2: os.getrandom().  This would be a 
simple wrapper over getrandom, only available on platforms that expose 
it.  It would provide a way to use both extant flags, GRND_RANDOM and  
GRND_NONBLOCK, though possibly not exactly mirroring the native API.


This would enable cryptography libraries to easily do what (IIUC) they 
regard as the "correct" thing on Linux for all supported versions of Python:


   if hasattr(os, "getrandom"):
bits = os.getrandom(n)
   else:
bits = os.urandom(n)

I'm not excited about adding a new function in 3.5.2, but on the other 
hand we are taking away this functionality they had in 3.5.0 and 3.5.1 
so only seems fair.  And the implementation of os.getrandom() should be 
very straightforward, and its semantics will mirror the native call, so 
I'm pretty confident we can get it solid in a couple of days, though we 
might slip 3.5.2rc1 by a day or two.


Guido: do you see this as an acceptable compromise?

Cryptographers: given that os.urandom() will no longer block in 3.5.2, 
do you want this?



Pointing out an alternate approach: Marc-Andre Lemburg proposes in issue 
#27279 ( http://bugs.python.org/issue27279 ) that we should add two 
"known best-practices" functions to get pseudo-random bits; one merely 
for pseudo random bits, the other for crypto-strength pseudo random 
bits.  While I think this is a fine idea, the exact spelling, semantics, 
and per-platform implementation of these functions is far from settled, 
and nobody is proposing that we do something like that for 3.5.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Fred Drake]

On Thu, Jun 9, 2016 at 6:16 PM, Ethan Furman  wrote:
> That's awfully close to antipathy [1], my path module on PyPI.

Good point.  Increasing confusion would not help.

> Besides, I liked the suggestion from the -ideas list: Python 2therescue. ;)

Nice; I like that too.  :-)


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Ethan Furman]

On 06/08/2016 02:40 PM, Fred Drake wrote:

On Wed, Jun 8, 2016 at 5:33 PM, Ryan Gonzalez  wrote:

What about something like "unpythonic" or similar?


Or perhaps... antipythy?


That's awfully close to antipathy [1], my path module on PyPI.

Besides, I liked the suggestion from the -ideas list: Python 2therescue. ;)

--
~Ethan~

[1] https://pypi.python.org/pypi/antipathy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 520: Ordered Class Definition Namespace

[Ethan Furman]

On 06/09/2016 02:39 PM, Nick Coghlan wrote:

On 7 June 2016 at 20:17, Eric Snow wrote:

On Tue, Jun 7, 2016 at 6:20 PM, Ethan Furman wrote:



If __definition_order__ is supposed to be immutable as well as read-only
then we should convert non-tuples to tuples.  No point in letting that
user bug slip through.


Do you mean if a class explicitly defines __definition_order__?  If
so, I'm not clear on how that would work.  It could be set to
anything, including None or a value that does not iterate into a
definition order.  If someone explicitly set __definition_order__ then
I think it should be used as-is.


I'm guessing Ethan is suggesting defining it as:

 __definition_order__ = tuple(ns["__definition_order__"])

When the attribute is present in the method body.


Yup, that it's it exactly.  Thanks, Nick!

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Brett Cannon]

On Thu, 9 Jun 2016 at 14:56 Nick Coghlan  wrote:

> On 8 June 2016 at 14:01, Neil Schemenauer  wrote:
> > [I've posted something about this on python-ideas but since I now
> > have some basic working code, I think it is more than an idea.]
> >
> > I think the uptake of Python 3 is starting to accelerate.  That's
> > good.  However, there are still millions or maybe billions of lines
> > of Python code that still needs to be ported.  It is beneficial to
> > the Python ecosystem if this code can get ported.
> >
> > My idea is to make a stepping stone version of Python, between 2.7.x
> > and 3.x that eases the porting job.  The high level goals are:
> >
> > - code coming out of 2to3 runs correctly on this modified Python
> >
> > - code that runs without warnings on this modified Python will run
> >   correctly on Python 3.x.
>
> As Victor noted, and as the porting guide describes in
> https://docs.python.org/3/howto/pyporting.html#update-your-code, we've
> determined that 2to3 isn't the best choice of tool for folks that
> can't afford to immediately drop Python 2 support.
>
> Once you switch to those now recommended more conservative migration
> tools, the tool suite you request already exists:
>
> - update your code with modernize or futurize
> - check it still runs on Python 2.7
> - check it doesn't generate warnings under 2.7's "-3" switch
> - check it passes "pylint --py3k"
> - check if it runs on Python 3.5
>

`python3.5 -bb` is best to help keep Python 2.7 compatibility, otherwise
what Nick said. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Smoothing the transition from Python 2 to 3

[Nick Coghlan]

On 8 June 2016 at 14:01, Neil Schemenauer  wrote:
> [I've posted something about this on python-ideas but since I now
> have some basic working code, I think it is more than an idea.]
>
> I think the uptake of Python 3 is starting to accelerate.  That's
> good.  However, there are still millions or maybe billions of lines
> of Python code that still needs to be ported.  It is beneficial to
> the Python ecosystem if this code can get ported.
>
> My idea is to make a stepping stone version of Python, between 2.7.x
> and 3.x that eases the porting job.  The high level goals are:
>
> - code coming out of 2to3 runs correctly on this modified Python
>
> - code that runs without warnings on this modified Python will run
>   correctly on Python 3.x.

As Victor noted, and as the porting guide describes in
https://docs.python.org/3/howto/pyporting.html#update-your-code, we've
determined that 2to3 isn't the best choice of tool for folks that
can't afford to immediately drop Python 2 support.

Once you switch to those now recommended more conservative migration
tools, the tool suite you request already exists:

- update your code with modernize or futurize
- check it still runs on Python 2.7
- check it doesn't generate warnings under 2.7's "-3" switch
- check it passes "pylint --py3k"
- check if it runs on Python 3.5

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 520: Ordered Class Definition Namespace

[Nick Coghlan]

On 7 June 2016 at 20:17, Eric Snow  wrote:
> On Tue, Jun 7, 2016 at 6:20 PM, Ethan Furman  wrote:
>> If __definition_order__ is supposed to be immutable as well as read-only
>> then we should convert non-tuples to tuples.  No point in letting that
>> user bug slip through.
>
> Do you mean if a class explicitly defines __definition_order__?  If
> so, I'm not clear on how that would work.  It could be set to
> anything, including None or a value that does not iterate into a
> definition order.  If someone explicitly set __definition_order__ then
> I think it should be used as-is.

I'm guessing Ethan is suggesting defining it as:

__definition_order__ = tuple(ns["__definition_order__"])

When the attribute is present in the method body.

That restriction would be comparable to what we do with __slots__ today:

>>> class C:
... __slots__ = 1
...
Traceback (most recent call last):
 File "", line 1, in 
TypeError: 'int' object is not iterable

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 468

[Émanuel Barry]

> From: zr...@fastmail.com
> Subject: [Python-Dev] PEP 468
> 
> Is there any further thoughts on including this in 3.6?  Similar to the
> recent discussion on OrderedDict namespaces for metaclasses, this would
> simplify / enable a number of type factory use cases where proper
> metaclasses are overkill. This feature would also be quite nice in say
> pandas where the (currently unspecified) field order used in the
> definition of frames is preserved in user-visible displays.

As stated by Guido (and pointed out in the PEP):

Making **kwds ordered is still open, but requires careful design and
implementation to avoid slowing down function calls that don't benefit.

The PEP has not been updated in a while, though. Python 3.5 has been
released, and with it a C implementation of OrderedDict.

Eric, are you still interested in this? IIRC that PEP was one of the
motivating use cases for implementing OrderedDict in C. Maybe it's time for
a second round of discussion on Python-ideas?

-Emanuel
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 468

[zreed]

Is there any further thoughts on including this in 3.6?  Similar to the
recent discussion on OrderedDict namespaces for metaclasses, this would
simplify / enable a number of type factory use cases where proper
metaclasses are overkill. This feature would also be quite nice in say
pandas where the (currently unspecified) field order used in the
definition of frames is preserved in user-visible displays.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Christian Heimes]

On 2016-06-09 19:14, Steven D'Aprano wrote:
> On Thu, Jun 09, 2016 at 12:39:00PM -0400, Donald Stufft wrote:
> 
>> There are three options for what do with os.urandom by default:
>>
>> * Allow it to silently return data that may or may not be 
>> cryptographically secure based on what the state of the urandom pool 
>> initialization looks like.
> 
> Just to be clear, this is only an option on Linux, right? All the other 
> major platforms block, whatever we decide to do on Linux. Including 
> Windows?

To best of my knowledge, Windows and OSX are already initialized when
Python is started. On other BSD platforms it is possible to get the
seeding state through the proc file system.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ben Leslie]

On 9 June 2016 at 13:29, Steven D'Aprano  wrote:
> On Thu, Jun 09, 2016 at 12:54:31PM -0400, Ben Leslie wrote:
>
>> I think an exception is much easier for a user to deal with from a
>> practical point of view. Trying to work out why a process has hung is
>> obviously possible, but not necessarily easy.
>>
>> Having a process crash due to an exception is very easy to diagnose by
>> comparison.
>
> That only makes sense if the application is going to block for (say)
> five or ten minutes. If it's going to block for three seconds, you might
> not even notice. At least not on a server.
>
> But what are you going to do when you catch that exception?
>
> - Sleep for a few seconds, and try again? That's just blocking.
>
> - Stop waiting on secure randomness, and use something low quality
>   and insecure? That's how you get exploits.
>
> - Crash?

What does a program do when on any exception? It really depends on the
program and the circumstances in which it is running.

But I would think that in most circumstances 'crash' is the answer.

In the circumstances where this is most likely going to occur (server
startup) you are almost certainly going to have some type of
supervisory program restarting the failed process. It will almost
certainly be logging the failure. Having logs filled with process
restarts due to this error until there is finally entropy is better
than it just hanging. At least that is what I'd prefer to diagnose.

I think the real solution here would be outside of Python; starting a
process that needs entropy when the system isn't ready yet is just as
silly as running a 'mount' on a disk where the driver is still
loading, or 'ifconfig' on a network interface where the network driver
isn't yet loaded. But that isn't really a problem that can be solved
in the context of Python.

Cheers,

Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nick Coghlan]

On 9 June 2016 at 04:25, Larry Hastings  wrote:
> A user reports that when starting CPython soon after startup on a fresh
> virtual machine, the process would hang for a long time.  Someone on the
> issue reported observed delays of over 90 seconds.  Later we found out: it
> wasn't 90 seconds before CPython became usable, these 90 seconds delays were
> before systemd timed out and simply killed the process.  It's not clear what
> the upper bound on the delay might be.
>
> The issue author had already identified the cause: CPython was blocking on
> getrandom() in order to initialize hash randomization.  On this fresh
> virtual machine the entropy pool started out uninitialized.  And since the
> only thing running on the machine was CPython, and since CPython was blocked
> on initialization, the entropy pool was initializing very, very slowly.

Further analysis (mentioned later in the original Python-3.5-on-Linux
bug report) suggested that this wasn't actually a generic "waiting for
the entropy pool to initialise" problem. Instead, the problem appeared
to be specifically that the Python script was being invoked *before
the Linux kernel had initialised the entropy pool* and the boot
process was waiting for that script to run before continuing on with
other tasks (like initialising the entropy pool). That meant
os.urandom() had nothing to do with it (since the affected script
wasn't generating random numbers), and the entire problem was that we
were blocking trying to initialise CPython's internal hashing.

Born from Victor's proposal to add a "wait for entropy?" flag to
os.urandom [1], the simplest proposal for a long term fix [2] posted
so far has been to:

1. make os.urandom raise BlockingIOError if kernel entropy is not available
2. don't rely on os.urandom for internal hash initialisation
3. don't rely on os.urandom for MT seeding in the random module

Linux is currently the only OS we know of where the BlockingIOError
would be a possible result, and the only known scenarios where it
could be raised are Linux init system scripts and some embedded
systems where the kernel doesn't have any good sources of entropy. In
both those cases, the lack of entropy is potentially a real problem,
and an exception lets the software author make an informed decision to
either wait for entropy (e.g. by polling os.urandom() until it
succeeds, or selecting on /dev/random) or else read directly from
/dev/urandom (potentially getting non-cryptographically secure bits)

The virtue of this approach is that it's entirely invisible for almost
all users, and the users that it does affect will start getting an
exception in Python 3.6+ rather than silently being handed
cryptographically non-secure random data.

Cheers,
Nick.

[1] http://bugs.python.org/issue27266
[2] http://bugs.python.org/issue27282


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Thu, Jun 09, 2016 at 06:21:32PM +0100, Paul Moore wrote:

> If we put the specific issue of applications that run very early in
> system startup to one side, is there a possibility of running out of
> entropy during normal system use? Even for a tiny duration?

With /dev/urandom, I believe the answer to that is no.

On most platforms other than Linux, /dev/urandom is exactly the same as 
/dev/random, and both can only block straight after the machine has 
booted up before enough entropy has been collected. Then they will run 
forever without blocking. (Or at least until you reboot.)

On Linux, /dev/random *will* block, at unpredictable times, but 
fortunately we're not using /dev/random. We're using Urandom. Apart from 
just after boot up, /dev/urandom on Linux will also run forever without 
blocking, just like the other platforms.

The critical difference is just after booting up:

- Linux /dev/urandom doesn't block, but it might return predictable, 
  poor-quality pseudo-random bytes (i.e. a potential exploit);

- Other OSes may block for potentially many minutes (i.e. a 
  potential DOS).

Two links which may help explain what's happening:

http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

http://security.stackexchange.com/a/42955



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Thu, Jun 09, 2016 at 12:54:31PM -0400, Ben Leslie wrote:

> I think an exception is much easier for a user to deal with from a
> practical point of view. Trying to work out why a process has hung is
> obviously possible, but not necessarily easy.
> 
> Having a process crash due to an exception is very easy to diagnose by
> comparison.

That only makes sense if the application is going to block for (say) 
five or ten minutes. If it's going to block for three seconds, you might 
not even notice. At least not on a server.

But what are you going to do when you catch that exception?

- Sleep for a few seconds, and try again? That's just blocking.

- Stop waiting on secure randomness, and use something low quality 
  and insecure? That's how you get exploits.

- Crash?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 1:14 PM, Steven D'Aprano  wrote:
> 
> On Thu, Jun 09, 2016 at 12:39:00PM -0400, Donald Stufft wrote:
> 
>> There are three options for what do with os.urandom by default:
>> 
>> * Allow it to silently return data that may or may not be 
>> cryptographically secure based on what the state of the urandom pool 
>> initialization looks like.
> 
> Just to be clear, this is only an option on Linux, right? All the other 
> major platforms block, whatever we decide to do on Linux. Including 
> Windows?

To my knowledge, all other major platforms block or otherwise ensure that 
/dev/urandom can never return anything but cryptographically secure random. [1]

> 
> 
> -- 
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

[1] I believe OpenBSD cannot block, but they inject randomness via the boot 
loader so that the system is never in a state where the kernel doesn’t have 
enough entropy.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 1:21 PM, Paul Moore  wrote:
> 
> On 9 June 2016 at 17:54, Ben Leslie  wrote:
>>> My opinion is that blocking is slightly better than raising an exception 
>>> because it matches what other OSs do, but that both blocking and raising an 
>>> exception is better than silently giving data that may or may not be 
>>> cryptographically secure.
>> 
>> I think an exception is much easier for a user to deal with from a
>> practical point of view. Trying to work out why a process has hung is
>> obviously possible, but not necessarily easy.
> 
> If we put the specific issue of applications that run very early in
> system startup to one side, is there a possibility of running out of
> entropy during normal system use? Even for a tiny duration? An
> exception may be better than a hanging process, but a random process
> crash in place of a wait of a few microseconds for the entropy buffer
> to fill up again not so much.
> 
> If we could predict whether the call was going to block for a
> microsecond, or for 20 minutes, I'd be OK with an exception for the
> latter case. But we can't predict the future, so unless the system
> call is guaranteed not to block except at system startup, then I
> prefer blocking over an exception.

/dev/urandom (and getrandom() on Linux) will never block once the pool
has been initialized. The concept of “running out of entropy” doesn’t
apply to it. Once it has entropy it’s good to go.

> 
> As for blocking vs returning less random results, I defer to others on that.
> 
> On 9 June 2016 at 18:14, Steven D'Aprano  wrote:
>> On Thu, Jun 09, 2016 at 12:39:00PM -0400, Donald Stufft wrote:
>> 
>>> There are three options for what do with os.urandom by default:
>>> 
>>> * Allow it to silently return data that may or may not be
>>> cryptographically secure based on what the state of the urandom pool
>>> initialization looks like.
>> 
>> Just to be clear, this is only an option on Linux, right? All the other
>> major platforms block, whatever we decide to do on Linux. Including
>> Windows?
> 
> That's what I understood, certainly. But the place where this was an
> issue in real life was a Python program being run during the startup
> sequence of the OS. That's never going to be possible on Windows, so
> I'd be cautious about drawing parallels with Windows in this situation
> (blocking on Windows may be fine because Python can never run when
> Windows could possibly have low entropy available).
> 
> Paul


—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Paul Moore]

On 9 June 2016 at 17:54, Ben Leslie  wrote:
>> My opinion is that blocking is slightly better than raising an exception 
>> because it matches what other OSs do, but that both blocking and raising an 
>> exception is better than silently giving data that may or may not be 
>> cryptographically secure.
>
> I think an exception is much easier for a user to deal with from a
> practical point of view. Trying to work out why a process has hung is
> obviously possible, but not necessarily easy.

If we put the specific issue of applications that run very early in
system startup to one side, is there a possibility of running out of
entropy during normal system use? Even for a tiny duration? An
exception may be better than a hanging process, but a random process
crash in place of a wait of a few microseconds for the entropy buffer
to fill up again not so much.

If we could predict whether the call was going to block for a
microsecond, or for 20 minutes, I'd be OK with an exception for the
latter case. But we can't predict the future, so unless the system
call is guaranteed not to block except at system startup, then I
prefer blocking over an exception.

As for blocking vs returning less random results, I defer to others on that.

On 9 June 2016 at 18:14, Steven D'Aprano  wrote:
> On Thu, Jun 09, 2016 at 12:39:00PM -0400, Donald Stufft wrote:
>
>> There are three options for what do with os.urandom by default:
>>
>> * Allow it to silently return data that may or may not be
>> cryptographically secure based on what the state of the urandom pool
>> initialization looks like.
>
> Just to be clear, this is only an option on Linux, right? All the other
> major platforms block, whatever we decide to do on Linux. Including
> Windows?

That's what I understood, certainly. But the place where this was an
issue in real life was a Python program being run during the startup
sequence of the OS. That's never going to be possible on Windows, so
I'd be cautious about drawing parallels with Windows in this situation
(blocking on Windows may be fine because Python can never run when
Windows could possibly have low entropy available).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Thu, Jun 09, 2016 at 12:39:00PM -0400, Donald Stufft wrote:

> There are three options for what do with os.urandom by default:
> 
> * Allow it to silently return data that may or may not be 
> cryptographically secure based on what the state of the urandom pool 
> initialization looks like.

Just to be clear, this is only an option on Linux, right? All the other 
major platforms block, whatever we decide to do on Linux. Including 
Windows?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ben Leslie]

On 9 June 2016 at 12:39, Donald Stufft  wrote:
>
>> On Jun 9, 2016, at 12:30 PM, Steven D'Aprano  wrote:
>>
>>>
>>> os.urandom
>>> --
>> [...]
>>> With that in mind, I think that we should, to the best of our ability given 
>>> the
>>> platform we're on, ensure that os.urandom does not return bytes that the OS
>>> does not think is cryptographically secure.
>>
>> Just to be clear, you're talking about having it block rather than raise
>> an exception, right?
>>
>> If so, that makes sense to me. That's already the behaviour on all major
>> platforms except Linux, so you're just bringing Linux into line with the
>> others. Those who want the non-blocking behaviour on Linux can just read
>> from /dev/urandom.
>
>
> There are three options for what do with os.urandom by default:
>
> * Allow it to silently return data that may or may not be cryptographically 
> secure based on what the state of the urandom pool initialization looks like.
> * Raise an exception if we determine that the pool isn’t initialized enough 
> to get secure random from it.
> * Block until the pool is initialized.
>
> Historically Python has done the first option on Linux (but not on other OSs) 
> because that was simply the only interface that Linux offered at all. In 
> 3.5.0 Victor changed the way os.urandom worked in a way that made it use the 
> third option (he wasn’t attempting to change the security properties, just 
> avoid using an FD, but it improved the security properties as well).
>
> My opinion is that blocking is slightly better than raising an exception 
> because it matches what other OSs do, but that both blocking and raising an 
> exception is better than silently giving data that may or may not be 
> cryptographically secure.

I think an exception is much easier for a user to deal with from a
practical point of view. Trying to work out why a process has hung is
obviously possible, but not necessarily easy.

Having a process crash due to an exception is very easy to diagnose by
comparison.

Cheers,

Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 12:30 PM, Steven D'Aprano  wrote:
> 
>> 
>> os.urandom
>> --
> [...]
>> With that in mind, I think that we should, to the best of our ability given 
>> the
>> platform we're on, ensure that os.urandom does not return bytes that the OS
>> does not think is cryptographically secure.
> 
> Just to be clear, you're talking about having it block rather than raise 
> an exception, right?
> 
> If so, that makes sense to me. That's already the behaviour on all major 
> platforms except Linux, so you're just bringing Linux into line with the 
> others. Those who want the non-blocking behaviour on Linux can just read 
> from /dev/urandom.


There are three options for what do with os.urandom by default:

* Allow it to silently return data that may or may not be cryptographically 
secure based on what the state of the urandom pool initialization looks like.
* Raise an exception if we determine that the pool isn’t initialized enough to 
get secure random from it.
* Block until the pool is initialized.

Historically Python has done the first option on Linux (but not on other OSs) 
because that was simply the only interface that Linux offered at all. In 3.5.0 
Victor changed the way os.urandom worked in a way that made it use the third 
option (he wasn’t attempting to change the security properties, just avoid 
using an FD, but it improved the security properties as well).

My opinion is that blocking is slightly better than raising an exception 
because it matches what other OSs do, but that both blocking and raising an 
exception is better than silently giving data that may or may not be 
cryptographically secure.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Steven D'Aprano]

On Thu, Jun 09, 2016 at 08:26:20AM -0400, Donald Stufft wrote:

> random.py
> -
> 
> In the abstract it doesn't hurt to seed MT with a CSPRNG, it just doesn't
> provide much (if any) benefit and in this case it is hurting us because of the
> cost on import (which will exist on other platforms as well no matter what we
> do here for Linux). There are a couple solutions to this problem:
> 
> * Use getrandom(GRND_NONBLOCK) for random.Random since it doesn't matter if we
>   get cryptographically secure random numbers or not.

+1 on this option (see below for rationale).

 
> * Switch it to use something other than a CSPRNG by default since it doesn't
>   need that.
[...]
> Between these options, I have a slight preference for switching it to use a 
> non
> CSPRNG, but I really don't care that much which of these options we pick. 
> Using
> random.Random is not secure and none of the above options meaningfully change
> the security posture of something that accidently uses it.

I don't think that is quite right, although it will depend on your 
definition of "meaningful".

PEP 506 says:

Demonstrated attacks against MT are typically against PHP 
applications. It is believed that PHP's version of MT is a 
significantly softer target than Python's version, due to
a poor seeding technique [17] . 

https://www.python.org/dev/peps/pep-0506/#id17

specifically that PHP seeds the MT with the time, while we use the 
output of a CSPRNG. Now, we all agree that MT is completely the wrong 
thing to use for secrets, good seeding or not, but *bad* seeding could 
make it a PHP-level soft target.

The point of PEP 506 is to move people away from using random.Random for 
their secrets, but we should expect that whatever we do, there will be 
some late adopters who are slow to get the message and continue to use 
it. I would not like us to weaken the seeding technique to the point 
that those folks become an attractive target.

I think that using getrandom(GRND_NONBLOCK) will be okay, provided that 
when the entropy pool is too low and getrandom falls back to something 
cryptographically weak, it's still better (hopefully significantly 
better) than seeding with the time.

My reasoning is that the sort of applications that could be targets of 
attacks against MT are unlikely to be started up early in the boot 
process, so they're almost always going to get good crypto seeds. On the 
rare occasion that they don't, well, there's only so far that I'm 
prepared to stand up for developer's right to be ignorant of security 
concerns in 2016, and that's where I draw the line.



> SipHash and the Interpreter Startup
> ---
[...]
> In the end, both of these choices make me happy and unhappy in different ways
> but I would lean towards adding a CLI flag for the special case and letting 
> the
> systemd script that caused this problem invoke their Python with that flag. I
> think this because:
> 
> * It leaves the interpreter so that it is secure by default, but provides the
>   relevant knobs to turn off this default in cases where a user doesn't need
>   or want it.
> * It solves the problem in a cross platform way, that doesn't rely on the
>   nuances of the CSPRNG interface on one particular supported platform.

Makes sense to me.

+1


> os.urandom
> --
[...]
> With that in mind, I think that we should, to the best of our ability given 
> the
> platform we're on, ensure that os.urandom does not return bytes that the OS
> does not think is cryptographically secure.

Just to be clear, you're talking about having it block rather than raise 
an exception, right?

If so, that makes sense to me. That's already the behaviour on all major 
platforms except Linux, so you're just bringing Linux into line with the 
others. Those who want the non-blocking behaviour on Linux can just read 
from /dev/urandom.

+1


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Terry Reedy]

On 6/9/2016 9:48 AM, Doug Hellmann wrote:



On Jun 9, 2016, at 9:27 AM, Cory Benfield 
wrote:



The problem here is that both definitions of ‘broken’ are unclear.
If we leave os.urandom() as it is, there is a small-but-nonzero
change that your program will hang, potentially indefinitely. If we
change it back, there is a small-but-nonzero chance your program
will generate you bad random numbers.

If we assume, for a moment, that os.urandom() doesn’t get called
during Python startup (that is that we adopt Christian’s approach
to deal with random and SipHash as separate concerns), what we’ve
boiled down to is: your application called os.urandom() so early
that you’ve got weak random numbers, does it hang or proceed? Those
are literally our two options.


I agree those are the two options. I want the application developer
to make the choice, not us.


I think the 'new API' should be a parameter, not a new function. With 
just two choices, 'wait' = True/False  could work.  If 'raise an 
exception' were added, then

'action (when good bits are not immediately available' =
'return (best possible)' or
'wait (until have good bits)' or
'raise (CryptBitsNotAvailable)'

In either case, there would then be the question of whether the default 
should match 3.5.0/1 or 3.4 and before.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

To expand on my idea of printing a warning, in 3.6 we could add a new
Warning exception for this purpose, so you'd have command-line control over
the behavior of os.urandom() by specifying -WXXX on your Python command
line. For 3.5.2 that's too fancy though -- we can't add a new exception.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Ethan Furman]

On 06/09/2016 04:25 AM, Larry Hastings wrote:


A problem has surfaced just this week in 3.5.1.  Obviously this is a
good time to fix it for 3.5.2.  But there's a big argument over what is
"broken" and what is an appropriate "fix".


Having read the thread thus far, here is my take on fixing it:

- Modify os.urandom() to raise an exception instead of blocking.
  Everyone seems to agree that this is a rare corner case, and
  being rare it would be easier (at least for me) to troubleshoot
  an exception instead of a VM (or whatever) hanging and then being
  killed.

- Add a CLI knob to not raise, but instead wait for initialization.
  I think this should be under the control of the user, who knows
  (or should) the environment that Python is running under, and not
  the developer who may have never dreamed his/her little script
  would be called first thing during bootup.  Maybe we just continue
  to use the hash seed parameter for this.

- Modify the functions that don't need cryptographically strong random
  bits to use the old style (reading directly from /dev/urandom?).

This seems like it should appease the security folks, yet still allow 
those in the trenches to (more) easily diagnose and work around the problem.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 11:52 AM, Guido van Rossum  wrote:
> 
> Wow. I have to decide an issue on which lots of people I respect disagree 
> strongly. So no matter how I decide some of you are going to hate me. Oh 
> well. :-(
> 
> So let's summarize the easy part first. It seems that there is actually 
> agreement that for the initialization of hash randomization and for the 
> random module's Mersenne Twister initialization it is not worth waiting.
> 
> That leaves direct calls to os.urandom(). I don't think this should block 
> either.

To be clear, it’s going to block until urandom has been initialized on most non 
Linux OSs, so either way if the requirement of someone calling os.urandom is 
“must never block”, then they can’t use os.urandom on most non Linux systems. 

> 
> I'm not a security expert. I'm not really an expert in anything. But I often 
> have a good sense for what users need or want. In this case it's clear what 
> users want: they don't want Python to hang waiting for random numbers.
> 
> Take an example from asyncio. If os.urandom() could block, then an ayncio 
> coroutine that wants to call it would have to move that call to a separate 
> thread using loop.run_in_executor() and await the resulting Future, just to 
> avoid blocking all I/O. But you can't test such code, because in practice 
> when you're there to test it, it will never block anyway. So nobody will 
> write it that way, and everybody's code will have a subtle bug (i.e. a 
> coroutine may block without letting other coroutines run). And it's not just 
> bare calls to os.urandom() -- it's any call to library code that might call 
> os.urandom(). Who documents whether their library call uses os.urandom()? 
> It's unknowable. And therein lies madness.
> 
> The problem with security experts is that they're always right when they say 
> you shouldn't do something. The only truly secure computer is one that's 
> disconnected and buried 6 feet under the ground. There's always a scenario 
> through which an attacker could exploit a certain behavior. And there's 
> always the possibility that the computer that's thus compromised is guarding 
> a list of Chinese dissidents, or a million credit card numbers, or the key 
> Apple uses to sign iPhone apps. But much more likely it just has my family 
> photos and 100 cloned GitHub projects.
> 
> And the only time when os.urandom() is going to block on me is probably when 
> I'm rebooting a development VM and wondering why it's so slow.
> 
> Maybe we can put in a warning when getrandom(..., GRND_NONBLOCK) returns 
> EAGAIN? And then award a prize to people who can make it print that warning. 
> Maybe we'll find a way to actually test this code.
> 
> -- 
> --Guido van Rossum (python.org/~guido )


—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Guido van Rossum]

Wow. I have to decide an issue on which lots of people I respect disagree
strongly. So no matter how I decide some of you are going to hate me. Oh
well. :-(

So let's summarize the easy part first. It seems that there is actually
agreement that for the initialization of hash randomization and for the
random module's Mersenne Twister initialization it is not worth waiting.

That leaves direct calls to os.urandom(). I don't think this should block
either.

I'm not a security expert. I'm not really an expert in anything. But I
often have a good sense for what users need or want. In this case it's
clear what users want: they don't want Python to hang waiting for random
numbers.

Take an example from asyncio. If os.urandom() could block, then an ayncio
coroutine that wants to call it would have to move that call to a separate
thread using loop.run_in_executor() and await the resulting Future, just to
avoid blocking all I/O. But you can't test such code, because in practice
when you're there to test it, it will never block anyway. So nobody will
write it that way, and everybody's code will have a subtle bug (i.e. a
coroutine may block without letting other coroutines run). And it's not
just bare calls to os.urandom() -- it's any call to library code that might
call os.urandom(). Who documents whether their library call uses
os.urandom()? It's unknowable. And therein lies madness.

The problem with security experts is that they're always right when they
say you shouldn't do something. The only truly secure computer is one
that's disconnected and buried 6 feet under the ground. There's always a
scenario through which an attacker could exploit a certain behavior. And
there's always the possibility that the computer that's thus compromised is
guarding a list of Chinese dissidents, or a million credit card numbers, or
the key Apple uses to sign iPhone apps. But much more likely it just has my
family photos and 100 cloned GitHub projects.

And the only time when os.urandom() is going to block on me is probably
when I'm rebooting a development VM and wondering why it's so slow.

Maybe we can put in a warning when getrandom(..., GRND_NONBLOCK) returns
EAGAIN? And then award a prize to people who can make it print that
warning. Maybe we'll find a way to actually test this code.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Colm Buckley]

Larry Hastings wrote:

On 3.4 and before, on Linux, os.urandom() would never block, but if the
> entropy pool was uninitialized it could return very-very-poor-quality
> random bits.  On 3.5.0 and 3.5.1, on Linux, when using the getrandom()
> call, it will instead block for an apparently unbounded period before
> returning high-quality random bits.


Just a point of information here. Ted Ts'o commented on the quality of the
pre-initialization bits; it's not a given that they're "very very poor
quality". Even before the per-boot entropy pool is initialized, the kernel
has a few sources of randomness available to it - viz: interrupt timings,
RDRAND (on x86) and a little per-machine data (uname -a). If RDRAND is
trusted, this is enough to provide quite significant entropy, however
that's not much help to all the ARM devices out there.

The most pressing issue from my perspective is the hash randomization
initialization; as there is currently nothing a script author can do to
influence its behavior (except setting PYTHONHASHSEED before invocation,
which might not be an option).

It should be possible, at least conceptually, for Python to be used to
implement /sbin/init. This isn't currently the case on Linux with Python
3.5.1 and Linux 3.17+

For what it's worth, I do agree with Larry that os.urandom() should hew as
closely as possible to the OS-specific urandom implementation. Adding an
optional "blocking" boolean flag might be a useful addition for 3.6.

Colm


-- 
Colm Buckley / c...@tuatha.org / +353 87 2469146
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 14:48, Doug Hellmann  wrote:
> 
> I agree those are the two options. I want the application developer to make 
> the choice, not us.

Right, but right now those two options aren’t available: only one of them is. 
And one way or another we’re taking an action here: either we’re leaving 
os.urandom() as it stands now, or reverting it back to the way it was in 3.4.0.

This means that you *do* want python-dev to make a choice: specifically, you 
want python-dev to make the choice that was made in 3.4.0, rather than the one 
that was made in 3.5.0. That’s fine, but we shouldn’t be pretending that either 
side is arguing for inaction or the status quo for Python 3.5 a choice was made 
with insufficient knowledge of the outcomes, and now we’re arguing about 
whether we can revert that choice. The difference is, now we *do* know about 
both outcomes, which means we are consciously choosing between them.

> All of which fails to be backwards compatible (new exceptions and hanging 
> behavior), which means you’re breaking apps.

Backwards compatible with what? Python 3.5.0 and 3.5.1 both have this 
behaviour, so I assume you mean “backward compatible with 3.4”. However, part 
of the point of a major release is that it doesn’t have to be backward 
compatible in this manner: Python breaks backward compatibility all the time in 
major releases.

I should point out that as far as I'm aware there are exactly two applications 
that suffer from this problem. One of them is Debian’s autopkgtest, which has 
resolved this problem by invoking Python with PYTHONHASHSEED=0. The other is 
systemd-cron, and frankly it does not seem at all unreasonable to suggest that 
perhaps systemd-cron should *maybe* hold off until the system’s CSPRNG gets 
seeded before it starts executing.

Cory


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 9:48 AM, Doug Hellmann  wrote:
> 
> All of which fails to be backwards compatible (new exceptions and hanging 
> behavior), which means you’re breaking apps. Introducing a new API lets the 
> developers who care about strong random values use them without breaking 
> anyone else.


I assert that the vast bulk of users of os.urandom are using it because they
care about strong random values, not because they care about the nuances of
it's behavior on Linux. You're suggesting that almost every [1] single use of
os.urandom in the wild should switch to this new API. Forcing the multitudes to
adapt for the minority is just pointless churn and pain. Besides, Python has
never held backwards compatibility sacred above all else and regularly breaks
it in X.Y+1 releases when there is good reason to do so. Just yesterday there
was discussion on removing bytes(n) from Python 3.x not because it's dangerous
in any way, but because it's behavior makes it slightly confusing in an
extremely obvious way in a PEP that appears like it has a reasonably good
chance of being accepted. 


[1] I would almost go as far as to call it every single use, but I'm sure
someone can dig up one person somewhere who purposely used this behavior. 

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Doug Hellmann]

> On Jun 9, 2016, at 9:27 AM, Cory Benfield  wrote:
> 
> 
>> On 9 Jun 2016, at 13:53, Doug Hellmann  wrote:
>> 
>> I agree with David. We shouldn't break existing behavior in a way
>> that might lead to someone else's software being unusable.
> 
> What does ‘usable’ mean? Does it mean “the code must execute from beginning 
> to end”? Or does it mean “the code must maintain the expected invariants”? If 
> it’s the second, what reasonably counts as “the expected invariants”?

The code must not cause the user’s computer to completely freeze in a way that 
makes their VM appear to be failing to boot?

> 
> The problem here is that both definitions of ‘broken’ are unclear. If we 
> leave os.urandom() as it is, there is a small-but-nonzero change that your 
> program will hang, potentially indefinitely. If we change it back, there is a 
> small-but-nonzero chance your program will generate you bad random numbers.
> 
> If we assume, for a moment, that os.urandom() doesn’t get called during 
> Python startup (that is that we adopt Christian’s approach to deal with 
> random and SipHash as separate concerns), what we’ve boiled down to is: your 
> application called os.urandom() so early that you’ve got weak random numbers, 
> does it hang or proceed? Those are literally our two options.

I agree those are the two options. I want the application developer to make the 
choice, not us.

> 
> These two options can be described a different way. If you didn’t actually 
> need strong random numbers but were affected by the hang, that program failed 
> obviously, and it failed closed. You *will* notice that your program didn’t 
> start up, you’ll investigate, and you’ll take action. On the other hand, if 
> you need strong random numbers but were affected by os.urandom() returning 
> bad random numbers, you almost certainly will *not* notice, and your program 
> will have failed *open*: that is, you are exposed to a security risk, and you 
> have no way to be alerted to that fact.
> 
> For my part, I think the first failure mode is *vastly* better than the 
> second, even if the first failure mode affects vastly more people than the 
> second one does. Failing early, obviously, and safely is IMO much, much 
> better than failing late, silently, and dangerously.
> 
> I’d argue that all the security disagreements that happen in this list boil 
> down to weighting that differently. For my part, I want code that expects to 
> be used in a secure context to fail *as loudly as possible* if it is unable 
> to operate securely. And for that reason:
> 
>> Adding a new API that does block allows anyone to call that when
>> they want guaranteed random values, and the decision about whether
>> to block or not can be placed in the application developer's hands.
> 
> I’d rather flip this around. Add a new API that *does not* block. Right now, 
> os.urandom() is trying to fill two niches, one of which is security focused. 
> I’d much rather decide that os.urandom() is the secure API and fail as loudly 
> as possible when people are using it insecurely than to decide that 
> os.urandom() is the *insecure* API and require changes.
> 
> This is because, again, people very rarely notice this kind of new API 
> introduction unless their code explodes when they migrate. If you think you 
> can find a way to blow up the secure crypto code only, I’m willing to have 
> that patch too, but otherwise I really think that those who expect this code 
> to be safe should be prioritised over those who expect it to be 100% 
> available.
> 
> My ideal solution: change os.urandom() to throw an exception if the kernel 
> CSPRNG is not seeded, and add a new function for saying you don’t care if the 
> CSPRNG isn’t seeded, with all the appropriate “don’t use this unless you’re 
> sure” warnings on it.

All of which fails to be backwards compatible (new exceptions and hanging 
behavior), which means you’re breaking apps. Introducing a new API lets the 
developers who care about strong random values use them without breaking anyone 
else.

Doug
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 13:53, Doug Hellmann  wrote:
> 
> I agree with David. We shouldn't break existing behavior in a way
> that might lead to someone else's software being unusable.

What does ‘usable’ mean? Does it mean “the code must execute from beginning to 
end”? Or does it mean “the code must maintain the expected invariants”? If it’s 
the second, what reasonably counts as “the expected invariants”?

The problem here is that both definitions of ‘broken’ are unclear. If we leave 
os.urandom() as it is, there is a small-but-nonzero change that your program 
will hang, potentially indefinitely. If we change it back, there is a 
small-but-nonzero chance your program will generate you bad random numbers.

If we assume, for a moment, that os.urandom() doesn’t get called during Python 
startup (that is that we adopt Christian’s approach to deal with random and 
SipHash as separate concerns), what we’ve boiled down to is: your application 
called os.urandom() so early that you’ve got weak random numbers, does it hang 
or proceed? Those are literally our two options.

These two options can be described a different way. If you didn’t actually need 
strong random numbers but were affected by the hang, that program failed 
obviously, and it failed closed. You *will* notice that your program didn’t 
start up, you’ll investigate, and you’ll take action. On the other hand, if you 
need strong random numbers but were affected by os.urandom() returning bad 
random numbers, you almost certainly will *not* notice, and your program will 
have failed *open*: that is, you are exposed to a security risk, and you have 
no way to be alerted to that fact.

For my part, I think the first failure mode is *vastly* better than the second, 
even if the first failure mode affects vastly more people than the second one 
does. Failing early, obviously, and safely is IMO much, much better than 
failing late, silently, and dangerously.

I’d argue that all the security disagreements that happen in this list boil 
down to weighting that differently. For my part, I want code that expects to be 
used in a secure context to fail *as loudly as possible* if it is unable to 
operate securely. And for that reason:

> Adding a new API that does block allows anyone to call that when
> they want guaranteed random values, and the decision about whether
> to block or not can be placed in the application developer's hands.

I’d rather flip this around. Add a new API that *does not* block. Right now, 
os.urandom() is trying to fill two niches, one of which is security focused. 
I’d much rather decide that os.urandom() is the secure API and fail as loudly 
as possible when people are using it insecurely than to decide that 
os.urandom() is the *insecure* API and require changes.

This is because, again, people very rarely notice this kind of new API 
introduction unless their code explodes when they migrate. If you think you can 
find a way to blow up the secure crypto code only, I’m willing to have that 
patch too, but otherwise I really think that those who expect this code to be 
safe should be prioritised over those who expect it to be 100% available.

My ideal solution: change os.urandom() to throw an exception if the kernel 
CSPRNG is not seeded, and add a new function for saying you don’t care if the 
CSPRNG isn’t seeded, with all the appropriate “don’t use this unless you’re 
sure” warnings on it.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 8:53 AM, Doug Hellmann  wrote:
> 
> Excerpts from R. David Murray's message of 2016-06-09 08:41:01 -0400:
>> On Thu, 09 Jun 2016 13:12:22 +0100, Cory Benfield  wrote:
>>> The Linux kernel can���t change this stuff easily because they mustn���t
>>> break userspace. Python *is* userspace, we can do what we like, and we
>> 
>> I don't have specific input on the rest of this discussion, but I disagree
>> strongly with this statement.  The environment in which python programs
>> run, ie: the python runtime and standard library, are *our* "userspace",
>> and the same constraints apply to our making changes there as apply
>> to the linux kernel and its userspace...even though we knowingly break
>> those constraints from time to time[*].
>> 
>> --David
>> 
>> [*] Which I think the twisted folks at least would argue we shouldn't
>> be doing :)
> 
> I agree with David. We shouldn't break existing behavior in a way
> that might lead to someone else's software being unusable.
> 
> Adding a new API that does block allows anyone to call that when
> they want guaranteed random values, and the decision about whether
> to block or not can be placed in the application developer's hands.
> 

I think this is a terrible compromise. The new API is going to be exactly the
same as the old API in 99.% of cases and it's fighting against the entire
software ecosystem's suggestion of what to use ("use urandom" is basically a
meme at this point). This is like saying that we can't switch to verifying
HTTPS by default because a one in a million connection might have different
behavior instead of being silently insecure.


—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Doug Hellmann]

Excerpts from R. David Murray's message of 2016-06-09 08:41:01 -0400:
> On Thu, 09 Jun 2016 13:12:22 +0100, Cory Benfield  wrote:
> > The Linux kernel can���t change this stuff easily because they mustn���t
> > break userspace. Python *is* userspace, we can do what we like, and we
> 
> I don't have specific input on the rest of this discussion, but I disagree
> strongly with this statement.  The environment in which python programs
> run, ie: the python runtime and standard library, are *our* "userspace",
> and the same constraints apply to our making changes there as apply
> to the linux kernel and its userspace...even though we knowingly break
> those constraints from time to time[*].
> 
> --David
> 
> [*] Which I think the twisted folks at least would argue we shouldn't
> be doing :)

I agree with David. We shouldn't break existing behavior in a way
that might lead to someone else's software being unusable.

Adding a new API that does block allows anyone to call that when
they want guaranteed random values, and the decision about whether
to block or not can be placed in the application developer's hands.

Christian's points about separating the various cases and solutions also
make sense.

Doug
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[R. David Murray]

On Thu, 09 Jun 2016 13:12:22 +0100, Cory Benfield  wrote:
> The Linux kernel can’t change this stuff easily because they mustn’t
> break userspace. Python *is* userspace, we can do what we like, and we

I don't have specific input on the rest of this discussion, but I disagree
strongly with this statement.  The environment in which python programs
run, ie: the python runtime and standard library, are *our* "userspace",
and the same constraints apply to our making changes there as apply
to the linux kernel and its userspace...even though we knowingly break
those constraints from time to time[*].

--David

[*] Which I think the twisted folks at least would argue we shouldn't
be doing :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 7:25 AM, Larry Hastings  wrote:
> 
> 6) Guido and Tim Peters already decided once that os.urandom() should behave 
> like /dev/urandom.
> 
> Issue #25003:
> http://bugs.python.org/issue25003 
To be exceedingly clear, in this issue the problem wasn’t that os.urandom was
blocking once, early on in the boot process before the kernel had initialized
it’s urandom pool. The problem was that the getentropy() function on Solaris
behaves more like /dev/random does on Linux. This behavior is something that
myself, and most security experts/cryptographers that I know of, think is bad
behavior (and indeed, most OSs have gotten rid of this behavior of /dev/random
and made /dev/random and /dev/urandom behave the same... except again for
Linux).

The ask here isn't to make Linux behave like Solaris did in that issue, it's to
use the newer, better, interface to make Linux use the more secure behavior
that most (all?) of the other modern OSs have already adopted.

—
Donald Stufft



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Donald Stufft]

> On Jun 9, 2016, at 7:25 AM, Larry Hastings  wrote:
> 
> A problem has surfaced just this week in 3.5.1.  Obviously this is a good 
> time to fix it for 3.5.2.  But there's a big argument over what is "broken" 
> and what is an appropriate "fix".


Couple clarifications:

random.py
-

In the abstract it doesn't hurt to seed MT with a CSPRNG, it just doesn't
provide much (if any) benefit and in this case it is hurting us because of the
cost on import (which will exist on other platforms as well no matter what we
do here for Linux). There are a couple solutions to this problem:

* Use getrandom(GRND_NONBLOCK) for random.Random since it doesn't matter if we
  get cryptographically secure random numbers or not.

* Switch it to use something other than a CSPRNG by default since it doesn't
  need that.

* Instead of seeding itself from os.urandom on import, have it lazily do that
  the first time one of the random.rand* functions are called.

* Do nothing, and say that ``import random`` relies on having the kernel's
  urandom pool initialized.

Between these options, I have a slight preference for switching it to use a non
CSPRNG, but I really don't care that much which of these options we pick. Using
random.Random is not secure and none of the above options meaningfully change
the security posture of something that accidently uses it.


SipHash and the Interpreter Startup
---

I have complicated thoughts on what SipHash should do. For something like, a
Django process, we never want it to be initialized with “bad” entropy, however
reading straight from /dev/urandom, or getrandom(GRND_NONBLOCK) means that we
might get that if we start the process early enough in the boot process. The
rub here is that I cannot think of a situation where by the time you’re at the
point you’re starting up something like Django, you’re even remotely likely to
not have an initialized random pool. The other side of this issue is that we
have Python scripts which do not need a secure random being passed to SipHash
running early enough in the boot process with systemd that we need to be able
to have SipHash initialization not block on waiting for /dev/urandom.

So I’m torn between the “Practicality beats Purity” mindset, which says we
should just let SipHash seed itself with whatever quality of random from the
urandom pool is currently available and the “Special cases aren’t special
enough to break the rules” mindset which says that we should just make it
easier for scripts in this edge case to declare they don’t care about hash
randomization to remove the need for it (in other words, a CLI flag that
matches PYTHONHASHSEED in functionality). An additional wrinkle in the mix is
that we cannot get non-blocking random on many (any?) modern OS besides Linux,
so we're going to run into this same problem if say, FreeBSD decides to put a
Python script early enough in the boot sequence.

In the end, both of these choices make me happy and unhappy in different ways
but I would lean towards adding a CLI flag for the special case and letting the
systemd script that caused this problem invoke their Python with that flag. I
think this because:

* It leaves the interpreter so that it is secure by default, but provides the
  relevant knobs to turn off this default in cases where a user doesn't need
  or want it.
* It solves the problem in a cross platform way, that doesn't rely on the
  nuances of the CSPRNG interface on one particular supported platform.


os.urandom
--

There have been a lot of proposals thrown around, and people pointing to
different sections of the documentation to justify different opinions. This is
easily the most contentious question we have here.

It is my belief that reading from urandom is the right thing to do for
generating cryptographically secure random numbers. This is a view point held
by every major security expert and cryptographer that I'm aware of. Most (all?)
major platforms besides Linux do not allow reading from their equivalent of
/dev/urandom until it has been successfully initialized and it is widely held
by all security experts and cryptographers that I'm aware of that this property
is a good one, and the Linux behavior of /dev/urandom is a wart/footgun but
that prior to getrandom() there simply wasn't a better option on Linux.

With that in mind, I think that we should, to the best of our ability given the
platform we're on, ensure that os.urandom does not return bytes that the OS
does not think is cryptographically secure.

In practice this means that os.urandom should do one of two things in the very
early boot process on Linux:

* Block waiting on the kernel to initialize the urandom pool, and then return
  the now secure random bytes given to us.
* Raise an exception saying that the pool has not been initialized and thus
  os.urandom is not ready yet.

The key point in both of these options is that os.urandom never [1] returns
bytes 

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Cory Benfield]

> On 9 Jun 2016, at 12:54, Christian Heimes  wrote:
> 
> Therefore I propose to fix problem 2 and 3:
> 
> - add a new random_seed member to _Py_HashSecret and use it to derive an
> initial Mersenne-Twister state for the default random instance of the
> random module.
> 
> - try CPRNG for _Py_HashSecret first, fall back to a user space RNG when
> the Kernel's CPRNG would block.
> 
> For some operating systems like Windows and OSX, we can assume that
> Kernel CPRNG is always available. For Linux we can use getrandom() in
> non-blocking mode and handle EWOULDBLOCK. On BSD the seed state can be
> queried from /proc.
> 

I am in agreement with Christian here.

Let me add: Larry has suggested that it’s ok that os.urandom() can degrade to 
weak random numbers in part because "os.urandom() doesn't actually guarantee 
it's suitable for cryptography.” That’s true, that is what the documentation 
says.

However, that documentation has been emphatically disagreed with by the entire 
Python ecosystem *including* the Python standard library. Both 
random.SystemRandom and the secrets module use os.urandom() to generate their 
random numbers. The secrets module says this right at the top:

"The secrets module is used for generating cryptographically strong random 
numbers suitable for managing data such as passwords, account authentication, 
security tokens, and related secrets.”

Regressing the behaviour in os.urandom() would mean that this statement is not 
unequivocally true but only situationally true. It would be more accurate to 
say “The secrets module should generate cryptographically strong random numbers 
most of the time”. So I’d argue that while os.urandom() does not make these 
promises, the rest of the standard library behaves like it does. While we’re 
here I should note that the cryptography project unequivocally recommends 
os.urandom[0], and that this aspect of Linux’s /dev/urandom behaviour is 
considered to be a dangerous misfeature by almost everyone in the crypto 
community.

The Linux kernel can’t change this stuff easily because they mustn’t break 
userspace. Python *is* userspace, we can do what we like, and we should be 
aiming to make sure that doing the obvious thing in Python amounts to doing the 
*right* thing. *Obviously* this shouldn’t block startup, and obviously we 
should fix that, but I disagree that we should be reverting the change to 
os.urandom().

Cory

[0]: https://cryptography.io/en/latest/random-numbers/



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Christian Heimes]

On 2016-06-09 13:25, Larry Hastings wrote:
> 
> A problem has surfaced just this week in 3.5.1.  Obviously this is a
> good time to fix it for 3.5.2.  But there's a big argument over what is
> "broken" and what is an appropriate "fix".
> 
> As 3.5 Release Manager, I can put my foot down and make rulings, and
> AFAIK the only way to overrule me is with the BDFL.  In two of three
> cases I've put my foot down.  In the third I'm pretty sure I'm right,
> but IIUC literally everyone with a stated opinion else disagrees with
> me.  So I thought it best I escalate it.  Note that 3.5.2 is going to
> wait until the issue is settled and any changes to behavior are written
> and checked in.
> 
> (Blanket disclaimer for the below: in some places I'm trying to
> communicate other's people positions.  I apologize if I misrepresented
> yours; please reply and correct my mistake.  Also, sorry for the length
> of this email.  But feel even sorrier for me: this debate has already
> eaten two days this week.)

Thanks for the digest, Larry.

I would appreciate if we could split the issue into three separate problems:

1) behavior of os.urandom()
2) initialization of _Py_HashSecret for byte, str and XML hash
randomization.
3) initialization of default random.random Mersenne-Twister

As of now 2 and 3 are the culprit for blocking starting. Both happen to
use _PyOS_URandom() either directly or indirectly through os.urandom().
We chose to use the OS random source because it was convenient. It is
not a necessity. The seed for Mersenne-Twister and the keys for hash
randomization don't have to be strong cryptographic values in all cases.
They just have to be hard-to-guess by an attacker. In case of scripts in
early boot, there are no viable attack scenarios.

Therefore I propose to fix problem 2 and 3:

- add a new random_seed member to _Py_HashSecret and use it to derive an
initial Mersenne-Twister state for the default random instance of the
random module.

- try CPRNG for _Py_HashSecret first, fall back to a user space RNG when
the Kernel's CPRNG would block.

For some operating systems like Windows and OSX, we can assume that
Kernel CPRNG is always available. For Linux we can use getrandom() in
non-blocking mode and handle EWOULDBLOCK. On BSD the seed state can be
queried from /proc.

Christian


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Victor Stinner]

I understood that Christian Heimes and/or Donald Stufft are interested
to work on a PEP.

2016-06-09 13:25 GMT+02:00 Larry Hastings :
> A problem has surfaced just this week in 3.5.1.  Obviously this is a good
> time to fix it for 3.5.2.  But there's a big argument over what is "broken"
> and what is an appropriate "fix".

IMHO the bug is now fixed in 3.5.2 as I explained at:
http://haypo-notes.readthedocs.io/pep_random.html#status-of-python-3-5-2


> THE PROBLEM
>
> Issue #26839:
>
> http://bugs.python.org/issue26839
>
> (warning, the issue is now astonishingly long, and exhausting to read, and
> various bits of it are factually wrong)

You may want to read my summary:
http://haypo-notes.readthedocs.io/pep_random.html


I'm not interested to reply to Larry's email point per point. IHMO a
formal PEP is now required for Python 3.6 (to enhance os.urandom and
clarify Python behaviour before urandom is initialized). Python 3.5.2
is fixed, there is no more urgency ;-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Larry Hastings]

A problem has surfaced just this week in 3.5.1.  Obviously this is a 
good time to fix it for 3.5.2.  But there's a big argument over what is 
"broken" and what is an appropriate "fix".


As 3.5 Release Manager, I can put my foot down and make rulings, and 
AFAIK the only way to overrule me is with the BDFL.  In two of three 
cases I've put my foot down.  In the third I'm pretty sure I'm right, 
but IIUC literally everyone with a stated opinion else disagrees with 
me.  So I thought it best I escalate it.  Note that 3.5.2 is going to 
wait until the issue is settled and any changes to behavior are written 
and checked in.


(Blanket disclaimer for the below: in some places I'm trying to 
communicate other's people positions.  I apologize if I misrepresented 
yours; please reply and correct my mistake.  Also, sorry for the length 
of this email.  But feel even sorrier for me: this debate has already 
eaten two days this week.)



BACKGROUND

For 3.5 os.urandom() was changed: instead of reading from /dev/urandom, 
it uses the new system call getrandom() where available.  This is a new 
system call on Linux (which has already been cloned by Solaris).  
getrandom(), as CPython uses it, reads from the same PRNG that 
/dev/urandom gets its bits from.  But because it's a system call you 
don't have to mess around with file handles.  Also it always works in 
chrooted environments.  Sounds like a fine idea.


Also for 3.5, several other places where CPython internally needs random 
bits were switched from reading from /dev/urandom to calling 
getrandom().  The two that I know of: choosing the seed for hash 
randomization, and initializing the default Mersenne Twister for the 
random module.


There's one subtle but important difference between /dev/urandom and 
getrandom().  At startup, Linux seeds the urandom PRNG from the entropy 
pool.  If the entropy pool is uninitialized, what happens? CPython's 
calls to getrandom() will block until the entropy pool is initialized, 
which is usually just a few seconds (or less) after startup.  But 
/dev/urandom *guarantees* that reads will *always* work.  If the entropy 
pool hasn't been initialized, it pulls numbers from the PRNG before it's 
been properly seeded.  What this results in depends on various aspects 
of the configuration (do you have ECC RAM? how long was the machine 
powered down? does the system have a correct realtime clock?).  In 
extreme circumstances this may mean the "random" numbers are shockingly 
predictable!


Under normal circumstances this minor difference is irrelevant. After 
all, when would the entropy pool ever be uninitialized?



THE PROBLEM

Issue #26839:

   http://bugs.python.org/issue26839

(warning, the issue is now astonishingly long, and exhausting to read, 
and various bits of it are factually wrong)


A user reports that when starting CPython soon after startup on a fresh 
virtual machine, the process would hang for a long time. Someone on the 
issue reported observed delays of over 90 seconds. Later we found out: 
it wasn't 90 seconds before CPython became usable, these 90 seconds 
delays were before systemd timed out and simply killed the process.  
It's not clear what the upper bound on the delay might be.


The issue author had already identified the cause: CPython was blocking 
on getrandom() in order to initialize hash randomization. On this fresh 
virtual machine the entropy pool started out uninitialized.  And since 
the only thing running on the machine was CPython, and since CPython was 
blocked on initialization, the entropy pool was initializing very, very 
slowly.


Other posters to the thread pointed out that the same thing would happen 
in "import random", if your code could get that far.  The constructor 
for the Random() object would seed the Mersenne Twister, which would 
call getrandom() and block.


Naturally, callers to os.urandom() could also block for an unbounded 
period for the same reason.



MY RULINGS SO FAR

1) The change in 3.5 that means "import random" may block for an 
unbounded period of time on Linux due to the switch to getrandom() must 
be backed out or amended so that it never blocks.


I *think* everyone agrees with this.  The Mersenne Twister is not a 
CPRNG, so seeding it with crypto-quality bits isn't necessary.  And 
unbounded delays are bad.



2) The change in 3.5 that means hash randomization initialization may 
block for an unbounded period of time on Linux due to the switch to 
getrandom() must be backed out or amended so that it never blocks.


I believe most people agree with me.  The cryptography experts 
disagree.  IIUC both Alex Gaynor and Christian Heimes feel the blocking 
is preferable to non-random hash "randomization".


Yes, the bad random data means the hashing will be predictable. Neither 
choice is exactly what you want.  But most people feel it's simply 
unreasonable that in extreme corner cases CPython can block for an 
unbounded amount of time before running user