[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[Roundup Robot]

Roundup Robot added the comment:

New changeset 835085cc28cd by Victor Stinner in branch '3.5':
Issue #25003: On Solaris 11.3 or newer, os.urandom() now uses the getrandom()
https://hg.python.org/cpython/rev/835085cc28cd

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[Roundup Robot]

Roundup Robot added the comment:

New changeset 202c827f86df by Victor Stinner in branch '2.7':
Issue #25003: os.urandom() doesn't use getentropy() on Solaris because
https://hg.python.org/cpython/rev/202c827f86df

New changeset 83dc79eeaf7f by Victor Stinner in branch '3.4':
Issue #25003: os.urandom() doesn't use getentropy() on Solaris because
https://hg.python.org/cpython/rev/83dc79eeaf7f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

STINNER Victor added the comment:

Ok, I pushed fixes for Python 2.7, 3.4, 3.5 and 3.6.

Summary for Solaris:

- Python 2.7, 3.4: read from /dev/urandom (use a private file descriptor)
- Python 3.5, 3.6: use the getrandom() function on Solaris 11.3 and newer (no 
file descriptor), or fallback on reading from /dev/urandom (use a private file 
descriptor)

getentropy() is no more used on Solaris.

@John Beck: It would be great if you could run test_os on the development 
branches 2.7, 3.4, 3.5 and default (3.6).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[John Beck]

John Beck added the comment:

Confirmed that test_os runs cleanly on Solaris 12, for each of:
* 2.7.10 (plus your patch from 98454:202c827f86df)
* 3.4.3 (plus your patch from 98455:83dc79eeaf7f)
* 3.5.0 (plus your patch from 98452:835085cc28cd)
* 3.6 (tip)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

STINNER Victor added the comment:

Thanks! I close the issue, it's now fixed.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

STINNER Victor added the comment:

"I have tested your patch with 3.5, and it works fine, although I did have to 
make a minor change to get test_os to pass.  In test_os.py you had: (...)"

Oh right, I fixed test_os too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[Roundup Robot]

Roundup Robot added the comment:

New changeset 221e09b89186 by Victor Stinner in branch 'default':
Issue #25003: Skip test_os.URandomFDTests on Solaris 11.3 and newer
https://hg.python.org/cpython/rev/221e09b89186

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[John Beck]

John Beck added the comment:

I have tested your patch with 3.5, and it works fine, although I did have to 
make a minor change to get test_os to pass.  In test_os.py you had:

...
USE_GETENTROPY = ((sysconfig.get_config_var('HAVE_GETENTROPY') == 1)
  and not sys.platform.startswith("sunos"))
HAVE_GETRANDOM = (sysconfig.get_config_var('HAVE_GETRANDOM_SYSCALL') == 1)
 
@unittest.skipIf(USE_GETENTROPY,
 "getentropy() does not use a file descriptor")
@unittest.skipIf(HAVE_GETRANDOM,
 "getrandom() does not use a file descriptor")
...

whereas I came up with this:

...
USE_GETENTROPY = ((sysconfig.get_config_var('HAVE_GETENTROPY') == 1)
  and not sys.platform.startswith("sunos"))
HAVE_GETRANDOM = (sysconfig.get_config_var('HAVE_GETRANDOM') == 1)
HAVE_GETRANDOM_SYSCALL = (sysconfig.get_config_var('HAVE_GETRANDOM_SYSCALL') == 
1)

@unittest.skipIf(USE_GETENTROPY,
 "getentropy() does not use a file descriptor")
@unittest.skipIf(HAVE_GETRANDOM,
 "getrandom() does not use a file descriptor")
@unittest.skipIf(HAVE_GETRANDOM_SYSCALL,
 "getrandom() does not use a file descriptor")
...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

STINNER Victor added the comment:

>  While testing this, I found out why I had needed EINVAL earlier (and still 
> do, for now): there is a bug in the Solaris implementation of getrandom(2).  
> If flags are 0 and the buffer size > 1024, then it fails with EINVAL.  That 
> is only supposed to happen for a buffer that big if GNRD_RANDOM is set in 
> flags but GNRD_NONBLOCK is not set.  So until that bug is fixed, ...

Ah! So it was useful to dig the EINVAL issue, it's a bug in the kernel :-)

You wrote that the final version of Solaris 11.3 and 12 was not released yet. 
Can we expect a fix in the kernel before the final version?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[John Beck]

John Beck added the comment:

I have queried the engineer who owns the kernel bug and will post an update 
once I hear back from him.  But as it is already almost midnight on Friday in 
his geo, it may well be Monday before I hear back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

STINNER Victor added the comment:

> But EINVAL should also be checked for to make sure the system call was 
> invoked with proper parameters.

py_getrandom() calls Py_FatalError() or raises an OSError on EINVAL. The error 
is not silently ignored.


> My builds of Python 3.5.0.X (don't recall whether X was a late Beta or 
> release candidate) were core dumping because Python was making that syscall 
> but not checking for EINVAL,

Py_FatalError() calls abort(). Usually, operating systems dump a core dump in 
this case. But it is the expected behaviour. Python now refuses to start if the 
OS random number generator doesn't work. There are similar checks on the system 
and monotonic clocks for example.

> ... and thus assuming the call was valid, when it was not.

Ah! Finally you explain the problem :-) I wrote py_getrandom() for Linux. I 
didn't expect other operating systems to implement the getrandom() syscall. I 
hardcoded the flags (0) for example.

py_getrandom() calls directly the syscall, because I like the new cool 
getrandom() syscall of Linux: it avoids the need of a private file descriptor. 
It would be much better to call a function of the C library, but the GNU C 
library didn't expose the function yet...

On Solaris, the function is available in C, no need to call directly the 
syscall. It would be better to call the C function, and check if it's available 
in configure.

Can you please try remove_get_entropy.patch + urandom_solaris.patch?

--
Added file: http://bugs.python.org/file40437/getrandom_solaris.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[John Beck]

John Beck added the comment:

Yes, those patches work, with a caveat.  While testing this, I found out why I 
had needed EINVAL earlier (and still do, for now): there is a bug in the 
Solaris implementation of getrandom(2).  If flags are 0 and the buffer size > 
1024, then it fails with EINVAL.  That is only supposed to happen for a buffer 
that big if GNRD_RANDOM is set in flags but GNRD_NONBLOCK is not set.  So until 
that bug is fixed, I have to patch py_getrandom() to treat EINVAL like ENOSYS 
and fall back to using /dev/urandom as if getrandom(2) were not supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris

[STINNER Victor]

Changes by STINNER Victor :


--
title: os.urandom() should call getrandom(2) not getentropy(2) -> os.urandom() 
should call getrandom(2) not getentropy(2) on Solaris

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[STINNER Victor]

STINNER Victor added the comment:

> Yes, the syscall was failing with EINVAL.

Sorry, I don't understand. Do you mean that calling a non-existant syscall on 
Solaris fails with EINVAL? Calling the getrandom() syscall on Solaris < 11.3 
fails with EINVAL?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[STINNER Victor]

Changes by STINNER Victor :


--
title: os.urandom() should call getrandom(2) not getentropy(2) on Solaris -> 
os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and 
newer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2) on Solaris 11.3 and newer

[John Beck]

John Beck added the comment:

Sorry, let me try to clarify.  ENOSYS is the correct errno value to check for, 
for the situation (e.g., Solaris < 11.3) of a non-existent system call. But 
EINVAL should also be checked for to make sure the system call was invoked with 
proper parameters.  My builds of Python 3.5.0.X (don't recall whether X was a 
late Beta or release candidate) were core dumping because Python was making 
that syscall but not checking for EINVAL, and thus assuming the call was valid, 
when it was not.  Sorry, I did not try to debug why it was invalid.  But once I 
added EINVAL, alongside ENOSYS, as a reason to set getrandom_works to 0, then 
python started behaving properly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[John Beck]

John Beck added the comment:

Yes, the syscall was failing with EINVAL.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[STINNER Victor]

STINNER Victor added the comment:

> Prior to applying this patch, I had needed to tweak py_getrandom() to 
> recognize EINVAL in addition to ENOSYS as sufficient reason to turn off 
> getrandom_works.

In which case getrandom() fails with EINVAL?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[John Beck]

John Beck added the comment:

@haypo: Yes, that patch works (applied to 3.5.0rc3; proxy problems are 
preventing me from updating my clone of the default branch at the moment) i.e., 
pyconfig.h shows:

#define HAVE_GETRANDOM_SYSCALL 1

To answer your other questions:
* Calling syscall(SYS_getrandom, buffer, size, 0) does work on Solaris.
* Our  does declare syscall().
* Solaris does not use the GNU C library, but our own libc, which does
  support getrandom(3) as of Solaris 11.3 (which will be released in
  the near future) and Solaris 12 (which is in Beta but has a release
  date in the more distant future).

Prior to applying this patch, I had needed to tweak py_getrandom() to recognize 
EINVAL in addition to ENOSYS as sufficient reason to turn off getrandom_works.  
I still have that tweak in place, but have not yet tried backing it out to see 
if things still behave with your patch.

Let me know if there is any more information I can provide, and thanks for your 
attention to this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[STINNER Victor]

STINNER Victor added the comment:

Thread on python-dev: 
https://mail.python.org/pipermail/python-dev/2015-September/141439.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Guido van Rossum]

Guido van Rossum added the comment:

Also, Theo believes that our Mersenne Twister is outdated and os.urandom() is 
the only reasonable alternative. So we might as well keep it fast.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Guido van Rossum]

Guido van Rossum added the comment:

To Theo it's probably inconceivable that anyone would be using random numbers 
for anything besides crypto security. :-) His favorite is arc4random() (but he 
notes it's not based on RC4 anymore, but uses chacha -- I have no idea what any 
of that means :-). He did say userland PRNG a few times.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Donald Stufft]

Donald Stufft added the comment:

(A)RC4 and ChaCha are just two stream ciphers that let you encrypt some data, 
they work by essentially producing a psuedo-random stream of data in a 
deterministic manner based off of a key, and than that is XOR'd with the data 
you want to encrypt. arc4random (ab)uses this and uses "real" entropy (e.g. 
randomness pulled from random noise on the network and such) as the "key" and 
then uses the psuedo-random stream of data as the values you get when you ask 
arc4random for some random data. The actual process is quite a bit more complex 
then that, but that's the basic gist.

Userspace PRNG's are not a very good idea for reasons better explained by an 
expert: http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

And yea, using MT for anything that needs a CSPRNG (that is, a 
Cryptographically Secure Psuedo Random Number Generator) is a real bad idea, 
because the numbers it outputs are not "really" random. I'm of a mind that the 
APIs should default to CSPRNGs (so ``random`` should default to SystemRandom) 
and using something like MT should be opt in via something like 
"UnsafeFastRandom) or something. That ship is almost certainly sailed at this 
point though.

--
nosy: +dstufft

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Donald Stufft]

Donald Stufft added the comment:

Oh yea, and (A)RC4 is broken and shouldn't be used for anything anymore, ChaCha 
is much better and is pretty great.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Tim Peters]

Tim Peters added the comment:

Guido, you're clearly talking with someone who knows too much ;-)  If we're 
using the Twister for _anything_ related to crypto-level randomness, then I'd 
appalled - it was utterly unsuitable for any such purpose from day 1.  But as a 
general-purpose generator for non-crypto uses, it remains an excellent choice, 
and still a nearly de facto standard for "almost all" such purposes.  There are 
general-purpose generators I like better now, but won't push in that direction 
until the Twister's "nearly de facto standard" status fades.  Better to be a 
follower than a leader in this area.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Guido van Rossum]

Changes by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[John Beck]

New submission from John Beck:

A recent Solaris build upgrade resulted in a massive slowdown of a package 
operation (our package client is written in Python).  Some DTrace revealed this 
was because os.urandom() calls had slowed down by a factor of over 300.  This 
turned out to be caused by an upgrade of Python from 2.7.9 to 2.7.10 which 
included:

- Issue #22585: On OpenBSD 5.6 and newer, os.urandom() now calls getentropy(),
  instead of reading /dev/urandom, to get pseudo-random bytes.

By adding ac_cv_func_getentropy=no to our configure options, we were able to 
get back the performance we had lost.  But our security experts warned:

---

OpenBSD only has getentropy(2) and we are compatible with that.
Linux has both getentropy(2) and getrandom(2)
Solaris has getentropy(2) and getrandom(2).

The bug is in Python it should not use getentropy(2) for the implementation of 
os.urandom() unless it builds its own DRBG (Deterministic Random Bit Generator) 
around that - which will mean it is "caching" and thus only calling 
getentropy() for seeding very infrequently.

You can not substitute getentropy() for getrandom(), if you need randomness 
then entropy does not suffice.

They are using getentropy(2) correctly in the implementation of 
_PyRandom_Init().

I would personally recommend that the upstream adds os.getentropy() changes
os.urandom() to use getrandom(buf, sz, GRND_NONBLOCK) and os.random() to use
getrandom(buf, sz, GRND_RANDOM).

As it is the upstream implementation is arguably a security vulnerability since 
you can not use entropy where you need randomness.
---

where by "upstream" he meant "python.org".  So I am filing this bug to request 
those changes.  I can probably prototype this in the reasonable near future, 
but I wanted to get the bug filed sooner rather than later.

--
components: Interpreter Core
messages: 249837
nosy: jbeck
priority: normal
severity: normal
status: open
title: os.urandom() should call getrandom(2) not getentropy(2)
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25003] os.urandom() should call getrandom(2) not getentropy(2)

[R. David Murray]

Changes by R. David Murray :


--
nosy: +haypo
stage:  -> needs patch
type:  -> security

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com