[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

> so i maintain that this issue is in fact very specific to the random module

I think that you misunderstood the issue. I'm now closing it as a duplicate of 
the issue #26839.

--

Marc-Andre Lemburg: Please continue the discussion on the issue #26839. I 
copied your latest message.

--
resolution:  -> duplicate
status: open -> closed
superseder:  -> python always calls getrandom() at start, causing long hang 
after boot

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

I still believe the underlying system API use should be fixed rather than all 
the different instances where it gets used.

getrandom() should not block. If it does on a platform, that's a bug on that 
platform and Python should revert to the alternative of using /dev/urandom 
directly (or whatever other source of randomness is available).

Disabling hash randomization is not a good workaround for the issue, since it 
will definitely pop up in other areas as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[jan matejek]

jan matejek added the comment:

unlike #26839, however, there is no workaround for "import random".
so i maintain that this issue is in fact very specific to the random module

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

The issue is more general than just "import random", Python reads entropy at 
startup to initialize a random seed for its randomized hash function: see the 
issue #26839.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Raymond Hettinger]

Raymond Hettinger added the comment:

FWIW, the random.patch from matejcik makes me uncomfortable.  It feels like a 
hack that obscures the code, would confound linters and type checkers, and 
would create more problems than it would solve.

--
assignee: rhettinger -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Thomas Petazzoni]

Thomas Petazzoni added the comment:

I can confirm that I'm affected by the same issue. Booting a simple Linux 
system on a Qemu ARM platform, the python startup hangs during 25 seconds due 
to the call to getrandom(). I am not doing anything with Python, just starting 
the Python interpreter:

# strace -t -o strace.log python
random: nonblocking pool is initialized
Python 3.5.0 (default, Dec 23 2015, 15:11:18) 
[GCC 5.1.1 20150608] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>  
# grep -A 2 getrandom strace.log 
14:43:50 
getrandom("\245\362a=\305\32Z\263\364\352j\223\0017\302q\361M\336+\2722>[", 24, 
0) = 24
14:44:35 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
14:44:35 mmap2(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
-1, 0) = 0x76baf000

As you can see, 25 seconds blocked due to the getrandom() system call. Makes 
the Python interpreter not really usable anymore. I would understand if Python 
would do when I need to generate cryptographically secure random numbers. But 
at this point, I am just starting the interpreter, nothing else.

This is a regression from Python 3.4.3.

--
nosy: +thomas-petazzoni

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Thomas Petazzoni]

Thomas Petazzoni added the comment:

Obviously I did my math wrong: it waits 45 seconds in getrandom(), not 25 
seconds. See my strace log.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

getrandom() is used to initialize the randomized hash function. Set
PYTHONHASHSEED env var to not use getrandom() at startup. But the hash
function will not randomized anymore :-/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

I knew the subtle difference between reading from /dev/urandom and
getrandom() syscall: the syscall hangs until /dev/urandom is feeded with
enough entropy. It should be documented in Whats New, os.urandom and maybe
also random doc. Not only python 3.5 was affected by the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Марк Коренберг]

Марк Коренберг added the comment:

Man getrandom() 

As of Linux 3.19, the following bug exists:

   *  Depending on CPU load, getrandom() does not react to interrupts
  before reading all bytes requested.


So, is it goot to use this syscall now?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

Hi,

Марк Коренберг added the comment:
> Man getrandom()
>
> As of Linux 3.19, the following bug exists:
>
>*  Depending on CPU load, getrandom() does not react to interrupts
>   before reading all bytes requested.
>
> So, is it goot to use this syscall now?

I saw a fix proposed on the LKML but it looks like it was not merged.
I don't know what to think about this bug.

getrandom(n, GRND_NONBLOCK) behaviour depends if /dev/urandom was
feeded with enough entropy and the value of n. It should not be
interrupted by signal for n <= 256.

Can you reproduce the bug? Which kind of applications can hang because
of this bug?

I would prefer to continue to use getrandom() syscall on Linux, avoid
using a file descriptor is really useful.

Maybe we can try to document the behaviour of os.urandom() for signal
handling? Or at least redirect users to getrandom() manual page.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

> The RNG is seeded from os.urandom, which as of python 3.5 uses the 
> potentially blocking getrandom() call.

Here is a patch for os.urandom() documentation. What do you think?

--
keywords: +patch
Added file: http://bugs.python.org/file40816/urandom.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[jan matejek]

jan matejek added the comment:

let me reiterate that what I consider a bug is the fact that "import random" 
statement calls os.urandom (which per the proposed documentation may sometimes 
block)

IOW, "import random" may sometimes block, even though it is not actually used 
at any point (could be pulled in through some dependencies)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

I think Jan has a point there. An import should not cause the whole interpreter 
to hang.

Wouldn't it be possible to have the getrandom() call be done lazily to avoid 
this and only have it block when the RNG from the random is actually being used 
?

Or alternatively, make things more robust by avoiding to call the API on 
systems which are known to have blocking problems and then reverting to using 
/dev/urandom directly instead ?

Note that the RNG does already use a fallback solution for systems which don't 
provide os.urandom. Also note that os.urandom() is documented (indirectly via 
man 4 urandom) to not be blocking. If it blocks on some systems, we should add 
a work-around for those, just like Python/random.c does for Solaris.

BTW: Is there a way to determine whether enough entropy has been gathered 
without doing a blocking call ? This could be used to find out whether 
getrandom() will potentially block.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[jan matejek]

jan matejek added the comment:

attaching a first draft of what i'd consider a solution? not sure if this is 
the right way to go, and i don't know how to write a test for an import 
statement

--
Added file: http://bugs.python.org/file40820/random.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[jan matejek]

jan matejek added the comment:

On 18.10.2015 23:09, STINNER Victor wrote:
> Hum ok, so your issue is specific to Linux.

yes, should have specified that, sorry

> Hum, the problem was already fixed some months/years ago: you must attach a 
> RNG virtio device to your VM. Python is just one example, a lot of 
> applications need entropy.

i disagree that this is a good solution; similar to your haveged 
suggestion, this is a workaround.

Unless a program specifically uses randomness, it should not need to 
read any entropy. For the python runtime itself, this is preventable by 
setting fixed PYTHONHASHSEED. For `random` module, there is no clean way 
to prevent it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

If your OS has no entropy at all, you will have much more severe
issue. For example, don't try to generate a SSH key or established a
SSL/TLS session.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[Марк Коренберг]

Марк Коренберг added the comment:

Just install rngd and setup it to user /dev/urandom as entropy source. I think 
thread is closed :)

--
nosy: +mmarkk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25420] "import random" blocks on entropy collection on Linux with low entropy

[STINNER Victor]

STINNER Victor added the comment:

"The RNG is seeded from os.urandom, which as of python 3.5 uses the potentially 
blocking getrandom() call."

Hum ok, so your issue is specific to Linux.

"This causes problems e.g. on our build VMs that don't have true entropy, so 
getrandom() blocks forever"

Hum, the problem was already fixed some months/years ago: you must attach a RNG 
virtio device to your VM. Python is just one example, a lot of applications 
need entropy.

"A possible workaround is to monkeypatch os.urandom (in this particular case, 
to return a string of zeroes and remove randomness entirely to get reproducible 
builds)"

An unsafe *workaround* is to install haveged, a daemon generating entropy using 
the CPU.

--
nosy: +haypo
title: "import random" blocks on entropy collection -> "import random" blocks 
on entropy collection on Linux with low entropy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com