[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Raymond Hettinger added the comment:

Sorry Steven, I'm going to mark this as rejected on the grounds that it is 
likely to do more harm than good.  We could in fact make the range larger but 
it easily creates terrible effects (encouraging bad design and creating a 
non-interruptable, long-running, total-memory-filling call).  

While we do allow ``2 ** 50 ** 50``, that call is more deliberately asking for 
trouble than getrandbits(2**60).  If someone really needed that number of bits, 
it isn't hard to multiple calls to getrandbits() and combine the results, 
deliberately and interruptably.

--
resolution:  -> rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The only case I know that would benefit is generating random data for tests. On 
my computer generating 2*28 bits with getrandbits() takes 2 sec (including 1 
sec for converting from bytes to int), plus 1.4 sec for converting from int to 
bytes. Special API would speed up generating random bytes by 3.4 times. I would 
request for this functionality, but I don't want to clutter the API.

As for getrandbits.diff, I think that we should follow the principle "we're all 
consenting adults here".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Raymond Hettinger added the comment:

> On the other hand, this limit isn't imposed elsewhere.

There are a number of places in the language with these limits.  In general, 
we're opening them up to wider limits if there are valid use cases and if it 
doesn't immediately shoot you in the foot like it does here.

Practicality is a reasonable design consideration.  Mitigation of harm is also 
reasonable design consideration.  Foolish consistently is, well, you know how 
the saying goes :-)

> Might having a method added to `randome.Random` that returns 
> random bits (as os.urandom does), be something Python project
> would consider accepting?

It isn't needed.  As far as I know, there has never been a user request for 
this functionality nor a presentation of use cases that benefit it.  The API is 
already bloated and we already have one way to do it with int.to_bytes().  
Also, we should keep this tracker entry focused on the OP's report and not 
meander.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Campbell Barton]

Campbell Barton added the comment:

@rhettinger, agree that very large ints in this case aren't going to give very 
usable results.

On the other hand, this limit isn't imposed elsewhere (you can power-of 
operator to create bigger numbers).

Nevertheless this isn't going to give good/usable performance.



Might having a method added to `randome.Random` that returns random bits (as 
os.urandom does), be something Python project would consider accepting?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Raymond Hettinger added the comment:

Since the downstream calls to PyMem_Malloc and _PyLong_FromByteArray both 
accept size_t for their sizing, there isn't a problem there.

That said, I think the current limitation nicely protects us from harm.  If you 
were to run getrandbits(2**60) it would take a long time, eat all your memory, 
trigger swaps until your harddrive was full, and you wouldn't be able to break 
out of the tight loop with a keyboard interrupt.

Even with the current limit, the resultant int object is ridiculously big in a 
way that is awkward to manipulate after it is created (don't bother trying to 
print it, jsonify it, or doing any interesting math it).

Also, if a person wants a lot of bits, it is effortless to make repeated calls 
getrandbits() using the current API.  Doing so would likely improve their code 
and be a better design (consuming bits as generated rather than creating them 
all at once and extracting them later).

In short, just because we can do it, doesn't mean we should.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Campbell Barton]

Campbell Barton added the comment:

> This probably isn't an issue on non-Windows or 64-bit systems.

In fact it is, the limitation applies to 64bit Linux too. (tested in CPython 
3.5.1)

--
nosy: +ideasman42

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[SilentGhost]

Changes by SilentGhost :


--
components: +Extension Modules, Windows -Library (Lib)
nosy: +mark.dickinson, paul.moore, rhettinger, steve.dower, tim.golden, 
zach.ware
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27072] random.getrandbits is limited to 2**31-1 bits on 64-bit Windows

[Steven Barker]

New submission from Steven Barker:

The C implementation of `_random.Random.getrandbits` is unnecessarily limited 
in the number of bits it can produce on 64-bit Windows systems. I learned about 
this issue in discussion of my answer to this stack overflow question:

 
http://stackoverflow.com/questions/37356338/is-there-a-predictable-replacement-for-os-urandom-using-pythons-random-module

The argument parsing code in `getrandbits` forces its Python `int` argument to 
fit into a C `int` variable. On my 64-bit Windows system, any value larger than 
`2**31-1` causes a `OverflowError`.

Since the number of bits is directly related to how much memory we need to 
allocate (in the non-fast case), I think `Py_ssize_t` would be more appropriate 
type than a regular `int`. This probably isn't an issue on non-Windows or 
64-bit systems, where `int` and `Py_ssize_t` will have the same size.

I'm attaching a very simple patch that changes the types of the relevant 
variables and the format code in the call to `PyArg_ParseTuple`. The code works 
and still passes its tests with the patch. I considered adding an additional 
test for this issue, but passing test cases would require allocations of 
several gigabytes of memory which seems a rather unfriendly thing to add in a 
test for a fairly minor issue.

This issue doesn't effect the pure Python implementation of 
`random.SystemRandom.getrandbits`, which already worked fine when large numbers 
of bits were requested. The documentation for `random.getrandbits` doesn't 
mention any limitation on the number of bits provided, so I don't imagine there 
will be backwards compatibility issues. I also don't expect the change to have 
any impact on third party `Random` replacement classes.

For convenience, here's the contents of the very short patch (which I'll also 
attach):

diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c
index fd6b230..3bf564f 100644
--- a/Modules/_randommodule.c
+++ b/Modules/_randommodule.c
@@ -348,12 +348,12 @@ random_setstate(RandomObject *self, PyObject *state)
 static PyObject *
 random_getrandbits(RandomObject *self, PyObject *args)
 {
-int k, i, words;
+Py_ssize_t k, i, words;
 PY_UINT32_T r;
 PY_UINT32_T *wordarray;
 PyObject *result;

-if (!PyArg_ParseTuple(args, "i:getrandbits", ))
+if (!PyArg_ParseTuple(args, "n:getrandbits", ))
 return NULL;

 if (k <= 0) {

--
components: Library (Lib)
files: getrandbits.diff
keywords: patch
messages: 265987
nosy: Steven.Barker
priority: normal
severity: normal
status: open
title: random.getrandbits is limited to 2**31-1 bits on 64-bit Windows
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file42919/getrandbits.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com