[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Martin Panter]

Martin Panter added the comment:

Thanks Raymond, now the test is noticeably faster and fine with my memory 
situation :)

--
stage:  -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Raymond Hettinger]

Raymond Hettinger added the comment:

A smaller value suffices for this test.  It was trying to make sure the 
underlying algorithms are as in-sync as possible without going to extremes.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3551fca2c6ae by Raymond Hettinger in branch '3.6':
Issue #28743:  Reduce memory consumption for random module tests
https://hg.python.org/cpython/rev/3551fca2c6ae

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Martin Panter]

Martin Panter added the comment:

Gareth, are you sure that fixes the main memory problem? Did you see Serhiy’s 
cum_weights list? Looking at the code, a list of every number from one to 13 
million will use more memory (to hold each unique integer) than the initial 
list of repeated ones.

I think the problem may also be causing buildbot failures:

Crash in test_random:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/5373/steps/test/logs/stdio
0:19:42 [368/404] test_random crashed -- running: test_zipfile (48 sec), 
test_tools (139 sec)
Traceback (most recent call last):
  [. . .]
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/libregrtest/runtest_mp.py",
 line 221, in run_tests_multiprocess
raise Exception(msg)
Exception: Child error on test_random: Exit code -9

Timeout in choices():
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/5464/steps/test/logs/stdio
0:17:05 [215/404] test_random crashed
Timeout (0:15:00)!
Thread 0x000802006400 (most recent call first):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_random.py", 
line 637 in test_choices_algorithms
  [. . .]
Traceback (most recent call last):
  [. . .]
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py",
 line 221, in run_tests_multiprocess
raise Exception(msg)
Exception: Child error on test_random: Exit code 1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Gareth Rees]

Gareth Rees added the comment:

In order for this to work, the __getitem__ method needs to be:

def __getitem__(self, key):
if 0 <= key < self.n:
return self.elem
else:
raise IndexError(key)

But unfortunately this is very bad for the performance of the test. The 
original code, with [1]*n:

Ran 1 test in 5.256s

With RepeatedSequence(1, n):

Ran 1 test in 33.620s

So that's no good. However, I notice that although the documentation of choices 
specifies that weights is a sequence, in fact it seems only to require an 
iterable:

cum_weights = list(_itertools.accumulate(weights))

so itertools.repeat works, and is faster than the original code:

Ran 1 test in 4.991s

Patch attached, in case it's acceptable to pass an iterable here.

--
keywords: +patch
Added file: http://bugs.python.org/file45546/issue28743.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The main memory consumer is a list of cumulated weights created inside 
choices().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Gareth Rees]

Gareth Rees added the comment:

Couldn't the test case use something like this to avoid allocating so much 
memory?

from collections.abc import Sequence

class RepeatedSequence(Sequence):
"""Immutable sequence of n repeats of elem."""
def __init__(self, elem, n):
self.elem = elem
self.n = n

def __getitem__(self, key):
return self.elem

def __len__(self):
return self.n

and then:

self.gen.choices(range(n), RepeatedSequence(1, n), k=1)

--
nosy: +Gareth.Rees

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This is a problem to me too.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28743] test_choices_algorithms() in test_random uses lots of memory

[Martin Panter]

New submission from Martin Panter:

Revision 32bfc8b6 added 
test.test_random.MersenneTwister_TestBasicOps.test_choices_algorithms(), which 
runs the following code:

n = 13132817  # 13 million
self.gen.choices(range(n), [1]*n, k=1)

When I build Python with the “--with-pydebug” configure option on x86-64 Linux, 
this call uses over 1.2 GB of memory. My computer only has 2 GiB, so it tends 
to slow down the whole operating system and/or trigger Linux’s out-of-memory 
killler. Especially if other tests are run concurrently.

Is it practical to reduce the magnitude of the test parameters, or optimize the 
implementation to use less memory? If not, perhaps we could hook into the 
mechanism that other tests use when the allocate large blocks of memory, to 
cause them to be skipped in low-memory situations.

--
components: Tests
messages: 281185
nosy: martin.panter, rhettinger
priority: normal
severity: normal
status: open
title: test_choices_algorithms() in test_random uses lots of memory
type: resource usage
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com