[issue24831] Load average in test suite too high

[STINNER Victor]

STINNER Victor  added the comment:

No activity for 5 years, so far this issue is more an open discussion than a 
list of concrete changes, so I close the issue as out of date.

If some tests are too CPU or I/O intensive, you can skip them using the -x 
option. Use --ignorefile to only skip specific test methods.

About the temporary directory, regrtest now has a --tempdir option.

--
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Irit Katriel]

Change by Irit Katriel :


--
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[STINNER Victor]

STINNER Victor added the comment:

There is not option to choose the temporary directory like $TMPDIR?

I had issues with /tmp when creating large files because /tmp is now a
tmpfs on Fedora by default.

Maybe we can blacklist some filesystems to choose the temporary directory.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Lib/test/regrtest.py:
# When tests are run from the Python build directory, it is best practice
# to keep the test files in a subfolder.  This eases the cleanup of leftover
# files using the make distclean command.
if sysconfig.is_python_build():
TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build')
else:
TEMPDIR = tempfile.gettempdir()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

Stefan Krah added the comment:

Actually, for ecryptfs we could just check for ~/.ecryptfs and
then use /tmp.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

Stefan Krah added the comment:

Yes, the current scheme is probably good for easy buildbot cleanup.


 Maybe we can blacklist some filesystems to choose the temporary directory.

If that's easily possible, it would be a good solution at least
for ecryptfs.


I've removed ecryptfs from my setup, since speedwise it's not
production-ready for encrypting an entire home directory.


The Ubuntu setup is a bit of a trap, since the encryption is
completely transparent to the user (i.e. mounting ecryptfs
occurs automatically on login).

So if you install Ubuntu as the first system on new hardware
and choose the ecryptfs option, after a while you forget about
ecryptfs entirely and blame any slowness on the hardware or
the software.

--
priority: high - normal

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

Stefan Krah added the comment:

It's very likely an I/O issue. There are several large file
tests that run even if 'largefile' is disabled. It already
helps a lot if the following tests are decorated with 
requires_resource('largefile'):

test_mmap: class LargeMmapTests()

test_io: large_file_ops()

test_largefile: all tests

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Should we perhaps create
temp files in /tmp rather than in a subdir of the build directory?

That would sound reasonable, I don't know why it hasn't been done like that 
from the start. Perhaps Ezio or Victor know.

--
nosy: +ezio.melotti, haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

Stefan Krah added the comment:

Thanks, this clears up the mystery! -- I made the mistake of choosing
the eCryptfs-homedir option when installing Ubuntu, so now I have eCryptfs on 
top of LVM on top of full-disk encryption.

So I can't create sparse files in my home directory, but I *can*
create sparse files with zero overhead in /tmp.


The test suite runs smoothly from /usr/local.  Should we perhaps create
temp files in /tmp rather than in a subdir of the build directory?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

New submission from Stefan Krah:

On my machine (Lenovo T400, Linux) the load average during running
the 2.7 test suite is about 0.5 (with some exceptions like test_io).
The system is still usable even during test_io.


In 3.6 the load average is  2.0, with some tests making
the system practically unusable (e.g. test_mmap).

--
components: Tests
messages: 248275
nosy: skrah
priority: high
severity: normal
stage: needs patch
status: open
title: Load average in test suite too high
type: resource usage
versions: Python 3.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Stefan Krah]

Stefan Krah added the comment:

Scratch the comment about test_io in 2.7: It also renders the
system unusable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Antoine Pitrou]

Antoine Pitrou added the comment:

Just to diagnose this, try to run the test suite with nice. If it still make 
the system unusable, then it's probably an I/O issue rather than CPU scheduling.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24831] Load average in test suite too high

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Any specific tests or all tests? 3.x has many new features in comparison with 
2.x, and new tests for these features. For example zipfile and tarfile tests 
run much longer due to testing with bzip2 and lzma compressions.

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24831
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com