#27890: runtests.py cleanup exception on Python 3.6
---------------------------------------------+------------------------
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Per [https://github.com/django/django/pull/8127#issuecomment-283322945
Tim's suggestion], I've looked into a cleanup exception that happens when
running tests on Python 3.6:
{{{
$ ./tests/runtests.py basic
Testing against Django installed in '/home/tim/code/django/django' with up
to 3 processes
....
Destroying test database for alias 'other'...
Traceback (most recent call last):
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 254,
in _run_finalizers
finalizer()
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 186,
in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
465, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
463, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/django_k0xziymh/pymp-i4s112bj'
}}}
What I found out:
- `multiprocessing` is involved. Indeed, running with `--parallel=1` does
not trigger this error.
- Jinja2 is involved. Running tests without Jinja2 installed does not
trigger this error.
What happens:
1. `multiprocessing` registers `atexit` handler.
1. `runtests.py` creates a temp dir (e.g. `/tmp/django_k0xziymh`) and
registers `atexit` handler to remove it.
1. `multiprocessing` creates a temp dir (e.g. `/tmp/django_k0xziymh/pymp-
i4s112bj`) which will be deleted in the handler it registered earlier.
1. Tests happen.
1. `runtests.py` exit handler deletes the temp dir.
1. `multiprocessing` exit handler tries to delete the inner temp dir but
it is already gone.
On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so
the error does not happen.
Jinja2 is imported by a chain of imports starting with `from django.test
import TestCase, TransactionTestCase` (full chain below).
On Python 3.6 Jinja2
[https://github.com/pallets/jinja/blob/master/jinja2/__init__.py#L74-L78
patches async support] which eventually imports `multiprocessing.util`
which registers the exit handler.
Possible solutions:
- Create the temp dir before importing `django` modules.
- Move some imports in `runtests.py` into functions, thereby delaying the
indirect import of Jinja2.
- Move some imports somewhere else into functions to break the import
chain.
- ...
The import chain leading up to the import of `multiprocessing.util` (with
uninteresting import machinery stack frames removed):
{{{
File "./runtests.py", line 18, in <module>
from django.test import TestCase, TransactionTestCase
File "/home/vytis/src/django/django/test/__init__.py", line 5, in
<module>
from django.test.client import Client, RequestFactory
File "/home/vytis/src/django/django/test/client.py", line 12, in
<module>
from django.core.handlers.base import BaseHandler
File "/home/vytis/src/django/django/core/handlers/base.py", line 7, in
<module>
from django.urls import get_resolver, set_urlconf
File "/home/vytis/src/django/django/urls/__init__.py", line 1, in
<module>
from .base import (
File "/home/vytis/src/django/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/home/vytis/src/django/django/urls/exceptions.py", line 1, in
<module>
from django.http import Http404
File "/home/vytis/src/django/django/http/__init__.py", line 5, in
<module>
from django.http.response import (
File "/home/vytis/src/django/django/http/response.py", line 13, in
<module>
from django.core.serializers.json import DjangoJSONEncoder
File "/home/vytis/src/django/django/core/serializers/__init__.py", line
23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/home/vytis/src/django/django/core/serializers/base.py", line 6,
in <module>
from django.db import models
File "/home/vytis/src/django/django/db/models/__init__.py", line 3, in
<module>
from django.db.models.aggregates import * # NOQA
File "/home/vytis/src/django/django/db/models/aggregates.py", line 5, in
<module>
from django.db.models.expressions import Func, Star
File "/home/vytis/src/django/django/db/models/expressions.py", line 6,
in <module>
from django.db.models import fields
File "/home/vytis/src/django/django/db/models/fields/__init__.py", line
11, in <module>
from django import forms
File "/home/vytis/src/django/django/forms/__init__.py", line 6, in
<module>
from django.forms.boundfield import * # NOQA
File "/home/vytis/src/django/django/forms/boundfield.py", line 5, in
<module>
from django.forms.widgets import Textarea, TextInput
File "/home/vytis/src/django/django/forms/widgets.py", line 21, in
<module>
from .renderers import get_default_renderer
File "/home/vytis/src/django/django/forms/renderers.py", line 11, in
<module>
from django.template.backends.jinja2 import Jinja2
File "/home/vytis/src/django/django/template/backends/jinja2.py", line
1, in <module>
import jinja2
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 81, in <module>
_patch_async()
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 77, in _patch_async
from jinja2.asyncsupport import patch_all
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/asyncsupport.py", line 13, in <module>
import asyncio
File "/opt/python/lib/python3.6/asyncio/__init__.py", line 21, in
<module>
from .base_events import *
File "/opt/python/lib/python3.6/asyncio/base_events.py", line 17, in
<module>
import concurrent.futures
File "/opt/python/lib/python3.6/concurrent/futures/__init__.py", line
17, in <module>
from concurrent.futures.process import ProcessPoolExecutor
File "/opt/python/lib/python3.6/concurrent/futures/process.py", line 55,
in <module>
from multiprocessing.connection import wait
File "/opt/python/lib/python3.6/multiprocessing/connection.py", line 23,
in <module>
from . import util
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/049.2d87343e217ad30e741ed0dbf2345bb1%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
