#20485: Find faster ways for test fixture loading
------------------------------------------------+------------------------
Reporter: akaariai | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Currently Django's test suite uses about 70s for fixture loading when
running full test suite on in-memory SQLite. Considering that after #20483
Django's test suite takes around 210s in total, this means 1/3 of time is
used for fixture loading. Most of the time in fixture loading is used for
finding the fixture file. Django needs to do effectively this for each
file, for each test:
{{{
for each app in INSTALLED_APPS:
for each suffix in combo(('json', 'yaml', 'xml') X ('bz2', 'gz',
'zip')):
try to open file() - if succeeds add to fixture files, else skip
}}}
Unsurprisingly this is expensive. In addition fixture loading after the
file has been found isn't that fast, it is actually somewhat faster to
create the models by SomeModel.objects.create() than by fixtures.
I have done some testing and it shows that of the 70s, around 50-60s
should be avoidable.
Some options:
- Somehow use absolute paths for the fixtures. Maybe by using
`path(__file__) + '/fixtures/' + fixture_name`. A subset of this is to
always use, say gz compressed files which removes the need to check for
combos.
- Limit fixture searching to current app (if requested): maybe by
something like './somefixture' -> search only from current app.
'somefixture' -> search everywhere.
- Cache the existing fixtures. That is, go only once through the
directories, check what files exists and cache that information, then
check against the cached information instead of trying to open each
possible combo in each possible directory.
- Do not use fixtures for Django's test data.
Of the above I like the last option as I find reading models created in
Python code much easier than definitions in fixtures (especially, what
relations does given instance have). Unfortunately converting the fixtures
takes time. In addition, I am sure some developers favour fixtures instead
of models created in Python code.
Marking as accepted as in "lets see what can be done here".
--
Ticket URL: <https://code.djangoproject.com/ticket/20485>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/051.3f2168f16b22f497e4267b7468d4da59%40djangoproject.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
