#27833: prefetch_related fails with SQLite when used with 1000 parent records
-------------------------------------+-------------------------------------
Reporter: Jason Barnabe | Owner: Raphael
| Michel
Type: Bug | Status: assigned
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by François Freitag):
* stage: Ready for checkin => Accepted
Comment:
As mentioned in the PR, I think there is one major issue:
{{{prefetch_related}}} expects the results of the prefetch query to be
unique.
Splitting the query in batches do not enforce uniqueness and can lead to
incorrect results. In short, if related objects are shared between several
instances, when instances are split into batches, nothing prevents a
related objects from appearing twice in the results. For example:
{{{
# Suppose there are 2 instances (A, B) and 2 related objects (1, 2)
A -> 1
B -> 2
A -> 1
}}}
Prefetching in batches of 2 results in prefetch queries for: {{{[A, B]}}}
then {{{[C]}}}.
Which gives the following mapping of related objects: {{{[(A, 1), (B,
2)]}}} then {{{[(A, 1)]}}}. The {{{rel_obj_cache}}} would then be:
{{{#!python
{
(1,): [A, A], # A should only be present once
(2,): [B]
}
}}}
A more detailed example is available on the PR.
Maybe the way forward is to use a set or a dict to store the related
queries' results (either for {{{all_related_objects}}} or for
{{{rel_obj_cache}}}).
--
Ticket URL: <https://code.djangoproject.com/ticket/27833#comment:9>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/070.f55012750bf9db94abbbe57782d8dd29%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
