#30207: Optimise paginator for tables with massive records
------------------------------------------------+------------------------
Reporter: M. Javidan Darugar | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
I had a problem with Paginator class to slice huge dataset. In case of
millions of record it can take 30 second to reach the last page. So I cam
up with a solution to overcome this problem.
The problem is that for slicing data, current solution needs to compute
whole list to render data `bottom:top`. The reason is that, current script
will generate for example `SELECT ID, COL_1, ..., COL_N, ... WHERE ...`
which has huge burden for database to slice the data. To overcome this
problem we can instead select primary keys and do the slicing step and
then fetch records that their pk are in that sliced list. Very simple but
very efficient solution. I improved the performance for our project
significantly using this approach. So form 30 seconds to only 2-3 seconds
for 8 million records.
In this ticket I propose same approach to improve django Paginator class.
--
Ticket URL: <https://code.djangoproject.com/ticket/30207>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/052.f8135a6e0d087b15741fe1462305c5fe%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
