Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by msphn):

 I couldn't find any good solution so I build myself a custom paginator.

 My database is mysql using innodb, I made use of innodb to get a fast and
 precise enough count of rows. Please be aware, that this might not be
 precise enough for everyone's usecase.

 My paginator shouldn't be understood as a patch, but it's an adequate
 workaround to speed up your django admin. I had no problem with that, and
 I can now easily browse my 5 million+ datasets.

 {{{#!python
 from django.core.paginator import Paginator

 from django.utils.functional import cached_property
 from django.db import connection


 class MysqlInnoDbPaginator(Paginator):
 @cached_property
 def count(self):
 with connection.cursor() as cursor:
 cursor.execute(
 "SELECT TABLE_ROWS FROM information_schema.tables WHERE
 TABLE_SCHEMA = %s AND TABLE_NAME = %s;",
 [connection.settings_dict['NAME'],
 self.object_list.model._meta.db_table]
 )

 row = cursor.fetchone()

 if row is not None:
 return row[0]
 else:
 return 0
 }}}


 Add this to your admin.ModelAdmin implementation. The
 show_full_result_count is very important, otherwise another count would
 happen.

 {{{#!python
 class FoobarAdmin(admin.ModelAdmin):
 paginator = MysqlInnoDbPaginator
 show_full_result_count = False
 }}}

 Also be aware, if you have multiple database configurations, you should
 modify it so it would find the right database name

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070180bd81c17d-ba8a5de7-9695-4cf2-9864-ea9468428a09-00%40eu-central-1.amazonses.com.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by elonzh):

 * cc: elonzh (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.557f78b6937cafe800bcc0b9498a886f%40djangoproject.com.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  dev
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Ondřej Chmelař):

 * cc: Ondřej Chmelař (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.28927b1a9c04d46f98377bfdaf62e534%40djangoproject.com.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Brett Higgins):

 In addition to slow COUNT queries, I've just noticed that the listing page
 calls `len()` on a potentially slow queryset:

 
https://github.com/django/django/blob/9a3454f6046b9b7591fd03e21cf6da0b23c57689/django/contrib/admin/options.py#L1816

 On my listing page, I have a couple nested lookups, so this becomes a
 complex query with four joins, and it can take tens of seconds to run for
 around 600k records. With too many more records than that, I'm running
 afoul of AWS Lambda timeouts (30s, just like the Heroku user mentioned
 above).

 All this to say - any solution to this slow COUNT issue should also
 address (avoid) this slow query as well.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.571c9963fa94be1855b4701545693ab2%40djangoproject.com.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by TapanGujjar):

 Is it possible to have a separate table that keeps the count of all the
 rows in other tables? I am new here and don't know how much work.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.72e3e49ed60fcb725fcc193077b769ae%40djangoproject.com.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by michi88):

 Ok, it only works when the number of total rows in the database is >
 `list_per_page`. When they are less I see multiple calls to
 `len(self.object_list)`. Which I don't mind as for those views performance
 is a non-issue.

 Nonetheless, this is all highly unreliable as it relies on the queryset
 cache also being used for rendering the actual list of objects.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.e997f8c4964da5ca5c04a936b474df81%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by michi88):

 Again spoke to early, sorry for the spam, it somehow only works for one
 particular admin view I have. Trying to figure out why that is the case.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.15be5168133ab90c7d4f29a188dd740f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by michi88):

 I indeed spoke too early. This did work for me as I combine it with
 `django-postgres-fuzzycount` as well which takes care of the unfiltered
 total count:

 https://github.com/stephenmcd/django-postgres-
 fuzzycount/blob/master/fuzzycount.py

 {{{
 #!python
 class FuzzyCountAdmin(ReplicaModelAdmin):
 """
  Models must have:

  fuzzy_count_manager = FuzzyCountManager()
 """

 def get_queryset(self, request):
 if hasattr(self.model, 'fuzzy_count_manager') and request.method
 == 'GET':
 qs = self.model.fuzzy_count_manager.get_queryset()
 else:
 qs = self.model._default_manager.get_queryset()
 # taken from Django source
 # TODO: this should be handled by some parameter to the
 ChangeList.
 ordering = self.get_ordering(request)
 if ordering:
 qs = qs.order_by(*ordering)
 return qs
 }}}

 This also works for the filtered requests:
 {{{
 #!python
 class LenCountPaginator(Paginator):

 @cached_property
 def count(self):
 if self.object_list.query.where or getattr(self.object_list.query,
 'having', None):
 return len(self.object_list)
 return self.object_list.count()
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.05cc5a9937fead2aac122cff81400530%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Tim Graham):

 `.count()` is called on the unfiltered queryset so calling `len()` on it
 would fetch all objects from the database, not just those on the current
 page.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.de9088f5a3b8798067f98b9c21c15323%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by michi88):

 What's wrong with just always calling `len(self.object_list)`?
 Using this saves me from the additional count of the filtered objects.
 This seems to work as the queryset is evaluated/fetched and cached on the
 `len(...)` call, after that another fetch is not needed to render the
 items in the admin list.
 {{{
 #!python
 class LenCountPaginator(Paginator):

 @cached_property
 def count(self):
 return len(self.object_list)
 }}}

 This is how de default Paginator does it:
 {{{
 #!python
 @cached_property
 def count(self):
 """
 Returns the total number of objects, across all pages.
 """
 try:
 return self.object_list.count()
 except (AttributeError, TypeError):
 # AttributeError if object_list has no count() method.
 # TypeError if object_list.count() requires arguments
 # (i.e. is of type list).
 return len(self.object_list)
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.fff6bf8a0389c5112330821cbf68b6b6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Jerome Leclanche):

 I've been using the following for a while:

 {{{
 #!python
 class EstimatedCountPaginator(Paginator):
 def __init__(self, *args, **kwargs):
 super().__init__(*args, **kwargs)
 self.object_list.count = self.count

 @cached_property
 def count(self):
 if self.object_list.query.where:
 return self.object_list.count()

 db_table = self.object_list.model._meta.db_table
 cursor = connections[self.object_list.db].cursor()
 cursor.execute("SELECT reltuples FROM pg_class WHERE
 relname = %s", (db_table, ))
 result = cursor.fetchone()
 if not result:
 return 0
 return int(result[0])
 }}}

 At the very least I believe such a paginator should be available by
 default in `django.contrib.postgres`.

 But going beyond that, the problem is that users are not always able to
 modify the affected admin objects. So there should be some way of setting
 the *default* paginator for all admin objects, I think.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.d7cf01ba51d8dbdf29aa3f5ff8383d00%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Ionel Cristian Mărieș):

 * cc: Ionel Cristian Mărieș (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.ee273c647daae91865b0e8f18fab5f39%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  1  |  Patch needs improvement:  1
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Josh Smeaton):

 * cc: josh.smeaton@… (added)
 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1


Comment:

 I spoke to Matthew at the pycon sprints and agreed that we should open
 this one back up if we had a PR to demonstrate. The PR needs some work
 still. I'll take this to the Mailing List.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.d9702a4af46ba913f9f196901b0796ef%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---
Changes (by Matthew Betts):

 * status:  closed => new
 * resolution:  fixed =>


Comment:

 As per Josh's comment a {{{count(*)}}} is still executed by the paginator
 when using the admin, even with {{{show_full_result_count=False}}}. This
 takes multiple seconds when querying tens of millions of rows. You can
 hack around the count(*) query by implementing a custom Paginator with
 either an [https://medium.com/squad-engineering/estimated-counts-for-
 faster-django-admin-change-list-963cbf43683e appromixiate] or a fixed
 count.

 {{{
 from django.core.paginator import Paginator
 class FixedCountPaginator(Paginator):

 @property
 def count(self):
 return 1000

 class MyModelAdmin(admin.ModelAdmin):
 show_full_result_count=False
 paginator = FixedCountPaginator
 }}}

 I'd like to fix this properly by either paginating without requesting the
 number of rows / pages or disabling pagination.

 {{{
 class MyModelAdmin(admin.ModelAdmin):
 show_full_result_count=False
 no_count = False
 }}}

 I've put up a [https://github.com/django/django/pull/8858/files pull
 request] to demonstrate the changes required to achieve this. It changes
 the pagination widget in the admin to display [Prev, PAGE, Next]. Changing
 this to a `show_pagination` field that disables pagination would also
 solve the {{{count(*)}}} issue and may be more consistent / straight
 forward.

 {{{
 class MyModelAdmin(admin.ModelAdmin):
 show_full_result_count=False
 show_pagination = True
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.fb3da72fc3afaadf6c26ed9adc4c4ed4%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+---
 Reporter:  LI Daobing |Owner:  Thomas Chaumeny
 Type:  New feature|   Status:  closed
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:  fixed
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+---

Comment (by Josh Smeaton):

 I don't think this ticket has been properly or fully fixed. The patch that
 landed only affects the display of filtered results. A count(*) is still
 executed every time you hit a page, which leads to timeouts when tables
 grow large.

 I think the idea of using a custom paginator is probably a good one, but
 that could probably be controlled by an option in model admin or
 something. I'm not reopening this ticket until I've got some feedback
 first.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.be81ac34490fb730c00b418c51fde738%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+-
 Reporter:  lidaobing  |Owner:  tchaumeny
 Type:  New feature|   Status:  closed
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:  fixed
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by Tim Graham ):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [changeset:"17557d068c43bd61cdc6c18caf250ffa469414a1"]:
 {{{
 #!CommitTicketReference repository=""
 revision="17557d068c43bd61cdc6c18caf250ffa469414a1"
 Fixed #8408 -- Added ModelAdmin.show_full_result_count to avoid COUNT()
 query.

 Thanks lidaobing for the suggestion.
 }}}

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.3fcea438fa95686e5a22be0637d2b326%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+-
 Reporter:  lidaobing  |Owner:  tchaumeny
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-

Comment (by aaugustin):

 Patch is here: https://github.com/django/django/pull/3260

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.f3b908b4798bf5298e028f2e281aed57%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+-
 Reporter:  lidaobing  |Owner:  tchaumeny
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+-
Changes (by tchaumeny):

 * owner:  oinopion => tchaumeny


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.0ef0ca7c3939ad1c3430ec9fa3ee4899%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  0
  Needs tests:  0  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by tchaumeny):

 * needs_docs:  1 => 0
 * needs_tests:  1 => 0


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.e21e3212d607eaaf471500c8c5304817%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by tchaumeny):

 * cc: t.chaumeny@… (added)


--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.d6913aff29d11f77ffe0a56a6d0f1f5e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Cerin):

 See here for some interesting workarounds of Django's inefficient result
 count mechanism: http://craiglabenz.me/2013/06/12/how-i-made-django-admin-
 scale/

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.17d5c82e0820eb96ed1c8dff335302c9%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by Cerin):

 +1 for this feature. I have a few databases with tables containing tens of
 millions of records, and browsing them in Django is painfully slow, even
 with caching and extensive indexing. These sections of admin takes
 *minutes* to load, and DDT is telling me that around 90% of the load time
 is just from a single "SELECT COUNT(*) ..." to display the total result
 count, which I don't care about.

 Yes, both PostgreSQL and MySQL have tricks for approximate counts, but
 only for total tables sizes, which doesn't help once you begin filtering.

 And can confirm that using InfinitePaginator from django-pagination and
 specifying a custom paginator=InfinitePaginator in your modeladmin does
 not work because admin still expects to receive a valid count, which
 causes InfinitePaginator to throw NotImplementedError exceptions.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.6e7dbd385e0ca5e5f4113e7d07be6d7c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by giuliettamasina):

 * cc: markus.magnuson@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.150d415c8fe356b1b0b81feeb1eb59b3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by oinopion):

 I'm working on this feature in my branch here:
 https://github.com/oinopion/django/compare/nocountpagination

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.1177e389d31a9ec3f8519635ad2af6c1%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: Add a new meta option: don't do count(*) in admin (was: add a new meta option: don't do count(*) in admin)

[Django]

#8408: Add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  oinopion
 Type:  New feature|   Status:  assigned
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by oinopion):

 * owner:  nobody => oinopion
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.7cb07cb19abbc384d604cfca33e99914%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by jonathan_livni):

 When django is hosted on Heroku, there's a time limit of 30 seconds for
 any web request. So instead of admin pages taking a very long time to
 load, you get admin pages which never load.

 [[Image(http://s2.postimg.org/onfy59xuh/Capture.jpg)]]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.5d092c3c41176bce5d407701b6cf6350%40djangoproject.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  new
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by craig.labenz@…):

 Using a table of 130M+ rows and thus obviously running into the same
 problem. The COUNT(*) query takes literally 10+ minutes to run in InnoDB-
 land.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.80610cc0285f525b34025f3ff6008823%40djangoproject.com?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by bb@…):

 +1 because I ran into this problem today.

 Using PostgreSQL and django 1.4 with a table that contains about 15
 million rows. executing "select count(*) from mytable" in the psql console
 takes about 40 seconds, even with an index on the primary key. This leads
 to timeouts and huge lag.

 I'm going to try MySQL to see if this is any better.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by danfairs):

 * cc: dan.fairs@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by slav0nic):

 * cc: slav0nic0@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by aaugustin):

 The admin displays the number of matching objects and the total number of
 objects.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by akaariai):

 Can you track down where the unrestricted qs.count() is issued, and if
 there is a reason for it?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  master
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by nova77):

 Apparently count(*) is called on the whole table even if you filter your
 view. For instance if you add "?id__in=555,666", you'll have two select
 count(*): one with the "where" clause and a plain one which runs on the
 whole table.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  SVN
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by gonz):

 * cc: gonz (removed)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  SVN
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+

Comment (by anonymous):

 The issue is not restricted to pagination. Seems like the admin interface
 calls count(*) at several other places.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
 Reporter:  lidaobing  |Owner:  nobody
 Type:  New feature|   Status:  reopened
Component:  contrib.admin  |  Version:  SVN
 Severity:  Normal |   Resolution:
 Keywords: | Triage Stage:  Accepted
Has patch:  1  |  Needs documentation:  1
  Needs tests:  1  |  Patch needs improvement:  0
Easy pickings:  0  |UI/UX:  0
---+
Changes (by adamnelson):

 * stage:  Design decision needed => Accepted


Comment:

 Switching back to 'accepted' - I think the switch to DDN was in error and
 simply sent this back into the icebox even though it was previously
 accepted by a core dev and has a draft patch.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
-+-
 Reporter:  lidaobing|Owner:  nobody
 Type:  New feature  |   Status:  reopened
Component:  contrib.admin|  Version:  SVN
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Design
Has patch:  1|  decision needed
  Needs tests:  1|  Needs documentation:  1
Easy pickings:  0|  Patch needs improvement:  0
 |UI/UX:  0
-+-
Changes (by kmike):

 * cc: kmike84@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
-+-
   Reporter:  lidaobing  |  Owner:  nobody
   Type:  New| Status:  reopened
  feature|  Component:  contrib.admin
  Milestone: |   Severity:  Normal
Version:  SVN|   Keywords:
 Resolution: |  Has patch:  1
   Triage Stage:  Design |Needs tests:  1
  decision needed|  Easy pickings:  0
Needs documentation:  1  |
Patch needs improvement:  0  |
  UI/UX:  0  |
-+-

Comment (by charles@…):

 This is not just MySQL. PostgreSQL has a separate query for cheap, inexact
 counts (with statistics collected during the most recent ANALYZE) -- but
 exact counts are expensive; I have a table where a `SELECT COUNT(*)` is
 typically running upward of 50 seconds.

 Perhaps on PostgreSQL we could run a query like `select reltuples from
 pg_class where relname=?;`, and use the traditional pagination system if
 the result is under 20,000 or so, or a newer one otherwise?

 See also http://wiki.postgresql.org/wiki/Slow_Counting

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
-+-
   Reporter:  lidaobing  |  Owner:  nobody
   Type:  New| Status:  reopened
  feature|  Component:  contrib.admin
  Milestone: |   Severity:  Normal
Version:  SVN|   Keywords:
 Resolution: |  Has patch:  1
   Triage Stage:  Design |Needs tests:  1
  decision needed|  Easy pickings:  0
Needs documentation:  1  |
Patch needs improvement:  0  |
  UI/UX:  0  |
-+-
Changes (by julien):

 * ui_ux:   => 0
 * type:  Uncategorized => New feature


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
-+-
   Reporter:  lidaobing  |  Owner:  nobody
   Type: | Status:  reopened
  Uncategorized  |  Component:  contrib.admin
  Milestone: |   Severity:  Normal
Version:  SVN|   Keywords:
 Resolution: |  Has patch:  1
   Triage Stage:  Design |Needs tests:  1
  decision needed|  Easy pickings:  0
Needs documentation:  1  |
Patch needs improvement:  0  |
-+-
Changes (by boxm):

 * needs_docs:  0 => 1
 * has_patch:  0 => 1
 * needs_tests:  0 => 1
 * stage:  Accepted => Design decision needed


Comment:

 OK, here's an attempt at a patch that fixes this.

 I've added a new UncountedPaginator to pagination.py, and updated the
 admin code to check whether the paginator has the appropriate attributes
 before using them, falling back on alternative implementations if not.

 Before I create test & doc updates, would like some feedback on whether
 this patch is going about things the right way and is something that might
 be accepted.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
-+---
   Reporter:  lidaobing  |  Owner:  nobody
   Type:  Uncategorized  | Status:  reopened
  Milestone: |  Component:  contrib.admin
Version:  SVN|   Severity:  Normal
 Resolution: |   Keywords:
   Triage Stage:  Accepted   |  Has patch:  0
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
-+---
Changes (by boxm):

 * status:  closed => reopened
 * severity:   => Normal
 * cc: boxm@… (added)
 * type:   => Uncategorized
 * version:  0.96 => SVN
 * easy:   => 0
 * resolution:  wontfix =>


Comment:

 Replying to [comment:17 lrekucki]:
 > Replying to [comment:16 kmike]:
 > > I think this can be marked as 'wontfix' because custom paginators can
 be used in admin now:
 > >
 
http://docs.djangoproject.com/en/dev/ref/contrib/admin/#django.contrib.admin.ModelAdmin.paginator
 >
 > Agreed :) Here's a link to a sample [http://code.google.com/p/django-
 pagination/source/browse/trunk/pagination/paginator.py?r=47 Paginator that
 doesn't use count]

 Reopening, as the resolution simply doesn't work. The example paginator
 given by kmike simply doesn't work as a ModelAdmin paginator, because the
 {% pagination %} tag and the ChangeList class (among others) assume you
 can count the number of items being paginated. See my comment on #4065 for
 the gory details.

 There are scenarios where you don't know how many results are being
 paginated, but can still do useful pagination links:

 - With InnoDB tables in MySQL, count(*) is frighteningly expensive, so
 while in principle you can count, in practice you can't
 - With NoSQL databases it can be impossible to do a count, but you can
 still paginate by range queries

 A proposed design for such a paginator would be to display (on page 12):

 1 2 3 . 10 11 '''12''' ... Next

 which is better than just next/prev.

 I'll see if I can create acceptable patches.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
+---
   Reporter:  lidaobing | Owner:  nobody
 Status:  closed| Milestone:
  Component:  django.contrib.admin  |   Version:  0.96  
 Resolution:  wontfix   |  Keywords:
   Triage Stage:  Accepted  | Has patch:  0 
Needs documentation:  0 |   Needs tests:  0 
Patch needs improvement:  0 |  
+---
Changes (by lrekucki):

  * status:  new => closed
  * resolution:  => wontfix


Comment:

 Replying to [comment:16 kmike]:
 > I think this can be marked as 'wontfix' because custom paginators can be
 used in admin now:
 >
 
http://docs.djangoproject.com/en/dev/ref/contrib/admin/#django.contrib.admin.ModelAdmin.paginator

 Agreed :) Here's a link to a sample [http://code.google.com/p/django-
 pagination/source/browse/trunk/pagination/paginator.py?r=47 Paginator that
 doesn't use count]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
+---
   Reporter:  lidaobing | Owner:  nobody
 Status:  new   | Milestone:
  Component:  django.contrib.admin  |   Version:  0.96  
 Resolution:|  Keywords:
   Triage Stage:  Accepted  | Has patch:  0 
Needs documentation:  0 |   Needs tests:  0 
Patch needs improvement:  0 |  
+---

Comment (by kmike):

 I think this can be marked as 'wontfix' because custom paginators can be
 used in admin now:
 
http://docs.djangoproject.com/en/dev/ref/contrib/admin/#django.contrib.admin.ModelAdmin.paginator

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
+---
   Reporter:  lidaobing | Owner:  nobody
 Status:  new   | Milestone:
  Component:  django.contrib.admin  |   Version:  0.96  
 Resolution:|  Keywords:
   Triage Stage:  Accepted  | Has patch:  0 
Needs documentation:  0 |   Needs tests:  0 
Patch needs improvement:  0 |  
+---
Changes (by davidfischer):

 * cc: djfische@… (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by marcob):

 * cc: marcob...@gmail.com (added)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Comment (by marcob):

 See #13643 for an ugly-monkey-patching workaround.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by ubernostrum):

  * milestone:  1.2 =>

Comment:

 1.2 is feature-frozen, moving this feature request off the milestone.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:  1.2   
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Comment (by anonymous):

 aside from the count problem, LIMIT item_count OFFSET 25000 (maybe slower
 numbers too already) is damn slow in MySQL ;)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-upda...@googlegroups.com.
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en.

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:  1.2   
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by ludo):

 * cc: ludo (added)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:  1.2   
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by toke):

 * cc: toke (added)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:  1.2   
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by gonz):

 * cc: gonz (added)

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody
Status:  new   | Milestone:  1.2   
 Component:  django.contrib.admin  |   Version:  0.96  
Resolution:|  Keywords:
 Stage:  Accepted  | Has_patch:  0 
Needs_docs:  0 |   Needs_tests:  0 
Needs_better_patch:  0 |  
---+
Changes (by mrts):

  * milestone:  => 1.2

Comment:

 Having problems with large datasets in admin myself (not sure if `count()`
 is the only reason though), proposing this for 1.2.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to 
django-updates+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---

Re: [Django] #8408: add a new meta option: don't do count(*) in admin

[Django]

#8408: add a new meta option: don't do count(*) in admin
---+
  Reporter:  lidaobing | Owner:  nobody  
Status:  new   | Milestone:  post-1.0
 Component:  django.contrib.admin  |   Version:  0.96
Resolution:|  Keywords:  
 Stage:  Accepted  | Has_patch:  0   
Needs_docs:  0 |   Needs_tests:  0   
Needs_better_patch:  0 |  
---+
Comment (by soroosh):

 This can be done without losing any functionality by setting a limit for
 count like this:
 SELECT count(*) from (select id from HUGE_TABLE LIMIT 1) as b;
 Then if the raws count is more than for example 1, SQL server stops
 sequential scanning through the table and paginatior can list pages around
 current page but not the last page. On my system postgreql takes 60
 seconds to return count of a table with 11,000,000 raws and this is
 suboptimal to wait a minute for each admin page to load.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-updates?hl=en
-~--~~~~--~~--~--~---