Re: Optimizing Prefetch for Postgres IN Limit

2018-07-27 Thread Ram J
Hi Xof, I realized what you meant regd the optimizer and understood why my query was doing a full table scan. The issue was that column through which I prefetching has a lot of NULLs so we have a partial index on NOT NULL and the optimizer is getting fooled by that. Adding a IS NOT NULL in the sam

Re: Optimizing Prefetch for Postgres IN Limit

2018-07-27 Thread Simon Charette
There's two open tickets to work around this issue. https://code.djangoproject.com/ticket/25464 which allows passing queryset override to be used for retrieval and another one that I can't currently find that allows specifying that a subquery should be used instead of a an IN clause. Simon L

Re: Optimizing Prefetch for Postgres IN Limit

2018-07-27 Thread Jason
well, prefetch explicitly does joining in python, as in the docs https://docs.djangoproject.com/en/2.0/ref/models/querysets/#django.db.models.query.QuerySet.prefetch_related prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python. sinc

Re: Optimizing Prefetch for Postgres IN Limit

2018-07-26 Thread Ram J
Hi Xof, The issue is that, when you do a naive Prefetch you are left with nothing but Django’s auto generated IN query. I was asking about he recommended way to make this into a JOIN, which is not obvious how to do with Django Also in my case the naive IN query on the PK *did not* generate a Inde

Re: Optimizing Prefetch for Postgres IN Limit

2018-07-26 Thread Christophe Pettus
> On Jul 25, 2018, at 02:59, Jason wrote: > > Where do you get that in the pg documentation? I can't find that anywhere > (google-fu may be failing me), and we do have some queries with more than 100 > values using IN. It's slightly more complicated than that. Above 100 entries, the Postgre

Re: Optimizing Prefetch for Postgres IN Limit

2018-07-25 Thread Jason
Where do you get that in the pg documentation? I can't find that anywhere (google-fu may be failing me), and we do have some queries with more than 100 values using IN. On Tuesday, July 24, 2018 at 11:00:48 PM UTC-4, Ram Jayaraman wrote: > > Is there a prescribed pattern to optimize Prefetches w

Optimizing Prefetch for Postgres IN Limit

2018-07-24 Thread Ram Jayaraman
Is there a prescribed pattern to optimize Prefetches when the expected Prefetch list is known to exceed 100 ? We are on Django 1.8 + Postgres 10. PG has a limit of 100 for values in IN queries after which the Index on the said column is not used. For ex: a typical Prefetch generating an IN query