Josh, I agree that silently not working is problematic but it has been this 
way since prefetch_related() was introduced.

Something to keep in mind as well is that silently turning it on would also 
perform P * C extra queries where P is the number of prefetches requested 
through prefetch_related() and C the number of chunks the results contains. 
This is non negligible IMO.

What I'd be in favor off is raising an exception on 
prefetch_related(*prefetches).iterator() in the next release at least to 
allow users using this pattern to adjust their code and then ship the 
optimization with all the warnings related to the interactions between 
prefetch_related(*prefetches) and iterator(chunk_size) in the next one.

I'm not completely completely against skipping the exception release phase 
entirely given there might be users out there accessing attributes expected 
to be prefetched on iterated instances and inadvertently performing tons of 
queries but the exception phase just feels safer given iterator() is 
usually used in memory constrained contexts.


Le vendredi 26 octobre 2018 19:27:55 UTC-4, Josh Smeaton a écrit :
> I tend to agree with Tobi. Prefetching silently not working on iterator 
> can be quite confusing, unless you have a good understanding of both APIs. 
> It might be possible to do what you're asking, but it'd mean that django is 
> now actually caching the result when it explicitly says it isn't - even if 
> the result is a much smaller moving cache. Prefetching chunk_size results 
> per chunk is unlikely to make a material difference to memory usage. Users 
> are usually concerned about the entire result set of the primary table.
> I don't know if you can change the API to make these suggested changes 
> without also impacting how we iterate over result sets - but I'd be 
> interested in seeing a proof of concept at the very least.
> On Monday, 15 October 2018 20:41:13 UTC+11, wrote:
>> Thank you for your feedback. I would like to answer some statements to 
>> either convince you or make it more clear, where my idea stems from:
>> The fundamental reason why iterator() cannot be used with 
>>> prefetch_related() is that the latter requires a set of model instance to 
>>> be materialized to work appropriately which chunk_size doesn't control at 
>>> all.
>>> In other words chunk_size only controls how many rows should be fetched 
>>> from the database cursor and kept into memory at a time. Even when this 
>>> parameter is used, iterator() will only materialize a single model instance 
>>> per yield.
>> It should be easily possible to change the involved code of ModelIterable 
>> to materialize the retrieved rows in batches. After materializing the batch 
>> / chunk, it could do the prefetching.
>>> Given that iterator() always ignored prefetch related lookups instead of 
>>> erroring out when they were specified make me feel like turning such a 
>>> feature on by default could be problematic as it could balloon the memory 
>>> usage which is the main reason why iterator is useful anyway.
>> I would argue, that users who thoughtlessly applied prefetching together 
>> with iterator now actually get, what they thought of: less DB query round 
>> trips traded against a little more memory usage.
>> Best,
>> Tobi

You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
To post to this group, send email to
Visit this group at
To view this discussion on the web visit
For more options, visit

Reply via email to