Thank you for your feedback. I would like to answer some statements to either convince you or make it more clear, where my idea stems from:
The fundamental reason why iterator() cannot be used with > prefetch_related() is that the latter requires a set of model instance to > be materialized to work appropriately which chunk_size doesn't control at > all. > In other words chunk_size only controls how many rows should be fetched > from the database cursor and kept into memory at a time. Even when this > parameter is used, iterator() will only materialize a single model instance > per yield. > It should be easily possible to change the involved code of ModelIterable to materialize the retrieved rows in batches. After materializing the batch / chunk, it could do the prefetching. > Given that iterator() always ignored prefetch related lookups instead of > erroring out when they were specified make me feel like turning such a > feature on by default could be problematic as it could balloon the memory > usage which is the main reason why iterator is useful anyway. > I would argue, that users who thoughtlessly applied prefetching together with iterator now actually get, what they thought of: less DB query round trips traded against a little more memory usage. Best, Tobi -- Truffls GmbH Chausseestraße 86, 10115 Berlin <https://www.google.com/maps/place/Truffls+GmbH/@52.5375244,13.3715239,17z/data=!3m1!4b1!4m5!3m4!1s0x47a851ece418ed87:0xdab2d728c831f006!8m2!3d52.5375212!4d13.3737126> https://truffls.de <https://truffls.de/> Vertretungsberechtigte Geschäftsführer: Clemens Dittrich, Matthes Dohmeyer Amtsgericht Charlottenburg, HRB 160036B Diese Nachricht ist vertraulich. Sie darf ausschließlich durch den vorgesehenen Empfänger und Adressaten gelesen, kopiert oder genutzt werden. Sollten Sie diese Nachricht versehentlich erhalten haben, bitten wir, den Absender (durch Antwort-E-Mail) hiervon unverzüglich zu informieren und die Nachricht zu löschen. Jede Nutzung oder Weitergabe des Inhalts dieser Nachricht ist unzulässig. This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return e-mail) immediately and delete this message. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bf7ce18b-db67-4bb9-9be6-2e6946201c58%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
