#36732: Sitemaps with i18n=True load the whole table in memory
-------------------------------------+-------------------------------------
Reporter: Julien Palard | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: contrib.sitemaps | Version: 5.2
Severity: Normal | Resolution:
Keywords: sitemap, memory | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Julien Palard:
Old description:
> Hi!
>
> It's a bit like: https://code.djangoproject.com/ticket/11572
>
> when enabling i18n on a sitemap, _items uses the following code:
>
> ```python
> # Create (item, lang_code) tuples for all items and
> languages.
> # This is necessary to paginate with all languages already
> considered.
> items = [
> (item, lang_code)
> for item in self.items()
> for lang_code in self.get_languages_for_item(item)
> ]
> ```
>
> The list comprehension loads the whole table times the number of
> languages in memory.
>
> We tried with a table containing two millions elements and two languages:
> it does not fit in 16GB of memory and the process gets killed.
New description:
Hi!
It's a bit like: https://code.djangoproject.com/ticket/11572
when enabling i18n on a sitemap, _items uses the following code:
{{{
# Create (item, lang_code) tuples for all items and languages.
# This is necessary to paginate with all languages already
considered.
items = [
(item, lang_code)
for item in self.items()
for lang_code in self.get_languages_for_item(item)
]
}}}
The list comprehension loads the whole table times the number of languages
in memory.
We tried with a table containing two millions elements and two languages:
it does not fit in 16GB of memory and the process gets killed.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/36732#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019a82a67237-53f9e585-5f6a-4a1a-ac1a-0c208909391f-000000%40eu-central-1.amazonses.com.
