Hi Dmitry, Please check that documentation page <https://docs.djangoproject.com/en/1.10/ref/models/querysets/#iterator>. As you can see iterator() loads all in the memory and after that returns an iterator.
Try to fetch records say by 50 - 100 items. Hope that helps. On Mon, Oct 10, 2016 at 2:14 PM, Горобец Дмитрий <[email protected]> wrote: > Hello. > > Thanks for advices. > > Crash occures, when I run Django command through python manage.py > my_command_name. > > It doesn't relate to nginx and even gunicorn, because, again, it's Django > command, which I call from shell. > > I switched off redis caching (I use cacheops), but it' didn't help too. > > пятница, 7 октября 2016 г., 13:21:48 UTC+5 пользователь Горобец Дмитрий > написал: > >> Hello. >> >> I have VPS with 2Gb RAM with django, gunicorn, mysql, nginx and redis. >> >> My app crashes with out of memory error, when I run django command on >> model, which has approximately 7 million records. It takes each premise >> object and updates one field by checking value with regular expression. >> Please, help optimize that command. >> >> class Command(BaseCommand): >> help = 'Updates premise.number_order' >> >> def handle(self, *args, **options): >> for premise in Premise.objects.iterator(): >> premise.number_order = premise.set_number_order() >> premise.save() >> >> self.stdout.write('Finished') >> >> >> # Method of Premise model >> def set_number_order(self): >> tr = { >> 'А': '.10', >> 'A': '.10', >> 'Б': '.20', >> 'В': '.30', >> 'Г': '.40', >> 'Д': '.50', >> 'Е': '.60', >> 'Ж': '.70', >> 'З': '.80', >> 'И': '.90', >> } >> >> only_digit = re.compile(r'^(?P<number>[0-9]{1,9})$') >> digit_with_separator = re.compile(r'^(?P<number>[0-9] >> {1,9})(?P<separator>[-|/])(?P<rest>\w+)$') >> digit_with_letter = re.compile(r'^(?P<number>[0-9] >> {1,9})(?P<letter>[А-Яа-я]+)') >> result = 0 >> title = self.title.strip().upper() >> >> if only_digit.match(title): >> number = only_digit.match(title).group('number') >> result = number + '.00' >> >> elif digit_with_separator.match(title): >> number = digit_with_separator.match(title).group('number') >> rest = digit_with_separator.match(title).group('rest') >> if rest[0].isalpha(): >> floating = tr.get(rest[0], '.90') >> result = number + floating >> >> elif rest[0].isdigit(): >> try: >> if rest[1].isdigit(): >> result = number + '.{}'.format(rest[:2]) >> else: >> result = number + '.0{}'.format(rest[0]) >> except IndexError: >> result = number + '.0{}'.format(rest[0]) >> >> elif digit_with_letter.match(title): >> number = digit_with_letter.match(title).group('number') >> letter = digit_with_letter.match(title).group('letter')[0] >> >> floating = tr.get(letter, '.90') >> result = number + floating >> >> return Decimal(result) >> > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/django-users. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/django-users/2e87323c-294c-4b82-89dd-783e36e23291%40googlegroups.com > <https://groups.google.com/d/msgid/django-users/2e87323c-294c-4b82-89dd-783e36e23291%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK52boUAy-8NfaoG8Hmvge2dNWps2baBc4tfHa0mOrV9MNEw8g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
