#33636: BulkProcessMixin on models.Model
-------------------------------------+-------------------------------------
Reporter: Myung Eui Yoon | Owner: Myung Eui
| Yoon
Type: New feature | Status: assigned
Component: Database layer | Version: 4.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk, model | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Myung Eui Yoon:
Old description:
> {{{
> """
> class BulkyModel(models.Model, ModelBulkProcessMixin):
> id = models.BigAutoField(primary_key=True)
> name = models.CharField(max_length=10, null=False)
>
> With ModelBulkProcessMixin , We could minimize memory usage.
> Without ModelBulkProcessMixin, We shold maintain bulk array size
> up to 100_000
> or manually maintain arraysize up to batch_size like 10_000
>
> if len(chunked_list)>10_000:
> Model.objects.bulk_create(chunked_list)
>
> and check remain in list again at the end.
>
> if len(chunked_list)>0:
> Model.objects.bulk_create(chunked_list)
> """
>
> names = [f"name-{num}" for num in range(100_000)]
>
> with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
> for name in names:
> bulk.add(BulkyModel(name=name))
>
> self.assertEqual(100_000, BulkyModel.objects.all().count())
> }}}
>
> https://github.com/django/django/pull/15577
New description:
{{{
class BulkyModel(models.Model, ModelBulkProcessMixin):
id = models.BigAutoField(primary_key=True)
name = models.CharField(max_length=10, null=False)
names = [f"name-{num}" for num in range(100_100_000)]
# Case : Raw bulk_create
objs = [BulkyModel(name=name) for name in names]
BulkyModel.objects.bulk_create(
objs
) # We should maintain big array size and this leades to OOM
error
# Case : Chunked bulk_create
objs = list()
for name in names:
obj = BulkyModel(name=name)
objs.append(obj)
if len(objs) > 10_1000:
BulkyModel.objects.bulk_create(objs)
objs.clear()
if len(objs) > 0:
BulkyModel.objects.bulk_create(objs)
objs.clear()
# Case : With ModelBulkProcessMixin
with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
for name in names:
bulk.add(BulkyModel(name=name))
}}}
https://github.com/django/django/pull/15577
--
--
Ticket URL: <https://code.djangoproject.com/ticket/33636#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/0107018018fc1b6e-ead86fae-1c13-472b-a233-3353ae0804f3-000000%40eu-central-1.amazonses.com.
