#33636: BulkProcessMixin on models.Model
-------------------------------------+-------------------------------------
Reporter: Myung Eui Yoon | Owner: Myung Eui
| Yoon
Type: New feature | Status: assigned
Component: Database layer | Version: 4.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk, model | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Myung Eui Yoon:
Old description:
> {{{
> class BulkyModel(models.Model, ModelBulkProcessMixin):
> id = models.BigAutoField(primary_key=True)
> name = models.CharField(max_length=10, null=False)
>
> names = [f"name-{num}" for num in range(100_100_000)]
>
> # Case : Raw bulk_create
> objs = [BulkyModel(name=name) for name in names]
> BulkyModel.objects.bulk_create(
> objs
> ) # We should maintain big array size and this leades to OOM
> error
>
> # Case : Chunked bulk_create
> objs = list()
> for name in names:
> obj = BulkyModel(name=name)
> objs.append(obj)
> if len(objs) > 10_1000:
> BulkyModel.objects.bulk_create(objs)
> objs.clear()
> if len(objs) > 0:
> BulkyModel.objects.bulk_create(objs)
> objs.clear()
>
> # Case : With ModelBulkProcessMixin
> with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
> for name in names:
> bulk.add(BulkyModel(name=name))
> }}}
>
> https://github.com/django/django/pull/15577
New description:
New Feature for convinient context manager for bulk create/update.
Just inherit MoModelBulkProcessMixin on Model class.
{{{
class BulkyModel(models.Model, ModelBulkProcessMixin):
id = models.BigAutoField(primary_key=True)
name = models.CharField(max_length=10, null=False)
names = [f"name-{num}" for num in range(100_100_000)]
# Case : Raw bulk_create
objs = [BulkyModel(name=name) for name in names]
BulkyModel.objects.bulk_create(
objs
) # We should maintain big array size and this leades to OOM
error
# Case : Chunked bulk_create
objs = list()
for name in names:
obj = BulkyModel(name=name)
objs.append(obj)
if len(objs) > 10_1000:
BulkyModel.objects.bulk_create(objs)
objs.clear()
if len(objs) > 0:
BulkyModel.objects.bulk_create(objs)
objs.clear()
# Case : With ModelBulkProcessMixin
with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
for name in names:
bulk.add(BulkyModel(name=name))
}}}
https://github.com/django/django/pull/15577
--
--
Ticket URL: <https://code.djangoproject.com/ticket/33636#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/0107018018fea9d0-b53cddd9-939e-42ca-b6e2-0c7520a8ffc1-000000%40eu-central-1.amazonses.com.
