Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  Nir
 Type:   |  Izraeli
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Nir Izraeli):

 * status:  new => assigned
 * owner:  nobody => Nir Izraeli
 * has_patch:  0 => 1


Comment:

 Claimed this issue and marked it as having a patch (assume a PR is
 considered a "patch" for tracking/reviewing purposes)

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.732be38123dac1bfdffc903f1b69d3bd%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Nir Izraeli):

 PR created and passing tests: https://github.com/django/django/pull/8540

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b05777e7fc6fca2d041d43bce43e24be%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Simon Charette):

 > Also, if I am to create a patch should I post it here first or create a
 PR for it?

 Create a PR for it and link it back to this ticket, it will allow you to
 run CI against and gather feedback more easily.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.91744abc284d2599949663e15fb2a78a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Nir Izraeli):

 I'm thinking of creating a `chunks` generator method, similar to ones
 implemented in, for example, the following Stack Overflow Answers:

 1. https://stackoverflow.com/a/24527424/1146713
 2. https://stackoverflow.com/a/8290514/1146713

 And then iterate over it in `bulk_create` and essentially run the existing
 `bulk_create` code for each `batch_size`-d chunks sequentially.

 WDYT?

 Also, if I am to create a patch should I post it here first or create a PR
 for it?

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.e2801d3f1ecc68fec7ff8586a7fbd880%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tim Graham):

 Feel free to offer a patch. Looking at the code of
 `QuerySet.bulk_create()`, I don't understand how your proposal would work.

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.05eaa26a0c9f8a41aeb8aa98f358d28c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28231: bulk_create: avoid iterating `objs` more than necessary when `batch_size` is provided (was: bulk_create: avoid iterating `objs` more than necessary when `bulk_size` is provided)

[Django]

#28231: bulk_create: avoid iterating `objs` more than necessary when 
`batch_size`
is provided
-+-
 Reporter:  Nir Izraeli  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  1.11
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Nir Izraeli:

Old description:

> When `bulk_size` provided in `bulk_create`, a user might assume (As I
> myself have) that `objs` iterable will not be iterated more than
> `bulk_size` times at once, and that no more than roughly `bulk_size`
> Model objects reside in memory at any given time.
>
> When using `bulk_create` for relatively big sets of objects provided by a
> generator object, it would be prefered to avoid iterating over the entire
> generator object. Moreover, if not iterating over the generator object is
> deemed unnecessary or out-of-scope for django it would be prefered to
> make a comment on said behavior in documentation.
>
> I suggest two possible solutions:
> 1. Document this behavior (`bulk_create` converts passed `objs` iterator
> to a list), or
> 2. Avoid doing so when `bulk_size` is given (or default is other than
> `None`. i.e. sqlite).
>
> I did not research the possibility of avoiding the list conversion, but
> if that solution is accepted by the community I volunteer to investigate
> further and claim this ticket.

New description:

 When `batch_size` provided in `bulk_create`, a user might assume (As I
 myself have) that `objs` iterable will not be iterated more than
 `batch_size` times at once, and that no more than roughly `batch_size`
 Model objects reside in memory at any given time.

 When using `bulk_create` for relatively big sets of objects provided by a
 generator object, it would be prefered to avoid iterating over the entire
 generator object. Moreover, if not iterating over the generator object is
 deemed unnecessary or out-of-scope for django it would be prefered to make
 a comment on said behavior in documentation.

 I suggest two possible solutions:
 1. Document this behavior (`bulk_create` converts passed `objs` iterator
 to a list), or
 2. Avoid doing so when `batch_size` is given (or default is other than
 `None`. i.e. sqlite).

 I did not research the possibility of avoiding the list conversion, but if
 that solution is accepted by the community I volunteer to investigate
 further and claim this ticket.

--

--
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/064.b863b9087d3b3bd1d598ce4f8d567d09%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.