Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Michael Viveros]

Victor Hooi, did you ever find a work-around?
I'm having the same problem of trying to efficiently create models that 
have a Many to One relationship (so that they are linked by a Foreign Key).

The ticket  referenced above 
is still open and I manually checked myself that using bulk_create does not 
update the pk_id field of a model.

One work-around I thought of was to have a dictionary from a Product to 
it's ProductImages.
In the main loop, create the ProductImage without specifying a Product and 
just map the ProductImage to it's Product using the dictionary.
Then call bulk_create on the Products.
Then loop through the dictionary, update each ProductImage's product to the 
newly created Product.
Then bulk_create the ProductImages.

On Tuesday, August 20, 2013 at 5:27:36 PM UTC-4, Victor Hooi wrote:
>
> Hi,
>
> Cool, thanks for the link to the ticket. Very interesting reading, and I 
> learnt something =).
>
> Apparently the ticket says the patch still needs docs/tests - guess it'll 
> be a while before this gets integrated then...hmm.
>
> Cheers,
> Victor
>
>
> On Wed, Aug 21, 2013 at 2:11 AM, Simon Charette  > wrote:
>
>> This is a known limitation of `bulk_create`: objects used for bulk 
>> creation are not assigned a primary key 
>> .
>>
>> Le mardi 20 août 2013 05:57:52 UTC-4, Victor Hooi a écrit :
>>
>>> Hi,
>>>
>>> *1. Bulk Creating Products/ProductImages*
>>>
>>> I have a Django custom management command that bulk-loads a list of 
>>> products and product images from a JSON file.
>>>
>>> I have a Product model, along with an ProductImage model - each Product 
>>> may have many ProductImages.
>>>
>>> class Product(models.Model):
 ...
 name = models.CharField(max_length=200, help_text='Name of this 
 product.')
 description = models.TextField(help_text='Short description of 
 this product.')
 external_id = models.CharField(max_length=100, unique=True, 
 help_text='Unique identifier for this product provided by the supplier.')
 brand = models.ForeignKey(Brand)
 supplier = models.ForeignKey(Supplier)
 selling_price = models.DecimalField(max_digits=10, 
 decimal_places=2, help_text='The price which we\'re selling this product 
 at.')
 original_price = models.DecimalField(max_digits=10, 
 decimal_places=2, help_text='The original retail price of this product, 
 before any discounts.')
 current_stock_level = models.PositiveIntegerField(default=0)
 current_sold_count = models.PositiveIntegerField(default=0)
 ...
>>>
>>>
>>> class ProductImage(models.Model):
 product = models.ForeignKey(Product, related_name='images')
 image = models.ImageField(max_length=200, 
 upload_to='product_images')
 # TODO - Do we actually want to make retina images a distinct 
 column in the database? May need to revisit this.
 retina_image = models.ImageField(max_length=200, 
 upload_to='product_images')
 size = models.ForeignKey(ImageSize)
 ordinal = models.PositiveIntegerField(default=0)
 class Meta:
 ordering = ['ordinal']
 unique_together = ('product', 'size', 'ordinal')
 def __unicode__(self):
 return u'%s image %i for %s' % (self.size, self.ordinal, 
 self.product.name)
>>>
>>>
>>> I have a single look that iterates through the JSON file, creating a 
>>> list of Products, as well as a list of ProductImages that link back to 
>>> those products. (The below is an extract)
>>>
>>> products = []
 product_images = []
 ...
 for product in json_data:
 brand, created_new_brand = Brand.objects.get_or_create(
 name=product['brand'])
 if created_new_brand:
 new_brands_created += 1
 ...
 current_product = Product(name = product['name'],
 brand = brand,
 supplier = supplier,
 description = 
 product['description'],
 ...
 )
 ...
 for image_set in product['images']:
 for size in image_sizes.keys():
 product_images.append(ProductImage(product = 
 current_product,
image = 
 image_set[size],
retina_image 
 = image_set[size],
size = 
 image_sizes[size],
ordinal = 
 image_set_counter,
   

Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Victor Hooi]

Hi,

Cool, thanks for the link to the ticket. Very interesting reading, and I
learnt something =).

Apparently the ticket says the patch still needs docs/tests - guess it'll
be a while before this gets integrated then...hmm.

Cheers,
Victor


On Wed, Aug 21, 2013 at 2:11 AM, Simon Charette wrote:

> This is a known limitation of `bulk_create`: objects used for bulk
> creation are not assigned a primary 
> key
> .
>
> Le mardi 20 août 2013 05:57:52 UTC-4, Victor Hooi a écrit :
>
>> Hi,
>>
>> *1. Bulk Creating Products/ProductImages*
>>
>> I have a Django custom management command that bulk-loads a list of
>> products and product images from a JSON file.
>>
>> I have a Product model, along with an ProductImage model - each Product
>> may have many ProductImages.
>>
>> class Product(models.Model):
>>> ...
>>> name = models.CharField(max_length=**200, help_text='Name of this
>>> product.')
>>> description = models.TextField(help_text='**Short description of
>>> this product.')
>>> external_id = models.CharField(max_length=**100, unique=True,
>>> help_text='Unique identifier for this product provided by the supplier.')
>>> brand = models.ForeignKey(Brand)
>>> supplier = models.ForeignKey(Supplier)
>>> selling_price = models.DecimalField(max_**digits=10,
>>> decimal_places=2, help_text='The price which we\'re selling this product
>>> at.')
>>> original_price = models.DecimalField(max_**digits=10,
>>> decimal_places=2, help_text='The original retail price of this product,
>>> before any discounts.')
>>> current_stock_level = models.PositiveIntegerField(**default=0)
>>> current_sold_count = models.PositiveIntegerField(**default=0)
>>> ...
>>
>>
>> class ProductImage(models.Model):
>>> product = models.ForeignKey(Product, related_name='images')
>>> image = models.ImageField(max_length=**200,
>>> upload_to='product_images')
>>> # TODO - Do we actually want to make retina images a distinct column
>>> in the database? May need to revisit this.
>>> retina_image = models.ImageField(max_length=**200,
>>> upload_to='product_images')
>>> size = models.ForeignKey(ImageSize)
>>> ordinal = models.PositiveIntegerField(**default=0)
>>> class Meta:
>>> ordering = ['ordinal']
>>> unique_together = ('product', 'size', 'ordinal')
>>> def __unicode__(self):
>>> return u'%s image %i for %s' % (self.size, self.ordinal,
>>> self.product.name)
>>
>>
>> I have a single look that iterates through the JSON file, creating a list
>> of Products, as well as a list of ProductImages that link back to those
>> products. (The below is an extract)
>>
>> products = []
>>> product_images = []
>>> ...
>>> for product in json_data:
>>> brand, created_new_brand = Brand.objects.get_or_create(**
>>> name=product['brand'])
>>> if created_new_brand:
>>> new_brands_created += 1
>>> ...
>>> current_product = Product(name = product['name'],
>>> brand = brand,
>>> supplier = supplier,
>>> description =
>>> product['description'],
>>> ...
>>> )
>>> ...
>>> for image_set in product['images']:
>>> for size in image_sizes.keys():
>>> product_images.append(**ProductImage(product =
>>> current_product,
>>>image =
>>> image_set[size],
>>>retina_image
>>> = image_set[size],
>>>size =
>>> image_sizes[size],
>>>ordinal =
>>> image_set_counter,
>>>))
>>> image_set_counter += 1
>>> ...
>>> Product.objects.bulk_create(**products)
>>> ...
>>> for product_image in product_images:
>>> product_image.save()
>>
>>
>> I then call bulk_create() on the products list, which succeeds.
>>
>> However, when I try to call .save() or bulk_create() on the ProductImage,
>> they fail, saying that product_id is NULL.
>>
>> IntegrityError: null value in column "product_id" violates not-null
>>> constraint
>>> DETAIL:  Failing row contains (8, null, https://cdn.foobar.com.au/**
>>> site_media/uploads/product_im.**..,
>>> https://cdn.foobar.com.au/**site_media/uploads/product_im.**..,
>>> 4, 0).
>>
>>
>> If I actually inspect one one of the items in the ProductImages list -
>> 

Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Victor Hooi]

Hi,

In the source file (JSON) - I have a list of Products dictionaries.

The images are one of the elements in that dictionary - hence, the link 
between them is that images is a child of Product.

Hence, when I loop through to create Product, I can create the images and 
link it there.

At the end, the ProductImage objects all have a valid Product field, but no 
valid Product_id field - perhaps I can create all the 
Products/ProductImages, do a bulk_create() on the Products, then do a 
lookup for each ProductImage for that Product and assign a valid product_id?

However, not sure whether doing that extra lookup is going to be any better 
than just iterating through and doing a save() each round and avoiding 
bulk_create() altogether.

Cheers,
Victor

On Tuesday, 20 August 2013 22:51:57 UTC+10, Daniel Roseman wrote:
>
> On Tuesday, 20 August 2013 12:10:34 UTC+1, Victor Hooi wrote:
>
>> Hi,
>>
>> 1. Hmm, in that case, using bulk_create may not work - I'm currently 
>> relying on generating the full lists of Products and ProductImages, then 
>> saving in a batch at the end.
>>
>> So I need to loop through and call .save() individually on each Product, 
>> then the corresponding ProductImages, before moving onto the next Product().
>>
>> Are there any workarounds for this, or any way I could still leverage on 
>> bulk_create()?
>>
>>
> Can you bulk save the Products first, then assign them to the 
> ProductImages and bulk save those?
> --
> DR.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Daniel Roseman]

On Tuesday, 20 August 2013 12:10:34 UTC+1, Victor Hooi wrote:

> Hi,
>
> 1. Hmm, in that case, using bulk_create may not work - I'm currently 
> relying on generating the full lists of Products and ProductImages, then 
> saving in a batch at the end.
>
> So I need to loop through and call .save() individually on each Product, 
> then the corresponding ProductImages, before moving onto the next Product().
>
> Are there any workarounds for this, or any way I could still leverage on 
> bulk_create()?
>
>
Can you bulk save the Products first, then assign them to the ProductImages 
and bulk save those?
--
DR.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Victor Hooi]

Hi,

1. Hmm, in that case, using bulk_create may not work - I'm currently 
relying on generating the full lists of Products and ProductImages, then 
saving in a batch at the end.

So I need to loop through and call .save() individually on each Product, 
then the corresponding ProductImages, before moving onto the next Product().

Are there any workarounds for this, or any way I could still leverage on 
bulk_create()?

2. Cool, thanks for the tip - completely forgotten about this - enumerate() 
will be very useful =).

Cheers,
Victor

On Tuesday, 20 August 2013 20:40:52 UTC+10, Daniel Roseman wrote:
>
> On Tuesday, 20 August 2013 10:57:52 UTC+1, Victor Hooi wrote:
>
>> Hi,
>>
>> *1. Bulk Creating Products/ProductImages*
>>
>> I have a Django custom management command that bulk-loads a list of 
>> products and product images from a JSON file.
>>
>> I have a Product model, along with an ProductImage model - each Product 
>> may have many ProductImages.
>>
>> class Product(models.Model):
>>> ...
>>> name = models.CharField(max_length=200, help_text='Name of this 
>>> product.')
>>> description = models.TextField(help_text='Short description of this 
>>> product.')
>>> external_id = models.CharField(max_length=100, unique=True, 
>>> help_text='Unique identifier for this product provided by the supplier.')
>>> brand = models.ForeignKey(Brand)
>>> supplier = models.ForeignKey(Supplier)
>>> selling_price = models.DecimalField(max_digits=10, decimal_places=2, 
>>> help_text='The price which we\'re selling this product at.')
>>> original_price = models.DecimalField(max_digits=10, 
>>> decimal_places=2, help_text='The original retail price of this product, 
>>> before any discounts.')
>>> current_stock_level = models.PositiveIntegerField(default=0)
>>> current_sold_count = models.PositiveIntegerField(default=0)
>>> ...
>>
>>
>> class ProductImage(models.Model):
>>> product = models.ForeignKey(Product, related_name='images')
>>> image = models.ImageField(max_length=200, upload_to='product_images')
>>> # TODO - Do we actually want to make retina images a distinct column 
>>> in the database? May need to revisit this.
>>> retina_image = models.ImageField(max_length=200, 
>>> upload_to='product_images')
>>> size = models.ForeignKey(ImageSize)
>>> ordinal = models.PositiveIntegerField(default=0)
>>> class Meta:
>>> ordering = ['ordinal']
>>> unique_together = ('product', 'size', 'ordinal')
>>> def __unicode__(self):
>>> return u'%s image %i for %s' % (self.size, self.ordinal, 
>>> self.product.name)
>>
>>
>> I have a single look that iterates through the JSON file, creating a list 
>> of Products, as well as a list of ProductImages that link back to those 
>> products. (The below is an extract)
>>
>> products = []
>>> product_images = []
>>> ...
>>> for product in json_data:
>>> brand, created_new_brand = 
>>> Brand.objects.get_or_create(name=product['brand'])
>>> if created_new_brand:
>>> new_brands_created += 1
>>> ...
>>> current_product = Product(name = product['name'],
>>> brand = brand,
>>> supplier = supplier,
>>> description = 
>>> product['description'],
>>> ...
>>> )
>>> ...
>>> for image_set in product['images']:
>>> for size in image_sizes.keys():
>>> product_images.append(ProductImage(product = 
>>> current_product,
>>>image = 
>>> image_set[size],
>>>retina_image 
>>> = image_set[size],
>>>size = 
>>> image_sizes[size],
>>>ordinal = 
>>> image_set_counter,
>>>))
>>> image_set_counter += 1
>>> ...
>>> Product.objects.bulk_create(products)
>>> ...
>>> for product_image in product_images:
>>> product_image.save()
>>
>>
>> I then call bulk_create() on the products list, which succeeds.
>>
>> However, when I try to call .save() or bulk_create() on the ProductImage, 
>> they fail, saying that product_id is NULL.
>>
>> IntegrityError: null value in column "product_id" violates not-null 
>>> constraint
>>> DETAIL:  Failing row contains (8, null, 
>>> https://cdn.foobar.com.au/site_media/uploads/product_im..., 
>>> https://cdn.foobar.com.au/site_media/uploads/product_im..., 4, 0).
>>
>>
>> If I actually inspect one one of the items in the ProductImages list - 
>> product_image.product seems to point 

Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Daniel Roseman]

On Tuesday, 20 August 2013 10:57:52 UTC+1, Victor Hooi wrote:

> Hi,
>
> *1. Bulk Creating Products/ProductImages*
>
> I have a Django custom management command that bulk-loads a list of 
> products and product images from a JSON file.
>
> I have a Product model, along with an ProductImage model - each Product 
> may have many ProductImages.
>
> class Product(models.Model):
>> ...
>> name = models.CharField(max_length=200, help_text='Name of this 
>> product.')
>> description = models.TextField(help_text='Short description of this 
>> product.')
>> external_id = models.CharField(max_length=100, unique=True, 
>> help_text='Unique identifier for this product provided by the supplier.')
>> brand = models.ForeignKey(Brand)
>> supplier = models.ForeignKey(Supplier)
>> selling_price = models.DecimalField(max_digits=10, decimal_places=2, 
>> help_text='The price which we\'re selling this product at.')
>> original_price = models.DecimalField(max_digits=10, decimal_places=2, 
>> help_text='The original retail price of this product, before any 
>> discounts.')
>> current_stock_level = models.PositiveIntegerField(default=0)
>> current_sold_count = models.PositiveIntegerField(default=0)
>> ...
>
>
> class ProductImage(models.Model):
>> product = models.ForeignKey(Product, related_name='images')
>> image = models.ImageField(max_length=200, upload_to='product_images')
>> # TODO - Do we actually want to make retina images a distinct column 
>> in the database? May need to revisit this.
>> retina_image = models.ImageField(max_length=200, 
>> upload_to='product_images')
>> size = models.ForeignKey(ImageSize)
>> ordinal = models.PositiveIntegerField(default=0)
>> class Meta:
>> ordering = ['ordinal']
>> unique_together = ('product', 'size', 'ordinal')
>> def __unicode__(self):
>> return u'%s image %i for %s' % (self.size, self.ordinal, 
>> self.product.name)
>
>
> I have a single look that iterates through the JSON file, creating a list 
> of Products, as well as a list of ProductImages that link back to those 
> products. (The below is an extract)
>
> products = []
>> product_images = []
>> ...
>> for product in json_data:
>> brand, created_new_brand = 
>> Brand.objects.get_or_create(name=product['brand'])
>> if created_new_brand:
>> new_brands_created += 1
>> ...
>> current_product = Product(name = product['name'],
>> brand = brand,
>> supplier = supplier,
>> description = 
>> product['description'],
>> ...
>> )
>> ...
>> for image_set in product['images']:
>> for size in image_sizes.keys():
>> product_images.append(ProductImage(product = 
>> current_product,
>>image = 
>> image_set[size],
>>retina_image = 
>> image_set[size],
>>size = 
>> image_sizes[size],
>>ordinal = 
>> image_set_counter,
>>))
>> image_set_counter += 1
>> ...
>> Product.objects.bulk_create(products)
>> ...
>> for product_image in product_images:
>> product_image.save()
>
>
> I then call bulk_create() on the products list, which succeeds.
>
> However, when I try to call .save() or bulk_create() on the ProductImage, 
> they fail, saying that product_id is NULL.
>
> IntegrityError: null value in column "product_id" violates not-null 
>> constraint
>> DETAIL:  Failing row contains (8, null, 
>> https://cdn.foobar.com.au/site_media/uploads/product_im..., 
>> https://cdn.foobar.com.au/site_media/uploads/product_im..., 4, 0).
>
>
> If I actually inspect one one of the items in the ProductImages list - 
> product_image.product seems to point to a valid product, however, 
> product_image.product_id seems to be None:
>
> ipdb> p product_image.product
>> 
>> ipdb> p product_image.product_id
>> None
>
>
> This is strange, because if I manually create the same ProductImage from 
> the shell, the product_image.product_id field is populated.
>
> I'm guessing this product_id field should resolve to the pk id of the 
> product in product_image.product, right?
>
> However, in my case, when I instantiate each ProductImage and set it to a 
> Product, that actual Product hasn't been saved to the database - I'm 
> guessing that Product.id hasn't been generated yet, and the ProductImage 
> object I have is somehow broken?
>
> Is there another way I can 

Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

[Victor Hooi]

Hi,

*1. Bulk Creating Products/ProductImages*

I have a Django custom management command that bulk-loads a list of 
products and product images from a JSON file.

I have a Product model, along with an ProductImage model - each Product may 
have many ProductImages.

class Product(models.Model):
> ...
> name = models.CharField(max_length=200, help_text='Name of this 
> product.')
> description = models.TextField(help_text='Short description of this 
> product.')
> external_id = models.CharField(max_length=100, unique=True, 
> help_text='Unique identifier for this product provided by the supplier.')
> brand = models.ForeignKey(Brand)
> supplier = models.ForeignKey(Supplier)
> selling_price = models.DecimalField(max_digits=10, decimal_places=2, 
> help_text='The price which we\'re selling this product at.')
> original_price = models.DecimalField(max_digits=10, decimal_places=2, 
> help_text='The original retail price of this product, before any 
> discounts.')
> current_stock_level = models.PositiveIntegerField(default=0)
> current_sold_count = models.PositiveIntegerField(default=0)
> ...


class ProductImage(models.Model):
> product = models.ForeignKey(Product, related_name='images')
> image = models.ImageField(max_length=200, upload_to='product_images')
> # TODO - Do we actually want to make retina images a distinct column 
> in the database? May need to revisit this.
> retina_image = models.ImageField(max_length=200, 
> upload_to='product_images')
> size = models.ForeignKey(ImageSize)
> ordinal = models.PositiveIntegerField(default=0)
> class Meta:
> ordering = ['ordinal']
> unique_together = ('product', 'size', 'ordinal')
> def __unicode__(self):
> return u'%s image %i for %s' % (self.size, self.ordinal, 
> self.product.name)


I have a single look that iterates through the JSON file, creating a list 
of Products, as well as a list of ProductImages that link back to those 
products. (The below is an extract)

products = []
> product_images = []
> ...
> for product in json_data:
> brand, created_new_brand = 
> Brand.objects.get_or_create(name=product['brand'])
> if created_new_brand:
> new_brands_created += 1
> ...
> current_product = Product(name = product['name'],
> brand = brand,
> supplier = supplier,
> description = 
> product['description'],
> ...
> )
> ...
> for image_set in product['images']:
> for size in image_sizes.keys():
> product_images.append(ProductImage(product = 
> current_product,
>image = 
> image_set[size],
>retina_image = 
> image_set[size],
>size = 
> image_sizes[size],
>ordinal = 
> image_set_counter,
>))
> image_set_counter += 1
> ...
> Product.objects.bulk_create(products)
> ...
> for product_image in product_images:
> product_image.save()


I then call bulk_create() on the products list, which succeeds.

However, when I try to call .save() or bulk_create() on the ProductImage, 
they fail, saying that product_id is NULL.

IntegrityError: null value in column "product_id" violates not-null 
> constraint
> DETAIL:  Failing row contains (8, null, 
> https://cdn.foobar.com.au/site_media/uploads/product_im..., 
> https://cdn.foobar.com.au/site_media/uploads/product_im..., 4, 0).


If I actually inspect one one of the items in the ProductImages list - 
product_image.product seems to point to a valid product, however, 
product_image.product_id seems to be None:

ipdb> p product_image.product
> 
> ipdb> p product_image.product_id
> None


This is strange, because if I manually create the same ProductImage from 
the shell, the product_image.product_id field is populated.

I'm guessing this product_id field should resolve to the pk id of the 
product in product_image.product, right?

However, in my case, when I instantiate each ProductImage and set it to a 
Product, that actual Product hasn't been saved to the database - I'm 
guessing that Product.id hasn't been generated yet, and the ProductImage 
object I have is somehow broken?

Is there another way I can achieve this then? (Loop through file, creating 
a list of Products and ProductImages, then bulk_creating them?)

*2. Image Set Counter*

Also, second question - in the last code segment above, I have a