Hi Dan, Thanks for the suggestion, it's a web scraper (run as a django management command) which then saves the data to the database via the Django ORM. Given it's a scraper rather than a form (or view) is the above suggested function an ok way to proceed or would you suggest something else is more appropriate/best practice?
On Friday, 6 November 2015 14:40:59 UTC, Dan Tagg wrote: > > Hi Yunti, > > > You could go up a level in the structure of your application and apply the > logic there, where there is more support. > > Are you using Django forms? The ModelForm class pretty much does what you > want, it examines form data, validating it against its type and any > validation rules you have set in the form or your model, compares it to the > instance's data in the database and only saves if there has been some kind > of change. > > Dan > > On 6 November 2015 at 13:47, Yunti <[email protected] <javascript:>> wrote: > >> Jani, >> >> Thanks for your reply - you explained it much more concisely than I did. >> :) >> >> Good to have it confirmed that update_or_create() doesn't quite do what I >> needed - I was confused as to whether it would or not. >> >> Thanks for taking the time to do that function, that looks ideal. I'll >> test it out. >> >> >> On Friday, 6 November 2015 12:52:11 UTC, Jani Tiainen wrote: >> >>> Your problem lies on the way Django actually carries out create or >>> update. >>> >>> As name suggest, create or update does either one. But that's what you >>> don't want - you want conditional update. >>> >>> Only update if certain fields have been changed. Well this can be done >>> few ways. >>> >>> So you want to do >>> "update_only_if_at_least_one_of_default_fields_changed_or_create" >>> >>> Operation is simple, if object is not found, create new one using >>> defaults if found, pull values as a dict, compare against >>> default values and if at least one differs do an update. Otherwise don't >>> do anything. >>> >>> So basically code would look something like this: >>> >>> update_if_changed_or_create(**kwargs): >>> defaults = kwargs.pop('defaults', None) >>> >>> qs = MyModel.objects.filter(**kwargs) >>> >>> if not qs: >>> obj = MyModel(**kwargs).save() >>> return obj, True # Created object >>> else if len(qs) == 1: >>> obj = qs[0] >>> changed = False >>> for k, v in defaults: >>> if getattr(obj, k) != v: >>> changed = True >>> setattr(obj, k, v) >>> if changed: >>> obj.save() >>> return obj, False # Updated object >>> else: >>> # Multiple objects... >>> >>> return obj, None # No change. >>> >>> >>> On 06.11.2015 14:08, Yunti wrote: >>> >>> Carsten , >>> >>> Thanks for your reply, >>> >>> A note about the last statement: If a Supplier object has the same >>> unique_id, and all >>> other fields (in `defaults`) are the same as well, logically there is no >>> difference >>> between updating and not updating – the result is the same. >>> >>> The entry in the database is the same - apart from the last_updated flag >>> if it's not rewritten over the top of it. This means I can check for new >>> data often and be alerted when there is an actual update (i.e. a change to >>> the data). If it rewrites the data everytime it checks then I have no idea >>> when data was actually updated. >>> >>> Have you checked? How? >>> In your create_or_update_if_diff() you seem to try to re-invent >>> update_or_create(), but >>> have you actually examined the results of the >>> >>> supplier, created = Supplier.objects.update_or_create(...) >>> >>> call? >>> >>> I checked by seeing that the last_updated field in the database was >>> updated everytime. (I suppose the issue could be with how that field gets >>> reset to the next time it's run- I didn't eliminate that possibility.) >>> >>> Yes I was worried that I might be recreating (a poor version) of >>> update_or_create() but it didn't seem to have the option where it wouldn't >>> write to the database if there was no change to the data. >>> Can it do this? And how would I verify when an item has been updated or >>> created (or neither) - could I output to the console? >>> >>> If it can how do I call it so it checks against all fields (unique_id >>> and defaults) and updates using the defaults if it finds a difference (and >>> creates if it doesn't find a unique_id)? >>> >>> I'm still not sure if this is possible and how to call the function, >>> particular how to pass in the remaining defaults to check against - >>> **kwargs = defaults isn't right but not sure what it should be. >>> >>> supplier, created = >>> Supplier.objects.update_or_create(unique_id=product_detail['supplierId'], >>> **kwargs=defaults, >>> defaults={ >>> 'name': >>> product_detail['supplierName'], >>> 'entity_name_1': >>> entity_name_1, >>> 'entity_name_2': >>> entity_name_1, >>> 'rating': >>> product_detail['supplierRating']}) >>> >>> On Thursday, 5 November 2015 20:05:39 UTC, Carsten Fuchs wrote: >>>> >>>> Hi Yunti, Am 05.11.2015 um 18:19 schrieb Yunti: > I have tried to use >>>> the update_or_create() method assuming that it would either, create > a >>>> new >>>> entry in the db if it found none or update an existing one if it found one >>>> and had > differences to the defaults passed in - or wouldn't update if >>>> there was no difference. A note about the last statement: If a Supplier >>>> object has the same unique_id, and all other fields (in `defaults`) are >>>> the >>>> same as well, logically there is no difference between updating and not >>>> updating – the result is the same. > However it just seemed to recreate >>>> entries each time even if there were no changes. Have you checked? How? In >>>> your create_or_update_if_diff() you seem to try to re-invent >>>> update_or_create(), but have you actually examined the results of the >>>> supplier, created = Supplier.objects.update_or_create(...) call? > I >>>> think >>>> the issue was that I wanted to: > 1) get an entry if all fields were the >>>> same, update_or_create() updates an object with the given kwargs, the >>>> match >>>> is not made against *all* fields (i.e. for the match the fields in >>>> `defaults` are not accounted for). > 2) or create a new entry if it didn't >>>> find an existing entry with the unique_id > 3) or if there was an entry >>>> with the same unique_id, update that entry with remaining > fields. >>>> update_or_create() should achieve this. It's hard to tell more without >>>> additional information, but >>>> https://docs.djangoproject.com/en/1.8/ref/models/querysets/#update-or-create >>>> >>>> explains the function well, including how it works. If you work through >>>> this in small steps, check examples and their (intermediate) results, you >>>> should be able to find what the original problem was. Best regards, >>>> Carsten >>> >>> -- You received this message because you are subscribed to the Google >>> Groups "Django users" group. To unsubscribe from this group and stop >>> receiving emails from it, send an email to >>> [email protected]. To post to this group, send email to >>> [email protected]. Visit this group at >>> http://groups.google.com/group/django-users. To view this discussion on >>> the web visit >>> https://groups.google.com/d/msgid/django-users/9b529e2d-7e2b-4194-a77c-8434efe6205d%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/django-users/9b529e2d-7e2b-4194-a77c-8434efe6205d%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/889c6480-98b3-415d-af92-490d11de5695%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/889c6480-98b3-415d-af92-490d11de5695%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3cea33db-f2e7-4739-a202-99a717bda092%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
