Hi list! I've been working for the past months or so on a project that reuses imported data.
This data is extremely unreliable. I have to set up a lot of stuff to normalize it throughout Django. However there is still a field I haven't covered decently, ForeignKeys. I have a good hundred tables, most of them with a bunch of foreignkeys from one to the other. Those foreignkeys are not always valid, they sometimes point to a row that doesn't exist anymore, or never existed. Nethertheless, I need those values intact. Example: If, in my application, I got a "Display" model that has a bunch of columns, such as "Alpha", "Scale" and "File". Alpha and Scale are standard float values. File is an integer that points to another table's (DisplayFiles) id column. However, not all IDs are valid/available. If id 12345 is not valid, I will want in my app to display the file as "Unknown (#12345)". For this, I have multiple options. - I could check integrity of data on DoesNotExist exception during the import. This is what I'm doing, but for it to work I had to setup a hack within django itself to pass actual meaningful values to the exception. Not ideal at all. - I could ALWAYS check integrity of data, but this is extremely slow. I used to do that, import times went up 500%. - I've been directed to this on #django a while ago: http://yeago.net/blog/2008/dec/13/importing-data-django-projects/ - I tried it, it created more problems than solved. I gave up after working on it very long without any kind of progress. - Another idea that came up recently was to setup some sort of LazyForeignKey. It would not check for integrity, and error more gently when called, sending placeholder values or the like. Discussed this with a couple of people, the idea seems good but I need the opinion of someone with more SQL experience. I'd love some feedback on that last point. I'm also open to all suggestions other than those above, as long as you keep in mind the following: - I'm not responsible for the imported data and am not able to modify it, only work around it - I need those invalid IDs intact. Thanks JL PS: Apologies if this gets sent twice, googlegroups gave a 500 when sending this message originally and it never seemed to appear. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
