On Thu, 2007-04-19 at 10:36 -0500, Tim Chase wrote: [...] > >> -be careful with tracebacks if, for some reason, you have a > >> query-set that brings back bajillions of records and you error > > ... > > Yeah, I don't really know what to do with this. It's hard to know how > > big something is (memory-wise) in python, and the debugging handlers > > is necessarily generic. > > When I recognize the problem, I tend to try and filter the > record-set before my crash-able code down to something more > manageable. Not always possible, and it tends to be reactive > rather than proactive, but it helps. > > Karen Tracey also suggested using a QuerySet wrapper that > prevented it from returning gobs of data when I mentioned this > problem on the list a month or two ago (Subject: "Preventing > tracebacks from pulling back gobs of data" if one's looking for > the thread)
Agreed. This is something I want to implement when I get time. I can't see any real downside to the change and it prevents some causes of browser freeze in the extreme cases. > > >> -the default admin tool is good for blogs, bad for bajillions of > >> records. This might be mitigated a bit with some judicious use > >> of limit_choices_to but I haven't mastered such yet > > > > raw_id_admin is your friend. > > Again, I'd like to do an auto-complete widget here some day... > > needing our data-entry folks to memorize IDs would give me more > grief that I need/want ;) So limit_choices_to combined with some > hypothetical solution to the hierarchy problem above would > produce an elegant solution. Auto-complete would also be > helpful, though might require JavaScript enabled which not all of > our devices support or have enabled. My gut feeling on this is that for really large data sets, the unmodified admin interface isn't going to be the appropriate tool. Some customisation will be required. Newforms-admin should make this a lot easier. If I had to do it today with the 0.96 release, I'd write my own mini-admin interface. It would be unnecessarily complicated to try and write a one-size-fits-all admin interface to also handle these huge cases because there is so much domain-specific customisation needed that it's either going to require programming or a few hundred config variables. > >> -PostgreSQL can do dumb things with your queries, which a little > >> jiggering/subclassing of the query object can mung into something > >> a little smarter (and much faster-running) > > > > Hmm. Can you give some examples? Are you saying the query planner is > > missing indexes or...? I haven't had this problem yet, but I bet it > > sucks. :) > > I think this was a PG thing, as the SQL query I would have > hand-coded was the same query that Django created according to my > query logs. I merely[*] had to invert some logic, moving it into > the ON clause of a join. It produced about the same results, but > instead of taking 5-10 minutes to run the query, it took 5-10 > seconds. I did EXPLAINs out the wazoo, checked and > double-checked my indicies, and shot a week trying to figure that > one out. I chalk it up to PG-optimizer hiccups. > > I can give query details if you're obcenely curious, but it > certainly didn't appear to be Django's fault. Somebody else mentioned this in the recent past (this year sometime) when hand-optimising some queries. In theory, it *shouldn't* make a difference to postgreSQL's query optimiser. Internally, it unfolds "ON" clauses into the WHERE clause to allow for better reordering as part of the optimisation. In practice, that's now two people who've noticed it with what I would consider serious data sets (more than 10K rows). The good news is that the QuerySet refactor uses "ON" conditions for joins (because it's easier), so this might improve as if by magic shortly. Regards, Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
