Couple of links which might be of interest ...
http://www.jeffknupp.com/blog/2012/02/14/profiling-django-applications-a-journey-from-1300-to-2-queries/
http://gun.io/blog/fast-as-fuck-django-part-1-using-a-profiler/
ymmv
Mike
On 31/10/2013 2:47pm, Carlos D. wrote:
So, first thing: I come from a C++ background and I've been building my
first site with Django. As I normally try to do in C++, I don't want to
store any more information than I have to since that can lead to big
problems with data consistency. In C++, I do this with private members
/ methods, and a defined public interface. For my Django project, I've
been using properties since it seems like a nice way to be able to
access data as if it were a member, but not actually store that
information. However, that brings me to my problem. Since so much of
the database querying has been abstracted away from me, I definitely
haven't been thinking about overhead with database access time. This is
definitely not a knock on Django, I find it to be a great thing -- I
only need to think about this now that I have some speed issues.
Anyway, I have multiple properties that end up calling a method that
is something like:
def _get_item_total(self, prop_id):
total = 0
for item in self.item_set.all():
att1 = item.att1 #Did this for speed? I thought this would
save a couple of hits on the DB.
total +=
(item.one_to_one.one_to_one.m2mfield_set.all().get(id=prop_id).value
* att1.quantity * att1.weight)
Note: I understand that the first two 1-1 fields seem redundant. We're
using an existing DB that we're slowly adding additional information to
some of the entries (we probably won't do all of them, even in a final
product). We don't want any entry that we haven't identified the
additional information for to be displayed in a drop down on another
model. Maybe there's a better way to do this, but that's what I've
implemented for now. If there are any other additional problems with
that line of code, I'd love to hear it!
Anyway, In some views, I'm looking at ~5 instances of models that each
have a 5 or so properties that use this method. Each the item set has,
on average, 5 items. This means that the line:
(item.one_to_one.one_to_one.m2mfield_set.all().get(id=specified_prop_id).value
* att1.quantity * att1.weight)
Is being hit 5*5*5 = 125 times for some views. If I assumed 5ms per
connection (is this reasonable??) this is ~0.6 seconds by itself just
for overhead opening and closing connections to the DB. There's also
whatever time is associated with executing this query (which seems like
it may be substantial given the number of tables I'm moving through).
The great news is that there seems to be many ways I might be able to
address this:
1. DB Connection pooling. (Might address time associated with opening
/ closing connection)
2. It sounds like Django may soon be supporting keeping a connection
open for a set period of time. (Same as above)
3. Switching my properties to actual columns in the DB. I should be
able to keep these consistent since many of the tables are actually
static information. I think there's one point where saving occurs
for any of the dynamic DB information that goes into
the calculation. (Should help with some overhead -- a complex query
would be reduced to a simple query).
4. Django's cache framework. (Could make significant improvements --
but this cache is only kept for a certain period of time. Some
users will still have to wait.)
5. Database tuning (Don't know much about this...)
6. It seems like it might be inefficient that each property is
individually moving through the first three tables (is this the
right way to say this??) since the prop_id isn't used until we get
to m2mfield_set_all(). I could condense the number of tables that
need to be traversed However, it seems like this would make the #
of calls to the DB equal to 2*n+1 rather than 2*n where n is the
number of prop_ids I'm using.
7. Maybe I need to check out the query this is making and write my own
custom query if it's being inefficient?
I guess my questions are:
* Is that line of code terrible? Should I be doing this a better way?
* If not, how do I do some time profiling to determine which of the
above I should do? Or should I just do each one and see if it improves?
--
You received this message because you are subscribed to the Google
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/9f1ff8a8-140a-4071-9c9f-551b55a2b204%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Django
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/52730952.6030308%40dewhirst.com.au.
For more options, visit https://groups.google.com/groups/opt_out.
