Re: ORM performance

[Michael P. Soulier]

On 04/09/14 Benjamin Scherrey said:

> The short answer to your question is no, the Django ORM is not inherently
> slower in that regard and it's very likely something you're doing. The

Given that it's basically

for obj in foo.objects.all():
obj.prop = new_value
obj.save()

I fail to see how it's something that I am doing. The whole thing runs in a
single transaction, and the only other addition would be Django's signals. I
believe I had none registered but I'll double check.

I should note that while sqlalchemy was 48 times faster, raw sql was roughly
100 times faster.

> Impossible to give you any more specific to your particular problem without
> seeing code, of course. That said, some common issues when grabbing
> individual model instances related to a larger query are often dramatically
> improved by using select_related() or fetch_related() as appropriate. Also

There is a foreign key involved in this model, not that it is being accessed,
but I can update the query and retry.

> Otherwise here's a decent little writeup of a good approach to providing
> better access to your ORM models:
> http://www.dabapps.com/blog/higher-level-query-api-django-orm/

I'll take a look, thanks.

Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20140904141000.GE11182%40digitaltorque.ca.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Michael P. Soulier]

On 03/09/14 Tom Lockhart said:

> I haven't had to deal with this myself, but the speed difference smacks of
> transactional issues. If you can run your loop by wrapping all of it or
> pieces of it (say, 100 or 1000 chunks) in a single transaction you'll
> probably see some significant speedup.

Yeah I tried that, and noticed no difference.

Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20140904140612.GD11182%40digitaltorque.ca.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Michael P. Soulier]

On 04/09/14 Tom Evans said:

> Is the update invariant? By using the ORM like this:

As I said, each update is unique and they cannot be batched.

> Are both Django and the sqlalchemy doing the same sort of update?

Yes. Identical.

Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20140904140538.GC11182%40digitaltorque.ca.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Tom Evans]

On Thu, Sep 4, 2014 at 12:22 AM, msoulier  wrote:
> Hi,
>
> I am looking at Django's performance with respect to modifying large numbers
> of objects, each in a unique way that cannot be batched. If I make a simple
> change to one of my Django models and save(), and then do the same thing in
> sqlalchemy, I notice a performance difference of about 48 times as far as
> the rate that the objects are processed to my postgresql db.
>
> The code is a simple property update and save, in a loop, trying to process
> as many objects as possible.

Is the update invariant? By using the ORM like this:

  for obj in MyObject.objects.all():
  obj.foo = 'hello'
  obj.save()

then you have to pull all the data for each object out of the
database, convert the raw DB column to it's python type, assign it to
a model instance, convert each python field back to its raw DB value
and save it in the database. That is N+1 queries, and many
conversions.

If the update is invariant, you can apply it without any of the
overhead, only 1 query and one conversion:

  MyObject.objects.all().update(foo='hello')

If the update doesn't depend on the other fields in the model, this
avoids some of the overhead, still N+1 queries but virtually no
conversion overhead:

  for pk in MyObject.objects.all().values_list('pk', flat=True):
  MyObject.objects.filter(pk=pk).update(foo=make_new_foo())

>
> Is the Django ORM known to be slower in this regard, or is it likely
> something that I'm doing?

Are both Django and the sqlalchemy doing the same sort of update?

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFHbX1J6GGtV8MAJ8vA8QRyC8KVuwacULPvLSRiY%2BRd%2BMwcOng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Sithembewena Lloyd Dube]

Thanks to the OP for asking this, and to all who answered. Ben, special
stars to you - you just shared some very valuable insight into efficieint
use of the ORM (wasn't obvious to me).

Kind regards,
Lloyd


On Thu, Sep 4, 2014 at 2:21 AM, Benjamin Scherrey 
wrote:

> The short answer to your question is no, the Django ORM is not inherently
> slower in that regard and it's very likely something you're doing. The
> useful answer is probably more complicated. :-) Naive usage of the ORM
> without an understanding of how it translates to SQL is likely to result in
> some really awful non-performant database requests for any reasonably
> complex models/queries. The good news is that it isn't very hard to get
> quite good performance out of it.
>
> Impossible to give you any more specific to your particular problem
> without seeing code, of course. That said, some common issues when grabbing
> individual model instances related to a larger query are often dramatically
> improved by using select_related() or fetch_related() as appropriate. Also
> for doing large amounts of writes/updates you should look into doing the
> transaction management yourself as Tom suggests. Postgres can really fly
> once you understand how the ORM works with it. MySQL does nicely as well
> but I have less experience with it. Ultimately, if your request can't be
> efficiently modeled with the ORM (rare but does happen) then you can use
> .extra() to pass in some direct SQL quite easily.
>
> Otherwise here's a decent little writeup of a good approach to providing
> better access to your ORM models:
> http://www.dabapps.com/blog/higher-level-query-api-django-orm/
>
> Good luck,
>
>   -- Ben
>
>
> On Thu, Sep 4, 2014 at 6:22 AM, msoulier 
> wrote:
>
>> Hi,
>>
>> I am looking at Django's performance with respect to modifying large
>> numbers of objects, each in a unique way that cannot be batched. If I make
>> a simple change to one of my Django models and save(), and then do the same
>> thing in sqlalchemy, I notice a performance difference of about 48 times as
>> far as the rate that the objects are processed to my postgresql db.
>>
>> The code is a simple property update and save, in a loop, trying to
>> process as many objects as possible.
>>
>> Is the Django ORM known to be slower in this regard, or is it likely
>> something that I'm doing?
>>
>> Thanks,
>> Mike
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/9049bff2-470e-4560-b93f-dee56bc924d4%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Chief Systems Architect Proteus Technologies 
> Chief Fan Biggest Fan Productions 
> Personal blog where I am not your demographic
> .
>
> This email intended solely for those who have received it. If you have
> received this email by accident - well lucky you!!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAHN%3D9D6ATHXVzDhwoXBx1xG0hS9n%3Do6egG450Js1Hrwv9KQTBg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,
Sithu Lloyd Dube

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAH-SnCCXzJYbdURYc2a-6ViouO8ZN4A-5FjJ%3Dk%3DSp2hNzgvNdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Benjamin Scherrey]

The short answer to your question is no, the Django ORM is not inherently
slower in that regard and it's very likely something you're doing. The
useful answer is probably more complicated. :-) Naive usage of the ORM
without an understanding of how it translates to SQL is likely to result in
some really awful non-performant database requests for any reasonably
complex models/queries. The good news is that it isn't very hard to get
quite good performance out of it.

Impossible to give you any more specific to your particular problem without
seeing code, of course. That said, some common issues when grabbing
individual model instances related to a larger query are often dramatically
improved by using select_related() or fetch_related() as appropriate. Also
for doing large amounts of writes/updates you should look into doing the
transaction management yourself as Tom suggests. Postgres can really fly
once you understand how the ORM works with it. MySQL does nicely as well
but I have less experience with it. Ultimately, if your request can't be
efficiently modeled with the ORM (rare but does happen) then you can use
.extra() to pass in some direct SQL quite easily.

Otherwise here's a decent little writeup of a good approach to providing
better access to your ORM models:
http://www.dabapps.com/blog/higher-level-query-api-django-orm/

Good luck,

  -- Ben


On Thu, Sep 4, 2014 at 6:22 AM, msoulier  wrote:

> Hi,
>
> I am looking at Django's performance with respect to modifying large
> numbers of objects, each in a unique way that cannot be batched. If I make
> a simple change to one of my Django models and save(), and then do the same
> thing in sqlalchemy, I notice a performance difference of about 48 times as
> far as the rate that the objects are processed to my postgresql db.
>
> The code is a simple property update and save, in a loop, trying to
> process as many objects as possible.
>
> Is the Django ORM known to be slower in this regard, or is it likely
> something that I'm doing?
>
> Thanks,
> Mike
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/9049bff2-470e-4560-b93f-dee56bc924d4%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Chief Systems Architect Proteus Technologies 
Chief Fan Biggest Fan Productions 
Personal blog where I am not your demographic
.

This email intended solely for those who have received it. If you have
received this email by accident - well lucky you!!

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAHN%3D9D6ATHXVzDhwoXBx1xG0hS9n%3Do6egG450Js1Hrwv9KQTBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM performance

[Tom Lockhart]

On 2014-09-03, at 4:22 PM, msoulier  wrote:

> Hi,
> 
> I am looking at Django's performance with respect to modifying large numbers 
> of objects, each in a unique way that cannot be batched. If I make a simple 
> change to one of my Django models and save(), and then do the same thing in 
> sqlalchemy, I notice a performance difference of about 48 times as far as the 
> rate that the objects are processed to my postgresql db.
> 
> The code is a simple property update and save, in a loop, trying to process 
> as many objects as possible.
> 
> Is the Django ORM known to be slower in this regard, or is it likely 
> something that I'm doing?

I haven't had to deal with this myself, but the speed difference smacks of 
transactional issues. If you can run your loop by wrapping all of it or pieces 
of it (say, 100 or 1000 chunks) in a single transaction you'll probably see 
some significant speedup.

https://docs.djangoproject.com/en/dev/topics/db/transactions/

hth

- Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/1193F2BD-2597-4D68-A8EE-D593FFD084EF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

ORM performance

[msoulier]

Hi,

I am looking at Django's performance with respect to modifying large 
numbers of objects, each in a unique way that cannot be batched. If I make 
a simple change to one of my Django models and save(), and then do the same 
thing in sqlalchemy, I notice a performance difference of about 48 times as 
far as the rate that the objects are processed to my postgresql db.

The code is a simple property update and save, in a loop, trying to process 
as many objects as possible.

Is the Django ORM known to be slower in this regard, or is it likely 
something that I'm doing?

Thanks,
Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9049bff2-470e-4560-b93f-dee56bc924d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Django ORM performance patch. Fixes #5420, #5768

[Dima Dogadaylo]

I made a patch for Django to add QuerySet.fields(*fields,
**related_fields) and make possible to load only some from master and
related models fields. It allows to tune various object list queries
when we need only limited subset of all fields, improve general
performance and decrease database load. As side effect of this patch
support of selecting fields from related models in QuerySet.values()
is implemented too. It was changed signature of this method from
values(*fields) to values(*fields, **related_fields) but the change is
backward compatible.

See more details at:
http://www.mysoftparade.com/blog/django_orm_performance_patch/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[Jarek Zgoda]

yezooz napisał(a):

>>> two words: intelligent caching
>>> Know what you're asking for commonly, and cache it up with memcache.
>>> That will do you a world of benefit.
>> Well of course caching will solve some of the problems, but cache
>> still needs to regenerated sometime...
> 
> I should say that I'm thinking about few millions of pageviews a day

Memcached is designed to handle such traffic, so use it intelligently. I
mean, don't rely only on page/view caching provided with Django out of
the box, use queryset result and object cache. With reasonable
expiration time, your db will be queried rarely and you'd get your
objects nearly instantly. The other things worth putting in cache are
computed values -- if you don't mind some delays. We do such things and
the only thing I can say is that it just works, we are able to handle
many thousands of requests per minute and the sites are much more
responsible.
Unfortunately, that approach would require deleting the objects from
cache everytime they are modified, but it's a fair tradeoff IMO.

-- 
Jarek Zgoda
Skype: jzgoda | GTalk: [EMAIL PROTECTED] | voice: +48228430101

"We read Knuth so you don't have to." (Tim Peters)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[Jeremy Dunck]

On 9/18/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> I have an app (0.96 so no select_related) that had a large initial

To be clear, select_related has existed for quite a while in Django --
0.91 at least.

>  I got a lot of mileage by caching just in a dictionary
> the foreign keys so that the ORM wasn't doing lots of redundant
> queries.

Hmm, maybe you'd be interested in this:
http://www.davidcramer.net/code/50/django-cachemanager.html#comments

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[[EMAIL PROTECTED]]

I have an app (0.96 so no select_related) that had a large initial
data load consisting of a couple core tables and several foreign key
relationships.  I got a lot of mileage by caching just in a dictionary
the foreign keys so that the ORM wasn't doing lots of redundant
queries.  I found the same strategy to work well when serving up the
pages that viewed this data.  In my case it was foods that had lots of
nutrient entries but the basic nutrient info (name, symbol etc) was
the same across each food.  The actual nutrient value per serving is
in a different table and not as easily "cached".

Since this data is unlikely to change, it is also a good candidate for
memcached but I haven't gotten their yet.

Also make sure you have your indexes tuned.  For me it was something I
thought of when working in pgAdmin and not while creating my model
definitions.  You can easily turn a 20 second query into a .2 second
one when you eliminate table scans.

On Sep 18, 3:43 pm, yezooz <[EMAIL PROTECTED]> wrote:
> hi all,
>
> I'm sure many of you run high traffic websites in Django and I'm very
> curious how you're avoiding performance hit caused by built-in ORM.
> Select_related not always working and it's still might generate dozens
> of select's.
> Are you just running custom SQL queries with joins ?
>
> greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[Tim Chase]

> Between caching, profiling, and having a good sense of what your

Profiling...the #1 thing to do.

Without profiling the app under "normal usage" (common actions & 
browsing patterns gleaned from the current deployment) scaled to 
high loads, you're just twiddling knobs and wasting time.

If the problem is in the queries, page-caching isn't going to 
help.  If each individual query is fine, but you have too many of 
them, caching database queries may help.  If it's bad python 
code, DB caching and twiddling your queries isn't going to save 
you either.

Until you know _where_ the problem resides, you can't fix it.

And it's also good to profile under load in a testing lab, rather 
than waiting for real-world load to drag your production 
machine(s) to their knees :)

-tim




--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[yezooz]

On Sep 18, 11:06 pm, "[EMAIL PROTECTED]"
<[EMAIL PROTECTED]> wrote:
> I want to say one word to you. Just one word.
>
> http://en.wikipedia.org/wiki/Denormalization

That's the word I'm familiar with :)
I'm using it, but not as often as I probably should.

So in your opinion Djano's ORM is good enough for large websites if
caching and denormalization are planned well ?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[[EMAIL PROTECTED]]

I want to say one word to you. Just one word.

http://en.wikipedia.org/wiki/Denormalization

On Sep 18, 10:57 pm, yezooz <[EMAIL PROTECTED]> wrote:
> On Sep 18, 10:34 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
>
> > Without knowing a hell of a lot more of the details of you site, what
> > you're trying to do, etc - nobody can reasonably answer this question.
> > The ORM is fine, but if it's not as fast as you need for certain
> > queries, you can always drop back to RAW sql to see if that will give
> > you speed. At the level of activity you're citing, you'll have to make
> > some serious choices between how personalized you'd like the site and
> > how much resource you can throw at the problem.
>
> > Between caching, profiling, and having a good sense of what your
> > current hardware can run to, you can scale out Django horizontally to
> > achieve the scale you need.
>
> To be more specific the site is can be compared to social networking
> site so there are some common elements for every user, but mostly it's
> user-specific data. At the moment the traffic is fairly low, but I'm
> expecting it to be close to numbers I've mentioned within few months.
> Do you guys know how ie. Pownce is solving this problem ?
>
>
>
> > -joe
>
> > On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
> > > On Sep 18, 10:23 pm, yezooz <[EMAIL PROTECTED]> wrote:
> > > > On Sep 18, 10:14 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
>
> > > > > two words: intelligent caching
>
> > > > > Know what you're asking for commonly, and cache it up with memcache.
> > > > > That will do you a world of benefit.
>
> > > > Well of course caching will solve some of the problems, but cache
> > > > still needs to regenerated sometime...
>
> > > I should say that I'm thinking about few millions of pageviews a day
>
> > > > > -joe
>
> > > > > On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
> > > > > > hi all,
>
> > > > > > I'm sure many of you run high traffic websites in Django and I'm 
> > > > > > very
> > > > > > curious how you're avoiding performance hit caused by built-in ORM.
> > > > > > Select_related not always working and it's still might generate 
> > > > > > dozens
> > > > > > of select's.
> > > > > > Are you just running custom SQL queries with joins ?
>
> > > > > > greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[yezooz]

On Sep 18, 10:34 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
> Without knowing a hell of a lot more of the details of you site, what
> you're trying to do, etc - nobody can reasonably answer this question.
> The ORM is fine, but if it's not as fast as you need for certain
> queries, you can always drop back to RAW sql to see if that will give
> you speed. At the level of activity you're citing, you'll have to make
> some serious choices between how personalized you'd like the site and
> how much resource you can throw at the problem.
>
> Between caching, profiling, and having a good sense of what your
> current hardware can run to, you can scale out Django horizontally to
> achieve the scale you need.

To be more specific the site is can be compared to social networking
site so there are some common elements for every user, but mostly it's
user-specific data. At the moment the traffic is fairly low, but I'm
expecting it to be close to numbers I've mentioned within few months.
Do you guys know how ie. Pownce is solving this problem ?

>
> -joe
>
> On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
>
>
> > On Sep 18, 10:23 pm, yezooz <[EMAIL PROTECTED]> wrote:
> > > On Sep 18, 10:14 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
>
> > > > two words: intelligent caching
>
> > > > Know what you're asking for commonly, and cache it up with memcache.
> > > > That will do you a world of benefit.
>
> > > Well of course caching will solve some of the problems, but cache
> > > still needs to regenerated sometime...
>
> > I should say that I'm thinking about few millions of pageviews a day
>
> > > > -joe
>
> > > > On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
> > > > > hi all,
>
> > > > > I'm sure many of you run high traffic websites in Django and I'm very
> > > > > curious how you're avoiding performance hit caused by built-in ORM.
> > > > > Select_related not always working and it's still might generate dozens
> > > > > of select's.
> > > > > Are you just running custom SQL queries with joins ?
>
> > > > > greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[Joseph Heck]

Without knowing a hell of a lot more of the details of you site, what
you're trying to do, etc - nobody can reasonably answer this question.
The ORM is fine, but if it's not as fast as you need for certain
queries, you can always drop back to RAW sql to see if that will give
you speed. At the level of activity you're citing, you'll have to make
some serious choices between how personalized you'd like the site and
how much resource you can throw at the problem.

Between caching, profiling, and having a good sense of what your
current hardware can run to, you can scale out Django horizontally to
achieve the scale you need.

-joe

On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
>
>
> On Sep 18, 10:23 pm, yezooz <[EMAIL PROTECTED]> wrote:
> > On Sep 18, 10:14 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
> >
> > > two words: intelligent caching
> >
> > > Know what you're asking for commonly, and cache it up with memcache.
> > > That will do you a world of benefit.
> >
> > Well of course caching will solve some of the problems, but cache
> > still needs to regenerated sometime...
>
> I should say that I'm thinking about few millions of pageviews a day
>
> >
> >
> >
> > > -joe
> >
> > > On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
> >
> > > > hi all,
> >
> > > > I'm sure many of you run high traffic websites in Django and I'm very
> > > > curious how you're avoiding performance hit caused by built-in ORM.
> > > > Select_related not always working and it's still might generate dozens
> > > > of select's.
> > > > Are you just running custom SQL queries with joins ?
> >
> > > > greetings
>
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[yezooz]

On Sep 18, 10:23 pm, yezooz <[EMAIL PROTECTED]> wrote:
> On Sep 18, 10:14 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
>
> > two words: intelligent caching
>
> > Know what you're asking for commonly, and cache it up with memcache.
> > That will do you a world of benefit.
>
> Well of course caching will solve some of the problems, but cache
> still needs to regenerated sometime...

I should say that I'm thinking about few millions of pageviews a day

>
>
>
> > -joe
>
> > On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
> > > hi all,
>
> > > I'm sure many of you run high traffic websites in Django and I'm very
> > > curious how you're avoiding performance hit caused by built-in ORM.
> > > Select_related not always working and it's still might generate dozens
> > > of select's.
> > > Are you just running custom SQL queries with joins ?
>
> > > greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[yezooz]

On Sep 18, 10:14 pm, "Joseph Heck" <[EMAIL PROTECTED]> wrote:
> two words: intelligent caching
>
> Know what you're asking for commonly, and cache it up with memcache.
> That will do you a world of benefit.

Well of course caching will solve some of the problems, but cache
still needs to regenerated sometime...

>
> -joe
>
> On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
>
>
> > hi all,
>
> > I'm sure many of you run high traffic websites in Django and I'm very
> > curious how you're avoiding performance hit caused by built-in ORM.
> > Select_related not always working and it's still might generate dozens
> > of select's.
> > Are you just running custom SQL queries with joins ?
>
> > greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

Re: ORM / Performance

[Joseph Heck]

two words: intelligent caching

Know what you're asking for commonly, and cache it up with memcache.
That will do you a world of benefit.

-joe

On 9/18/07, yezooz <[EMAIL PROTECTED]> wrote:
>
> hi all,
>
> I'm sure many of you run high traffic websites in Django and I'm very
> curious how you're avoiding performance hit caused by built-in ORM.
> Select_related not always working and it's still might generate dozens
> of select's.
> Are you just running custom SQL queries with joins ?
>
> greetings
>
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---

ORM / Performance

[yezooz]

hi all,

I'm sure many of you run high traffic websites in Django and I'm very
curious how you're avoiding performance hit caused by built-in ORM.
Select_related not always working and it's still might generate dozens
of select's.
Are you just running custom SQL queries with joins ?

greetings


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---