Re: URL dispatcher slow?

[Moonlight]

static[0]   

 138017 function calls in 0.818 seconds

   Ordered by: internal time
   List reduced from 79 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 10000.0620.0000.4900.000 base.py:72(get_response)
 20000.0470.0000.1290.000 base.py:240(get_script_name)
 10000.0420.0000.7580.001 wsgi.py:210(__call__)
 10000.0400.0000.2470.000 urlresolvers.py:293(resolve)
 10000.0390.0000.1360.000 wsgi.py:129(__init__)
 20000.0310.0000.0520.000 
trans_real.py:212(get_language)
 70000.0310.0000.0440.000 functional.py:182(inner)
 30000.0290.0000.0880.000 encoding.py:54(force_unicode)
140020.0290.0000.0290.000 {isinstance}
 10000.0270.0000.0810.000 __init__.py:564(__init__)


static[-1]  

 1929017 function calls in 10.654 seconds

   Ordered by: internal time
   List reduced from 80 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2010002.9980.0004.9280.000 
trans_real.py:212(get_language)
   201.2880.0008.3780.000 urlresolvers.py:195(resolve)
 10001.1750.0019.9730.010 urlresolvers.py:293(resolve)
   4080001.0850.0001.0850.000 {getattr}
   2060000.8790.0001.2310.000 functional.py:182(inner)
   2010000.8780.0006.5770.000 urlresolvers.py:153(regex)
   2010000.7550.0005.6820.000 __init__.py:128(get_language)
   2010000.5470.0000.5470.000 {method 'search' of 
'_sre.SRE_Pattern' objects}
   1990000.2900.0000.2900.000 {method 'append' of 'list' 
objects}
 10000.0940.000   10.2720.010 base.py:72(get_response)


dynamic[0]

 3745017 function calls in 20.459 seconds

   Ordered by: internal time
   List reduced from 80 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   4020006.0060.0009.8660.000 
trans_real.py:212(get_language)
   4010002.5280.000   16.7000.000 urlresolvers.py:195(resolve)
 10002.3010.002   19.7330.020 urlresolvers.py:293(resolve)
   812.1620.0002.1620.000 {getattr}
   4020001.7350.000   13.1070.000 urlresolvers.py:153(regex)
   4070001.7310.0002.4240.000 functional.py:182(inner)
   4020001.4890.000   11.3550.000 __init__.py:128(get_language)
   4020001.0980.0001.0980.000 {method 'search' of 
'_sre.SRE_Pattern' objects}
   400.5710.0000.5710.000 {method 'append' of 'list' 
objects}
 10000.1370.000   20.0760.020 base.py:72(get_response)


dynamic[-1]

 3916017 function calls in 29.633 seconds

   Ordered by: internal time
   List reduced from 80 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   4210008.6500.000   14.2670.000 
trans_real.py:212(get_language)
   423.6930.000   24.2790.000 urlresolvers.py:195(resolve)
 10003.2990.003   28.6060.029 urlresolvers.py:293(resolve)
   8480003.1120.0003.1120.000 {getattr}
   4260002.5520.0003.5920.000 functional.py:182(inner)
   4210002.5270.000   18.9930.000 urlresolvers.py:153(regex)
   4210002.1760.000   16.4430.000 __init__.py:128(get_language)
   4210001.6390.0001.6390.000 {method 'search' of 
'_sre.SRE_Pattern' objects}
   4190000.8050.0000.8050.000 {method 'append' of 'list' 
objects}
 10000.2110.000   29.1030.029 base.py:72(get_response)


seo[0]

 1941017 function calls in 14.737 seconds

   Ordered by: internal time
   List reduced from 80 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2020004.1620.0006.8550.000 
trans_real.py:212(get_language)
   2010001.7870.000   11.5990.000 urlresolvers.py:195(resolve)
 10001.5940.002   13.7970.014 urlresolvers.py:293(resolve)
   411.4970.0001.4970.000 {getattr}
   2070001.2430.0001.7470.000 functional.py:182(inner)
   2020001.2300.0009.1640.000 urlresolvers.py:153(regex)
   2020001.0560.0007.9110.000 __init__.py:128(get_language)
   2020000.6930.0000.6930.000 {method 'search' of 
'_sre.SRE_Pattern' objects}
   200.4020.0000.4020.000 {method 'append' of 'list' 
objects}
 10000.1430.000   14.2210.014 base.py:72(get_response)


seo[-1]

 2022017 function calls in 15.401 seconds

   Ordered by: internal time
   List reduced from 80 to 10 due to restriction <10>

   ncalls  tottime  

Re: URL dispatcher slow?

[Tom Evans]

On Thu, Oct 11, 2012 at 10:58 PM, Łukasz Rekucki  wrote:
> On 11 October 2012 10:20, Russell Keith-Magee  wrote:
>>
>> And don't just say "Why are Django's URL resolvers slow?". Do some
>> profiling, and come back with an analysis of where the time is being
>> spent and/or wasted.
>
> FWIW, here's a link to a cProfile result for the mentioned
> benchmark[1] on Django 1.4.1 and CPython 2.7.3. A quick look shows
> that we're calling get_language() 1.5mln times (read: for every
> pattern), so that's definitely going to slow down things.
>
> I'll try running the same with 1.3 and maybe 1.4 without the locale
> mixin and post if I find anything interesting.
>
> [1]: https://gist.github.com/3875701
>
>

It definitely doesn't help; ideally language code should be calculated
once per call to resolve, and the same value used throughout.
Hard-coding a value of 'en' for language code in
LocaleRegexProvider.regex gives an idea of what gains would be:

Stock django:

static[0] msecrps  tcalls  funcs
django1583   6318 143 69

static[-1]msecrps  tcalls  funcs
django   208794791934 70


Hard coded 'en' in LocaleRegexProvider.regex:

static[0] msecrps  tcalls  funcs
django1424   7023 133 67

static[-1]msecrps  tcalls  funcs
django4972   2011 929 68

static[0] represents tests matching the first URL in the urlconf,
static[-1] represents tests matching the approximately 100th URL in
the urlconf.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Moonlight]

> FWIW, here's a link to a cProfile result for the mentioned 
> benchmark[1] on Django 1.4.1 and CPython 2.7.3. A quick look shows 
> that we're calling get_language() 1.5mln times (read: for every 
> pattern), so that's definitely going to slow down things. 
>

Happy to see we moved it a bit forward! Well done!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/ZnKPiKqj-tgJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Tom Evans]

On Thu, Oct 11, 2012 at 2:53 PM, Tom Evans  wrote:
> On Wed, Oct 10, 2012 at 7:52 AM, Moonlight  wrote:
>> Here is an article comparing various URL dispatchers:
>>
>> http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html
>>
>> What cause django URL dispatcher that much... slow?
>>
>
> Now that I've looked in detail at the test, it is because the test is
> nonsensical. Each time it tests the URLs, it constructs a fresh WSGI
> application. Each fresh application has to compile each URL in the
> urlconf before using it. It then destroys the application, and starts
> another one up.
>
> In a normal django application, you do not start from fresh on each
> request. This explains why the performance degradation from
> "static[0]" to "static[-1] " - which is the difference between testing
> the first url in the urlconf and the last url in the urlconf - is so
> pathological, each time it is run Django is recompiling each and every
> URL regexp in the urlconf (as well as _all_ other work django does at
> server start).
>
> This is testing something, but it is not testing django as a web
> application, or as Django is intended to be run.
>
> Cheers
>
> Tom

I was wrong, the WSGI app is initialized correctly just once.
Something pathological is happening with this test case, the
performance of the resolver looks decidedly dicey as number of
patterns grows.

Something for the weekend…

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Łukasz Rekucki]

On 11 October 2012 10:20, Russell Keith-Magee  wrote:
>
> And don't just say "Why are Django's URL resolvers slow?". Do some
> profiling, and come back with an analysis of where the time is being
> spent and/or wasted.

FWIW, here's a link to a cProfile result for the mentioned
benchmark[1] on Django 1.4.1 and CPython 2.7.3. A quick look shows
that we're calling get_language() 1.5mln times (read: for every
pattern), so that's definitely going to slow down things.

I'll try running the same with 1.3 and maybe 1.4 without the locale
mixin and post if I find anything interesting.

[1]: https://gist.github.com/3875701


-- 
Łukasz Rekucki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Sean Bleier]

JKM, I'm wondering if it would benefit the community to house
djangobench under https://github.com/django to give it more
visibility.  Just a thought.

On Thu, Oct 11, 2012 at 9:45 AM, ptone  wrote:
>
>
> On Thursday, October 11, 2012 1:21:09 AM UTC-7, Russell Keith-Magee wrote:
>>
>>
>> That said - could Django be faster? Sure. *Anything* can be improved.
>>
>> So, if you've got a suggestion, make it. If you think URL resolving is
>> the source of the problem, propose a way to improve the speed of URL
>> resolvers. If you think the template language is the problem, propose
>> a fix.
>
>
> To do my part to increase the signal:noise.
>
> I'll point out that just as writing a failing test is a great way to augment
> a bug report - contributing or improving a benchmark for:
>
> https://github.com/jacobian/djangobench
>
> would be a good way to contribute to a discussion around a particular
> performance improvement you're interested in seeing happen.
>
> -Preston
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-developers/-/kI7jNl7gYwEJ.
>
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Marco Paolini]

On 11/10/2012 15:53, Tom Evans wrote:

On Wed, Oct 10, 2012 at 7:52 AM, Moonlight  wrote:

Here is an article comparing various URL dispatchers:

http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html

What cause django URL dispatcher that much... slow?



Now that I've looked in detail at the test, it is because the test is
nonsensical. Each time it tests the URLs, it constructs a fresh WSGI
application. Each fresh application has to compile each URL in the
urlconf before using it. It then destroys the application, and starts
another one up.

no, the wsgi handler is instatiated only once and outside the timeit call


--
markopaol...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Aymeric Augustin]

2012/10/11 Tom Evans 

> Now that I've looked in detail at the test, it is because the test is
> nonsensical. Each time it tests the URLs, it constructs a fresh WSGI
> application. Each fresh application has to compile each URL in the
> urlconf before using it. It then destroys the application, and starts
> another one up.
>

Hi Tom,

The performance gap was really huge, roughly two orders of magnitude.
I had a gut feeling that the implementation of the URL resolver couldn't
explain it fully. This makes much more sense.

So, thank you for checking the benchmark's source code!

(I have to admit that I was too lazy and didn't care enough to do it
myself.)

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Michael]

This conversation is getting nasty. Django's URL dispatcher is what it is.
If you can get the flexibility that the current URL dispatcher provides
while making it faster, I imagine everyone on this thread would be
impressed.

There is no reason to use Django's URL dispatcher. It is quite easy to map
a WSGI path to the view functions. Heck, you can even write your own URL
dispatcher and let us all see your method. Maybe it will get traction.

I have not seen performance bottlenecks on templates or URL dispatchers. My
systems tend to bail due to the database or IO first. So benchmarks are
great and something to keep an eye on while adding new features or making
changes, but I am not going to worry about the abstract performance of URL
Dispatchers until it is clear that it accounts for a clear problem in the
request/response cycle.

Other than that we are seeing problems that are not actually there and
nothing will ever get done.


On Thu, Oct 11, 2012 at 10:01 AM, Alex Gaynor  wrote:

>
>
> On Thu, Oct 11, 2012 at 6:52 AM, Daniel Sokolowski <
> daniel.sokolow...@klinsight.com> wrote:
>
>>   I absolutely agree with: if we were looking for speed we wouldn't use
>> python at all (period).
>>
>>
> Speak for yourself.
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Daniel Sokolowski]

Was that necessary?  I am tuning out of this conversion, it is becoming 
hostile. 
From: Alex Gaynor 
Sent: Thursday, October 11, 2012 10:01 AM
To: django-developers@googlegroups.com 
Subject: Re: URL dispatcher slow?




On Thu, Oct 11, 2012 at 6:52 AM, Daniel Sokolowski 
<daniel.sokolow...@klinsight.com> wrote:

  I absolutely agree with: if we were looking for speed we wouldn't use python 
at all (period).


Speak for yourself.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to say 
it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Alex Gaynor]

On Thu, Oct 11, 2012 at 6:52 AM, Daniel Sokolowski <
daniel.sokolow...@klinsight.com> wrote:

>   I absolutely agree with: if we were looking for speed we wouldn't use
> python at all (period).
>
>
Speak for yourself.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Tom Evans]

On Wed, Oct 10, 2012 at 7:52 AM, Moonlight  wrote:
> Here is an article comparing various URL dispatchers:
>
> http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html
>
> What cause django URL dispatcher that much... slow?
>

Now that I've looked in detail at the test, it is because the test is
nonsensical. Each time it tests the URLs, it constructs a fresh WSGI
application. Each fresh application has to compile each URL in the
urlconf before using it. It then destroys the application, and starts
another one up.

In a normal django application, you do not start from fresh on each
request. This explains why the performance degradation from
"static[0]" to "static[-1] " - which is the difference between testing
the first url in the urlconf and the last url in the urlconf - is so
pathological, each time it is run Django is recompiling each and every
URL regexp in the urlconf (as well as _all_ other work django does at
server start).

This is testing something, but it is not testing django as a web
application, or as Django is intended to be run.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Daniel Sokolowski]

I absolutely agree with: if we were looking for speed we wouldn't use python at 
all (period).

From: Florian Apolloner 
Sent: Wednesday, October 10, 2012 6:29 PM
To: django-developers@googlegroups.com 
Subject: Re: URL dispatcher slow?

Oh cmon,

please stop playing a socket puppet for the wheezy.web author. A web framework 
consists of more than just a win in speed (an the author of wheezy.web can 
argue whatever he wants that "basic" stuff stuff should be fast) -- if we were 
looking for speed we wouldn't use python at all (period). The goal is to be 
fast enough and be able to scale out horizontally, everything else is not 
really interesting in the case of web development. While I won't deny that 
Django isn't the fastest framework out there it certainly gets it's job done 
more than good enough and that's all it matters. I don't know what your goal 
is, but please stop posting such meaningless comparisons on django-dev…

That said, if you really think Django's urlresolver is to slow, go ahead, 
profile it and improve it, we certainly won't say no to speed improvements…

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/DgNcKfL2AeUJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Daniel Sokolowski
http://webdesign.danols.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Yo-Yo Ma]

I only now realized that this thread had started in a bit of a trolling 
fashion, and that there was a similar thread this week. That helps to explain 
the slightly more defensive stance.

Django can survive (and thrive) just fine in its current, reasonably performant 
state, but performance should have a priority, rather than being in a sort of 
"its fine as-is" stasis. 

Benchmarks like the ones posted aren't that helpful, but it doesn't change the 
fact that there tends to be an anti-performance sentiment in this group. When 
you look at the latest Python 3.3.x release, you see that there are many 
performance improvements that are clearly not due to incidental changes. I'm 
merely suggesting that an initiative to better the performance of Django, 
perhaps by benchmarking to find some of the weakest points, determining the 
lowest hanging fruit, and creating tickets in trac for them. I would love to 
help optimize any given part of Django and submit a patch, but not if I'm 
afraid that it will never be applied and/or the ticket will be marked as "works 
for me", etc. Help give us who care about performance the most a real chance to 
make some improvements. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/bxfyOKgoRogJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Fw: URL dispatcher slow?

[Kkk Kkk]

somehow it went to wrong address... sorry, here it is.

- Forwarded Message -


>This is why I disagree with your conjecture that because Django is
>slower than another framework, therefore Django is slow. That is not
>what is shown, only that the other frameworks do certain things
> faster.

I think we have the same standpoint of view here. May be it is faster somewhere 
else... but where? If it is slower here and there... it is probaly just slow, 
isn't it? May be the word 'slow' is not correct... if the framework does more 
for me in URL dispatch I want to know what that more means... I know it can do 
reverse url but that feature is not used in url dispatch.

>Open source is wonderful, because you are allowed the choice of
>framework. You can choose full featured, and accept that those
>features will cost time, or you can choose lightweight, and not use
>those features. There is no magical mid-way point where you get all
>the features and all the speed.

Agreed. We are free to talk about this and that is good. It would be better if 
someone also listen. If the difference between lightweight and full featured 
impacts performance that much I want to see which features cause that. Let 
itemize the list. Honesty I tried... but just can not related them to the facts 
faced.




 From: Tom Evans <tevans...@googlemail.com>
To: Moonlight <moonlight_13_...@yahoo.com> 
Sent: Thursday, October 11, 2012 3:46 PM
Subject: Re: URL dispatcher slow?
 
On Thu, Oct 11, 2012 at 1:07 PM, Moonlight <moonlight_13_...@yahoo.com> wrote:
>> So, the benchmarks are interesting. They tell us which stacks are
>> fully featured, and which stacks are very lightweight. Apart from
>> that, they don't tell us much at all - is Django's template engine
>> slow, or is it about right for the work it does? This benchmark
>> doesn't tell us that, it only says it is slower than a bare bones
>> template engine, which is unsurprising, and shouldn't be a cause for
>> concern.
>
>
> It is sort of no sense. Django is fully featured and we do not care... some
> other are fast because they are lightweight... tell me which 'full
 featured'
> feature prevents it from become better. I guess for the #1 framework it is
> important to be leader... hmm... not sure what you mean by 'right for the
> work it does'... who needs a template that doesn't do what you need?

You are absolutely right - personally, I do not care one iota how fast
Django routes requests. I know that it does it fast enough, and that
improving the speed of that tiny subsection of code is not going to
improve the performance of my web application. This part of code makes
up a disproportionally small part of the request cycle, even if it was
twice as fast, the overall improvement would be negligible.

This is why I disagree with your conjecture that because Django is
slower than another framework, therefore Django is slow. That is not
what is shown, only that the other frameworks do certain things
faster.

>From your previous post, talking about
 urlresolver:

> I definitely would appreciate if someone from core team I guess or in the 
> community finally have a look there

Django's url resolver _is_ fast. It does a lot more work than the
other frameworks under test, which is why it is subjectively slower
than them in that test, but that is not relevant. It could be made
faster, if it stopped doing the useful features that we rely on to
build our projects.

Open source is wonderful, because you are allowed the choice of
framework. You can choose full featured, and accept that those
features will cost time, or you can choose lightweight, and not use
those features. There is no magical mid-way point where you get all
the features and all the speed.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Moonlight]

>
> So, the benchmarks are interesting. They tell us which stacks are 
> fully featured, and which stacks are very lightweight. Apart from 
> that, they don't tell us much at all - is Django's template engine 
> slow, or is it about right for the work it does? This benchmark 
> doesn't tell us that, it only says it is slower than a bare bones 
> template engine, which is unsurprising, and shouldn't be a cause for 
> concern.
>
 
It is sort of no sense. Django is fully featured and we do not care... some 
other are fast because they are lightweight... tell me which 'full 
featured' feature prevents it from become better. I guess for the #1 
framework it is important to be leader... hmm... not sure what you mean by 
'right for the work it does'... who needs a template that doesn't do what 
you need?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/YAcwku7WouYJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Moonlight]

Well I am not that good to get it fixed it django... it quite easy get 
lost there. But I definitely would appreciate if someone from core team I 
guess or in the community finally have a look there...

That said, if you really think Django's urlresolver is to slow, go ahead, 
> profile it and improve it, we certainly won't say no to speed improvements…
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/CZfjypvjpPQJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Tom Evans]

On Thu, Oct 11, 2012 at 7:02 AM, Yo-Yo Ma  wrote:
> Why does every conversation about Django's performance met with "GTFO we 
> don't care"? (that was a rhetorical question :). I'd venture to guess that 
> most "It's fast enough for me!" responses are predicated on experiences that 
> can be likened to personal blog development, rather than large scale, 10+ 
> server deployments (e.g., Disqus, et al).

At $JOB, we used to write our websites in C++. No, really, we used to
write all our web apps as part of custom apache modules that used our
own C++ framework. If I compared our C++ url routing code against any
of those python stacks tested, it would absolutely murder them. Yet we
don't write new websites in C++ - why?

The expensive parts of a website are not routing requests, it is DB
queries and waiting to write responses to clients. When you look at it
like that, the speed of "grunt" parts of your framework is simply not
relevant. Django's template system is considerably faster than our
XSLT templates we used in C++ (that doesn't paint XSLT in a good
light).

So, the benchmarks are interesting. They tell us which stacks are
fully featured, and which stacks are very lightweight. Apart from
that, they don't tell us much at all - is Django's template engine
slow, or is it about right for the work it does? This benchmark
doesn't tell us that, it only says it is slower than a bare bones
template engine, which is unsurprising, and shouldn't be a cause for
concern.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Russell Keith-Magee]

On Thu, Oct 11, 2012 at 2:02 PM, Yo-Yo Ma  wrote:
> Why does every conversation about Django's performance met with "GTFO we 
> don't care"? (that was a rhetorical question :).

Ok - If this is where the conversation is starting, it's going to go
downhill *very* fast.

Consider this a warning to *everyone* on this thread: we expect *all*
participants to maintain a civil and professional tone. Anyone
engaging in ad hominem attacks, straw man arguments, or any other sort
of offensive behaviour will be banned from posting. I'm not directing
this at anyone in particular - yet. But this is the second thread in a
week that has started down this path, and I want to make the rules
*absolutely* clear before anything gets out of control.

Back to the topic at hand:

We are interested in any concrete proposal that serves to improve
Django's performance.

However, we have no interest in optimisations for the sake of beating
some arbitrary benchmark score.

In addition, any claim that Django is "too slow" for practical
purposes is *demonstrably* false. There are *thousands* of sites in
production that are quite happily using Django. Many of them are
*very* large sites, serving *enormous* amounts of traffic. So, let's
dispense with the hyperbole that implies Django performance is so bad
it is only suitable for toy sites.

That said - could Django be faster? Sure. *Anything* can be improved.

So, if you've got a suggestion, make it. If you think URL resolving is
the source of the problem, propose a way to improve the speed of URL
resolvers. If you think the template language is the problem, propose
a fix.

And don't just say "Why are Django's URL resolvers slow?". Do some
profiling, and come back with an analysis of where the time is being
spent and/or wasted. Come up with a proposal for how Django's URL
resolvers could be made pluggable so that you can swap in a faster
resolver as an option.

We'll happily discuss concrete suggestions. We won't engage in fishing
expeditions.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Florian Apolloner]

Hi,

I won't answer your rheotircal questions ;)

On Thursday, October 11, 2012 8:02:19 AM UTC+2, Yo-Yo Ma wrote:
>
> BUT... Django is NOT that fast. 
>

We do know that (you know that and probably everyone else too) and I 
already said in my post that Django isn't the fastest framework out there.

The list goes on. What is the harm in discussing the weak points of Django? 
> Performance is probably the only major weak point in Django right now 
> (aside from the lack of schema migration in core, which is coming soon 
> anyway). 
>

There is no harm in discussing weak points in Django, we do it all the 
time. I just don't think that copy pasting a link (and that more than once) 
to a benchmark which is (imo) just there as a marketing strategy for 
another framework (note: I am not saying the benchmarks are a fake, I am 
just saying they probably aren't representive) adds real value to the 
discussion.

I have no solution or patches in my pocket, but I can say with absolute 
> certainty that performance will never, ever get better when discourse is in 
> a monologue format. 
>

I agree to some extend, but I think you shouldn't forget that many core 
devs use Django on somewhat large-scale deployments (read: definetely way 
over a blog) and performance issues are fixed as needed. But yes, as you 
said stuff like templates are probably slower than they should be :/

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/iVOJwI_v07kJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Alex Ogier]

On Wed, Oct 10, 2012 at 6:29 PM, Florian Apolloner wrote:

> That said, if you really think Django's urlresolver is to slow, go ahead,
> profile it and improve it, we certainly won't say no to speed improvements…
>

That's not really fair. Django core *will* say no to speed improvements if
they necessitate any backwards incompatibilities, which due to early design
decisions they often will.

The main reason Django's url resolver is slow is that it was designed to
allow some very flexible patterns. Just like middleware is slow because it
is globally configured and called on every request. And the template
language is slow because it allows arbitrary textual replacements and has a
flexible import system that defeats compilation.

Django has committed itself to backwards compatibility, which means it
tends to get larger and heavier over time, and decisions that were made
long ago have lasting impact. Speed is a real feature even in python, but
Django has very consciously chosen to prioritize other things. I consider
it a valid argument to say, "Django cares more about developer productivity
and backwards compatibility than framework speed," but "Fix it yourself" is
a hackneyed open-source truism, and "Speed doesn't matter" is more or less
self-evidently false -- especially to someone evaluating frameworks, who
may not yet have a strong opinion on what features are game-changers for
his application.

Best,
Alex Ogier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Yo-Yo Ma]

Why does every conversation about Django's performance met with "GTFO we don't 
care"? (that was a rhetorical question :). I'd venture to guess that most "It's 
fast enough for me!" responses are predicated on experiences that can be 
likened to personal blog development, rather than large scale, 10+ server 
deployments (e.g., Disqus, et al). 

If you had great abs, nice pecs, ripped legs, a nice deltoids, and 4" 
circumference biceps, you'd *probably* start to care when people said, "hey, 
what about those biceps...", no?

Are CPUs, RAM, and electricity free? (that too was a rhetorical question :)

Django is an *epic* (not in the literal sense :) framework, which offers a 
tremendous amount of functionality with nary an inkling of logical restraint. 
You can accomplish basically anything with Django without running into 
arbitrary limitations (e.g., have a gander at the limitations in most 
frameworks' URL dispatchers, just as a relevant example). To shorten this, from 
a features / ease of use standpoint, Django is just plain awesome.

BUT... Django is NOT that fast.

1) Django templates are unbearably slow (doesn't matter for a blog or something 
simple, but try something with lots of inline model formsets in a page and 
quite a few includes, and before you know it, you're wasting a decent 
percentage of your CPU on templates).

2) URL dispatching

3) Middleware is applied liberally (vs the a la carté, more efficient decorator 
approach to AOS)

The list goes on. What is the harm in discussing the weak points of Django? 
Performance is probably the only major weak point in Django right now (aside 
from the lack of schema migration in core, which is coming soon anyway).

I have no solution or patches in my pocket, but I can say with absolute 
certainty that performance will never, ever get better when discourse is in a 
monologue format.

That's all.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/uZ9r1EtZnj4J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Florian Apolloner]

Oh cmon,

please stop playing a socket puppet for the wheezy.web author. A web 
framework consists of more than just a win in speed (an the author of 
wheezy.web can argue whatever he wants that "basic" stuff stuff should be 
fast) -- if we were looking for speed we wouldn't use python at all 
(period). The goal is to be fast enough and be able to scale out 
horizontally, everything else is not really interesting in the case of web 
development. While I won't deny that Django isn't the fastest framework out 
there it certainly gets it's job done more than good enough and that's all 
it matters. I don't know what your goal is, but please stop posting such 
meaningless comparisons on django-dev…

That said, if you really think Django's urlresolver is to slow, go ahead, 
profile it and improve it, we certainly won't say no to speed improvements…

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/DgNcKfL2AeUJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Daniel Sokolowski]

Yes, wheezy.web is much more bare metal compared to Django : 
http://packages.python.org/wheezy.web/tutorial.html

From: Daniel Sokolowski 
Sent: Wednesday, October 10, 2012 10:32 AM
To: django-developers@googlegroups.com 
Subject: Re: URL dispatcher slow?

The middlewares appear to be disabled and the test bypasses the template system 
too: 
https://bitbucket.org/akorn/helloworld/src/c3f2d44dfca7/02-routing/django/helloworld/settings.py?at=default
 but yes Aymeric is right that Django provides a lot of convenience by default. 
In my opinion I rather take convenience as speed has not been a issue to me,  
but one thing though is that these benchmarks are  transparent enough to give 
them some validity. 

From: Aymeric Augustin 
Sent: Wednesday, October 10, 2012 9:30 AM
To: django-developers@googlegroups.com 
Subject: Re: URL dispatcher slow?

2012/10/10 Moonlight <moonlight_13_...@yahoo.com>

  Here is an article comparing various URL dispatchers:

  http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html

  What cause django URL dispatcher that much... slow?

Django's URL dispatcher is more complex than others. It provides advances 
features such as reversing, automatic i18n, and namespaces. Of course, these 
come at the expense of speed.

Note that this benchmark doesn't test URL dispatchers; it tests full stack 
requests. With Django's default settings, lots of convenient features such as 
middleware are enabled. This is likely to skew the results.

It'd be interesting to analyze what parts of the URL resolver are slow. If you 
really want to know what going on, go ahead and profile it. I'd be interested 
in the results (even though Django's performance has absolutely never been a 
problem for me).

As a side note, I read these benchmarks with a grain of salt. They're developed 
by the author of wheezy.web and consistently show it as a clear winner. Where's 
the line between testing and marketing?

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Daniel Sokolowski
http://webdesign.danols.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Daniel Sokolowski]

The middlewares appear to be disabled and the test bypasses the template system 
too: 
https://bitbucket.org/akorn/helloworld/src/c3f2d44dfca7/02-routing/django/helloworld/settings.py?at=default
 but yes Aymeric is right that Django provides a lot of convenience by default. 
In my opinion I rather take convenience as speed has not been a issue to me,  
but one thing though is that these benchmarks are  transparent enough to give 
them some validity. 

From: Aymeric Augustin 
Sent: Wednesday, October 10, 2012 9:30 AM
To: django-developers@googlegroups.com 
Subject: Re: URL dispatcher slow?

2012/10/10 Moonlight <moonlight_13_...@yahoo.com>

  Here is an article comparing various URL dispatchers:

  http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html

  What cause django URL dispatcher that much... slow?

Django's URL dispatcher is more complex than others. It provides advances 
features such as reversing, automatic i18n, and namespaces. Of course, these 
come at the expense of speed.

Note that this benchmark doesn't test URL dispatchers; it tests full stack 
requests. With Django's default settings, lots of convenient features such as 
middleware are enabled. This is likely to skew the results.

It'd be interesting to analyze what parts of the URL resolver are slow. If you 
really want to know what going on, go ahead and profile it. I'd be interested 
in the results (even though Django's performance has absolutely never been a 
problem for me).

As a side note, I read these benchmarks with a grain of salt. They're developed 
by the author of wheezy.web and consistently show it as a clear winner. Where's 
the line between testing and marketing?

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Daniel Sokolowski
http://webdesign.danols.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: URL dispatcher slow?

[Aymeric Augustin]

2012/10/10 Moonlight 

> Here is an article comparing various URL dispatchers:
>
> http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html
>
> What cause django URL dispatcher that much... slow?


Django's URL dispatcher is more complex than others. It provides advances
features such as reversing, automatic i18n, and namespaces. Of course,
these come at the expense of speed.

Note that this benchmark doesn't test URL dispatchers; it tests full stack
requests. With Django's default settings, lots of convenient features such
as middleware are enabled. This is likely to skew the results.

It'd be interesting to analyze what parts of the URL resolver are slow. If
you really want to know what going on, go ahead and profile it. I'd be
interested in the results (even though Django's performance has absolutely
never been a problem for me).

As a side note, I read these benchmarks with a grain of salt. They're
developed by the author of wheezy.web and consistently show it as a clear
winner. Where's the line between testing and marketing?

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

URL dispatcher slow?

[Moonlight]

Here is an article comparing various URL dispatchers:

http://mindref.blogspot.com/2012/10/python-web-routing-benchmark.html

What cause django URL dispatcher that much... slow?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/xJFGtYmJsBYJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.