Re: Django and Cython

[Curtis Maloney]

On 08/02/2018 07:02 AM, Daniel Anechitoaie wrote:
I'm looking at frameworks like https://vibora.io/ that make use of 
Cython to greatly improve the performance and I'm just wondering if 
something like this would work with Django?
Was there any discussion (taken into consideration) about using Cython 
to boost the performance of certain aspects of the framework?

Would it help make it faster?

I really love Python and Django for me is the best web framework out 
there, but you have to notice how fast other frameworks from other 
programming languages are.

Would using Cython help boost the performance?


There was a discussion some time ago about perhaps cythonising some of 
the template engine and other select parts of Django.  Someone did some 
eexcellent independant work on this, and you should be able to find 
their results in the mailing list archives.


However, I think you'll find the common wisdom on speeding up Django is 
to reach for tools like PyPy, which can provide hot-spot JIT 
compilation, thus focusing performance work on the parts where your app 
needs it.


Further to that, my own experiences of optimising and scaling django 
have shown often the performance bottlenecks are not where you expect.


As anyone in the industry will tell you, beware of benchmarks not 
testing what you _actually_ do.


One time we got a dramatic improvement in site resource usage by moving 
a user setting (like preferred language] into a cookie instead of the 
user session. This allowed many requests to never access the session 
store at all, avoiding a round-trip.


Another time, a site trebled in throughput (on average) by simply 
upgrading Django.  As is often the case, a smarter algorithm (in this 
case, in how QuerySets made copies) will yield improvements orders of 
magnitude more than minor optimisations.


As has been mentioned in other posts, async IO is often viewed as a 
solution to performance problems.  This is a furphy; asyncio can help 
improve resource utilisation when you have an IO-bound workload.


It can allow fewer processes to scale more readily to consume more CPU 
by avoiding blocking on IO.  It's trues that the majority of web sites 
are IO bound [waiting for DB, disk, or even other web sites].


However, this doesn't necessarily make each individual request any 
faster - in many cases, it can actually be slower.


As is always the case with improving performance, measurement and 
careful analysis are requried.


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d1fbf79c-6861-0ac4-d48e-f0316d0f3ca6%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: Django and Cython

[Tom Forbes]

The people showing you benchmarks comparing Django to ‘NodeJS’ are
comparing apples to oranges. Django is not an asynchronous framework (yet!)
so you cannot fairly compare the number of raw requests per second the two
handle and present that as evidence one is better than the other. Each one
has it’s strengths and weaknesses but Django wins hands down in a number of
places. Plus it’s not JavaScript, so that’s one thing Django has going for
it!

I’ve had immense success with uvloop and asyncio which are much more
comparable to NodeJS. Here are some synthetic benchmarks showing it handing
over 100,000 requests per second:
https://magic.io/blog/uvloop-blazing-fast-python-networking/. I would
recommend playing around with asyncio and aiohttp to build some small
services and benchmark them yourself, you might be surprised at the speed.

Regarding Cython: while some very specific parts of Django could
potentially benefit from the speedup Cython would bring we would need to
maintain two separate versions: A Cython one and a fallback Python one.
This would add a maintenance burden for an uncertain gain and even if you
implemented this the raw requests per second Django would reach would not
be close to that of an asynchronous framework with the same resources.
Typically Django apps are not CPU bound by the Django code itself, and a
lot of time is spent waiting on the network for the database or other
services.




On 3 August 2018 at 18:18:59, Daniel Anechitoaie (
daniel.anechito...@gmail.com) wrote:

So the fact that frameworks like Vibora and webapps developed in other
faster programming languages like GO/NodeJS/etc. support so many more req/s
than Django it's relevant only in benchmarks?
And in real life/live web apps the difference in performance won't be
noticeable?

I have to mention again that I really don't try to start any flame war and
I'm legit interested in this.
As a huge Python fan and unfortunately lone Python dev among my peers they
all flame me when I bring Python up and show me all kind of benchmarks how
fast NodeJS (as this is what they use) is vs Python.
I'm just trying to make sure I have ground to stand upon.


On Thursday, August 2, 2018 at 3:31:37 PM UTC+3, Jason Johns wrote:
>
> There was a discussion a while back about this https://groups.google.
> com/forum/#!searchin/django-developers/cython/django-
> developers/Fi4U602GxHA/mE50LOPkBgAJ
>
> tl;dr not sure what benefits Django would get from it, since the
> bottlenecks you experience are most likely non-Django/Python parts of your
> project, such as networking latency, db queries, connection initiation,
> etc.  In addition, pypy is an alternative interpreter that you can drop in
> for up to 3.5.x Python versions.
>
> On Wednesday, August 1, 2018 at 5:21:38 PM UTC-4, Daniel Anechitoaie
> wrote:
>>
>> I'm looking at frameworks like https://vibora.io/ that make use of
>> Cython to greatly improve the performance and I'm just wondering if
>> something like this would work with Django?
>> Was there any discussion (taken into consideration) about using Cython to
>> boost the performance of certain aspects of the framework?
>> Would it help make it faster?
>>
>> I really love Python and Django for me is the best web framework out
>> there, but you have to notice how fast other frameworks from other
>> programming languages are.
>> Would using Cython help boost the performance?
>>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/24d4d03d-d9ed-4d7f-9a69-8066846f090e%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNkcVcjHmuzJ35NAN2mLGhHbWopWribGj3_mGU4hDqcyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django and Cython

[Jason Johns]

Not at all, just that Cython itself is probably not the approach to take.  
Andrew Godwin recently posted a proposal for a Django ASGI roadmap 
 and 
there's a good amount of work to be done there.

Benchmarks have their place.  But if they're not matching real world 
problems, then I don't see much use of them other than stroking egos of 
their developers.  And my personal experience says that most of the 
problems with Django's slowness have most of the bottlenecks with db 
connections, networking latency and other things outside of Python.  And 
those problems inside Python might not be the best things to leverage 
Cython for.

I get the issue of wanting to show up to your peers with node, but 
personally, I just don't find it worth it.  It has its use cases, but more 
and more I see it being squashed into a use case its really not suited for 
simply because the creator or team are very one-dimensional as far as 
languages..  Use the right tool for the job, and just ignore the fanbois.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3a82c5de-b3d0-449a-8e6b-2f008fd9c803%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django and Cython

[Daniel Anechitoaie]

So the fact that frameworks like Vibora and webapps developed in other 
faster programming languages like GO/NodeJS/etc. support so many more req/s 
than Django it's relevant only in benchmarks?
And in real life/live web apps the difference in performance won't be 
noticeable?

I have to mention again that I really don't try to start any flame war and 
I'm legit interested in this.
As a huge Python fan and unfortunately lone Python dev among my peers they 
all flame me when I bring Python up and show me all kind of benchmarks how 
fast NodeJS (as this is what they use) is vs Python. 
I'm just trying to make sure I have ground to stand upon. 


On Thursday, August 2, 2018 at 3:31:37 PM UTC+3, Jason Johns wrote:
>
> There was a discussion a while back about this 
> https://groups.google.com/forum/#!searchin/django-developers/cython/django-developers/Fi4U602GxHA/mE50LOPkBgAJ
>
> tl;dr not sure what benefits Django would get from it, since the 
> bottlenecks you experience are most likely non-Django/Python parts of your 
> project, such as networking latency, db queries, connection initiation, 
> etc.  In addition, pypy is an alternative interpreter that you can drop in 
> for up to 3.5.x Python versions.
>
> On Wednesday, August 1, 2018 at 5:21:38 PM UTC-4, Daniel Anechitoaie wrote:
>>
>> I'm looking at frameworks like https://vibora.io/ that make use of 
>> Cython to greatly improve the performance and I'm just wondering if 
>> something like this would work with Django?
>> Was there any discussion (taken into consideration) about using Cython to 
>> boost the performance of certain aspects of the framework?
>> Would it help make it faster?
>>
>> I really love Python and Django for me is the best web framework out 
>> there, but you have to notice how fast other frameworks from other 
>> programming languages are.
>> Would using Cython help boost the performance?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/24d4d03d-d9ed-4d7f-9a69-8066846f090e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django and Cython

[Jason Johns]

There was a discussion a while back about this 
https://groups.google.com/forum/#!searchin/django-developers/cython/django-developers/Fi4U602GxHA/mE50LOPkBgAJ

tl;dr not sure what benefits Django would get from it, since the 
bottlenecks you experience are most likely non-Django/Python parts of your 
project, such as networking latency, db queries, connection initiation, 
etc.  In addition, pypy is an alternative interpreter that you can drop in 
for up to 3.5.x Python versions.

On Wednesday, August 1, 2018 at 5:21:38 PM UTC-4, Daniel Anechitoaie wrote:
>
> I'm looking at frameworks like https://vibora.io/ that make use of Cython 
> to greatly improve the performance and I'm just wondering if something like 
> this would work with Django?
> Was there any discussion (taken into consideration) about using Cython to 
> boost the performance of certain aspects of the framework?
> Would it help make it faster?
>
> I really love Python and Django for me is the best web framework out 
> there, but you have to notice how fast other frameworks from other 
> programming languages are.
> Would using Cython help boost the performance?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7173fb49-48d3-48c0-a711-0671af91d297%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.