Hey Karl and Russell, Thank you for your quick replies! Karl: Agreed, duplicate CI runs would have to be performed (which would double the time or double the number of runners required). As I understand it the Django project itself would not distribute pre-compiled wheels, the setup.py Cython 'stuff' would handle this (and fail gracefully if anything goes wrong, like no C compiler being available). The type annotations sounds interesting and would alleviate a fair bit of engineering effort, keeping two duplicate copies in sync sounded horrible.
Russell: The point release issues certainly sound troubling and almost make me want to give up based on that alone. Can you elaborate on them at all - where you doing anything particularly crazy or complex with your cythonified code? You are of course right about the speed of the Python code itself being a bottleneck and the usual suspects like the database are more important to optimize. However I don't think it's necessarily always a waste of time to explore optimizing some Python/django functions, if that only means simply moving them to a separate file and running Cython on them to get a speed boost. That's the dream at least, but it's rarely that simple in practice. After perusing the Django code for some functions that look like they could benefit from Cython it seems a lot are tightly coupled and could not be extracted without a bit of effort. Plus the engineering/ci/release overhead would be considerable. So, perhaps it seems this is just a pipe dream and not worth the effort. Thanks for replying anyway! Tom On 21 May 2017 23:36, "Russell Keith-Magee" <[email protected]> wrote: Hi Tom, My immediate reaction is No, for three reasons: 1. My experience has been that Cython isn’t especially stable. Admittedly, I haven’t looked at it for a couple of years, but when I did, I ended up getting caught in some really nasty bugs that came back and forth between micro versions. 2. Even if Cython *was* stable: The execution speed of your Django stack is almost certainly *not* the bottleneck of your application. Query time, database transfer time, and just basic client-end connection latency will, for most applications, be a *much* bigger performance problem than the execution time of the Python stack. 3. Even if the Django code in your app *was* your bottleneck, switching to PyPy as your interpreter will almost certainly give you better performance for less engineering effort. If you want to do some experimentation, by all means go right ahead; however, I would caution you that any patch you produce will need to demonstrate a *significant* improvement in real-world use cases for us to adopt the engineering overhead of integrating Cython into Django’s runtime environment. Yours Russ Magee %-) On 21 May 2017, 2:59 PM -0700, Tom Forbes <[email protected]>, wrote: Hello, There was a very interesting talk at Pycon about using Cython to speed up hotspots in Python programs: https://www.youtube.com/watch?v=_1MSX7V28Po It got me wondering about possibly using Cython in selected places within Django. I realize since Django was first released the distribution situation was a bit more wild-west, resulting in part to Django not relying on any third party dependencies. But that situation is rapidly changing (see https://github.com/django/deps/blob/master/draft/0007-depend ency-policy.rst#background-and-motivation) and with these changes could it also be a time to investigate Cython usage for select parts of Django? Several popular projects use Cython 'speedup' modules with pure-python fallbacks with great success, for example aiohttp ( https://github.com/aio-libs/aiohttp/blob/master/setup.py#L20). I did some quick and dirty profiling of the 'django.utils.html.escape' function and found that by simply including Cython as part of the build, and with no syntax changes, the function executes twice as fast. There are lots of considerations to take into account (like ensuring the Cython functions are in sync with the fallback ones), but it seems that it could make a big difference with small, self contained functions (like html.escape or html.escapejs) that are executed frequently as part of a request. Other functions that might be worth looking at include core.http.mutliparser.parse_header or utils.baseconv.BaseConverter.convert. My question is: this this something that's worth exploring, or is it outside of the realms of possibility? -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/ms gid/django-developers/CAFNZOJOsAdL422Ntj4cUkYF1bjqUBdMAXp33x Z%3DapSwqXMasvA%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CAFNZOJOsAdL422Ntj4cUkYF1bjqUBdMAXp33xZ%3DapSwqXMasvA%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/ms gid/django-developers/0c83b001-6654-4c6d-a134-e5dd7a6b8dc8%40Spark <https://groups.google.com/d/msgid/django-developers/0c83b001-6654-4c6d-a134-e5dd7a6b8dc8%40Spark?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJPuYAt-KypkthydrAD0SJbyPbPqY5r8r2CWnJJoQ4SBLQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
