Hi, I have downloaded the actual source code, and I have probably forgot to mention that I'm using django 1.7.10. It seems like the compiler.py module got a little bit improved since then, what used to be a big and highly inefficient loop with many if conditions inside was reduced to a small loop with a single if (it could be improved even more, taking the if out of the loop and doing two separate loops based on the if condition, avoiding checking the same if condition over and over when you already know its value, which is very slow on python since there's no JIT optimizing this kind of loops.). On another side, it also seems like ValuesListQuerySet was changed to ValuesListIterable, but the functionallity remains the same. So I should probably do the tests with this version to see if the call to results_iter is still a big deal for a values_list query.
I really wish that values_list (and values?) would be used as a high performance query option with optimizations in mind, rather than just the implicit improvements when returning a list of tuples rather than model instances which are very expensive for large queries. El domingo, 15 de noviembre de 2015, 23:10:08 (UTC-3), Cristiano Coelho escribió: > > After some testing I have found out that even when using values_list to > prevent massive object creation when fetching data, it is from 2 to 3 times > slower tan using directly a cursor.execute query through the django cursor. > The issue started here > http://stackoverflow.com/questions/33726004/django-queryset-vs-raw-query-performance > > when trying to improve some queries that were looking very slow on apache > benchemark testing. > It seems like compiler.results_iter, called from ValuesListQuerySet is > very, very slow, due to all the for loops in Python code compared to when > using a raw query through a C connector (like mysqlclient), there's just > too much boilerplate that might be posible to be removed. > As an example, a very ugly work around to those critical queries, was > defining a function like this, which would convery a queryset into > something usable by a cursor, which performs very, very efficiently, at > least with mysqlclient connector. > > q = qs.query.get_compiler(qs.db).as_sql() > with connection.cursor() as c: > c.execute(q[0], q[1]) > for r in c: > yield r > > Would it be possible to have something similar to values_list, but that > executes directly through a cursor, improving performance? I'm sure it will > be less flexible than values_list, but the extra performance will be nice. > > > > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2b0a8832-b81f-4591-a5c3-75569383c26d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
