Is it possible to open up multiple database connections inside of one
request in Django? I am using Django 1.6 and Python3.3.
In my use case, I have a web application that executes raw SQL against a
data warehouse we have. For example, one page may require five different
SQL queries to be executed to pull down all the datasets necessary to
display as charts, tables, etc.
Using only one database connection and executing the SQL queries
synchronously is too slow, so I am trying to use threads (specifically the
concurrent.futures package in stdlib) to run my queries asynchronously. In
particular, I have code that looks like:
from django.db import connection
from concurrent.futures import ThreadPoolExecutor, as_completed
def execute_query(sql):
cursor = connection.cursor()
…
rows = cursor.fetchall()
return rows
def execute_queries_asynchronously(list_of_sql_queries):
datasets = []
with ThreadPoolExecutor(max_workers=3) as executor:
futures = [executor.submit(execute_query, query) for query in
list_of_sql_queries]
for future in as_completed(futures):
datasets.append(future.result())
return datasets
When I run the above code in shell, total execution time improves
noticeably which is expected (about 2x-3x faster). However, when I put this
code in my web application, I get an almost negligible performance gain, if
any. It doesn’t seem like Django is opening up more DB connections. I’d
like Django to open up a new connection or pull from the connection pool
for every worker thread. Does the Django connection handler limit the
number of available DB connections to a request to only one?
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/cb7ebe95-1565-4f4b-84cd-e25a36a7497a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
