Re: Threads and db connection handling question

So in the end, i have to manually close the database connection after my threads in my threadpool are done executing, even though, django created them automatically when the threads were first created? This is just plain bad design. On Tuesday, June 21, 2016 at 6:04:11 PM UTC+5:30, Aymeric

Re: Threads and db connection handling question

On Tuesday, June 21, 2016 at 2:34:11 PM UTC+2, Aymeric Augustin wrote: > Regarding database connections, if you weren’t using Django, you’d have to > create a database connection with something like `psycopg2.connect()` and > dispose of it with `connection.close()` when you no longer need

Re: Threads and db connection handling question

On 21 Jun 2016, at 12:53, Marcin Nowak wrote: > It looks like it is completely unreliable in async environment, and the worst > thing is that I can't force Django to create new connection within the > thread/process. > > Well, they'll say that Django is not designed

Re: Threads and db connection handling question

> From version to version it turns more into handy tool for blog > creators/webdesigners > Don't get me wrong. For smal/short-term projects (up to 1-2yr of operation) I'm still using Django and I would recommend it everyone for that kind of job. For long-term projects the first thing that

Re: Threads and db connection handling question

On Saturday, June 4, 2016 at 12:12:42 AM UTC+2, Cristiano Coelho wrote: > > Aymeric, I have never said anything about connection pool, I'm talking > about thread pooling to perform async work > I have similar requirements and issues with Django. It looks like it is completely unreliable in

Re: Threads and db connection handling question

Aymeric, I have never said anything about connection pool, I'm talking about thread pooling to perform async work (without the need to spawn a new thread every time and have control over the amount data that is offloaded) and the behaviour of django connections when used on a separate thread

Re: Threads and db connection handling question

On Friday, June 3, 2016 at 1:48:05 PM UTC+2, Cristiano Coelho wrote: > > El viernes, 3 de junio de 2016, 4:41:16 (UTC-3), Florian Apolloner > escribió: >> >> >> Yes, SIGTERM is a signal Python should handle nicely, SIGKILL will just >> nuke the process from earth and prevent any cleanup I

Re: Threads and db connection handling question

Hello, I have to say that, as the author of the “persistent connections” feature, I am confused by repeated references to a “connection pool” in this discussion. I chose *not* to implement a connection pool because these aren’t easy to get right. Users who need a connection pool are better off

Re: Threads and db connection handling question

On Friday, June 3, 2016 at 2:17:01 AM UTC+2, Cristiano Coelho wrote: > > Now that you mention django code, does > connection.close_if_unusable_or_obsolete() always close the connection, or > does it also handle the case where persistent connections are used (and so > the connection is not

Re: Threads and db connection handling question

On Friday, June 3, 2016 at 12:48:26 AM UTC+2, Cristiano Coelho wrote: > > So just to be sure, is SIGTERM actually propagated to python code so it > can gracefully kill all threads, garbage collect and close connections? > Would a SIGKILL actually prevent any kind of cleanup leaving a chance

Re: Threads and db connection handling question

Some of the pools might have some high load (such as the one that handles logging to the database and other sources) so opening and closing a connection for each call might end up bad. Now that you mention django code, does connection.close_if_unusable_or_obsolete() always close the

Re: Threads and db connection handling question

Do you expect your background threads to be equivalent to or greater than the number of requests you're normally servicing? Usually background tasks are much less frequent than the web requests, so a little overhead w/r/t database connections isn't even going to be noticed. Looking at what Django

Re: Threads and db connection handling question

I'm not starting threads by hand but rather a pool which handles any threads for me, I basically just send a function to the pool and leave it run. You are right, I could wrap every function sent to the pool with the code you proposed, but I also don't want to open and close a connection on

Re: Threads and db connection handling question

I'm still a bit confused. What is the advantage of having connections closed automatically when the thread exits? It seems to me that you can quickly solve your problem by modifying your thread start routines: from django.db import connection from contextlib import closing def my_thread_start():

Re: Threads and db connection handling question

Florian, Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere some time ago that SIGTERM would instantly finish any pending request, so I assumed it would also kill any thread in not a really nice way. However now that you mention it, there's one SIGKILL from the apache

Re: Threads and db connection handling question

On Friday, June 3, 2016 at 12:32:15 AM UTC+2, Florian Apolloner wrote: > > while getting killed with SIGTERM (dunno if the postgres protocol has keep > alive support on the protocol level, most likely not). As long as you are > not sending a SIGTERM > Ups, now I am myself mixing up SIGKILL and

Re: Threads and db connection handling question

On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote: > > Not always, for example, on amazon elastic beasntalk when you either > restart the app server or upload a new version, it basically kills apache > and all WSGI processes through a sigterm > A SIGTERM is a normal signal

Re: Threads and db connection handling question

El jueves, 2 de junio de 2016, 11:48:33 (UTC-3), Florian Apolloner escribió: > > > No it would not be great at all, connections could theoretically shared > between threads etc… In general Django has no way of knowing when you want > to close it. In the end a "dying" thread* which is not

Re: Threads and db connection handling question

On Thursday, June 2, 2016 at 5:15:55 PM UTC+2, Aymeric Augustin wrote: > > > On 02 Jun 2016, at 13:39, Cristiano Coelho > wrote: > > > > it would be great for them to be automatically closed/disposed for you > on thread's death, which right now seems to happen some

Re: Threads and db connection handling question

Hi Cristiano, > On 02 Jun 2016, at 13:39, Cristiano Coelho wrote: > > it would be great for them to be automatically closed/disposed for you on > thread's death, which right now seems to happen some times, and some times > not, leaking connections (something I'm

Re: Threads and db connection handling question

On Thursday, June 2, 2016 at 1:39:51 PM UTC+2, Cristiano Coelho wrote: > > So what was stated on the stack overflow post that connections are somehow > closed only at the end of a request through the request end signal is still > the actual behavior? > Dunno, I do not read SO. Connections are

Re: Threads and db connection handling question

So what was stated on the stack overflow post that connections are somehow closed only at the end of a request through the request end signal is still the actual behavior? Any best / suggested practices on how to handle connections on threads that are not part of the request cycle? Considering

Re: Threads and db connection handling question

Hi, Django does not really use a Pool from which you can check out connections and return them back to the pool. Every thread has it's own connection which is initialized on first use -- there is nothing which closes connections (or checks for their lifetime with persistent connections)

Re: Threads and db connection handling question

Yes, mainly some things that needs to be done async and you don't want to keep up the request for those such as email sending or logging. So this work is just offloaded to a thread pool and the request returns instantly. Also when you are using auto scaling services from amazon and since each

Re: Threads and db connection handling question

That's pretty close but on a much more difficult level, since it is about multi processing? Things starts to get very odd with multi processing and django, compared to threads that you can easily launch a new thread or pool without any special work, it is just the database connections handling

Re: Threads and db connection handling question

Here's a ticket requesting the documentation you seek (as far as I understand): https://code.djangoproject.com/ticket/20562 On Wednesday, June 1, 2016 at 6:34:05 PM UTC-4, Cristiano Coelho wrote: > > Let me start saying sorry if this actually belongs to django-users rather > than developers. >

Threads and db connection handling question

Let me start saying sorry if this actually belongs to django-users rather than developers. I'm curious about (and if someone can point me to the code) how exactly django handles database connections (in particular when persistent connections are used). As I can tell from the docs, they are