Re: Threads and db connection handling question

[Cristiano Coelho]

Some of the pools might have some high load (such as the one that handles 
logging to the database and other sources) so opening and closing a 
connection for each call might end up bad.

Now that you mention django code, does 
connection.close_if_unusable_or_obsolete() always close the connection, or 
does it also handle the case where persistent connections are used (and so 
the connection is not closed if it is alive and in good state) ? If so, 
would it possible to simply replicate what django does on every request 
start and end as a wrapper of every function that's sent to the pool? That 
way connections are re used if possible and recycled/closed if they go bad.

Looking at the code this seems to be called on every request start and end.

def close_old_connections(**kwargs):
for conn in connections.all():
conn.close_if_unusable_or_obsolete()
signals.request_started.connect(close_old_connections)
signals.request_finished.connect(close_old_connections)

I guess something could be done but just with that thread's connection, 
then all functions sent to the pool will need to be sent through a wrapper 
that does this before and after every call.

El jueves, 2 de junio de 2016, 20:50:13 (UTC-3), Stephen Butler escribió:
>
> Do you expect your background threads to be equivalent to or greater than 
> the number of requests you're normally servicing? Usually background tasks 
> are much less frequent than the web requests, so a little overhead w/r/t 
> database connections isn't even going to be noticed.
>
> Looking at what Django does, at the start and end of each request it calls 
> connection.close_if_unusable_or_obsolete(). That function does careful 
> checks to see if the connection is even worth using. Unless you do 
> something similar in your thread_start (adding more complication than I've 
> suggested), having a TLS connection will cause more problems than it saves 
> you. To make this work in general you'd at least need a hook at the point 
> the thread is removed from and added back to the pool, not when the thread 
> exits.
>
> Also, the connection won't be opened unless you actually do something that 
> needs it.
>
> Personally, I think this sounds like something you're trying to optimize 
> before you've profiled that the benefit it worth it.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3d668979-fae3-4a77-b9b4-fe9e09f763fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Stephen J. Butler]

Do you expect your background threads to be equivalent to or greater than
the number of requests you're normally servicing? Usually background tasks
are much less frequent than the web requests, so a little overhead w/r/t
database connections isn't even going to be noticed.

Looking at what Django does, at the start and end of each request it calls
connection.close_if_unusable_or_obsolete(). That function does careful
checks to see if the connection is even worth using. Unless you do
something similar in your thread_start (adding more complication than I've
suggested), having a TLS connection will cause more problems than it saves
you. To make this work in general you'd at least need a hook at the point
the thread is removed from and added back to the pool, not when the thread
exits.

Also, the connection won't be opened unless you actually do something that
needs it.

Personally, I think this sounds like something you're trying to optimize
before you've profiled that the benefit it worth it.



On Thu, Jun 2, 2016 at 6:33 PM, Cristiano Coelho 
wrote:

> I'm not starting threads by hand but rather a pool which handles any
> threads for me, I basically just send a function to the pool and leave it
> run.
> You are right, I could wrap every function sent to the pool with the code
> you proposed, but I also don't want to open and close a connection on every
> function call, but instead only when the thread from the pool is no longer
> required and disposed, pretty much on application exist, although the pool
> handler can do what ever it wants with the threads.
>
> El jueves, 2 de junio de 2016, 20:22:21 (UTC-3), Stephen Butler escribió:
>>
>> I'm still a bit confused. What is the advantage of having connections
>> closed automatically when the thread exits? It seems to me that you can
>> quickly solve your problem by modifying your thread start routines:
>>
>> from django.db import connection
>> from contextlib import closing
>>
>> def my_thread_start():
>> with closing(connection):
>> # do normal work
>>
>> You can even create a quick decorator if that's too much modification.
>>
>> On Thu, Jun 2, 2016 at 5:48 PM, Cristiano Coelho 
>> wrote:
>>
>>> Florian,
>>>
>>> Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere
>>> some time ago that SIGTERM would instantly finish any pending request, so I
>>> assumed it would also kill any thread in not a really nice way. However now
>>> that you mention it, there's one SIGKILL from the apache logs (compared to
>>> the thousands of sigterm due to restarts). However the connections that
>>> were somehow stuck and never close dated from about 2 weeks ago, yes, there
>>> were connections that were opened 2 weeks ago and never closed, even if
>>> apache was restarded many times every day!
>>>
>>> About thread pool, I'm talking about python's thread pool I'm using to
>>> offload work, not any django's pool, and these pools are the ones I have no
>>> control over its threads as they are completely managed by the thread pool
>>> library.
>>>
>>> 3 days has passed since I noticed those hanged out connections and the
>>> issue didn't repeat again yet, maybe it was some really odd condition that
>>> caused them, but the thread pool's threads connections are indeed being
>>> correctly closed on servers restart, so a very odd case created those
>>> hanged connections.
>>>
>>> So just to be sure, is SIGTERM actually propagated to python code so it
>>> can gracefully kill all threads, garbage collect and close connections?
>>> Would a SIGKILL actually prevent any kind of cleanup leaving a chance for
>>> python/django leave some connections opened?
>>>
>>> Maybe this is a postgres issue instead that happened for some very odd
>>> reason.
>>>
>>>
>>> Finally, would it be possible through any kind of callbacks of the
>>> thread local object to fire a connection close before a thread dies? This
>>> would certainly help rather than waiting for the connection to get garbage
>>> collected. You mentioned that connections could end up being shared by
>>> threads, but I don't see that being something being done in django at all.
>>>
>>>
>>> El jueves, 2 de junio de 2016, 19:32:15 (UTC-3), Florian Apolloner
>>> escribió:

 On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote:
>
> Not always, for example, on amazon elastic beasntalk when you either
> restart the app server or upload a new version, it basically kills apache
> and all WSGI processes through a sigterm
>

 A SIGTERM is a normal signal an should cause a proper shutdown.


> so those thread pools are probably killed in a bad way and you don't
> really have control over that.
>

 Absolutely not, you are mixing up SIGTERM and SIGKILL.


> Also you don't really have control on the life of a thread pool
> thread, so a given thread could be gracefully stopped by 

Re: Threads and db connection handling question

[Cristiano Coelho]

I'm not starting threads by hand but rather a pool which handles any 
threads for me, I basically just send a function to the pool and leave it 
run.
You are right, I could wrap every function sent to the pool with the code 
you proposed, but I also don't want to open and close a connection on every 
function call, but instead only when the thread from the pool is no longer 
required and disposed, pretty much on application exist, although the pool 
handler can do what ever it wants with the threads.

El jueves, 2 de junio de 2016, 20:22:21 (UTC-3), Stephen Butler escribió:
>
> I'm still a bit confused. What is the advantage of having connections 
> closed automatically when the thread exits? It seems to me that you can 
> quickly solve your problem by modifying your thread start routines:
>
> from django.db import connection
> from contextlib import closing
>
> def my_thread_start():
> with closing(connection):
> # do normal work
>
> You can even create a quick decorator if that's too much modification.
>
> On Thu, Jun 2, 2016 at 5:48 PM, Cristiano Coelho  > wrote:
>
>> Florian, 
>>
>> Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere 
>> some time ago that SIGTERM would instantly finish any pending request, so I 
>> assumed it would also kill any thread in not a really nice way. However now 
>> that you mention it, there's one SIGKILL from the apache logs (compared to 
>> the thousands of sigterm due to restarts). However the connections that 
>> were somehow stuck and never close dated from about 2 weeks ago, yes, there 
>> were connections that were opened 2 weeks ago and never closed, even if 
>> apache was restarded many times every day!
>>
>> About thread pool, I'm talking about python's thread pool I'm using to 
>> offload work, not any django's pool, and these pools are the ones I have no 
>> control over its threads as they are completely managed by the thread pool 
>> library.
>>
>> 3 days has passed since I noticed those hanged out connections and the 
>> issue didn't repeat again yet, maybe it was some really odd condition that 
>> caused them, but the thread pool's threads connections are indeed being 
>> correctly closed on servers restart, so a very odd case created those 
>> hanged connections.
>>
>> So just to be sure, is SIGTERM actually propagated to python code so it 
>> can gracefully kill all threads, garbage collect and close connections? 
>> Would a SIGKILL actually prevent any kind of cleanup leaving a chance for 
>> python/django leave some connections opened?
>>
>> Maybe this is a postgres issue instead that happened for some very odd 
>> reason.
>>
>>
>> Finally, would it be possible through any kind of callbacks of the thread 
>> local object to fire a connection close before a thread dies? This would 
>> certainly help rather than waiting for the connection to get garbage 
>> collected. You mentioned that connections could end up being shared by 
>> threads, but I don't see that being something being done in django at all.
>>
>>
>> El jueves, 2 de junio de 2016, 19:32:15 (UTC-3), Florian Apolloner 
>> escribió:
>>>
>>> On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote:

 Not always, for example, on amazon elastic beasntalk when you either 
 restart the app server or upload a new version, it basically kills apache 
 and all WSGI processes through a sigterm

>>>
>>> A SIGTERM is a normal signal an should cause a proper shutdown.
>>>  
>>>
 so those thread pools are probably killed in a bad way and you don't 
 really have control over that.

>>>
>>> Absolutely not, you are mixing up SIGTERM and SIGKILL.
>>>  
>>>
 Also you don't really have control on the life of a thread pool thread, 
 so a given thread could be gracefully stopped by the pool implementation

>>>
>>> Once again: there is no pool. 
>>>
>>> As ayneric pointed out, it seems like those connections are correctly 
 closed most of the time when a thread dies, but for some reason, postgres 
 would keep some connections opened.

>>>
>>> If a connection is closed properly, postgres will close it accordingly. 
>>> The only way possible for a connection to stay open while the app is gone 
>>> is that you are running into tcp timeouts while getting killed with SIGTERM 
>>> (dunno if the postgres protocol has keep alive support on the protocol 
>>> level, most likely not). As long as you are not sending a SIGTERM, python 
>>> should clean up properly which should call garbage collection, which then 
>>> again should delete all connections and therefore close the connection. Any 
>>> other behavior seems like a bug in Python (or maybe Django, but as long as 
>>> Python shuts down properly I think we are fine).
>>>
>>> Are there any rare cases where even if the thread is stopped the 
 connection won't be closed? The only thing I can think of are that those 
 threads are never garbage 

Re: Threads and db connection handling question

[Stephen J. Butler]

I'm still a bit confused. What is the advantage of having connections
closed automatically when the thread exits? It seems to me that you can
quickly solve your problem by modifying your thread start routines:

from django.db import connection
from contextlib import closing

def my_thread_start():
with closing(connection):
# do normal work

You can even create a quick decorator if that's too much modification.

On Thu, Jun 2, 2016 at 5:48 PM, Cristiano Coelho 
wrote:

> Florian,
>
> Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere
> some time ago that SIGTERM would instantly finish any pending request, so I
> assumed it would also kill any thread in not a really nice way. However now
> that you mention it, there's one SIGKILL from the apache logs (compared to
> the thousands of sigterm due to restarts). However the connections that
> were somehow stuck and never close dated from about 2 weeks ago, yes, there
> were connections that were opened 2 weeks ago and never closed, even if
> apache was restarded many times every day!
>
> About thread pool, I'm talking about python's thread pool I'm using to
> offload work, not any django's pool, and these pools are the ones I have no
> control over its threads as they are completely managed by the thread pool
> library.
>
> 3 days has passed since I noticed those hanged out connections and the
> issue didn't repeat again yet, maybe it was some really odd condition that
> caused them, but the thread pool's threads connections are indeed being
> correctly closed on servers restart, so a very odd case created those
> hanged connections.
>
> So just to be sure, is SIGTERM actually propagated to python code so it
> can gracefully kill all threads, garbage collect and close connections?
> Would a SIGKILL actually prevent any kind of cleanup leaving a chance for
> python/django leave some connections opened?
>
> Maybe this is a postgres issue instead that happened for some very odd
> reason.
>
>
> Finally, would it be possible through any kind of callbacks of the thread
> local object to fire a connection close before a thread dies? This would
> certainly help rather than waiting for the connection to get garbage
> collected. You mentioned that connections could end up being shared by
> threads, but I don't see that being something being done in django at all.
>
>
> El jueves, 2 de junio de 2016, 19:32:15 (UTC-3), Florian Apolloner
> escribió:
>>
>> On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote:
>>>
>>> Not always, for example, on amazon elastic beasntalk when you either
>>> restart the app server or upload a new version, it basically kills apache
>>> and all WSGI processes through a sigterm
>>>
>>
>> A SIGTERM is a normal signal an should cause a proper shutdown.
>>
>>
>>> so those thread pools are probably killed in a bad way and you don't
>>> really have control over that.
>>>
>>
>> Absolutely not, you are mixing up SIGTERM and SIGKILL.
>>
>>
>>> Also you don't really have control on the life of a thread pool thread,
>>> so a given thread could be gracefully stopped by the pool implementation
>>>
>>
>> Once again: there is no pool.
>>
>> As ayneric pointed out, it seems like those connections are correctly
>>> closed most of the time when a thread dies, but for some reason, postgres
>>> would keep some connections opened.
>>>
>>
>> If a connection is closed properly, postgres will close it accordingly.
>> The only way possible for a connection to stay open while the app is gone
>> is that you are running into tcp timeouts while getting killed with SIGTERM
>> (dunno if the postgres protocol has keep alive support on the protocol
>> level, most likely not). As long as you are not sending a SIGTERM, python
>> should clean up properly which should call garbage collection, which then
>> again should delete all connections and therefore close the connection. Any
>> other behavior seems like a bug in Python (or maybe Django, but as long as
>> Python shuts down properly I think we are fine).
>>
>> Are there any rare cases where even if the thread is stopped the
>>> connection won't be closed? The only thing I can think of are that those
>>> threads are never garbage collected or something.
>>>
>>
>> Depends on the python version you are using, especially thread local
>> behavior changed a lot…
>>
>> Cheers,
>> Florian
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/562bf2f9-8a44-495c-bacd-9a42012aa011%40googlegroups.com
> 

Re: Threads and db connection handling question

[Cristiano Coelho]

Florian, 

Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere 
some time ago that SIGTERM would instantly finish any pending request, so I 
assumed it would also kill any thread in not a really nice way. However now 
that you mention it, there's one SIGKILL from the apache logs (compared to 
the thousands of sigterm due to restarts). However the connections that 
were somehow stuck and never close dated from about 2 weeks ago, yes, there 
were connections that were opened 2 weeks ago and never closed, even if 
apache was restarded many times every day!

About thread pool, I'm talking about python's thread pool I'm using to 
offload work, not any django's pool, and these pools are the ones I have no 
control over its threads as they are completely managed by the thread pool 
library.

3 days has passed since I noticed those hanged out connections and the 
issue didn't repeat again yet, maybe it was some really odd condition that 
caused them, but the thread pool's threads connections are indeed being 
correctly closed on servers restart, so a very odd case created those 
hanged connections.

So just to be sure, is SIGTERM actually propagated to python code so it can 
gracefully kill all threads, garbage collect and close connections? Would a 
SIGKILL actually prevent any kind of cleanup leaving a chance for 
python/django leave some connections opened?

Maybe this is a postgres issue instead that happened for some very odd 
reason.


Finally, would it be possible through any kind of callbacks of the thread 
local object to fire a connection close before a thread dies? This would 
certainly help rather than waiting for the connection to get garbage 
collected. You mentioned that connections could end up being shared by 
threads, but I don't see that being something being done in django at all.


El jueves, 2 de junio de 2016, 19:32:15 (UTC-3), Florian Apolloner escribió:
>
> On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote:
>>
>> Not always, for example, on amazon elastic beasntalk when you either 
>> restart the app server or upload a new version, it basically kills apache 
>> and all WSGI processes through a sigterm
>>
>
> A SIGTERM is a normal signal an should cause a proper shutdown.
>  
>
>> so those thread pools are probably killed in a bad way and you don't 
>> really have control over that.
>>
>
> Absolutely not, you are mixing up SIGTERM and SIGKILL.
>  
>
>> Also you don't really have control on the life of a thread pool thread, 
>> so a given thread could be gracefully stopped by the pool implementation
>>
>
> Once again: there is no pool. 
>
> As ayneric pointed out, it seems like those connections are correctly 
>> closed most of the time when a thread dies, but for some reason, postgres 
>> would keep some connections opened.
>>
>
> If a connection is closed properly, postgres will close it accordingly. 
> The only way possible for a connection to stay open while the app is gone 
> is that you are running into tcp timeouts while getting killed with SIGTERM 
> (dunno if the postgres protocol has keep alive support on the protocol 
> level, most likely not). As long as you are not sending a SIGTERM, python 
> should clean up properly which should call garbage collection, which then 
> again should delete all connections and therefore close the connection. Any 
> other behavior seems like a bug in Python (or maybe Django, but as long as 
> Python shuts down properly I think we are fine).
>
> Are there any rare cases where even if the thread is stopped the 
>> connection won't be closed? The only thing I can think of are that those 
>> threads are never garbage collected or something.
>>
>
> Depends on the python version you are using, especially thread local 
> behavior changed a lot…
>
> Cheers,
> Florian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/562bf2f9-8a44-495c-bacd-9a42012aa011%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Friday, June 3, 2016 at 12:32:15 AM UTC+2, Florian Apolloner wrote:
>
> while getting killed with SIGTERM (dunno if the postgres protocol has keep 
> alive support on the protocol level, most likely not). As long as you are 
> not sending a SIGTERM
>

Ups, now I am myself mixing up SIGKILL and SIGTERM -- I ment SIGKILL here 
which cannot be caught by the program itself. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/49f12755-1e01-427b-b28b-3a551eafdfe0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote:
>
> Not always, for example, on amazon elastic beasntalk when you either 
> restart the app server or upload a new version, it basically kills apache 
> and all WSGI processes through a sigterm
>

A SIGTERM is a normal signal an should cause a proper shutdown.
 

> so those thread pools are probably killed in a bad way and you don't 
> really have control over that.
>

Absolutely not, you are mixing up SIGTERM and SIGKILL.
 

> Also you don't really have control on the life of a thread pool thread, so 
> a given thread could be gracefully stopped by the pool implementation
>

Once again: there is no pool. 

As ayneric pointed out, it seems like those connections are correctly 
> closed most of the time when a thread dies, but for some reason, postgres 
> would keep some connections opened.
>

If a connection is closed properly, postgres will close it accordingly. The 
only way possible for a connection to stay open while the app is gone is 
that you are running into tcp timeouts while getting killed with SIGTERM 
(dunno if the postgres protocol has keep alive support on the protocol 
level, most likely not). As long as you are not sending a SIGTERM, python 
should clean up properly which should call garbage collection, which then 
again should delete all connections and therefore close the connection. Any 
other behavior seems like a bug in Python (or maybe Django, but as long as 
Python shuts down properly I think we are fine).

Are there any rare cases where even if the thread is stopped the connection 
> won't be closed? The only thing I can think of are that those threads are 
> never garbage collected or something.
>

Depends on the python version you are using, especially thread local 
behavior changed a lot…

Cheers,
Florian 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0387d845-c490-4527-b33f-e0caf99696a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Cristiano Coelho]

El jueves, 2 de junio de 2016, 11:48:33 (UTC-3), Florian Apolloner escribió:
>
>
> No it would not be great at all, connections could theoretically shared 
> between threads etc… In general Django has no way of knowing when you want 
> to close it. In the end a "dying" thread* which is not properly closed is 
> a bug in your code anyways.*
>
> Cheers,
> Florian
>  
>

Not always, for example, on amazon elastic beasntalk when you either 
restart the app server or upload a new version, it basically kills apache 
and all WSGI processes through a sigterm, so those thread pools are 
probably killed in a bad way and you don't really have control over that. 
Also you don't really have control on the life of a thread pool thread, so 
a given thread could be gracefully stopped by the pool implementation but 
you can't really do any cleanup code before it happens for that thread (at 
least not that I'm aware of for multiprocessing.pool.ThreadPool)

As ayneric pointed out, it seems like those connections are correctly 
closed most of the time when a thread dies, but for some reason, postgres 
would keep some connections opened. Are there any rare cases where even if 
the thread is stopped the connection won't be closed? The only thing I can 
think of are that those threads are never garbage collected or something.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/797440be-cc41-44ac-b382-10dd5ea2cb5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Thursday, June 2, 2016 at 5:15:55 PM UTC+2, Aymeric Augustin wrote:
>
> > On 02 Jun 2016, at 13:39, Cristiano Coelho  > wrote: 
> > 
> > it would be great for them to be automatically closed/disposed for you 
> on thread's death, which right now seems to happen some times, and some 
> times not, leaking connections (something I'm trying to figure out what's 
> going on). 
>
> In the scenario you’re discussing here, connections are closed when Python 
> garbage collects them, which can cause seemingly random behavior. 
>

Assuming a new enough python (2.7.1 iirc), that should reliably happen once 
a thread is destroyed -- earlier versions would keep the connection till 
the actual thread local went out of scope. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4ae7f872-46da-4999-be0d-ce688b01b170%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Aymeric Augustin]

Hi Cristiano,

> On 02 Jun 2016, at 13:39, Cristiano Coelho  wrote:
> 
> it would be great for them to be automatically closed/disposed for you on 
> thread's death, which right now seems to happen some times, and some times 
> not, leaking connections (something I'm trying to figure out what's going on).

In the scenario you’re discussing here, connections are closed when Python 
garbage collects them, which can cause seemingly random behavior.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/E4C7D2F2-2153-4189-BDEE-4AB2464F8E80%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts on upgrade pain & deprecations.

[Tom Christie]

> I'm against adding custom behavior in Django

That's entirely reasonable, yes.

In which case, what do folks think of the following suggestions:

* Linking prominently to the 'upgrading django 
' section 
from at the begining of every new version's release notes.
* Tweaking the upgrading django section so that it recommends *first* 
running your existing test case with warnings enabled and resolving those 
at that point, *before* upgrading django, and *then* upgrading and running 
the tests again, at which point you don't necessarily need warnings 
enabled. (As it currently stands it's rather the wrong way around)
* Tweaking the upgrading django section so that it also mentions making 
sure that your test runner isn't silencing the warning output. (eg "Using 
the py.test runner, you probably want to use something like 
`PYTHONWARNINGS=default py.test tests -s`")

Cheers,

  Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/88cc77e2-fe2b-480d-8378-9fbd33e7745a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Thursday, June 2, 2016 at 1:39:51 PM UTC+2, Cristiano Coelho wrote:
>
> So what was stated on the stack overflow post that connections are somehow 
> closed only at the end of a request through the request end signal is still 
> the actual behavior?
>

Dunno, I do not read SO. Connections are closed at the start and end of 
request depending on the configuration.
 

> Any best / suggested practices on how to handle connections on threads 
> that are not part of the request cycle? 
>

Same as in any other (non-Django) program: Manually call 
connection.connect/close() at the relevant points.
 

> Considering the connections are automatically opened for you, it would be 
> great for them to be automatically closed/disposed for you on thread's death
>

No it would not be great at all, connections could theoretically shared 
between threads etc… In general Django has no way of knowing when you want 
to close it. In the end a "dying" thread which is not properly closed is a 
bug in your code anyways.

Cheers,
Florian
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d88b9c64-e51e-4714-8f36-d6091ee06765%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts on upgrade pain & deprecations.

[Tim Graham]

 
>
> Aside: I think the `.urls = ...` -> `@override_settings(ROOT_URLCONF=...)` 
> change to test cases is missing from those notes?
>

The 1.10 release notes say, "SimpleTestCase.urls is removed."
 

> I expected to have seen a deprecation warning when running the tests under 
> 1.9, although wasn't exactly sure if and when those warnings would be 
> displayed.
>

Deprecation warnings are loud by default when using "manage.py test", 
however, I've removed that behavior starting with Django 1.11 as per 
https://docs.djangoproject.com/en/dev/releases/1.11/#deprecating-warnings-are-no-longer-loud-by-default.
 
I'll probably duplicate those tips in the "how to upgrade Django" doc 
(https://docs.djangoproject.com/en/dev/howto/upgrade-version/).

(The behavior still isn't ideal then as each instance of warning is output 
> once every time for each occurrence, rather than once and once only)
>

I don't think it's a sensible default to see each warning once rather than 
each time the deprecated behavior is invoked. For some warnings it might 
make sense, but for others, then you'd have to rerun the test suite after 
fixing each warning to see if the warning pops up elsewhere (unless I'm 
missing your rationale?).
 

> The upshot of all this is that I believe many of our users are likely to 
> be hitting problems with upgrades because they're not seeing deprecation 
> warnings, and then get hit by a behavioral change that they weren't 
> expecting to have to deal with, and end up having to debug the cause of.
>

I guess this is currently limited to anyone not using "manage.py test", but 
will affect all users starting with Django 1.11 per the previously 
mentioned changed.
 

> For instance, a user is running under 1.9 and wants to upgrade to 1.10. 
> Could we have a reliable and documented mechanism by which they'd first run 
> the tests under 1.9, and have any Deprecation warnings treated as errors?
>
If they're running 1.8, could we document how to run the tests so that any 
> PendingDeprecation warnings are treated as errors?
>

warnings.simplefilter("error", RemovedInDjango20Warning)

With the latest deprecation/release schedule, I think technique described 
in the 1.11 release notes should work for the majority of cases. If 
projects want to support consecutive LTS releases, they shouldn't worry 
about deprecation warnings until a "dot zero" release, at which point they 
should follow the tip and fix all warnings.
 

> Could we collate instances of these warnings, rather than displaying them 
> in an overly verbose fashion?
>

I'm against adding custom behavior in Django (like what's been removed in 
https://github.com/django/django/commit/5b94b17feff15a9f0345f92fc0568bfe7038e3a3)
 
that makes warnings work differently from in Python.
 

> An example of the sort of thing we could do might be to have an 
> environment variable `DJANGO_UPGRADE`, that if set to a version number, 
> forces the relevant class of deprecation / pending deprecation warnings to 
> be written to a log file or similar. I don't think that interface is quite 
> right, but suggesting it as an example of how we might make our user's 
> lives easier.
>

I've never had trouble with the console output format myself, but I suppose 
you could experiment with some third-party package that does what you want 
and we'll see if people find it useful.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1ede7c88-41d2-40ca-a1da-a6e3f345ad49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts on upgrade pain & deprecations.

[Josh Smeaton]

I like the general idea. Going with your low tech solution we could have 
snippets for the X most popular test runners to treat warnings as errors. 
Perhaps ./manage test.py --warnings-as-errors or similar for the interface 
django provides. Every time django makes a release there are some that 
express frustration at deprecations, so making the fixing process easier 
can only help there. 

On Thursday, 2 June 2016 22:12:54 UTC+10, Tom Christie wrote:
>
> The low tech solution to this, may just be to have the release notes 
> recommend running your test suite using
> `PYTHONWARNINGS=once`, and making sure not to swallow any output, eg. with 
> py.test that'd be...
>
> PYTHONWARNINGS=once py.test tests -s
>
>
> I'm fairly sure that making that point explicitly would save an awful lot 
> of folks a more painful upgrade, by making it more clear how to see 
> upcoming issues without getting bitten by them and having to debug without 
> the help of an associated warning.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/78834b25-d2a4-4752-af1d-580e07815302%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Possible Bug (MYSQL)

[Josh Smeaton]

Hi Paulo,

Have you opened a ticket on Trac  for this 
yet? If not, please do so. This mailing list isn't really for verifying 
bugs. That said, does the query work if you provide an actual alias for the 
aggregate?

*Company.objects.annotate(max_pk=Max('employee__pk')).filter(employee__pk=F('max_pk'))*

I suspect there's an ordering issue when doing field resolving that's not 
checking the annotated_select dict first. Maybe. Open up a ticket on Trac 
and we can discuss further there.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e59a-d885-4178-9458-3fd149145369%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts on upgrade pain & deprecations.

[Tom Christie]

The low tech solution to this, may just be to have the release notes 
recommend running your test suite using
`PYTHONWARNINGS=once`, and making sure not to swallow any output, eg. with 
py.test that'd be...

PYTHONWARNINGS=once py.test tests -s


I'm fairly sure that making that point explicitly would save an awful lot 
of folks a more painful upgrade, by making it more clear how to see 
upcoming issues without getting bitten by them and having to debug without 
the help of an associated warning.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c23e2934-b929-41b5-b3d4-572f0be8a66f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Cristiano Coelho]

So what was stated on the stack overflow post that connections are somehow 
closed only at the end of a request through the request end signal is still 
the actual behavior?

Any best / suggested practices on how to handle connections on threads that 
are not part of the request cycle? Considering the connections are 
automatically opened for you, it would be great for them to be 
automatically closed/disposed for you on thread's death, which right now 
seems to happen some times, and some times not, leaking connections 
(something I'm trying to figure out what's going on).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/36c63148-a6c1-44f2-b862-57a1cb750c01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Some thoughts on upgrade pain & deprecations.

[Tom Christie]

I've just finished upgrading Django REST framework to support 1.10, and 
it's given me some food for thought on things that can make Django upgrades 
painful.

Firstly, here's a couple of things that I needed to track down in order to 
upgrade that weren't obvious at first...

* A bunch of test views started returning 404s.
* None of the app templates were being found.
* `user` was no longer being passed into template contexts.

All of these were due to changes that have been part of the standard 
deprecation cycle, which is fair enough, however they weren't necessarily 
all that easy to debug. Figuring them out involves digging through the set 
of removed features 
, 
and trying to figure out which of those would result in the behavior change 
that's been seen.

Aside: I think the `.urls = ...` -> `@override_settings(ROOT_URLCONF=...)` 
change to test cases is missing from those notes?

I expected to have seen a deprecation warning when running the tests under 
1.9, although wasn't exactly sure if and when those warnings would be 
displayed.

Seeing the deprecation warnings under 1.9 would have made the upgrade far 
simpler, as I'd be warned about the changes I needed to make, rather than 
seeing a behavioral change and having to debug what might have caused it.

Figuring out how to ensure that the tests properly displayed deprecation 
warnings (under py.test) wasn't all that easy either, eventually I got 
there with...

   python -Wd [...]/py.test tests/ -s

(The behavior still isn't ideal then as each instance of warning is output 
once every time for each occurrence, rather than once and once only)

The upshot of all this is that I believe many of our users are likely to be 
hitting problems with upgrades because they're not seeing deprecation 
warnings, and then get hit by a behavioral change that they weren't 
expecting to have to deal with, and end up having to debug the cause of.

I'm wondering if we could do more to help our users here?

For instance, a user is running under 1.9 and wants to upgrade to 1.10. 
Could we have a reliable and documented mechanism by which they'd first run 
the tests under 1.9, and have any Deprecation warnings treated as errors?

If they're running 1.8, could we document how to run the tests so that any 
PendingDeprecation warnings are treated as errors?

Could we collate instances of these warnings, rather than displaying them 
in an overly verbose fashion?

python -Werror sort of achieves this, but doesn't differentiate between 
deprecation and pending deprecation.

An example of the sort of thing we could do might be to have an environment 
variable `DJANGO_UPGRADE`, that if set to a version number, forces the 
relevant class of deprecation / pending deprecation warnings to be written 
to a log file or similar. I don't think that interface is quite right, but 
suggesting it as an example of how we might make our user's lives easier.

Any other suggestions or thoughts in this area?

  Tom


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dd7a2cda-12fd-42e4-8ecb-cfec60c1d69f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

Hi,

Django does not really use a Pool from which you can check out connections 
and return them back to the pool. Every thread has it's own connection 
which is initialized on first use -- there is nothing which closes 
connections (or checks for their lifetime with persistent connections) 
automatically if you are outside of a standard request/response cycle -- 
that is on you to ensure.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4182f220-3424-4bef-902b-ccea9d1b23f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.