Re: Should we require pytz for timezone support in Django?

[Philip James]

Is there consensus on making pytz required? (I, personally, am in favor, 
since I too can't remember the last project I did that didn't need it.)

If it's required, should it be vendored?

On Thursday, May 19, 2016 at 11:10:09 PM UTC-7, akki wrote:
>
> Corresponding ticket - #26622 
> .
>
> On Tuesday, 17 May 2016 05:51:29 UTC+5:30, Josh Smeaton wrote:
>>
>> While writing timezone tests for 
>> https://github.com/django/django/pull/6243 I ran into some issues where 
>> pytz seemed to be required for just about every database and platform 
>> combination except postgres on linux. The docs for timezone support (
>> https://docs.djangoproject.com/en/dev/topics/i18n/timezones/) say:
>>
>> Installing pytz  is highly recommended, 
>>> but may not be mandatory depending on your particular database backend, 
>>> operating system and time zone. If you encounter an exception querying 
>>> dates or times, please try installing it before filing a bug. 
>>
>>
>> I tried to find some documentation on the broader internet that would 
>> more accurately describe when and for what platforms pytz is actually 
>> required for timezone support, but was unable to find anything. I've opened 
>> a new ticket https://code.djangoproject.com/ticket/26622 to clarify more 
>> explicitly when pytz is required.
>>
>> However, I'm wondering if we should just go all-in on pytz and require it 
>> for timezone support. I've not run a Django site with timezone support 
>> without pytz, so I'm not sure what exact limitations exist or why you'd 
>> want to use timezone support without pytz. I'd be interested in hearing use 
>> cases for not requiring pytz if there are any. Explicitly requiring pytz 
>> will make test writing a lot easier, and will remove any doubt about 
>> whether or not pytz is required for users.
>>
>> Josh 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/16ce6fb6-c787-452b-a1b3-5f9891f8c80e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Cristiano Coelho]

Aymeric, I have never said anything about connection pool, I'm talking 
about thread pooling to perform async work (without the need to spawn a new 
thread every time and have control over the amount data that is offloaded) 
and the behaviour of django connections when used on a separate thread 
that's not part of the request/response cycle, since if you don't take 
care, everything will work as expected because django is nice enough to 
open db connections for you even if it is from a user spawned thread, but 
it won't close them for you, so when working with user spawned threads 
special care needs to be taken on the db connections.

I have added changes to the thread's pool code so it always wraps functions 
between connection.close_if_unusable_or_obsolete()  so the thread pool 
threads can always have a healthy connection and hopefully close invalid 
ones. I don't know if this is related to the connection leak I got that I 
can't seem to reproduce now but I will be keepin an eye.

It would be good to have a big warning on the django docs about doing 
queries on threads that are spawned by the user and are not part of a 
request/response cycle, leting the users know they have to explicitly close 
them.

El viernes, 3 de junio de 2016, 10:01:38 (UTC-3), Aymeric Augustin escribió:
>
> Hello,
>
> I have to say that, as the author of the “persistent connections” feature, 
> I am confused by repeated references to a “connection pool” in this 
> discussion. I chose *not* to implement a connection pool because these 
> aren’t easy to get right. Users who need a connection pool are better off 
> with a third-party option such as pgpool.
>
> When persistent connections are enabled, each thread uses a persistent 
> connection to the database — as opposed to one connection per HTTP request. 
> That said connections aren’t shared or pooled between threads. This 
> guarantees that each connection dies when the thread that opened it dies.
>
> In practice, Django opens a database connection per thread and keeps it 
> open after each request. When the next request comes in, if the connection 
> still works (this is tested with something like “SELECT 1”) and its maximum 
> age isn’t reached, Django re-uses it; otherwise it closes it and opens a 
> new one. This is what the “close_if_unusable_or_obsolete” function does — 
> as the name says :-)
>
> I hope this helps,
>
> -- 
> Aymeric.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/78b396c7-3085-416a-9fa9-568e4aedf836%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Friday, June 3, 2016 at 1:48:05 PM UTC+2, Cristiano Coelho wrote:
>
> El viernes, 3 de junio de 2016, 4:41:16 (UTC-3), Florian Apolloner 
> escribió:
>>
>>
>> Yes, SIGTERM is a signal Python should handle nicely, SIGKILL will just 
>> nuke the process from earth and prevent any cleanup I think. Note that the 
>> OS will clean up though (sooner or later). As for the open connections: 
>> Check on postgres where from they are and then check on that machine to 
>> which process they belong and maybe use gdb to find out more.
>>  
>>
>
> When I saw the issue the first thing I checked was the source IP and was 
> the amazon machine as expected, then I'm almost sure those connections were 
> used by the thread pools
>

Don't be almost sure, investigate were that connection is coming from… 
 

> The only issue I see here on the postgres side would be that you have very 
>> long timeouts configured.
>>
>
> Looks like postgres doesn't have any kind of connection timeout and you 
> need to use pg bouncer for that.
>

If the app died indeed, tcp keep alive should help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ea3288a8-e67f-4319-9f67-d2e59cc342bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Aymeric Augustin]

Hello,

I have to say that, as the author of the “persistent connections” feature, I am 
confused by repeated references to a “connection pool” in this discussion. I 
chose *not* to implement a connection pool because these aren’t easy to get 
right. Users who need a connection pool are better off with a third-party 
option such as pgpool.

When persistent connections are enabled, each thread uses a persistent 
connection to the database — as opposed to one connection per HTTP request. 
That said connections aren’t shared or pooled between threads. This guarantees 
that each connection dies when the thread that opened it dies.

In practice, Django opens a database connection per thread and keeps it open 
after each request. When the next request comes in, if the connection still 
works (this is tested with something like “SELECT 1”) and its maximum age isn’t 
reached, Django re-uses it; otherwise it closes it and opens a new one. This is 
what the “close_if_unusable_or_obsolete” function does — as the name says :-)

I hope this helps,

-- 
Aymeric.

> On 03 Jun 2016, at 13:48, Cristiano Coelho  wrote:
> 
> 
> El viernes, 3 de junio de 2016, 4:41:16 (UTC-3), Florian Apolloner escribió:
> 
> Yes, SIGTERM is a signal Python should handle nicely, SIGKILL will just nuke 
> the process from earth and prevent any cleanup I think. Note that the OS will 
> clean up though (sooner or later). As for the open connections: Check on 
> postgres where from they are and then check on that machine to which process 
> they belong and maybe use gdb to find out more.
>  
> 
> When I saw the issue the first thing I checked was the source IP and was the 
> amazon machine as expected, then I'm almost sure those connections were used 
> by the thread pools because the last statement was a 'commit' where normally 
> most of the statements listed through the list connections query are selects 
> and those pools always end up with an insert so the commit makes sense.
> 
> The only issue I see here on the postgres side would be that you have very 
> long timeouts configured.
> 
> Looks like postgres doesn't have any kind of connection timeout and you need 
> to use pg bouncer for that.
>  
> 
> Well, I guess that with the connection.close_if_unusable_or_obsolete() change 
> it should improve the connection handling of the thread pools.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/5fbbb59e-4ed6-4795-b8e8-c97da791d3d1%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/E5C1BFEB-CB5A-4E5D-A450-22F1361761A1%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Friday, June 3, 2016 at 2:17:01 AM UTC+2, Cristiano Coelho wrote:
>
> Now that you mention django code, does 
> connection.close_if_unusable_or_obsolete() always close the connection, or 
> does it also handle the case where persistent connections are used (and so 
> the connection is not closed if it is alive and in good state) ? 
>

Yes, it does handle persistent connections properly, otherwise Django would 
not work ;) 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ab632e95-c07d-4cea-a6cc-9b6633fa262a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Threads and db connection handling question

[Florian Apolloner]

On Friday, June 3, 2016 at 12:48:26 AM UTC+2, Cristiano Coelho wrote:
>
> So just to be sure, is SIGTERM actually propagated to python code so it 
> can gracefully kill all threads, garbage collect and close connections? 
> Would a SIGKILL actually prevent any kind of cleanup leaving a chance for 
> python/django leave some connections opened?
>

Yes, SIGTERM is a signal Python should handle nicely, SIGKILL will just 
nuke the process from earth and prevent any cleanup I think. Note that the 
OS will clean up though (sooner or later). As for the open connections: 
Check on postgres where from they are and then check on that machine to 
which process they belong and maybe use gdb to find out more.
 

> Maybe this is a postgres issue instead that happened for some very odd 
> reason.
>

The only issue I see here on the postgres side would be that you have very 
long timeouts configured.
 

> Finally, would it be possible through any kind of callbacks of the thread 
> local object to fire a connection close before a thread dies? This would 
> certainly help rather than waiting for the connection to get garbage 
> collected. 
>

Killing a thread and garbage collection should happen at the same time 
roughly due to refcounting imo.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9ce40d73-7e0c-4060-b3a9-7704b91c5d48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.