Re: Django blocking on writing logs

[Avraham Serour]

it doesn't seem to have any relation to a synchronous logger

On Tue, Apr 10, 2018 at 1:16 AM, Ameni Bejaoui 
wrote:

> I face this problem when i want to test the django interface in the web
> browser local domain :/ what should i do ? i change even the port to 8080
> the same problem
>
> 2018-04-09 20:27 GMT+01:00 Avraham Serour :
>
>> It seems you are right, I found this SO thread dealing with the same
>> question: https://stackoverflow.com/questions/34349797/are-
>> log-statements-blocking-in-django-python
>>
>> In pratical terms I use sentry, the python client claims to use a
>> threaded async approach, I suggest taking a look at their implementation,
>> or just plugging sentry to your app.
>>
>> On Thu, Apr 5, 2018 at 7:51 PM, Daniel Arndt 
>> wrote:
>>
>>> Hi all,
>>>
>>> Although not explicitly mentioned anywhere, I've come to the conclusion
>>> Django blocks while writing logs with some handlers (it uses Python
>>> logging, https://docs.python.org/3/howto/logging-cookbook.ht
>>> ml#dealing-with-handlers-that-block).
>>>
>>> I can't find anything that explicitly states Django handles this already
>>> (in the code or documentation) and I believe it to be a problem in the
>>> platform I am working on. Justification for this belief is loose at best,
>>> but it's the best lead I have (specifically, the SysLogHandler seems like
>>> it might be blocking on sending to a UDP port, which seems crazy but
>>> apparently is possible)
>>>
>>> I'm setting trying to see if I can solve this, but there seems to be
>>> very little details on this subject, and that alone worries me. So I have a
>>> couple questions for anyone willing to entertain me:
>>>
>>>
>>>1. Is it silly to be worried about this? My gut says yes, simply
>>>because nobody else is. At the same time I see no real downside to 
>>> logging
>>>asynchronously (is that true?). There are potential upsides. It might not
>>>solve the real problem (what is causing buffers to back up in the first
>>>place? likely crazy I/O usage) but it might speed up handling of requests
>>>during heavy I/O.
>>>2. Have you done anything similar to this? How did you approach it?
>>>3. The QueueHandler suggested in the Python cookbook doesn't seem to
>>>be easy to stuff into Django. We have Django running behind Gunicorn, so 
>>> my
>>>thought was to have Gunicorn spin up a QueueListener process, and then 
>>> have
>>>Gunicorn pass the queue to each worker similar to this cookbook:
>>>https://docs.python.org/3/howto/logging-cookbook.h
>>>tml#logging-to-a-single-file-from-multiple-processes
>>>
>>> 
>>>-- I'm interested on any thoughts on this approach. Especially if there 
>>> is
>>>a better way.
>>>
>>> Cheers,
>>> Dan
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-users.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-users/ce269c78-bd6b-4e06-b9de-2da6ccc79cd8%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-users/CAFWa6tKBZ5Nq8BwOVBWxpyg%3DS6nzV_%3DhXF_
>> km9A-bT_pFXfbtw%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/CAHBt3Ny%2BspWTP4-um4j%3DmtU1z-
> 1HC0jeij9HTtaV_N77Q_R%3DRQ%40mail.gmail.com
> 

Re: Django blocking on writing logs

[Avraham Serour]

It seems you are right, I found this SO thread dealing with the same
question:
https://stackoverflow.com/questions/34349797/are-log-statements-blocking-in-django-python

In pratical terms I use sentry, the python client claims to use a threaded
async approach, I suggest taking a look at their implementation, or just
plugging sentry to your app.

On Thu, Apr 5, 2018 at 7:51 PM, Daniel Arndt  wrote:

> Hi all,
>
> Although not explicitly mentioned anywhere, I've come to the conclusion
> Django blocks while writing logs with some handlers (it uses Python
> logging, https://docs.python.org/3/howto/logging-cookbook.
> html#dealing-with-handlers-that-block).
>
> I can't find anything that explicitly states Django handles this already
> (in the code or documentation) and I believe it to be a problem in the
> platform I am working on. Justification for this belief is loose at best,
> but it's the best lead I have (specifically, the SysLogHandler seems like
> it might be blocking on sending to a UDP port, which seems crazy but
> apparently is possible)
>
> I'm setting trying to see if I can solve this, but there seems to be very
> little details on this subject, and that alone worries me. So I have a
> couple questions for anyone willing to entertain me:
>
>
>1. Is it silly to be worried about this? My gut says yes, simply
>because nobody else is. At the same time I see no real downside to logging
>asynchronously (is that true?). There are potential upsides. It might not
>solve the real problem (what is causing buffers to back up in the first
>place? likely crazy I/O usage) but it might speed up handling of requests
>during heavy I/O.
>2. Have you done anything similar to this? How did you approach it?
>3. The QueueHandler suggested in the Python cookbook doesn't seem to
>be easy to stuff into Django. We have Django running behind Gunicorn, so my
>thought was to have Gunicorn spin up a QueueListener process, and then have
>Gunicorn pass the queue to each worker similar to this cookbook:
>https://docs.python.org/3/howto/logging-cookbook.
>html#logging-to-a-single-file-from-multiple-processes
>
> 
>-- I'm interested on any thoughts on this approach. Especially if there is
>a better way.
>
> Cheers,
> Dan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/ce269c78-bd6b-4e06-b9de-2da6ccc79cd8%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFWa6tKBZ5Nq8BwOVBWxpyg%3DS6nzV_%3DhXF_km9A-bT_pFXfbtw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django blocking on writing logs

[David Bauer]

No I gave up on Django, don't know what it does, and don't have the money
to take a class to learn more, so I just gave up for right now, when I see
some money I might be able to take a class

On Thu, Apr 5, 2018 at 9:51 AM, Daniel Arndt  wrote:

> Hi all,
>
> Although not explicitly mentioned anywhere, I've come to the conclusion
> Django blocks while writing logs with some handlers (it uses Python
> logging, https://docs.python.org/3/howto/logging-cookbook.
> html#dealing-with-handlers-that-block).
>
> I can't find anything that explicitly states Django handles this already
> (in the code or documentation) and I believe it to be a problem in the
> platform I am working on. Justification for this belief is loose at best,
> but it's the best lead I have (specifically, the SysLogHandler seems like
> it might be blocking on sending to a UDP port, which seems crazy but
> apparently is possible)
>
> I'm setting trying to see if I can solve this, but there seems to be very
> little details on this subject, and that alone worries me. So I have a
> couple questions for anyone willing to entertain me:
>
>
>1. Is it silly to be worried about this? My gut says yes, simply
>because nobody else is. At the same time I see no real downside to logging
>asynchronously (is that true?). There are potential upsides. It might not
>solve the real problem (what is causing buffers to back up in the first
>place? likely crazy I/O usage) but it might speed up handling of requests
>during heavy I/O.
>2. Have you done anything similar to this? How did you approach it?
>3. The QueueHandler suggested in the Python cookbook doesn't seem to
>be easy to stuff into Django. We have Django running behind Gunicorn, so my
>thought was to have Gunicorn spin up a QueueListener process, and then have
>Gunicorn pass the queue to each worker similar to this cookbook:
>https://docs.python.org/3/howto/logging-cookbook.
>html#logging-to-a-single-file-from-multiple-processes
>
> 
>-- I'm interested on any thoughts on this approach. Especially if there is
>a better way.
>
> Cheers,
> Dan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/ce269c78-bd6b-4e06-b9de-2da6ccc79cd8%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
David A. Bauer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAJ%3DpNhEDYg_DWxorXDCtbCkLtMQfxatuh16apEFwfwf%3DREtW-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Django blocking on writing logs

[Daniel Arndt]

Hi all,

Although not explicitly mentioned anywhere, I've come to the conclusion 
Django blocks while writing logs with some handlers (it uses Python 
logging, 
https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block).

I can't find anything that explicitly states Django handles this already 
(in the code or documentation) and I believe it to be a problem in the 
platform I am working on. Justification for this belief is loose at best, 
but it's the best lead I have (specifically, the SysLogHandler seems like 
it might be blocking on sending to a UDP port, which seems crazy but 
apparently is possible)

I'm setting trying to see if I can solve this, but there seems to be very 
little details on this subject, and that alone worries me. So I have a 
couple questions for anyone willing to entertain me:


   1. Is it silly to be worried about this? My gut says yes, simply because 
   nobody else is. At the same time I see no real downside to logging 
   asynchronously (is that true?). There are potential upsides. It might not 
   solve the real problem (what is causing buffers to back up in the first 
   place? likely crazy I/O usage) but it might speed up handling of requests 
   during heavy I/O.
   2. Have you done anything similar to this? How did you approach it?
   3. The QueueHandler suggested in the Python cookbook doesn't seem to be 
   easy to stuff into Django. We have Django running behind Gunicorn, so my 
   thought was to have Gunicorn spin up a QueueListener process, and then have 
   Gunicorn pass the queue to each worker similar to this 
   cookbook: 
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
 
   -- I'm interested on any thoughts on this approach. Especially if there is 
   a better way.

Cheers,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ce269c78-bd6b-4e06-b9de-2da6ccc79cd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.