Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-13 Thread Carsten Fuchs 
Hello Graham,

many thanks for your kind and detailed help!

Best regards,
Carsten


Am 10.02.23 um 01:07 schrieb Graham Dumpleton:
> Use WSGIApplicationGroup to force the use of the main Python sub interpreter. 
> That is only safe way to deal with it.
> 
> I have created issue against CPython itself to separately highlight the 
> problem.
> 
> 101767.png
> sqlite callback functions can deadlock in Python subinterpreters · Issue 
> #101767 · python/cpython 
> github.com 
> 
> 
> 
> You might also look at:
> 
> 59956.png
> [subinterpreters] Make the PyGILState API compatible with subinterpreters · 
> Issue #59956 · python/cpython 
> github.com 
> 
> 
> 
> which is the long running issue about the underlying problem in CPython.
> 
> Graham
> 

-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/modwsgi/11ec9ac0-a3b7-b677-4190-ee44b53c6a98%40cafu.de.

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Carsten Fuchs 
Argh, sorry, I think I may have spoken too soon: The deadsnakes PPA does not 
contain Python 3.10 for Ubuntu 22.04 and I'm not really prepared to compile 
from source.

Would it help you, mod_wsgi or someone else if I pursued installing the latest 
Python 3.10 point release for further testing?

Otherwise, now that the reason for the problem is clear, I consider to just use 
the `WSGIApplicationGroup %{GLOBAL}` or switch to another database in order to 
evade the issue.

Best regards,
Carsten


Am 08.02.23 um 21:53 schrieb Carsten Fuchs:
> Many thanks for your help and the detailed information!
> My Python version is 3.10.6.
> I'll try to switch to the latest point release next, but it will a couple of 
> days before I can do it. Will let you know the results as soon as I got it 
> done.
> 
> Best regards,
> Carsten
> 
> 
> 
> Am 08.02.23 um 21:28 schrieb Graham Dumpleton:
>> Possibly related issues:
>>
>> https://bugs.archlinux.org/task/72979 
>>  
>> https://bugs.python.org/issue46006 
>>
>> There are possibly some known issues with sub interpreters in code used by 
>> sqlite module there.
>>
>> What exact version of Python 3.10 as you using? Can you update to latest 
>> patch revision of it?
>>
>> Graham
>>
>>> On 9 Feb 2023, at 7:22 am, Graham Dumpleton  
>>> wrote:
>>>
>>> More specifically, if you look at the stack trace you see:
>>>
>>> #6  0x7f68aefe6292 in PyEval_RestoreThread () from 
>>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>>> #7  0x7f68af02f1dc in PyGILState_Ensure () from 
>>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>>> #8  0x7f68ac30d6e3 in ?? () from 
>>> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
>>> #9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
>>> target:/lib/x86_64-linux-gnu/libsqlite3.so.0
>>>
>>> Note how that function ends up calling PyGILState_Ensure(). That is a big 
>>> no no when working with Python sub interpreters.
>>>
>>> In other words, it uses the simplified GIL state API for some callback into 
>>> Python from C code of SQLite and using that  when in a sub interpreter can 
>>> cause deadlocks.
>>>
>>> Application Issues — mod_wsgi 4.9.4 documentation 
>>> 
>>> modwsgi.readthedocs.io 
>>> 
>>>  
>>> 
>>>
>>> 
>>>
>>> So need to find the code for that function in sqlite module and work out 
>>> what it does.
>>>
>>> Graham
>>>
 On 9 Feb 2023, at 7:09 am, Graham Dumpleton  
 wrote:

 The key bit of information in that is the call to 
 sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python 
 is doing something really strange that isn't working with sub 
 interpreters. This will need some research as it may be a bug they have 
 introduced in Python 3.10 for the sqlite module.

 Graham

> On 9 Feb 2023, at 7:02 am, Carsten Fuchs  wrote:
>
> Ok, I got a stack trace. My understanding of this is limited, but the 
> thread seems to be stuck at polling something without timeout? (gdb 
> reports that file ../sysdeps/unix/sysv/linux/poll.c cannot be found)
>
> The full session follows. (Using plaintext email today in the hope that 
> this helps with readibility. Please let me know if I should send 
> differently instead.)
> Between the two "thread apply all bt" commands I waited about two minutes:
>
>
> $ sudo gdb /usr/sbin/apache2 60501
> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
> Copyright (C) 2022 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
>   .
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/apache2...
> (No debugging symbols found in /usr/sbin/apache2)
> Attaching to program: /usr/sbin/apache2, process 60501
> [New LWP 60532]
> [New LWP 60533]
> [New LWP

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Carsten Fuchs 
Many thanks for your help and the detailed information!
My Python version is 3.10.6.
I'll try to switch to the latest point release next, but it will a couple of 
days before I can do it. Will let you know the results as soon as I got it done.

Best regards,
Carsten



Am 08.02.23 um 21:28 schrieb Graham Dumpleton:
> Possibly related issues:
> 
> https://bugs.archlinux.org/task/72979  
> https://bugs.python.org/issue46006 
> 
> There are possibly some known issues with sub interpreters in code used by 
> sqlite module there.
> 
> What exact version of Python 3.10 as you using? Can you update to latest 
> patch revision of it?
> 
> Graham
> 
>> On 9 Feb 2023, at 7:22 am, Graham Dumpleton  
>> wrote:
>>
>> More specifically, if you look at the stack trace you see:
>>
>> #6  0x7f68aefe6292 in PyEval_RestoreThread () from 
>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #7  0x7f68af02f1dc in PyGILState_Ensure () from 
>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #8  0x7f68ac30d6e3 in ?? () from 
>> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
>> #9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
>> target:/lib/x86_64-linux-gnu/libsqlite3.so.0
>>
>> Note how that function ends up calling PyGILState_Ensure(). That is a big no 
>> no when working with Python sub interpreters.
>>
>> In other words, it uses the simplified GIL state API for some callback into 
>> Python from C code of SQLite and using that  when in a sub interpreter can 
>> cause deadlocks.
>>
>> Application Issues — mod_wsgi 4.9.4 documentation 
>> 
>> modwsgi.readthedocs.io 
>> 
>>   
>> 
>>
>> 
>>
>> So need to find the code for that function in sqlite module and work out 
>> what it does.
>>
>> Graham
>>
>>> On 9 Feb 2023, at 7:09 am, Graham Dumpleton  
>>> wrote:
>>>
>>> The key bit of information in that is the call to 
>>> sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python 
>>> is doing something really strange that isn't working with sub interpreters. 
>>> This will need some research as it may be a bug they have introduced in 
>>> Python 3.10 for the sqlite module.
>>>
>>> Graham
>>>
 On 9 Feb 2023, at 7:02 am, Carsten Fuchs  wrote:

 Ok, I got a stack trace. My understanding of this is limited, but the 
 thread seems to be stuck at polling something without timeout? (gdb 
 reports that file ../sysdeps/unix/sysv/linux/poll.c cannot be found)

 The full session follows. (Using plaintext email today in the hope that 
 this helps with readibility. Please let me know if I should send 
 differently instead.)
 Between the two "thread apply all bt" commands I waited about two minutes:


 $ sudo gdb /usr/sbin/apache2 60501
 GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
 Copyright (C) 2022 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later 
 
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.
 Type "show copying" and "show warranty" for details.
 This GDB was configured as "x86_64-linux-gnu".
 Type "show configuration" for configuration details.
 For bug reporting instructions, please see:
 .
 Find the GDB manual and other documentation resources online at:
   .

 For help, type "help".
 Type "apropos word" to search for commands related to "word"...
 Reading symbols from /usr/sbin/apache2...
 (No debugging symbols found in /usr/sbin/apache2)
 Attaching to program: /usr/sbin/apache2, process 60501
 [New LWP 60532]
 [New LWP 60533]
 [New LWP 60534]
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 0x7f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) 
 at ../sysdeps/unix/sysv/linux/poll.c:29
 29../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht gefunden.
 (gdb) thread apply all bt

 Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
 #0  __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, 
 abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 
 <_PyRuntime+424>) at ./nptl/futex-internal.c:57
 #1  

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Graham Dumpleton 
Possibly related issues:

https://bugs.archlinux.org/task/72979 
https://bugs.python.org/issue46006

There are possibly some known issues with sub interpreters in code used by 
sqlite module there.

What exact version of Python 3.10 as you using? Can you update to latest patch 
revision of it?

Graham

> On 9 Feb 2023, at 7:22 am, Graham Dumpleton  
> wrote:
> 
> More specifically, if you look at the stack trace you see:
> 
> #6  0x7f68aefe6292 in PyEval_RestoreThread () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #7  0x7f68af02f1dc in PyGILState_Ensure () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #8  0x7f68ac30d6e3 in ?? () from 
> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
> #9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
> target:/lib/x86_64-linux-gnu/libsqlite3.so.0
> 
> Note how that function ends up calling PyGILState_Ensure(). That is a big no 
> no when working with Python sub interpreters.
> 
> In other words, it uses the simplified GIL state API for some callback into 
> Python from C code of SQLite and using that  when in a sub interpreter can 
> cause deadlocks.
> 
> https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api
> 
> So need to find the code for that function in sqlite module and work out what 
> it does.
> 
> Graham
> 
>> On 9 Feb 2023, at 7:09 am, Graham Dumpleton  
>> wrote:
>> 
>> The key bit of information in that is the call to 
>> sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python is 
>> doing something really strange that isn't working with sub interpreters. 
>> This will need some research as it may be a bug they have introduced in 
>> Python 3.10 for the sqlite module.
>> 
>> Graham
>> 
>>> On 9 Feb 2023, at 7:02 am, Carsten Fuchs  wrote:
>>> 
>>> Ok, I got a stack trace. My understanding of this is limited, but the 
>>> thread seems to be stuck at polling something without timeout? (gdb reports 
>>> that file ../sysdeps/unix/sysv/linux/poll.c cannot be found)
>>> 
>>> The full session follows. (Using plaintext email today in the hope that 
>>> this helps with readibility. Please let me know if I should send 
>>> differently instead.)
>>> Between the two "thread apply all bt" commands I waited about two minutes:
>>> 
>>> 
>>> $ sudo gdb /usr/sbin/apache2 60501
>>> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
>>> Copyright (C) 2022 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later 
>>> 
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.
>>> Type "show copying" and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> Type "show configuration" for configuration details.
>>> For bug reporting instructions, please see:
>>> .
>>> Find the GDB manual and other documentation resources online at:
>>>   .
>>> 
>>> For help, type "help".
>>> Type "apropos word" to search for commands related to "word"...
>>> Reading symbols from /usr/sbin/apache2...
>>> (No debugging symbols found in /usr/sbin/apache2)
>>> Attaching to program: /usr/sbin/apache2, process 60501
>>> [New LWP 60532]
>>> [New LWP 60533]
>>> [New LWP 60534]
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> 0x7f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) 
>>> at ../sysdeps/unix/sysv/linux/poll.c:29
>>> 29  ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht 
>>> gefunden.
>>> (gdb) thread apply all bt
>>> 
>>> Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
>>> #0  __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, 
>>> abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 
>>> <_PyRuntime+424>) at ./nptl/futex-internal.c:57
>>> #1  __futex_abstimed_wait_common (cancel=true, private=-1359973880, 
>>> abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, 
>>> futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
>>> #2  __GI___futex_abstimed_wait_cancelable64 
>>> (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, 
>>> expected=expected@entry=0, clockid=clockid@entry=1, 
>>> abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at 
>>> ./nptl/futex-internal.c:139
>>> #3  0x7f68af6a6f1b in __pthread_cond_wait_common 
>>> (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, 
>>> cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
>>> #4  ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, 
>>> mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at 
>>> ./nptl/pthread_cond_wait.c:652
>>> #5  0x7f68aefe5df5 in ?? () from 
>>>

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Graham Dumpleton 
More specifically, if you look at the stack trace you see:

#6  0x7f68aefe6292 in PyEval_RestoreThread () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7  0x7f68af02f1dc in PyGILState_Ensure () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8  0x7f68ac30d6e3 in ?? () from 
target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
target:/lib/x86_64-linux-gnu/libsqlite3.so.0

Note how that function ends up calling PyGILState_Ensure(). That is a big no no 
when working with Python sub interpreters.

In other words, it uses the simplified GIL state API for some callback into 
Python from C code of SQLite and using that  when in a sub interpreter can 
cause deadlocks.

https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api

So need to find the code for that function in sqlite module and work out what 
it does.

Graham

> On 9 Feb 2023, at 7:09 am, Graham Dumpleton  
> wrote:
> 
> The key bit of information in that is the call to 
> sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python is 
> doing something really strange that isn't working with sub interpreters. This 
> will need some research as it may be a bug they have introduced in Python 
> 3.10 for the sqlite module.
> 
> Graham
> 
>> On 9 Feb 2023, at 7:02 am, Carsten Fuchs  wrote:
>> 
>> Ok, I got a stack trace. My understanding of this is limited, but the thread 
>> seems to be stuck at polling something without timeout? (gdb reports that 
>> file ../sysdeps/unix/sysv/linux/poll.c cannot be found)
>> 
>> The full session follows. (Using plaintext email today in the hope that this 
>> helps with readibility. Please let me know if I should send differently 
>> instead.)
>> Between the two "thread apply all bt" commands I waited about two minutes:
>> 
>> 
>> $ sudo gdb /usr/sbin/apache2 60501
>> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
>> Copyright (C) 2022 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later 
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> Type "show copying" and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> .
>> Find the GDB manual and other documentation resources online at:
>>   .
>> 
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from /usr/sbin/apache2...
>> (No debugging symbols found in /usr/sbin/apache2)
>> Attaching to program: /usr/sbin/apache2, process 60501
>> [New LWP 60532]
>> [New LWP 60533]
>> [New LWP 60534]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> 0x7f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) 
>> at ../sysdeps/unix/sysv/linux/poll.c:29
>> 29   ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht 
>> gefunden.
>> (gdb) thread apply all bt
>> 
>> Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
>> #0  __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, 
>> abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 
>> <_PyRuntime+424>) at ./nptl/futex-internal.c:57
>> #1  __futex_abstimed_wait_common (cancel=true, private=-1359973880, 
>> abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, 
>> futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
>> #2  __GI___futex_abstimed_wait_cancelable64 
>> (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, 
>> expected=expected@entry=0, clockid=clockid@entry=1, 
>> abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at 
>> ./nptl/futex-internal.c:139
>> #3  0x7f68af6a6f1b in __pthread_cond_wait_common 
>> (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, 
>> cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
>> #4  ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, 
>> mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at 
>> ./nptl/pthread_cond_wait.c:652
>> #5  0x7f68aefe5df5 in ?? () from 
>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #6  0x7f68aefe6292 in PyEval_RestoreThread () from 
>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #7  0x7f68af02f1dc in PyGILState_Ensure () from 
>> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #8  0x7f68ac30d6e3 in ?? () from 
>> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
>> #9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
>>

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Graham Dumpleton 
The key bit of information in that is the call to 
sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python is 
doing something really strange that isn't working with sub interpreters. This 
will need some research as it may be a bug they have introduced in Python 3.10 
for the sqlite module.

Graham

> On 9 Feb 2023, at 7:02 am, Carsten Fuchs  wrote:
> 
> Ok, I got a stack trace. My understanding of this is limited, but the thread 
> seems to be stuck at polling something without timeout? (gdb reports that 
> file ../sysdeps/unix/sysv/linux/poll.c cannot be found)
> 
> The full session follows. (Using plaintext email today in the hope that this 
> helps with readibility. Please let me know if I should send differently 
> instead.)
> Between the two "thread apply all bt" commands I waited about two minutes:
> 
> 
> $ sudo gdb /usr/sbin/apache2 60501
> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
> Copyright (C) 2022 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/apache2...
> (No debugging symbols found in /usr/sbin/apache2)
> Attaching to program: /usr/sbin/apache2, process 60501
> [New LWP 60532]
> [New LWP 60533]
> [New LWP 60534]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x7f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at 
> ../sysdeps/unix/sysv/linux/poll.c:29
> 29../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht 
> gefunden.
> (gdb) thread apply all bt
> 
> Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
> #0  __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, 
> abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 
> <_PyRuntime+424>) at ./nptl/futex-internal.c:57
> #1  __futex_abstimed_wait_common (cancel=true, private=-1359973880, 
> abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, 
> futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
> #2  __GI___futex_abstimed_wait_cancelable64 
> (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, 
> expected=expected@entry=0, clockid=clockid@entry=1, 
> abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at 
> ./nptl/futex-internal.c:139
> #3  0x7f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, 
> clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 
> <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
> #4  ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, 
> mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at 
> ./nptl/pthread_cond_wait.c:652
> #5  0x7f68aefe5df5 in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #6  0x7f68aefe6292 in PyEval_RestoreThread () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #7  0x7f68af02f1dc in PyGILState_Ensure () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #8  0x7f68ac30d6e3 in ?? () from 
> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
> #9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
> target:/lib/x86_64-linux-gnu/libsqlite3.so.0
> #10 0x7f68a750b2d6 in ?? () from 
> target:/lib/x86_64-linux-gnu/libsqlite3.so.0
> #11 0x7f68ac3087d6 in ?? () from 
> target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
> #12 0x7f68aef118ba in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #13 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #14 0x7f68aefe781f in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #15 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #16 0x7f68aefe781f in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #17 0x7f68aee97766 in _PyEval_EvalFrameDefault () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #18 0x7f68aefe781f in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #19 0x7f68aee9c9b8 in _PyEval_EvalFrameDefault () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
> #20 0x7f68aefe781f in ?? () from 
> target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-08 Thread Carsten Fuchs 
Ok, I got a stack trace. My understanding of this is limited, but the thread 
seems to be stuck at polling something without timeout? (gdb reports that file 
../sysdeps/unix/sysv/linux/poll.c cannot be found)

The full session follows. (Using plaintext email today in the hope that this 
helps with readibility. Please let me know if I should send differently 
instead.)
Between the two "thread apply all bt" commands I waited about two minutes:


$ sudo gdb /usr/sbin/apache2 60501
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...
(No debugging symbols found in /usr/sbin/apache2)
Attaching to program: /usr/sbin/apache2, process 60501
[New LWP 60532]
[New LWP 60533]
[New LWP 60534]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x7f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at 
../sysdeps/unix/sysv/linux/poll.c:29
29  ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht 
gefunden.
(gdb) thread apply all bt

Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
#0  __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, 
abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 
<_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=-1359973880, 
abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, 
futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 
(futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, 
expected=expected@entry=0, clockid=clockid@entry=1, 
abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at 
./nptl/futex-internal.c:139
#3  0x7f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, 
clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 
<_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, 
mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at 
./nptl/pthread_cond_wait.c:652
#5  0x7f68aefe5df5 in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#6  0x7f68aefe6292 in PyEval_RestoreThread () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7  0x7f68af02f1dc in PyGILState_Ensure () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8  0x7f68ac30d6e3 in ?? () from 
target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9  0x7f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from 
target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#10 0x7f68a750b2d6 in ?? () from 
target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#11 0x7f68ac3087d6 in ?? () from 
target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#12 0x7f68aef118ba in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#13 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#14 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#15 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#16 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#17 0x7f68aee97766 in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#18 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#19 0x7f68aee9c9b8 in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#20 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#21 0x7f68aee97766 in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#22 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#23 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#24 0x7f68aefe781f in ?? () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#25 0x7f68aee9aeee in _PyEval_EvalFrameDefault () from 
target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#26 0x7f68aefe781f in ?? () from

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-07 Thread Graham Dumpleton 
Only other thing can suggest if you know point at which it is hanging, is to 
test and extract C stack trace by attaching gdb to the running process. This 
may or may not work depending on your system setup and what tools are installed.

https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb

That may give some clues.

Graham

> On 8 Feb 2023, at 8:42 am, Graham Dumpleton  
> wrote:
> 
> Is any of your code using ctypes module for Apache, or any other module which 
> is used to call into C code by writing Python code only?
> 
> Graham
> 
>> On 8 Feb 2023, at 2:55 am, Carsten Fuchs  wrote:
>> 
>> Hello,
>> 
>> hmmm. There are no .so files there:
>> 
>> $ pwd
>> /home/carsten/.virtualenvs/HallCam-web/lib/python3.10/site-packages
>> 
>> $ dir
>> insgesamt 120
>> drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:41 asgiref
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 asgiref-3.6.0.dist-info
>> drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:41 dateutil
>> drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:40 _distutils_hack
>> drwxrwxr-x 18 carsten carsten  4096 Feb  6 20:41 django
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 Django-4.1.5.dist-info
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
>> django_improved_user-2.0a2.dist-info
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
>> django_widget_tweaks-1.4.12.dist-info
>> drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 improved_user
>> drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:40 pip
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 pip-22.0.2.dist-info
>> drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:40 pkg_resources
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 __pycache__
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
>> python_dateutil-2.8.2.dist-info
>> drwxrwxr-x  7 carsten carsten  4096 Feb  6 20:40 setuptools
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 setuptools-59.6.0.dist-info
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 six-1.16.0.dist-info
>> drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:41 sqlparse
>> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 sqlparse-0.4.3.dist-info
>> drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 widget_tweaks
>> -rw-rw-r--  1 carsten carsten   152 Feb  6 20:40 distutils-precedence.pth
>> -rw-rw-r--  1 carsten carsten 34549 Feb  6 20:41 six.py
>> 
>> $ find -iname "*.so"
>> 
>> The last command returned no output, but I additionally looked into the 
>> directories also in many other ways to make sure that I didn't run a wrong 
>> search. Hmmm.
>> 
>> Best regards,
>> Carsten
>> 
>> 
>> Am 07.02.23 um 00:55 schrieb Graham Dumpleton:
>>> Go into the lib/python3.10/site-packages directory of your Python virtual 
>>> environment. What .so files exist in that directory? These are the packages 
>>> with C extensions. It is only code in C   extensions that should 
>>> normally be able to trigger a Python deadlock.
>>> 
>>> Graham
>>> 
 On 7 Feb 2023, at 7:14 am, Carsten Fuchs  
  wrote:
 
 Hello Graham,
 
 many thanks for your detailed reply!
 
 I must add that my Django project is a simple webcam application: A webcam 
 sends POST requests to upload the images, the views show them. A single 
 webcam is connected over a very slow internet connection (32 kBit/s), so a 
 single POST can take several minutes. However, that never was and still 
 doesn't seem to be a problem: Uploads still succeed, it's only the GET 
 requests that fail. The GET requests in turn do more than just sending 
 some HTML with the image URLs: especially they use the Pillow library to 
 lazily produce thumbnails, put timestamps into the images, etc.
 
 Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed 
 the related code, made a new virtuelenv without Pillow from scratch and 
 restarted Apache. Unfortunately, it didn't help – please see the error log 
 below. The timeout occurs only after the browser has received a large 
 portion, possibly all, of the view's response.
 
 `pip list` shows
 
 Package  Version
  ---
 asgiref  3.6.0
 Django   4.1.5
 django-improved-user 2.0a2
 django-widget-tweaks 1.4.12
 pip  22.0.2
 python-dateutil  2.8.2
 setuptools   59.6.0
 six  1.16.0
 sqlparse 0.4.3
 
 
 Is any of these know for causing problems like numpy?
 
 Best regards,
 Carsten
 
 The above mentioned error log:
 
 [Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid
  140714434877312] AH00489: Apache/2.4.52 (Ubuntu) 
 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
 [Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-07 Thread Graham Dumpleton 
Is any of your code using ctypes module for Apache, or any other module which 
is used to call into C code by writing Python code only?

Graham

> On 8 Feb 2023, at 2:55 am, Carsten Fuchs  wrote:
> 
> Hello,
> 
> hmmm. There are no .so files there:
> 
> $ pwd
> /home/carsten/.virtualenvs/HallCam-web/lib/python3.10/site-packages
> 
> $ dir
> insgesamt 120
> drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:41 asgiref
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 asgiref-3.6.0.dist-info
> drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:41 dateutil
> drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:40 _distutils_hack
> drwxrwxr-x 18 carsten carsten  4096 Feb  6 20:41 django
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 Django-4.1.5.dist-info
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
> django_improved_user-2.0a2.dist-info
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
> django_widget_tweaks-1.4.12.dist-info
> drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 improved_user
> drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:40 pip
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 pip-22.0.2.dist-info
> drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:40 pkg_resources
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 __pycache__
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
> python_dateutil-2.8.2.dist-info
> drwxrwxr-x  7 carsten carsten  4096 Feb  6 20:40 setuptools
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 setuptools-59.6.0.dist-info
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 six-1.16.0.dist-info
> drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:41 sqlparse
> drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 sqlparse-0.4.3.dist-info
> drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 widget_tweaks
> -rw-rw-r--  1 carsten carsten   152 Feb  6 20:40 distutils-precedence.pth
> -rw-rw-r--  1 carsten carsten 34549 Feb  6 20:41 six.py
> 
> $ find -iname "*.so"
> 
> The last command returned no output, but I additionally looked into the 
> directories also in many other ways to make sure that I didn't run a wrong 
> search. Hmmm.
> 
> Best regards,
> Carsten
> 
> 
> Am 07.02.23 um 00:55 schrieb Graham Dumpleton:
>> Go into the lib/python3.10/site-packages directory of your Python virtual 
>> environment. What .so files exist in that directory? These are the packages 
>> with C extensions. It is only code in C extensions that should normally be 
>> able to trigger a Python deadlock.
>> 
>> Graham
>> 
>>> On 7 Feb 2023, at 7:14 am, Carsten Fuchs  
>>>  wrote:
>>> 
>>> Hello Graham,
>>> 
>>> many thanks for your detailed reply!
>>> 
>>> I must add that my Django project is a simple webcam application: A webcam 
>>> sends POST requests to upload the images, the views show them. A single 
>>> webcam is connected over a very slow internet connection (32 kBit/s), so a 
>>> single POST can take several minutes. However, that never was and still 
>>> doesn't seem to be a problem: Uploads still succeed, it's only the GET 
>>> requests that fail. The GET requests in turn do more than just sending some 
>>> HTML with the image URLs: especially they use the Pillow library to lazily 
>>> produce thumbnails, put timestamps into the images, etc.
>>> 
>>> Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed 
>>> the related code, made a new virtuelenv without Pillow from scratch and 
>>> restarted Apache. Unfortunately, it didn't help – please see the error log 
>>> below. The timeout occurs only after the browser has received a large 
>>> portion, possibly all, of the view's response.
>>> 
>>> `pip list` shows
>>> 
>>> Package  Version
>>>  ---
>>> asgiref  3.6.0
>>> Django   4.1.5
>>> django-improved-user 2.0a2
>>> django-widget-tweaks 1.4.12
>>> pip  22.0.2
>>> python-dateutil  2.8.2
>>> setuptools   59.6.0
>>> six  1.16.0
>>> sqlparse 0.4.3
>>> 
>>> 
>>> Is any of these know for causing problems like numpy?
>>> 
>>> Best regards,
>>> Carsten
>>> 
>>> The above mentioned error log:
>>> 
>>> [Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid 
>>> 140714434877312] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 
>>> configured -- resuming normal operations
>>> [Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid 
>>> 140714434877312] AH00094: Command line: '/usr/sbin/apache2'
>>> [Mon Feb 06 20:52:20.380741 2023] [wsgi:info] [pid 25845:tid 
>>> 140714434877312] mod_wsgi (pid=25845): Attach interpreter ''.
>>> [Mon Feb 06 20:52:20.395024 2023] [wsgi:info] [pid 25846:tid 
>>> 140714434877312] mod_wsgi (pid=25846): Attach interpreter ''.
>>> [Mon Feb 06 20:52:20.395771 2023] [wsgi:info] [pid 25845:tid 
>>> 140714434877312] mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' 
>>> to path.
>>> [Mon Feb 06 20:52:20.411711 2023] [wsgi:info] [pid 25846:tid 
>>> 140714434877312] mod_wsgi (pid=25846):

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-07 Thread Carsten Fuchs 
Hello,

hmmm. There are no .so files there:

$ pwd
/home/carsten/.virtualenvs/HallCam-web/lib/python3.10/site-packages

$ dir
insgesamt 120
drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:41 asgiref
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 asgiref-3.6.0.dist-info
drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:41 dateutil
drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:40 _distutils_hack
drwxrwxr-x 18 carsten carsten  4096 Feb  6 20:41 django
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 Django-4.1.5.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
django_improved_user-2.0a2.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 
django_widget_tweaks-1.4.12.dist-info
drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 improved_user
drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:40 pip
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 pip-22.0.2.dist-info
drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:40 pkg_resources
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 __pycache__
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 python_dateutil-2.8.2.dist-info
drwxrwxr-x  7 carsten carsten  4096 Feb  6 20:40 setuptools
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 setuptools-59.6.0.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 six-1.16.0.dist-info
drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:41 sqlparse
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 sqlparse-0.4.3.dist-info
drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 widget_tweaks
-rw-rw-r--  1 carsten carsten   152 Feb  6 20:40 distutils-precedence.pth
-rw-rw-r--  1 carsten carsten 34549 Feb  6 20:41 six.py

$ find -iname "*.so"

The last command returned no output, but I additionally looked into the 
directories also in many other ways to make sure that I didn't run a wrong 
search. Hmmm.

Best regards,
Carsten

Am 07.02.23 um 00:55 schrieb Graham Dumpleton:
> Go into the lib/python3.10/site-packages directory of your Python virtual 
> environment. What .so files exist in that directory? These are the packages 
> with C extensions. It is only code in C extensions that should normally be 
> able to trigger a Python deadlock.
>
> Graham
>
>> On 7 Feb 2023, at 7:14 am, Carsten Fuchs  wrote:
>>
>> Hello Graham,
>>
>> many thanks for your detailed reply!
>>
>> I must add that my Django project is a simple webcam application: A webcam 
>> sends POST requests to upload the images, the views show them. A single 
>> webcam is connected over a very slow internet connection (32 kBit/s), so a 
>> single POST can take several minutes. However, that never was and still 
>> doesn't seem to be a problem: Uploads still succeed, it's only the GET 
>> requests that fail. The GET requests in turn do more than just sending some 
>> HTML with the image URLs: especially they use the Pillow library to lazily 
>> produce thumbnails, put timestamps into the images, etc.
>>
>> Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed 
>> the related code, made a new virtuelenv without Pillow from scratch and 
>> restarted Apache. Unfortunately, it didn't help – please see the error log 
>> below. The timeout occurs only after the browser has received a large 
>> portion, possibly all, of the view's response.
>>
>> `pip list` shows
>>
>> Package  Version
>>  ---
>> asgiref  3.6.0
>> Django   4.1.5
>> django-improved-user 2.0a2
>> django-widget-tweaks 1.4.12
>> pip  22.0.2
>> python-dateutil  2.8.2
>> setuptools   59.6.0
>> six  1.16.0
>> sqlparse 0.4.3
>>
>> Is any of these know for causing problems like numpy?
>>
>> Best regards,
>> Carsten
>>
>> The above mentioned error log:
>>
>> [Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid 
>> 140714434877312] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 
>> configured -- resuming normal operations
>> [Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid 
>> 140714434877312] AH00094: Command line: '/usr/sbin/apache2'
>> [Mon Feb 06 20:52:20.380741 2023] [wsgi:info] [pid 25845:tid 
>> 140714434877312] mod_wsgi (pid=25845): Attach interpreter ''.
>> [Mon Feb 06 20:52:20.395024 2023] [wsgi:info] [pid 25846:tid 
>> 140714434877312] mod_wsgi (pid=25846): Attach interpreter ''.
>> [Mon Feb 06 20:52:20.395771 2023] [wsgi:info] [pid 25845:tid 
>> 140714434877312] mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to 
>> path.
>> [Mon Feb 06 20:52:20.411711 2023] [wsgi:info] [pid 25846:tid 
>> 140714434877312] mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to 
>> path.
>> [Mon Feb 06 20:52:48.621284 2023] [wsgi:info] [pid 25845:tid 
>> 140714369189440] mod_wsgi (pid=25845): Create interpreter 
>> '192.168.1.222:32228|'.
>> [Mon Feb 06 20:52:48.634464 2023] [wsgi:info] [pid 25845:tid 
>> 140714369189440] mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to 
>> path.
>> [Mon Feb 06 20:52:48.635248 2023] [wsgi:info] [pid

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-06 Thread Graham Dumpleton 
Go into the lib/python3.10/site-packages directory of your Python virtual 
environment. What .so files exist in that directory? These are the packages 
with C extensions. It is only code in C extensions that should normally be able 
to trigger a Python deadlock.

Graham

> On 7 Feb 2023, at 7:14 am, Carsten Fuchs  wrote:
> 
> Hello Graham,
> 
> many thanks for your detailed reply!
> 
> I must add that my Django project is a simple webcam application: A webcam 
> sends POST requests to upload the images, the views show them. A single 
> webcam is connected over a very slow internet connection (32 kBit/s), so a 
> single POST can take several minutes. However, that never was and still 
> doesn't seem to be a problem: Uploads still succeed, it's only the GET 
> requests that fail. The GET requests in turn do more than just sending some 
> HTML with the image URLs: especially they use the Pillow library to lazily 
> produce thumbnails, put timestamps into the images, etc.
> 
> Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed 
> the related code, made a new virtuelenv without Pillow from scratch and 
> restarted Apache. Unfortunately, it didn't help – please see the error log 
> below. The timeout occurs only after the browser has received a large 
> portion, possibly all, of the view's response.
> 
> `pip list` shows
> 
> Package  Version
>  ---
> asgiref  3.6.0
> Django   4.1.5
> django-improved-user 2.0a2
> django-widget-tweaks 1.4.12
> pip  22.0.2
> python-dateutil  2.8.2
> setuptools   59.6.0
> six  1.16.0
> sqlparse 0.4.3
> 
> 
> Is any of these know for causing problems like numpy?
> 
> Best regards,
> Carsten
> 
> The above mentioned error log:
> 
> [Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid 
> 140714434877312] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 
> configured -- resuming normal operations
> [Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid 
> 140714434877312] AH00094: Command line: '/usr/sbin/apache2'
> [Mon Feb 06 20:52:20.380741 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
> mod_wsgi (pid=25845): Attach interpreter ''.
> [Mon Feb 06 20:52:20.395024 2023] [wsgi:info] [pid 25846:tid 140714434877312] 
> mod_wsgi (pid=25846): Attach interpreter ''.
> [Mon Feb 06 20:52:20.395771 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
> mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
> [Mon Feb 06 20:52:20.411711 2023] [wsgi:info] [pid 25846:tid 140714434877312] 
> mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
> [Mon Feb 06 20:52:48.621284 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
> mod_wsgi (pid=25845): Create interpreter '192.168.1.222:32228|'.
> [Mon Feb 06 20:52:48.634464 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
> mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
> [Mon Feb 06 20:52:48.635248 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
> [remote 88.75.25.178:42162] mod_wsgi (pid=25845, process='cf-hallcam-site', 
> application='192.168.1.222:32228|'): Loading Python script file 
> '/home/carsten/HallCam/web/HallCam/wsgi.py'.
> [Mon Feb 06 20:57:40.661043 2023] [wsgi:info] [pid 25845:tid 140714419545664] 
> mod_wsgi (pid=25845): Daemon process request time limit exceeded, stopping 
> process 'cf-hallcam-site'.
> [Mon Feb 06 20:57:40.661184 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
> mod_wsgi (pid=25845): Shutdown requested 'cf-hallcam-site'.
> [Mon Feb 06 20:57:45.661421 2023] [wsgi:info] [pid 25845:tid 140714177934912] 
> mod_wsgi (pid=25845): Aborting process 'cf-hallcam-site'.
> [Mon Feb 06 20:57:45.661464 2023] [wsgi:info] [pid 25845:tid 140714177934912] 
> mod_wsgi (pid=25845): Exiting process 'cf-hallcam-site'.
> [Mon Feb 06 20:57:45.666987 2023] [wsgi:error] [pid 25847:tid 
> 140714226689600] [client 37.81.109.237:49940] Truncated or oversized response 
> headers received from daemon process 'cf-hallcam-site': 
> /home/carsten/HallCam/web/HallCam/wsgi.py
> [Mon Feb 06 20:57:45.667005 2023] [wsgi:error] [pid 25848:tid 
> 140714335794752] (104)Connection reset by peer: [client 88.75.25.178:42162] 
> mod_wsgi (pid=25848): Failed to proxy response from daemon., referer: 
> http://vdzuggmrroo5k7e9.myfritz.net:32228/
> [Mon Feb 06 20:57:46.621804 2023] [wsgi:info] [pid 25977:tid 140714434877312] 
> mod_wsgi (pid=25977): Attach interpreter ''.
> [Mon Feb 06 20:57:46.633564 2023] [wsgi:info] [pid 25977:tid 140714434877312] 
> mod_wsgi (pid=25977): Adding '/home/carsten/HallCam/web' to path.
> [Mon Feb 06 21:00:01.811479 2023] [wsgi:info] [pid 25846:tid 140714360796736] 
> mod_wsgi (pid=25846): Create interpreter '192.168.1.222:32228|'.
> [Mon Feb 06 21:00:01.824795 2023] [wsgi:info] [pid 25846:tid 140714360796736] 
> mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
> [Mon Feb 06 21:00:01.825535

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-06 Thread Carsten Fuchs 
Hello Graham,

many thanks for your detailed reply!

I must add that my Django project is a simple webcam application: A webcam 
sends POST requests to upload the images, the views show them. A single webcam 
is connected over a very slow internet connection (32 kBit/s), so a single POST 
can take several minutes. However, that never was and still doesn't seem to be 
a problem: Uploads still succeed, it's only the GET requests that fail. The GET 
requests in turn do more than just sending some HTML with the image URLs: 
especially they use the Pillow library to lazily produce thumbnails, put 
timestamps into the images, etc.

Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed the 
related code, made a new virtuelenv without Pillow from scratch and restarted 
Apache. Unfortunately, it didn't help – please see the error log below. The 
timeout occurs only after the browser has received a large portion, possibly 
all, of the view's response.

`pip list` shows

Package  Version
 ---
asgiref  3.6.0
Django   4.1.5
django-improved-user 2.0a2
django-widget-tweaks 1.4.12
pip  22.0.2
python-dateutil  2.8.2
setuptools   59.6.0
six  1.16.0
sqlparse 0.4.3

Is any of these know for causing problems like numpy?

Best regards,
Carsten

The above mentioned error log:

[Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid 
140714434877312] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 
configured -- resuming normal operations
[Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid 140714434877312] 
AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 06 20:52:20.380741 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
mod_wsgi (pid=25845): Attach interpreter ''.
[Mon Feb 06 20:52:20.395024 2023] [wsgi:info] [pid 25846:tid 140714434877312] 
mod_wsgi (pid=25846): Attach interpreter ''.
[Mon Feb 06 20:52:20.395771 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:20.411711 2023] [wsgi:info] [pid 25846:tid 140714434877312] 
mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:48.621284 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
mod_wsgi (pid=25845): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 20:52:48.634464 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:48.635248 2023] [wsgi:info] [pid 25845:tid 140714369189440] 
[remote 88.75.25.178:42162] mod_wsgi (pid=25845, process='cf-hallcam-site', 
application='192.168.1.222:32228|'): Loading Python script file 
'/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Mon Feb 06 20:57:40.661043 2023] [wsgi:info] [pid 25845:tid 140714419545664] 
mod_wsgi (pid=25845): Daemon process request time limit exceeded, stopping 
process 'cf-hallcam-site'.
[Mon Feb 06 20:57:40.661184 2023] [wsgi:info] [pid 25845:tid 140714434877312] 
mod_wsgi (pid=25845): Shutdown requested 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.661421 2023] [wsgi:info] [pid 25845:tid 140714177934912] 
mod_wsgi (pid=25845): Aborting process 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.661464 2023] [wsgi:info] [pid 25845:tid 140714177934912] 
mod_wsgi (pid=25845): Exiting process 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.666987 2023] [wsgi:error] [pid 25847:tid 140714226689600] 
[client 37.81.109.237:49940] Truncated or oversized response headers received 
from daemon process 'cf-hallcam-site': /home/carsten/HallCam/web/HallCam/wsgi.py
[Mon Feb 06 20:57:45.667005 2023] [wsgi:error] [pid 25848:tid 140714335794752] 
(104)Connection reset by peer: [client 88.75.25.178:42162] mod_wsgi 
(pid=25848): Failed to proxy response from daemon., referer: 
http://vdzuggmrroo5k7e9.myfritz.net:32228/
[Mon Feb 06 20:57:46.621804 2023] [wsgi:info] [pid 25977:tid 140714434877312] 
mod_wsgi (pid=25977): Attach interpreter ''.
[Mon Feb 06 20:57:46.633564 2023] [wsgi:info] [pid 25977:tid 140714434877312] 
mod_wsgi (pid=25977): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:00:01.811479 2023] [wsgi:info] [pid 25846:tid 140714360796736] 
mod_wsgi (pid=25846): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 21:00:01.824795 2023] [wsgi:info] [pid 25846:tid 140714360796736] 
mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:00:01.825535 2023] [wsgi:info] [pid 25846:tid 140714360796736] 
[remote 37.81.109.237:49942] mod_wsgi (pid=25846, process='cf-hallcam-site', 
application='192.168.1.222:32228|'): Loading Python script file 
'/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Mon Feb 06 21:00:02.506063 2023] [wsgi:info] [pid 25977:tid 140714402760256] 
mod_wsgi (pid=25977): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 21:00:02.519453 2023] [wsgi:info] [pid 25977:tid 140714402760256] 
mod_wsgi (pid=25977): Adding

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-05 Thread Graham Dumpleton 
Two points here to clarify.

In your case the final error is:

  Daemon process deadlock timer expired

This means that the full Python interpreter locked up. In this case the request 
timeout may not apply and the feature where by stack traces can be dumped might 
not happen. Depends on what lead up to the issue.

By default the deadlock timeout is 300 seconds.

deadlock-timeout=sss
Defines the maximum number of seconds allowed to pass before the daemon process 
is shutdown and restarted after a potential deadlock on the Python GIL has been 
detected. The default is 300 seconds.

This option exists to combat the problem of a daemon process freezing as the 
result of a rogue Python C extension module which doesn’t properly release the 
Python GIL when entering into a blocking or long running operation.

This can occur when you are using third party Python packages which aren't 
designed to work in Python sub interpreters. More details in:

https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api
The solution as you found is to set:

  WSGIApplicationGroup %{GLOBAL}

So likely when you updated Python versions, some third party package you are 
using has shifted to a newer version which breaks in this respect, or which 
runs code different when using newer Python version.

As to request timeout if it were coming into play, the timeout is 30 seconds 
when you have daemon processes which are single threaded.

For multithreaded daemon process the timeout can trigger after 30 seconds as 
uses an average across all active request handler threads.

request-timeout=sss
Defines the maximum number of seconds that a request is allowed to run before 
the daemon process is restarted. This can be used to recover from a scenario 
where a request blocks indefinitely, and where if all request threads were 
consumed in this way, would result in the whole WSGI application process being 
blocked.

How this option is seen to behave is different depending on whether a daemon 
process uses only one thread, or more than one thread for handling requests, as 
set by the threads option.

If there is only a single thread, and so the process can only handle one 
request at a time, as soon as the timeout has passed, a restart of the process 
will be initiated.

If there is more than one thread, the request timeout is applied to the average 
running time for any requests, across all threads. This means that a request 
can run longer than the request timeout. This is done to reduce the possibility 
of interupting other running requests, and causing a user to see a failure. So 
where there is still capacity to handle more requests, restarting of the 
process will be delayed if possible.


In your case though because hitting a full lock up of the Python interpreter, 
even though the request timeout may still have triggered, it couldn't dump 
stack traces as doing so requires getting the global Python interpreter lock, 
which wasn't being released and thus threads to dump stack traces got stuck.

To understand the cause further and identify which third party package may be 
the problem, I would need to see what extra Python packages you are installing. 
Note that numpy can cause this, so if using anything which relies on that, that 
could be the reason.

Graham

> On 6 Feb 2023, at 3:35 am, Carsten Fuchs  wrote:
> 
> Adding to my recent post:
> 
> It seems that the `request-timeout=30` is effective, after all: The error 
> messages have changed, but there is no stack trace.
> For completeness, here is the site config and error log excerpt:
> 
> Site config:
> 
> 
> ServerAdmin webmaster@localhost
> DocumentRoot /var/www/html
> 
> # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
> # error, crit, alert, emerg.
> # It is also possible to configure the loglevel for particular
> # modules, e.g.
> #LogLevel info ssl:warn
> LogLevel info
> 
> ErrorLog ${APACHE_LOG_DIR}/error.log
> CustomLog ${APACHE_LOG_DIR}/access.log combined
> 
> # Siehe https://bz.apache.org/bugzilla/show_bug.cgi?id=45023
> # Hierfür muss mod_headers aktiv sein: sudo a2enmod headers
> RequestHeader edit "If-None-Match" '^"((.*)-gzip)"$' '"$1", "$2"'
> 
> Alias /static/ /var/www/HallCam-static/
> 
> Require all granted
> 
> 
> Alias /media/ /var/www/HallCam-media/
> 
> Require all granted
> 
> 
> WSGIDaemonProcess cf-hallcam-site request-timeout=30 user=carsten 
> group=carsten processes=2 display-name=%{GROUP} 
> python-home=/home/carsten/.virtualenvs/HallCam-web 
> python-path=/home/carsten/HallCam/web
> WSGIProcessGroup cf-hallcam-site
> # WSGIApplicationGroup %{GLOBAL}
> 
> WSGIScriptAlias / /home/carsten/HallCam/web/HallCam/wsgi.py
> 
> 
> Require all granted
> 
> 
> 
> 
> 
> From `/var/log/apache2/error.log`:
> 
> [Sun Feb 05 17:18:55.470532 2023]

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-02-05 Thread Carsten Fuchs 
Adding to my recent post:

It seems that the `request-timeout=30` is effective, after all: The error 
messages have changed, but there is no stack trace.
For completeness, here is the site config and error log excerpt:

Site config:


ServerAdmin webmaster@localhost
DocumentRoot /var/www/html

# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
LogLevel info

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

# Siehe https://bz.apache.org/bugzilla/show_bug.cgi?id=45023
# Hierfür muss mod_headers aktiv sein: sudo a2enmod headers
RequestHeader edit "If-None-Match" '^"((.*)-gzip)"$' '"$1", "$2"'

Alias /static/ /var/www/HallCam-static/

Require all granted


Alias /media/ /var/www/HallCam-media/

Require all granted


WSGIDaemonProcess cf-hallcam-site request-timeout=30 user=carsten 
group=carsten processes=2 display-name=%{GROUP} 
python-home=/home/carsten/.virtualenvs/HallCam-web 
python-path=/home/carsten/HallCam/web
WSGIProcessGroup cf-hallcam-site
# WSGIApplicationGroup %{GLOBAL}

WSGIScriptAlias / /home/carsten/HallCam/web/HallCam/wsgi.py


Require all granted





>From `/var/log/apache2/error.log`:

[Sun Feb 05 17:18:55.470532 2023] [mpm_event:notice] [pid 635:tid 
140277991982976] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 
configured -- resuming normal operations
[Sun Feb 05 17:18:55.690430 2023] [core:notice] [pid 635:tid 
140277991982976] AH00094: Command line: '/usr/sbin/apache2'
[Sun Feb 05 17:18:56.338443 2023] [wsgi:info] [pid 638:tid 140277991982976] 
mod_wsgi (pid=638): Attach interpreter ''.
[Sun Feb 05 17:18:56.342866 2023] [wsgi:info] [pid 636:tid 140277991982976] 
mod_wsgi (pid=636): Attach interpreter ''.
[Sun Feb 05 17:18:56.366079 2023] [wsgi:info] [pid 636:tid 140277991982976] 
mod_wsgi (pid=636): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:18:56.371777 2023] [wsgi:info] [pid 638:tid 140277991982976] 
mod_wsgi (pid=638): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:24.823860 2023] [wsgi:info] [pid 636:tid 140277959865920] 
mod_wsgi (pid=636): Create interpreter '192.168.1.222:32228|'.
[Sun Feb 05 17:19:24.837012 2023] [wsgi:info] [pid 636:tid 140277959865920] 
mod_wsgi (pid=636): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:24.837752 2023] [wsgi:info] [pid 636:tid 140277959865920] 
[remote 88.75.25.178:50558] mod_wsgi (pid=636, process='cf-hallcam-site', 
application='192.168.1.222:32228|'): Loading Python script file 
'/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Sun Feb 05 17:19:43.356895 2023] [wsgi:info] [pid 638:tid 140277867546176] 
mod_wsgi (pid=638): Create interpreter '192.168.1.222:32228|'.
[Sun Feb 05 17:19:43.369787 2023] [wsgi:info] [pid 638:tid 140277867546176] 
mod_wsgi (pid=638): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:43.370532 2023] [wsgi:info] [pid 638:tid 140277867546176] 
[remote 88.75.25.178:48210] mod_wsgi (pid=638, process='cf-hallcam-site', 
application='192.168.1.222:32228|'): Loading Python script file 
'/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Sun Feb 05 17:23:21.631286 2023] [wsgi:info] [pid 636:tid 140277976651328] 
mod_wsgi (pid=636): Daemon process request time limit exceeded, stopping 
process 'cf-hallcam-site'.
[Sun Feb 05 17:23:21.631405 2023] [wsgi:info] [pid 636:tid 140277991982976] 
mod_wsgi (pid=636): Shutdown requested 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.631650 2023] [wsgi:info] [pid 636:tid 140277591307840] 
mod_wsgi (pid=636): Aborting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.631699 2023] [wsgi:info] [pid 636:tid 140277591307840] 
mod_wsgi (pid=636): Exiting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.714616 2023] [wsgi:error] [pid 639:tid 
140277725460032] [client 88.75.25.178:48224] Truncated or oversized 
response headers received from daemon process 'cf-hallcam-site': 
/home/carsten/HallCam/web/HallCam/wsgi.py, referer: 
http://vdzuggmrroo5k7e9.myfritz.net:32228/upload/
[Sun Feb 05 17:23:26.714952 2023] [wsgi:error] [pid 639:tid 
140277959865920] (104)Connection reset by peer: [client 88.75.25.178:50558] 
mod_wsgi (pid=639): Failed to proxy response from daemon., referer: 
http://vdzuggmrroo5k7e9.myfritz.net:32228/upload/
[Sun Feb 05 17:23:26.983714 2023] [wsgi:info] [pid 885:tid 140277991982976] 
mod_wsgi (pid=885): Attach interpreter ''.
[Sun Feb 05 17:23:26.994980 2023] [wsgi:info] [pid 885:tid 140277991982976] 
mod_wsgi (pid=885): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:23:37.641476 2023] [wsgi:info] [pid 638:tid 140277976651328] 
mod_wsgi (pid=638): Daemon process request time limit exceeded, stopping 
process 'cf-hallcam-site'.
[Sun Feb 05 17:23:37.641609 2023] [wsgi:info] [pid 638:tid

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-01-24 Thread A 
Dear Graham,

It was a matter of adding that line and everything fell into place!


Thanks a lot,
A

On Monday, January 23, 2023 at 8:21:44 PM UTC+1 Graham Dumpleton wrote:

> Your configuration means only a single request can be handled at a time by 
> the daemon process, so if you have a very long running request, any other 
> requests would block waiting.
>
> Even if not a long running request, a problem may be that you are using a 
> Python package that isn't designed to work in a Python sub interpreter. 
> This could cause that package to hang and so everything blocks up (same as 
> long running request at that point) for that request.
>
> Application Issues — mod_wsgi 4.9.4 documentation 
> 
> modwsgi.readthedocs.io 
> 
> [image: favicon.ico] 
> 
>  
> 
>
> Try the solution described in the docs of forcing the use of the main 
> Python interpreter by adding:
>
> WSGIApplicationGroup %{GLOBAL}
>
> If that doesn't help you will need to work out where your code is 
> blocking. For that try adding:
>
> request-timeout=30
>
> option to WSGIDaemonProcess directive.
>
> This will cause daemon process in your case to restart after 30 seconds 
> when it blocks and in doing that will attempt to log to error log file the 
> stack traces of where your code was blocked.
>
> Graham
>
> On 24 Jan 2023, at 3:27 am, A  wrote:
>
> It keeps timing out and I've been trying to solve it to no avail.
> Here my modwsgi.conf
>
> 
>   ServerName localhost
>   ServerAlias --.com *.--.com
>   
>   Define project_name --
>   Define user -
>   
>   Define project_path /srv/http/fosware
>   Define wsgi_path/srv/http/fosware/fosware
>   Define environment_path /srv/http/fosware/venv
>   
>   WSGIDaemonProcess ${user}-${project_name} user=${user} group=${user} 
> processes=1 threads=1 python-eggs=/tmp/python-eggs/ 
> python-path=${project_path}:${environment_path}/lib/python2.7/site-packages
>   WSGIProcessGroup ${user}-${project_name}
>
>   WSGIScriptAlias / ${wsgi_path}/wsgi.py
>
>   
> 
> Order allow,deny
> Allow from all
> 
> = 2.3>
> Require all granted
> 
>   
>   
>
>   Alias /static ${project_path}/static
>   
>  Require all granted
>  SetHandler None
>  FileETag none
>  Options FollowSymLinks
>   
>
>   Alias /media ${project_path}/media
>   
>  Require all granted
>  SetHandler None
>  FileETag none
>  Options FollowSymLinks
>  ErrorDocument 404 /error404
>   
>
>   ErrorLog /var/log/httpd/${user}-${project_name}-error.log
>   LogLevel info
>   CustomLog /var/log/httpd/${user}-${project_name}-access.log combined
>  
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to modwsgi+u...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/modwsgi/02fb2171-ca59-4053-9be3-8ff75e9cf9edn%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/modwsgi/782fe863-b465-4cca-84fc-f0752328fea4n%40googlegroups.com.

Re: [modwsgi] Timeout when reading response header from daemon process.

2023-01-23 Thread Graham Dumpleton 
Your configuration means only a single request can be handled at a time by the 
daemon process, so if you have a very long running request, any other requests 
would block waiting.

Even if not a long running request, a problem may be that you are using a 
Python package that isn't designed to work in a Python sub interpreter. This 
could cause that package to hang and so everything blocks up (same as long 
running request at that point) for that request.

https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api
Try the solution described in the docs of forcing the use of the main Python 
interpreter by adding:

WSGIApplicationGroup %{GLOBAL}

If that doesn't help you will need to work out where your code is blocking. For 
that try adding:

request-timeout=30

option to WSGIDaemonProcess directive.

This will cause daemon process in your case to restart after 30 seconds when it 
blocks and in doing that will attempt to log to error log file the stack traces 
of where your code was blocked.

Graham

> On 24 Jan 2023, at 3:27 am, A  wrote:
> 
> It keeps timing out and I've been trying to solve it to no avail.
> Here my modwsgi.conf
> 
> 
>   ServerName localhost
>   ServerAlias --.com *.--.com
>   
>   Define project_name --
>   Define user -
>   
>   Define project_path /srv/http/fosware
>   Define wsgi_path/srv/http/fosware/fosware
>   Define environment_path /srv/http/fosware/venv
>   
>   WSGIDaemonProcess ${user}-${project_name} user=${user} group=${user} 
> processes=1 threads=1 python-eggs=/tmp/python-eggs/ 
> python-path=${project_path}:${environment_path}/lib/python2.7/site-packages
>   WSGIProcessGroup ${user}-${project_name}
> 
>   WSGIScriptAlias / ${wsgi_path}/wsgi.py
> 
>   
> 
> Order allow,deny
> Allow from all
> 
> = 2.3>
> Require all granted
> 
>   
>   
> 
>   Alias /static ${project_path}/static
>   
>  Require all granted
>  SetHandler None
>  FileETag none
>  Options FollowSymLinks
>   
> 
>   Alias /media ${project_path}/media
>   
>  Require all granted
>  SetHandler None
>  FileETag none
>  Options FollowSymLinks
>  ErrorDocument 404 /error404
>   
> 
>   ErrorLog /var/log/httpd/${user}-${project_name}-error.log
>   LogLevel info
>   CustomLog /var/log/httpd/${user}-${project_name}-access.log combined
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to modwsgi+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/modwsgi/02fb2171-ca59-4053-9be3-8ff75e9cf9edn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/modwsgi/23758818-6AB0-4FBB-903E-AF251D51655D%40gmail.com.

[modwsgi] Timeout when reading response header from daemon process.

2023-01-23 Thread A 
It keeps timing out and I've been trying to solve it to no avail.
Here my modwsgi.conf


  ServerName localhost
  ServerAlias --.com *.--.com
  
  Define project_name --
  Define user -
  
  Define project_path /srv/http/fosware
  Define wsgi_path/srv/http/fosware/fosware
  Define environment_path /srv/http/fosware/venv
  
  WSGIDaemonProcess ${user}-${project_name} user=${user} group=${user} 
processes=1 threads=1 python-eggs=/tmp/python-eggs/ 
python-path=${project_path}:${environment_path}/lib/python2.7/site-packages
  WSGIProcessGroup ${user}-${project_name}

  WSGIScriptAlias / ${wsgi_path}/wsgi.py

  

Order allow,deny
Allow from all

= 2.3>
Require all granted

  
  

  Alias /static ${project_path}/static
  
 Require all granted
 SetHandler None
 FileETag none
 Options FollowSymLinks
  

  Alias /media ${project_path}/media
  
 Require all granted
 SetHandler None
 FileETag none
 Options FollowSymLinks
 ErrorDocument 404 /error404
  

  ErrorLog /var/log/httpd/${user}-${project_name}-error.log
  LogLevel info
  CustomLog /var/log/httpd/${user}-${project_name}-access.log combined


-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/modwsgi/02fb2171-ca59-4053-9be3-8ff75e9cf9edn%40googlegroups.com.