Hi Graham,
Here are the version of softwares being used.
*Python -> 3.6*
*mod_wsgi -> 4.6.4*
Sujai@machine0009:/usr/lib64/httpd/modules> ldd mod_wsgi.so
linux-vdso.so.1 => (0x00007ffc74cd7000)
libpython3.6m.so.1.0 =>
/data/ev/Python-3.6.3s/lib/libpython3.6m.so.1.0 (0x00007fbd76c66000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbd76a4a000)
libc.so.6 => /lib64/libc.so.6 (0x00007fbd7667d000)
libssl.so.1.0.0 => /data/dev/openssl-1.0.2sh/lib/libssl.so.1.0.0
(0x00007fbd7640b000)
libcrypto.so.1.0.0 =>
/data/dev/openssl-1.0.2sh/lib/libcrypto.so.1.0.0 (0x00007fbd75fbd000)
libz.so.1 => /data/dev/Python-3.6.3s/lib/libz.so.1
(0x00007fbd75da2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fbd75b9e000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fbd7599b000)
libm.so.6 => /lib64/libm.so.6 (0x00007fbd75699000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbd773fa000)
Sujai@machine0009 :/usr/lib64/httpd/modules> strings mod_wsgi.so | grep -w
-A 1 "wsgi_init"
wsgi_init
4.6.4
*Stack Trace of the Error*
2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent
call last):
File
"/data/dev/deployment/production/release/server-python/sf_django/sf_app/views/utils/__init__.py",
line 135, in function_wrapper
return_value = func(caller_object, request, *args, **kwargs)
File
"/data/dev/deployment/production/release/server-python/sf_django/starfish_app/views/api/api_id.py",
line 27, in get
response = get_sub_discipline_wise_api_ids(user_itss_id,
parameters["bs_revision_id"])
File
"/data/dev/deployment/production/release/server-python/sf_django/starfish_app/core/api/api_id.py",
line 21, in get_sub_discipline_wise_api_ids
log_user_access(user_itss_id, func_param={"bs_revision_id":
bs_revision_id})
File
"/data/dev/deployment/production/release/server-python/sf_django/starfish_app/core/api/api_util.py",
line 15, in log_user_access
caller_func_name = inspect.stack()[1][3] # retrieves the name of the
function that calls this function
File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in
stack
return getouterframes(sys._getframe(1), context)
File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in
getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in
getframeinfo
lines, lnum = findsource(frame)
File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 825, in
findsource
if pat.match(lines[lnum]): break
IndexError: list index out of range
In what case the interpreter will have half destroyed code. Won't the
'apapche reload' perform the process shutdown gracefully (maening, all
existing connections to a thread would be served before the thread redeems
itself)?
Thanks & regards,
Sujaikumar
On Thu, May 16, 2019 at 8:15 AM Graham Dumpleton <[email protected]>
wrote:
> As already mentioned, if it is only on process shutdown (caused by a
> reload), then it is possibly because it is trying to execute code when the
> interpreter has been half destroyed. You cut off the file system paths and
> so I can't see what:
>
> File ".....core/api/api_util.py", line 15, in log_user_access
>
> is likely a part of (if open source project) and infer anything about
> whether a separate thread is running. So a more complete stack trace may
> help.
>
> Also, given you are using an ancient Python version, what mod_wsgi version
> are you using?
>
> On 16 May 2019, at 12:41 pm, Sujai Kumar <[email protected]> wrote:
>
> Hi Graham,
>
> Yes, mod_wsgi is pulling the right python. I have verified them before
> writing to you. As mentioned earlier, the system is working fine for more
> than a year now. This issue happened suddenly and there were no code change
> recently and hence the concern. The only thing, that I see as external
> factor is 'apache reload' (as part of logrotate simultaneously). Are there
> any other things that we need to check? I shall send the complete stack
> trace in some time.
>
> Thanks & regards,
> Sujaikumar
>
> On Thu, 16 May 2019, 8:03 am Graham Dumpleton, <[email protected]>
> wrote:
>
>> Are you sure you mod_wsgi is pulling in the correct Python shared
>> library, given that it looks like you are using a custom Python
>> installation.
>>
>> Is Python 3.6 installed as system Python?
>>
>> Are you wanting to actually use a separate Python installation you
>> installed from source code?
>>
>> Did you compile mod_wsgi from source code against any separate Python
>> installation? If you did that, was LD_RUN_PATH environment variable set to
>> the lib directory for that Python installation where the Python shared
>> library is, when mod_wsgi was being built, to ensure that at run time it
>> would use that library and not the system Python one.
>>
>> If you are getting the wrong Python shared library pulled in, and there
>> are patch level differences, you could see all sorts of strange issues.
>>
>> On 16 May 2019, at 11:36 am, Sujai Kumar <[email protected]> wrote:
>>
>> Thanks for looking into this.
>>
>> Well, the stack trace doesn't have much information. The issue happens
>> when caller function name was about to be logged (using the inspect
>> module). This was a working application for over an year and suddenly the
>> situation has occurred.
>>
>> My inference is that the python state has gone bad due to the apache
>> reload that happens simultaneously due to logrotate configs. My
>> understanding is that, on apache reload, the python processes goes
>> gracefully down rereads the apache configuration(if any) and comes up. Why
>> would existing python library (inspect module) give the 'list index out of
>> range' error? Is there any other way that this situation could happen?
>>
>> Again the apache restart (systemctl restart httpd) has to be done
>> manually to bring the python process back to normal state.
>>
>> Thanks & regards,
>> Sujaikumar
>>
>> On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <
>> [email protected]> wrote:
>>
>>> Hard to tell just from the information you have given because you have
>>> truncated the file system paths so can't tell what package this code is
>>> executed from. The stack trace doesn't even look complete.
>>>
>>> At a guess you are hitting an issue where there if there is a background
>>> thread running when the process is shutdown, that thread can start up when
>>> the interpreter is in a half destroyed state. This can result in strange
>>> things happening due to the interpreter having started to destroy objects
>>> and clear our modules.
>>>
>>> Is this only happening when the processes are being shutdown?
>>>
>>> Graham
>>>
>>> On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected]> wrote:
>>>
>>> Hello All,
>>>
>>> We have a Apache/Python/Django using modwsgi. We hit upon a situation
>>> where the following error is happening in the site,
>>>
>>> *Error Log*
>>>
>>> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most
>>> recent call last):
>>> File ".....core/api/api_util.py", line 15, in log_user_access
>>> caller_func_name = inspect.stack()[1][3] # retrieves the name of
>>> the function that calls this function
>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack
>>> return getouterframes(sys._getframe(1), context)
>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in
>>> getouterframes
>>> frameinfo = (frame,) + getframeinfo(frame, context)
>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in
>>> getframeinfo
>>> lines, lnum = findsource(frame)
>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in
>>> findsource
>>> if pat.match(lines[lnum]): break
>>> IndexError: list index out of range
>>>
>>> *My understanding*
>>> I think that the issue is happening because of apache getting reloaded
>>> due to below configurations. At the same time the logrotate triggers
>>> 'apache reload' around the same time.
>>>
>>> *Quick Fix*
>>> The quick fix that we had to do was to restart the apache. Just after
>>> the restart the application started working.
>>>
>>> *Question:*
>>> What has caused this situation and how to avoid this occurring in future
>>> so that we are not reloading it manually? Thanks,
>>>
>>> *Logrotate Configs*
>>>
>>>
>>> [Sujai@machine0009 logrotate.d]$ cat apache
>>>
>>> # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE
>>>
>>> # OVERWRITTEN.
>>>
>>>
>>> /opt/www/logs/*log {
>>>
>>> compress
>>>
>>> daily
>>>
>>> dateext
>>>
>>> missingok
>>>
>>> olddir /opt/www/logs/roll
>>>
>>> rotate 31
>>>
>>> sharedscripts
>>>
>>> postrotate
>>>
>>> /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true
>>>
>>> endscript
>>>
>>> }
>>>
>>> [Sujai@machine0009 logrotate.d]$ cat httpd
>>>
>>> /var/log/httpd/*log {
>>>
>>> missingok
>>>
>>> notifempty
>>>
>>> sharedscripts
>>>
>>> delaycompress
>>>
>>> postrotate
>>>
>>> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null ||
>>> true
>>>
>>> endscript
>>>
>>> }
>>>
>>>
>>> Thanks & regards,
>>> Sujaikumar
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at https://groups.google.com/group/modwsgi.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com
>>> <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at https://groups.google.com/group/modwsgi.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com
>>> <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at https://groups.google.com/group/modwsgi.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com
>> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at https://groups.google.com/group/modwsgi.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com
>> <https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/modwsgi.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com
> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/modwsgi.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/4C85A26F-8162-4000-AB0C-108970B5FDAE%40gmail.com
> <https://groups.google.com/d/msgid/modwsgi/4C85A26F-8162-4000-AB0C-108970B5FDAE%40gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/CAEpWo-7tg4z4cYbytkQXbPOVg7XOKY3sCNy1PohGVQ%2BmDre5%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
