Haha, for some reason I read it as Python 2.6 when I said ancient.
Anyway, can you grab section of code around line 825 in
/data/dev/Python-3.6.3s/lib/python3.6/inspect.py.
Python 3.6.7 doesn't match that code at that line number, so wondering whether
you are using older patch revision which was since updated. In 3.6.7 it has:
if iscode(object):
if not hasattr(object, 'co_firstlineno'):
raise OSError('could not find function definition')
lnum = object.co_firstlineno - 1. #### <----- This is line 825
pat =
re.compile(r'^(\s*def\s)|(\s*async\s+def\s)|(.*(?<!\w)lambda(:|\s))|^(\s*@)')
while lnum > 0:
if pat.match(lines[lnum]): break
lnum = lnum - 1
return lines, lnum
In your stack trace, line 825 is 3 lines later. Has the code in that function
changed, with your version not having the "while lnum > 0"?
> On 16 May 2019, at 1:11 pm, Sujai Kumar <[email protected]> wrote:
>
> Hi Graham,
>
> Here are the version of softwares being used.
> Python -> 3.6
> mod_wsgi -> 4.6.4
> Sujai@machine0009:/usr/lib64/httpd/modules> ldd mod_wsgi.so
> linux-vdso.so.1 => (0x00007ffc74cd7000)
> libpython3.6m.so.1.0 =>
> /data/ev/Python-3.6.3s/lib/libpython3.6m.so.1.0 (0x00007fbd76c66000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbd76a4a000)
> libc.so.6 => /lib64/libc.so.6 (0x00007fbd7667d000)
> libssl.so.1.0.0 => /data/dev/openssl-1.0.2sh/lib/libssl.so.1.0.0
> (0x00007fbd7640b000)
> libcrypto.so.1.0.0 =>
> /data/dev/openssl-1.0.2sh/lib/libcrypto.so.1.0.0 (0x00007fbd75fbd000)
> libz.so.1 => /data/dev/Python-3.6.3s/lib/libz.so.1
> (0x00007fbd75da2000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00007fbd75b9e000)
> libutil.so.1 => /lib64/libutil.so.1 (0x00007fbd7599b000)
> libm.so.6 => /lib64/libm.so.6 (0x00007fbd75699000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fbd773fa000)
> Sujai@machine0009 :/usr/lib64/httpd/modules> strings mod_wsgi.so | grep -w -A
> 1 "wsgi_init"
> wsgi_init
> 4.6.4
>
> Stack Trace of the Error
> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent call
> last):
> File
> "/data/dev/deployment/production/release/server-python/sf_django/sf_app/views/utils/__init__.py",
> line 135, in function_wrapper
> return_value = func(caller_object, request, *args, **kwargs)
> File
> "/data/dev/deployment/production/release/server-python/sf_django/starfish_app/views/api/api_id.py",
> line 27, in get
> response = get_sub_discipline_wise_api_ids(user_itss_id,
> parameters["bs_revision_id"])
> File
> "/data/dev/deployment/production/release/server-python/sf_django/starfish_app/core/api/api_id.py",
> line 21, in get_sub_discipline_wise_api_ids
> log_user_access(user_itss_id, func_param={"bs_revision_id":
> bs_revision_id})
> File
> "/data/dev/deployment/production/release/server-python/sf_django/starfish_app/core/api/api_util.py",
> line 15, in log_user_access
> caller_func_name = inspect.stack()[1][3] # retrieves the name of the
> function that calls this function
> File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack
> return getouterframes(sys._getframe(1), context)
> File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in
> getouterframes
> frameinfo = (frame,) + getframeinfo(frame, context)
> File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in
> getframeinfo
> lines, lnum = findsource(frame)
> File "/data/dev/Python-3.6.3s/lib/python3.6/inspect.py", line 825, in
> findsource
> if pat.match(lines[lnum]): break
> IndexError: list index out of range
>
> In what case the interpreter will have half destroyed code. Won't the
> 'apapche reload' perform the process shutdown gracefully (maening, all
> existing connections to a thread would be served before the thread redeems
> itself)?
>
> Thanks & regards,
> Sujaikumar
>
> On Thu, May 16, 2019 at 8:15 AM Graham Dumpleton <[email protected]
> <mailto:[email protected]>> wrote:
> As already mentioned, if it is only on process shutdown (caused by a reload),
> then it is possibly because it is trying to execute code when the interpreter
> has been half destroyed. You cut off the file system paths and so I can't see
> what:
>
> File ".....core/api/api_util.py", line 15, in log_user_access
>
> is likely a part of (if open source project) and infer anything about whether
> a separate thread is running. So a more complete stack trace may help.
>
> Also, given you are using an ancient Python version, what mod_wsgi version
> are you using?
>
>> On 16 May 2019, at 12:41 pm, Sujai Kumar <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi Graham,
>>
>> Yes, mod_wsgi is pulling the right python. I have verified them before
>> writing to you. As mentioned earlier, the system is working fine for more
>> than a year now. This issue happened suddenly and there were no code change
>> recently and hence the concern. The only thing, that I see as external
>> factor is 'apache reload' (as part of logrotate simultaneously). Are there
>> any other things that we need to check? I shall send the complete stack
>> trace in some time.
>>
>> Thanks & regards,
>> Sujaikumar
>>
>> On Thu, 16 May 2019, 8:03 am Graham Dumpleton, <[email protected]
>> <mailto:[email protected]>> wrote:
>> Are you sure you mod_wsgi is pulling in the correct Python shared library,
>> given that it looks like you are using a custom Python installation.
>>
>> Is Python 3.6 installed as system Python?
>>
>> Are you wanting to actually use a separate Python installation you installed
>> from source code?
>>
>> Did you compile mod_wsgi from source code against any separate Python
>> installation? If you did that, was LD_RUN_PATH environment variable set to
>> the lib directory for that Python installation where the Python shared
>> library is, when mod_wsgi was being built, to ensure that at run time it
>> would use that library and not the system Python one.
>>
>> If you are getting the wrong Python shared library pulled in, and there are
>> patch level differences, you could see all sorts of strange issues.
>>
>>> On 16 May 2019, at 11:36 am, Sujai Kumar <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Thanks for looking into this.
>>>
>>> Well, the stack trace doesn't have much information. The issue happens when
>>> caller function name was about to be logged (using the inspect module).
>>> This was a working application for over an year and suddenly the situation
>>> has occurred.
>>>
>>> My inference is that the python state has gone bad due to the apache reload
>>> that happens simultaneously due to logrotate configs. My understanding is
>>> that, on apache reload, the python processes goes gracefully down rereads
>>> the apache configuration(if any) and comes up. Why would existing python
>>> library (inspect module) give the 'list index out of range' error? Is there
>>> any other way that this situation could happen?
>>>
>>> Again the apache restart (systemctl restart httpd) has to be done manually
>>> to bring the python process back to normal state.
>>>
>>> Thanks & regards,
>>> Sujaikumar
>>>
>>> On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <[email protected]
>>> <mailto:[email protected]>> wrote:
>>> Hard to tell just from the information you have given because you have
>>> truncated the file system paths so can't tell what package this code is
>>> executed from. The stack trace doesn't even look complete.
>>>
>>> At a guess you are hitting an issue where there if there is a background
>>> thread running when the process is shutdown, that thread can start up when
>>> the interpreter is in a half destroyed state. This can result in strange
>>> things happening due to the interpreter having started to destroy objects
>>> and clear our modules.
>>>
>>> Is this only happening when the processes are being shutdown?
>>>
>>> Graham
>>>
>>>> On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Hello All,
>>>>
>>>> We have a Apache/Python/Django using modwsgi. We hit upon a situation
>>>> where the following error is happening in the site,
>>>>
>>>> Error Log
>>>>
>>>> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent
>>>> call last):
>>>> File ".....core/api/api_util.py", line 15, in log_user_access
>>>> caller_func_name = inspect.stack()[1][3] # retrieves the name of the
>>>> function that calls this function
>>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack
>>>> return getouterframes(sys._getframe(1), context)
>>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in
>>>> getouterframes
>>>> frameinfo = (frame,) + getframeinfo(frame, context)
>>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in
>>>> getframeinfo
>>>> lines, lnum = findsource(frame)
>>>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in
>>>> findsource
>>>> if pat.match(lines[lnum]): break
>>>> IndexError: list index out of range
>>>>
>>>> My understanding
>>>> I think that the issue is happening because of apache getting reloaded due
>>>> to below configurations. At the same time the logrotate triggers 'apache
>>>> reload' around the same time.
>>>>
>>>> Quick Fix
>>>> The quick fix that we had to do was to restart the apache. Just after the
>>>> restart the application started working.
>>>>
>>>> Question:
>>>> What has caused this situation and how to avoid this occurring in future
>>>> so that we are not reloading it manually? Thanks,
>>>>
>>>> Logrotate Configs
>>>>
>>>>
>>>>
>>>> [Sujai@machine0009 logrotate.d]$ cat apache
>>>>
>>>> # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE
>>>>
>>>> # OVERWRITTEN.
>>>>
>>>>
>>>>
>>>> /opt/www/logs/*log {
>>>>
>>>> compress
>>>>
>>>> daily
>>>>
>>>> dateext
>>>>
>>>> missingok
>>>>
>>>> olddir /opt/www/logs/roll
>>>>
>>>> rotate 31
>>>>
>>>> sharedscripts
>>>>
>>>> postrotate
>>>>
>>>> /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true
>>>>
>>>> endscript
>>>>
>>>> }
>>>>
>>>> [Sujai@machine0009 logrotate.d]$ cat httpd
>>>>
>>>> /var/log/httpd/*log {
>>>>
>>>> missingok
>>>>
>>>> notifempty
>>>>
>>>> sharedscripts
>>>>
>>>> delaycompress
>>>>
>>>> postrotate
>>>>
>>>> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
>>>>
>>>> endscript
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> Thanks & regards,
>>>> Sujaikumar
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected]
>>>> <mailto:[email protected]>.
>>>> To post to this group, send email to [email protected]
>>>> <mailto:[email protected]>.
>>>> Visit this group at https://groups.google.com/group/modwsgi
>>>> <https://groups.google.com/group/modwsgi>.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>> For more options, visit https://groups.google.com/d/optout
>>>> <https://groups.google.com/d/optout>.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected]
>>> <mailto:[email protected]>.
>>> To post to this group, send email to [email protected]
>>> <mailto:[email protected]>.
>>> Visit this group at https://groups.google.com/group/modwsgi
>>> <https://groups.google.com/group/modwsgi>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer>.
>>> For more options, visit https://groups.google.com/d/optout
>>> <https://groups.google.com/d/optout>.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected]
>>> <mailto:[email protected]>.
>>> To post to this group, send email to [email protected]
>>> <mailto:[email protected]>.
>>> Visit this group at https://groups.google.com/group/modwsgi
>>> <https://groups.google.com/group/modwsgi>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>>> For more options, visit https://groups.google.com/d/optout
>>> <https://groups.google.com/d/optout>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To post to this group, send email to [email protected]
>> <mailto:[email protected]>.
>> Visit this group at https://groups.google.com/group/modwsgi
>> <https://groups.google.com/group/modwsgi>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com?utm_medium=email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To post to this group, send email to [email protected]
>> <mailto:[email protected]>.
>> Visit this group at https://groups.google.com/group/modwsgi
>> <https://groups.google.com/group/modwsgi>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> Visit this group at https://groups.google.com/group/modwsgi
> <https://groups.google.com/group/modwsgi>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/4C85A26F-8162-4000-AB0C-108970B5FDAE%40gmail.com
>
> <https://groups.google.com/d/msgid/modwsgi/4C85A26F-8162-4000-AB0C-108970B5FDAE%40gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> Visit this group at https://groups.google.com/group/modwsgi
> <https://groups.google.com/group/modwsgi>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/CAEpWo-7tg4z4cYbytkQXbPOVg7XOKY3sCNy1PohGVQ%2BmDre5%2Bg%40mail.gmail.com
>
> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-7tg4z4cYbytkQXbPOVg7XOKY3sCNy1PohGVQ%2BmDre5%2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/DB703E53-07CC-4848-BC16-D72AF019ACD8%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
