As already mentioned, if it is only on process shutdown (caused by a reload), then it is possibly because it is trying to execute code when the interpreter has been half destroyed. You cut off the file system paths and so I can't see what:
File ".....core/api/api_util.py", line 15, in log_user_access is likely a part of (if open source project) and infer anything about whether a separate thread is running. So a more complete stack trace may help. Also, given you are using an ancient Python version, what mod_wsgi version are you using? > On 16 May 2019, at 12:41 pm, Sujai Kumar <[email protected]> wrote: > > Hi Graham, > > Yes, mod_wsgi is pulling the right python. I have verified them before > writing to you. As mentioned earlier, the system is working fine for more > than a year now. This issue happened suddenly and there were no code change > recently and hence the concern. The only thing, that I see as external factor > is 'apache reload' (as part of logrotate simultaneously). Are there any other > things that we need to check? I shall send the complete stack trace in some > time. > > Thanks & regards, > Sujaikumar > > On Thu, 16 May 2019, 8:03 am Graham Dumpleton, <[email protected] > <mailto:[email protected]>> wrote: > Are you sure you mod_wsgi is pulling in the correct Python shared library, > given that it looks like you are using a custom Python installation. > > Is Python 3.6 installed as system Python? > > Are you wanting to actually use a separate Python installation you installed > from source code? > > Did you compile mod_wsgi from source code against any separate Python > installation? If you did that, was LD_RUN_PATH environment variable set to > the lib directory for that Python installation where the Python shared > library is, when mod_wsgi was being built, to ensure that at run time it > would use that library and not the system Python one. > > If you are getting the wrong Python shared library pulled in, and there are > patch level differences, you could see all sorts of strange issues. > >> On 16 May 2019, at 11:36 am, Sujai Kumar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thanks for looking into this. >> >> Well, the stack trace doesn't have much information. The issue happens when >> caller function name was about to be logged (using the inspect module). This >> was a working application for over an year and suddenly the situation has >> occurred. >> >> My inference is that the python state has gone bad due to the apache reload >> that happens simultaneously due to logrotate configs. My understanding is >> that, on apache reload, the python processes goes gracefully down rereads >> the apache configuration(if any) and comes up. Why would existing python >> library (inspect module) give the 'list index out of range' error? Is there >> any other way that this situation could happen? >> >> Again the apache restart (systemctl restart httpd) has to be done manually >> to bring the python process back to normal state. >> >> Thanks & regards, >> Sujaikumar >> >> On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <[email protected] >> <mailto:[email protected]>> wrote: >> Hard to tell just from the information you have given because you have >> truncated the file system paths so can't tell what package this code is >> executed from. The stack trace doesn't even look complete. >> >> At a guess you are hitting an issue where there if there is a background >> thread running when the process is shutdown, that thread can start up when >> the interpreter is in a half destroyed state. This can result in strange >> things happening due to the interpreter having started to destroy objects >> and clear our modules. >> >> Is this only happening when the processes are being shutdown? >> >> Graham >> >>> On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hello All, >>> >>> We have a Apache/Python/Django using modwsgi. We hit upon a situation where >>> the following error is happening in the site, >>> >>> Error Log >>> >>> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent >>> call last): >>> File ".....core/api/api_util.py", line 15, in log_user_access >>> caller_func_name = inspect.stack()[1][3] # retrieves the name of the >>> function that calls this function >>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack >>> return getouterframes(sys._getframe(1), context) >>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in >>> getouterframes >>> frameinfo = (frame,) + getframeinfo(frame, context) >>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in >>> getframeinfo >>> lines, lnum = findsource(frame) >>> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in findsource >>> if pat.match(lines[lnum]): break >>> IndexError: list index out of range >>> >>> My understanding >>> I think that the issue is happening because of apache getting reloaded due >>> to below configurations. At the same time the logrotate triggers 'apache >>> reload' around the same time. >>> >>> Quick Fix >>> The quick fix that we had to do was to restart the apache. Just after the >>> restart the application started working. >>> >>> Question: >>> What has caused this situation and how to avoid this occurring in future so >>> that we are not reloading it manually? Thanks, >>> >>> Logrotate Configs >>> >>> >>> >>> [Sujai@machine0009 logrotate.d]$ cat apache >>> >>> # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE >>> >>> # OVERWRITTEN. >>> >>> >>> >>> /opt/www/logs/*log { >>> >>> compress >>> >>> daily >>> >>> dateext >>> >>> missingok >>> >>> olddir /opt/www/logs/roll >>> >>> rotate 31 >>> >>> sharedscripts >>> >>> postrotate >>> >>> /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true >>> >>> endscript >>> >>> } >>> >>> [Sujai@machine0009 logrotate.d]$ cat httpd >>> >>> /var/log/httpd/*log { >>> >>> missingok >>> >>> notifempty >>> >>> sharedscripts >>> >>> delaycompress >>> >>> postrotate >>> >>> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true >>> >>> endscript >>> >>> } >>> >>> >>> >>> Thanks & regards, >>> Sujaikumar >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com >> >> <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com > > <https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com > > <https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/4C85A26F-8162-4000-AB0C-108970B5FDAE%40gmail.com. For more options, visit https://groups.google.com/d/optout.
