Are you sure you mod_wsgi is pulling in the correct Python shared library, given that it looks like you are using a custom Python installation.
Is Python 3.6 installed as system Python? Are you wanting to actually use a separate Python installation you installed from source code? Did you compile mod_wsgi from source code against any separate Python installation? If you did that, was LD_RUN_PATH environment variable set to the lib directory for that Python installation where the Python shared library is, when mod_wsgi was being built, to ensure that at run time it would use that library and not the system Python one. If you are getting the wrong Python shared library pulled in, and there are patch level differences, you could see all sorts of strange issues. > On 16 May 2019, at 11:36 am, Sujai Kumar <[email protected]> wrote: > > Thanks for looking into this. > > Well, the stack trace doesn't have much information. The issue happens when > caller function name was about to be logged (using the inspect module). This > was a working application for over an year and suddenly the situation has > occurred. > > My inference is that the python state has gone bad due to the apache reload > that happens simultaneously due to logrotate configs. My understanding is > that, on apache reload, the python processes goes gracefully down rereads the > apache configuration(if any) and comes up. Why would existing python library > (inspect module) give the 'list index out of range' error? Is there any other > way that this situation could happen? > > Again the apache restart (systemctl restart httpd) has to be done manually to > bring the python process back to normal state. > > Thanks & regards, > Sujaikumar > > On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <[email protected] > <mailto:[email protected]>> wrote: > Hard to tell just from the information you have given because you have > truncated the file system paths so can't tell what package this code is > executed from. The stack trace doesn't even look complete. > > At a guess you are hitting an issue where there if there is a background > thread running when the process is shutdown, that thread can start up when > the interpreter is in a half destroyed state. This can result in strange > things happening due to the interpreter having started to destroy objects and > clear our modules. > > Is this only happening when the processes are being shutdown? > > Graham > >> On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello All, >> >> We have a Apache/Python/Django using modwsgi. We hit upon a situation where >> the following error is happening in the site, >> >> Error Log >> >> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent >> call last): >> File ".....core/api/api_util.py", line 15, in log_user_access >> caller_func_name = inspect.stack()[1][3] # retrieves the name of the >> function that calls this function >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack >> return getouterframes(sys._getframe(1), context) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in >> getouterframes >> frameinfo = (frame,) + getframeinfo(frame, context) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in >> getframeinfo >> lines, lnum = findsource(frame) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in findsource >> if pat.match(lines[lnum]): break >> IndexError: list index out of range >> >> My understanding >> I think that the issue is happening because of apache getting reloaded due >> to below configurations. At the same time the logrotate triggers 'apache >> reload' around the same time. >> >> Quick Fix >> The quick fix that we had to do was to restart the apache. Just after the >> restart the application started working. >> >> Question: >> What has caused this situation and how to avoid this occurring in future so >> that we are not reloading it manually? Thanks, >> >> Logrotate Configs >> >> >> >> [Sujai@machine0009 logrotate.d]$ cat apache >> >> # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE >> >> # OVERWRITTEN. >> >> >> >> /opt/www/logs/*log { >> >> compress >> >> daily >> >> dateext >> >> missingok >> >> olddir /opt/www/logs/roll >> >> rotate 31 >> >> sharedscripts >> >> postrotate >> >> /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true >> >> endscript >> >> } >> >> [Sujai@machine0009 logrotate.d]$ cat httpd >> >> /var/log/httpd/*log { >> >> missingok >> >> notifempty >> >> sharedscripts >> >> delaycompress >> >> postrotate >> >> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true >> >> endscript >> >> } >> >> >> >> Thanks & regards, >> Sujaikumar >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com > > <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com > > <https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com. For more options, visit https://groups.google.com/d/optout.
