Hi Graham, Yes, mod_wsgi is pulling the right python. I have verified them before writing to you. As mentioned earlier, the system is working fine for more than a year now. This issue happened suddenly and there were no code change recently and hence the concern. The only thing, that I see as external factor is 'apache reload' (as part of logrotate simultaneously). Are there any other things that we need to check? I shall send the complete stack trace in some time.
Thanks & regards, Sujaikumar On Thu, 16 May 2019, 8:03 am Graham Dumpleton, <[email protected]> wrote: > Are you sure you mod_wsgi is pulling in the correct Python shared library, > given that it looks like you are using a custom Python installation. > > Is Python 3.6 installed as system Python? > > Are you wanting to actually use a separate Python installation you > installed from source code? > > Did you compile mod_wsgi from source code against any separate Python > installation? If you did that, was LD_RUN_PATH environment variable set to > the lib directory for that Python installation where the Python shared > library is, when mod_wsgi was being built, to ensure that at run time it > would use that library and not the system Python one. > > If you are getting the wrong Python shared library pulled in, and there > are patch level differences, you could see all sorts of strange issues. > > On 16 May 2019, at 11:36 am, Sujai Kumar <[email protected]> wrote: > > Thanks for looking into this. > > Well, the stack trace doesn't have much information. The issue happens > when caller function name was about to be logged (using the inspect > module). This was a working application for over an year and suddenly the > situation has occurred. > > My inference is that the python state has gone bad due to the apache > reload that happens simultaneously due to logrotate configs. My > understanding is that, on apache reload, the python processes goes > gracefully down rereads the apache configuration(if any) and comes up. Why > would existing python library (inspect module) give the 'list index out of > range' error? Is there any other way that this situation could happen? > > Again the apache restart (systemctl restart httpd) has to be done manually > to bring the python process back to normal state. > > Thanks & regards, > Sujaikumar > > On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <[email protected]> > wrote: > >> Hard to tell just from the information you have given because you have >> truncated the file system paths so can't tell what package this code is >> executed from. The stack trace doesn't even look complete. >> >> At a guess you are hitting an issue where there if there is a background >> thread running when the process is shutdown, that thread can start up when >> the interpreter is in a half destroyed state. This can result in strange >> things happening due to the interpreter having started to destroy objects >> and clear our modules. >> >> Is this only happening when the processes are being shutdown? >> >> Graham >> >> On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected]> wrote: >> >> Hello All, >> >> We have a Apache/Python/Django using modwsgi. We hit upon a situation >> where the following error is happening in the site, >> >> *Error Log* >> >> 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent >> call last): >> File ".....core/api/api_util.py", line 15, in log_user_access >> caller_func_name = inspect.stack()[1][3] # retrieves the name of the >> function that calls this function >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack >> return getouterframes(sys._getframe(1), context) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in >> getouterframes >> frameinfo = (frame,) + getframeinfo(frame, context) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in >> getframeinfo >> lines, lnum = findsource(frame) >> File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in >> findsource >> if pat.match(lines[lnum]): break >> IndexError: list index out of range >> >> *My understanding* >> I think that the issue is happening because of apache getting reloaded >> due to below configurations. At the same time the logrotate triggers >> 'apache reload' around the same time. >> >> *Quick Fix* >> The quick fix that we had to do was to restart the apache. Just after the >> restart the application started working. >> >> *Question:* >> What has caused this situation and how to avoid this occurring in future >> so that we are not reloading it manually? Thanks, >> >> *Logrotate Configs* >> >> >> [Sujai@machine0009 logrotate.d]$ cat apache >> >> # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE >> >> # OVERWRITTEN. >> >> >> /opt/www/logs/*log { >> >> compress >> >> daily >> >> dateext >> >> missingok >> >> olddir /opt/www/logs/roll >> >> rotate 31 >> >> sharedscripts >> >> postrotate >> >> /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true >> >> endscript >> >> } >> >> [Sujai@machine0009 logrotate.d]$ cat httpd >> >> /var/log/httpd/*log { >> >> missingok >> >> notifempty >> >> sharedscripts >> >> delaycompress >> >> postrotate >> >> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || >> true >> >> endscript >> >> } >> >> >> Thanks & regards, >> Sujaikumar >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/modwsgi. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com >> <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/modwsgi. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com >> <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com > <https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com > <https://groups.google.com/d/msgid/modwsgi/8D1DA36D-513A-4E2D-88B1-9E3290EE2D9E%40gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAEpWo-6Bgm1KZT35Pda6eZX9Via5AfX_ZV__yu%2BUCfpGYPUSKA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
