Thanks for looking into this. Well, the stack trace doesn't have much information. The issue happens when caller function name was about to be logged (using the inspect module). This was a working application for over an year and suddenly the situation has occurred.
My inference is that the python state has gone bad due to the apache reload that happens simultaneously due to logrotate configs. My understanding is that, on apache reload, the python processes goes gracefully down rereads the apache configuration(if any) and comes up. Why would existing python library (inspect module) give the 'list index out of range' error? Is there any other way that this situation could happen? Again the apache restart (systemctl restart httpd) has to be done manually to bring the python process back to normal state. Thanks & regards, Sujaikumar On Thu, 16 May 2019, 3:57 am Graham Dumpleton, <[email protected]> wrote: > Hard to tell just from the information you have given because you have > truncated the file system paths so can't tell what package this code is > executed from. The stack trace doesn't even look complete. > > At a guess you are hitting an issue where there if there is a background > thread running when the process is shutdown, that thread can start up when > the interpreter is in a half destroyed state. This can result in strange > things happening due to the interpreter having started to destroy objects > and clear our modules. > > Is this only happening when the processes are being shutdown? > > Graham > > On 16 May 2019, at 1:44 am, Sujai Kumar <[email protected]> wrote: > > Hello All, > > We have a Apache/Python/Django using modwsgi. We hit upon a situation > where the following error is happening in the site, > > *Error Log* > > 2019-05-09 09:18:08,471 - root - ERROR] - Error : Traceback (most recent > call last): > File ".....core/api/api_util.py", line 15, in log_user_access > caller_func_name = inspect.stack()[1][3] # retrieves the name of the > function that calls this function > File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1491, in stack > return getouterframes(sys._getframe(1), context) > File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1468, in > getouterframes > frameinfo = (frame,) + getframeinfo(frame, context) > File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 1442, in > getframeinfo > lines, lnum = findsource(frame) > File ".../Python-3.6.3s/lib/python3.6/inspect.py", line 825, in > findsource > if pat.match(lines[lnum]): break > IndexError: list index out of range > > *My understanding* > I think that the issue is happening because of apache getting reloaded due > to below configurations. At the same time the logrotate triggers 'apache > reload' around the same time. > > *Quick Fix* > The quick fix that we had to do was to restart the apache. Just after the > restart the application started working. > > *Question:* > What has caused this situation and how to avoid this occurring in future > so that we are not reloading it manually? Thanks, > > *Logrotate Configs* > > > [Sujai@machine0009 logrotate.d]$ cat apache > > # THIS FILE IS AUTOMATICALLY DISTRIBUTED BY PUPPET. ANY CHANGES WILL BE > > # OVERWRITTEN. > > > /opt/www/logs/*log { > > compress > > daily > > dateext > > missingok > > olddir /opt/www/logs/roll > > rotate 31 > > sharedscripts > > postrotate > > /bin/systemctl reload httpd.service >/dev/null 2>/dev/null || true > > endscript > > } > > [Sujai@machine0009 logrotate.d]$ cat httpd > > /var/log/httpd/*log { > > missingok > > notifempty > > sharedscripts > > delaycompress > > postrotate > > /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true > > endscript > > } > > > Thanks & regards, > Sujaikumar > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com > <https://groups.google.com/d/msgid/modwsgi/b75f3ce9-7408-4be2-b35f-e131e78bfb2e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com > <https://groups.google.com/d/msgid/modwsgi/970534A1-DBCA-4C56-8A8C-7A9104DBF8B4%40gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAEpWo-5qrsR59MReynDFCS6DRm3vH4OvCmZDXc_CZV3kt%2BCLag%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
