On Jan 7, 4:35 pm, Graham Dumpleton <[email protected]>
wrote:
> 2010/1/8 inaf <[email protected]>:
>
>
>
>
>
> > Graham,
>
> > Thank you very much for your quick response and help... very much
> > appreciated.. see below for my replies..
>
> >> > I think I read pretty much whatever there is to read about these
> >> > errors in this group but still cannot understand what is causing this
> >> > problem in my case..
>
> >> > Every once in a while (very very rarely, compared to the amount of
> >> > traffic being served) I get the following error:
>
> >> Your selection of error log messages is confusing.
>
> > I tried to select the lines that show the specific errors that I am
> > getting from mod_wsgi daemon so that it is not confusing.. I guess I
> > was wrong :)
>
> >> > [Wed Jan 06 22:38:31 2010] [warn] (14)Bad address: mod_wsgi
> >> > (pid=18735): Unable to stat target WSGI script '(null)'.
> >> > [Wed Jan 06 22:38:31 2010] [alert] (14)Bad address: mod_wsgi
> >> > (pid=18735): Request origin could not be validated.
>
> >> Both the above error messages are generated within mod_wsgi daemon
> >> mode process.
>
> >> The first error message, because it says '(null)' indicates that
> >> SCRIPT_FILENAME was missing in data passed to mod_wsgi daemon process
> >> however I don't understand how that could occur at this point.
>
> >> The second error message indicates that some mod_wsgi validation to
> >> protect against malicious attempts to execute arbitrary script as user
> >> that mod_wsgi daemon process runs has failed. This can be a side
> >> effect of corruption indicated from message above, or technically
> >> could indicate an attempt by external code to connect to mod_wsgi
> >> listener sockets directly and try and fake up requests for execution.
>
> >> > [Wed Jan 06 22:38:31 2010] [error] [client 3.181.52.54] Premature end
> >> > of script headers: <script_name>.wsgi
>
> >> This though now is a likely indicator that mod_wsgi daemon process crashed.
>
> >> > [Wed Jan 06 22:38:32 2010] [notice] child pid 2710 exit signal
> >> > Segmentation fault (11)
>
> >> Problem now is that the pid of the process that crashed doesn't match
> >> that in which original error messages occurred, thus why it is a bit
> >> confusing.
>
> > Yes, I see it every time the error occurs.. the pids are always
> > different..
>
> Odd. Can you post a larger snippet of error log from around when event occurs.
Regarding the pids not matching, I found out that the seg fault is for
siteminder agent pid.. but there is an interesting coincidence where
mod_wsgi daemon throws these errors and shortly after the seg fault
comes for siteminder agent.. I also confirmed that it is not always
the case.. so mod_wsgi errors are not always followed by a seg fault
error..
As far as the error logs are concerned, these are pretty much the only
ones along with siteminder lines. Please see below:
After upgrade:
(nothing before these lines for a while)
[Thu Jan 07 13:57:05 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 13:57:05 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
[Thu Jan 07 13:57:06 2010] [notice] child pid 31237 exit signal
Segmentation fault (11)
(nothing after the lines above for a while)
..........
[07/Jan/2010:14:46:28] [Information] SiteMinder Agent
SiteMinder agent is enabled.
[07/Jan/2010:14:46:28] [Information] SiteMinder Agent
Configuration file path:
'/appl/apache1/conf/WebAgent.conf'.
[Thu Jan 07 14:46:28 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 14:46:28 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676200496 using key 0x6b000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550469672 using key 0x6c000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 681279577 using key 0xc8000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 681279577 using key 0xc8000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676397111 using key 0x66000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676429880 using key 0x67000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550633517 using key 0x65000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676462649 using key 0x68000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676495418 using key 0x69000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676266034 using key 0x32000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550502441 using key 0x61000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676331573 using key 0x33000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550567979 using key 0x62000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676364342 using key 0x34000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550600748 using key 0x63000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSem::getSem] Attached to semaphore 676298804 using key 0x6a000dd5
[07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
[CSmSharedSegment::smalloc] Attached to shared memory segment
550535210 using key 0x69000dd5
[07/Jan/2010:14:46:29] [Information] SiteMinder Agent
SiteMinder agent is running.
[Thu Jan 07 14:49:06 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 14:49:06 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
........
[Thu Jan 07 15:00:03 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 15:00:03 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
[Thu Jan 07 15:00:08 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 15:00:08 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
[Thu Jan 07 15:00:45 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 15:00:45 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
[Thu Jan 07 15:00:50 2010] [alert] mod_wsgi (pid=21148): Request
origin could not be validated.
[Thu Jan 07 15:00:50 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
.....
[Thu Jan 07 16:33:42 2010] [alert] mod_wsgi (pid=14480): Request
origin could not be validated.
[Thu Jan 07 16:33:42 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
.........
[Thu Jan 07 17:15:03 2010] [alert] mod_wsgi (pid=27655): Request
origin could not be validated.
[Thu Jan 07 17:15:03 2010] [error] [client 3.49.42.185] Premature end
of script headers: <script_name>.wsgi
*** glibc detected *** malloc(): memory corruption: 0x08435ab8 ***
I know it looks like I am cherry picking these lines but there is
nothing above or below in the logs other than the first snippet where
siteminder specific lines.
Log level is debug now so will see what can be captured..
>
>
>
>
>
> >> > I have siteminder agent running as well and I notice that bunch of seg
> >> > fault errors associated to that follows along with malloc errors...
>
> >> > Here's my configuration:
>
> >> > Apache/2.0.59 (Unix) mod_jk/1.2.18 mod_wsgi/2.6 Python/2.5.4
> >> > configured
>
> >> > WSGIApplicationGroup %{GLOBAL}
> >> > WSGIDaemonProcess wsgi processes=1 threads=1 display-name=%{GROUP}
>
> >> You don't need 'processes=1' as will default to single process and
> >> using 'processes=1' instead of allow it to default has subtle side
> >> affect of setting 'wsgi.multiprocess' to True. You should only use
> >> 'processes=1' if load balancing across many Apache instances where
> >> each has only single process in daemon process group for that
> >> application.
>
> > I have 4 apaches running on the box with the same wsgi configuration..
> > the box has 4 cores hence 4 apaches.. I have only 3 simple wsgi
> > scripts running.. one of them is used for testing, another one is
> > actively used in production and the third one is only hit by a back
> > end script to refresh data in a singleton object, which is used by
> > others for only read.. so I guess it is ok to keep processes=1?
>
> Just because you have four cores doesn't mean you need to run multiple
> Apache instances. Apache is already multiprocess and you can within
> one mod_wsgi daemon process group specify multiple processes as well.
> So, running multiple Apache instances on same box with same
> configuration is not necessary to make the most of those cores. Have a
> read of:
>
> http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modws...
>
> But then, if running multiple Apache instances so you can restart each
> without interfering with the others, then that is a different issue.
> Even then, ensure you read:
>
> http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
>
> Because for mod_wsgi hosted Python applications at least, there are
> various ways you can trigger reloading of application without needing
> to restart whole of Apache.
>
>
The reason for multiple apaches is to be able to isolate issues.. it
is a long story :)
I followed your advice on processes and changed the configuration to
the following:
WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess wsgi threads=1 display-name=%{GROUP}
WSGIProcessGroup wsgi
Did not seem to help (not sure if it was expected to help)..
>
>
>
> >> > WSGIProcessGroup wsgi
>
> >> > Any help and insight would be much appreciated..
>
> >> I can only suggest trying mod_wsgi 3.1.
>
> > Just did.. monitoring to see if I get any errors..
>
> >> Other than that don't really have an answer. It looks like memory
> >> corruption but whether the source is mod_wsgi, another Apache module
> >> or a Python C extension module, don't know.
>
> >> What third party Python modules do you use which may have a C
> >> extension module component?
>
> >> Anyway, will have a think about it some more and see if can come up
> >> with any suggestions of things to look for or try. A snippet of log
> >> file covering a longer amount of time may be a good point.
>
> >> Graham
>
> > Another question I had was whether slow network connections might
> > cause this issue.. what are your thoughts on that?
>
> Not a process crash as you are seeing.
>
> Graham- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
Thanks,
-Cem
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
