2010/1/8 inaf <[email protected]>:
> Regarding the pids not matching, I found out that the seg fault is for
> siteminder agent pid.. but there is an interesting coincidence where
> mod_wsgi daemon throws these errors and shortly after the seg fault
> comes for siteminder agent.. I also confirmed that it is not always
> the case.. so mod_wsgi errors are not always followed by a seg fault
> error..
For this site minder, you have a mod_siteminder module loaded into
Apache. Correct?
This still stinks of memory corruption. Specifically, some Apache
module is keeping a pointer to memory contained in an Apache memory
pool after the memory pool is released. That memory area is then being
given out mod_wsgi as per request scratch space. The other Apache
module is then scribbling in the memory.
What doesn't make sense is why it is always replacing it with
'<script_name>'. Are you doing that to protect what the actual paths
are or is that really what the error log files say?
If it is always that, then perhaps not memory corruption but some
Apache module deliberately updating r->filename in Apache request
structure and replacing it with a new string encompassing that string.
Can you clarify whether you are modifying the logs or whether that is
the actual value.
Graham
> As far as the error logs are concerned, these are pretty much the only
> ones along with siteminder lines. Please see below:
>
>
> After upgrade:
>
> (nothing before these lines for a while)
> [Thu Jan 07 13:57:05 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 13:57:05 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> [Thu Jan 07 13:57:06 2010] [notice] child pid 31237 exit signal
> Segmentation fault (11)
> (nothing after the lines above for a while)
>
> ..........
>
>
> [07/Jan/2010:14:46:28] [Information] SiteMinder Agent
> SiteMinder agent is enabled.
> [07/Jan/2010:14:46:28] [Information] SiteMinder Agent
> Configuration file path:
> '/appl/apache1/conf/WebAgent.conf'.
> [Thu Jan 07 14:46:28 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 14:46:28 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676200496 using key 0x6b000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550469672 using key 0x6c000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 681279577 using key 0xc8000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 681279577 using key 0xc8000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676397111 using key 0x66000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676429880 using key 0x67000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550633517 using key 0x65000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676462649 using key 0x68000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676495418 using key 0x69000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676266034 using key 0x32000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550502441 using key 0x61000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676331573 using key 0x33000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550567979 using key 0x62000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676364342 using key 0x34000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550600748 using key 0x63000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSem::getSem] Attached to semaphore 676298804 using key 0x6a000dd5
> [07/Jan/2010:14:46:29] [Info] [CA WebAgent IPC] [10627]
> [CSmSharedSegment::smalloc] Attached to shared memory segment
> 550535210 using key 0x69000dd5
> [07/Jan/2010:14:46:29] [Information] SiteMinder Agent
> SiteMinder agent is running.
> [Thu Jan 07 14:49:06 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 14:49:06 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
>
> ........
>
> [Thu Jan 07 15:00:03 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 15:00:03 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> [Thu Jan 07 15:00:08 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 15:00:08 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> [Thu Jan 07 15:00:45 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 15:00:45 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> [Thu Jan 07 15:00:50 2010] [alert] mod_wsgi (pid=21148): Request
> origin could not be validated.
> [Thu Jan 07 15:00:50 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
>
> .....
>
> [Thu Jan 07 16:33:42 2010] [alert] mod_wsgi (pid=14480): Request
> origin could not be validated.
> [Thu Jan 07 16:33:42 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
>
>
> .........
>
>
> [Thu Jan 07 17:15:03 2010] [alert] mod_wsgi (pid=27655): Request
> origin could not be validated.
> [Thu Jan 07 17:15:03 2010] [error] [client 3.49.42.185] Premature end
> of script headers: <script_name>.wsgi
> *** glibc detected *** malloc(): memory corruption: 0x08435ab8 ***
>
> I know it looks like I am cherry picking these lines but there is
> nothing above or below in the logs other than the first snippet where
> siteminder specific lines.
>
> Log level is debug now so will see what can be captured..
>
>>
>>
>>
>>
>>
>> >> > I have siteminder agent running as well and I notice that bunch of seg
>> >> > fault errors associated to that follows along with malloc errors...
>>
>> >> > Here's my configuration:
>>
>> >> > Apache/2.0.59 (Unix) mod_jk/1.2.18 mod_wsgi/2.6 Python/2.5.4
>> >> > configured
>>
>> >> > WSGIApplicationGroup %{GLOBAL}
>> >> > WSGIDaemonProcess wsgi processes=1 threads=1 display-name=%{GROUP}
>>
>> >> You don't need 'processes=1' as will default to single process and
>> >> using 'processes=1' instead of allow it to default has subtle side
>> >> affect of setting 'wsgi.multiprocess' to True. You should only use
>> >> 'processes=1' if load balancing across many Apache instances where
>> >> each has only single process in daemon process group for that
>> >> application.
>>
>> > I have 4 apaches running on the box with the same wsgi configuration..
>> > the box has 4 cores hence 4 apaches.. I have only 3 simple wsgi
>> > scripts running.. one of them is used for testing, another one is
>> > actively used in production and the third one is only hit by a back
>> > end script to refresh data in a singleton object, which is used by
>> > others for only read.. so I guess it is ok to keep processes=1?
>>
>> Just because you have four cores doesn't mean you need to run multiple
>> Apache instances. Apache is already multiprocess and you can within
>> one mod_wsgi daemon process group specify multiple processes as well.
>> So, running multiple Apache instances on same box with same
>> configuration is not necessary to make the most of those cores. Have a
>> read of:
>>
>> http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modws...
>>
>> But then, if running multiple Apache instances so you can restart each
>> without interfering with the others, then that is a different issue.
>> Even then, ensure you read:
>>
>> http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
>>
>> Because for mod_wsgi hosted Python applications at least, there are
>> various ways you can trigger reloading of application without needing
>> to restart whole of Apache.
>>
>>
>
> The reason for multiple apaches is to be able to isolate issues.. it
> is a long story :)
>
> I followed your advice on processes and changed the configuration to
> the following:
>
> WSGIApplicationGroup %{GLOBAL}
> WSGIDaemonProcess wsgi threads=1 display-name=%{GROUP}
> WSGIProcessGroup wsgi
>
> Did not seem to help (not sure if it was expected to help)..
>
>
>>
>>
>>
>> >> > WSGIProcessGroup wsgi
>>
>> >> > Any help and insight would be much appreciated..
>>
>> >> I can only suggest trying mod_wsgi 3.1.
>>
>> > Just did.. monitoring to see if I get any errors..
>>
>> >> Other than that don't really have an answer. It looks like memory
>> >> corruption but whether the source is mod_wsgi, another Apache module
>> >> or a Python C extension module, don't know.
>>
>> >> What third party Python modules do you use which may have a C
>> >> extension module component?
>>
>> >> Anyway, will have a think about it some more and see if can come up
>> >> with any suggestions of things to look for or try. A snippet of log
>> >> file covering a longer amount of time may be a good point.
>>
>> >> Graham
>>
>> > Another question I had was whether slow network connections might
>> > cause this issue.. what are your thoughts on that?
>>
>> Not a process crash as you are seeing.
>>
>> Graham- Hide quoted text -
>>
>> - Show quoted text -- Hide quoted text -
>>
>> - Show quoted text -- Hide quoted text -
>>
>> - Show quoted text -
>
> Thanks,
> -Cem
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
