Giving up for the night. Getting memory corruption all over the place now and can’t even get sane stack traces from gdb. Nothing making sense. Will start over tomorrow.
> On 15 Aug 2016, at 11:30 PM, Stefan Nastic <[email protected]> wrote: > > tried again and no luck .... > > Just to remind you that this happens immediately upon the firs request after > restarting the apache, so if you are referring to gc below, I don't see how > it can kick in and cause a problem. Just a thought .... > > Stefan > > On 15-Aug-16 15:16, Graham Dumpleton wrote: >> I am not confident that next change will solve the problem, but can you try >> pulling down source code again and try. >> >> I can’t recreate it so far after this change, but then that is what happened >> with the first change and it only started occurring again after you and >> someone else said it didn’t solve problem. Very confusing. >> >> The problem seems to relate to strange things going on when Python objects >> are being destroyed. Python 3 appears to behave differently with the new >> streams implementation. >> >> Graham >> >>> On 15 Aug 2016, at 10:51 PM, Stefan Nastic <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi, >>> >>> I have just tested it (4.5.5) and am still getting the same error. >>> >>> Best, >>> >>> Stefan >>> >>> On 15-Aug-16 14:46, Graham Dumpleton wrote: >>>> Problem only affects Python 3.X. Verification of fix being worked on right >>>> now. >>>> >>>> If you are able to download: >>>> >>>> https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip >>>> <https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip> >>>> >>>> and try with that it would help immensely. >>>> >>>> Thanks. >>>> >>>> Graham >>>> >>>>> On 15 Aug 2016, at 10:43 PM, Stefan Nastic <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> sorry for a bit delayed response, but I was traveling the last week and >>>>> had a very limited internet connectivity. >>>>> >>>>> Now back to the issue. I tested the version 4.5.4 and in short there is >>>>> some problem as I get seg fault in the apache. I have checked 4.5.4. for >>>>> the most common issues such as loading mod_python, linking against shared >>>>> libraries, etc. and there everything seam to be fine. I have also tested >>>>> the same compilation and installation procedure with 4.5.3 and it works >>>>> fine. >>>>> >>>>> So I suspect that this is a codding error, most probably accessing the >>>>> protected memory region when you try to access/reproduce Log ID seed. >>>>> However, this is only my assumption since I have not had a chance to look >>>>> at your code. >>>>> >>>>> For debugging purposes .... >>>>> >>>>> Apache error log: >>>>> >>>>> [mpm_event:notice] [pid 25563:tid 140401132873600] AH00494: SIGHUP >>>>> received. Attempting to restart >>>>> [mpm_event:notice] [pid 25563:tid 140401132873600] AH00489: Apache/2.4.10 >>>>> (Debian) mod_wsgi/4.5.4 Python/3.4.2 configured -- resuming normal >>>>> operations >>>>> [core:notice] [pid 25563:tid 140401132873600] AH00094: Command line: >>>>> '/usr/sbin/apache2' >>>>> [core:notice] [pid 25563:tid 140401132873600] AH00052: child pid 27051 >>>>> exit signal Segmentation fault (11) >>>>> >>>>> OS: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4 >>>>> (2016-02-29) x86_64 GNU/Linux >>>>> >>>>> APACHE: >>>>> Server version: Apache/2.4.10 (Debian) >>>>> Server built: Nov 28 2015 14:05:48 >>>>> Server's Module Magic Number: >>>>> Server loaded: APR 1.5.1, APR-UTIL 1.5.4 >>>>> Compiled using: APR 1.5.1, APR-UTIL 1.5.4 >>>>> Architecture: 64-bit >>>>> Server MPM: event >>>>> threaded: yes (fixed thread count) >>>>> forked: yes (variable process count) >>>>> Server compiled with.... >>>>> -D APR_HAS_SENDFILE >>>>> -D APR_HAS_MMAP >>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>>> -D APR_USE_SYSVSEM_SERIALIZE >>>>> -D APR_USE_PTHREAD_SERIALIZE >>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>>> -D APR_HAS_OTHER_CHILD >>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>>> -D DYNAMIC_MODULE_LIMIT=256 >>>>> -D HTTPD_ROOT="/etc/apache2" >>>>> -D SUEXEC_BIN="/usr/lib/apache2/suexec" >>>>> -D DEFAULT_PIDLOG="/var/run/apache2.pid" >>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>>> -D DEFAULT_ERRORLOG="logs/error_log" >>>>> -D AP_TYPES_CONFIG_FILE="mime.types" >>>>> -D SERVER_CONFIG_FILE="apache2.conf" >>>>> >>>>> Python: >>>>> Python 3.4.2 >>>>> >>>>> >>>>> Linked libs for mod_wsgi.so: >>>>> >>>>> linux-vdso.so.1 (0x00007ffd436c5000) >>>>> libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 >>>>> (0x00007f79fea6f000) >>>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> (0x00007f79fe852000) >>>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f79fe64e000) >>>>> libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f79fe44b000) >>>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f79fe14a000) >>>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f79fdd9f000) >>>>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f79fdb97000) >>>>> libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f79fd96e000) >>>>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f79fd753000) >>>>> /lib64/ld-linux-x86-64.so.2 (0x00007f79ff2ea000) >>>>> >>>>> >>>>> >>>>> >>>>> Hope this helps! >>>>> Best, >>>>> Stefan >>>>> >>>>> On Sunday, August 7, 2016 at 2:16:31 AM UTC+2, Graham Dumpleton wrote: >>>>> Believe this should all now be fixed on develop branch of mod_wsgi. >>>>> Please test and let me know if is now working as you expect so I can get >>>>> a new release out. >>>>> >>>>> If anyone is able to test off develop with Apache 2.0 and 2.2 (not just >>>>> 2.4), that would be most appreciated. >>>>> >>>>> Graham >>>>> >>>>>> On 6 Aug 2016, at 3:32 PM, Graham Dumpleton <graham.d...@ <>gmail.com >>>>>> <http://gmail.com/>> wrote: >>>>>> >>>>>> Getting closer. Anything output to sys.stdout and sys.stderr from within >>>>>> a request handler will now be routed via wsgi.errors. Your %L format >>>>>> will work. The remote address shows IP correctly, but is not showing the >>>>>> remote port correctly, instead showing 0. >>>>>> >>>>>> Please start testing using develop branch and let me know of any >>>>>> unexpected problems. >>>>>> >>>>>> Graham >>>>>> >>>>>>> On 6 Aug 2016, at 1:59 PM, Graham Dumpleton <graham.d...@ <>gmail.com >>>>>>> <http://gmail.com/>> wrote: >>>>>>> >>>>>>> So I have it sort of working when using stdout/stderr, but there is one >>>>>>> issue which means still need to do some work if can decide what should >>>>>>> be done. >>>>>>> >>>>>>> niaZC/85C0Q AH01628: authorization result: granted (no directives) >>>>>>> niaZC/85C0Q AH01628: authorization result: granted (no directives) >>>>>>> niaZC/85C0Q global message >>>>>>> niaZC/85C0Q request message >>>>>>> >>>>>>> X+fwC/85C0Q AH01628: authorization result: granted (no directives) >>>>>>> X+fwC/85C0Q AH01628: authorization result: granted (no directives) >>>>>>> X+fwC/85C0Q queued messageglobal message >>>>>>> X+fwC/85C0Q request message >>>>>>> >>>>>>> The problem is the ‘queued message’. This is with application of: >>>>>>> >>>>>>> def application(environ, start_response): >>>>>>> print('global message') >>>>>>> print('request message', file=environ['wsgi.errors']) >>>>>>> print('queued message', end=‘') >>>>>>> >>>>>>> The ‘queued message’ is one which there was no newline terminator. >>>>>>> >>>>>>> What normally occurs is that messages get buffered up until a newline >>>>>>> terminator is encountered, which then acts to flush it out and write it >>>>>>> to Apache log API. This is important else if you do something like: >>>>>>> >>>>>>> print(1, 2, 3, 4) >>>>>>> >>>>>>> They will all get printed on separate log lines, which isn’t what you >>>>>>> would expect. >>>>>>> >>>>>>> The log ID only gets pulled in when you thus flush out what is buffered. >>>>>>> >>>>>>> There a couple of problems around this. >>>>>>> >>>>>>> Each of those numbers in that last print() statement get buffered >>>>>>> separately. If another request handler does the same thing, they can >>>>>>> get interleaved, as there is only one stdout. When they get flushed >>>>>>> out, the log ID will be for whatever request sent the newline first. >>>>>>> This data will get attributed against wrong request. >>>>>>> >>>>>>> Similar issue is where a request never sent a new line like above, in >>>>>>> this case it ends up at start of data for next request. >>>>>>> >>>>>>> One solution which should work out reasonably cleanly is that if in a >>>>>>> request, reach back and grab wsgi.errors and divert output via that, >>>>>>> which is per request and is flushed at the end of the request. At least >>>>>>> I think that will work. >>>>>>> >>>>>>> More later. >>>>>>> >>>>>>> Graham >>>>>>> >>>>>>>> On 5 Aug 2016, at 11:34 PM, Graham Dumpleton <graham.d...@ <>gmail.com >>>>>>>> <http://gmail.com/>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On 5 Aug 2016, at 10:18 PM, Graham Dumpleton <graham.d...@ >>>>>>>>> <>gmail.com <http://gmail.com/>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 5 Aug 2016, at 9:54 PM, Stefan Nastic <stefan....@ <>gmail.com >>>>>>>>>> <http://gmail.com/>> wrote: >>>>>>>>>> >>>>>>>>>> Is there a way to configure the logging with mod_wsgi and Django >>>>>>>>>> without losing the request information, such as remote client IP or >>>>>>>>>> unique log ID AND without using environ['wsgi.errors']? Other >>>>>>>>>> solutions such as logging wrapper would be also acceptable... >>>>>>>>>> >>>>>>>>>> Please also see related SO discussions: >>>>>>>>>> http://stackoverflow.com/questions/38786532/logging-with-apache-2-4-mod-wsgi-and-django >>>>>>>>>> >>>>>>>>>> <http://stackoverflow.com/questions/38786532/logging-with-apache-2-4-mod-wsgi-and-django> >>>>>>>>>> http://stackoverflow.com/questions/38767989/apache-2-4-error-log-entries-incomplete >>>>>>>>>> >>>>>>>>>> <http://stackoverflow.com/questions/38767989/apache-2-4-error-log-entries-incomplete> >>>>>>>>> >>>>>>>>> Short answer, no. >>>>>>>>> >>>>>>>>> The two basic problems are detailed in: >>>>>>>>> >>>>>>>>> https://github.com/GrahamDumpleton/mod_wsgi/issues/144 >>>>>>>>> <https://github.com/GrahamDumpleton/mod_wsgi/issues/144> >>>>>>>>> https://github.com/GrahamDumpleton/mod_wsgi/issues/145 >>>>>>>>> <https://github.com/GrahamDumpleton/mod_wsgi/issues/145> >>>>>>>>> >>>>>>>>> but the solution even for the first is far from simple. >>>>>>>>> >>>>>>>>> The problem is that the request and connection log IDs are only >>>>>>>>> generated the first time a message is logged via the Apache log API. >>>>>>>>> In the case of proxying a request from the Apache child worker >>>>>>>>> process to a daemon mode process, there wouldn’t have been any >>>>>>>>> messages logged and so nothing has triggered the generation of the >>>>>>>>> log IDs. This means they aren’t available to transfer across to the >>>>>>>>> daemon process such that they could be reconstructed into the >>>>>>>>> connection and request records to fake up things so that logging >>>>>>>>> works in daemon mode. A solution may be for mod_wsgi to forcibly >>>>>>>>> cause the generation of the log IDs on every request using an Apache >>>>>>>>> API call, even though they may not be required. The implications of >>>>>>>>> doing this need to be looked at. Alternatively, one works out what >>>>>>>>> seed information from a request is used to generate the log ID so can >>>>>>>>> ensure that is being passed across and added to the fake connection >>>>>>>>> and request objects that the logging will eventually use. >>>>>>>> >>>>>>>> Copying across underlying seed information isn’t possible because part >>>>>>>> of the calculation involves turning a C pointer to a data structure >>>>>>>> for the thread into an integer. One can’t set the C pointer to same >>>>>>>> value of daemon process side because if something then tries to access >>>>>>>> via it, process will crash. >>>>>>>> >>>>>>>> It is possible to force the generation of the connection and request >>>>>>>> IDs though if not already set and pass those across. Luckily is not so >>>>>>>> hard. >>>>>>>> >>>>>>>> The develop branch of the repo now has changes which will do that at >>>>>>>> least, with the connection and request ID available in the per request >>>>>>>> WSGI environ dictionary as mod_wsgi.connection_id and >>>>>>>> mod_wsgi.request_id. Any messages logged via wsgi.errors will now show >>>>>>>> the correct log ID. >>>>>>>> >>>>>>>>> You also keep saying remote client IP doesn’t show. On my testing on >>>>>>>>> MacOS X it does and the client IP is transferred across to daemon >>>>>>>>> mode, so not sure what the issue is there as cannot replicate it at >>>>>>>>> this point. >>>>>>>>> >>>>>>>>> So trying to address even the log ID issues for wsgi.errors is going >>>>>>>>> to take some time and work. Linking messages to stdout/stderr back to >>>>>>>>> requests is going to be even more complicated. >>>>>>>> >>>>>>>> >>>>>>>> The stdout/stderr issues needs more investigation. One thing in our >>>>>>>> favour is that I had already added a thread local for tracking request >>>>>>>> state due to metrics collection. I don’t know that that gives access >>>>>>>> to the raw Apache request object though which is needed, so would >>>>>>>> likely need extending. >>>>>>>> >>>>>>>> If you really need something though, you can try caching those IDs >>>>>>>> from the per request WSGI environ dictionary and using those in >>>>>>>> messages somehow. >>>>>>>> >>>>>>>> Graham >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <mailto:[email protected]>. >>>>> To post to this group, send email to [email protected] >>>>> <mailto:[email protected]>. >>>>> Visit this group at https://groups.google.com/group/modwsgi >>>>> <https://groups.google.com/group/modwsgi>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "modwsgi" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe >>>> <https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] >>>> <mailto:[email protected]>. >>>> To post to this group, send email to [email protected] >>>> <mailto:[email protected]>. >>>> Visit this group at https://groups.google.com/group/modwsgi >>>> <https://groups.google.com/group/modwsgi>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "modwsgi" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe >> <https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
