New set of changes on ‘develop’ branch of mod_wsgi repo.
Try again with latest:
• https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip
or:
•
https://codeload.github.com/GrahamDumpleton/mod_wsgi/tar.gz/develop#mod_wsgi-4.5.6
<https://codeload.github.com/GrahamDumpleton/mod_wsgi/tar.gz/develop#mod_wsgi-4.5.6>
Thanks.
Graham
> On 15 Aug 2016, at 11:40 PM, Graham Dumpleton <[email protected]>
> wrote:
>
> Giving up for the night. Getting memory corruption all over the place now and
> can’t even get sane stack traces from gdb. Nothing making sense. Will start
> over tomorrow.
>
>> On 15 Aug 2016, at 11:30 PM, Stefan Nastic <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> tried again and no luck ....
>>
>> Just to remind you that this happens immediately upon the firs request after
>> restarting the apache, so if you are referring to gc below, I don't see how
>> it can kick in and cause a problem. Just a thought ....
>>
>> Stefan
>>
>> On 15-Aug-16 15:16, Graham Dumpleton wrote:
>>> I am not confident that next change will solve the problem, but can you try
>>> pulling down source code again and try.
>>>
>>> I can’t recreate it so far after this change, but then that is what
>>> happened with the first change and it only started occurring again after
>>> you and someone else said it didn’t solve problem. Very confusing.
>>>
>>> The problem seems to relate to strange things going on when Python objects
>>> are being destroyed. Python 3 appears to behave differently with the new
>>> streams implementation.
>>>
>>> Graham
>>>
>>>> On 15 Aug 2016, at 10:51 PM, Stefan Nastic <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have just tested it (4.5.5) and am still getting the same error.
>>>>
>>>> Best,
>>>>
>>>> Stefan
>>>>
>>>> On 15-Aug-16 14:46, Graham Dumpleton wrote:
>>>>> Problem only affects Python 3.X. Verification of fix being worked on
>>>>> right now.
>>>>>
>>>>> If you are able to download:
>>>>>
>>>>> https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip
>>>>> <https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip>
>>>>>
>>>>> and try with that it would help immensely.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Graham
>>>>>
>>>>>> On 15 Aug 2016, at 10:43 PM, Stefan Nastic <[email protected]
>>>>>> <mailto:[email protected]>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> sorry for a bit delayed response, but I was traveling the last week and
>>>>>> had a very limited internet connectivity.
>>>>>>
>>>>>> Now back to the issue. I tested the version 4.5.4 and in short there is
>>>>>> some problem as I get seg fault in the apache. I have checked 4.5.4.
>>>>>> for the most common issues such as loading mod_python, linking against
>>>>>> shared libraries, etc. and there everything seam to be fine. I have
>>>>>> also tested the same compilation and installation procedure with 4.5.3
>>>>>> and it works fine.
>>>>>>
>>>>>> So I suspect that this is a codding error, most probably accessing the
>>>>>> protected memory region when you try to access/reproduce Log ID seed.
>>>>>> However, this is only my assumption since I have not had a chance to
>>>>>> look at your code.
>>>>>>
>>>>>> For debugging purposes ....
>>>>>>
>>>>>> Apache error log:
>>>>>>
>>>>>> [mpm_event:notice] [pid 25563:tid 140401132873600] AH00494: SIGHUP
>>>>>> received. Attempting to restart
>>>>>> [mpm_event:notice] [pid 25563:tid 140401132873600] AH00489:
>>>>>> Apache/2.4.10 (Debian) mod_wsgi/4.5.4 Python/3.4.2 configured --
>>>>>> resuming normal operations
>>>>>> [core:notice] [pid 25563:tid 140401132873600] AH00094: Command line:
>>>>>> '/usr/sbin/apache2'
>>>>>> [core:notice] [pid 25563:tid 140401132873600] AH00052: child pid 27051
>>>>>> exit signal Segmentation fault (11)
>>>>>>
>>>>>> OS: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4
>>>>>> (2016-02-29) x86_64 GNU/Linux
>>>>>>
>>>>>> APACHE:
>>>>>> Server version: Apache/2.4.10 (Debian)
>>>>>> Server built: Nov 28 2015 14:05:48
>>>>>> Server's Module Magic Number:
>>>>>> Server loaded: APR 1.5.1, APR-UTIL 1.5.4
>>>>>> Compiled using: APR 1.5.1, APR-UTIL 1.5.4
>>>>>> Architecture: 64-bit
>>>>>> Server MPM: event
>>>>>> threaded: yes (fixed thread count)
>>>>>> forked: yes (variable process count)
>>>>>> Server compiled with....
>>>>>> -D APR_HAS_SENDFILE
>>>>>> -D APR_HAS_MMAP
>>>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>>>> -D APR_USE_SYSVSEM_SERIALIZE
>>>>>> -D APR_USE_PTHREAD_SERIALIZE
>>>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>>>> -D APR_HAS_OTHER_CHILD
>>>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>>>> -D DYNAMIC_MODULE_LIMIT=256
>>>>>> -D HTTPD_ROOT="/etc/apache2"
>>>>>> -D SUEXEC_BIN="/usr/lib/apache2/suexec"
>>>>>> -D DEFAULT_PIDLOG="/var/run/apache2.pid"
>>>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>>>> -D DEFAULT_ERRORLOG="logs/error_log"
>>>>>> -D AP_TYPES_CONFIG_FILE="mime.types"
>>>>>> -D SERVER_CONFIG_FILE="apache2.conf"
>>>>>>
>>>>>> Python:
>>>>>> Python 3.4.2
>>>>>>
>>>>>>
>>>>>> Linked libs for mod_wsgi.so:
>>>>>>
>>>>>> linux-vdso.so.1 (0x00007ffd436c5000)
>>>>>> libpython3.4m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
>>>>>> (0x00007f79fea6f000)
>>>>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> (0x00007f79fe852000)
>>>>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f79fe64e000)
>>>>>> libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f79fe44b000)
>>>>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f79fe14a000)
>>>>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f79fdd9f000)
>>>>>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f79fdb97000)
>>>>>> libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f79fd96e000)
>>>>>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f79fd753000)
>>>>>> /lib64/ld-linux-x86-64.so.2 (0x00007f79ff2ea000)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hope this helps!
>>>>>> Best,
>>>>>> Stefan
>>>>>>
>>>>>> On Sunday, August 7, 2016 at 2:16:31 AM UTC+2, Graham Dumpleton wrote:
>>>>>> Believe this should all now be fixed on develop branch of mod_wsgi.
>>>>>> Please test and let me know if is now working as you expect so I can get
>>>>>> a new release out.
>>>>>>
>>>>>> If anyone is able to test off develop with Apache 2.0 and 2.2 (not just
>>>>>> 2.4), that would be most appreciated.
>>>>>>
>>>>>> Graham
>>>>>>
>>>>>>> On 6 Aug 2016, at 3:32 PM, Graham Dumpleton <graham.d...@ <>gmail.com
>>>>>>> <http://gmail.com/>> wrote:
>>>>>>>
>>>>>>> Getting closer. Anything output to sys.stdout and sys.stderr from
>>>>>>> within a request handler will now be routed via wsgi.errors. Your %L
>>>>>>> format will work. The remote address shows IP correctly, but is not
>>>>>>> showing the remote port correctly, instead showing 0.
>>>>>>>
>>>>>>> Please start testing using develop branch and let me know of any
>>>>>>> unexpected problems.
>>>>>>>
>>>>>>> Graham
>>>>>>>
>>>>>>>> On 6 Aug 2016, at 1:59 PM, Graham Dumpleton <graham.d...@ <>gmail.com
>>>>>>>> <http://gmail.com/>> wrote:
>>>>>>>>
>>>>>>>> So I have it sort of working when using stdout/stderr, but there is
>>>>>>>> one issue which means still need to do some work if can decide what
>>>>>>>> should be done.
>>>>>>>>
>>>>>>>> niaZC/85C0Q AH01628: authorization result: granted (no directives)
>>>>>>>> niaZC/85C0Q AH01628: authorization result: granted (no directives)
>>>>>>>> niaZC/85C0Q global message
>>>>>>>> niaZC/85C0Q request message
>>>>>>>>
>>>>>>>> X+fwC/85C0Q AH01628: authorization result: granted (no directives)
>>>>>>>> X+fwC/85C0Q AH01628: authorization result: granted (no directives)
>>>>>>>> X+fwC/85C0Q queued messageglobal message
>>>>>>>> X+fwC/85C0Q request message
>>>>>>>>
>>>>>>>> The problem is the ‘queued message’. This is with application of:
>>>>>>>>
>>>>>>>> def application(environ, start_response):
>>>>>>>> print('global message')
>>>>>>>> print('request message', file=environ['wsgi.errors'])
>>>>>>>> print('queued message', end=‘')
>>>>>>>>
>>>>>>>> The ‘queued message’ is one which there was no newline terminator.
>>>>>>>>
>>>>>>>> What normally occurs is that messages get buffered up until a newline
>>>>>>>> terminator is encountered, which then acts to flush it out and write
>>>>>>>> it to Apache log API. This is important else if you do something like:
>>>>>>>>
>>>>>>>> print(1, 2, 3, 4)
>>>>>>>>
>>>>>>>> They will all get printed on separate log lines, which isn’t what you
>>>>>>>> would expect.
>>>>>>>>
>>>>>>>> The log ID only gets pulled in when you thus flush out what is
>>>>>>>> buffered.
>>>>>>>>
>>>>>>>> There a couple of problems around this.
>>>>>>>>
>>>>>>>> Each of those numbers in that last print() statement get buffered
>>>>>>>> separately. If another request handler does the same thing, they can
>>>>>>>> get interleaved, as there is only one stdout. When they get flushed
>>>>>>>> out, the log ID will be for whatever request sent the newline first.
>>>>>>>> This data will get attributed against wrong request.
>>>>>>>>
>>>>>>>> Similar issue is where a request never sent a new line like above, in
>>>>>>>> this case it ends up at start of data for next request.
>>>>>>>>
>>>>>>>> One solution which should work out reasonably cleanly is that if in a
>>>>>>>> request, reach back and grab wsgi.errors and divert output via that,
>>>>>>>> which is per request and is flushed at the end of the request.
>>>>>>>> At least I think
>>>>>>>> that will work.
>>>>>>>>
>>>>>>>> More later.
>>>>>>>>
>>>>>>>> Graham
>>>>>>>>
>>>>>>>>> On 5 Aug 2016, at 11:34 PM, Graham Dumpleton <graham.d...@
>>>>>>>>> <>gmail.com <http://gmail.com/>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 5 Aug 2016, at 10:18 PM, Graham Dumpleton <graham.d...@
>>>>>>>>>> <>gmail.com <http://gmail.com/>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 5 Aug 2016, at 9:54 PM, Stefan Nastic <stefan....@ <>gmail.com
>>>>>>>>>>> <http://gmail.com/>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Is there a way to configure the logging with mod_wsgi and Django
>>>>>>>>>>> without losing the request information, such as remote client IP or
>>>>>>>>>>> unique log ID AND without using environ['wsgi.errors']? Other
>>>>>>>>>>> solutions such as logging wrapper would be also acceptable...
>>>>>>>>>>>
>>>>>>>>>>> Please also see related SO discussions:
>>>>>>>>>>> http://stackoverflow.com/questions/38786532/logging-with-apache-2-4-mod-wsgi-and-django
>>>>>>>>>>>
>>>>>>>>>>> <http://stackoverflow.com/questions/38786532/logging-with-apache-2-4-mod-wsgi-and-django>
>>>>>>>>>>> http://stackoverflow.com/questions/38767989/apache-2-4-error-log-entries-incomplete
>>>>>>>>>>>
>>>>>>>>>>> <http://stackoverflow.com/questions/38767989/apache-2-4-error-log-entries-incomplete>
>>>>>>>>>>
>>>>>>>>>> Short answer, no.
>>>>>>>>>>
>>>>>>>>>> The two basic problems are detailed in:
>>>>>>>>>>
>>>>>>>>>> https://github.com/GrahamDumpleton/mod_wsgi/issues/144
>>>>>>>>>> <https://github.com/GrahamDumpleton/mod_wsgi/issues/144>
>>>>>>>>>> https://github.com/GrahamDumpleton/mod_wsgi/issues/145
>>>>>>>>>> <https://github.com/GrahamDumpleton/mod_wsgi/issues/145>
>>>>>>>>>>
>>>>>>>>>> but the solution even for the first is far from simple.
>>>>>>>>>>
>>>>>>>>>> The problem is that the request and connection log IDs are only
>>>>>>>>>> generated the first time a message is logged via the Apache log API.
>>>>>>>>>> In the case of proxying a request from the Apache child worker
>>>>>>>>>> process to a daemon mode process, there wouldn’t have been any
>>>>>>>>>> messages logged and so nothing has triggered the generation of the
>>>>>>>>>> log IDs. This means they aren’t available to transfer across to the
>>>>>>>>>> daemon process such that they could be reconstructed into the
>>>>>>>>>> connection and request records to fake up things so that logging
>>>>>>>>>> works in daemon mode. A solution may be for mod_wsgi to forcibly
>>>>>>>>>> cause the generation of the log IDs on every request using an Apache
>>>>>>>>>> API call, even though they may not be required. The implications of
>>>>>>>>>> doing this need to be looked at. Alternatively, one works out what
>>>>>>>>>> seed information from a request is used to generate the log ID so
>>>>>>>>>> can ensure that is being passed across and added to the fake
>>>>>>>>>> connection and request objects that the logging will eventually use.
>>>>>>>>>
>>>>>>>>> Copying across underlying seed information isn’t possible because
>>>>>>>>> part of the calculation involves turning a C pointer to a data
>>>>>>>>> structure for the thread into an integer. One can’t set the C pointer
>>>>>>>>> to same value of daemon process side because if something then tries
>>>>>>>>> to access via it, process will crash.
>>>>>>>>>
>>>>>>>>> It is possible to force the generation of the connection and request
>>>>>>>>> IDs though if not already set and pass those across. Luckily is not
>>>>>>>>> so hard.
>>>>>>>>>
>>>>>>>>> The develop branch of the repo now has changes which will do that at
>>>>>>>>> least, with the connection and request ID available in the per
>>>>>>>>> request WSGI environ dictionary as mod_wsgi.connection_id and
>>>>>>>>> mod_wsgi.request_id. Any messages logged via wsgi.errors will now
>>>>>>>>> show the correct log ID.
>>>>>>>>>
>>>>>>>>>> You also keep saying remote client IP doesn’t show. On my testing on
>>>>>>>>>> MacOS X it does and the client IP is transferred across to daemon
>>>>>>>>>> mode, so not sure what the issue is there as cannot replicate it at
>>>>>>>>>> this point.
>>>>>>>>>>
>>>>>>>>>> So trying to address even the log ID issues for wsgi.errors is going
>>>>>>>>>> to take some time and work. Linking messages to stdout/stderr back
>>>>>>>>>> to requests is going to be even more complicated.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The stdout/stderr issues needs more investigation. One thing in our
>>>>>>>>> favour is that I had already added a thread local for tracking
>>>>>>>>> request state due to metrics collection. I don’t know that that gives
>>>>>>>>> access to the raw Apache request object though which is needed, so
>>>>>>>>> would likely need extending.
>>>>>>>>>
>>>>>>>>> If you really need something though, you can try caching those IDs
>>>>>>>>> from the per request WSGI environ dictionary and using those in
>>>>>>>>> messages somehow.
>>>>>>>>>
>>>>>>>>> Graham
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "modwsgi" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to [email protected]
>>>>>> <mailto:[email protected]>.
>>>>>> To post to this group, send email to [email protected]
>>>>>> <mailto:[email protected]>.
>>>>>> Visit this group at https://groups.google.com/group/modwsgi
>>>>>> <https://groups.google.com/group/modwsgi>.
>>>>>> For more options, visit https://groups.google.com/d/optout
>>>>>> <https://groups.google.com/d/optout>.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "modwsgi" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe
>>>>> <https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe>.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> [email protected]
>>>>> <mailto:[email protected]>.
>>>>> To post to this group, send email to [email protected]
>>>>> <mailto:[email protected]>.
>>>>> Visit this group at https://groups.google.com/group/modwsgi
>>>>> <https://groups.google.com/group/modwsgi>.
>>>>> For more options, visit https://groups.google.com/d/optout
>>>>> <https://groups.google.com/d/optout>.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected]
>>>> <mailto:[email protected]>.
>>>> To post to this group, send email to [email protected]
>>>> <mailto:[email protected]>.
>>>> Visit this group at https://groups.google.com/group/modwsgi
>>>> <https://groups.google.com/group/modwsgi>.
>>>> For more options, visit https://groups.google.com/d/optout
>>>> <https://groups.google.com/d/optout>.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "modwsgi" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe
>>> <https://groups.google.com/d/topic/modwsgi/6I2jiEQomh8/unsubscribe>.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected]
>>> <mailto:[email protected]>.
>>> To post to this group, send email to [email protected]
>>> <mailto:[email protected]>.
>>> Visit this group at https://groups.google.com/group/modwsgi
>>> <https://groups.google.com/group/modwsgi>.
>>> For more options, visit https://groups.google.com/d/optout
>>> <https://groups.google.com/d/optout>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <mailto:[email protected]>.
>> To post to this group, send email to [email protected]
>> <mailto:[email protected]>.
>> Visit this group at https://groups.google.com/group/modwsgi
>> <https://groups.google.com/group/modwsgi>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
