OK, thanks. I was kind of afraid of that but found it hard to accept that going into the third decade of the 21st century, a major software project like Apache would not be able to handle unicode in its log files.
On Tuesday, September 3, 2019 at 11:53:04 PM UTC-6, Graham Dumpleton wrote: > > The string will be getting output correctly from Python. The problem is > that the Apache logging system when passed any strings which contain > characters outside of the ASCII character set, will convert them to encoded > byte values. So that string is the escaped UTF-8 encoded value of the byte > string. > > If you need the original original strings, you should use the Python > logging module and set it up to save log output to a separate file. > > On 4 Sep 2019, at 3:14 pm, '[email protected] <javascript:>' via modwsgi < > [email protected] <javascript:>> wrote: > > Unable to write unicode log messages > > I have a Flask app running with Apache and modwsgi with uses the Python > logging module to write log messages to stderr. These messages sometimes > contain non-ascii text (Japanese specifically). When run with the Flask > development server (no Apache, no modwsgi) the log messages are fine, eg: > ERROR:root:test message: 日本語 > > When run under Apache/modwsgi, the message appearing in the Apache log > file is escaped bytes string: > ERROR:root:test message: \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e > > I am using Ubuntu-18.04, apache2-2.4.29, libapache2-mod-wsgi-py3-4.5.17, > python3-3.6.7, flask-0.12.2, all installed from the usual Ubuntu > repositories. > > What I have done: > - Lots of googling. Similar problems seem to be old (eg python-2), > involve unicode encode or decode exceptions (not a problem in my > case) and the suggested solutions I tried don't fix my problem. > - Added following settings to the WSGIDaemonProcess line in the > mod_wsgi config file: > locale=en_US.UTF-8 lang=en_US.UTF-8 > - In /etc/apache2/envvars, uncommented line as directed: > ## Uncomment the following line to use the system default locale > instead: > . /etc/default/locale > - In /etc/apache2/envvars, added below the two lines above the lines: > export LANG='en_US.UTF-8' > export LC_ALL='en_US.UTF-8' > export PYTHONIOENCODING=UTF-8 > - Printed out following values in the python app code just before > the problem logging line: > sys.filesystemencoding = utf-8 > sys.defaultencoding = utf-8 > type(sys.stderr) = <class '_io.TextIOWrapper'>, encoding = utf-8 > locale = ('en_US', 'UTF-8') > - In the app, wrote the log message line directly to sys.stderr > without going through the logging module. The results were > still escaped bytes rather then the correct encoded unicode text. > > None of the above made any difference. What now??? > > Thanks for any advice! > > Addendum: here is a minimal test program. > > ======== test.py ======== > import sys, logging > from flask import Flask > App = Flask(__name__) > L = logging.getLogger > logging.basicConfig() > > @App.route('/') > def hello_world(): > L().error("test message: 日本語") > return 'Hello, World!' > > if __name__ == '__main__': > App.run (debug=True) > > ======== test.wsgi ======== > import sys, os > appdir, _ = os.path.split (os.path.abspath(__file__)) > sys.path.insert (0, appdir) > from test import App as application > > ======== test.conf ======== > WSGIDaemonProcess jmtest processes=2 threads=4 \ > display-name=apache2-jmtest \ > locale=en_US.UTF-8 lang=en_US.UTF-8 > WSGIProcessGroup jmtest > > WSGIScriptAlias /jmtest /home/.../jmtest/test.wsgi > <Directory /home/.../jmtest> > Require all granted > </Directory> > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/538fb660-7145-49d0-89da-8eaf7e1f694a%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/538fb660-7145-49d0-89da-8eaf7e1f694a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/1d2a375a-d191-4e73-b59a-6f6a106080be%40googlegroups.com.
