Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All,
The solution / explanation follows.
Thanks to Graham Dumpleton, the author of mod_wsgi (the WSGI module for
Apache2) the source of the problem could be traced back to variables in
Apache2. Below are the details reproduced from
https://groups.google.com/forum/#!topic/modwsgi/4wdfCOnMUkU
Now, everything is indeed UTF-8 !
Thanks again to each and everyone.
Best regards,
René
Solution
There is a file /etc/apache2/envvars referred to by /etc/apache2/apache2.conf.
In that file, I found the following lines:
## The locale used by some modules like mod_dav
export LANG=C
## Uncomment the following line to use the system default locale instead:
#. /etc/default/locale
As I don't need mod_dav, neither is it compiled with Apache2 ($apache2ctl -l),
neither is it loaded with Apache2 ($apache2ctl -M), I commented / uncommented
the 2 lines so that it now looks like:
#export LANG=C
. /etc/default/locale
export LANG
After a stop/start of Apache2, everything works fine and when I put the code:
from locale import getpreferredencoding
prefcoding = getpreferredencoding()
from os import environ
lang = environ[LANG]
g = open('envresults', 'a')
g.write('LANG: ' + lang + '\n')
g.write('PrefCod: ' + prefcoding + '\n')
in my WSGI application, it gives me the same as the interpreter:
rse@Alibaba:~/test$ cat envresults
LANG: en_US.UTF-8
PrefCod: UTF-8
rse@Alibaba:~/test$
-*- The End -*-
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Chris Angelico ros...@gmail.com: Apache itself most likely is running with LANG=C and other environmental changes. It's not a Python-specific thing. The topic is discussed also at: URL: http://blog.dscpl.com.au/2014/0 9/setting-lang-and-lcall-when-using.html. Personally, I think the C locale is the only safe choice for Apache. CGI shell scripts, in particular, can act in suprising ways when the locale is changed. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Wednesday, August 26, 2015 at 5:59:12 PM UTC+2, Chris Angelico wrote:
On Thu, Aug 27, 2015 at 1:02 AM, RAH wrote:
7) Now, that source of the problem is known, I must investigate why my
run-time environment differs from the interpreter environment. I know it is
the same machine, same Python 3.4.0. As the mod_wsgi module in Apache2
initiates Python for the run-time, I will look there around.
First place to look would be the environment. If os.environ[LANG]
has C when you run under Apache, that would be the explanation. But
explicitly choosing the encoding is the best way for other reasons
anyway, and it solves the problem, so researching this is merely a
matter of curiosity.
ChrisA
Hello Chris,
Thanks for your further input.
os.environ{LANG} returns `en_US.UTF-8`, exactly the same as asking in Bash
`echo $LANG`. But again this is the interpreter.
Now if I ask the same os.environ[LANG] within my application, it returns `C`
So, again, there is a marked difference between what the interpreter shows and
what the run-time shows.
In the meantime, I have checked the configuration directives of mod_wsgi. There
is nothing there to set or choose a particular environment. And I wonder if
there is a reason to check the Apache2 directives, like SetEnv ? Indeed Apache2
doesn't know anything about Python.
On the other hand the server I use (Ubuntu) still has Python 2.7 installed and
I can't remove it with apt-get. I believe Ubuntu needs python a lot. The Python
3.4 has been installed separately and it could be that a doubtful configuration
subsists.
Nevertheless I can't figure out why calling Python in the shell (interactive
mode) or letting mod_wsgi start the same Python provide two different
environments. I guess I must investigate that part because mod_wsgi gets to
Python in what I would call 'auto-discovery' mode. Obviously it gets the same
version 3.4.0 but maybe it picks up some 2.7 configuration files because the
installation of 3.4 next to 2.7 might not be perfect. I'll look at it.
Thank you Chris. If I find something I'll post it here.
René
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On 26/08/15 04:19, RAH wrote: UnicodeEncodeError: 'ascii' codec can't encode character '\\xc7' in position 15: ordinal not in range(128) (Hi all, this is my first post to the list) This can be a frustrating issue to resolve, but your issue might be solved with this environment variable: PYTHONIOENCODING=UTF-8 Regards, pd -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Thu, Aug 27, 2015 at 1:02 AM, RAH rene.heym...@gmail.com wrote: 7) Now, that source of the problem is known, I must investigate why my run-time environment differs from the interpreter environment. I know it is the same machine, same Python 3.4.0. As the mod_wsgi module in Apache2 initiates Python for the run-time, I will look there around. First place to look would be the environment. If os.environ[LANG] has C when you run under Apache, that would be the explanation. But explicitly choosing the encoding is the best way for other reasons anyway, and it solves the problem, so researching this is merely a matter of curiosity. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Wed, Aug 26, 2015 at 7:12 PM, RAH rene.heym...@gmail.com wrote: By the way my whole installation is defaulted to UTF-8: HTML:headmeta charset=utf-8/head Javascript: script type=text/javascript charset=UTF-8 PuTTY: TranslationCharacter set translationRemote character setUTF-8 Python code: # -*- coding: utf-8 -*- Linux: $ echo $LANG --- en_US.UTF-8 Check that from actually inside your web script - os.environ[LANG] should be en_US.UTF-8. If it isn't, then that may be where your difference between web and interactive is, and so you'll need to be more explicit about encodings. It's common for background processes to have restricted environments, sometimes with LANG=C or similar. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Chris, I can confirm it is Python 3. Here is the line from the Apache2 log: [Wed Aug 26 10:28:01.508194 2015] [mpm_worker:notice] [pid 1120:tid 3074398848] AH00292: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/4.4.13 Python/3.4.0 configured -- resuming normal operations As a matter of fact, I connect to the same machine that runs Apache2/mod_wsgi/Python via PuTTY and when I open the Python interpreter it responds: Python 3.4.0 (default, Jun 19 2015, 14:18:46) [GCC 4.8.2] on linux Type help, copyright, credits or license for more information. Hence exactly the same Python 3.4.0 version. By the way my whole installation is defaulted to UTF-8: HTML:headmeta charset=utf-8/head Javascript: script type=text/javascript charset=UTF-8 PuTTY: TranslationCharacter set translationRemote character setUTF-8 Python code: # -*- coding: utf-8 -*- Linux: $ echo $LANG --- en_US.UTF-8 Finally, I also checked the coding of the `request_body` as written in the binary file (`logbytes`) and `Ç` is indeed coded as C387 (hex) or `é` is correctly written as C3A9 (hex). Thank you Chris, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Chris, Thank you. I got the answer (at least a partial one) that I will share in a while. I will first respond to the other posts I received to thank each and everyone. Please stay tuned. René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Thank you Pete. Indeed it has to do with choice of encoding. I'll be back in a short while with more details. Cheers, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear Dieter, Indeed there is a difference. I will share my discoveries in a while after I respond to each one. Be in touch soon. Thanks. René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Thank you Chris. I'll share my findings in a moment. Please bear with me a bit more time. Cheers, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All, First, thanks to each and everyone. There is indeed a solution by I haven't yet found the root of the problem (I'll come back to that at the end of my post). 1) After many trials and errors, I found that the problem was with the write() function in `h.write(str(d) + '\n')` and not with the argument itself which is a perfect string. 2) Reading the documentation it refers to the open() function and its preferred encoding. 3) I checked with the interpreter and got: rse@Alibaba:~/test$ python Python 3.4.0 (default, Jun 19 2015, 14:18:46) [GCC 4.8.2] on linux Type help, copyright, credits or license for more information. from locale import * getpreferredencoding() 'UTF-8' 4) I knew everything was set up with UTF-8 (refer my first answer to Chris K.) so I couldn't believe it ! Another dead end ? 5) I had to make sure it was the same within the application, so I added a couple of statements to get and record the preferred encoding. And lo and behold I got his: rse@Alibaba:~/test$ cat type ANSI_X3.4-1968class 'bytes' class 'str' class 'dict' rse@Alibaba:~/test$ So, here the getpreferredencoding() function returns ANSI_X3.4-1968 instead of UTF-8 !? 6) The solution is then obvious: open the file by specifying the encoding; a suggestion made already by Chris A. 7) Now, that source of the problem is known, I must investigate why my run-time environment differs from the interpreter environment. I know it is the same machine, same Python 3.4.0. As the mod_wsgi module in Apache2 initiates Python for the run-time, I will look there around. Dear All, Thank you each and everyone for your contribution. I would suggest to close this subject. If I get a solution around mod_wsgi + python I will post it. Kind regards to all, René -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Thu, Aug 27, 2015 at 5:23 AM, RAH rene.heym...@gmail.com wrote: Nevertheless I can't figure out why calling Python in the shell (interactive mode) or letting mod_wsgi start the same Python provide two different environments. I guess I must investigate that part because mod_wsgi gets to Python in what I would call 'auto-discovery' mode. Obviously it gets the same version 3.4.0 but maybe it picks up some 2.7 configuration files because the installation of 3.4 next to 2.7 might not be perfect. I'll look at it. Apache itself most likely is running with LANG=C and other environmental changes. It's not a Python-specific thing. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
RAH rene.heym...@gmail.com writes:
I experienced an incomprehensible behavior (I've spent already many hours on
this subject): the `file.write('string')` provides an error in run mode and
not when interpreted at the console.
Maybe, I can explain the behavior: the interactive interpreter uses magic
to determine the console's encoding and automatically uses this
for console output. No such magic in non-interactive interpreter use.
Therefore, you can get UnicodeEncoding problems in non-interactive use
which you do not see in interactive use.
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Wed, Aug 26, 2015 at 7:19 AM, RAH rene.heym...@gmail.com wrote:
rb = request_body.decode() # string
I'd recommend avoiding this operation in Python 2. As of Python 3,
omitting the encoding means UTF-8, but in Python 2 it means use the
default encoding, and that often causes problems in scripts that run
in various background environments. Instead, explicitly say what
encoding you're using:
rb = request_body.decode(UTF-8)
Conversely, if you're using Python 3 for this, the default encoding is
coming from this line:
h = open('logdict', 'a')
Again, if you want this to be a text file with a specific encoding, say so:
h = open('logdict', 'a', encoding='UTF-8')
Give that a try and see if your problems disappear. If not, this
should at least poke them with a pointy stick.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
file.write() of non-ASCII characters differs in Interpreted Python than in script run
Dear All,
I experienced an incomprehensible behavior (I've spent already many hours on
this subject): the `file.write('string')` provides an error in run mode and not
when interpreted at the console. The string must contain non-ASCII characters.
If all ASCII, there is no error.
The following example shows what I can see. I must overlook something because I
cannot think Python makes a difference between interpreted and run modes and
yet ... Can someone please check that subject.
Thank you in advance.
René
Code extract from WSGI application (reply.py)
=
request_body = environ['wsgi.input'].read(request_body_size)# bytes
rb = request_body.decode() # string
d = parse_qs(rb)# dict
f = open('logbytes', 'ab')
g = open('logstr', 'a')
h = open('logdict', 'a')
f.write(request_body)
g.write(str(type(request_body)) + '\t' + str(type(rb)) + '\t' +
str(type(d)) + '\n')
h.write(str(d) + '\n') --- line 28 of the application
h.close()
g.close()
f.close()
Tail of Apache2 error.log
=
[Tue Aug 25 20:24:04.657933 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] File reply.py, line 28, in application
[Tue Aug 25 20:24:04.658001 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] h.write(str(d) + '\\n')
[Tue Aug 25 20:24:04.658201 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] UnicodeEncodeError: 'ascii' codec can't encode
character '\\xc7' in position 15: ordinal not in range(128)
Checking what has been logged
=
rse@Alibaba:~/test$ cat logbytes
userName=Ça va ! --- this was indeed the input (notice the
french C + cedilla)
Unicode U+00C7ALT-0199UTF-8 C387
Reading the logbytes file one can verify
that Ç is indeed represented by the 2 bytes
\xC3 and \x87
rse@Alibaba:~/test$ cat logstr
class 'bytes'class 'str'class 'dict'
rse@Alibaba:~/test$ cat logdict
rse@Alibaba:~/test$ --- Obviously empty because of error
Trying similar code within the Python interpreter
=
rse@Alibaba:~/test$ python
Python 3.4.0 (default, Jun 19 2015, 14:18:46)
[GCC 4.8.2] on linux
Type help, copyright, credits or license for more information.
di = {'userName': ['Ça va !']}--- A dictionary
str(di)
{'userName': ['Ça va !']} --- and its string representation
type(str(di))
class 'str' --- Is a string indeed
fi = open('essai', 'a')
fi.write(str(di) + '\n')
26 --- It works well
fi.close()
Checking what has been written
==
rse@Alibaba:~/test$ cat essai
{'userName': ['Ça va !']} --- The result is correct
rse@Alibaba:~/test$
No error if all ASCII
=
If the input is `userName=Rene` for instance then there is no error and the
`logdict' does indeed then contain the text of the dictionary
`{'userName': ['Rene']}`
--
https://mail.python.org/mailman/listinfo/python-list
Re: file.write() of non-ASCII characters differs in Interpreted Python than in script run
On Tue, Aug 25, 2015 at 2:19 PM, RAH rene.heym...@gmail.com wrote:
Dear All,
I experienced an incomprehensible behavior (I've spent already many hours on
this subject): the `file.write('string')` provides an error in run mode and
not when interpreted at the console. The string must contain non-ASCII
characters. If all ASCII, there is no error.
The following example shows what I can see. I must overlook something because
I cannot think Python makes a difference between interpreted and run modes
and yet ... Can someone please check that subject.
Thank you in advance.
René
Code extract from WSGI application (reply.py)
=
request_body = environ['wsgi.input'].read(request_body_size)# bytes
rb = request_body.decode() # string
d = parse_qs(rb)# dict
f = open('logbytes', 'ab')
g = open('logstr', 'a')
h = open('logdict', 'a')
f.write(request_body)
g.write(str(type(request_body)) + '\t' + str(type(rb)) + '\t' +
str(type(d)) + '\n')
h.write(str(d) + '\n') --- line 28 of the application
h.close()
g.close()
f.close()
Tail of Apache2 error.log
=
[Tue Aug 25 20:24:04.657933 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] File reply.py, line 28, in application
[Tue Aug 25 20:24:04.658001 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] h.write(str(d) + '\\n')
[Tue Aug 25 20:24:04.658201 2015] [wsgi:error] [pid 3677:tid 3029764928]
[remote 192.168.1.5:27575] UnicodeEncodeError: 'ascii' codec can't encode
character '\\xc7' in position 15: ordinal not in range(128)
What version of Python is Apache2 using? From the looks of the error,
it is probably using some version of Python2, in which case you'll
need to manually encode the string and pick an encoding for the file
(via an encoding argument to the open function). I'd recommend using
UTF-8.
You can log out the value of sys.version to find out the version number.
Trying similar code within the Python interpreter
=
rse@Alibaba:~/test$ python
Python 3.4.0 (default, Jun 19 2015, 14:18:46)
[GCC 4.8.2] on linux
Type help, copyright, credits or license for more information.
di = {'userName': ['Ça va !']}--- A dictionary
str(di)
{'userName': ['Ça va !']} --- and its string representation
type(str(di))
class 'str' --- Is a string indeed
fi = open('essai', 'a')
fi.write(str(di) + '\n')
26 --- It works well
fi.close()
In this run, you are using Python 3.4, which defaults to UTF-8.
--
https://mail.python.org/mailman/listinfo/python-list
