UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
I am using Eclipse to write my python scripts and when i run them from inside eclipse they work fine without errors. But almost in every script that handle some form of special characters like swedish åäö and chinese characters etc i get Unicode errors when running the script externally with

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
: Thanks, your crystal ball seems to be right :P On Tuesday, February 12, 2013 12:43:00 PM UTC+1, Steven D'Aprano wrote: Magnus Pettersson wrote: I am using Eclipse to write my python scripts and when i run them from inside eclipse they work fine without errors. But almost in every

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
) On Tuesday, February 12, 2013 12:01:19 PM UTC+1, Andrew Berg wrote: On 2013.02.12 04:43, Magnus Pettersson wrote: I am using Eclipse to write my python scripts and when i run them from inside eclipse they work fine without errors. But almost in every script that handle some form of special

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
Are you sure you are writing the same data? That would mean that pydev changes the default encoding -- which is evil. A portable approach would be to use codecs.open() or io.open() instead of the built-in: import io with io.open(filepath, a) as f: ...

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
What encoding is this file? Since you're appending to it, you really need to match the pre-existing encoding, or the next program to deal with it is in big trouble. So using the io.open() without the encoding= keyword is probably a mistake. The .txt file is in UTF-8 I have got it

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
You don't show the code that actually does the io.open(), nor the url.encode, so I'm not going to guess what you're actually doing. Hmm im not sure what you mean but I wrote all code needed in a previous post so maybe you missed that one :) In short I basically just have: import io

Re: UnicodeEncodeError when not running script from IDE

2013-02-12 Thread Magnus Pettersson
Thanks a lot Steven, you gave me a good AHA experience! :) Now I understand why I had to use encoding when calling the urllib2! So basically Eclipse PyDev does this in the background for me, and its console supports utf-8, so thats why i never had to think about it before (and why some scripts