Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Yeah I figured that its python as per the message. I didn't know I had to add it to every source that has the chars but thanks for the tip. I was trying it out to see if the issue was related to the database encoding or not, a UTF-8 conversion on the data completely fixed my issue. This is the

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Michael Radziej
Hi, this has nothing to do with Django at all, it's a python thing. The encoding declaration needs to go on top of EVERY source code file if you the source code contains non-ASCII encoding. This has NOTHING to do with Django encoding or the encoding your database uses! Now, slowly: When you excha

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
# Django settings for mandala project. # -*- coding: utf-8 -*- But django says its not declared. Where should I put this? -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send a

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Michael I already have this on top of my settings.py file all along, but djanog does not pick it up.. #django project # encoding line is 2nd in settings.py -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Michael Radziej
Hi Radomir, > So when I changed the encoding in the sql file from LATIN1 to UTF-8 I get > import errors: > > (1 row) > > ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x67 0x75 > CONTEXT: COPY locations_location, line 31 > setval Well, this is not a UTF8 sequence, but probably iso-8

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Thanks a bunch. Yeah when I did a full convert to UTF-8 I no longer have the issue. I still have the case sensitive login issue which came with the PostgreSQL upgrade. Do you have the same issue? Is this normal? Logging in as Rad no longer works, I have to use exact case as it is in the databa

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Xavier Ordoquy
I'm using full utf8. Whenever I need to import data, I make sure they are utf8 encoded. an extract from a database dump: SET statement_timeout = 0; SET client_encoding = 'UTF8'; SET standard_conforming_strings = off; SET check_function_bodies = false; SET client_min_messages = warning; SET escape

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
This fixed it iconv -f ISO_8859-1 -t UTF-8 mandala.postgres.sql > mandala.postgres.utf8.sql Now it takes the accents in as utf-8. I guess it didnt' like the LATIN1 encoding mixed with UTF-8 -- You received this message because you are subscribed to the Google Groups "Django users" group. To

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Xavier , which encoding do you use for french characters with django? UTF-8 does not like *é*, But it says it exists in utf-8:U+00E9éc3 a9LATIN SMALL LETTER E WITH ACUTE http://www.utf8-chartable.de -- You received this message because you are subscribed to the Google Groups "Django users" gr

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Xavier, what do you mean file is latin1 encoded? You lost me. By the way, one of the Non UTF-8 data that it complains about is the accent over the e in Neguac: * Néguac* New Brunswick See my error above given by psql db < file.sql -- You received this message because you are subscribed to t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
So when I changed the encoding in the sql file from LATIN1 to UTF-8 I get import errors: (1 row) ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x67 0x75 CONTEXT: COPY locations_location, line 31 setval So what encoding can I use that is safe with french accents but at the same t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Xavier Ordoquy
Hi Le 22 août 2013 à 09:52, Radomir Wojcik a écrit : > Are french characters UTF-8 Compliant? As I said, yes. Been using Django + Postgres with french texts and faced no issue so far. You also have a decent Django french user base which would have noticed that sort of issues. Back on our topi

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
So is it normal that without adding the u' to a string that contains french accents will give the error: ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) ? Are french characters UTF-8 Compliant? Would this insertion process not fail if the data contained within t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
MySQL would also somehow make the built in django contrib views login case insensitive, and now its case sensitive. Since it takes the request as a parameter and request.POST is immutable I will have to write my own login without using the batteries built in solution since I cannot do a .lower()

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
This must be my problem all along: *By the way, if you're using python 2, you shouldn't be using 'string' notation for character strings and should be using the u'string' one. 'string' is a binary string while u'string' is a text string. This is misleading with python 2 as there's implicit conv

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
> > Russ like I said before, I used COPY to get the data back in and it was > accepted by postgres with the schema it crated using django (See step 5 > below) 1) Export each table individually from MySQL into csv format. 2) Have Django re-create the models schema from django on its own. 3) E

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
> > From what I read, SQLite is production grade and its the most used > database in the world.. besides the point. So other than Postgres not picking up my encoding (UTF-8 or Latin-1) , both did not work. How can I get the login to be case insensitive again? This will really be a bummer if I

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 2:06 PM, Radomir Wojcik wrote: > Its not a problem with the conversion process. >> > Well, with all due respect -- yes, it is. Allow me to assure you that PostgreSQL handles UTF-8 and unicode just fine. If you're getting errors, it's either a problem with your original sour

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Thanks Xavier, I had this at the top of my settings.py file, tried changing it to latin-1 and it didn't make a difference. # -*- coding: utf-8 -*- So right now I am using the UTF-8 database as shown above: mandala | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | The data contains speci

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 2:11 PM, Radomir Wojcik wrote: > Also why did from django.contrib.auth.views.login become case sensitive? I > have users that login with User and user, in PostgreSQL this is treated as > 2 different things. Using MySQL it always saw it as one.. all very > frustrating. > Dj

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Also getting a bunch of these too: Non-ASCII character '\xc3' in file /var/web/mandala_env/mandala/alarms/views.py on line 329, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details (views.py, line 329) Is this normal? PostgreSQL is more strict with character types

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Xavier Ordoquy
Hi, Le 22 août 2013 à 08:25, Radomir Wojcik a écrit : > Adding the unicode u to the 'string' as such u'string' fixes the issue but > now I have to add this u to every string that will potentially have a > character with a french accent in it? How come MySQL didn't need this? There > must be

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Adding the unicode u to the 'string' as such u'string' fixes the issue but now I have to add this u to every string that will potentially have a character with a french accent in it? How come MySQL didn't need this? There must be a straight forward answer to this, something that will make Post

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Also why did from django.contrib.auth.views.login become case sensitive? I have users that login with User and user, in PostgreSQL this is treated as 2 different things. Using MySQL it always saw it as one.. all very frustrating. I will try SQLite tomorrow maybe instead, been up 2 days straigh

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
> > Its not a problem with the conversion process. I re-created the schema > using django syncdb and then I went table by table putting the data back in > using csv files I exported from MySQL. I took the long way to get this all > in and PostgreSQL had no problems taking my csv data as input.

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 1:37 PM, Radomir Wojcik wrote: > > And I tried with both UTF8 and LATIN1 database encodings, both produce the > same error: > > List of databases >Name| Owner | Encoding | Collate |Ctype| Access > privileg > es > >

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
And I tried with both UTF8 and LATIN1 database encodings, both produce the same error: List of databases Name| Owner | Encoding | Collate |Ctype| Access privileg es ---+--+--+-+-+---

Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
my code worked fine up until I switched to PostgreSQL from MySQL. Now I am bombarded with these errors whenever I have french accents in my strings. My database is correctly configured with LATIN1 (as UTF8 didn't like the accents when trying to import them from MySQL, the default was UTF8). My