Re: [SQLObject] Unicode fun with SQLObject and MySQLdb
Dear Viktor, the final solution was that we have to build a new python-mysql dbapi, because there was a bug regarding db_encoding. you can follow up the bug history in python-mysql project on sf.net now it is working for me as well. thank you anyway. *** REPLY SEPARATOR *** On 2006. 09. 03. at 16:29 Binzberger Viktor wrote: >Dear Ivan, > >I've encountered the same problem earlier under Linux. The problem seems >to be that the "db_encoding" parameter is set by the _mysql.so library >based on some compile-time setting. It can be overridden by creating a >my.cnf file which contains at least > >[client] >default-character-set = utf8 >(...) > >and then forcing SQLObject to read that file: > >sqlobject.dburi="mysql://xxx:[EMAIL >PROTECTED]:3306/testscan_tg?read_default_file=my.cnf" > > >This helped for me. >Viktor > > >- Original Message - >From: "Ivan Horvath" <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> >To: <mailto:[email protected]>> >Sent: Thursday, August 24, 2006 5:14 PM >Subject: Re: [SQLObject] Unicode fun with SQLObject and MySQLdb > > > Dear Nadav, > > > > unfortunately i receive now the same error, when i try to insert > > some non ascii characters. > > the strange is that my system gives the db_encoding as latin-1, > > but my table collation is utf-8. > > > > so do you have any solution inserting non ascii characters to a > > unicode mysql table? > > > > Monday, July 10, 2006, 1:00:42 PM, you wrote: > > > > NS> Hi, > > > > NS> Thanks, but I get the same exception. It seems that the only role of > > NS> use_unicode is to force StringCol's to be UnicodeCol's. My >columns are > > NS> already UnicodeCol's. > > > > NS> Nadav > > > > NS> On 7/10/06, Ivan Horvath <[EMAIL PROTECTED] ><mailto:[EMAIL PROTECTED]>> wrote: > >>> Dear Nadav, > >>> > >>> try using the connection string with use_unicode parameter > >>> e.g. > >>> connString = "mysql://root:[EMAIL PROTECTED]/test?use_unicode=1" > >>> __connection__ = connectionForURI(connString) > >>> > >>> > >>> Saturday, July 8, 2006, 12:54:35 PM, you wrote: > >>> > >>> NS> Hi, > >>> > >>> NS> I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL >5.0.22 > >>> NS> on Arch Linux. > >>> NS> I am using fields of type UnicodeCol. When I try to insert such >object > >>> NS> to the database I get > >>> NS> an exception which originates in MySQLdb: > >>> > >>> NS> File > >>> NS> >"/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", > >>> NS> line 66, in _executeRetry > >>> NS> return cursor.execute(query) > >>> NS> File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line > >>> NS> 146, in execute > >>> NS> query = query.encode(charset) > >>> NS> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in >position > >>> NS> 196: ordinal not in range(128) > >>> > >>> NS> I tried to pass the data both as unicode strings and as normal >strings > >>> NS> with utf8 encoding. > >>> > >>> NS> It seems like SQLObject is sending the query to MySQLdb as a >normal > >>> NS> string, encoded in utf8. But, MySQLdb excepts it to be a unicode > >>> NS> string, because lines 145-156 in cursors.py are: > >>> > >>> NS> charset = db.character_set_name() > >>> NS> query = query.encode(charset) > >>> > >>> NS> Also, it turns out that the value of charset is 'latin1'. I am not > >>> NS> sure why it is so, but I could force it to be 'utf8' if I >changed the > >>> NS> connect() call in makeConnection() method of mysqlconnection.py to > >>> NS> have a keyword argument charset='utf8'. > >>> NS> Of course it does not help, since SQLObject already sends an >encoded string. > >>> > >>> NS> Any Ideas? > >>> > >>> NS> Thanks, > >>> NS> Nadav > >>> > >>> NS> Using Tomcat but need to do more? Ne
Re: [SQLObject] Unicode fun with SQLObject and MySQLdb
Dear Ivan, I've encountered the same problem earlier under Linux. The problem seems to be that the "db_encoding" parameter is set by the _mysql.so library based on some compile-time setting. It can be overridden by creating a my.cnf file which contains at least [client] default-character-set = utf8 (...) and then forcing SQLObject to read that file: sqlobject.dburi="mysql://xxx:[EMAIL PROTECTED]:3306/testscan_tg?read_default_file=my.cnf" This helped for me. Viktor - Original Message - From: "Ivan Horvath" <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> To: mailto:[email protected]>> Sent: Thursday, August 24, 2006 5:14 PM Subject: Re: [SQLObject] Unicode fun with SQLObject and MySQLdb > Dear Nadav, > > unfortunately i receive now the same error, when i try to insert > some non ascii characters. > the strange is that my system gives the db_encoding as latin-1, > but my table collation is utf-8. > > so do you have any solution inserting non ascii characters to a > unicode mysql table? > > Monday, July 10, 2006, 1:00:42 PM, you wrote: > > NS> Hi, > > NS> Thanks, but I get the same exception. It seems that the only role of > NS> use_unicode is to force StringCol's to be UnicodeCol's. My columns are > NS> already UnicodeCol's. > > NS> Nadav > > NS> On 7/10/06, Ivan Horvath <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: >>> Dear Nadav, >>> >>> try using the connection string with use_unicode parameter >>> e.g. >>> connString = "mysql://root:[EMAIL PROTECTED]/test?use_unicode=1" >>> __connection__ = connectionForURI(connString) >>> >>> >>> Saturday, July 8, 2006, 12:54:35 PM, you wrote: >>> >>> NS> Hi, >>> >>> NS> I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL 5.0.22 >>> NS> on Arch Linux. >>> NS> I am using fields of type UnicodeCol. When I try to insert such object >>> NS> to the database I get >>> NS> an exception which originates in MySQLdb: >>> >>> NS> File >>> NS> "/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", >>> NS> line 66, in _executeRetry >>> NS> return cursor.execute(query) >>> NS> File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line >>> NS> 146, in execute >>> NS> query = query.encode(charset) >>> NS> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position >>> NS> 196: ordinal not in range(128) >>> >>> NS> I tried to pass the data both as unicode strings and as normal strings >>> NS> with utf8 encoding. >>> >>> NS> It seems like SQLObject is sending the query to MySQLdb as a normal >>> NS> string, encoded in utf8. But, MySQLdb excepts it to be a unicode >>> NS> string, because lines 145-156 in cursors.py are: >>> >>> NS> charset = db.character_set_name() >>> NS> query = query.encode(charset) >>> >>> NS> Also, it turns out that the value of charset is 'latin1'. I am not >>> NS> sure why it is so, but I could force it to be 'utf8' if I changed the >>> NS> connect() call in makeConnection() method of mysqlconnection.py to >>> NS> have a keyword argument charset='utf8'. >>> NS> Of course it does not help, since SQLObject already sends an encoded string. >>> >>> NS> Any Ideas? >>> >>> NS> Thanks, >>> NS> Nadav >>> >>> NS> Using Tomcat but need to do more? Need to support web services, security? >>> NS> Get stuff done quickly with pre-integrated technology to make your job easier >>> NS> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >>> NS> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 <http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642> >>> NS> ___ >>> NS> sqlobject-discuss mailing list >>> NS> [email protected] <mailto:[email protected]> >>> NS> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss >>> >>>
Re: [SQLObject] Unicode fun with SQLObject and MySQLdb
Dear Nadav, unfortunately i receive now the same error, when i try to insert some non ascii characters. the strange is that my system gives the db_encoding as latin-1, but my table collation is utf-8. so do you have any solution inserting non ascii characters to a unicode mysql table? Monday, July 10, 2006, 1:00:42 PM, you wrote: NS> Hi, NS> Thanks, but I get the same exception. It seems that the only role of NS> use_unicode is to force StringCol's to be UnicodeCol's. My columns are NS> already UnicodeCol's. NS> Nadav NS> On 7/10/06, Ivan Horvath <[EMAIL PROTECTED]> wrote: >> Dear Nadav, >> >> try using the connection string with use_unicode parameter >> e.g. >> connString = "mysql://root:[EMAIL PROTECTED]/test?use_unicode=1" >> __connection__ = connectionForURI(connString) >> >> >> Saturday, July 8, 2006, 12:54:35 PM, you wrote: >> >> NS> Hi, >> >> NS> I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL 5.0.22 >> NS> on Arch Linux. >> NS> I am using fields of type UnicodeCol. When I try to insert such object >> NS> to the database I get >> NS> an exception which originates in MySQLdb: >> >> NS> File >> NS> >> "/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", >> NS> line 66, in _executeRetry >> NS> return cursor.execute(query) >> NS> File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line >> NS> 146, in execute >> NS> query = query.encode(charset) >> NS> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position >> NS> 196: ordinal not in range(128) >> >> NS> I tried to pass the data both as unicode strings and as normal strings >> NS> with utf8 encoding. >> >> NS> It seems like SQLObject is sending the query to MySQLdb as a normal >> NS> string, encoded in utf8. But, MySQLdb excepts it to be a unicode >> NS> string, because lines 145-156 in cursors.py are: >> >> NS> charset = db.character_set_name() >> NS> query = query.encode(charset) >> >> NS> Also, it turns out that the value of charset is 'latin1'. I am not >> NS> sure why it is so, but I could force it to be 'utf8' if I changed the >> NS> connect() call in makeConnection() method of mysqlconnection.py to >> NS> have a keyword argument charset='utf8'. >> NS> Of course it does not help, since SQLObject already sends an encoded >> string. >> >> NS> Any Ideas? >> >> NS> Thanks, >> NS> Nadav >> >> NS> Using Tomcat but need to do more? Need to support web services, security? >> NS> Get stuff done quickly with pre-integrated technology to make your job >> easier >> NS> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> NS> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> NS> ___ >> NS> sqlobject-discuss mailing list >> NS> [email protected] >> NS> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss >> >> >> >> -- >> Best regards, >> Ivan Horvath >> Chief Programmer >> >> Alcatel ISD >> PMS 2000 Product Team >> >> H-1116, Budapest >> Kondorfa u. 10. >> Tel.: +36 1 209 9546 >> Fax.: +36 1 209 9599 >> Mobil: +36 30 257 0235 >> VOIP: +49 511 676 478010 >> mailto:[EMAIL PROTECTED] >> >> Anyone who has never made a mistake has never tried anything new. >> >> /Albert Einstein/ >> >> >> >> - >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> ___ >> sqlobject-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss >> NS> - NS> Using Tomcat but need to do more? Need to support web services, security? NS> Get stuff done quickly with pre-integrated technology to make your job easier NS> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo NS> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 NS> ___ NS> sqlobject-discuss mailing list NS> [email protected] NS> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss -- Best regards, Ivan Horvath Chief Programmer Alcatel ISD PMS 2000 Product Team H-1116, Budapest Kondorfa u. 10. Tel.: +36 1 209 9546 Fax.: +36 1 209 9599 Mobil: +36 30 257 0235 VOIP: +49 511 676 478010 mailto:[EMAIL PROTECTED] Anyone who has never made a mistake has never tried anything new. /Albert Einstein/ - Using Tomcat but need t
Re: [SQLObject] Unicode fun with SQLObject and MySQLdb
Hi, Thanks, but I get the same exception. It seems that the only role of use_unicode is to force StringCol's to be UnicodeCol's. My columns are already UnicodeCol's. Nadav On 7/10/06, Ivan Horvath <[EMAIL PROTECTED]> wrote: > Dear Nadav, > > try using the connection string with use_unicode parameter > e.g. > connString = "mysql://root:[EMAIL PROTECTED]/test?use_unicode=1" > __connection__ = connectionForURI(connString) > > > Saturday, July 8, 2006, 12:54:35 PM, you wrote: > > NS> Hi, > > NS> I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL 5.0.22 > NS> on Arch Linux. > NS> I am using fields of type UnicodeCol. When I try to insert such object > NS> to the database I get > NS> an exception which originates in MySQLdb: > > NS> File > NS> > "/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", > NS> line 66, in _executeRetry > NS> return cursor.execute(query) > NS> File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line > NS> 146, in execute > NS> query = query.encode(charset) > NS> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position > NS> 196: ordinal not in range(128) > > NS> I tried to pass the data both as unicode strings and as normal strings > NS> with utf8 encoding. > > NS> It seems like SQLObject is sending the query to MySQLdb as a normal > NS> string, encoded in utf8. But, MySQLdb excepts it to be a unicode > NS> string, because lines 145-156 in cursors.py are: > > NS> charset = db.character_set_name() > NS> query = query.encode(charset) > > NS> Also, it turns out that the value of charset is 'latin1'. I am not > NS> sure why it is so, but I could force it to be 'utf8' if I changed the > NS> connect() call in makeConnection() method of mysqlconnection.py to > NS> have a keyword argument charset='utf8'. > NS> Of course it does not help, since SQLObject already sends an encoded > string. > > NS> Any Ideas? > > NS> Thanks, > NS> Nadav > > NS> Using Tomcat but need to do more? Need to support web services, security? > NS> Get stuff done quickly with pre-integrated technology to make your job > easier > NS> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > NS> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > NS> ___ > NS> sqlobject-discuss mailing list > NS> [email protected] > NS> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > > > > -- > Best regards, > Ivan Horvath > Chief Programmer > > Alcatel ISD > PMS 2000 Product Team > > H-1116, Budapest > Kondorfa u. 10. > Tel.: +36 1 209 9546 > Fax.: +36 1 209 9599 > Mobil: +36 30 257 0235 > VOIP: +49 511 676 478010 > mailto:[EMAIL PROTECTED] > > Anyone who has never made a mistake has never tried anything new. > > /Albert Einstein/ > > > > - > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ___ > sqlobject-discuss mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ sqlobject-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Unicode fun with SQLObject and MySQLdb
Dear Nadav, try using the connection string with use_unicode parameter e.g. connString = "mysql://root:[EMAIL PROTECTED]/test?use_unicode=1" __connection__ = connectionForURI(connString) Saturday, July 8, 2006, 12:54:35 PM, you wrote: NS> Hi, NS> I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL 5.0.22 NS> on Arch Linux. NS> I am using fields of type UnicodeCol. When I try to insert such object NS> to the database I get NS> an exception which originates in MySQLdb: NS> File NS> "/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", NS> line 66, in _executeRetry NS> return cursor.execute(query) NS> File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line NS> 146, in execute NS> query = query.encode(charset) NS> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position NS> 196: ordinal not in range(128) NS> I tried to pass the data both as unicode strings and as normal strings NS> with utf8 encoding. NS> It seems like SQLObject is sending the query to MySQLdb as a normal NS> string, encoded in utf8. But, MySQLdb excepts it to be a unicode NS> string, because lines 145-156 in cursors.py are: NS> charset = db.character_set_name() NS> query = query.encode(charset) NS> Also, it turns out that the value of charset is 'latin1'. I am not NS> sure why it is so, but I could force it to be 'utf8' if I changed the NS> connect() call in makeConnection() method of mysqlconnection.py to NS> have a keyword argument charset='utf8'. NS> Of course it does not help, since SQLObject already sends an encoded string. NS> Any Ideas? NS> Thanks, NS> Nadav NS> Using Tomcat but need to do more? Need to support web services, security? NS> Get stuff done quickly with pre-integrated technology to make your job easier NS> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo NS> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 NS> ___ NS> sqlobject-discuss mailing list NS> [email protected] NS> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss -- Best regards, Ivan Horvath Chief Programmer Alcatel ISD PMS 2000 Product Team H-1116, Budapest Kondorfa u. 10. Tel.: +36 1 209 9546 Fax.: +36 1 209 9599 Mobil: +36 30 257 0235 VOIP: +49 511 676 478010 mailto:[EMAIL PROTECTED] Anyone who has never made a mistake has never tried anything new. /Albert Einstein/ - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ sqlobject-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
[SQLObject] Unicode fun with SQLObject and MySQLdb
Hi, I am using SQLObject-0.8dev_r1814, MySQLdb 1.2.1_p2 and MySQL 5.0.22 on Arch Linux. I am using fields of type UnicodeCol. When I try to insert such object to the database I get an exception which originates in MySQLdb: File "/usr/lib/python2.4/site-packages/SQLObject-0.8dev_r1814-py2.4.egg/sqlobject/mysql/mysqlconnection.py", line 66, in _executeRetry return cursor.execute(query) File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 146, in execute query = query.encode(charset) UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 196: ordinal not in range(128) I tried to pass the data both as unicode strings and as normal strings with utf8 encoding. It seems like SQLObject is sending the query to MySQLdb as a normal string, encoded in utf8. But, MySQLdb excepts it to be a unicode string, because lines 145-156 in cursors.py are: charset = db.character_set_name() query = query.encode(charset) Also, it turns out that the value of charset is 'latin1'. I am not sure why it is so, but I could force it to be 'utf8' if I changed the connect() call in makeConnection() method of mysqlconnection.py to have a keyword argument charset='utf8'. Of course it does not help, since SQLObject already sends an encoded string. Any Ideas? Thanks, Nadav Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ sqlobject-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
