first of all, the stack trace suggests you have not set the "encoding"
parameter on create_engine() as it's still using UTF-8.
If you mean that a single database column may have different encodings
in different rows, you want to do your own encoding/decoding with
"encoding errors" set to something liberal like "ignore". You also
need to use your own custom type, as below:
from sqlalchemy import types
class MyEncodedType(types.TypeDecorator):
impl = String
def process_bind_param(self, value, dialect):
assert isinstance(value, unicode)
return value.encode('latin-1')
def process_result_value(self, value, dialect):
return value.decode('latin-1', 'ignore')
then use MyEncodedType() as the type for all your columns which
contain random encoding. No convert_unicode setting should be used
on your engine as this type replaces that usage.
On Jun 26, 2008, at 6:55 PM, Hermann Himmelbauer wrote:
>
> Hi,
> I'm trying to access a database via SA, which contains varchars with
> different, arbitrary encodings. Most of them are ascii or ISO-8859-2
> encoded,
> however, many are windows-1252 encoded and there are also some other
> weird
> ones.
>
> In my engine setup, I set the encoding to latin1 and set
> convert_unicode to
> True, as I my application requires the database values in unicode
> format.
>
> If SA now tries to retrieve such a key, the following traceback
> occurs:
>
> ------------------
> File "/home/dusty/prog/python_modules/sqlalchemy/engine/base.py",
> line 1605,
> in _get_col
> return processor(row[index])
> File "/home/dusty/prog/python_modules/sqlalchemy/databases/
> maxdb.py", line
> 112, in process
> return value.decode(dialect.encoding)
>
> File "/local/home/dusty/python/Python-2.4.4/lib/python2.4/encodings/
> utf_8.py",
> line 16, in decode
> return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-6:
> invalid
> data
> -----------------
>
> What can I do? It's not so important that all characters are correctly
> displayed, but it's vital that such improper encodings do not crash my
> application. Perhaps, there's some "universal" encoding that is able
> to deal
> with such problems?
>
> Best Regards,
> Hermann
>
> --
> [EMAIL PROTECTED]
> GPG key ID: 299893C7 (on keyservers)
> FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~----------~----~----~----~------~----~------~--~---
