> On Nov 29, 2014, at 3:52 AM, Staszek <[email protected]> wrote:
>
> Hi
>
> We are experiencing strange UnicodeEncodeError with MySQL + Google App
> Engine. The stack trace looks like this (I edited the application name
> for security reasons, sorry):
>
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/orm/query.py",
>> line 2280, in all
>> return list(self)
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/orm/loading.py",
>> line 72, in instances
>> rows = [process[0](row, None) for row in fetch]
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/orm/loading.py",
>> line 451, in _instance
>> populate_state(state, dict_, row, isnew, only_load_props)
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/orm/loading.py",
>> line 305, in populate_state
>> populator(state, dict_, row)
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/orm/strategies.py",
>> line 154, in fetch_col
>> dict_[key] = row[col]
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/engine/result.py",
>> line 89, in __getitem__
>> return processor(self._row[index])
>> File
>> "/base/data/home/apps/s~app/1-0.379828850176861603/sqlalchemy/processors.py",
>> line 66, in process
>> return decoder(value, errors)[0]
>> File
>> "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/encodings/utf_8.py",
>> line 16, in decode
>> return codecs.utf_8_decode(input, errors, True)
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\u0142' in
>> position 26: ordinal not in range(128)
>
> The table, which is being queried using all():
>
>> opList = dbSession.query(PendingOperation).filter_by(opcode=opcode).all()
looks easy enough to reproduce:
import codecs
x = u'\u0142'
codecs.utf_8_decode(x)
that is, in Py2K when you call decode() on a unicode object already, things go
wrong.
SQLAlchemy shouldn’t be attempting to run this decode operation unless the
MySQL driver used here is acting in a flaky way, that is, SQLAlchemy did a test
on first connect to see if a Unicode() type comes back as u’’ or not. That
it’s trying to do the decode means that it didn’t, yet then on this column it
is.
The MySQL driver in use here would be of top importance in determining why this
might be happening, I guess this is the “gaerdbms” driver?
So the best approach for MySQL is to ensure that the driver is predictable
about unicode include, specify use_unicode=0 in the URL (see
http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#unicode).
>
> looks quite ordinary:
>
>> class PendingOperation(Base, AbstractEntity):
>> __tablename__ = 'pendingops'
>> __table_args__ = {'mysql_engine': 'InnoDB'}
>>
>> [... some constants ...]
>>
>> id = Column(BigInteger, autoincrement=True, primary_key=True)
>> idStrSha512 = Column(String(length=128), nullable=True, unique=True,
>> index=True)
>> createTS = Column(DateTime(), nullable=False,
>> default=datetime.datetime.utcnow)
>> opcode = Column(Integer, nullable=False, default=0)
>> opcode2 = Column(Integer, default=0)
>> aid1 = Column(BigInteger, nullable=True, default=None)
>>
>>
>> aid2 = Column(BigInteger, nullable=True, default=None)
>> as1 = Column(Unicode(length=100))
>> as2 = Column(Unicode(length=100))
>> utxt = Column(Unicode(length=10000))
>>
>> [... some methods ...]
>
> The problem seems to be with as2 column.
>
> What's weird is that:
>
> 1. I was unable to reproduce this locally (using Google App Engine SDK +
> my local MySQL installation).
>
> 2. The problem resolves by itself (from time to time): it just starts to
> work, then stops again!
>
> Has anybody seen anything like this before??
>
> This is SQLAlchemy 0.9.1.
>
> --
> http://people.eisenbits.com/~stf/
> http://www.eisenbits.com/
>
> OpenPGP: 80FC 1824 2EA4 9223 A986 DB4E 934E FEA0 F492 A63B
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
