Robert Smallshire wrote:
Hi Michael,
I'm trying to get some commercial code for a simple object
database we
have written for Python 2.6 to work with IronPython 2.6. In
Python 2.6
the return type of pickle.dumps() is str, which is of course a byte
string. In IronPython 2.6 it is also str, which is of course a
unicode string. This 'compatibility' is fine until I put those
strings into a database, at which point my interoperability between
CPython and IronPython goes off the rails.
How is this actually a problem?
I mean, can you provide a specific example of where a string in
IronPython doesn't behave as a byte string in CPython. I'm sure there
are such examples, but those may be bugs that the IPy team
can fix. In
practise I've encountered these problems very rarely.
My opening paragraph may be ambiguously worded - by 'interoperability' I
didn't mean the ability to run the same code unchanged on CPython and
IronPython (I have to change the code anyway to use a different database
adapter) - I meant interoperability between pickles persisted into a
database from both IronPython and CPython.
So are you telling the database that it is binary data or text?
Is the question how do I go from a pickle string in IronPython to a byte
array that I can pass to the database adaptor without going through an
explicit encode (which will transform the data)?
(One technique would be to explicitly use pickle protocol 0 which is
less efficient but only creates ascii characters - this is actually the
default. Another alternative would be to use JSON or YAML instead of
pickle.)
Here is an example of getting a byte array from a binary pickle in
IronPython:
>>> import pickle
>>> class A(object):
... b = 'hello'
... c = (None, 'fish', 7.2, 7j)
... a = {1: 2}
...
>>> p = pickle.dumps(A(), protocol=2)
>>> p
u'\x80\x02c__main__\nA\nq\x00)\x81q\x01}q\x02b.'
>>> from System import Array, Byte
>>> a = Array[Byte](tuple(Byte(ord(c)) for c in p))
>>> a
Array[Byte]((<System.Byte object at 0x0000000000000033 [128]>,
<System.Byte obje...
I hope this is at least slightly helpful. :-)
Michael
My basic issue is that the 'str' unavoidably implies certain semantics when
calling .NET APIs from IronPython. These APIs interpret str as text rather
than just bytes, which therefore gets transformed by various text encodings,
such as UTF-8 to UTF-16. Such encodings are undesirable for my pickled data
since the result is no longer necessarily a valid pickle. I suppose the
intention in Python 3.0 is that 'bytes' doesn't carry any semantics with it,
its just data, which is why pickle.dumps() in Python 3.0 returns bytes
rather than str.
I want to push plain old byte arrays into the database from both CPython and
IronPython, so I can avoid any head-scratching confusion with database
adapters and/or databases inappropriately encoding or decoding my data.
For example "data = [ord(c) for c in some_string]" has behaved as
expected many times for me in IronPython (and could help you turn
strings into bytes).
Thanks. I'll try something based on that.
Is this a theoretical problem at this stage or an actual problem?
Its an actual problem with SQLiteParameter.Value from the SQLite ADO.NET
provider. I think our original CPython code is a bit sloppy with respect to
the distinction between text strings and byte arrays, so I'll probably need
to tighten things up on both sides.
Would you agree tha using unicode() and bytes() everywhere and avoiding
str() gives code that has the same meaning in Python 2.6, IronPython 2.6 and
Python 3.0? Do you think this would be a good guideline to follow until we
can leave Python 2.x behind?
Many thanks,
Rob
--
http://www.ironpythoninaction.com/
_______________________________________________
Users mailing list
Users@lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
