hi,
We recently upgraded from SQLAlchemy to 0.4.8 to 0.9.3 also we upgrade
psycopg2 etc and we are using postgresql 9.2
We run in a very strange problem that the unpickle of one of the columns is
giving different result when we do this in a python procedure call.
When I run in the in the orm with sqlalchemy i get this back:
We defined our pickle processor as this:
class FailablePickleType(types.PickleType):
"""This syntax here is that it should replace all PickleType by this
type, that's why we sublclass PickleType and also implement
adapt.
"""
def process_bind_param(self, value, dialect):
dumps = pickle.dumps
def process(value):
try:
if value is None:
return None
return dumps(value)
except:
return None
return process
def result_processor(self, value, dialect):
loads = pickle.loads
def process(value):
try:
if value is None:
return None
return loads(value)
except (EOFError, IndexError):
try:
return loads(bytea.decode(value))
except EOFError:
return None
return None
return process
def compare_values(self, x, y):
try:
if isinstance(x, dict) and isinstance(y, dict):
return x == y
else:
return super(FailablePickleType, self).compare_values(x, y)
except:
return False
def adapt(self, impltype):
return FailablePickleType()
Then I have a table:
calculation_cache_transaction_table = Table(
'calculation_cache_transaction', metadata,
Column('id', Integer, primary_key=True),
Column('id_transaction', Integer, ForeignKey("transaction.id"),
index=True, nullable=False),
Column('dictionary', FailablePickleType(), nullable=True),
UniqueConstraint('id_transaction')
so when I use sqlalchemy to load some objects of my table (not all I get
different results from some rows) then when I run the code in side
postgresql with python procedural function
This is what I get in SQLAlchemy
{'calculateTaxesByID': {('totalPriceEx()',): {}}, 'averageCost': {None:
Decimal('7.4838709677419355')}, 'totalPriceEx': {None: Decimal('17.50')},
'getBasePriceEx': {None: Decimal('17.50')}}
This is the result when I rand the following code in Postgresql
"{'calculateTaxesByID': {('totalPriceEx()',): {}}}"
As you can see I also added some older technique of depickle fall back code
from sqlalchmey 0.4.8 with postgresql 8.4 the "decode" function but i had
that fallback also here above in the SQLALchmey orm code: So I feel the
code 2 paths are identical.
def decode(data):
diter = iter(data)
output = []
next = diter.next
for x in diter:
if x == "\\":
try:
y = next()
except StopIteration:
raise ValueError("incomplete backslash sequence")
if y == "\\":
pass
elif y.isdigit():
try:
os = ''.join((y, next(), next()))
except StopIteration:
raise ValueError("incomplete backslash sequence")
try:
x = chr(int(os, base=8))
except ValueError:
raise ValueError("invalid bytea octal sequence
{0}".format(os))
else:
raise ValueError("invalid backslash follow '{0}'".format(y))
output.append(x)
return ''.join(output)
def co_un_pickle(value):
try:
if value is None:
return None
a = pickle.loads(value)
return a
except (EOFError, IndexError):
try:
a = pickle.loads(decode(value))
return a
except EOFError:
return None
CREATE OR REPLACE FUNCTION util.unpickle(data bytea)
RETURNS text AS
$BODY$
from bytea import co_un_pickle
return unicode(co_un_pickle(data)).encode('raw_unicode_escape')
$BODY$
LANGUAGE plpythonu STABLE
COST 100;
ALTER FUNCTION util.unpickle(bytea)
My question for sqlalchmy or is this psycopg2 question?
is there any data manipulation that is happened before the pickle.loads
function gets called? so is there any processing happening before
def result_processor(self, value, dialect):
I put a break point in the debugger and looked on the stack but didn't see
anything getting done to the data at first quickly looking now.
Which i could get some inside in this?
thanks
marc
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
