On Thursday, October 16, 2014 11:52:39 PM UTC-4, Marc Van Olmen wrote:
>
> hi,
>
> We recently upgraded from SQLAlchemy to 0.4.8 to 0.9.3 also we upgrade
> psycopg2 etc and we are using postgresql 9.2
>
> We run in a very strange problem that the unpickle of one of the columns
> is giving different result when we do this in a python procedure call.
>
> When I run in the in the orm with sqlalchemy i get this back:
>
> We defined our pickle processor as this:
>
> class FailablePickleType(types.PickleType):
> """This syntax here is that it should replace all PickleType by this
> type, that's why we sublclass PickleType and also implement
> adapt.
> """
>
> def process_bind_param(self, value, dialect):
> dumps = pickle.dumps
>
> def process(value):
> try:
> if value is None:
> return None
> return dumps(value)
> except:
> return None
> return process
>
> def result_processor(self, value, dialect):
> loads = pickle.loads
>
> def process(value):
> try:
> if value is None:
> return None
> return loads(value)
> except (EOFError, IndexError):
> try:
> return loads(bytea.decode(value))
> except EOFError:
> return None
>
> return None
> return process
>
> def compare_values(self, x, y):
> try:
> if isinstance(x, dict) and isinstance(y, dict):
> return x == y
> else:
> return super(FailablePickleType, self).compare_values(x, y)
> except:
> return False
>
> def adapt(self, impltype):
> return FailablePickleType()
>
>
> Then I have a table:
>
> calculation_cache_transaction_table = Table(
> 'calculation_cache_transaction', metadata,
> Column('id', Integer, primary_key=True),
> Column('id_transaction', Integer, ForeignKey("transaction.id"),
> index=True, nullable=False),
> Column('dictionary', FailablePickleType(), nullable=True),
> UniqueConstraint('id_transaction')
>
>
What do you expect to fail? Do you expect the bytestring in the column to
become corrupted somehow?
>
> so when I use sqlalchemy to load some objects of my table (not all I get
> different results from some rows) then when I run the code in side
> postgresql with python procedural function
>
> This is what I get in SQLAlchemy
>
>
> {'calculateTaxesByID': {('totalPriceEx()',): {}}, 'averageCost': {None:
> Decimal('7.4838709677419355')}, 'totalPriceEx': {None: Decimal('17.50')},
> 'getBasePriceEx': {None: Decimal('17.50')}}
>
> This is the result when I rand the following code in Postgresql
>
> "{'calculateTaxesByID': {('totalPriceEx()',): {}}}"
>
What is the raw bytestring in the column which is decoded as the two
different values above?
>
>
> As you can see I also added some older technique of depickle fall back
> code from sqlalchmey 0.4.8 with postgresql 8.4 the "decode" function but i
> had that fallback also here above in the SQLALchmey orm code: So I feel the
> code 2 paths are identical.
>
>
> def decode(data):
> diter = iter(data)
> output = []
> next = diter.next
> for x in diter:
> if x == "\\":
> try:
> y = next()
> except StopIteration:
> raise ValueError("incomplete backslash sequence")
> if y == "\\":
> pass
> elif y.isdigit():
> try:
> os = ''.join((y, next(), next()))
> except StopIteration:
> raise ValueError("incomplete backslash sequence")
> try:
> x = chr(int(os, base=8))
> except ValueError:
> raise ValueError("invalid bytea octal sequence
> {0}".format(os))
> else:
> raise ValueError("invalid backslash follow
> '{0}'".format(y))
> output.append(x)
> return ''.join(output)
>
>
> def co_un_pickle(value):
> try:
> if value is None:
> return None
> a = pickle.loads(value)
> return a
> except (EOFError, IndexError):
> try:
> a = pickle.loads(decode(value))
> return a
> except EOFError:
> return None
>
>
>
> CREATE OR REPLACE FUNCTION util.unpickle(data bytea)
> RETURNS text AS
> $BODY$
> from bytea import co_un_pickle
> return unicode(co_un_pickle(data)).encode('raw_unicode_escape')
> $BODY$
> LANGUAGE plpythonu STABLE
> COST 100;
> ALTER FUNCTION util.unpickle(bytea)
>
>
>
> My question for sqlalchmy or is this psycopg2 question?
>
You should be able to answer that question by running your decoder on a
value retrieved directly from a psycopg2 connection.
>
> is there any data manipulation that is happened before the pickle.loads
> function gets called? so is there any processing happening before
>
> def result_processor(self, value, dialect):
>
> I put a break point in the debugger and looked on the stack but didn't see
> anything getting done to the data at first quickly looking now.
>
> Which i could get some inside in this?
>
> thanks
>
> marc
>
>
>
>
>
>
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
