I've got a recipe for what will work well for us. I imagine it could
be useful for others, although I left out the actual serialization
mechanism, since that will likely be very project specific.
I'd be happy to put this on the wiki, but if you wanted to look it over
first, you are more than welcome (I'd prefer your feedback). If you are
busy, I can just post it and hope someone may find it useful.
Thank again for your help,
Kent
On 9/7/2010 7:28 PM, Michael Bayer wrote:
On Sep 7, 2010, at 6:41 PM, Kent Bower wrote:
Two items:
* How does the orm currently determine whether it is safe to try
get() (e.i. there are no "funny join conditions")? If you point me
to the function where decision takes place, I can probably answer
this myself....
it compares the join condition of the relationship() to that of the
clause which the Mapper uses when it issues get(), then stores that
away as a flag for future consultation. It's very unusual for a
many-to-one relationship to be based on something other than a simple
foreign-key->primary key relationship, though.
* When I build up the primary key from the foreign key, is there an
efficient way to build a composite key in the correct order to pass
to get()? (I thought maybe "synchronize_pairs", but that maybe has
to do with getting the direction consistent instead?)
Well if you aren't using any composite primary keys in many-to-ones,
you wouldn't even have to worry about this. Otherwise, the two
collections to correlate would be property.local_remote_pairs and
property.mapper.primary_key. Perhaps make a dictionary out of
dict([(r, l) for l, r in prop.local_remote_pairs]) and your PK value
would be [getattr(instance,
prop.parent.get_property_by_column(mydict[p]).key) for p in
property.mapper.primary_key].
Or if you want to get ambitious you can just copy roughly whats in
strategies.py on line 605 but then you're digging into
internals....and looking at that now I'm wondering if
strategy._equated_columns is really different than local_remote_pairs
at all...
Thanks again, you've been much help!
On 9/7/2010 5:03 PM, Michael Bayer wrote:
On Sep 7, 2010, at 4:38 PM, Kent Bower wrote:
Don't want to strangle me, but when the orm (lazy)loads a MANYTONE
object, it doesn't go to the database if the object is in the
session. Can I get "with_parent()" to behave this way, or would I
need to specifically build up the primary key of the related object
and call query.get()?
the latter. You can use get() for all many to ones if you aren't
using any funny join conditions.
On 9/7/2010 10:25 AM, Michael Bayer wrote:
On Sep 7, 2010, at 10:12 AM, Kent Bower wrote:
Mike, in your proof of concept, when __getstate__ detected
transient, why did you need to make a copy of self.__dict__?
"self.__dict__.copy()"
i was modifying the __dict__ from what would be expected in a
non-serialized object, so that was to leave the original object
being serialized unchanged.
On 9/6/2010 2:35 PM, Michael Bayer wrote:
On Sep 6, 2010, at 2:11 PM, Kent Bower wrote:
Also, I was hoping you would tell me whether this would be a candidate for
subclassing InstrumentedAttribute? Would that make more sense or providing custom
__getstate__& __setstate__ ?
__getstate__ / __setstate__ are pretty much what I like to use for pickle
stuff, unless some exotic situation makes me have to use __reduce__. One
problem with the recipe is that theres no 'deferred' loading of attributes.
So in that sense playing with InstrumentedAttribute would give you a chance to
put a callable in there that does what you want.
There is also the possibility that __setstate__ can load up callables into the
instance_state using state.set_callable(). This is a callable that triggers
when you access the attribute that is otherwise None. There's a little bit of
fanfare required to get that callable to assign to the attribute in the right
way. Attached is an example of that. This is all a little more shaky since
the state/callable API isn't really public. Hasn't changed for awhile but
there's no guarantee.
Thanks for your help, hopefully I'll be able to contribute such a recipe.
Kent
Since sqla won't load that for me in the case of transient, I need to load the
relation manually (unless you feel like enhancing that as well).
its not an enhancement - it was a broken behavior that was specifically
removed. The transient object has no session, so therefore no SQL can be
emitted - there's no context established.
Now I can manually emulate the obj being persistent with your changes for
On Sep 6, 2010, at 10:58 AM, Michael Bayer<mike...@zzzcomputing.com> wrote:
On Sep 6, 2010, at 9:06 AM, Kent wrote:
with_parent seems to add a join condition.
OK, so I guess you read the docs which is why you thought it joined and why you didn't
realize it doesn't work for transient. r20b6ce05f194 changes all that so that
with_parent() accepts transient objects and will do the "look at the
attributes" thing. The docs are updated as this method does use the lazy loader
SQL mechanism, not a join.
Is there a way to get at
the query object that would be rendered from a lazy load (or what
"subqueryload" would render on the subsequent load), but on a
transient object, if i supply the session?
even though not "recommended", can it make sqla believe my transient
object is detached by setting its state key?
There are reasons i do not want to add this to the session and
disabling autoflush would also cause problems.
On Sep 3, 9:58 am, Michael Bayer<mike...@zzzcomputing.com> wrote:
On Sep 3, 2010, at 9:36 AM, Kent wrote:
For the case of customerid = '7', that is a simple problem, but when
it is a more complex join condition, we only wanted to define this
condition in one single place in our application (namely, the orm).
That way, if or when that changes, developers don't need to search for
other places in the app that needed to manually duplicate the logic of
the orm join condition.
If I supplied the DBSession to sqla, it would know how to create the
proper Query object for this lazyload. Can you point me in the right
direction (even if where you point me is not currently part of the
public API)?
Query has the with_parent() method for this use case.
Thanks again,
Kent
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this
groupathttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email tosqlalch...@googlegroups.com.
To unsubscribe from this group, send email
tosqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the
Google Groups "sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To post to this group, send email to sqlalchemy@googlegroups.com
<mailto:sqlalchemy@googlegroups.com>.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
<mailto:sqlalchemy+unsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
"""
If your project uses serialization techniques with sqlalchemy, you may
find the need to load persistent relations from transient or pending objects.
For example, you may have deserialized an object in order to perform calculations with it.
You may have no intention to add it to the database at this point (transient)
or you may be working in a controlled environment with session.autoflush == False (pending)
while you do validations or other calculations before the session.flush()
"""
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm.properties import RelationshipProperty
from sqlalchemy.orm.strategies import LazyLoader
from sqlalchemy.orm import attributes
import logging
log = logging.getLogger(__name__)
class InstallCustomizedAttribute(InstrumentationManager):
def post_configure_attribute(self, class_, attr, inst):
"""
Set up function to be invoked when relations are 'get'ed on possibly
transient objects, so we can manually query these related objects
"""
if isinstance(inst.property, RelationshipProperty):
default_loader = inst.impl.callable_
def create_getattr_when_never_set(state):
if state.has_identity:
# this is a persistent (or detached) object, so instead
# of a custom function, return default sqla functionality
return default_loader(state)
def getattr_when_never_set(passive):
prop = inst.property
get_local_property_by_col = prop.parent.get_property_by_column
dict_ = state.obj().__dict__
qry = DBSession.query(prop.mapper)
# Caution: not part of public API, but sqla already knows
# if we should "use_get" instead of qry, so let's ask...
if prop._get_strategy(LazyLoader).use_get:
# Work out the fk
rem_pk_ident = []
local_fk_from_remote = dict([(r, l) for l, r in prop.local_remote_pairs])
# Loop over the ordered primary key of related property
for remote_pk in prop.mapper.primary_key:
val = dict_.get(get_local_property_by_col(local_fk_from_remote[remote_pk]).key, None)
if val is None:
# fk not fully set, so no lookup
# (Note: if you allow_partial_pks, need extra logic here)
return None
rem_pk_ident.append(val)
result = qry.get(rem_pk_ident)
else:
table = prop.parent.mapped_table
for l, r in prop.local_remote_pairs:
if l.table == table and \
dict_.get(get_local_property_by_col(l).key, None) is None:
# fk not fully set, so no lookup
# (Note: if you allow_partial_pks, need extra logic here)
return None
result = qry.with_parent(state.obj(), attr).all()
if not prop.uselist:
l = len(result)
if l:
if l > 1:
log.error(
"Multiple rows returned with "
"uselist=False for attribute '%s' "
% prop)
result = result[0]
else:
result = None
# populate the result via dict. this produces no
# net change during flush, relying upon the FK
dict_[attr] = result
# tell the InstrumentedAttribute "its been set"
return attributes.ATTR_WAS_SET
return getattr_when_never_set
inst.impl.callable_ = create_getattr_when_never_set
DeclareBase = declarative_base()
class Base(object):
# comment out the next line to compare the "normal" behavior
__sa_instrumentation_manager__ = InstallCustomizedAttribute
pass
class Shipping(Base, DeclareBase):
__tablename__ = 'shipping'
id = Column(Integer, primary_key=True)
type = Column(Unicode)
def __repr__(self):
return '<Shipping: %s "%s">' % (self.id, self.type)
class Promotion(Base, DeclareBase):
__tablename__ = 'promotion'
id = Column(Unicode, primary_key=True)
discount = Column(Numeric)
def __repr__(self):
return '<Promotion: %s discount: %s>' % (self.id, self.discount)
class ShippingZip(Base, DeclareBase):
__tablename__ = 'shipzip'
ship_id = Column(Integer, ForeignKey('shipping.id'), primary_key=True)
zip_code = Column(Integer, primary_key=True)
class Order(Base, DeclareBase):
__tablename__ = 'order'
id = Column(Integer, primary_key=True)
promotion_id = Column(Unicode, ForeignKey('promotion.id'))
zip_code = Column(Integer)
promotion = relationship("Promotion")
shippingchoices = relationship("Shipping",
secondary=ShippingZip.__table__,
primaryjoin= zip_code==ShippingZip.zip_code,
foreign_keys=[ShippingZip.zip_code, Shipping.id])
def __repr__(self):
return '<Order: %s>' % self.id
engine = create_engine('sqlite://', echo=False)
Order.metadata.create_all(engine)
DBSession = Session(engine)
# add static, persistent data
data = (Shipping(type=u'Next Day', id=1),
ShippingZip(zip_code=29332, ship_id=1),
ShippingZip(zip_code=29000, ship_id=1),
Shipping(type=u'Ground', id=2),
ShippingZip(zip_code=29332, ship_id=2),
ShippingZip(zip_code=35444, ship_id=2),
Shipping(type=u'Slow', id=3),
ShippingZip(zip_code=88005, ship_id=3),
ShippingZip(zip_code=35444, ship_id=3),
Promotion(id=u'COUPON_A', discount=5),
Promotion(id=u'PROMO_B', discount=25),
Promotion(id=u'PROMO_C', discount=35),
)
for obj in data:
DBSession.merge(obj)
DBSession.commit()
DBSession.expunge_all()
# Demo
engine.echo=True
# Your deserialization routines create an order with some fk
# data populated
ord=Order()
ord.zip_code = 29332
ord.promotion_id = u'PROMO_B'
# Now you can reference the persistent data on the transient object as needed,
# so long as the fks have been set:
print "\n(Transient) ord.promotion = %r\n" % ord.promotion
print "\n(Transient) ord.shippingchoices = %r" % ord.shippingchoices
# If a many to one is in the session, we don't need a trip to the database:
another_ord = Order(zip_code=88005, promotion_id=u'PROMO_B')
DBSession.autoflush=False
DBSession.add(another_ord)
print "\n(Pending) another_ord.promotion = %r" % another_ord.promotion
