On Sunday, July 17, 2016 at 8:47:11 AM UTC+2, Martijn van Oosterhout wrote:
>
>
> I'll play a bit and see what I can get to work. Thanks again.
>
>
So, I have a chance to play and got something that actually works quite
nicely, see below. Two things:
- I switched to referencing the primary key of the original object
directly, because some of our relationships are a bit more complex.
- Chained relationships don't work.
But in any case, even this improves performance greatly.
from itertools import groupby, islice
from sqlalchemy.orm import attributes, object_session
from sqlalchemy import tuple_
def yielded_load(query, attrs, N=1000):
# Note: query must return only a single object (for now anyway)
main_query = query.yield_per(N)
main_res = iter(main_query)
while True:
# Fetch block of results from query
objs = list(islice(main_res, N))
if not objs:
break
for attr in attrs:
target = attr.prop.mapper
pk = attr.prop.parent.primary_key
# Generate query that joins against original table
child_q = object_session(objs[0]).query(target, *pk).order_by(*
pk)
if attr.prop.order_by:
child_q = child_q.order_by(*attr.prop.order_by)
keys = [[getattr(obj, col.key) for col in pk] for obj in objs]
child_q = child_q.join(attr).filter(tuple_(*pk).in_(keys))
collections = dict((k, [r[0] for r in v]) for k, v in groupby(
child_q,
lambda x:tuple([getattr(x, c.key) for c in pk])
))
for obj in objs:
attributes.set_committed_value(
obj,
attr.key,
collections.get(
tuple(getattr(obj, c.key) for c in pk),
())
)
for obj in objs:
yield obj
--
Martijn
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
