better yet, try out the trunk r3993, the patch I attached has a few quirks in it.
On Jan 2, 2008, at 5:45 PM, Michael Bayer wrote: > Heres a patch against 0.4.2 you can try. > > > > > <instances_yields.patch> > > > > just call session.query(Foo).yield_per(100).<filter, order by, etc>, > and evaluate the returned Query object in an iterative context. > > > On Jan 2, 2008, at 4:41 PM, Artur Siekielski wrote: > >> >> Hi. >> I have a table with many records (about 1mln), mapping is >> straightforward - one DB record corresponds to one Python object of >> the mapped class. I have to do something with every record. Code like >> this doesn't work: >> >> for inst in session.query(Bar): >> foo(inst) >> >> because all instances are loaded into memory before 'foo' is called - >> out of memory error or poor performance. Using offsets and limits >> doesnt' work good too, because doing eg. OFFSET 500000 causes very >> poor performance (memory usage is low, but executing SELECT takes >> much >> time; I'm using PostgreSQL). SQL Expression sql.select supports lazy >> yielding records, but the only way I can automatically transform >> fetched tuple into mapped class' object is calling >> Query.instances(cursor) method which loads all instances again (no >> possibility to load a part of result set). The only solution I have >> found is this: >> >> def generateAll(domainCls, table, session=sa.Session, minId=None, >> maxId=None): >> if minId is None: >> minId = session.query(domainCls).min(table.c.id) >> if maxId is None: >> maxId = session.query(domainCls).max(table.c.id) >> for id in xrange(minId, maxId + 1): >> inst = session.query(domainCls).get(id) >> if inst is not None: >> yield inst >> >> Am I missing something or there is no better way? I understand that >> in >> general case loading all instances at once is required, but I think >> that in cases like mine there should be some possibility for lazily >> loading instances. >> >> --~--~---------~--~----~------------~-------~--~----~ >> You received this message because you are subscribed to the Google >> Groups "sqlalchemy" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to [EMAIL PROTECTED] >> For more options, visit this group at >> http://groups.google.com/group/sqlalchemy?hl=en >> -~----------~----~----~----~------~----~------~--~--- >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
