On Thu, 27 Mar 2008 10:24:10 -0400 Michael Bayer <[EMAIL PROTECTED]> wrote: > > On Mar 27, 2008, at 7:39 AM, crybaby wrote: > > > > Correct me if I am wrong, as of right now, relationship between > > multiple db shards, like sorting, grouping and joins have to be done > > in application level. > > that is correct; the ShardedQuery currently just concatentates > results from each individual shard into a list and returns the > result.
I've just sort of taken a random interest in this particular problem, so forgive me if this seems out of the blue :) I assume this concatenation happens in ShardedQuery._execute_and_instances, at the line reading: partial = partial + list(...) The bits inside that list() calls up to Query.instances for each shard in the for loop, which seems to return an iterator. Assuming the desired ordering can be figured out from that spot, and a python comparison function generated from it, creating a generator that performs a merge would be just the thing, no? For one thing, the entire result set wouldn't be sucked into RAM at once, as it appears to be doing here. And if the various shard queries emit sorted results, you just need a merge, which can be done incrementally with a generator. Does this seem like a patch worth writing? Any big gotchas that might come up if (say) I were to attempt to implement it? Kyle --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
