On Sep 2, 2009, at 6:48 PM, Marek Szuba wrote:
> Namshin Kim <[email protected]> wrote: > >> I ran the megatest with checkFirstID=False option, and it still uses >> almost 2.5GB memory. Maybe it would be the minimum memory >> requirements. > This was sort of expected, as looking at the code of the megatest in > questions shows all keys are iterated over even if the AnnotationDB > constructor does not (which is what checkFirstID does) - so it is > quite likely this is indeed simply how much memory performing this > particular operation needs. Upon reflection, I think we can make iter() stop calling keys(), and eliminate this whole issue. It was calling keys() to avoid the possibility that while the iterator is continuing to run, another query might reuse the same cursor, interrupting the iteration results prematurely, I think we fixed a subtle bug some time back in which this problem occurred. Fortunately, we now have a better solution open to us: DBServerInfo provides a method for allocating a new cursor whenever you want. That means the iter method can allocate a new cursor for itself, which will isolate it from any other queries that might be performed while the iterator is still running. In that case there is no need to call keys() to get the entire list of results before returning the first iterator result. Marek, it would be nice if you could find the original issue that addressed this "incomplete iteration" bug, and make sure that we have a test for it in the test suite. I'm almost positive we do. Then we can try out this new idea and make sure it doesn't break the old "iterator / query isolation" test. -- Chris --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pygr-dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pygr-dev?hl=en -~----------~----~----~----~------~----~------~--~---
