On Aug 5, 2009, at 9:05 AM, C. Titus Brown wrote:
> This appears to be this problem, > > http://bugs.python.org/issue5736 > > caused in this case by anydbm returning a dbm.dbm object, which is not > iterable. Yuck. What a mess Python has inflicted on us: anydbm is supposed to provide a transparent interface to multiple backends, but even basic capabilities like iteration and __contains__ don't work for backends like gdbm and dbm. As a result, the standard module shelve won't work correctly either for those backends. And Python is taking away the bsddb backend, the one backend that actually worked correctly with shelve... Pygr currently provides a workaround for gdbm's iterator failure. For dbm there appears to be no way to access iteration (from Python) except via iter(db.keys()), thereby loading the entire set of keys into memory. Yuck. This will work, but it's not highly scalable. In addition, asking "key in db" doesn't work on dbm objects, which again will break shelve. This gets us into the business of subclassing shelve.Shelf to provide improved has_key(), __contains__() and get() methods (we already subclass it to provide an improved __iter__() method for gdbm). All of those are easy, but it makes you wonder, when will the madness end? > > This is one of those errors that could cause serious grief to > n00bies ;). Absolutely. Pygr is trying to clear away all these annoyances that get between people and real work. > > At least three possible solutions: > > - require bsddb > > - put in code to "test" to see if what anydbm returns can give an > iterator, and if not, give a very clear error message; > > - try to write our own wrapper code to "fix" the problem. > > I vote for #2 or #1. Note that bsddb is deprecated in 2.7 and going > away (gone?) in 3.x, but we can deal with that... later. I'd like to > get in something for 8.0 final if possible. Option #1 puts us on a collision course with Python's avowed policy to deprecate bsddb. It also means users won't be able to install Pygr unless they are able to install bsddb. I am concerned that installing bsddb may in some cases be tricky (I've heard of problems on various platforms...). Option #2 is not much of an improvement. The user will be able to install Pygr without bsddb, but all the file storages (SequenceFileDB, NLMSA etc.) use shelve. So if shelve doesn't actually work on their box, they won't be able to do much of anything! So I end up at Option #3, reluctantly. It looks quite easy to solve both dbm's iteration and __contains__ bugs. But then there is the fact that the iter() scalability will be poor (because your only choice is to call iter(db.keys()). And we keep hitting new problems with the backends (first gdbm, now dbm, next time ???). When will the madness end? It would be great if Python would solve all these backend problems in Python 2.7. So I conclude that we should eventually move away from all these bsddb / dbm style backends entirely, and use sqlite as the backend for shelve. sqlite looks like it has a future in Python; the other backends seem like throwbacks to the past. I believe Istvan looked at sqlite performance some time ago; I think it was pretty good. What do other people think? -- Chris --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pygr-dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pygr-dev?hl=en -~----------~----~----~----~------~----~------~--~---
