[issue3783] dbm.sqlite proof of concept

Change by Erlend E. Aasland : -- nosy: +erlendaasland ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3783] dbm.sqlite proof of concept

Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3783] dbm.sqlite proof of concept

Gerhard Häring added the comment: This wiki page is out of date. It appears that SQlite is now threadsafe by default: http://www.sqlite.org/threadsafe.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783

[issue3783] dbm.sqlite proof of concept

Changes by Florent Xicluna florent.xicl...@gmail.com: -- versions: +Python 3.4 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Ray.Allen ysj@gmail.com: -- versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___

[issue3783] dbm.sqlite proof of concept

Changes by Éric Araujo mer...@netwok.org: -- nosy: +merwok ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing

[issue3783] dbm.sqlite proof of concept

Changes by Gregory P. Smith g...@krypto.org: -- nosy: -gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro s...@pobox.com: -- nosy: -skip.montanaro ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___ Python-bugs-list

[issue3783] dbm.sqlite proof of concept

Changes by Jesús Cea Avión j...@jcea.es: ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3783] dbm.sqlite proof of concept

Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___ Python-bugs-list

[issue3783] dbm.sqlite proof of concept

Changes by Antoine Pitrou pit...@free.fr: -- type: behavior - feature request ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___

[issue3783] dbm.sqlite proof of concept

Runar Tenfjord runar.tenfj...@gmail.com added the comment: Multi threading: According to http://www.sqlite.org/cvstrac/wiki?p=MultiThreading we need to keep a connection for each thread to support multi threaded access to the database. I came across this when deploying an application in a

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou pit...@free.fr added the comment: It would be nice to try to advance this at PyCon, or at another time. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Runar Tenfjord runar.tenfj...@gmail.com added the comment: By utilizing triggers on inserts and deletes it is possible to keep track of the size and speed up __len__ by 10 x. SQL: CREATE TABLE IF NOT EXISTS info (key TEXT UNIQUE NOT NULL, value INTEGER NOT NULL); INSERT OR IGNORE INTO

[issue3783] dbm.sqlite proof of concept

Changes by Antoine Pitrou pit...@free.fr: -- stage: - needs patch versions: +Python 3.2 -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Matthias Klose d...@debian.org added the comment: is there any chance for inclusion in 3.1? -- nosy: +doko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Unassigning. The code works but no one seems to be pushing for or caring about inclusion in Py3.1. If commits are delayed, then you might as well adopt the dbdict.py approach instead (reading the file in once at the

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: FWIW, I put an alternative in the sandbox /dbm_sqlite/alt/dbdict.py and am attaching a copy here. The idea is to emulate gdbm's fast mode and delay all writes until closing. That lets us subclass from dict and get high-speed

[issue3783] dbm.sqlite proof of concept

Changes by Jean-Paul Calderone exar...@divmod.com: -- nosy: -exarkun ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___ ___ Python-bugs-list

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou pit...@free.fr added the comment: I think issuing 'SELECT MAX(ROWID)' to compute the length of the table is not correct if some rows get deleted in the table. I've found a thread about it here: http://osdir.com/ml/db.sqlite.general/2004-03/msg00329.html In that thread someone

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: That's a bummer. Changing this method to __bool__ and then setting __len__ back to count(*). Added file: http://bugs.python.org/file12933/dbsqlite.py ___ Python tracker

[issue3783] dbm.sqlite proof of concept

Changes by Raymond Hettinger rhettin...@users.sourceforge.net: Removed file: http://bugs.python.org/file12931/dbsqlite.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Raymond Hettinger rhettin...@users.sourceforge.net: Removed file: http://bugs.python.org/file12896/dbsqlite.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Raymond Hettinger rhettin...@users.sourceforge.net: Removed file: http://bugs.python.org/file12891/tmp_dev_shelver.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Here's an updated patch (it's also in the sandbox): * Added a sync() method to support shelves. * Removed commits on granular sets and gets. * Optimized __len__ and __contains__. Added file:

[issue3783] dbm.sqlite proof of concept

Skip Montanaro s...@pobox.com added the comment: Unassigning myself. I don't have time for this. I've taken my sandbox version about as far as I can, and the subject of this ticket has gone a bit far afield from just adding a sqlite module to the dbm pkg. -- assignee: skip.montanaro -

[issue3783] dbm.sqlite proof of concept

Changes by Raymond Hettinger rhettin...@users.sourceforge.net: -- assignee: - rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Raymond Hettinger rhettin...@users.sourceforge.net: Added file: http://bugs.python.org/file12896/dbsqlite.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Am also working on a patch for this and would like to coordinate. My first draft is attached: Added file: http://bugs.python.org/file12891/tmp_dev_shelver.py ___ Python tracker

[issue3783] dbm.sqlite proof of concept

Jean-Paul Calderone exar...@divmod.com added the comment: Some comments on tmp_dev_shelver.py... Regarding SQLhash.__init__, it would be better to avoid relying on the sqlite_master table by using the CREATE TABLE IF NOT EXISTS form of table creation. Setting the isolation_level in __init__

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Thanks for looking at this. I'll do an update in the next few days. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Skip Montanaro s...@pobox.com added the comment: Hopefully I'm not picking at a scab here. I updated the dbm.sqlite module in the sandbox. It now orders by rowid instead of by key. (I saw no performance penalty for the small table sizes I was using to ordering. I switched from ordering by key

[issue3783] dbm.sqlite proof of concept

Erno Kuusela [EMAIL PROTECTED] added the comment: I'm looking for a bsddb-shelve replacement (because of we bsddb corruption problems), and decided to give this a try. Don't overlook the free locking you get from sqlite when evaluating this for inclusion! A small bug: from sq_dict import

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: Thank you for the report (fixed in the newly attached version) :) . Added file: http://bugs.python.org/file11602/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Josiah I know that no one hear likes my particular implementation Josiah (though it offers more or less the full interface)... I think implementing as much of the bsddb interface as you can is fine. I agree people used first, next, last,

[issue3783] dbm.sqlite proof of concept

Gerhard Häring [EMAIL PROTECTED] added the comment: I like Skip's version better, because it's closer to the dbm specification instead of trying to mimic bsddb (first, last, etc.). I'd like to keep such things out. I've made a few changes to the sandbox project which I will check in later

[issue3783] dbm.sqlite proof of concept

Gerhard Häring [EMAIL PROTECTED] added the comment: One question about Josiah's _check_value(). SQLite is less crippled than other simplistic databases and also supports integers, reals and blobs in addition to strings. Shouldn't we make this accessible to users? Or is compatibility with other

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Gerhard What's all this ORDER BY in both your implementations about? Gerhard The dbm spec says nothing about keys being ordered AFAIC. Can Gerhard we get rid of these? I'd like to guarantee that zip(db.keys(), db.values() ==

[issue3783] dbm.sqlite proof of concept

Gerhard Häring [EMAIL PROTECTED] added the comment: I'd like to guarantee that zip(db.keys(), db.values() == db.items(). Ok. If that isn't guaranteed elsewhere just drop it here? FWIW that will also work without the ORDER BY, because you're getting the rows back in the same ORDER. Something

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: I'd like to guarantee that zip(db.keys(), db.values() == db.items(). Antoine It doesn't sound very useful, and it may hurt performance on Antoine big tables. Actually, I think Python guarantees (for dicts at least - other mappings

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Gerhard FWIW that will also work without the ORDER BY, because you're Gerhard getting the rows back in the same ORDER. Something cheaper Gerhard would also be ORDER BY ROWID. I still propose to just do Gerhard without the ORDER BY.

[issue3783] dbm.sqlite proof of concept

Gerhard Häring [EMAIL PROTECTED] added the comment: Skip Montanaro wrote: Skip Montanaro [EMAIL PROTECTED] added the comment: Gerhard FWIW that will also work without the ORDER BY, because you're Gerhard getting the rows back in the same ORDER. Something cheaper Gerhard would also

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou [EMAIL PROTECTED] added the comment: Le jeudi 11 septembre 2008 à 13:48 +, Skip Montanaro a écrit : Actually, I think Python guarantees (for dicts at least - other mappings should probably follow suit) that if you call keys() then call values() without making any changes to

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou [EMAIL PROTECTED] added the comment: I might add that calling keys() then values() is suboptimal, because it will issue two SQL queries while calling items() will issue just one. ___ Python tracker [EMAIL PROTECTED]

[issue3783] dbm.sqlite proof of concept

Changes by Josiah Carlson [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11412/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: I like Skip's version better, because it's closer to the dbm specification instead of trying to mimic bsddb (first, last, etc.). I'd like to keep such things out. dbm.sqlite is meant as a potential replacement of dbm.bsddb. Since people

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: As long as SQLite guarantees that the ordering is identical, then sure, dump the ORDER BY clause. Gerhard It doesn't guarantee it, but the implementation behaves like Gerhard this. If the behavior isn't guaranteed, I think you

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Antoine I might add that calling keys() then values() is suboptimal, Antoine because it will issue two SQL queries while calling items() Antoine will issue just one. Well, sure, but heaven only knows what an application programmer will

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou [EMAIL PROTECTED] added the comment: Well, sure, but heaven only knows what an application programmer will do... If the docs clearly explain that there is no guarantee, we don't need heaven. I would find it strange to potentially ruin performance just for a guarantee which has

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: I would find it strange to potentially ruin performance just for a guarantee which has no useful purpose. Benchmarks to prove or disprove performance changes? Subclasses to offer different order by semantics (see the version I uploaded for

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou [EMAIL PROTECTED] added the comment: Benchmarks to prove or disprove performance changes? Agreed, benchmarks should be run. Subclasses to offer different order by semantics (see the version I uploaded for an example)? If you like, but ordering semantics is something which

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Well, sure, but heaven only knows what an application programmer will do... Antoine If the docs clearly explain that there is no guarantee, we Antoine don't need heaven. I would find it strange to potentially ruin Antoine

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: If you like, but ordering semantics is something which is just as easily done in Python, so I don't understand the point of integrating it in the dbm layer... Actually, the db layer is *exactly* where it should be implemented, especially

[issue3783] dbm.sqlite proof of concept

Changes by Josiah Carlson [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11467/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Josiah Carlson [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11470/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: Here's a version with views from Python 3.0 for keys, values, and items :) . I know that no one hear likes my particular implementation (though it offers more or less the full interface), but the Keys, Values, and Items classes in the

[issue3783] dbm.sqlite proof of concept

Changes by Gregory P. Smith [EMAIL PROTECTED]: -- nosy: +gregory.p.smith ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list

[issue3783] dbm.sqlite proof of concept

Raymond Hettinger [EMAIL PROTECTED] added the comment: I would like to see something like this go into 3.0 so that shelves don't become useless for Windows users. -- nosy: +rhettinger ___ Python tracker [EMAIL PROTECTED]

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: Here's an alternate version with most of bsddb's interface intact. -- nosy: +josiahcarlson Added file: http://bugs.python.org/file11412/sq_dict.py ___ Python tracker [EMAIL PROTECTED]

[issue3783] dbm.sqlite proof of concept

Gregory P. Smith [EMAIL PROTECTED] added the comment: sq_dict review: have sqlite quote/escape self._mtn before using it with a python %s substitution. or pass it into the sql query function as a positional ? parameter like you do for keys and values. (avoid sql injection) raise a TypeError

[issue3783] dbm.sqlite proof of concept

Josiah Carlson [EMAIL PROTECTED] added the comment: I tried passing the db name as a parameter with '?', it doesn't always work. Also, there shouldn't be any SQL injection issues here unless someone designed their system wrong (if a third party is allowed to pass the name of a db table into

[issue3783] dbm.sqlite proof of concept

Antoine Pitrou [EMAIL PROTECTED] added the comment: It would be more efficient to base keys() on iterkeys() than the reverse, IMO. Other than that, why not open a branch or at least upload full-fledged patch files? :) -- nosy: +pitrou ___ Python

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Antoine It would be more efficient to base keys() on iterkeys() than the Antoine reverse, IMO. True. I was just modifying the dumbdbm implementation. Antoine Other than that, why not open a branch or at least upload Antoine

[issue3783] dbm.sqlite proof of concept

Changes by Jesús Cea Avión [EMAIL PROTECTED]: -- nosy: +jcea ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list

[issue3783] dbm.sqlite proof of concept

Changes by Gregory Burd [EMAIL PROTECTED]: -- nosy: +gregburd ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: OK, I made a sandbox project out of it: svn+ssh://[EMAIL PROTECTED]/sandbox/trunk/dbm_sqlite Hack away! -- assignee: - skip.montanaro ___ Python tracker [EMAIL PROTECTED]

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11386/dbm.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11387/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11388/test_dbm_sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

New submission from Skip Montanaro [EMAIL PROTECTED]: Based on recent discussions about ridding Python of bsddb I decided to see how hard it would be to implement a barebones dbm.sqlite module. Turns out, not very hard. No docs. No test cases. Caveat emptor. But I think it can serve as at

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Attaching corrected module. Added file: http://bugs.python.org/file11383/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11380/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Attaching test cases based on dumbdbm tests. Added file: http://bugs.python.org/file11384/test_dbm_sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Another slight revision to the module. Added file: http://bugs.python.org/file11385/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11383/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Trivial doc diffs against 3.0b3 doc. Added file: http://bugs.python.org/file11386/dbm.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Another tweak - add values() Added file: http://bugs.python.org/file11387/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Skip Montanaro [EMAIL PROTECTED] added the comment: Updated test cases Added file: http://bugs.python.org/file11388/test_dbm_sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11384/test_dbm_sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___

[issue3783] dbm.sqlite proof of concept

Changes by Skip Montanaro [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11385/sqlite.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___