Hello community, here is the log from the commit of package python-lmdb for openSUSE:Factory checked in at 2019-07-26 12:39:49 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-lmdb (Old) and /work/SRC/openSUSE:Factory/.python-lmdb.new.4126 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-lmdb" Fri Jul 26 12:39:49 2019 rev:3 rq:718281 version:0.96 Changes: -------- --- /work/SRC/openSUSE:Factory/python-lmdb/python-lmdb.changes 2019-06-12 13:14:18.304816307 +0200 +++ /work/SRC/openSUSE:Factory/.python-lmdb.new.4126/python-lmdb.changes 2019-07-26 12:39:52.873925046 +0200 @@ -1,0 +2,11 @@ +Wed Jul 24 13:54:02 UTC 2019 - Martin Herkt <[email protected]> + +- Update to v0.96 + * Doc updates. + * More removal of code for now-unsupported Python versions. + * Only preload the value with the GIL unlocked when the value is + actually requested. This significantly improves read + performance to retrieve keys with large values when the value + isn't retrieved. + +------------------------------------------------------------------- Old: ---- lmdb-0.95.tar.gz New: ---- lmdb-0.96.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-lmdb.spec ++++++ --- /var/tmp/diff_new_pack.QKowfO/_old 2019-07-26 12:39:54.645924013 +0200 +++ /var/tmp/diff_new_pack.QKowfO/_new 2019-07-26 12:39:54.669923999 +0200 @@ -18,7 +18,7 @@ %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-lmdb -Version: 0.95 +Version: 0.96 Release: 0 Summary: Universal Python binding for the LMDB 'Lightning' Database License: OLDAP-2.8 ++++++ lmdb-0.95.tar.gz -> lmdb-0.96.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/ChangeLog new/lmdb-0.96/ChangeLog --- old/lmdb-0.95/ChangeLog 2019-06-08 02:57:33.000000000 +0200 +++ new/lmdb-0.96/ChangeLog 2019-07-15 04:24:30.000000000 +0200 @@ -1,5 +1,16 @@ -2018-06-08 v0.95 +2019-07-14 v0.96 +* First release under new maintainer, Nic Watson. + +* Doc updates. + +* More removal of code for now-unsupported Python versions. + +* Only preload the value with the GIL unlocked when the value is actually + requested. This significantly improves read performance to retrieve keys + with large values when the value isn't retrieved. Reported by Dan Patton. + +2019-06-08 v0.95 * The minimum supported version of Python is now 2.7. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/PKG-INFO new/lmdb-0.96/PKG-INFO --- old/lmdb-0.95/PKG-INFO 2019-06-08 02:58:23.000000000 +0200 +++ new/lmdb-0.96/PKG-INFO 2019-07-15 04:25:00.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: lmdb -Version: 0.95 +Version: 0.96 Summary: Universal Python binding for the LMDB 'Lightning' Database Home-page: http://github.com/dw/py-lmdb/ Author: David Wilson diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/README.md new/lmdb-0.96/README.md --- old/lmdb-0.95/README.md 2017-07-16 13:24:56.000000000 +0200 +++ new/lmdb-0.96/README.md 2019-07-15 04:24:30.000000000 +0200 @@ -1,23 +1,24 @@ +This is a universal Python binding for the LMDB ‘Lightning’ Database. -# py-lmdb Needs a Maintainer! +See [the documentation](https://lmdb.readthedocs.io) for more information. -I simply don't have time for this project right now, and still the issues keep -piling in. Are you a heavy py-lmdb user and understand most bits of the API? -Got some spare time to give a binding you use a little love? Dab hand at C and -CFFI? Access to a Visual Studio build machine? Please drop me an e-mail: dw at -botanicus dot net. TLC and hand-holding will be provided as necessary, I just -have no bandwidth left to write new code. +# py-lmdb Has a New Maintainer! +Hi. My name is Nic Watson. With lots of help from David, the author and original maintainer, I'm taking over maintenance of py-lmdb. Please be patient as we work out the details of the handover. ### CI State | Platform | Branch | Status | | -------- | ------ | ------ | -| UNIX | ``master`` | [](https://travis-ci.org/dw/py-lmdb/branches) | -| Windows | ``master`` | [](https://ci.appveyor.com/project/dw/py-lmdb/branch/master) | -| UNIX | ``release`` | [](https://travis-ci.org/dw/py-lmdb/branches) | -| Windows | ``release`` | [](https://ci.appveyor.com/project/dw/py-lmdb/branch/release) | +| UNIX | ``master`` | [](https://travis-ci.org/jnwatson/py-lmdb/branches) | +| Windows | ``master`` | [](https://ci.appveyor.com/project/NicWatson/py-lmdb/branch/master) | +| UNIX | ``release`` | [](https://travis-ci.org/jnwatson/py-lmdb/branches) | +| Windows | ``release`` | [](https://ci.appveyor.com/project/NicWatson/py-lmdb/branch/release) | If you care whether the tests are passing, check out the repository and execute the tests under your desired target Python release, as the Travis CI build has a bad habit of breaking due to external factors approximately every 3 months. + +# Sponsored by The Vertex Project + +My current employer, [The Vertex Project](https://vertex.link/) is generously sponsoring my time to maintain py-lmdb. If you like open source and systems programming in Python, check us out. We could definitely use some more hands. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/docs/index.rst new/lmdb-0.96/docs/index.rst --- old/lmdb-0.95/docs/index.rst 2019-06-08 02:57:33.000000000 +0200 +++ new/lmdb-0.96/docs/index.rst 2019-07-15 04:24:30.000000000 +0200 @@ -12,7 +12,7 @@ <http://symas.com/mdb/>`_. Two variants are provided and automatically selected during install: a `CFFI <https://cffi.readthedocs.io/en/release-0.5/>`_ variant that supports `PyPy <http://www.pypy.org/>`_ and all versions of CPython >=2.7, -and a C extension that supports CPython >=2.7 and >=3.3. Both variants +and a C extension that supports CPython >=2.7 and >=3.4. Both variants provide the same interface. LMDB is a tiny database with some excellent properties: @@ -352,7 +352,8 @@ :py:class:`Environment`. Most :py:class:`Environment` methods are thread-safe, and may be called -concurrently, except for :py:meth:`Environment.close`. +concurrently, except for :py:meth:`Environment.close`. Running `close` at the +same time as other database operations may crash the interpreter. A write :py:class:`Transaction` may only be used from the thread it was created on. @@ -364,12 +365,37 @@ any thread except the thread that currently owns its associated :py:class:`Transaction`. +Limitations running on 32-bit Processes ++++++++++++++++++++++++++++++++++++++++ +32-bit processes (for example 32-bit builds of Python on Windows) are severely +limited in the amount of virtual memory that can be mapped in. This is +particularly true for any 32-bit process but is particularly true for +Python running on Windows and long running processes. + +Virtual address space fragmentation is a significant issue for mapping files +into memory, a requirement for lmdb, as lmdb requires a contiguous range of +virtual addresses. See +https://web.archive.org/web/20170701204304/http://forthescience.org/blog/2014/08/16/python-and-memory-fragmentation +for more information and a solution that potentially gives another 50% of +virtual address space on Windows. + +Importantly, using a 32-bit instance of Python (even with the OS being 64-bits) +means that the maximum size file that can be ever be mapped into memory is +around 1.1 GiB, and that number decreases as the python process lives and +allocates/deallocates memory. That means the DB file you can open now might not +be the DB file you can open in a hour, given the same process. + +On Windows, You can see the see the precise maximum mapping size by using the +SysInternals tool VMMap, then selecting your Python process, then selecting the +"free" row, then sorting by size. + +This is not a problem at all for 64-bit processes. Interface +++++++++ .. py:function:: lmdb.open(path, **kwargs) - + Shortcut for :py:class:`Environment` constructor. .. autofunction:: lmdb.version @@ -655,21 +681,3 @@ .. include:: ../LICENSE :literal: - - -.. raw:: html - - <script type="text/javascript"> - var _paq = _paq || []; - _paq.push(["trackPageView"]); - _paq.push(["enableLinkTracking"]); - - (function() { - var u=(("https:" == document.location.protocol) ? "https" : "http") + "://37.187.23.96/tr/"; - _paq.push(["setTrackerUrl", u+"ep"]); - _paq.push(["setSiteId", "2"]); - var d=document, g=d.createElement("script"), s=d.getElementsByTagName("script")[0]; g.type="text/javascript"; - g.defer=true; g.async=true; g.src=u+"js"; s.parentNode.insertBefore(g,s); - })(); - </script> - <noscript><p><img src="http://37.187.23.96/tr/ep?idsite=2"; style="border:0" alt="" /></p></noscript> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/lmdb/__init__.py new/lmdb-0.96/lmdb/__init__.py --- old/lmdb-0.95/lmdb/__init__.py 2019-06-08 02:57:33.000000000 +0200 +++ new/lmdb-0.96/lmdb/__init__.py 2019-07-15 04:24:45.000000000 +0200 @@ -50,10 +50,5 @@ from lmdb.cffi import __all__ from lmdb.cffi import __doc__ -__version__ = '0.95' +__version__ = '0.96' -# Hack to support Python v2.5 'python -mlmdb' -if __name__ == '__main__': - import lmdb.tool - import atexit - atexit.register(lmdb.tool.main) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/lmdb/_config.py new/lmdb-0.96/lmdb/_config.py --- old/lmdb-0.95/lmdb/_config.py 2019-06-08 02:58:23.000000000 +0200 +++ new/lmdb-0.96/lmdb/_config.py 2019-07-15 04:25:00.000000000 +0200 @@ -1,2 +1,2 @@ -CONFIG = dict((('extra_compile_args', ['-UNDEBUG']), ('extra_sources', ['lib/mdb.c', 'lib/midl.c']), ('extra_library_dirs', []), ('extra_include_dirs', ['lib/py-lmdb', 'lib']), ('libraries', []))) +CONFIG = dict((('extra_compile_args', ['-UNDEBUG', '-w']), ('extra_sources', ['lib/mdb.c', 'lib/midl.c']), ('extra_library_dirs', []), ('extra_include_dirs', ['lib/py-lmdb', 'lib']), ('libraries', []))) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/lmdb/cffi.py new/lmdb-0.96/lmdb/cffi.py --- old/lmdb-0.95/lmdb/cffi.py 2019-06-08 02:57:33.000000000 +0200 +++ new/lmdb-0.96/lmdb/cffi.py 2019-07-15 04:24:30.000000000 +0200 @@ -275,6 +275,10 @@ static int pymdb_cursor_put(MDB_cursor *cursor, char *key_s, size_t keylen, char *val_s, size_t vallen, int flags); + + // Prefaults a range + static void preload(int rc, void *x, size_t size); + ''' _CFFI_VERIFY = ''' @@ -289,7 +293,6 @@ { MDB_val key = {keylen, key_s}; int rc = mdb_get(txn, dbi, &key, val_out); - preload(rc, val_out->mv_data, val_out->mv_size); return rc; } @@ -324,7 +327,6 @@ MDB_val tmp_data = {data_len, data_s}; int rc = mdb_cursor_get(cursor, &tmp_key, &tmp_data, op); if(! rc) { - preload(rc, tmp_data.mv_data, tmp_data.mv_size); *key = tmp_key; *data = tmp_data; } @@ -338,6 +340,7 @@ MDB_val tmpval = {vallen, val_s}; return mdb_cursor_put(cursor, &tmpkey, &tmpval, flags); } + ''' if not lmdb._reading_docs(): @@ -535,6 +538,9 @@ """Convert a MDB_val cdata to Python bytes.""" return _ffi.buffer(mv.mv_data, mv.mv_size)[:] +def preload(mv): + _lib.preload(0, mv.mv_data, mv.mv_size) + def enable_drop_gil(): """Deprecated.""" @@ -559,6 +565,12 @@ simultaneous write transaction is allowed, however there is no limit on the number of read transactions even when a write transaction exists. + This class is aliased to `lmdb.open`. + + It is a serious error to have open the same LMDB file in the same process at + the same time. Failure to heed this may lead to data corruption and + interpreter crash. + Equivalent to `mdb_env_open() <http://symas.com/mdb/doc/group__mdb.html#ga1fe2740e25b1689dc412e7b9faadba1b>`_ @@ -1401,6 +1413,8 @@ if rc == _lib.MDB_NOTFOUND: return default raise _error("mdb_cursor_get", rc) + + preload(self._val) return self._to_py(self._val) def put(self, key, value, dupdata=True, overwrite=True, append=False, @@ -1424,7 +1438,9 @@ overwrite any existing matching key. `overwrite`: - If ``False``, do not overwrite any existing matching key. + If ``False``, do not overwrite any existing matching key. If + False and writing to a dupsort=True database, this will not add a value + to the key and this function will return False. `append`: If ``True``, append the pair to the end of the database without @@ -1642,6 +1658,7 @@ # Must refresh `key` and `val` following mutation. if self._last_mutation != self.txn._mutations: self._cursor_get(_lib.MDB_GET_CURRENT) + preload(self._val) return self._to_py(self._val) def item(self): @@ -1649,6 +1666,7 @@ # Must refresh `key` and `val` following mutation. if self._last_mutation != self.txn._mutations: self._cursor_get(_lib.MDB_GET_CURRENT) + preload(self._val) return self._to_py(self._key), self._to_py(self._val) def _iter(self, op, keys, values): @@ -2161,6 +2179,7 @@ """ if self.db._flags & _lib.MDB_DUPSORT: if self._cursor_get_kv(_lib.MDB_SET_KEY, key, EMPTY_BYTES): + preload(self._val) old = _mvstr(self._val) self.delete(True) else: @@ -2178,6 +2197,7 @@ raise _error("mdb_cursor_put", rc) self._cursor_get(_lib.MDB_GET_CURRENT) + preload(self._val) old = _mvstr(self._val) rc = _lib.pymdb_cursor_put(self._cur, key, keylen, val, len(val), 0) self.txn._mutations += 1 @@ -2198,6 +2218,7 @@ Bytestring key to delete. """ if self._cursor_get_kv(_lib.MDB_SET_KEY, key, EMPTY_BYTES): + preload(self._val) old = _mvstr(self._val) rc = _lib.mdb_cursor_del(self._cur, 0) self.txn._mutations += 1 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/lmdb/cpython.c new/lmdb-0.96/lmdb/cpython.c --- old/lmdb-0.95/lmdb/cpython.c 2019-06-08 02:57:33.000000000 +0200 +++ new/lmdb-0.96/lmdb/cpython.c 2019-07-15 04:24:30.000000000 +0200 @@ -28,6 +28,15 @@ #include <errno.h> #include <stdarg.h> #include <string.h> + +#ifdef _WIN32 +# define bool int +# define true 1 +# define false 0 +#else +# include <stdbool.h> +#endif + #include <sys/stat.h> #include "Python.h" @@ -1933,7 +1942,6 @@ Py_BEGIN_ALLOW_THREADS; rc = mdb_cursor_get(self->curs, &self->key, &self->val, op); - preload(rc, self->val.mv_data, self->val.mv_size); Py_END_ALLOW_THREADS; self->positioned = rc == 0; @@ -2072,6 +2080,7 @@ PyObject *key; PyObject *val; PyObject *tup; + int rc = 0; if(! self->valid) { return err_invalid(); @@ -2084,6 +2093,9 @@ as_buffer = self->trans->flags & TRANS_BUFFERS; key = obj_from_val(&self->key, as_buffer); + Py_BEGIN_ALLOW_THREADS; + preload(rc, self->val.mv_data, self->val.mv_size); + Py_END_ALLOW_THREADS; val = obj_from_val(&self->val, as_buffer); tup = PyTuple_New(2); if(tup && key && val) { @@ -2335,7 +2347,7 @@ static PyObject * do_cursor_replace(CursorObject *self, MDB_val *key, MDB_val *val) { - int rc; + int rc = 0; PyObject *old; MDB_val newval = *val; @@ -2345,6 +2357,9 @@ return NULL; } if(self->positioned) { + Py_BEGIN_ALLOW_THREADS; + preload(rc, self->val.mv_data, self->val.mv_size); + Py_END_ALLOW_THREADS; if(! ((old = obj_from_val(&self->val, 0)))) { return NULL; } @@ -2420,7 +2435,7 @@ {"key", ARG_BUF, OFFSET(cursor_pop, key)}, }; PyObject *old; - int rc; + int rc = 0; static PyObject *cache = NULL; if(parse_args(self->valid, SPECSIZE(), argspec, &cache, args, kwds, &arg)) { @@ -2434,6 +2449,9 @@ if(! self->positioned) { Py_RETURN_NONE; } + Py_BEGIN_ALLOW_THREADS; + preload(rc, self->val.mv_data, self->val.mv_size); + Py_END_ALLOW_THREADS; if(! ((old = obj_from_val(&self->val, 0)))) { return NULL; } @@ -2550,9 +2568,13 @@ } /* Must refresh `key` and `val` following mutation. */ if(self->last_mutation != self->trans->mutations && - _cursor_get_c(self, MDB_GET_CURRENT)) { + _cursor_get_c(self, MDB_GET_CURRENT)) { return NULL; } + Py_BEGIN_ALLOW_THREADS; + preload(0, self->val.mv_data, self->val.mv_size); + Py_END_ALLOW_THREADS; + return obj_from_val(&self->val, self->trans->flags & TRANS_BUFFERS); } @@ -3343,7 +3365,7 @@ }; CursorObject *cursor; PyObject *old; - int rc; + int rc = 0; static PyObject *cache = NULL; if(parse_args(self->valid, SPECSIZE(), argspec, &cache, args, kwds, &arg)) { @@ -3366,6 +3388,8 @@ Py_DECREF((PyObject *)cursor); Py_RETURN_NONE; } + + preload(rc, cursor->val.mv_data, cursor->val.mv_size); if(! ((old = obj_from_val(&cursor->val, 0)))) { Py_DECREF((PyObject *)cursor); return NULL; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/lmdb.egg-info/PKG-INFO new/lmdb-0.96/lmdb.egg-info/PKG-INFO --- old/lmdb-0.95/lmdb.egg-info/PKG-INFO 2019-06-08 02:58:23.000000000 +0200 +++ new/lmdb-0.96/lmdb.egg-info/PKG-INFO 2019-07-15 04:25:00.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: lmdb -Version: 0.95 +Version: 0.96 Summary: Universal Python binding for the LMDB 'Lightning' Database Home-page: http://github.com/dw/py-lmdb/ Author: David Wilson diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/tests/cursor_test.py new/lmdb-0.96/tests/cursor_test.py --- old/lmdb-0.95/tests/cursor_test.py 2016-03-17 11:25:53.000000000 +0100 +++ new/lmdb-0.96/tests/cursor_test.py 2019-07-15 04:24:30.000000000 +0200 @@ -24,6 +24,7 @@ from __future__ import absolute_import from __future__ import with_statement +import sys import unittest import testlib @@ -181,7 +182,7 @@ assert B('x') == self.c.replace(B('a'), B('y')) -class ContextManagerTest(CursorTestBase): +class ContextManagerTest2(CursorTestBase): def test_enter(self): with self.c as c: assert c is self.c @@ -217,6 +218,53 @@ self.assertRaises(Exception, lambda: self.c.put(B('a'), B('a'))) +GiB = 1024 * 1024 * 1024 + +class PreloadTest(CursorTestBase): + + def setUp(self, redo=False): + env_args = {'writemap': True, 'map_size': GiB} + if not redo: + self.path, self.env = testlib.temp_env(**env_args) + else: + self.path, self.env = testlib.temp_env(path=self.path, **env_args) + self.txn = self.env.begin(write=True) + self.c = self.txn.cursor() + + @unittest.skipIf(sys.platform != 'linux', "test only works on Linux") + def test_preload(self): + """ + Test that reading just the key doesn't prefault the value contents, but + reading the data does. + """ + + import resource + self.c.put(B('a'), B('a') * (256 * 1024 * 1024)) + self.txn.commit() + self.env.close() + # Just reading the data is obviously going to fault the value in. The + # point is to fault it in while the GIL is unlocked. We use the buffers + # API so that we're not actually copying the data in. This doesn't + # actually show that we're prefaulting with the GIL unlocked, but it + # does prove we prefault at all, and in 2 correct places. + self.path, self.env = testlib.temp_env(path=self.path, writemap=True) + self.txn = self.env.begin(write=True, buffers=True) + self.c = self.txn.cursor() + minflts_before = resource.getrusage(resource.RUSAGE_THREAD)[6] + self.c.set_key(B('a')) + assert self.c.key() == B('a') + minflts_after_key = resource.getrusage(resource.RUSAGE_THREAD)[6] + + self.c.value() + minflts_after_value = resource.getrusage(resource.RUSAGE_THREAD)[6] + + epsilon = 5 + + # Setting the position doesn't prefault the data + assert minflts_after_key - minflts_before < epsilon + + # Getting the value does prefault the data, even if we only get it by pointer + assert minflts_after_value - minflts_after_key > 1000 if __name__ == '__main__': unittest.main() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lmdb-0.95/tests/testlib.py new/lmdb-0.96/tests/testlib.py --- old/lmdb-0.95/tests/testlib.py 2016-10-17 20:33:34.000000000 +0200 +++ new/lmdb-0.96/tests/testlib.py 2019-07-15 04:24:30.000000000 +0200 @@ -116,8 +116,6 @@ B = lambda s: s except TypeError: # Python3.x, requires encoding parameter. B = lambda s: bytes(s, 'ascii') -except NameError: # Python<=2.5. - B = lambda s: s # BL('s1', 's2') -> ['bytes1', 'bytes2'] BL = lambda *args: list(map(B, args))
