Re: New Python bindings (was: Crash with Python bindings)
David Bremner writes: > That's not an itch I personally have, but as I said in the next > paragraph, if someone want's to take on the project of maintaining a > wheel, we'll render the same kind of assistance we give *BSD/Linux/MacOS > package maintainers. We're happy to look at (reasonable) things we can > do to make downstream projects life easier. Fair enough. No problem. I am going to assume that the notmuch library is reasonable stable, and backward incompatable changes are kept to a minimum with proper updating of the shared library SONAME. If this is not the case, ignore the rest of this email. Ideally the python bindings should be in a git repository that is separate from the C library. This means you don't have to release new python bindings for every new source code release of notmuch. You only need to make a new release if supporting new features or a new release that breaks backword compatability. It also will make it easier to build the python libraries standalone using the installed versions of the C library, which I suspect might make pypi support a lot easier. I might be able to get time to look at this sometime myself, if nobody beats me to it. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Brian May writes: > David Bremner writes: > >> We tried this before, and it didn't work out very well. Bindings tend to >> depend on a strict matching of versions with the underlying library, so >> distributing them seperately doesn't really make sense to me. You need >> the underlying libraries, so why not get the matching bindings from the >> same place? We found that the situation was exacerbated by the fact >> that no-one cared about updating the bindings on pypi. > [...] > Unfortunately, I think many people will not even consider using a python > library unless it has up-to-date bindings available on pypi. That's not an itch I personally have, but as I said in the next paragraph, if someone want's to take on the project of maintaining a wheel, we'll render the same kind of assistance we give *BSD/Linux/MacOS package maintainers. We're happy to look at (reasonable) things we can do to make downstream projects life easier. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
David Bremner writes: > We tried this before, and it didn't work out very well. Bindings tend to > depend on a strict matching of versions with the underlying library, so > distributing them seperately doesn't really make sense to me. You need > the underlying libraries, so why not get the matching bindings from the > same place? We found that the situation was exacerbated by the fact > that no-one cared about updating the bindings on pypi. I believe that is the purpose of Python Wheels. https://pythonwheels.com/ pypi is the defecto standard for distributing Python code for use in Python applications. It means packages that use notmuch just need to list it as a dependancy in 'requirements.txt' and a 'pip install -r requirements.txt' will install everything required (if inside a virtualenv no root access required even). There are also various solutions to get automatic deploys to pypi, for example through travis: https://docs.travis-ci.com/user/deployment/pypi/ Unfortunately, I think many people will not even consider using a python library unless it has up-to-date bindings available on pypi. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 28 2018, David Bremner wrote: > Brian May writes: > >> I can into this thread late. However, my priorities for python bindings >> would be: > > [...] >> * Packages should be available from pypi.python.org >> > > We tried this before, and it didn't work out very well. Bindings tend to > depend on a strict matching of versions with the underlying library, so > distributing them seperately doesn't really make sense to me. You need > the underlying libraries, so why not get the matching bindings from the > same place? We found that the situation was exacerbated by the fact > that no-one cared about updating the bindings on pypi. I did build a #if LIBNOTMUCH_MAJOR_VERSION < 5 #error libnotmuch version not supported by notdb #endif into my current bindings which kind of allows you to do this to some, hopefully reasonable, level. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 28 2018, Brian May wrote: > Justus Winter writes: > >>> This is exactly what I have fixed in my alternative bindings which I >>> created around the end of last year [0]. So we do have an idea of how >>> to fix this, at the time I said I do believe that it's possible to also >>> do this for the existing bindings even though it is a lot of work. >>> After some talking between dkg and me we got to a way forward which >>> proposed this, but I must admit that after messing a little with getting >>> a pytest run integrated into the notmuch test-suite instead of using tox >>> I lost momentum on the project and didn't advance any further. >> >> I'm sorry that I didn't speak up when you announced your work. I'm >> actually excited about a new set of bindings for Python. I agree with >> using cffi. I briefly looked at the code, and I believe it is much >> nicer than what we currently have. > > I can into this thread late. However, my priorities for python bindings > would be: > > * I don't care about anything less then Python 3.5. > * Reliable Python 3.6 support important. > * Packages should be available from pypi.python.org Well, the 1st two should already be covered on my https://github.com/flub/notmuch/tree/cffi branch. There's obviously still room for improvement. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Brian May writes: > I can into this thread late. However, my priorities for python bindings > would be: [...] > * Packages should be available from pypi.python.org > We tried this before, and it didn't work out very well. Bindings tend to depend on a strict matching of versions with the underlying library, so distributing them seperately doesn't really make sense to me. You need the underlying libraries, so why not get the matching bindings from the same place? We found that the situation was exacerbated by the fact that no-one cared about updating the bindings on pypi. Projects like numpy seem to get around this by distributing compiled shared libraries on pypi. That's fine if someone wants to do it, but it looks like "just another distro" to me, and not really an upstream problem. I guess we'd entertain minor tweaks to the build system to support that, but probably not a wholesale conversion. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Justus Winter writes: >> This is exactly what I have fixed in my alternative bindings which I >> created around the end of last year [0]. So we do have an idea of how >> to fix this, at the time I said I do believe that it's possible to also >> do this for the existing bindings even though it is a lot of work. >> After some talking between dkg and me we got to a way forward which >> proposed this, but I must admit that after messing a little with getting >> a pytest run integrated into the notmuch test-suite instead of using tox >> I lost momentum on the project and didn't advance any further. > > I'm sorry that I didn't speak up when you announced your work. I'm > actually excited about a new set of bindings for Python. I agree with > using cffi. I briefly looked at the code, and I believe it is much > nicer than what we currently have. I can into this thread late. However, my priorities for python bindings would be: * I don't care about anything less then Python 3.5. * Reliable Python 3.6 support important. * Packages should be available from pypi.python.org I have found the current official bindings unreliable with Python 3.x, and tend to cause aborts on exiting and/or fail to save updates. As such, for now I have downgraded to Python 2.7 for now. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 21 2018, Justus Winter wrote: > > Floris Bruynooghe writes: > >> This is exactly what I have fixed in my alternative bindings which I >> created around the end of last year [0]. So we do have an idea of how >> to fix this, at the time I said I do believe that it's possible to also >> do this for the existing bindings even though it is a lot of work. >> After some talking between dkg and me we got to a way forward which >> proposed this, but I must admit that after messing a little with getting >> a pytest run integrated into the notmuch test-suite instead of using tox >> I lost momentum on the project and didn't advance any further. > > I'm sorry that I didn't speak up when you announced your work. I'm > actually excited about a new set of bindings for Python. I agree with > using cffi. I briefly looked at the code, and I believe it is much > nicer than what we currently have. Nice to hear, thanks! > I trust that it works fine with Python 3, does it? The version I made so far *only* works on Python 3. Mostly because it was easier, but it also allows some API niceties. > The testsuite cannot depend on pulling stuff from the net simply because > build servers typically do not have access to it. That is a hard > requirement. Sure I understand that. See another part of this thread on this though. Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
New Python bindings (was: Crash with Python bindings)
Hi Floris :) Floris Bruynooghe writes: > This is exactly what I have fixed in my alternative bindings which I > created around the end of last year [0]. So we do have an idea of how > to fix this, at the time I said I do believe that it's possible to also > do this for the existing bindings even though it is a lot of work. > After some talking between dkg and me we got to a way forward which > proposed this, but I must admit that after messing a little with getting > a pytest run integrated into the notmuch test-suite instead of using tox > I lost momentum on the project and didn't advance any further. I'm sorry that I didn't speak up when you announced your work. I'm actually excited about a new set of bindings for Python. I agree with using cffi. I briefly looked at the code, and I believe it is much nicer than what we currently have. I trust that it works fine with Python 3, does it? The testsuite cannot depend on pulling stuff from the net simply because build servers typically do not have access to it. That is a hard requirement. I don't remember what kind of tests there are for the current bindings (are there any...?), but shouldn't these just continue to work for the time being? Thanks, Justus signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
Daniel Kahn Gillmor writes: > On Fri 2018-03-16 19:30:37 +0100, Floris Bruynooghe wrote: >> If someone can hook pytest runs with various python versions into the >> notmuch test suit I'd be very much obliged and probably have another go >> at this as it's still an interesting problem and gives a nice way >> forward. > > I don't really know what this request means -- so maybe that means that > i'm not the right person for the task, which is fine. > > but it's also possible that the right person for the task *also* doesn't > know what you're asking for, so if you could elaborate a bit further > i think that would be super helpful :) Fair enough :) Here a somewhat more long-form version of this: Before even attempting to refactor the existing bindings to use cffi as a backend instead off ctypes and/or adding the changes needed to track the lifetime of objects correctly I would like to be able to write full unitttest-level tests for the bindings to be able to guarantee that no user-level APIs are broken. In my version of the bindings I did this the traditional Python way: using pytest for writing unittest and using tox to invoke the tests for the various supported versions of python. One of the feedback items I got from the patch I sent last time was that the project would be reluctant to adopt this and would like to avoid virtualenv and pip with their behaviour of downloading things over the network. Instead wishing for it to use a system python which should have the available tools already installed (i.e. pytest). And all this just integrated in the existing test suite. So my last attempt at this looks like I made a test/T391-pytest.sh file with the idea of running a subtest for each python version, with the subtest being a ``pythonX.Y -m pytest bindings/python/tests/`` so it'd run the entire test. To be nice this also needs to be hooked up so that the subtests get skipped when a python version is not available, or is missing pytest itself. So while trying to figure this out is where I got distracted last time and started working more on other things. Kind Regards, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
On Fri 2018-03-16 19:30:37 +0100, Floris Bruynooghe wrote: > If someone can hook pytest runs with various python versions into the > notmuch test suit I'd be very much obliged and probably have another go > at this as it's still an interesting problem and gives a nice way > forward. I don't really know what this request means -- so maybe that means that i'm not the right person for the task, which is fine. but it's also possible that the right person for the task *also* doesn't know what you're asking for, so if you could elaborate a bit further i think that would be super helpful :) thanks for looking into this further! --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
Hi all, David Bremner writes: > "W. Trevor King" writes: > >> you can avoid the abort (which happens when q.__del__ is called after >> db.__del__). We could make that sort of cleanup easier with context >> managers for Query objects (we have them for databases since [3]), and >> they look like the only object that keep an internal database >> reference: >> >> with Database() as db: >> with Query(db, "*") as q: >> # do something with q >> db.close() >> > > I'm reminded [1] that this problem still exists. If noone has any idea > of a fix, should we document one of the workarounds? This is exactly what I have fixed in my alternative bindings which I created around the end of last year [0]. So we do have an idea of how to fix this, at the time I said I do believe that it's possible to also do this for the existing bindings even though it is a lot of work. After some talking between dkg and me we got to a way forward which proposed this, but I must admit that after messing a little with getting a pytest run integrated into the notmuch test-suite instead of using tox I lost momentum on the project and didn't advance any further. If someone can hook pytest runs with various python versions into the notmuch test suit I'd be very much obliged and probably have another go at this as it's still an interesting problem and gives a nice way forward. Cheers, Floris [0] https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
David Bremner writes: > "W. Trevor King" writes: > >> you can avoid the abort (which happens when q.__del__ is called after >> db.__del__). We could make that sort of cleanup easier with context >> managers for Query objects (we have them for databases since [3]), and >> they look like the only object that keep an internal database >> reference: >> >> with Database() as db: >> with Query(db, "*") as q: >> # do something with q >> db.close() So while this shouldn't crash of course, this code is wrong. The context manager closes the database, so doing db.close() at the end of the block is superfluous. > I'm reminded [1] that this problem still exists. If noone has any idea > of a fix, should we document one of the workarounds? I don't remember the details, but the different semantics of garbage collection and talloc was problematic. In essence, every wrapping Python object must keep a reference to its parent (as in parent in the talloc hierarchy). The bug report [1] sounds like that the crash happens at interpreter shutdown, where iirc the objects destructors are called in arbitrary order. Justus signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
"W. Trevor King" writes: > you can avoid the abort (which happens when q.__del__ is called after > db.__del__). We could make that sort of cleanup easier with context > managers for Query objects (we have them for databases since [3]), and > they look like the only object that keep an internal database > reference: > > with Database() as db: > with Query(db, "*") as q: > # do something with q > db.close() > I'm reminded [1] that this problem still exists. If noone has any idea of a fix, should we document one of the workarounds? [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=893057 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Binding access to ~/.notmuch-config (was: Crash with Python bindings)
On Tue, Jan 12, 2016 at 03:03:15PM -0400, David Bremner wrote: > Nothing to do with Konrad's crash, but I consider the fact that the > python bindings read ~/.notmuch-config to be a kind of layering > violation, since that file belongs to the CLI, while the bindings > are supposed to provide access to libnotmuch. I think of ~/.notmuch-config as being shared between all client code, and in that view it makes sense to have both the CLI and Python bindings (and other bindings) access it to figure out how to configure their library access calls. Having a separate config file for each client to point at the default database path seems like more trouble than it's worth, as does adding a library function for “reach into some local config and return the default database path”. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
David Bremner writes: >> No path means path=None, which stands for the path from >> ~/.notmuch-config. That's exactly what I want. Is there some reason not >> to rely on this mechanism? > > Oh sorry, I'm (obviously) not that familiar with the python bindings. Nothing to do with Konrad's crash, but I consider the fact that the python bindings read ~/.notmuch-config to be a kind of layering violation, since that file belongs to the CLI, while the bindings are supposed to provide access to libnotmuch. Whether this is a real problem or just an aesthetic one, I'm not sure, but I thought I'd mention it since we are thinking of various config related issues. Obviously the location of the database is not one of the things it makes sense to store in the database. I can imagine scenarios where the bindings might be usable without the CLI, but they seem fairly artificial so far, since it seems like almost everyone needs notmuch-new / notmuch-insert. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
On Tue, Jan 12, 2016 at 03:23:46PM +0100, Konrad Hinsen wrote: > Hi Justus, > > > So I guess what happens is that Python3 changed how the > > interpreter environment is torn down and they actually destroy the > > 'q' object. If that is so, then your data is indeed safe. > > That reminds me of a recent change in object finalization in Python > 3: > > https://www.python.org/dev/peps/pep-0442/ I'm pretty sure that is what's going on. The PEP landed in Python 3.4, explaining why I don't see this issue in 3.3 [1]. And it has [2]: In particular, this PEP obsoletes the current guideline that "objects with a __del__ method should not be part of a reference cycle". I'm not sure what the best way is to fix __del__ in our Python bindings, but if you manage them explicitly: db = Database() q = Query(db, "*") del q db.close() del db you can avoid the abort (which happens when q.__del__ is called after db.__del__). We could make that sort of cleanup easier with context managers for Query objects (we have them for databases since [3]), and they look like the only object that keep an internal database reference: with Database() as db: with Query(db, "*") as q: # do something with q db.close() Cheers, Trevor [1]: id:20160112180813.ga20...@odin.tremily.us [2]: https://www.python.org/dev/peps/pep-0442/#impact [3]: 36ce7e3c (python: implement the context manager protocol for database objects, 2012-02-15, v0.12) -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
On Tue, Jan 12, 2016 at 10:41:57AM +0100, Konrad Hinsen wrote: > -- > from notmuch import Query, Database > > def foo(bar): > pass > > db = Database() > q = Query(db, "*") > db.close() > -- > > Running this script (Python 3.5, MacOS X) yields: > > [1]22478 abort pydev3 Temp/notmuch_test.py > … > The strange part is that what causes the crash is the presence of the > function foo(), even though it is never called. Remove foo and the > script runs fine. It is also necessary to create a Query object. Adding some more data-points, I see the same results with Python 3.4.3 on Gentoo, although stderr just gets “Aborted”. All permutations seem to work on Python 3.3.5. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
Konrad Hinsen writes: > David Bremner writes: > >>> from notmuch import Query, Database >>> >>> def foo(bar): >>> pass >>> >>> db = Database() >>> q = Query(db, "*") >>> db.close() >> >> Do you really call the constructor without a path? Or are you censoring >> the script for some reason? > > No path means path=None, which stands for the path from > ~/.notmuch-config. That's exactly what I want. Is there some reason not > to rely on this mechanism? Oh sorry, I'm (obviously) not that familiar with the python bindings. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
Hi Justus, > So I guess what happens is that Python3 changed how the interpreter > environment is torn down and they actually destroy the 'q' object. If > that is so, then your data is indeed safe. That reminds me of a recent change in object finalization in Python 3: https://www.python.org/dev/peps/pep-0442/ I'll look into that, thanks for the gdb exploration! Cheers, Konrad. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
David Bremner writes: >> from notmuch import Query, Database >> >> def foo(bar): >> pass >> >> db = Database() >> q = Query(db, "*") >> db.close() > > Do you really call the constructor without a path? Or are you censoring > the script for some reason? No path means path=None, which stands for the path from ~/.notmuch-config. That's exactly what I want. Is there some reason not to rely on this mechanism? Cheers, Konrad. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Crash with Python bindings
Konrad Hinsen writes: > Hi everyone, > > I have been writing quite a few Python scripts for notmuch before > running into a strange bug. Here is a minimal script producing it: > > -- > from notmuch import Query, Database > > def foo(bar): > pass > > db = Database() > q = Query(db, "*") > db.close() Do you really call the constructor without a path? Or are you censoring the script for some reason? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Crash with Python bindings
Hi everyone, I have been writing quite a few Python scripts for notmuch before running into a strange bug. Here is a minimal script producing it: -- from notmuch import Query, Database def foo(bar): pass db = Database() q = Query(db, "*") db.close() -- Running this script (Python 3.5, MacOS X) yields: [1]22478 abort pydev3 Temp/notmuch_test.py The crash actually happens *after* db.close(), when the Python interpreter exists, and therefore I suspect that no data is lost, but I hesitate to use scripts with that behavior in production use. The strange part is that what causes the crash is the presence of the function foo(), even though it is never called. Remove foo and the script runs fine. It is also necessary to create a Query object. The combination of a function definition (any) and the creation of a Query object yields the crash. This looks like a memory management issue, but I didn't explore it any further. Cheers, Konrad. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
