Re: New Python bindings (notmuch2 module) fail to exclude tags
Floris Bruynooghe writes: > > I have a database with 3 messages organised in 2 threads and the > following tags: > > msg1: all > +- msg3: all, spam > msg2: all > > I query '*', so all messages with exclude_tags=['spam']. Querying this > with various flags gives me: > > NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect > NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect > NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3 > NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3 > > Did I misunderstand the docs? Are these results correct? It sounds a bit strange. Can you check against the results of "notmuch search --output=messages", with the corresponding flag? The numbers would make more sense if searching for threads rather than messages. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: New Python bindings (notmuch2 module) fail to exclude tags
Hi All,
On Sat 21 Nov 2020 at 18:51 -0300, Jorge P. de Morais Neto wrote:
> Hi Floris.
>
> Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:
>
>> Looking at the implementation I don't seem much that could have gone
>> wrong. However I did notice the bindings fail to check the return code
>> in one call where it probably should, you could try with this patch?
>>
>> diff --git a/bindings/python-cffi/notmuch2/_database.py
>> b/bindings/python-cffi/notmuch2/_database.py
>> index 5ab0f20a..5dbfe68e 100644
>> --- a/bindings/python-cffi/notmuch2/_database.py
>> +++ b/bindings/python-cffi/notmuch2/_database.py
>> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
>> for tag in exclude_tags:
>> if isinstance(tag, str):
>> tag = str.encode('utf-8')
>> -capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> +ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> +if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
>> + capi.lib.NOTMUCH_STATUS_IGNORED]:
>> +raise errors.NotmuchError(ret)
>> return querymod.Query(self, query_p)
>>
>> def messages(self, query, *,
>>
> After applying your patch, the call to nm_db.count_messages() fails:
> AttributeError: cffi library 'notmuch2._capi' has no function, constant
> or global variable named 'NOTMUCH_STATUS_IGNORED'
>
> However, I then found the bug. The patch below fixes it.
>
> $ diff -u notmuch2-orig/_database.py notmuch2/_database.py
> --- notmuch2-orig/_database.py2020-11-21 18:02:17.560240619 -0300
> +++ notmuch2/_database.py 2020-11-21 18:43:44.827879141 -0300
> @@ -578,7 +578,7 @@
> if exclude_tags is not None:
> for tag in exclude_tags:
> if isinstance(tag, str):
> -tag = str.encode('utf-8')
> +tag = tag.encode('utf-8')
> capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> return querymod.Query(self, query_p)
>
>
> However, I think you should *also* add the error checking. The only
> reason my patch omits error checking is that I don't know how to define
> capi.lib.NOTMUCH_STATUS_IGNORED.
Yup, that was missing from _build.py so wouldn't have been available.
Before fixing this I tried to write a test, but I don't know if this
behaves correctly:
I have a database with 3 messages organised in 2 threads and the
following tags:
msg1: all
+- msg3: all, spam
msg2: all
I query '*', so all messages with exclude_tags=['spam']. Querying this
with various flags gives me:
NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect
NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect
NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3
NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3
Did I misunderstand the docs? Are these results correct?
Cheers,
Floris
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: New Python bindings (notmuch2 module) fail to exclude tags
Hi Floris.
Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:
> Looking at the implementation I don't seem much that could have gone
> wrong. However I did notice the bindings fail to check the return code
> in one call where it probably should, you could try with this patch?
>
> diff --git a/bindings/python-cffi/notmuch2/_database.py
> b/bindings/python-cffi/notmuch2/_database.py
> index 5ab0f20a..5dbfe68e 100644
> --- a/bindings/python-cffi/notmuch2/_database.py
> +++ b/bindings/python-cffi/notmuch2/_database.py
> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
> for tag in exclude_tags:
> if isinstance(tag, str):
> tag = str.encode('utf-8')
> -capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> +ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> +if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
> + capi.lib.NOTMUCH_STATUS_IGNORED]:
> +raise errors.NotmuchError(ret)
> return querymod.Query(self, query_p)
>
> def messages(self, query, *,
>
After applying your patch, the call to nm_db.count_messages() fails:
AttributeError: cffi library 'notmuch2._capi' has no function, constant or
global variable named 'NOTMUCH_STATUS_IGNORED'
However, I then found the bug. The patch below fixes it.
$ diff -u notmuch2-orig/_database.py notmuch2/_database.py
--- notmuch2-orig/_database.py 2020-11-21 18:02:17.560240619 -0300
+++ notmuch2/_database.py 2020-11-21 18:43:44.827879141 -0300
@@ -578,7 +578,7 @@
if exclude_tags is not None:
for tag in exclude_tags:
if isinstance(tag, str):
-tag = str.encode('utf-8')
+tag = tag.encode('utf-8')
capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
return querymod.Query(self, query_p)
However, I think you should *also* add the error checking. The only
reason my patch omits error checking is that I don't know how to define
capi.lib.NOTMUCH_STATUS_IGNORED. After you apply my patch and also add
the error checking, you could send me the combined patch, so I don't
have to wait for the next release to get the desirable error checking.
Regards
--
-
New Python bindings (notmuch2 module) fail to exclude tags
Hi. I am trying to migrate my Python3 script to the new Python bindings
(notmuch2 module). However, I cannot obtain a count of messages
matching a query excluding messages that have an exclude tag. From the
command line:
$ notmuch count 'is:.bf_spam'
0
The CLI command correctly counts zero messages having '.bf_spam' tag,
because all such messages also have the excluded 'spam' tag. But from
Python:
$ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import notmuch2
>>> nm_db=notmuch2.Database()
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>> omit_excluded=nm_db.EXCLUDE.FALSE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>> omit_excluded=nm_db.EXCLUDE.TRUE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>> omit_excluded=nm_db.EXCLUDE.ALL)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>> omit_excluded=nm_db.EXCLUDE.FLAG)
379
Am I doing something wrong?
Regards
--
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: New Python bindings (notmuch2 module) fail to exclude tags
Hi Jorge,
On Fri 20 Nov 2020 at 12:54 -0300, Jorge P. de Morais Neto wrote:
> Hi. I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module). However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag. From the
> command line:
> $ notmuch count 'is:.bf_spam'
> 0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag. But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44)
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.FLAG)
> 379
Personally I find the description of these flags in notmuch.h not very
clear and don't claim to understand them (this probably explains the
really bad docstring, we should improve those). But I expected to at
least see the EXCLUDE.TRUE one, the default, to work.
Looking at the implementation I don't seem much that could have gone
wrong. However I did notice the bindings fail to check the return code
in one call where it probably should, you could try with this patch?
diff --git a/bindings/python-cffi/notmuch2/_database.py
b/bindings/python-cffi/notmuch2/_database.py
index 5ab0f20a..5dbfe68e 100644
--- a/bindings/python-cffi/notmuch2/_database.py
+++ b/bindings/python-cffi/notmuch2/_database.py
@@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
for tag in exclude_tags:
if isinstance(tag, str):
tag = str.encode('utf-8')
-capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
+ capi.lib.NOTMUCH_STATUS_IGNORED]:
+raise errors.NotmuchError(ret)
return querymod.Query(self, query_p)
def messages(self, query, *,
To experiment you could also raise the exception for
NOTMUCH_STATUS_IGNORED, even though for the bindings to silently swallow
this is probably the best (this is really the bindings being more
limiting than the C lib, if this bothers ppl we could add another
keyword arg to let this raise as well).
While the above patch is probably good, I feel like it's clutching at
straws and unlikely to solve your problem. Could you try to reproduce
this in plain C or maybe even in pretend-plain-c by using the
capi.lib.notmuch_* calls directly?
Cheers,
Floris
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: New Python bindings (notmuch2 module) fail to exclude tags
"Jorge P. de Morais Neto" writes:
> Hi. I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module). However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag. From the
> command line:
> $ notmuch count 'is:.bf_spam'
> 0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag. But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44)
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',),
>>>> omit_excluded=nm_db.EXCLUDE.FLAG)
> 379
>
It looks like a bug, but I'm not very expert with the python bindings,
so I've CCed the author of the new bindings. It would be great if you
could make a small test case that does not rely on your mail store.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/2] doc: add new python bindings to main documentatation tree.
Oh, this is very nice! I've been thinking for a while I should attempt
this. Great to see it being done!
On Sat 11 Jul 2020 at 10:20 -0300, David Bremner wrote:
> A seperate conf.py and doc directory will be needed if someone wants
> to build the bindings docs separately from notmuch.
> ---
> configure | 4
> doc/conf.py | 8
> doc/index.rst | 1 +
> doc/python-bindings.rst | 5 +
> 4 files changed, 18 insertions(+)
> create mode 100644 doc/python-bindings.rst
>
> diff --git a/configure b/configure
> index 177432db..36fe4a9d 100755
> --- a/configure
> +++ b/configure
> @@ -801,6 +801,7 @@ if [ $have_python3 -eq 1 ]; then
> if "$python" -c 'import cffi,setuptools; cffi.FFI().verify()' >/dev/null
> 2>&1; then
> printf "Yes.\n"
> have_python3_cffi=1
> +WITH_PYTHON_DOCS=1
> else
> printf "No (will not install CFFI-based python bindings).\n"
> fi
> @@ -1554,6 +1555,9 @@ EOF
> if [ $WITH_EMACS = "1" ]; then
> printf "tags.add('WITH_EMACS')\n" >> sphinx.config
> fi
> +if [ $WITH_PYTHON_DOCS = "1" ]; then
> +printf "tags.add('WITH_PYTHON')\n" >> sphinx.config
> +fi
> printf "rsti_dir = '%s'\n" $(realpath emacs) >> sphinx.config
>
> # Finally, after everything configured, inform the user how to continue.
> diff --git a/doc/conf.py b/doc/conf.py
> index fdff2a2c..94e266af 100644
> --- a/doc/conf.py
> +++ b/doc/conf.py
> @@ -4,6 +4,8 @@
> import sys
> import os
>
> +extensions = [ 'sphinx.ext.autodoc' ]
> +
> # The suffix of source filenames.
> source_suffix = '.rst'
>
> @@ -22,6 +24,9 @@ for pathdir in ['.', '..']:
> with open(version_file,'r') as infile:
> version=infile.read().replace('\n','')
>
> +# for autodoc
> +sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi',
> 'notmuch2'))
> +
> # read generated config
> for pathdir in ['.', '..']:
> conf_file = os.path.join(location,pathdir,'sphinx.config')
> @@ -50,6 +55,9 @@ else:
> # the docstring include files
> exclude_patterns.append('notmuch-emacs.rst')
>
> +if not tags.has('WITH_PYTHON'):
> +exclude_patterns.append('python-bindings.rst')
> +
> # The name of the Pygments (syntax highlighting) style to use.
> pygments_style = 'sphinx'
>
> diff --git a/doc/index.rst b/doc/index.rst
> index 4440d93a..a3bf3480 100644
> --- a/doc/index.rst
> +++ b/doc/index.rst
> @@ -26,6 +26,7 @@ Contents:
> man7/notmuch-search-terms
> man1/notmuch-show
> man1/notmuch-tag
> + python-bindings
>
> Indices and tables
> ==
> diff --git a/doc/python-bindings.rst b/doc/python-bindings.rst
> new file mode 100644
> index ..e1ad26ad
> --- /dev/null
> +++ b/doc/python-bindings.rst
> @@ -0,0 +1,5 @@
> +Python Bindings
> +===
> +
> +.. automodule:: notmuch2
> + :members:
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 2/2] doc: add new python bindings to main documentatation tree.
A seperate conf.py and doc directory will be needed if someone wants
to build the bindings docs separately from notmuch.
---
configure | 4
doc/conf.py | 8
doc/index.rst | 1 +
doc/python-bindings.rst | 5 +
4 files changed, 18 insertions(+)
create mode 100644 doc/python-bindings.rst
diff --git a/configure b/configure
index 177432db..36fe4a9d 100755
--- a/configure
+++ b/configure
@@ -801,6 +801,7 @@ if [ $have_python3 -eq 1 ]; then
if "$python" -c 'import cffi,setuptools; cffi.FFI().verify()' >/dev/null
2>&1; then
printf "Yes.\n"
have_python3_cffi=1
+WITH_PYTHON_DOCS=1
else
printf "No (will not install CFFI-based python bindings).\n"
fi
@@ -1554,6 +1555,9 @@ EOF
if [ $WITH_EMACS = "1" ]; then
printf "tags.add('WITH_EMACS')\n" >> sphinx.config
fi
+if [ $WITH_PYTHON_DOCS = "1" ]; then
+printf "tags.add('WITH_PYTHON')\n" >> sphinx.config
+fi
printf "rsti_dir = '%s'\n" $(realpath emacs) >> sphinx.config
# Finally, after everything configured, inform the user how to continue.
diff --git a/doc/conf.py b/doc/conf.py
index fdff2a2c..94e266af 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -4,6 +4,8 @@
import sys
import os
+extensions = [ 'sphinx.ext.autodoc' ]
+
# The suffix of source filenames.
source_suffix = '.rst'
@@ -22,6 +24,9 @@ for pathdir in ['.', '..']:
with open(version_file,'r') as infile:
version=infile.read().replace('\n','')
+# for autodoc
+sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi',
'notmuch2'))
+
# read generated config
for pathdir in ['.', '..']:
conf_file = os.path.join(location,pathdir,'sphinx.config')
@@ -50,6 +55,9 @@ else:
# the docstring include files
exclude_patterns.append('notmuch-emacs.rst')
+if not tags.has('WITH_PYTHON'):
+exclude_patterns.append('python-bindings.rst')
+
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
diff --git a/doc/index.rst b/doc/index.rst
index 4440d93a..a3bf3480 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -26,6 +26,7 @@ Contents:
man7/notmuch-search-terms
man1/notmuch-show
man1/notmuch-tag
+ python-bindings
Indices and tables
==
diff --git a/doc/python-bindings.rst b/doc/python-bindings.rst
new file mode 100644
index ..e1ad26ad
--- /dev/null
+++ b/doc/python-bindings.rst
@@ -0,0 +1,5 @@
+Python Bindings
+===
+
+.. automodule:: notmuch2
+ :members:
--
2.27.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: status of the new python bindings
Hi Anton, Great that you're looking at this API! My apologies for the late response, this slipped by me probably as I was bulk marking things as read when I came back from a few weeks away. On Thu 07 May 2020 at 15:57 +0200, Anton Khirnov wrote: > 1) What is the logic behind choosing whether something is exported as > a property or as a method? E.g. Database.needs_upgrade is a property, > while Database.revision() is a method. In my own python code, I tend to > use @property for things that are "cheap" - i.e. do not involve > (significant) IO or heavy computation and methods for those that do. But > both of the above attributes involve library calls, presumably(?) of > similar complexity. Would be nice if this was consistent. Choosing when something should be a property is sometimes indeed tricky and in general I agree you're right to expect property calls to be not too expensive. But I wouldn't go so far to consider every call into libnotmuch as expensive. I think there is also an element of how static something is. Usually I expect properties to behave more like attributes, that is after all the point of them, and thus I don't expect them to change often without explicitly manipulating them. So that could justifies why I choose Database.version as a property while Database.revision() is a method. The revision changes all the time as a side-effect of other things. I think it also justifies generally why tags are exposed as properties. Database.needs_upgrade is indeed a more questionable choice, especially given its naming which is more of a question being asked and thus probably more suitable as a method. It does change rarely, but does so by interaction with another method - Database.upgrade(). So I would agree that this perhaps wasn't the best choice and maybe that was better as a method. > 2) Atomic transactions are now exported as a context manager, which is > nice and convenient for the usual use cases, but AFAIU does not have the > same power. E.g. my tagging script does the tagging as a single atomic > transaction and has a "dry-run" mode in which it omits the end_atomic() > call, which is documented to throw away all the changes. This seems to > not be possible with the new bindings. > Would it be okay to add bindings for explicitly calling > start/end_atomic()? Or is my approach considered invalid? This is indeed a good usecase that I overlooked, I stopped reading at the part where is says being and end atomic must always be used in pairs... But yes, we should add support for this too. What would you think of a .abort() on the context manager instead of directly exposing the start/end? E.g.: db = notmuch2.Database() with db.atomic() as txn: # do stuff txn.abort() However if I read the docs correctly the transaction only actually is aborted once the database is closed? But when I try this: with db.atomic(): # do stuff db.close() I discover that after calling .close() it's very easy to segfault by making other calls, e.g. the above currently segfaults. I thought this wasn't so bad but perhaps .close() should immediately destroy the database too. This would again disallow some funky behaviour that the raw C API allows I guess where some objects might still be valid to use. But this is really hard to get right I think and I'd prefer to vote on the side of safety in the Python bindings. Still, it could be reasonable to make AtomicContext.abort() also close the database and then make sure it doesn't call notmuch_database_end_atomic() on exit. I tried this locally and it seems reasonable. What do you think? Lastly you can do this today by calling the C API directly: db = notmuch2.Database() notmuch2._capi.notmuch_database_being_atomic(db._db_p) Yes, that uses a lot of internal stuff but it's also pretty advanced usage. I'm only suggesting you this to maybe unblock you in the short term. > 3) There seem to be no bindings for notmuch_database_set_config(). There are still bits of the API missing indeed. Would be great to add them but would be even better to have someone who needs it to validate the API. There were recently some good patches posted to expose the config stuff, would be good to see that merged indeed. > 4) The setup for building the documentation seems to be missing. Hmm, yes. The docstrings are written to work with sphinx autodoc but a doc directory with sphinx configured never got added. Perhaps we should do this so that people can at least build docs locally. > Anything else of note that remains to be implemented? Probably. Thanks for caring! Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
status of the new python bindings
Hi, I've started tinkering with the "new" Python bindings (python-cffi / python-notmuch2) and have a couple questions/comments about them: 1) What is the logic behind choosing whether something is exported as a property or as a method? E.g. Database.needs_upgrade is a property, while Database.revision() is a method. In my own python code, I tend to use @property for things that are "cheap" - i.e. do not involve (significant) IO or heavy computation and methods for those that do. But both of the above attributes involve library calls, presumably(?) of similar complexity. Would be nice if this was consistent. 2) Atomic transactions are now exported as a context manager, which is nice and convenient for the usual use cases, but AFAIU does not have the same power. E.g. my tagging script does the tagging as a single atomic transaction and has a "dry-run" mode in which it omits the end_atomic() call, which is documented to throw away all the changes. This seems to not be possible with the new bindings. Would it be okay to add bindings for explicitly calling start/end_atomic()? Or is my approach considered invalid? 3) There seem to be no bindings for notmuch_database_set_config(). 4) The setup for building the documentation seems to be missing. Anything else of note that remains to be implemented? -- Anton Khirnov ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
David Bremner writes: > That's not an itch I personally have, but as I said in the next > paragraph, if someone want's to take on the project of maintaining a > wheel, we'll render the same kind of assistance we give *BSD/Linux/MacOS > package maintainers. We're happy to look at (reasonable) things we can > do to make downstream projects life easier. Fair enough. No problem. I am going to assume that the notmuch library is reasonable stable, and backward incompatable changes are kept to a minimum with proper updating of the shared library SONAME. If this is not the case, ignore the rest of this email. Ideally the python bindings should be in a git repository that is separate from the C library. This means you don't have to release new python bindings for every new source code release of notmuch. You only need to make a new release if supporting new features or a new release that breaks backword compatability. It also will make it easier to build the python libraries standalone using the installed versions of the C library, which I suspect might make pypi support a lot easier. I might be able to get time to look at this sometime myself, if nobody beats me to it. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Brian May writes: > David Bremner writes: > >> We tried this before, and it didn't work out very well. Bindings tend to >> depend on a strict matching of versions with the underlying library, so >> distributing them seperately doesn't really make sense to me. You need >> the underlying libraries, so why not get the matching bindings from the >> same place? We found that the situation was exacerbated by the fact >> that no-one cared about updating the bindings on pypi. > [...] > Unfortunately, I think many people will not even consider using a python > library unless it has up-to-date bindings available on pypi. That's not an itch I personally have, but as I said in the next paragraph, if someone want's to take on the project of maintaining a wheel, we'll render the same kind of assistance we give *BSD/Linux/MacOS package maintainers. We're happy to look at (reasonable) things we can do to make downstream projects life easier. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
David Bremner writes: > We tried this before, and it didn't work out very well. Bindings tend to > depend on a strict matching of versions with the underlying library, so > distributing them seperately doesn't really make sense to me. You need > the underlying libraries, so why not get the matching bindings from the > same place? We found that the situation was exacerbated by the fact > that no-one cared about updating the bindings on pypi. I believe that is the purpose of Python Wheels. https://pythonwheels.com/ pypi is the defecto standard for distributing Python code for use in Python applications. It means packages that use notmuch just need to list it as a dependancy in 'requirements.txt' and a 'pip install -r requirements.txt' will install everything required (if inside a virtualenv no root access required even). There are also various solutions to get automatic deploys to pypi, for example through travis: https://docs.travis-ci.com/user/deployment/pypi/ Unfortunately, I think many people will not even consider using a python library unless it has up-to-date bindings available on pypi. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 28 2018, David Bremner wrote: > Brian May writes: > >> I can into this thread late. However, my priorities for python bindings >> would be: > > [...] >> * Packages should be available from pypi.python.org >> > > We tried this before, and it didn't work out very well. Bindings tend to > depend on a strict matching of versions with the underlying library, so > distributing them seperately doesn't really make sense to me. You need > the underlying libraries, so why not get the matching bindings from the > same place? We found that the situation was exacerbated by the fact > that no-one cared about updating the bindings on pypi. I did build a #if LIBNOTMUCH_MAJOR_VERSION < 5 #error libnotmuch version not supported by notdb #endif into my current bindings which kind of allows you to do this to some, hopefully reasonable, level. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 28 2018, Brian May wrote: > Justus Winter writes: > >>> This is exactly what I have fixed in my alternative bindings which I >>> created around the end of last year [0]. So we do have an idea of how >>> to fix this, at the time I said I do believe that it's possible to also >>> do this for the existing bindings even though it is a lot of work. >>> After some talking between dkg and me we got to a way forward which >>> proposed this, but I must admit that after messing a little with getting >>> a pytest run integrated into the notmuch test-suite instead of using tox >>> I lost momentum on the project and didn't advance any further. >> >> I'm sorry that I didn't speak up when you announced your work. I'm >> actually excited about a new set of bindings for Python. I agree with >> using cffi. I briefly looked at the code, and I believe it is much >> nicer than what we currently have. > > I can into this thread late. However, my priorities for python bindings > would be: > > * I don't care about anything less then Python 3.5. > * Reliable Python 3.6 support important. > * Packages should be available from pypi.python.org Well, the 1st two should already be covered on my https://github.com/flub/notmuch/tree/cffi branch. There's obviously still room for improvement. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings
On Wed, Mar 28 2018, Justus Winter wrote: > Floris Bruynooghe writes: > >> On Wed, Mar 21 2018, Justus Winter wrote: >>> >>> Floris Bruynooghe writes: >>> This is exactly what I have fixed in my alternative bindings which I created around the end of last year [0]. So we do have an idea of how to fix this, at the time I said I do believe that it's possible to also do this for the existing bindings even though it is a lot of work. After some talking between dkg and me we got to a way forward which proposed this, but I must admit that after messing a little with getting a pytest run integrated into the notmuch test-suite instead of using tox I lost momentum on the project and didn't advance any further. >>> >>> I'm sorry that I didn't speak up when you announced your work. I'm >>> actually excited about a new set of bindings for Python. I agree with >>> using cffi. I briefly looked at the code, and I believe it is much >>> nicer than what we currently have. >> >> Nice to hear, thanks! > > Thanks for all the work :) > >>> I trust that it works fine with Python 3, does it? >> >> The version I made so far *only* works on Python 3. Mostly because it >> was easier, but it also allows some API niceties. > > Reasonable choice. Which versions of Python 3 are supported? I am also > writing bindings and I am wondering which versions to target. Personally I consider python3.5, pypy3.5 and python3.6 the ones to target if I have no other constraints, which was the case here. For upstreaming into notmuch proper there are naturally other constraints ;-) But unless you need something specific I think 3.4 is when py3k became the better version than 2.7, everything below that is probably not worth it. All IMHO obviously. Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Brian May writes: > I can into this thread late. However, my priorities for python bindings > would be: [...] > * Packages should be available from pypi.python.org > We tried this before, and it didn't work out very well. Bindings tend to depend on a strict matching of versions with the underlying library, so distributing them seperately doesn't really make sense to me. You need the underlying libraries, so why not get the matching bindings from the same place? We found that the situation was exacerbated by the fact that no-one cared about updating the bindings on pypi. Projects like numpy seem to get around this by distributing compiled shared libraries on pypi. That's fine if someone wants to do it, but it looks like "just another distro" to me, and not really an upstream problem. I guess we'd entertain minor tweaks to the build system to support that, but probably not a wholesale conversion. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
Justus Winter writes: >> This is exactly what I have fixed in my alternative bindings which I >> created around the end of last year [0]. So we do have an idea of how >> to fix this, at the time I said I do believe that it's possible to also >> do this for the existing bindings even though it is a lot of work. >> After some talking between dkg and me we got to a way forward which >> proposed this, but I must admit that after messing a little with getting >> a pytest run integrated into the notmuch test-suite instead of using tox >> I lost momentum on the project and didn't advance any further. > > I'm sorry that I didn't speak up when you announced your work. I'm > actually excited about a new set of bindings for Python. I agree with > using cffi. I briefly looked at the code, and I believe it is much > nicer than what we currently have. I can into this thread late. However, my priorities for python bindings would be: * I don't care about anything less then Python 3.5. * Reliable Python 3.6 support important. * Packages should be available from pypi.python.org I have found the current official bindings unreliable with Python 3.x, and tend to cause aborts on exiting and/or fail to save updates. As such, for now I have downgraded to Python 2.7 for now. -- Brian May https://linuxpenguins.xyz/brian/ ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings
Floris Bruynooghe writes: > On Wed, Mar 21 2018, Justus Winter wrote: >> >> Floris Bruynooghe writes: >> >>> This is exactly what I have fixed in my alternative bindings which I >>> created around the end of last year [0]. So we do have an idea of how >>> to fix this, at the time I said I do believe that it's possible to also >>> do this for the existing bindings even though it is a lot of work. >>> After some talking between dkg and me we got to a way forward which >>> proposed this, but I must admit that after messing a little with getting >>> a pytest run integrated into the notmuch test-suite instead of using tox >>> I lost momentum on the project and didn't advance any further. >> >> I'm sorry that I didn't speak up when you announced your work. I'm >> actually excited about a new set of bindings for Python. I agree with >> using cffi. I briefly looked at the code, and I believe it is much >> nicer than what we currently have. > > Nice to hear, thanks! Thanks for all the work :) >> I trust that it works fine with Python 3, does it? > > The version I made so far *only* works on Python 3. Mostly because it > was easier, but it also allows some API niceties. Reasonable choice. Which versions of Python 3 are supported? I am also writing bindings and I am wondering which versions to target. Cheers, Justus signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: New Python bindings (was: Crash with Python bindings)
On Wed, Mar 21 2018, Justus Winter wrote: > > Floris Bruynooghe writes: > >> This is exactly what I have fixed in my alternative bindings which I >> created around the end of last year [0]. So we do have an idea of how >> to fix this, at the time I said I do believe that it's possible to also >> do this for the existing bindings even though it is a lot of work. >> After some talking between dkg and me we got to a way forward which >> proposed this, but I must admit that after messing a little with getting >> a pytest run integrated into the notmuch test-suite instead of using tox >> I lost momentum on the project and didn't advance any further. > > I'm sorry that I didn't speak up when you announced your work. I'm > actually excited about a new set of bindings for Python. I agree with > using cffi. I briefly looked at the code, and I believe it is much > nicer than what we currently have. Nice to hear, thanks! > I trust that it works fine with Python 3, does it? The version I made so far *only* works on Python 3. Mostly because it was easier, but it also allows some API niceties. > The testsuite cannot depend on pulling stuff from the net simply because > build servers typically do not have access to it. That is a hard > requirement. Sure I understand that. See another part of this thread on this though. Cheers, Floris ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
New Python bindings (was: Crash with Python bindings)
Hi Floris :) Floris Bruynooghe writes: > This is exactly what I have fixed in my alternative bindings which I > created around the end of last year [0]. So we do have an idea of how > to fix this, at the time I said I do believe that it's possible to also > do this for the existing bindings even though it is a lot of work. > After some talking between dkg and me we got to a way forward which > proposed this, but I must admit that after messing a little with getting > a pytest run integrated into the notmuch test-suite instead of using tox > I lost momentum on the project and didn't advance any further. I'm sorry that I didn't speak up when you announced your work. I'm actually excited about a new set of bindings for Python. I agree with using cffi. I briefly looked at the code, and I believe it is much nicer than what we currently have. I trust that it works fine with Python 3, does it? The testsuite cannot depend on pulling stuff from the net simply because build servers typically do not have access to it. That is a hard requirement. I don't remember what kind of tests there are for the current bindings (are there any...?), but shouldn't these just continue to work for the time being? Thanks, Justus signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 4/6] test: Add tests for new python bindings
The tests where adopted from the tests for the corresponding C functions
in test/T590-libconfig.sh.
---
test/T390-python.sh | 68 +
1 file changed, 68 insertions(+)
diff --git a/test/T390-python.sh b/test/T390-python.sh
index a93a7f34..725a00c9 100755
--- a/test/T390-python.sh
+++ b/test/T390-python.sh
@@ -74,4 +74,72 @@ EOF
notmuch search --sort=oldest-first --output=messages "tuÄÅáÄÄÃ" | sed s/^id:// > EXPECTED
test_expect_equal_file EXPECTED OUTPUT
+# TODO currently these tests for setting and getting config values are
+# somewhat interdependent. This is because the config values stored in the
+# database are not cleaned up after each test, so they remain there for the
+# next test. The ./README file states that this can happen so it seems kind
+# of ok.
+
+test_begin_subtest "set and get config values"
+test_python <<'EOF'
+import notmuch
+db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
+db.set_config('testkey1', 'testvalue1')
+db.set_config('testkey2', 'testvalue2')
+v1 = db.get_config('testkey1')
+v2 = db.get_config('testkey2')
+print('testkey1 = ' + v1)
+print('testkey2 = ' + v2)
+EOF
+cat <<'EOF' >EXPECTED
+testkey1 = testvalue1
+testkey2 = testvalue2
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "get_config_list with no match returns empty generator"
+test_python <<'EOF'
+import notmuch
+db = notmuch.Database()
+v = db.get_config_list('nonexistent')
+print(list(v) == [])
+EOF
+test_expect_equal "$(cat OUTPUT)" "True"
+
+test_begin_subtest "get_config_list with no arguments returns all pairs"
+test_python <<'EOF'
+import notmuch
+db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
+db.set_config("zzzafter", "afterval")
+db.set_config("aaabefore", "beforeval")
+v = db.get_config_list()
+for index, keyval in enumerate(v):
+key, val = keyval
+print('{}: {} => {}'.format(index, key, val))
+EOF
+cat <<'EOF' >EXPECTED
+0: aaabefore => beforeval
+1: testkey1 => testvalue1
+2: testkey2 => testvalue2
+3: zzzafter => afterval
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "get_config_list prefix is used to match keys"
+test_python <<'EOF'
+import notmuch
+db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
+db.set_config('testkey1', 'testvalue1')
+db.set_config('testkey2', 'testvalue2')
+v = db.get_config_list('testkey')
+for index, keyval in enumerate(v):
+key, val = keyval
+print('{}: {} => {}'.format(index, key, val))
+EOF
+cat <<'EOF' >EXPECTED
+0: testkey1 => testvalue1
+1: testkey2 => testvalue2
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
