Re: python CFFI bindings integration into notmuch build/test

2019-11-17 Thread David Bremner
Floris Bruynooghe  writes:

>
> I still have a bunch of non-existing messages with weird unicode message
> IDs in them, but no associated filename just an empty string.  AFAIK
> notmuch_database_remove_message() needs a filename though, I've tried
> removing the empty file which does not fail but doesn't clear those
> database entries.  The messages are not ghosts either.  Any idea how I
> might be able to clear those bad messages from the database?

In principle running "notmuch new" should do it, if there are no files
on disk corresponding to those messages. If that doesn't work, you might
have to use "xapian-delve" to figure out what terms are attached to
those message documents.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-17 Thread Floris Bruynooghe
On Sat 16 Nov 2019 at 10:51 -0500, David Bremner wrote:

> Floris Bruynooghe  writes:
>> Anyway, this looks good.  Would you like some changes, e.g. the rename
>> to notmuch2 or so as patches?  What's the next step.
>
> If you could look at the rename that would be great. Patches on top of
> wip/cffi, or a ref for me to pull both sound fine.
>
>> Kind of unrelated, but in my attempt to run the full notmuch test suite
>> (cd tests; make test) I somehow ended up with lots of weird tags in my
>> notmuch database and made the emacs UI pretty horrible - though it seems
>> I haven't lost any email.  I'm sure that was my fault somehow even
>> though I was just trying to follow the readme.  But if anyone has any
>> hints how to recover from this that could save me some time :)
>
> Hmm. I've never encountered that, but maybe it has to do with killing
> the environment? The test harness uses the environment variable
> NOTMUCH_CONFIG to point to a test database, and if that was deleted, it
> would look in ~/.notmuch-config.

Yeah, I probably managed to make it use ~/.notmuch-config somehow.

> Still that would really only be tags from your pytest tests, and you'd
> probably recognize those?

The were a bunch of weird characters that looked like unicode
test-cases, the pytest tests don't try to test notmuch only the bindings
so doesn't really make up weird strings IIRC.  I recovered with
something like "notmuch dump | cleanup_tags.py | notmuch restore", oh
well.

I still have a bunch of non-existing messages with weird unicode message
IDs in them, but no associated filename just an empty string.  AFAIK
notmuch_database_remove_message() needs a filename though, I've tried
removing the empty file which does not fail but doesn't clear those
database entries.  The messages are not ghosts either.  Any idea how I
might be able to clear those bad messages from the database?

Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-16 Thread David Bremner
Floris Bruynooghe  writes:
> Anyway, this looks good.  Would you like some changes, e.g. the rename
> to notmuch2 or so as patches?  What's the next step.

If you could look at the rename that would be great. Patches on top of
wip/cffi, or a ref for me to pull both sound fine.

> Kind of unrelated, but in my attempt to run the full notmuch test suite
> (cd tests; make test) I somehow ended up with lots of weird tags in my
> notmuch database and made the emacs UI pretty horrible - though it seems
> I haven't lost any email.  I'm sure that was my fault somehow even
> though I was just trying to follow the readme.  But if anyone has any
> hints how to recover from this that could save me some time :)

Hmm. I've never encountered that, but maybe it has to do with killing
the environment? The test harness uses the environment variable
NOTMUCH_CONFIG to point to a test database, and if that was deleted, it
would look in ~/.notmuch-config.

Still that would really only be tags from your pytest tests, and you'd
probably recognize those?

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-16 Thread David Bremner
Tomi Ollila  writes:

> On Thu, Nov 14 2019, Floris Bruynooghe wrote:
>
>> Hi,
>>
>> Thanks for carrying on with this!
>>
>> I'm a little confused with how to follow this, is the current state of
>> the code somewhere in a repo/branch where I can try things out and make
>> changes from?
>
> In git://notmuchmail.org/git/notmuch David has ref origin/wip/cffi
> which contains related changes -- You can fetch the code while waiting
> for more collaboration instructions from David.
>

That should be just a normal branch. I see

 % git ls-remote origin | grep cffi
 2a326a866ccd917f1d62bc7f2b475a0e2d26f6c8   refs/heads/wip/cffi

Anyway, for now I'm leary of reposting the whole series since we're
mainly discussing my mucking about, not Florian's largish patch.



___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-14 Thread Floris Bruynooghe
On Thu 14 Nov 2019 at 23:24 +0100, Floris Bruynooghe wrote:

> On Thu 14 Nov 2019 at 22:20 +0200, Tomi Ollila wrote:
>> In git://notmuchmail.org/git/notmuch David has ref origin/wip/cffi
>> which contains related changes -- You can fetch the code while waiting
>> for more collaboration instructions from David.
>
> Aha, thanks!

So I can run the tests using `make` and `cd tests;
./T-391-pytest-cffi.sh`.  I can also run the tests still individually
using something like:

make
set PATH (pwd) $PATH  # I use fish
pushd bindings/python-cffi
pew workon notmuch  # how I manage python virtualenvs
pip install -e .
pip install pytest pytest-cov
pytest test/test_base.py

And all works.  I added a little function to the conftest.py to show
which notmuch it's testing with:

def pytest_report_header():
vers = subprocess.run(['notmuch', '--version'],
  capture_output=True, text=True)
which = subprocess.run(['which', 'notmuch'],
   capture_output=True, text=True)
return ['{} ({})'.format(vers.stdout.strip(), which.stdout.strip())]

Maybe you find that useful too.

Anyway, this looks good.  Would you like some changes, e.g. the rename
to notmuch2 or so as patches?  What's the next step.



Kind of unrelated, but in my attempt to run the full notmuch test suite
(cd tests; make test) I somehow ended up with lots of weird tags in my
notmuch database and made the emacs UI pretty horrible - though it seems
I haven't lost any email.  I'm sure that was my fault somehow even
though I was just trying to follow the readme.  But if anyone has any
hints how to recover from this that could save me some time :)

Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-14 Thread Floris Bruynooghe
On Thu 14 Nov 2019 at 22:20 +0200, Tomi Ollila wrote:
> In git://notmuchmail.org/git/notmuch David has ref origin/wip/cffi
> which contains related changes -- You can fetch the code while waiting
> for more collaboration instructions from David.

Aha, thanks!

Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-14 Thread Tomi Ollila
On Thu, Nov 14 2019, Floris Bruynooghe wrote:

> Hi,
>
> Thanks for carrying on with this!
>
> I'm a little confused with how to follow this, is the current state of
> the code somewhere in a repo/branch where I can try things out and make
> changes from?

In git://notmuchmail.org/git/notmuch David has ref origin/wip/cffi
which contains related changes -- You can fetch the code while waiting
for more collaboration instructions from David.

> On Tue 05 Nov 2019 at 22:22 -0400, David Bremner wrote:
>
>> Tomi Ollila  writes:
>>
>>
>>> alternative:
>>>
>>>...
>>>print('Invoking: {}'.format(' '.join(cmd)))
>>> 
>>>def preexec_fn(): os.environ['NOTMUCH_CONFIG'] = str(cfg_fname) 
>>>
>>>proc = subprocess.run(cmd, timeout=5, preexec_fn=preexec_fn)
>>>...
>>>
>>> The unix fork ... (here preexec_fn called in child) ... exec model is
>>> superior to any other alternative ! =D
>>
>> I don't consider myself a good judge of python style, so I'll defer to
>> Floris on that one.
>
> I'd have gone with what David wrote to be fair, mostly because it's
> pretty simple to see what's going on where for the preexec_fn I'd have
> to look at the docs.  Not sure if this has much to do with Python style
> though, more with how much you like Unix tricks.

Yes, the simple approach taken (by env copy) is fine. It is also good to
know these powerful alternatives that may be useful elsewhere =D

>
>
> Cheers,
> Floris

Tomi

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-14 Thread Floris Bruynooghe
Hi,

Thanks for carrying on with this!

I'm a little confused with how to follow this, is the current state of
the code somewhere in a repo/branch where I can try things out and make
changes from?

On Tue 05 Nov 2019 at 22:22 -0400, David Bremner wrote:

> Tomi Ollila  writes:
>
>
>> alternative:
>>
>>...
>>print('Invoking: {}'.format(' '.join(cmd)))
>> 
>>def preexec_fn(): os.environ['NOTMUCH_CONFIG'] = str(cfg_fname) 
>>
>>proc = subprocess.run(cmd, timeout=5, preexec_fn=preexec_fn)
>>...
>>
>> The unix fork ... (here preexec_fn called in child) ... exec model is
>> superior to any other alternative ! =D
>
> I don't consider myself a good judge of python style, so I'll defer to
> Floris on that one.

I'd have gone with what David wrote to be fair, mostly because it's
pretty simple to see what's going on where for the preexec_fn I'd have
to look at the docs.  Not sure if this has much to do with Python style
though, more with how much you like Unix tricks.


Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-05 Thread David Bremner
Tomi Ollila  writes:


> alternative:
>
>...
>print('Invoking: {}'.format(' '.join(cmd)))
> 
>def preexec_fn(): os.environ['NOTMUCH_CONFIG'] = str(cfg_fname) 
>
>proc = subprocess.run(cmd, timeout=5, preexec_fn=preexec_fn)
>...
>
> The unix fork ... (here preexec_fn called in child) ... exec model is
> superior to any other alternative ! =D

I don't consider myself a good judge of python style, so I'll defer to
Floris on that one.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-05 Thread Tomi Ollila
On Sun, Nov 03 2019, David Bremner wrote:

>  
> @@ -32,10 +33,11 @@ def notmuch(maildir):
>  """
>  cfg_fname = maildir.path / 'notmuch-config'
>  cmd = ['notmuch'] + list(args)
> -print('Invoking: {}'.format(' '.join(cmd)))
> +env = os.environ.copy()
> +env['NOTMUCH_CONFIG'] = str(cfg_fname)
>  proc = subprocess.run(cmd,
>timeout=5,
> -  env={'NOTMUCH_CONFIG': str(cfg_fname)})
> +  env=env)

Tried lambda (to have preexe_fn have cfg_fname in context), did not work,
nested function also seems to hold context:

alternative:

   ...
   print('Invoking: {}'.format(' '.join(cmd)))

   def preexec_fn(): os.environ['NOTMUCH_CONFIG'] = str(cfg_fname) 

   proc = subprocess.run(cmd, timeout=5, preexec_fn=preexec_fn)
   ...

The unix fork ... (here preexec_fn called in child) ... exec model is
superior to any other alternative ! =D


Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: python CFFI bindings integration into notmuch build/test

2019-11-04 Thread David Bremner
David Bremner  writes:

> This series is on top of a slightly modified version [1] of
>
>  id:20191008210312.20685-2-f...@devork.be
>
> If you want to try out the new bindings, you can try the branch wip/cffi in 
> the repo
> https://git.notmuchmail.org/git/notmuch
>
> One current annoyance is that the version of pytest on travis is too
> old for what is specified in tox.ini.
>

I fixed this by moving the travis env to bionic (2018.x).  This involved
dropping dkg's notmuch ppa. Daniel, do you happen to know if there is
anything in that repo that we should still keep when building in ubuntu
bionic? FWIW, the build succeeds without that PPA.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


python CFFI bindings integration into notmuch build/test

2019-11-03 Thread David Bremner
This series is on top of a slightly modified version [1] of

 id:20191008210312.20685-2-f...@devork.be

If you want to try out the new bindings, you can try the branch wip/cffi in the 
repo
https://git.notmuchmail.org/git/notmuch

One current annoyance is that the version of pytest on travis is too
old for what is specified in tox.ini.

[1]: the diff is as follows.

diff --git a/bindings/python-cffi/notdb/_build.py 
b/bindings/python-cffi/notdb/_build.py
index 97ad7808..6be7e5b1 100644
--- a/bindings/python-cffi/notdb/_build.py
+++ b/bindings/python-cffi/notdb/_build.py
@@ -13,6 +13,8 @@ ffibuilder.set_source(
 #error libnotmuch version not supported by notdb
 #endif
 """,
+include_dirs=['../../lib'],
+library_dirs=['../../lib'],
 libraries=['notmuch'],
 )
 ffibuilder.cdef(
diff --git a/bindings/python-cffi/tests/conftest.py 
b/bindings/python-cffi/tests/conftest.py
index 1b7bbc35..aa940947 100644
--- a/bindings/python-cffi/tests/conftest.py
+++ b/bindings/python-cffi/tests/conftest.py
@@ -5,6 +5,7 @@ import socket
 import subprocess
 import textwrap
 import time
+import os
 
 import pytest
 
@@ -32,10 +33,11 @@ def notmuch(maildir):
 """
 cfg_fname = maildir.path / 'notmuch-config'
 cmd = ['notmuch'] + list(args)
-print('Invoking: {}'.format(' '.join(cmd)))
+env = os.environ.copy()
+env['NOTMUCH_CONFIG'] = str(cfg_fname)
 proc = subprocess.run(cmd,
   timeout=5,
-  env={'NOTMUCH_CONFIG': str(cfg_fname)})
+  env=env)
 proc.check_returncode()
 return run

t

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch