Re: 'notmuch search thread:<>' lists multiple threads
"Naveen N. Rao" writes: > Greetings-- > If I search for threads matching a specific thread-id, I am seeing > multiple results: > > $ notmuch search --output=threads thread:000c4d20 > thread:000c4d1e > thread:000c4d20 This looks like a bug to me. I was able to replicate it in my own mail store with the script at the end of the message. I haven't completely analyzed the situation yet, but one thing I noticed is that in all "bad threads", there are files with duplicate message-ids. Typical output looks like ╭─ zancas:software/upstream/notmuch/test ╰─ (git)-[master]-% notmuch search thread:0001760a thread:000175e5 November 03 [1/2(3)] 1...@gmx.us; Bug#846042: VTK 8 (unread) thread:0001760a 2016-11-27 [1/2(3)] 1...@gmx.us; Bug#846042: virtual/meta package for python-vtk (unread) At least some of this mail data is public, but I'm not sure if the bad threading is reproducible or not; I want to run a complete census overnight before I reindex. Even if the bug is non-deterministic, it probably lives in lib/add-message.cc -- count=0 success=0 for id in $(notmuch search --output=threads '*'); do count=$((count +1)) matches=$((`notmuch search --output=threads "$id" | wc -l`)) if [ "$matches" = 1 ]; then success=$((success + 1)) else echo "bad thread: $id" fi if [ $((count % 1000)) -eq 0 ]; then echo $count; fi done echo "count=$count success=$success" ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] doc: add a section on quoting to notmuch-search-terms(7)
I think we've diverged enough from the Xapian query parser that we can't rely on that syntax description [1]. As far as I can tell, [1] also only discusses quotes in the context of phrases. [1]: https://xapian.org/docs/queryparser.html --- doc/man7/notmuch-search-terms.rst | 22 ++ 1 file changed, 22 insertions(+) diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst index b4d784a3..a4e5abfc 100644 --- a/doc/man7/notmuch-search-terms.rst +++ b/doc/man7/notmuch-search-terms.rst @@ -273,6 +273,28 @@ Both of these will match a subject "Free Delicious Pizza" while will not. +Quoting +--- + +Double quotes are also used by the notmuch query parser to protect +boolean terms or regular expressions containing spaces or other +special characters, e.g. + +:: + + tag:"a tag" + +:: + + folder:"/^.*/(Junk|Spam)$/" + +As with phrases, you will probably have to protect the double quotes +from the shell e.g. + +:: + + % notmuch search 'folder:"/^.*/(Junk|Spam)$/"' + DATE AND TIME SEARCH -- 2.16.3 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 1/2] configure: detect which versions of python can run pytest
On Sun, Apr 08 2018, Tomi Ollila wrote:
> On Sat, Apr 07 2018, David Bremner wrote:
>
>> Based on a patch from Florian [1].
>>
>> [1]: id:py3ibmevqnna@devork.be
>> ---
>> configure | 20 +++-
>> 1 file changed, 19 insertions(+), 1 deletion(-)
>>
>> diff --git a/configure b/configure
>> index b177b141..c14e6f8b 100755
>> --- a/configure
>> +++ b/configure
>
> ...
>
>> # Name of python interpreter
>> NOTMUCH_PYTHON=${python}
>> +NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'
>
> NOTMUCH_PYTEST_PYTHONS="${pytest_pythons}"
Argh, forget this >;/
Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 1/2] configure: detect which versions of python can run pytest
On Sat, Apr 07 2018, David Bremner wrote:
> Based on a patch from Florian [1].
>
> [1]: id:py3ibmevqnna@devork.be
> ---
> configure | 20 +++-
> 1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/configure b/configure
> index b177b141..c14e6f8b 100755
> --- a/configure
> +++ b/configure
...
> # Name of python interpreter
> NOTMUCH_PYTHON=${python}
> +NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'
NOTMUCH_PYTEST_PYTHONS="${pytest_pythons}"
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
remix of pytest runner
I didn't love the idea of an evergrowing list of variables for versions of python, since to add or delete a version we'd have to update several places, even with the use of functions. This version uses one variable, which because configure is supposed to be fairly portable is a space seperated list of names of python binaries to run. It also replaces inlines functions into the bodies of loops, not for efficiency but just to avoid some (perceived?) expansion subtleties. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/2] configure: detect which versions of python can run pytest
Based on a patch from Florian [1].
[1]: id:py3ibmevqnna@devork.be
---
configure | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/configure b/configure
index b177b141..c14e6f8b 100755
--- a/configure
+++ b/configure
@@ -571,6 +571,24 @@ if [ $have_python -eq 0 ]; then
errors=$((errors + 1))
fi
+pytest_pythons=""
+if [ $have_python -eq 1 ]; then
+for bin in python2.7 python3.5 python3.6 pypy3.5; do
+printf "Checking for pytest ($bin)... "
+if command -v $bin > /dev/null; then
+if $bin -c 'import pytest' >/dev/null 2>&1; then
+pytest_pythons="${pytest_pythons} $bin"
+printf "Yes.\n"
+else
+printf "No (skipping $bin tests).\n"
+fi
+else
+printf "No (skipping $bin tests).\n"
+fi
+done
+pytest_pythons="${pytest_pythons# }"
+fi
+
printf "Checking for valgrind development files... "
if pkg-config --exists valgrind; then
printf "Yes.\n"
@@ -808,7 +826,6 @@ else
fi
rm -f compat/have_canonicalize_file_name
-
printf "Checking for getline... "
if ${CC} -o compat/have_getline "$srcdir"/compat/have_getline.c > /dev/null
2>&1
then
@@ -1233,6 +1250,7 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
# Name of python interpreter
NOTMUCH_PYTHON=${python}
+NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'
# Are the ruby development files (and ruby) available? If not skip
# building/testing ruby bindings.
--
2.16.3
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/2] test: pytest runner for the test suite
The 'test_subtest_known_broken' should be removed when there are
actual tests to run.
Based on a function from Tomi [1]
[1]: id:m2r2nq23r9@guru.guru-group.fi
---
test/T391-pytest.sh | 14 ++
1 file changed, 14 insertions(+)
create mode 100755 test/T391-pytest.sh
diff --git a/test/T391-pytest.sh b/test/T391-pytest.sh
new file mode 100755
index ..9ac7aabe
--- /dev/null
+++ b/test/T391-pytest.sh
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+test_description="python bindings (pytest)"
+. $(dirname "$0")/test-lib.sh || exit 1
+
+test_require_external_prereq ${NOTMUCH_PYTHON}
+
+for bin in ${NOTMUCH_PYTEST_PYTHONS}; do
+test_begin_subtest "pytest ($bin)"
+ test_subtest_known_broken
+ PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}"
\
+ test_expect_success "$bin -m pytest
$NOTMUCH_SRCDIR/bindings/python"
+done
+
+test_done
--
2.16.3
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database corruption after clean rebuild
On Sat, Apr 07, 2018 at 12:17:39PM -0300, David Bremner wrote: > Javier Garcia writes: > > > The following is a solid workaround I've stumbled upon. Afew no longer > > complains and database corruption is gone. > > > > $ notmuch compact > > $ xapian-check ~/.mail/.notmuch/xapian > > > > No errors found > > Right, I should have thought of compaction, that's a workaround Olly > mentioned before. That strongly suggests that you are hitting the known > Xapian bug. Yes - the error exactly matches that too. Cheers, Olly ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: experimenting with pytest tests
On Sat, Apr 07 2018, Floris Bruynooghe wrote:
> Hi,
>
> From another conversation on this list I've dug up my earlier attempts
> at integrating a pytest run into the notmuch testing suite. I'd be
> grateful for some guidance on whether this is the right way to go about
> things.
good stuff, some comments inline
>
> Here's the diff of configure:
>
> diff --git a/configure b/configure
> index c5e2ffed..8aa57b83 100755
> --- a/configure
> +++ b/configure
> @@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
> errors=$((errors + 1))
> fi
>
> +check_python() {
perhaps check_pytest()
> +local bin=$1
> +local var=$(echo $bin | tr -d '.')
I'm not so happy of the above, but good enough in this context,
alternarives are somewhat overkill
(e.g. calling like `check_python -v python27 python2.7`
or even `check_python python2 7` or `check_python python 2 7` or
`check_python python 2 . 7` ;)
> +printf "Checking for $bin (with: pytest)... "
> +if command -v $bin > /dev/null; then
> +if $bin -c 'import pytest' >/dev/null 2>&1; then
> +eval have_$var=1
> +eval $var=$bin
> +printf "Yes.\n"
> +else
> +printf "No (skipping $bin tests).\n"
> +fi
> +else
> +printf "No (skipping $bin tests).\n"
> +fi
> +}
> +
> +check_python python2.7
perhaps check_python python3.4 -- that is what EPEL for RHEL/centos7
provides (hmm, it seems python 3.6 is coming to EPEL, see
http://www.nic.funet.fi/pub/mirrors/fedora.redhat.com/pub/epel/7/x86_64/Packages/p/
(or any other mirror), anyway that looks a bit incomplete so far)
> +check_python python3.5
> +check_python python3.6
> +check_python pypy3.5
> +
> printf "Checking for valgrind development files... "
> if pkg-config --exists valgrind; then
> printf "Yes.\n"
> @@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
>
> # Name of python interpreter
> NOTMUCH_PYTHON=${python}
> +NOTMUCH_PYTHON27=${python27-}
> +NOTMUCH_PYTHON35=${python35-}
> +NOTMUCH_PYTHON36=${python36-}
> +NOTMUCH_PYPY35=${pypy35-}
perhaps NOTMUCH_PYTEST_PYPY35=${pytest_pypy35-} ...
> And I was then also trying to introduce a test/T391-pytest.sh file.
> What I'm trying to do in this file, but it doesn't currently work, is to
> have one test, with 4 subtests where each subtest is a pytest run with a
> particular python version. But if the NOTMUCH_PYTHON27 (etc) is not
> found in sh.config then the subtest should be skipped. I'm not really
> familiar enough with test-lib.sh to do this correctly without a bunch of
> more looking into it, but maybe someone else knows how to do this?
> Anyway, here my failing attempt:
We could start just with if test -n "$NOTMUCH_PYTEST_PYTHON27"
and iterate from that...
> #!/usr/bin/env bash
> test_description="python unittests"
> . ./test-lib.sh || exit 1
>
>
> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
> test_begin_subtest "${NOTMUCH_PYTHON27}"
> (
> cd "$NOTMUCH_SRCDIR/bindings/python"
> PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON27 -m pytest
> )
> test_expect_equal $? 0
> }
The above could look like...
run_pytest() {
test_begin_subtest "$1"
PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
"$1" -m pytest $NOTMUCH_SRCDIR/bindings/python
test_expect_equal $? 0
}
if test -n "$NOTMUCH_PYTEST_PYTHON27"; then
run_pytest "$NOTMUCH_PYTEST_PYTHON27"
fi
(dropped short-circuit &&, since if we ever added `set -e` the
unhandled || -case would make execution stop there)
(we might want to add export PYTHONDONTWRITEBYTECODE=donot so
$NOTMUCH_SRCDIR/bindings/python is not polluted with *.pyc files,
and we might want to think how to handle all the output files
pytest spits out...)
if the whole test script is for pytest run, PYTHONPATH could
also be put into environment variable "global" to this
particular script.
> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
> test_begin_subtest "${NOTMUCH_PYTHON27}"
> (
> cd "$NOTMUCH_SRCDIR/bindings/python"
> PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON27 -m pytest
> )
> test_expect_equal $? 0
> }
> test_require_external_prereq ${NOTMUCH_PYTHON35} && {
> test_begin_subtest "${NOTMUCH_PYTHON35}"
> (
> cd "$NOTMUCH_SRCDIR/bindings/python"
> PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON35 -m pytest
> )
> test_expect_equal $? 0
> }
>
>
> test_done
>
>
> Any tips on whether this is the right direction?
>
> Many thanks,
> Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database corruption after clean rebuild
Javier Garcia writes: > Unfortunately I can't share my emails without the approval of other > parties. The minimum subsets that trigger the error are in the range of > 1000-5000 mails, so asking each and everyone of them is out of my reach. > I tried to replicate the problem using just spam folders without success. > > The following is a solid workaround I've stumbled upon. Afew no longer > complains and database corruption is gone. > > $ notmuch compact > $ xapian-check ~/.mail/.notmuch/xapian > > No errors found Right, I should have thought of compaction, that's a workaround Olly mentioned before. That strongly suggests that you are hitting the known Xapian bug. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database corruption after clean rebuild
Unfortunately I can't share my emails without the approval of other parties. The minimum subsets that trigger the error are in the range of 1000-5000 mails, so asking each and everyone of them is out of my reach. I tried to replicate the problem using just spam folders without success. The following is a solid workaround I've stumbled upon. Afew no longer complains and database corruption is gone. $ notmuch compact $ xapian-check ~/.mail/.notmuch/xapian No errors found I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it. I will wait and test again in a few weeks. If you are interested in my setup, the error happens with this minimal configuration. #~/.config/afew/config [Filter.1] query = 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)' tags = +inbox;-new message = Messages in INBOX folder are tagged as inbox [Filter.2] query = '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)' tags = -inbox message = Messages not in INBOX folder cannot be inbox #~/.notmuch-config [database] path=/home-path/.mail [new] tags=new On 07/04/18 12:51, David Bremner wrote: > Javier Garcia writes: > >> I've applied the path to notmuch 0.26.1 without success. >> >> $ rm -rf ~/.mail/.notmuch >> $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH >> ./notmuch new >> Found 20065 total files (that's not much mail). >> Processed 20065 total files in 58s (341 files/sec.). >> Added 19605 new messages to the database. >> >> $ xapian-check .mail/.notmuch/xapian/ >> docdata: >> blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 >> B-tree checked okay >> docdata table structure checked OK >> termlist: >> blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 >> xapian-check: DatabaseError: 1 unused block(s) missing from the free >> list, first is 0 > OK, so probably not related to reference loops (although that patch is > not very well tested). It's not clear how notmuch is triggering it, but > this looks like the same bug in Xapian that olly fixed recently [1]. > > A possible next step is to try building xapian master, and linking > notmuch against that. > > Maybe Patrick or Justus (in copy) has some idea why you're only seeing > problems in afew. > > Another debugging direction is to try to duplicate your problem with > some subset of mail that you're willing to share (bisection is the usual > strategy). > > [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
experimenting with pytest tests
Hi,
From another conversation on this list I've dug up my earlier attempts
at integrating a pytest run into the notmuch testing suite. I'd be
grateful for some guidance on whether this is the right way to go about
things.
Here's the diff of configure:
diff --git a/configure b/configure
index c5e2ffed..8aa57b83 100755
--- a/configure
+++ b/configure
@@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
errors=$((errors + 1))
fi
+check_python() {
+local bin=$1
+local var=$(echo $bin | tr -d '.')
+printf "Checking for $bin (with: pytest)... "
+if command -v $bin > /dev/null; then
+if $bin -c 'import pytest' >/dev/null 2>&1; then
+eval have_$var=1
+eval $var=$bin
+printf "Yes.\n"
+else
+printf "No (skipping $bin tests).\n"
+fi
+else
+printf "No (skipping $bin tests).\n"
+fi
+}
+
+check_python python2.7
+check_python python3.5
+check_python python3.6
+check_python pypy3.5
+
printf "Checking for valgrind development files... "
if pkg-config --exists valgrind; then
printf "Yes.\n"
@@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
# Name of python interpreter
NOTMUCH_PYTHON=${python}
+NOTMUCH_PYTHON27=${python27-}
+NOTMUCH_PYTHON35=${python35-}
+NOTMUCH_PYTHON36=${python36-}
+NOTMUCH_PYPY35=${pypy35-}
And I was then also trying to introduce a test/T391-pytest.sh file.
What I'm trying to do in this file, but it doesn't currently work, is to
have one test, with 4 subtests where each subtest is a pytest run with a
particular python version. But if the NOTMUCH_PYTHON27 (etc) is not
found in sh.config then the subtest should be skipped. I'm not really
familiar enough with test-lib.sh to do this correctly without a bunch of
more looking into it, but maybe someone else knows how to do this?
Anyway, here my failing attempt:
#!/usr/bin/env bash
test_description="python unittests"
. ./test-lib.sh || exit 1
test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
test_begin_subtest "${NOTMUCH_PYTHON27}"
(
cd "$NOTMUCH_SRCDIR/bindings/python"
PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
$NOTMUCH_PYTHON27 -m pytest
)
test_expect_equal $? 0
}
test_require_external_prereq ${NOTMUCH_PYTHON35} && {
test_begin_subtest "${NOTMUCH_PYTHON35}"
(
cd "$NOTMUCH_SRCDIR/bindings/python"
PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
$NOTMUCH_PYTHON35 -m pytest
)
test_expect_equal $? 0
}
test_done
Any tips on whether this is the right direction?
Many thanks,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database corruption after clean rebuild
Javier Garcia writes: > I've applied the path to notmuch 0.26.1 without success. > > $ rm -rf ~/.mail/.notmuch > $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH > ./notmuch new > Found 20065 total files (that's not much mail). > Processed 20065 total files in 58s (341 files/sec.). > Added 19605 new messages to the database. > > $ xapian-check .mail/.notmuch/xapian/ > docdata: > blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 > B-tree checked okay > docdata table structure checked OK > termlist: > blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 > xapian-check: DatabaseError: 1 unused block(s) missing from the free > list, first is 0 OK, so probably not related to reference loops (although that patch is not very well tested). It's not clear how notmuch is triggering it, but this looks like the same bug in Xapian that olly fixed recently [1]. A possible next step is to try building xapian master, and linking notmuch against that. Maybe Patrick or Justus (in copy) has some idea why you're only seeing problems in afew. Another debugging direction is to try to duplicate your problem with some subset of mail that you're willing to share (bisection is the usual strategy). [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Database corruption after clean rebuild
I've applied the path to notmuch 0.26.1 without success. $ rm -rf ~/.mail/.notmuch $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH ./notmuch new Found 20065 total files (that's not much mail). Processed 20065 total files in 58s (341 files/sec.). Added 19605 new messages to the database. $ xapian-check .mail/.notmuch/xapian/ docdata: blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748 xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0 With or without the patch, the "corrupted" database works fine most of the time. For instance this works: $ notmuch tag +new2 -- tag:new It's just that afew can't work with the db in this state, it complains saying the database is corrupted. $ rm -rf ~/.mail/.notmuch $ notmuch new $ afew -tn -vv terminate called after throwing an instance of 'Xapian::DatabaseCorruptError' Aborted (core dumped) Afew doesn't always crash, even though the database is always corrupted. Afew crashes when it's called just after a fresh notmuch database is built and randomly thereafter. The error is always the same. The following one-liner can solve most of afew crashes. It works well for the cases in which afew is called right after notmuch database creation. For random crashes is not as effective. $ notmuch tag +new2 -- tag:new On 07/04/18 03:49, David Bremner wrote: > Javier Garcia writes: > >> I can't build a healthy database for notmuch. My mail directory has >> quite a few mails, around 20,000. >> >> $ rm -rf ~/.mail/.notmuch >> $ notmuch new >> $ xapian-check ~/.mail/.notmuch/xapian/ >>> docdata: >>> blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0 >>> B-tree checked okay >>> docdata table structure checked OK >>> >>> termlist: >>> blocksize=8K items=43520 firstunused=8291 revision=2 levels=2 root=748 >>> xapian-check: DatabaseError: 1 unused block(s) missing from the free >> list, first is 0 >> > There was recently a similar report that turned out to be related to a > reference loop in the mail. Do you actually have any symptoms of > database corruption other than the message about the free list? if not, > it might be worth trying the attached patch, which attempts to break > reference loops. > ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
