Re: Meaning of offset in notmuch search --output=files --offset=
Am Sa., 25. Mai 2024 um 12:06 Uhr schrieb David Bremner : > > Teemu Likonen writes: > > > > >> Perhaps the manual page needs a few more words to make it clear. > > > > --offset=[-]N > > Skip displaying the first N results. With the leading '-', > > start at the Nth result from the end. > > > > What if we change the first sentence to "Skip displaying the first N > > search results"? > > For me personally that doesn't make it that much clearer, but I'm > willing to follow a concensus if there is one. ... because it was clear to you before, already ;-) Once you are aware of the conceptual difference between "search" and "output", the change makes it clearer. Maybe even "Skip displaying output for the first N search results", in an effort to hint at said difference? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: I don't understand the results of this search query, is this a bug?
Am Mo., 13. Mai 2024 um 23:38 Uhr schrieb Carl Worth : > > Hi Renaud, > > I was able to see similar behavior in my own mail store. And I agree > that this behavior is confusing! > > The documentation for the --files option of notmuch search documents the > cause (and predicts that this will be confusing): > > In general, I'm not a fan of software documenting "this may be > confusing". That suggests the authors of the documentation know that the :-) > software is not behaving as the user intends, so it would be preferable > for the software to behave as intended. That said, I also understand the > implementation details that lead to this behavior. So I wouldn't be > opposed to improving the behavior of notmuch to reduce this behavior, > (but that implementation might not be trivial or even fully feasible). I wouldn't call it an implementation detail in this case, though, rather than the guiding principle of notmuch: it is all about *messages* as identified by a mid. Consequently, notmuch stores information by message, searches by message and outputs information by message. This in turn has consequences, for better or worse, e.g. when different mail files with the same mid have different (maildir) flags. But without grasping the main guiding principle you'll get confused sooner or later. As soon as you introduce "do what I mean" into the CLI design the outcome depends on the "I" implementing it, who may "mean" very different things compared to the "I" using the CLI. This creates confusion which cannot be resolved by pointing out a guiding principle, but rather "when we do x it is often convenient to imply y and that's why do z". You can witness that to some extent in git's CLI. Also, dwim'ing in the case at hand seems difficult - you'd have to extract "path:" tokens from a possibly complex query, track logical operators applying to them and filter the output accordingly. Compare that to "find -type f dirWhichIWant" which would have solved OP's use case ... Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
"Detected x file renames" for copies
Hi there, I noticed the following: - have file f indexed by notmuch - copy message file to a different folder using `notmuch insert --folder=Y < f` - run notmuch new This results in "Detected x file renames" if you do that for x files. I have not checked whether this depends on folder names or on which file copy notmuch considers as "the" message file. Probably this comes from being able to find the message by filename or not in notmuch.c:remove_filename(). I find "renames" confusing because it's the same message when you actually move files, i.e. delete the original. But I"m wondering whether it's worth investigating. I know not to be surprised now :) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: BUG: Python's Message.header fails for empty headers
Am Di., 9. Jan. 2024 um 13:38 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > >> > >> I agree the bindings documentation does not make much sense. I suspect > >> that the bindings should follow the underlying library and return "" if > >> the library does. I don't use the bindings that much, so I am curious > >> what others think. > > > > I might be misunderstanding the OP,and I didn't check the RFC, but > > isn't there a difference between a missing header and an empty header? > > Are you suggesting the library should change as well? I don't know. Missing vs. empty are two different cases. It may be that in terms of e-mail standards (RFC), a missing header is equivalent to an empty one. Then one can argue whether an empty header is a missing header (raise error) or a missing header is an empty header (return None) ... Or one may distinguish between mandatory headers and optional ones. I'd really check the RFC. In Python, there is a difference between a dictionary key with an empty value and a key which is not present, of course. So, if I think of the headers as a dictionary, I would expect to differentiate between those cases, unless the data which the dict represents (e-mail headers) treats them as equal by RFC. > > If there is, this may come down to the difference between testing for > > an empty string, None or False in dynamically typed python ... > > But it does make sense for the bindings to return an empty string or > > None for an empty header and LookUpError for a missing header. I have > > not checked whether our bindings in fact do. > > > > AFAICT it checks explicitely for NULL, but then throws LookupError on > any falsy return from capi.ffi.string > > ret = capi.lib.notmuch_message_get_header(self._msg_p, name) > if ret == capi.ffi.NULL: > raise errors.NullPointerError() > hdr = capi.ffi.string(ret) > if not hdr: > raise LookupError > return hdr.decode(encoding='utf-8') That clarifies what OP observed :) ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: BUG: Python's Message.header fails for empty headers
Am Di., 9. Jan. 2024 um 00:09 Uhr schrieb David Bremner : > > Vojtěch Káně writes: > > > At first, this sounds reasonable: the subject is empty, so it is > > effectively missing. That would indicate a bug in Lieer itself and would > > be fixed by a try-catch block. Notmuch's source for Message.header, > > however, states: > > > >>:returns: The header value, an empty string if the header is not present. > >>:rtype: str > > > > This makes an impression that no error should be raised and a harmless > > value (at least for the above-mentioned code) should be returned. Yet > > the docs continue with > > > >>:raises LookupError: if the header is not present. > > > > completely contradicting itself. > > > > And so here the questions: > > Is my confusion justified? What is the expected nm's behavior? Can we > > fix the docs and possible the implementation? > > > > I agree the bindings documentation does not make much sense. I suspect > that the bindings should follow the underlying library and return "" if > the library does. I don't use the bindings that much, so I am curious > what others think. I might be misunderstanding the OP,and I didn't check the RFC, but isn't there a difference between a missing header and an empty header? If there is, this may come down to the difference between testing for an empty string, None or False in dynamically typed python ... But it does make sense for the bindings to return an empty string or None for an empty header and LookUpError for a missing header. I have not checked whether our bindings in fact do. Also, note that *sending* via lieer (i.e. via GMail API) provides more pitfalls. For example, message IDs are rewritten, which makes it unusable for patch series. Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Difficulty understanding maildir.synchronize_flags behavior
Am Di., 12. Dez. 2023 um 10:58 Uhr schrieb Andrew Todd : > Hello, > > I'm trying to set up notmuch, and I think that the behavior I want > should be possible (and is desirable), but I can't seem to make it work. > I also can't find any reference online suggesting that it *shouldn't* > work, and I would have thought that it's a very common use case for > people using multiple mail clients with one IMAP server. > > Running Arch Linux, so on latest (currently 0.38.1). Working with the > CLI and Emacs frontend. > > I'm synchronizing mail from a remote IMAP server to local maildir using > isync/mbsync. > > What I want is for messages that do not have maildir files ending in > ':2,S' to be tagged as 'unread' in notmuch. > > In other words, I want notmuch to respect the read/unread state of > messages from the IMAP server and not make any other changes. > > My understanding is that this is the purpose of the > maildir.synchronize_flags option. I have explicitly set this to true in > my config: > > [maildir] > synchronize_flags=true > > Moreover, I have also turned off all default tags when running notmuch new: > > [new] > tags= > > The default is `unread;inbox`. I understand you don't want the inbox tag, but have you tried with `unread` here? The manual is not overly clear about the interaction between this config and flag sync, but AFAIU, `notmuch new` sets `unread` and then possibly unsets it during the flag sync (maybe in one atomic db write, i.e. you don't see this). By removing `unread` from `new.tags` you keep `notmuch new` from doing that first step. And just in case this feels like a "duh moment" to you - we all have them ;) Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Advanced search with wildcard using notmuch for mutt
Am Do., 30. Nov. 2023 um 12:37 Uhr schrieb io :
>
>
>
> i have an html email with this sentence 'xycfe11cg64d_2501034012' within
> the body of the message.
> no result found when i search for '2501034012'
> i have even tried using '*2501034012*' (wildcard)
>
> Notmuch doesn't support regular expressions ("wildcards") when searching
the body of messages, see the man page for `notmuch-search-terms`. It
computes stems of words and indexes (and searches) those.
Using xapian commands, one could extract all stems and grep those for a
term which one "remembers partially" (often happened to me), and then feed
that into notmuch. Might be worthwhile scripting or even integrating into
notmuch (sexp?).
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 0/4] test: T380 rework
So, with the key-value pairs sorted by both, I resumed testing for Python 3.1.13 and encountered failing T380 which gave me some a deja-vue due to its confusing messages: ``` T380-atomicity: Testing atomicity cat: outcount: No such file or directory /builddir/build/BUILD/notmuch-0.38.1/test/T380-atomicity.sh: line 79: ((: i < : syntax error: operand expected (error token is "< ") PASS "notmuch new" is idempotent under arbitrary aborts FAIL detected >10 abort points test -gt 10 /builddir/build/BUILD/notmuch-0.38.1/test/test-lib.sh: line 701: test: -gt: unary operator expected ``` And that is why this is a reply to the old thread where I suggested making this less confusing, because everything reported is not the actual failure, but the consequence of not safe-guarding against a failed test setup. The *real cause* is most likely that `import gdb` fails in `atomicity.py` because it's not ready for py 3.13 yet. But maybe it's time to reconsider some bits of the old series? We ended up discussing "echo vs printf" and never addressed the real issues here. Cheers, Michael P.S.: There are also a few lines like ``` Error: database path '/builddir/build/BUILD/notmuch-0.38.1/test/tmp.T400-hooks/database.85' does not exist or is not a directory. ``` sprinkled in the test output between PASS tests, apparently without making any test fail. I don't know whether I never noticed for a apassing test suite, or whether this is new. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T610 failing on Fedora rawhide
Am Fr., 24. Nov. 2023 um 14:57 Uhr schrieb David Bremner :
> Michael J Gruber writes:
>
> > Hi there,
> >
> > during my first tests for Python 3.13 (hooray...) I noticed that some
> tests
> > in T610 started to fail independent of that. It seems that with notmuch
> > 0.38.1 on current Fedora rawhide, `notmuch_message_get_properties()`
> > returns properties in a different order, while the tests expect a
> specific
> > order. So I'm wondering:
> > - Is there a particular order which the interface promises to deliver?
> > - If yes: What could cause it to be different?
> > If not there's some work to do making the tests independent of the order
> ...
> > This is not glib again, is it?
>
> Can you try the following patch?
>
> diff --git a/lib/string-map.c b/lib/string-map.c
> index e3a81b4f..99bc2ea2 100644
> --- a/lib/string-map.c
> +++ b/lib/string-map.c
> @@ -86,10 +86,14 @@ _notmuch_string_map_append (notmuch_string_map_t *map,
> static int
> cmppair (const void *pa, const void *pb)
> {
> +int cmp = 0;
> notmuch_string_pair_t *a = (notmuch_string_pair_t *) pa;
> notmuch_string_pair_t *b = (notmuch_string_pair_t *) pb;
>
> -return strcmp (a->key, b->key);
> +cmp = strcmp (a->key, b->key);
> +if (cmp == 0)
> + cmp = strcmp (a->value, b->value);
> +return cmp;
> }
>
> static void
>
So I guess we did not quite sort completely before, ey? ;-)
That patch makes T610 pass again, thanks. It's a good change independent of
any glibc promises, I think.
Glad we got this sorted out!
Cheers
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
T610 failing on Fedora rawhide
Hi there,
during my first tests for Python 3.13 (hooray...) I noticed that some tests
in T610 started to fail independent of that. It seems that with notmuch
0.38.1 on current Fedora rawhide, `notmuch_message_get_properties()`
returns properties in a different order, while the tests expect a specific
order. So I'm wondering:
- Is there a particular order which the interface promises to deliver?
- If yes: What could cause it to be different?
If not there's some work to do making the tests independent of the order ...
This is not glib again, is it?
Cheers,
Michael
https://kojipkgs.fedoraproject.org//work/tasks/3691/109443691/build.log
T610-message-property: Testing message property API
PASS notmuch_message_{add,get,remove}_property
PASS testing string map binary search (via message properties)
PASS notmuch_message_get_properties: empty list
PASS notmuch_message_properties: one value
PASS notmuch_message_remove_all_properties
PASS notmuch_message_properties: multiple values
FAIL notmuch_message_properties: prefix
--- T610-message-property.7.EXPECTED 2023-11-23 12:39:05.368791821 +
+++ T610-message-property.7.OUTPUT 2023-11-23 12:39:05.370791961 +
@@ -1,7 +1,7 @@
== stdout ==
testkey1 = alice
-testkey1 = bob
testkey1 = testvalue2
+testkey1 = bob
testkey3 = alice3
testkey3 = bob3
testkey3 = testvalue3
PASS notmuch_message_properties: modify during iteration
FAIL dump message properties
--- T610-message-property.9.PROPERTIES 2023-11-23 12:39:05.787821095 +
+++ T610-message-property.9.OUTPUT 2023-11-23 12:39:05.789821235 +
@@ -1 +1 @@
-#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=alice testkey1=bob testkey1=testvalue2 testkey3=alice3
testkey3=bob3 testkey3=testvalue3
+#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=testvalue2 testkey1=bob testkey1=alice testkey3=alice3
testkey3=bob3 testkey3=testvalue3
FAIL dump _only_ message properties
--- T610-message-property.10.EXPECTED 2023-11-23 12:39:05.815823052 +
+++ T610-message-property.10.OUTPUT 2023-11-23 12:39:05.817823191 +
@@ -1,2 +1,2 @@
#notmuch-dump batch-tag:3 properties
-#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=alice testkey1=bob testkey1=testvalue2 testkey3=alice3
testkey3=bob3 testkey3=testvalue3
+#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=testvalue2 testkey1=bob testkey1=alice testkey3=alice3
testkey3=bob3 testkey3=testvalue3
FAIL restore missing message property (single line)
--- T610-message-property.11.PROPERTIES 2023-11-23 12:39:06.042838911 +
+++ T610-message-property.11.OUTPUT 2023-11-23 12:39:06.043838981 +
@@ -1 +1 @@
-#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=alice testkey1=bob testkey1=testvalue2 testkey3=alice3
testkey3=bob3 testkey3=testvalue3
+#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=testvalue2 testkey1=bob testkey1=alice testkey3=alice3
testkey3=bob3 testkey3=testvalue3
FAIL restore missing message property (full dump)
--- T610-message-property.12.PROPERTIES 2023-11-23 12:39:06.276855260 +
+++ T610-message-property.12.OUTPUT 2023-11-23 12:39:06.277855330 +
@@ -1 +1 @@
-#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=alice testkey1=bob testkey1=testvalue2 testkey3=alice3
testkey3=bob3 testkey3=testvalue3
+#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=testvalue2 testkey1=bob testkey1=alice testkey3=alice3
testkey3=bob3 testkey3=testvalue3
FAIL restore clear extra message property
--- T610-message-property.13.PROPERTIES 2023-11-23 12:39:06.485869862 +
+++ T610-message-property.13.OUTPUT 2023-11-23 12:39:06.487870002 +
@@ -1 +1 @@
-#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=alice testkey1=bob testkey1=testvalue2 testkey3=alice3
testkey3=bob3 testkey3=testvalue3
+#= 4efc743a.3060...@april.org
fancy%20key%20with%20%c3%a1cc%c3%a8nts=import%20value%20with%20=
testkey1=testvalue2 testkey1=bob testkey1=alice testkey3=alice3
testkey3=bob3 testkey3=testvalue3
PASS test 'property:' queries: empty
PASS test 'property:' queries: single message
FAIL msg.get_property (python)
--- T610-message-property.16.EXPECTED 2023-11-23 12:39:06.601877966 +
+++ T610-message-property.16.OUTPUT 2023-11-23 12:39:06.603878106 +
@@ -1,2 +1,2 @@
-testkey1 = alice
+testkey1 = testvalue2
testkey3 = alice3
FAIL msg.get_properties (python)
--- T610-message-property.17.EXPECTED 2023-11-23 12:39:06.677883276 +
+++ T610-message-property.17.OUTPUT 2023-11-23 12:39:06.678883346 +
@@ -1,3 +1,3 @@
-testkey1 = alice
-testkey1 = bob
testkey1
Re: [PATCH 26/27] emacs: avoid binding unnamed commands in keymaps
Am Sa., 28. Okt. 2023 um 06:22 Uhr schrieb Ryan Tate : > > Jonas Bernoulli writes: > > > - -(defun notmuch-tree-close-message-pane-and (func) - "Close > > message pane and execute FUNC. > > I am confused why a function used in config files and documented > on the notmuch website (to this moment) as an example of how to > configure something would be removed, and to be removed without > any announcement. > > I argue that a user (who is not literally an author of notmuch) > should be able to reliably use the software, which involves configuring > it and having that configuration continue to work. It's not like it was > remotely easy getting that config setup in the first place. > ___ > With all due respect, I suggest you try to (re-)read your posts again through the eyes of a notmuch author or contributor, for a change of perspective. Do you still consider your tone appropriate? Bug reports and constructive suggestions are always welcome here. In fact, I experienced the notmuch community as absolutely welcoming and zero-drama. Let's keep it that way. Maybe you can help us by rephrasing your complaints into suggestions for documentation updates, and thus help others getting less confused? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: v2 Image toggle fixes for emacs 29.1
Am So., 1. Okt. 2023 um 13:13 Uhr schrieb David Bremner : > > I have applied this series to release and master (and uploaded a > pre-release for 0.38.1) Is "pre" the new "rc", or how is this supposed to sort? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Fixed Message-ID trouble
Am Mo., 25. Sept. 2023 um 12:53 Uhr schrieb Teemu Likonen : > > * 2023-09-25 11:54:07+0300, Teemu Likonen wrote: > > > Some person on debian-user mailing list seems to be sending messages > > with fixed Message-ID field: the same ID in different messages. In > > Notmuch it is creating trouble because it connects unrelated threads to > > one. The person has different messages in different threads but Notmuch > > thinks they are the same message because the Message-ID is the same. > > > > This is potentially a "denial of service" for Notmuch. Well, not quite, > > but is harmful nonetheless. How would a Notmuch user fix the mess or > > protect himself against it? > > I am no longer sure if this issue is caused by fixed "Message-ID" or > wrong "References" or "In-Reply-To" values. Anyway, someone has created > real mess anyway because Notmuch combines originally separate threads > now and forever. Yes, several sources of different badness ... Still, if I understand correctly, a new message with a pre-existing mid ends up being registered by notmuch as a second file for the "same" message irrespective of differences in the actual files. For message copies which you receive via different paths (say directly plus via an ml) this may or may not be what you want. Used intentionally, it may create harm - how do other mailers handle this? Show them in parallel in the same thread (but as individual messages)? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: notmuch breaks on \. in config file with upgrade from glib2 2.76.1 to 2.76.5
Hi there [snip] > Last night, I filed > https://bugzilla.redhat.com/show_bug.cgi?id=2237562. Later, I found > this about glib 2.77 introducing regressions: > https://bugzilla.redhat.com/show_bug.cgi?id=2225257; looks like Fedora > backported enough of that into 2.76.5 to cause similar issues in > relation to 2.76.1, even though a patchlevel release of glib shouldn't > be changing behaviors. Fedora has no related patches in 2.76.5-1 at all (only hmac/eperm). So, if that's the same regression as in 2.77 it was introduced earlier, and purely in upstream. > I presume that 'notmuch config set' should be the preferred way to > modify the config file - but since it IS a human-readable file, > notmuch should do a much better job of reporting errors whenever > glib's gkeyfile API cannot parse the file (even if that failure to > parse is caused by an unintended regression in glib behavior for > rejecting something it used to accept). Yes. This round of glib2 gave us quite some headaches to get config back working at all. The typical response from glib2 upstream was that what we called regressions were fixes to wrong behaviour in glib2 and that we should not rely on established behaviour (my words) but only on the documentation, the latter exposing a sense of humour which I do appreciate at times. In particular, glib2's read and write results are the authoritative answer. And probably the older glib2 was "wrong" in what it accepted leniently ... Does notmuch even get the chance to read partially? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Do., 31. Aug. 2023 um 17:17 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > > > I still get those issues. OTOH, skipping T460.14 did not show any > > adverse side effects. So I'll do that for emacs29. > > I might be nice to mark some tests ignored rather than skipped so that > > we notice when they do not fail sporadically any more. That is, *if* > > we look at the output of a passing test suite ... > > > > It is possible to selectively mark tests as broken, but it requires > patching the test suite, and it sets a failing exit code if those tests > start passing. Yes, that's why I wrote "ignore". Something like NOTMUCH_IGNORE_TESTS which runs the test, outputs the diff on fail, but "succeeds" without counting towards pass/fail, and reports the number of ignored pass/fail separately - basically "known_broken" without the "known/expectation". I just don't know whether it's worth it. Other folks disable a whole test suite when they want to get a package update going ... Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Do., 31. Aug. 2023 um 15:16 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > Am Sa., 26. Aug. 2023 um 16:41 Uhr schrieb David Bremner > > : > >> > >> Michael J Gruber writes: > >> > >> > > >> > I tried the current 0.38rc1 on COPR, and unfortunately I get the same > >> > T460 failure (fedora-eln-aarch64 and fedora-rawhide-x86_64 this time, > >> > out of 35 buildroots). > >> > Did you get your fails with emacs 29 only, or with earlier emacs? > >> > >> I only tested emacs 29; it would be some different incantation to > >> semi-disable native compilation for emacs 28.x. Are you seeing those > >> same failures (where emacs attempting to write into /usr/bin) on older > >> emacs? > > > > No, I see them only on Fedora rawhide/ELN and Fedora 39, but not on > > the current release 38 or earlier. Emacs 29/28 is one difference and > > an obvious guess as the cause, but it could be dtach or whatnot. > > Hmm. I just built 200 times in sbuild (chroot) so I guess I am no longer > able to reproduce the issue on Debian, fwiw. I still get those issues. OTOH, skipping T460.14 did not show any adverse side effects. So I'll do that for emacs29. I might be nice to mark some tests ignored rather than skipped so that we notice when they do not fail sporadically any more. That is, *if* we look at the output of a passing test suite ... Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Sa., 26. Aug. 2023 um 16:41 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > > > I tried the current 0.38rc1 on COPR, and unfortunately I get the same > > T460 failure (fedora-eln-aarch64 and fedora-rawhide-x86_64 this time, > > out of 35 buildroots). > > Did you get your fails with emacs 29 only, or with earlier emacs? > > I only tested emacs 29; it would be some different incantation to > semi-disable native compilation for emacs 28.x. Are you seeing those > same failures (where emacs attempting to write into /usr/bin) on older > emacs? No, I see them only on Fedora rawhide/ELN and Fedora 39, but not on the current release 38 or earlier. Emacs 29/28 is one difference and an obvious guess as the cause, but it could be dtach or whatnot. I get the same failures with notmuch 0.37+your patch on koji now (rawhide, f39; not f38), sporadically. I'm confident it's only in the test suite, so I can disable that test on Fedora for the release build. (Will have to test whether the failures creep up somewhere else then.) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Sa., 26. Aug. 2023 um 00:28 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > It took more runs to get some fails now, and archs vary, so I still > > think its a time out. And no way to get it locally so far. > > I can duplicate it locally about once every 40 runs of the complete test > suite. > > > ENOLISP (for me) but could it be the case that notmuch-test-wait can > > abort its while loop too early if the first buffer write takes longer > > than the timeout, or if some other process writes (because the process > > parameter is nil)? Is something different for emacs 29 in this regard? > > Any clues from sbuild? > > Can you try the attached patch? It needs more testing, but I did get 140 > runs of the test suite without an error. I tried the current 0.38rc1 on COPR, and unfortunately I get the same T460 failure (fedora-eln-aarch64 and fedora-rawhide-x86_64 this time, out of 35 buildroots). Did you get your fails with emacs 29 only, or with earlier emacs? Trying with 0.37+patches on KOJI right now. There's also one patch I want to send out before release, hopefully in a minute or two ;-) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Do., 24. Aug. 2023 um 17:10 Uhr schrieb David Bremner : > > David Bremner writes: > > > I just saw this when running in debian's "sbuild" isolated build > > environment. So my current guess is that this has to do with HOME > > pointing somewhere nonexistent. Is that also the case in COPR? > > > > d > > I realized that we override HOME inside the tests anyway, so emacs > should think there is some writable HOME in any case. I did notice that > the tests trigger a bunch of emacs native compilation (because the > caching happens in the temporary $HOME, which gets blown away every > time). Also, $HOME is set in all my build envs (pass or fail), and permissions are the same. Bummer. It took more runs to get some fails now, and archs vary, so I still think its a time out. And no way to get it locally so far. ENOLISP (for me) but could it be the case that notmuch-test-wait can abort its while loop too early if the first buffer write takes longer than the timeout, or if some other process writes (because the process parameter is nil)? Is something different for emacs 29 in this regard? Any clues from sbuild? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T460: new sporadic failures with emacs 29
Am Do., 24. Aug. 2023 um 16:01 Uhr schrieb David Bremner : > > David Bremner writes: > > > Michael J Gruber writes: > > > >> -notmuch@notmuchmail.org > >> -http://notmuchmail.org/mailman/listinfo/notmuch > >> *ERROR*: Opening output file: Permission denied, /usr/bin/OUTPUT > >> PASS Stash id > >> ``` > >> > >> That "/usr/bin/OUTPUT" looks strange and smells like a mis-expanded > >> variable. > > > > Yes, that's pretty weird. The only writes to "OUTPUT" are relative to > > emacs default-directory. Not sure how that could be set to /usr/bin; > > possible some weird script involved with starting emacs? > > I just saw this when running in debian's "sbuild" isolated build > environment. So my current guess is that this has to do with HOME > pointing somewhere nonexistent. Is that also the case in COPR? > I encountered this on koji (the main Fedora infra), too, and am trying with an increased wait (1 rather than 0.1) right now. Dunno by how much this increases test suite run times. HOME could be an issue only if some builder VMs are set-up differently, I guess? They shouldn't be (which does not necessarily mean they aren't). Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
T460: new sporadic failures with emacs 29
Hi there, I'm sorry to report that I'm getting new sporadic (some archs sometimes) failures on Fedora 39+ only, i.e. with emacs 29, on COPR. ``` T460-emacs-tree: Testing emacs tree view interface PASS Basic notmuch-tree view in emacs ... PASS Tree view of a single thread (from show) FAIL Message window of tree view --- T460-emacs-tree.14.notmuch-tree-show-window 2023-08-24 10:41:07.938464748 + +++ T460-emacs-tree.14.OUTPUT 2023-08-24 10:41:07.938464748 + @@ -1,41 +0,0 @@ -Lars Kellogg-Stedman (2009-11-17) (inbox signed) ... -notmuch@notmuchmail.org -http://notmuchmail.org/mailman/listinfo/notmuch *ERROR*: Opening output file: Permission denied, /usr/bin/OUTPUT PASS Stash id ``` That "/usr/bin/OUTPUT" looks strange and smells like a mis-expanded variable. Why sporadically, though? The emacs test wait 0.1 before writing - I dunno why, but those waits are fragile and make me nervous about even keeping the tests for release builds. I guess due to its load, COPR is prone to exposing timing issues. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: plans for releasing 0.38
Am So., 20. Aug. 2023 um 19:54 Uhr schrieb David Bremner : > > > I just posted the last patch needed to unbreak the test suite for me > with Emac 29.x and glib 2.77.x. As soon as these (or something similar) > are applied to master, I plan to feature freeze 0.38. There is less new > stuff than a typical release, but it has been almost a year since the > last release. Thanks! I had planned to look at the glib issues after my vacation, only to see you've solved them already meanwhile ;) Together with the patch for emacs 29 (on top of our usual spec file), current notmuch master builds (and tests) nicely on old and new Fedoras (37, 38, 39[branched] and rawhide), (RH)EL 8, 9 and its development versions (CentOS Stream and ELN) with a mix of glib and emacs versions. Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/2] test: showcase thread-unsafe s-expression query parser
Am Sa., 22. Juli 2023 um 20:48 Uhr schrieb David Bremner : > > Kevin Boulain writes: > > > The test fails quite reliably for me: > > T810-tsan: Testing run code with TSan enabled against the library > >PASS create > >PASS query > >FAIL sexp query > > I can't get this test to fail at all. I'm not sure what to conclude from > that. This is on a 20 core system running linux. This might be completely unrelated, but I recall a strange problem I had with asan (not tsan): We build without asan, but if asan is present for whatever reason then the tests are run nonetheless. So we had to disable asan tests explicitly. Is your sexp compiled accordingly? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T040 fails on Fedora 39 rebuild
Am Fr., 21. Juli 2023 um 03:40 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > I'm not sure we should or can work around 2.77.0's behaviour. (It > > might get triggered by the add/remove trick.) > > Will FC39 definitely ship with 2.77.0 ? There is still plenty of time, so I hope that 2.77.1 will land soon and I can prod them to carry that (or the fix). If it lands the Fedora notmuch rebuild against it is in my hands. At first I thought we (notmuch) might have a flushing problem or such, which we should prevent. But this clearly looks like a glib2 regression now which caused others' CI to fail, too. At least you're prepared in case of incoming reports now ;-) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T040 fails on Fedora 39 rebuild
Am Do., 20. Juli 2023 um 20:54 Uhr schrieb Michael J Gruber : > > Hi there, > > funny failures again during the mass rebuild for the (not quite yet) > upcoming Fedora rebuild. This test passed just a few weeks ago, so the > failure is related to something else changing. But still it shows that > the order in which `notmuch config` produces new config can depend on > something else. It looks a bit like a buffering problem, but what do I > know ... In any case, this is with notmuch 0.37, and I did not see any > commit in master which could change that. > > I vaguely remember we had g_key_file ordering issues before - maybe in > another project? > > The group comment for the next group is written too early, or the keys > for the current group are written too late. This results in a > functionally equivalent config with "misplaced" comments. > > Test still passes on Fedora 38. Glib2 versions are: > > 2.76.4-1.fc38 > 2.77.0-6.fc39 In fact, glib2 may come with a fix when 2.77.1 is released: https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3380 I'm not sure we should or can work around 2.77.0's behaviour. (It might get triggered by the add/remove trick.) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
T040 fails on Fedora 39 rebuild
Hi there, funny failures again during the mass rebuild for the (not quite yet) upcoming Fedora rebuild. This test passed just a few weeks ago, so the failure is related to something else changing. But still it shows that the order in which `notmuch config` produces new config can depend on something else. It looks a bit like a buffering problem, but what do I know ... In any case, this is with notmuch 0.37, and I did not see any commit in master which could change that. I vaguely remember we had g_key_file ordering issues before - maybe in another project? The group comment for the next group is written too early, or the keys for the current group are written too late. This results in a functionally equivalent config with "misplaced" comments. Test still passes on Fedora 38. Glib2 versions are: 2.76.4-1.fc38 2.77.0-6.fc39 Cheers Michael ``` T040-setup: Testing "notmuch setup" PASS Notmuch new without a config suggests notmuch setup FAIL Create a new config interactively --- T040-setup.2.config-with-comments 2023-07-20 16:58:16.513709261 + +++ T040-setup.2.new-notmuch-config 2023-07-20 16:58:16.513709261 + @@ -1,7 +1,3 @@ -# .notmuch-config - Configuration file for the notmuch mail system -# -# For more information about notmuch, see https://notmuchmail.org - # Database configuration # # The only value supported here is 'path' which should be the top-level @@ -11,8 +7,6 @@ # configured here named ".notmuch". # [database] -path=/path/to/maildir - # User configuration # # Here is where you can let notmuch know how you would like to be @@ -28,11 +22,8 @@ # recipient list of replies, and will set the From address based on the # address to which the original email was addressed. # +path=/path/to/maildir [user] -name=Test Suite -primary_email=test.su...@example.com -other_email=another.su...@example.com - # Configuration for "notmuch new" # # The following options are supported here: @@ -47,9 +38,10 @@ # names will be ignored, independent of its depth/location # in the mail store. # +name=Test Suite +primary_email=test.su...@example.com +other_email=another.su...@example.com [new] -tags=foo;bar; - # Search configuration # # The following option is supported here: @@ -59,9 +51,8 @@ # search results by default. Using an excluded tag in a # query will override that exclusion. # +tags=foo;bar; [search] -exclude_tags=baz - # Maildir compatibility configuration # # The following option is supported here: @@ -83,4 +74,5 @@ # and update tags, while the "notmuch tag" and "notmuch restore" # commands will notice tag changes and update flags in filenames # +exclude_tags=baz [maildir] PASS setup consistent with config-set for single items ``` ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] python: adjust legacy bindings to py 3.12
Am So., 9. Juli 2023 um 16:57 Uhr schrieb David Bremner : > > michaeljgruber+grubix+...@gmail.com writes: > > > From: Michael J Gruber > > > > Py 3.12 finally pulled the plug on the `SafeConfigParser` class which > > has been deprecated since py 3.2. > > Applied to master. > > Apropos, how many people are still using the legacy bindings? > We are still shipping them in Fedora (but, simply put, we have no py2 any more), and that's how I noticed the issue. Also, `afew` is not fully converted yet. It's WISP. S = slow ;-) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: "notmuch compact" questions
Am Sa., 17. Juni 2023 um 23:49 Uhr schrieb Andy Smith : > > Hi, > > I'm using v0.31.4 on Debian 11. I have ~3.9 million messages in my > archive and the notmuch database currently takes up 85GiB (though > actually "only" 51GiB due to btrfs zstd:1 compression). Wow ;) > I did remove a few hundred thousand messages from my archive but the > space used by the database did not go down at all. > > If I ran "notmuch compact" should I expect any space to be > reclaimed? > > The manual page says that this builds a new copy of the database and > then switches them over. Does that imply that I will need nearly the > same amount of space again to perform the compact, until it finishes > and the old database is discarded? Yes. > The manual page says that the new database is built "in a temporary > directory". Where is that directory exactly? Is it inside the > current notmuch database directory or is it in $TMPDIR? I ask Your notmuch db has a xpian subdir, by default named `xapian`. The compact is done in a new dir `xapian.compact` which is a sibling dir to `xapian`. > because it looks like I'll need to make sure that there about 50GiB > of space available wherever that is. > > I'm aware that this procedure is going to take a really really long > time. If my machine should crash, or the notmuch process runs out of > memory or something, will my database be left in a functional state? You can tell `notmuch compact` to move the existing db to a backup dir (on the same file system) after a successful compaction. So, if you don't want to take the risk of `notmuch compact` wrongly considering the process to be successful, just specify a backup dir. Either the compaction fails, in which case the original db is untouched. (In case of a hard crash a file lock might be stale, but that should be it.) Or it succeeds and the old db is in the backup dir. Disclaimer: Back up manually to a different fs before ... or just take a btrfs snapshot. > If I have to reindex it, that is going to take even longer, so I > have to think about how much I want what is probably a very marginal > amount of space back! On my comparatively small db, gains are typically substantial (postlist, position), even though I compact from time to time. Expect 4 tables to be compactified (if you want to start a trial run, say). Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Am Di., 30. Mai 2023 um 18:57 Uhr schrieb Michael J Gruber : > > Am Sa., 27. Mai 2023 um 14:31 Uhr schrieb David Bremner : > > > > Michael J Gruber writes: > > > > > > Are all gpg related tests emacs based? Either gpg or emacs is the red > > > herring here, or both ... > > > > The issue (at least on rawhide) seems to be the interaction between gpg > > and emacs > > > > https://dev.gnupg.org/T6481 > > That was the perfect hint, it seems! > On F39 failing: > T315-emacs-tagging This one I can fix by serializing the test suite. For whatever reasons it fails on newer Fedoras when looping through the different tree mode tests (in parallel). > T350-crypto > T357-index-decryption And these I can fix by using gnupg2 2.4.2 plus the upstream fix from the bug you pointed me to. Hooray! This means that with gnupg2 2.4.1 and 2.4.2 the test suite may fail (I guess emacs/gpg2 will be broken), unless your distro back-ports that fix. (I submitted it for Fedora.) Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Am Sa., 27. Mai 2023 um 14:31 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > > Are all gpg related tests emacs based? Either gpg or emacs is the red > > herring here, or both ... > > The issue (at least on rawhide) seems to be the interaction between gpg > and emacs > > https://dev.gnupg.org/T6481 > > A fix seems to be in progress. Do you have problems with gpg versions > other than 2.4.1? Yes. "It's complicated". Packages in Fedora copr buildroots: epel-8 gnupg2-2.2.20-3.el8_6 emacs-1:26.1-10.el8_8.2 epel-9 gnupg2-2.3.3-2.el9_0emacs-27.2-8.el9_2.1 fedora-37 gnupg2-2.3.8-1.fc37 emacs-1:28.2-3.fc37 fedora-38 gnupg2-2.4.0-3.fc38 emacs-28.2-4.fc38 fedora-39 gnupg2-2.4.1-1.fc39 emacs-28.2-6.fc39 (39=rawhide) RHEL/EPEL/ELN do not have "dtach" and thus skip the pertaining tests! (I'll try this with dtach one day.) notmuch master 0.37^62.gd155f29e (increased lisp timeout 120) (workee epel-8 epel-9) no workee fedora-37 fedora-38 fedora-39 notmuch 0.37 (standard lisp timeout 0.1) no workee fedora-39 everything else works notmuch 0.37 (increased lisp timeout 120) (workee epel-8 epel-9) no workee fedora-37 fedora-38 fedora-39 That is, on F37 and F38 failing: T315-emacs-tagging T460-emacs-tree On F39 failing: T315-emacs-tagging T350-crypto T357-index-decryption Now, I don't know how the timeouts interact and whether increasing the lisp timeout makes the overall timeout kick-in when it should not, of course. Maybe this produces "false fails" on F37 and F38 because you expect some lisp code to "hang" rather than return, and with the increased timeout the test gets killed instead? "Fixing" T350/T357 for the wrong reasons there, I guess. Based on that new hypothesis, I omitted the lisp timeout again, and sure enough, it's only fedora-39/rawhide failing again with T350/T357. So here is hoping that gnupg2-2.4.0 vs gnupg2-2.4.1 indeed makes the difference! Thanks, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Am Sa., 27. Mai 2023 um 14:59 Uhr schrieb David Bremner : > > David Bremner writes: > > > > > I'm not sure if this is the main issue, but in commit bf8aa34324cc91a > > that key was replaced by an ed255519 key > > 9A3AFE6C60065A148FD4B58A7E6ABE924645CC60 > > > > Did the fedora sources get out of synch somehow? > > > > d > > > > PS. Currently spinning up a Fedora Rawhide VM to see if I can duplicate > > the issue without the packaging infrastructure. > > Sorry, that's a silly question, that change is only in master so far. > Not silly at all. I'm seeing this with release (my local mock tests; Fedora koji) and master (Fedora copr). Both build an rpm from a complete tarball (as released resp. git-archive [via rpkg]), so the test suite is in-sync. Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
David Bremner venit, vidit, dixit 2023-05-12 21:17:45: > Michael J Gruber writes: > > > oh well, attachments ... > > > > Can you encrypt to the key 6D92612D94E46381 interactively using an > approriately simplified version of that command? Took me a while, sorry. In that chroot: ``` sh-5.2# gpg --no-tty --import ./test/gnupg-secret-key.asc gpg: directory '/builddir/.gnupg' created gpg: /builddir/.gnupg/trustdb.gpg: trustdb created gpg: key 6D92612D94E46381: public key "Notmuch Test Suite (INSECURE!)" imported gpg: key 6D92612D94E46381: secret key imported gpg: Total number processed: 1 gpg: imported: 1 gpg: secret keys read: 1 gpg: secret keys imported: 1 sh-5.2# echo supersecret | gpg -ear 6D92612D94E46381 gpg: C44D36DEAD54AB16: There is no assurance this key belongs to the named user sub rsa1024/C44D36DEAD54AB16 2011-02-05 Notmuch Test Suite (INSECURE!) Primary key fingerprint: 5AEA B11F 5E33 DCE8 75DD B75B 6D92 612D 94E4 6381 Subkey fingerprint: 8987 5467 478A A81C EBD5 2E7E C44D 36DE AD54 AB16 It is NOT certain that the key belongs to the person named in the user ID. If you *really* know what you are doing, you may answer the next question with yes. Use this key anyway? (y/N) ``` Confirming that works, of course. Also, `gpg --always-trust -ear 6D92612D94E46381` works. ``` sh-5.2# printf '%s:6:\n' "$FINGERPRINT" | gpg --quiet --batch --no-tty --import-ownertrust gpg: inserting ownertrust of 6 ``` (like test-lib.sh does) and then encryption works - no questions asked. So, all that works. Are all gpg related tests emacs based? Either gpg or emacs is the red herring here, or both ... Unfortunately I have no clue about emacs/lisp and cannot dig further there. I just know it's 100% reproducible (for f39 mock on f38, all fedoras in copr, but not f38 mock on f38). Stomped. Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
[now from the subscribed address, sorry] David Bremner venit, vidit, dixit 2023-05-12 21:17:45: > Michael J Gruber writes: > > > oh well, attachments ... > > > > Can you encrypt to the key 6D92612D94E46381 interactively using an > approriately simplified version of that command? Took me a while, sorry. In that chroot: ``` sh-5.2# gpg --no-tty --import ./test/gnupg-secret-key.asc gpg: directory '/builddir/.gnupg' created gpg: /builddir/.gnupg/trustdb.gpg: trustdb created gpg: key 6D92612D94E46381: public key "Notmuch Test Suite (INSECURE!)" imported gpg: key 6D92612D94E46381: secret key imported gpg: Total number processed: 1 gpg: imported: 1 gpg: secret keys read: 1 gpg: secret keys imported: 1 sh-5.2# echo supersecret | gpg -ear 6D92612D94E46381 gpg: C44D36DEAD54AB16: There is no assurance this key belongs to the named user sub rsa1024/C44D36DEAD54AB16 2011-02-05 Notmuch Test Suite (INSECURE!) Primary key fingerprint: 5AEA B11F 5E33 DCE8 75DD B75B 6D92 612D 94E4 6381 Subkey fingerprint: 8987 5467 478A A81C EBD5 2E7E C44D 36DE AD54 AB16 It is NOT certain that the key belongs to the person named in the user ID. If you *really* know what you are doing, you may answer the next question with yes. Use this key anyway? (y/N) ``` Confirming that works, of course. Also, `gpg --always-trust -ear 6D92612D94E46381` works. ``` sh-5.2# printf '%s:6:\n' "$FINGERPRINT" | gpg --quiet --batch --no-tty --import-ownertrust gpg: inserting ownertrust of 6 ``` (like test-lib.sh does) and then encryption works - no questions asked. So, all that works. Are all gpg related tests emacs based? Either gpg or emacs is the red herring here, or both ... Unfortunately I have no clue about emacs/lisp and cannot dig further there. I just know it's 100% reproducible (for f39 mock on f38, all fedoras in copr, but not f38 mock on f38). Stomped. Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
oh well, attachments ... Am Fr., 12. Mai 2023 um 19:42 Uhr schrieb Michael J Gruber : > > Am Do., 11. Mai 2023 um 22:49 Uhr schrieb David Bremner : > > > > Michael J Gruber writes: > > > ... > > > T350-crypto: Testing PGP/MIME signature verification and decryption > ... > > > PASS signature verification with signer key unavailable > > > ``` > > > There the suite "hangs" for about 2 minutes, followed by > > > > This sounds suspiciously like "timeout" kicking in and killing a test > > that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the > > environment to some smaller/larger number to test this. > > Yes, when I set the timeout to 10m it hangs for 10min there, > > > The first subtest in T357 is also sending an encrypted message, so it > > Same after that one. I"ll attach "hang1" and "hang2" which are the > process lists during those hangs. Are those subtests the only ones (or > rather, the first subtests in these tests, and those tests the only > ones) sending encrypted messages? > > > looks like some bad interaction between gpg and emacs. Maybe you can try > > sending an encrypted message from emacs interactively in your chroot > > environment. > > > > 0) you will first need to run > > > >gpg --no-tty --import ./test/openpgp4-secret-key.asc > > > > this will create a keyring etc... in the chroot > > > > 1) You can use the script > > > > ./devel/try-emacs-mua -q > > > > > > 2) Copy the following into *scratch* and run with C-j after the last > > closing paren (this is just copied from the test suite internals) > > > > (let ((message-send-mail-function (lambda () t)) > > (mail-host-address "example.com")) > > (notmuch-mua-mail) > > (message-goto-to) > > (insert "test_su...@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -") > > (message-goto-subject) > > (insert "My Subject") > > (message-goto-body) > > (insert "a body") > > (mml-secure-message-encrypt) > > (let ((mml-secure-smime-sign-with-sender t) > > (mml-secure-openpgp-sign-with-sender t)) > > (notmuch-mua-send-and-exit))) > > > > You will probably need to answer "c" to create the sent-mail folder at > > the prompt. > > Thanks for these instructions. I don't get a prompt: Right after C-j, > that emacs session hangs indefinitely. There was also something about > "require nnheader" or something like that. "hang3" is the hanging > process, which seems to be the same as for the other runs. > > > > At least for T350 this is strange because several subtests ran and > > > passed! This indicates a race or a wrong signal trap. > > > > Note that those files are summaries created by test_done, so it's not > > that surprising that they are not there, since test_done is not reached > > in those files. > > I assumed the subtests would timeout separately and the test framework > would catch that. If the test script itself is killed then yes, that > outcome is expected :) hang1 Description: Binary data hang3 Description: Binary data hang2 Description: Binary data ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Am Do., 11. Mai 2023 um 22:49 Uhr schrieb David Bremner : > > Michael J Gruber writes: > ... > > T350-crypto: Testing PGP/MIME signature verification and decryption ... > > PASS signature verification with signer key unavailable > > ``` > > There the suite "hangs" for about 2 minutes, followed by > > This sounds suspiciously like "timeout" kicking in and killing a test > that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the > environment to some smaller/larger number to test this. Yes, when I set the timeout to 10m it hangs for 10min there, > The first subtest in T357 is also sending an encrypted message, so it Same after that one. I"ll attach "hang1" and "hang2" which are the process lists during those hangs. Are those subtests the only ones (or rather, the first subtests in these tests, and those tests the only ones) sending encrypted messages? > looks like some bad interaction between gpg and emacs. Maybe you can try > sending an encrypted message from emacs interactively in your chroot > environment. > > 0) you will first need to run > >gpg --no-tty --import ./test/openpgp4-secret-key.asc > > this will create a keyring etc... in the chroot > > 1) You can use the script > > ./devel/try-emacs-mua -q > > > 2) Copy the following into *scratch* and run with C-j after the last > closing paren (this is just copied from the test suite internals) > > (let ((message-send-mail-function (lambda () t)) > (mail-host-address "example.com")) > (notmuch-mua-mail) > (message-goto-to) > (insert "test_su...@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -") > (message-goto-subject) > (insert "My Subject") > (message-goto-body) > (insert "a body") > (mml-secure-message-encrypt) > (let ((mml-secure-smime-sign-with-sender t) > (mml-secure-openpgp-sign-with-sender t)) > (notmuch-mua-send-and-exit))) > > You will probably need to answer "c" to create the sent-mail folder at > the prompt. Thanks for these instructions. I don't get a prompt: Right after C-j, that emacs session hangs indefinitely. There was also something about "require nnheader" or something like that. "hang3" is the hanging process, which seems to be the same as for the other runs. > > At least for T350 this is strange because several subtests ran and > > passed! This indicates a race or a wrong signal trap. > > Note that those files are summaries created by test_done, so it's not > that surprising that they are not there, since test_done is not reached > in those files. I assumed the subtests would timeout separately and the test framework would catch that. If the test script itself is killed then yes, that outcome is expected :) ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
T350-crypto T357-index-decryption: possible race condition?
Hi there, my regular notmuch test builds recently started to fail, more concretely: the test suite fails because some subtests are KILLed. Building notmuch 0.37 with my usual spec-file as a rawhide-mock build (a local chroot for the development "version" of Fedora which will become Fedora 39) I see: ``` T350-crypto: Testing PGP/MIME signature verification and decryption PASS emacs delivery of signed message via fcc PASS emacs delivery of signed message via fcc and smtp PASS signed part content-type indexing PASS signature verification PASS detection of modified signed contents PASS corrupted pgp/mime signature PASS signature verification without full user ID validity PASS signature verification with signer key unavailable ``` There the suite "hangs" for about 2 minutes, followed by ``` FATAL: /builddir/build/BUILD/notmuch-0.37/test/T350-crypto.sh: interrupted by signal 15 ``` It proceeds until ``` T357-index-decryption: Testing indexing decrypted mail ``` and hangs again for about 2 minutes, followed by ``` FATAL: /builddir/build/BUILD/notmuch-0.37/test/T357-index-decryption.sh: interrupted by signal 15 ``` In the end, the suite complains: ``` '/builddir/build/BUILD/notmuch-0.37/test/test-results/T350-crypto' does not exist! '/builddir/build/BUILD/notmuch-0.37/test/test-results/T357-index-decryption' does not exist! ``` At least for T350 this is strange because several subtests ran and passed! This indicates a race or a wrong signal trap. The same problem happens with notmuch 0.37 in Fedora's infrastructure (koji rawhide, e.g. https://koji.fedoraproject.org/koji/taskinfo?taskID=101014703). Curiously, everything seems to work with notmuch 0.37 in Fedora 38, which is the current release, in both koji and locally in mock. BUT: In Fedora's secondary test-bed (copr) and with notmuch from git, these kind of errors happen on released fedora versions, too. This was kind of erratic, but I suspected something related to emacs 28 and test timeouts. So I increased the timeout in the test lisp lib (see below), hoping for the better, but getting the worse, at least deterministically worse: With this change, the test suite fails reliably (the two mentioned above plus T315-emacs-tagging) on all Fedoras (with Emacs 28) and passes on epel (with Emacs 27), see for example: https://copr.fedorainfracloud.org/coprs/mjg/notmuch-git/build/5908525/ Now, emacs is not the only difference, and the complete test result directory disappearing is still strange, and really all that is strange. Help please ;) (There is another problem related to Python 3.12 which I'll address separately - rawhide still carries 3.11.) ``` --- test/test-lib.el | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/test-lib.el b/test/test-lib.el index 79a9d4d6..39ade9b9 100644 --- a/test/test-lib.el +++ b/test/test-lib.el @@ -39,7 +39,7 @@ (defun notmuch-test-wait () "Wait for process completion." (while (get-buffer-process (current-buffer)) -(accept-process-output nil 0.1))) +(accept-process-output nil 120))) (defun test-output ( filename) "Save current buffer to file FILENAME. Default FILENAME is OUTPUT." ``` ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Recommended method to pair Notmuch with Neovim?
James Cook venit, vidit, dixit 2023-04-19 04:19:25: > On Wed, Apr 19, 2023 at 12:47:24AM +0200, Ralph Seichter wrote: > > Jon Fineman wrote: > > > > > [database] > > > path=/home/jjf/Maildir > > > > > > > and is your Notmuch DB directory /full/path/to/your/Maildir/.notmuch , > > > > i.e. nested inside your Maildir folder? > > > Yes. > > > > Then our respective directory paths are similar. aerc still does not > > work for me, though. I had suspected that the nested directories might > > be causing trouble for me, but that's apparently not the case. > > > > -Ralph > > You could try tracing aerc's systems calls (e.g. ktrace on BSD, strace > on Linux) to see what file it's trying to access. > > Incidentally, my current favourite notmuch client is neomutt. It > probably helps that I'm already used to it. I tried notmuch-emacs and > aerc, and they were okay, though aerc seemed a bit buggy (but that was a > while ago). I tried meli but it wasn't ready yet (maybe it is now?). I > haven't tried notmuch-vim. Is there any good documentation on neomutt's notmuch support beyond what is on their website? I currently use alot for its great notmuch support (in particular tag completion with search and tag inyterfaces, multiple "tabs"/views) but find myself switching to neomutt for its better mime support (plus s/mime) and features like thread split & join, so far mostly maildir backed. neomutt's doc probably tell you everything you need to know to set up a notmuch-friendly user experience, but ... I know multiple views will not be a thing in neomutt. Easy tagging and nm-based search are a must, though. Being able to switch to the "maildir view" of a message (so that you can rearrange messages within your maildir folder tree) would be a plus. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: inbox-update: new competition of notmuch-lore
Hi Felipe > I'm moving from mbsync to public-inbox and I find there aren't many tools to > make it work with notmuch. Looking at that, too. > I gave a try to notmuch-lore [1] but I found it too slow and had a couple of > issues. > > So I wrote my own script to convert public-inbox mailing lists to Maildir > format: notmuch-tools/inbox-update [2]. > > It's much faster at the initial clone, it deals with deleted mails, and YAML > is > a much better configuration format. Looking at both scripts: Is the speed-up mainly due to `git cat-file` vs. `git show`? > Also, you can configure which epochs you want to fetch (notmuch-lore fetches > all of them). > > One thing it doesn't yet do is trim the repository once the mails have been > converted, but that's probably easy to add later on. What kind of trimming are you thinking about here? Partial history? I guess this shows that public-inbox's repo format is simply not the best choice for the purpose of mail readers. It is optimised for other uses, and I always wondered why they use a non-bare repo at all. That single file path m at the root creates absolutely meaningless diffs. And the commit message doubles the info which is present in the blob. notes-ref could have served better for inspiration of public-inbox. (Barking up the wrong tree, I know.) There are even tools in the public-inbox eco system which feed that info into a xapian db, though not notmuch-like, as if notmuch hadn't existed already. What I'm dreaming of is a notmuch "storage backend" which is git object db based rather than maildir based, and compatible with public-inbox (at least with the use case, i.e. v3 or v4...). I mean - why do we need a checkout of basically immutable files which are stored in blobs already, just so that notmuch can index them? We need them for the MUAs, I know, and we would need a solution for them, too. Or simply a tree in public-inbox which allows clients to use a mere checkout ... Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Data loss
Am Di., 4. Apr. 2023 um 00:54 Uhr schrieb Carl Worth : > > Ouch. > Yes, we're feeling with you, Fulvio :| > It's really unfortunate if notmuch-mutt makes it that easy to throw away > your email. > > That sounds like a nasty bug that should be fixed in that program. To be fair to notmuch-mutt: ``` --output-dir DIR Store search results as (symlink) messages under maildir DIR. Beware: DIR will be overwritten. (Default: F<~/.cache/notmuch/mutt/results/>) ``` So the default is safe, and the warning is there. It would be a bug only if notmuch-mutt descended up and deleted a root dir or such. Maybe naming the option `scratch-dir` or `cache-dir` or such could have helped, but otoh the name of the default is there. Note that neomutt incorporates notmuch-mutt's functionality, it might be a better choice for this and other reasons. > As for recovering, I suppose there _is_ a fair amount of detail in your > notmuch index from all of the position-indexed terms, (as long as you > haven't run "notmuch new" again since the mail was deleted). > > If you still have a large Xapian database in the notmuch database > directory, it would be theoretically possible to recover a lot of the > email content. But I don't know that anyone has ever written a recovery > tool to help with that process. > That is an interesting question for the insiders, indeed - can e-mail bodies be recovered fully (minus capitalisation, stemming)? In Fulvio's case I would try a file-system dump (if it's not too late) and a recovery tool depending on the fs type. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Data loss
Am Mo., 3. Apr. 2023 um 20:17 Uhr schrieb Fulvio Pizzigoni : > > > > > > > Hi Carl e thanks for your prompt answer. > > As you suggested, I add notmuch@notmuchmail.org email as well. > > This is what I did: > fulvio@linux:~$ notmuch setup > Your full name [fulvio]: > Your primary email address [my address]: return > Additional email address [Press 'Enter' if none]: return > Top-level directory of your email archive [/home/fulvio/.mutt]: return > Tags to apply to all new messages (separated by spaces) [unread inbox]: return > Tags to exclude when searching messages (separated by spaces) [deleted spam]: > return > fulvio@linux:~$ > > After this my .mutt directory (~ 4 GB di mail-boxess) appears so: > fulvio@linux:~$ ll .mutt > totale 12 > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp > fulvio@linux:~$ du -h .mutt > 4,0K.mutt/new > 4,0K.mutt/tmp > 4,0K.mutt/cur > 16K .mutt > > That's all. > > Terrible! > > What happened? > > Thanks in advance. > > Fulvio First of all: Do you have a back-up? notmuch itself does not delete mails, as Carl pointed out. In addition to notmuch you mentioned notmuch-mutt. Did you run that manually or using some mutt config? It creates a maildir of symlinks with search results. In order to do so, it deletes the maildir ... Usually this sits in a cache dir, though. Michael > On Mon, Apr 03, 2023 at 09:27:22AM -0700, Carl Worth wrote: > > Hi Fulvio, > > > > I've never used notmuch-mutt. > > > > But notmuch itself doesn't delete any mail. It's really paranoid that > > way, (knowing how valuable mail is). > > > > I would suggest you write an email to the notmuch@notmuchmail.org > > mailing list where you will be able to reach more people likely to have > > experience with all of the software you were using. > > > > And if you could provide more details on the actual steps you used, that > > would be useful. For example, you said "configuration process" and "at > > the end". But what actual commands were you running on those steps, for > > example? > > > > -Carl > > > > On Mon, Apr 03 2023, Fulvio Pizzigoni wrote: > > > Hi. > > > > > > I have installed packages notmuch and notmuch-mutt. > > > > > > During configuration process I have indicated the directory used from > > > Mutt for your mail-boxes. > > > > > > The configuration process have not indicated any allert. > > > > > > At the end the mail-boxes in that directory was removed; 3 new > > > empty sub-directory were created: cur, new, tmp. > > > > > > Was the data irremediably lost? > > > > > > Thanks in advance. > > > > > > Fulvio Pizzigoni > ___ > notmuch mailing list -- notmuch@notmuchmail.org > To unsubscribe send an email to notmuch-le...@notmuchmail.org ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Am Mi., 29. März 2023 um 10:41 Uhr schrieb Felipe Contreras : > > Hi, > > I noticed you promoted notmuch-git as a user tool to toy around with it. > > Very quickly I realized that most of what it does is something I've > been working on for at least 10 years: making git work with other > tools. > > I presume you haven't heard of git remote-helpers [1], because they do > precisely what notmuch-git is trying to do. > Hi Felipe that's an interesting idea for sure. When I came across `notmuch-git` first I wondered whether it rather should be`git-notmuch`, i.e. a subcommand to `git`. I admit that - given its preexistence as nmbug - I was never quite sure what to use it for. Maybe sync tags for mail stores whose content you sync otherwise? `public-inbox` came to my mind in this context, too. (I wondered about an nm backend for that, i.e. a public-inbox backed mailstore for notmuch, without multiple checkouts.) So, if we consider the notmuch database (more precisely: the dump output) as a "remote", then what is the history? I understand that we can transfer and transform its content in the form of blobs as specific paths encoding mid etc. Is the history stored by current `notmuch-git` something secondary (say, like the history of notes refs in git) which can be discarded? Note that I haven't looked at your code thoroughly yet (I'm not a rubyist), and I'm all for using git tools to do gittish things and more; I'm just wondering whether fast-import/export cover what current `notmuch-git` intends to do. They are probably the best tool for "cloning" an existing nm-db into a git repo of mid-tag associations. And if all you want is a gittish transport for nm tags then that's probably perfect! `notmuch-git` seems to be about handling both updates (commit etc) and queries (log etc), too, as a wrapper to git commands. Those may be candidates for other git tools, such as aliases, diff helpers, textconv and such. In summary, I think a notmuch-git repo is more than a conversion of notmuch-dump output (it adds history and commit messages; we have a "one-sided inverse" only), and the notmuch-git command is more than a converter between the respective data stores. It smells more like `git-lfs` or other filter-based approaches, storing the real objects outside of the git repo. But I feel I know too little about `notmuch-git`'s purpose so far. Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/tree: use two argument form of setq-local
Am Di., 21. Feb. 2023 um 12:49 Uhr schrieb David Bremner : > > Apparently the macro setq-local only takes two arguments in Emacs 26.1 > --- > emacs/notmuch-tree.el | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el > index 14775d59..b58fa6a6 100644 > --- a/emacs/notmuch-tree.el > +++ b/emacs/notmuch-tree.el > @@ -1451,11 +1451,11 @@ notmuch-tree buffers, just set >(unless (derived-mode-p 'notmuch-tree-mode) > (user-error "notmuch-tree-outline-mode is only meaningful for notmuch > trees!")) >(if notmuch-tree-outline-mode > - (progn (setq-local outline-regexp "^[^\n]+" > -outline-level #'notmuch-tree-outline--level) > + (progn (setq-local outline-regexp "^[^\n]+") > +(setq-local outline-level #'notmuch-tree-outline--level) > (notmuch-tree-outline--set-visibility)) > -(setq-local outline-regexp (default-value 'outline-regexp) > - outline-level (default-value 'outline-level > +(setq-local outline-regexp (default-value 'outline-regexp)) > +(setq-localoutline-level (default-value 'outline-level > Thanks, notmuch.git master plus this patch builds happily again on EPEL 8 (and 9, and ...). Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Test breakage on EPEL 8 with tree-outline-mode
Hi there, the test suite started to fail on EPEL 8 with the new tree-outline-mode: ``` EMACS emacs/notmuch-tree.elc In notmuch-tree-mode: emacs/notmuch-tree.el:1088:9:Warning: reference to free variable `notmuch-tree-outline-enabled' In notmuch-tree-outline--set-visibility: emacs/notmuch-tree.el:1322:14:Warning: reference to free variable `notmuch-tree-outline-mode' emacs/notmuch-tree.el:1415:20:Error: Wrong number of arguments: (2 . 2), 4 make: *** [emacs/Makefile.local:87: emacs/notmuch-tree.elc] Error 1 ``` Does the implementation require a specific emacs version? EPEL 8 has emacs 26.1. I am not going to update official notmuch packages for EPEL 8 anyways, but others might care about older emacs. (I do provide notmuch-git test packages for EPEL 8 so far.) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Am Di., 14. Feb. 2023 um 02:47 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > That is really weird: > > ``` > > xapian-delve -t G00021229 . > > Posting List for term 'G00021229' (termfreq 115, collfreq 0, > > wdf_max 0): 146259 ... > > ``` > > with 115 record numbers, all different. > > Doing `xapian-delve -1r` for each of them and grepping for the G-lines > > gives 115 times that correct thread id. > > Grepping for the Q-lines and notmuch-searching for the message ids > > gives only 5 results (the expected ones). Apparantly, there are bogus > > mail records which that thread points to. > > 1) Do those "bogus" records have a "Tghost" term? That would be for > messages that are known via references, but not actually in the local > database. This is a bug / feature of the current implementation, it > counts all messages known, whether or not local copies exist. Yes, the extra ones all are ghosts, and I slowly remember that they scared me in the past already ... These ghosts appear to be pretty common. It happens all the time that I am joined to an existing discussion thread where I do not have all references. I'd go as far as to say that counting ghosts as thread members makes this useless for me. On the other hand, notmuch's own count gets this right. And getting different counts is even more confusing. > 2) Do they have more than one G term? That suggests a bug somewhere. We > actually have a test in the test suite [1] for that, but of course that is > with a simple artificial database. No, they all have one. But their sheer number looks suspicious: those 5 "real" e-mails have maybe 20 reference headers in total, and some of them refer to some of those 5. Grepping the account store for those references gives me around that number. Where do the 110 ghosts (90 extra) come from which this thread points to? Still scared by them ... we need ghost busters! Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Am Mo., 13. Feb. 2023 um 21:23 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > > It has 5, as confirmed by the search output and that of `notmuch > > count`. But it is matched by `count 115`. > > `xapian-check` is happy. (There used to be some issue with additional > > thread entries at some point.) > > > > Michael > > A simple test to try is > > % xapian-delve -t G00021229 \ > ~/.local/share/notmuch/default/xapian > > adjusting your database path as needed. > > If that says "termfreq 115", then something is broken (or at least > confusing) about your database (possibly related to the previous issues > with threading). In that case I'm curious if there are 115 distinct > record numbers. You can find all of the thread-ids attached to a given > message with > > % xapian-delve -1r 267585 ~/.local/share/notmuch/default/xapian | grep ^G > > where 267585 is an example record number in my database. That is really weird: ``` xapian-delve -t G00021229 . Posting List for term 'G00021229' (termfreq 115, collfreq 0, wdf_max 0): 146259 ... ``` with 115 record numbers, all different. Doing `xapian-delve -1r` for each of them and grepping for the G-lines gives 115 times that correct thread id. Grepping for the Q-lines and notmuch-searching for the message ids gives only 5 results (the expected ones). Apparantly, there are bogus mail records which that thread points to. I guess I should recreate the db, if I only knew how lieer deals with a reindexed mail store ... (The thread and the 5 message sit in an mbsynced folder, but lieer syncs other folders with that same db). Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Am Mo., 13. Feb. 2023 um 17:32 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > I am getting a few surprising matches, e.g. > > ``` > > notmuch search --query=sexp '(thread (count 115)))' > > thread:000000021229 2021-05-17 [5/5] Michael J Gruber ... redacted > > notmuch count --exclude=false thread:00021229 > > 5 > > ``` > > It could be some database issues, of course. Or me misunderstanding > > something :) > > Hmm. I don't see any strange matches for that particular query, just a > thread that actually has 115 messages. But there could also be bugs of > course. Does xapin-check complain about your database? It has 5, as confirmed by the search output and that of `notmuch count`. But it is matched by `count 115`. `xapian-check` is happy. (There used to be some issue with additional thread entries at some point.) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Am Mo., 13. Feb. 2023 um 13:26 Uhr schrieb David Bremner : > > So for this only supports counting messages in threads, and the sexp > based query parser. It seems useful to expand it to other fields > (from, e.g.). I'm not sure how motivated I am to shim this into the > infix query parser, but we will see how it goes. This certainly looks interesting, and not easy to get by scripting around the existing commands. It is kinda special, so having it in sexp only seems okay. I am getting a few surprising matches, e.g. ``` notmuch search --query=sexp '(thread (count 115)))' thread:00021229 2021-05-17 [5/5] Michael J Gruber ... redacted notmuch count --exclude=false thread:00021229 5 ``` It could be some database issues, of course. Or me misunderstanding something :) Patch 1/2 is crlf garbled, by the way. Applies cleanly after removing the extra ^Ms. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: ## error when running notmuch from emacsclient, but not from standalone emacs instance
Am Mi., 18. Jan. 2023 um 15:40 Uhr schrieb David Bremner : > > ricardomart...@riseup.net writes: > > > > > When I try to run notmuch from the emacsclient instance (M-x notmuch > > ), I get an error > > "notmuch count --batch failed > > Please check that the notmuch CLI is new enough to support `count > > --batch'. In general we recommend running matching versions of > > the CLI and emacs interface." > > > > I agree it is strange strange to have problems only affecting > emacsclient. The only thing I can think of off hand is that starting > emacsclient does not reload the emacs initialization file, so make sure > you run both experiments with a fresh restart of emacs. > > What is the output of M-x notmuch-version for both emacsclient and emacs? ... and how are you starting the emacs server? Depending on that, server and standalone may get a different PATH from the environment so that one uses the system notmuch and the other your personal install. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test failure in Ubuntu 22.04 and 22.10 (new test)
Am Do., 6. Okt. 2022 um 18:34 Uhr schrieb David Bremner : > > Michael J Gruber writes: > > > > > Yes, lto-wrapper calls make. > > > > Are we compiling test functions on the fly during the test? In that > > case we need to make sure that each test depends on the build > > products, or else the test helper compilation and its users might run > > in parallel ... > > Yes, we compile C code on the fly during the run of the tests. I'm not > really clear on what race condition you are anticipating, as neither the > compilation nor the other parts of the test are directly run by make. > Execution is sequential within each T*.sh file. Unless gcc is returning > before it has finished compilation (which I think we'd all agree would > be gcc bug), I don't see how a race can arise there. One thing I can > imagine happening is gcc's recursive invocation of make somehow fails > under make -j, possibly something to do with violated assumptions about > the jobserver and/or environment variables. What I mean is: make calls T*.sh T*.sh calls gcc gcc calls make (for lto) Could it be that within a parallel make session, that gcc-make-call gets delegated to the master make jobserver and thus gcc returns too early? Wild speculation, I admit. I haven't checked the code, but having those testhelpers as prerequisites of the test scripts may help in that case. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] test: replace aging OpenPGP key used in the test suite
Am Do., 22. Sept. 2022 um 12:14 Uhr schrieb Justus Winter : > > Michael J Gruber writes: > > > Am Do., 22. Sept. 2022 um 10:47 Uhr schrieb Justus Winter > > : > >> > >> This replaces the old OpenPGPv4 key that is used in the test suite > >> with a more modern OpenPGPv4 key. All cryptographic artifacts in the > > > > Both v4? Only one key file is named v4. > > Yes, the old key was also a v4 key. In this context, OpenPGP v4 was > standardized in 1998. So when the old key was created, v4 was and has > been for a long time *the* version of OpenPGP. It didn't seem to make > sense to specify the version. > > Now, v5 is around the corner, so it makes sense to make the version > explicit. That'll help when we introduce v5 artifacts. > > >> @@ -6,7 +6,7 @@ Message-ID: > >> MIME-Version: 1.0 > >> Content-Type: multipart/signed; boundary="=-=-="; > >> protocol="application/pgp-signature"; > >> - micalg=pgp-sha512 > >> + micalg=pgp-sha256 > > > > You are downgrading the hash algo here and in the other regenerated > > signatures. This is not wrong per-se, I'm just wondering whether it is > > intentional (or forced by the standard) when the aim of this series is > > future-proofing. sha256 is the current "replacement" for sha1, which > > means it's the one which will be replaced next ;) > > Yes I am. It happened when I re-created the signature. Recreating the > artifacts was somewhat tedious (I'm working on tooling for that, but the > changes to notmuch I created by hand), so I opted for the easiest fix. > > WRT future proofing: SHA256 is the only mandatory to implement hash > algorithm in v5 OpenPGP. Therefore, when SHA256 falls, we will > hopefully have specified v6 OpenPGP which moved to a new MTI hash > algorithm. So, for a v4 OpenPGP artifact, SHA256 is and will forever be > more than appropriate. > > Best, > Justus Thanks for clarifying, sounds good to me! Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] test: replace aging OpenPGP key used in the test suite
Am Do., 22. Sept. 2022 um 10:47 Uhr schrieb Justus Winter : > > This replaces the old OpenPGPv4 key that is used in the test suite > with a more modern OpenPGPv4 key. All cryptographic artifacts in the Both v4? Only one key file is named v4. > @@ -6,7 +6,7 @@ Message-ID: > MIME-Version: 1.0 > Content-Type: multipart/signed; boundary="=-=-="; > protocol="application/pgp-signature"; > - micalg=pgp-sha512 > + micalg=pgp-sha256 You are downgrading the hash algo here and in the other regenerated signatures. This is not wrong per-se, I'm just wondering whether it is intentional (or forced by the standard) when the aim of this series is future-proofing. sha256 is the current "replacement" for sha1, which means it's the one which will be replaced next ;) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: best practices for keeping a notmuch tag in sync with maildir folder
Am Mo., 5. Sept. 2022 um 13:33 Uhr schrieb David Bremner : > > Bence Ferdinandy writes: > > > are there any best practices for keeping notmuch tags synced with an imap > > folder? It seems to me that simply moving a mail file in the shell from one > > folder to the other is not something that works well, at least with mbsync > > the moved file didn't appear in my webmail (I would like to keep a few tags > > mapped one-to-one with folders so I have a rudimentary access to some tags > > on my phone and webmail). > > As far as I know (as a non-mbsync user) this is only a problem for > mbsync. Notmuch itself is fine with moving (or copying) files between > folders. You will have to run notmuch-new afterwards. If I remember > correctly mbsync encodes an IMAP UUID into the file name, but I don't > know the details. I guess that the general mbsync instructions for how > to move messages should work fine with notmuch. Yes, that is the main issue, as well as "by design" if you use mbsyncs "native" scheme, which is the fastest option. Alternatively, you can use the "alternative" scheme ;) You can also make sure to drop the uuid part from the filename when you move things around. But for keeping things in-sync you might want to look at something like https://github.com/afewmail/afew which allows you to specify mappings. Personally, I ended up mapping only coarsely (things like Inbox, Spam, Trash, Drafts) and using tag based search in notmuch. Well, plus mapping some list headers to tags, moving messages between accounts based on labels, and ... It's really versatile, and the information is "stored in the tags" (plus folders), i.e. it does not tie you to an extra db or such. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: notmuch release 0.37 now available
Thanks for another fine release, and for being a pleasant upstream to work with! For Fedora Linux/RHEL/etc folks: notmuch 0.37 is available in rawhide and f37 (branched), as well as in updates-testing for the current release f36, all with sfsexp support. It will not go to any EPEL branch (per guidelines), they will remain at 0.36 (el9) resp. 0.35 (el8). As always, you can get the most current version for any supported Fedora Linux or EPEL branch from COPR: https://copr.fedorainfracloud.org/coprs/mjg/notmuch This includes EPEL 9 (with sfsexp support) and EPEL 8 (without), all built from the same source package. For now, I have also enabled CentOS Stream and ELN which fail to build because of missing requirements (the devil is in the devel packages; I will sort this out or disable those chroots again). So don't get worried by the overall "red light", builds for all other branches are "green" (with the test suite enabled). Cheers Michael (mjg) ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: nmweb HTML injection
Am Mo., 22. Aug. 2022 um 09:22 Uhr schrieb Jakub Wilk : > > See: https://nmbug.notmuchmail.org/nmweb/search/markup%20where%20appropriate > > and from the mail subject was dumped without escaping into HTML. > Interesting :) The body is htmlescape()ed, but the subject header is used as is. I should be escaped too. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 4/4] CLI/git: replace calls to notmuch-search with database access
Am Do., 7. Juli 2022 um 17:59 Uhr schrieb David Bremner : > > Tomi Ollila writes: > > > On Sun, Jul 03 2022, David Bremner wrote: > > > > I've trying to think if there were a way to somehow run only one notmuch > > command instead of notmuch search on all maeby-deleted files -- or > > alternatively attempt to load python bindings and in case of failure use > > the notmuch-search methid... > > > > We could run one notmuch-dump command to get a list of the message-ids > in the database, and build a dictionary in memory. Might be a bit slow? > Here it would take 10-15s just do the dump. But certainly faster than > 500K execs of notmuch search When I first saw that `notmuch-git` is implemented in python and calls out to `notmuch search` I wondered: Why doesn't it use the python bindings? I don't think "building the project partially and installing by copying parts somewhere" is a use case that the design implementation has to cater for, especially if this incurs performance penalties. Scripting around `notmuch dump` does not make things better. I do understand that you want lean dependencies server side, but having python there isn't really uncommon, is it? If building and installing from git via `make install` is too much of a hassle we should probably work on reducing the hassle ;) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] doc: replace symlink with duplicate page for nmbug
Am Do., 30. Juni 2022 um 13:20 Uhr schrieb David Bremner :
>
> This automatically propagates to the info version of the pages, which
> saves having maintain two sets of symlinks.
> ---
>
> What do you think about this as an alternative approach? It seems
> slightly more maintainable to me, although I am willing to be
> convinced otherwise. If it is an improvement we could do the same for
> the notmuch-setup page.
Looks OK to me. I just followed the man example (by linking), after I
had noticed the asymmetry, but copies work fine, too.
You probably also want to add them to `make install`.
> doc/Makefile.local | 3 ++-
> doc/conf.py| 4
> 2 files changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/doc/Makefile.local b/doc/Makefile.local
> index 2f67f4de..123ef10b 100644
> --- a/doc/Makefile.local
> +++ b/doc/Makefile.local
> @@ -19,6 +19,7 @@ MAN_RST_FILES := $(MAN1_RST) $(MAN5_RST) $(MAN7_RST)
> ALL_RST_FILES := $(MAN_RST_FILES) $(srcdir)/doc/notmuch-emacs.rst
>
> MAN1_ROFF := $(patsubst
> $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN1_RST:.rst=.1))
> +MAN1_ROFF := $(MAN1_ROFF) $(DOCBUILDDIR)/man/man1/nmbug.1
> MAN5_ROFF := $(patsubst
> $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN5_RST:.rst=.5))
> MAN7_ROFF := $(patsubst
> $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN7_RST:.rst=.7))
> MAN_ROFF_FILES := $(MAN1_ROFF) $(MAN5_ROFF) $(MAN7_ROFF)
> @@ -26,6 +27,7 @@ MAN_ROFF_FILES := $(MAN1_ROFF) $(MAN5_ROFF) $(MAN7_ROFF)
> MAN_GZIP_FILES := $(addsuffix .gz,${MAN_ROFF_FILES})
>
> MAN1_TEXI := $(patsubst
> $(srcdir)/doc/man1/%.rst,$(DOCBUILDDIR)/texinfo/%.texi,$(MAN1_RST))
> +MAN1_TEXI := $(MAN1_TEXI) $(DOCBUILDDIR)/texinfo/nmbug.info
> MAN5_TEXI := $(patsubst
> $(srcdir)/doc/man5/%.rst,$(DOCBUILDDIR)/texinfo/%.texi,$(MAN5_RST))
> MAN7_TEXI := $(patsubst
> $(srcdir)/doc/man7/%.rst,$(DOCBUILDDIR)/texinfo/%.texi,$(MAN7_RST))
> INFO_TEXI_FILES := $(MAN1_TEXI) $(MAN5_TEXI) $(MAN7_TEXI)
> @@ -131,7 +133,6 @@ install-man: ${MAN_GZIP_FILES}
> install -m0644 $(filter %.5.gz,$(MAN_GZIP_FILES))
> $(DESTDIR)/$(mandir)/man5
> install -m0644 $(filter %.7.gz,$(MAN_GZIP_FILES))
> $(DESTDIR)/$(mandir)/man7
> cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch.1.gz notmuch-setup.1.gz
> - cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch-git.1.gz nmbug.1.gz
> endif
>
> ifneq ($(HAVE_SPHINX)$(HAVE_MAKEINFO),11)
> diff --git a/doc/conf.py b/doc/conf.py
> index f01c0058..ed27c3f4 100644
> --- a/doc/conf.py
> +++ b/doc/conf.py
> @@ -127,6 +127,10 @@ man_pages = [
> u'manage notmuch tags with git',
> [notmuch_authors], 1),
>
> +('man1/notmuch-git', 'nmbug',
> + u'manage notmuch bugs with git',
> + [notmuch_authors], 1),
> +
> ('man5/notmuch-hooks', 'notmuch-hooks',
> u'hooks for notmuch',
> [notmuch_authors], 5),
> --
> 2.35.2
>
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: sexp and strings
Am Mo., 13. Juni 2022 um 11:31 Uhr schrieb erik colson : > > Michael J Gruber writes: > > > That search works without the macro, but not as a macro: notmuch > > computes an empty `Query()` for this (as per `NOTMUCH_DEBUG_QUERY=1`). > > I'm not sure whether this is intended or an artefact of the > > implementation, since both the macro and the regex need > > expansion/evaluation before being fed to xapian, and the order > > matters. > > > > Defining `D=(macro (dossier) ((tag ,dossier)))` and calling it with > > `(D (rx d))` works, btw (but is not what you want, obviously), so > > something tells me lazy evaluation of macros is not completely lazy ;) > > Thanks for checking this out Michael. I decided to write an emacs lisp > function which I keybind and which prompts for the variable, and then > launches notmuch-search with that: > > (defun ec/notmuch-search-dossier () > "Zoek mails van een dossier" > (interactive >(let* ((dossier (read-no-blanks-input "Dossier:")) > (zoek (concat "tag:/" dossier "/"))) > (notmuch-search zoek nil nil nil nil > > This works like a charm ;) Well, I am of course open to enhancement > suggestions! > > Also I upgraded my OS to fedora36 and now I use the standard notmuch > package instead of compiling it myself. So I don't have sexp support > anymore and therefor I am moving the squeries I added to my notmuch > config into emacs lisp functions. FYI: I submitted sfsexp to fedora (review pending), and as soon as that is in, I will adjust the notmuch package. (Until then there is copr mjg/notmuch-sfsexp at your own risk ;)) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: sexp and strings
Am Mo., 13. Juni 2022 um 01:20 Uhr schrieb erik colson : > > Hi, > > I would like to define a squery in my notmuch configuration which would > ease a query I often use. The query is > > tag:/d/ > > where d are decimal numbers. > Now I would like to shorten this to > > D d > > wherefor I was thinking of using a macro like: > > D=(macro (dossier) ((tag (regex ,dossier > > But this doesn't seem to do the job. > Any ideas how I can achieve this ? That search works without the macro, but not as a macro: notmuch computes an empty `Query()` for this (as per `NOTMUCH_DEBUG_QUERY=1`). I'm not sure whether this is intended or an artefact of the implementation, since both the macro and the regex need expansion/evaluation before being fed to xapian, and the order matters. Defining `D=(macro (dossier) ((tag ,dossier)))` and calling it with `(D (rx d))` works, btw (but is not what you want, obviously), so something tells me lazy evaluation of macros is not completely lazy ;) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: contact instead of (to or from)
[rearranged for bottom post style] Am Fr., 10. Juni 2022 um 05:51 Uhr schrieb 杨令 : > On Fri 2022-06-10 00:35:12, erik colson wrote: > > Date: Fri, 10 Jun 2022 00:35:12 +0200 > > From: erik colson > > To: notmuch@notmuchmail.org > > Subject: contact instead of (to or from) > > > > Hi, > >> > > Coming from mu4e I loved the query "contact" which includes "to" or > > "from" the contact in the results. > > Is there something equivalent in notmuch? > ⮩ Hi, Erik, I don't think notmuch has a built-in `contact` query, but I > think it's easy to build a saved search. > > (add-to-list 'notmuch-saved-searches '(:name "query" :query "to:... or > from:..." :key "r")) > > `:key` part is optional. > > -- > L.Yang If you have notmuch with sexp queries there is a pure notmuch solution: notmuch config set squery.Contact '(macro (x) (or (from ,x) (to ,x)))' Use it with either of the following forms: notmuch count --query=sexp -- '(Contact erik)' notmuch count sexp:'"(Contact erik)"' Fun fact: I learned this by mistaking your emacs lisp for sexp first and vaguely remembering the existence of sexp macros, then rtfming ... Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 2/2] lib: add sexp: prefix to Xapian (infix) query parser.
This breaks builds if you don't have sfexp, unfortunately. (And since there is no sfsexp release with the recent fixes, there is no sfsexp in Fedora, for example.) Not declaring _notmuch_sexp_string_to_xapian_query() twice in lib/database-private.h (once unconditionally, once depending on HAVE_SFSEXP) may be part of the fix, as well as guarding it in lib/query.cc. But I'm not sure what SexpFieldProcessor::operator() should do when sfsexp is not available - throw a Xapian error? Am Fr., 15. Apr. 2022 um 13:32 Uhr schrieb David Bremner : > David Bremner writes: > > > This is analogous to the "infix" prefix provided by the s-expression > > based query parser. > > series applied to master. > ___ > notmuch mailing list -- notmuch@notmuchmail.org > To unsubscribe send an email to notmuch-le...@notmuchmail.org > ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/2] test/smime: fix signature verification test with newer gmime.
Am Mo., 11. Apr. 2022 um 02:36 Uhr schrieb Daniel Kahn Gillmor <
d...@debian.org>:
> Thanks, Bremner!
>
> This series looks reasonable to me. nice clever hack to reuse the
> gmime embedded .c source for the test.
>
Just so that others don't have to be wondering, too: notmuch does not embed
gmime sources and does not reuse them. (It would be very wrong to do so.)
notmuch has a file `_check_gmime_cert.c` which is used for configure checks
and which David cleverly amended to check for the return format of
signature checks (when compiled against the libgmime3).
If I read 2/2 correctly, though, then T355-smime does not adjust its
expected textual outcome to the results of the check, but rather marks the
test "known broken" if the signature check does not return the "new"
format. In other words: Unless you have a very new unpatched gmime,
T355-sime does not "really" do this subtest any more - it is happy as soon
as it fails for any reason.
> a bit of a tweak below:
>
> On Sat 2022-04-09 09:34:53 -0300, David Bremner wrote:
> > + printf "Checking for GMime new email format... "
> > + if ${CC} -DCHECK_EMAIL ${CFLAGS} ${gmime_cflags}
> _check_gmime_cert.c ${gmime_ldflags} -o _check_email &&
> > + GNUPGHOME=${TEMP_GPG} ./_check_email; then
> > + gmime_new_email_format=1
> > + printf "Yes.\n"
> > + else
> > + gmime_new_email_format=0
> > + printf "No (some tests will be skipped).\n"
> > + fi
> > else
> > printf 'No.\nFailed to set up gpgsm for testing X.509 certificate
> validity support.\n'
> > errors=$((errors + 1))
>
> Words like "new" have a tendency to get, well, old.
>
> I'd say
>
>"Checking GMime emits email addresses from certs without angle
> brackets..."
>
> And i'd name the variable gmime_cert_addresses_have_angle_brackets (so
> "1" effectively means "probably a stale, deprecated version of GMime").
>
> Then change the rest of the tests to match.
>
> This is kind of an aesthetic choice -- i'd be fine with the original
> patch too. but it seems safer to just identify the out-of-date stuff
> when it happens, rather than identifying the current stuff.
>
> --dkg
> ___
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-le...@notmuchmail.org
>
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/4] test: due not pass T380.1 for the wrong reasons
David Bremner venit, vidit, dixit 2022-02-20 00:02:40: > Tomi Ollila writes: > > > > > Wat? afaik echo is builtin in every modern bourne shell derivative... > > > > (I tested: > > $ bash -c 'builtin echo foo' > > foo > > $ bash -c 'export PATH=/tmp; echo foo; ls' > > foo > > bash: ls: command not found > > ) > > Oops. That's what I get for believing "which". Which is another tale of > woe, of course. Builtin in zsh, and not in bash. Not that it matters > here, but probably why it doesn't know about bash builtins. echo is both a builtin (for bash etc) and a command: $ type echo echo is a shell builtin $ which echo /usr/bin/echo The latter is part of GNU coreutils (sh-utils) and often used when one wants to avoid shell pecularities. printf is a builtin Due to the way the builtins were specified (or underspecified in the case of echo) in POSIX, the printf builtin is more portable than the echo builtin. As a rule of thumb, the echo builtin is OK if you don't need options (nor special control sequences) but want a newline at the end of the output anyways; use printf if you need options or format strings. Alternatively, require GNU coreutils and use /usr/bin/echo. Cheers Michael P.S.: I haven't forgotten about the series but it needs quite a bit of rework as we found out. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] test: allow to use --full-sync
David Bremner venit, vidit, dixit 2022-02-16 14:04:17: > Michael J Gruber writes: > > > Some build infrastructure appears to habe problems with mtime/stat, > > have Huh, this looks as if my mother tongue slipped in, while it's just key proximity. > > leading to spurious failures (noticed on s390x and aarch64 with epel8). > > Allow the test suite to be run with --full-sync so that release builds > > can use the test suite while avoiding spurious failures. > > > > Signed-off-by: Michael J Gruber > > --- > > test/README | 8 > > test/test-lib-common.sh | 7 ++- > > test/test-lib.sh| 2 +- > > 3 files changed, 15 insertions(+), 2 deletions(-) > > > > diff --git a/test/README b/test/README > > index 10f127cb..1de79b78 100644 > > --- a/test/README > > +++ b/test/README > > @@ -110,6 +110,14 @@ printed on screen. This printing can be disabled by > > setting the > > NOTMUCH_TEST_QUIET variable to a non-null value. Message on test > > failures and skips are still printed. > > typo in variable name. Not here, but in fact in the actual patch ;) > How confident are you that this actually fixes your intermittent > failures? There are quite a few literal invocations of "notmuch new" in > the test suite. Would those need to be changed as well? About 80% confident ... The patch covers all areas in which I've ever witnessed the spurious FAILs. I have not had a single failure with the patch. Many of the other invocations are not immediately after message creation. Some of them are immediately after, and - given that notmuch new catches the case of same time - I do not understand why this happens at all. That accounts for the other 20%. I don't mind carrying this locally and retrying without for the next round of notmuch updates. Having the patch here on-list may help someone else in the future in any case. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH] test: allow to use --full-sync
Some build infrastructure appears to habe problems with mtime/stat,
leading to spurious failures (noticed on s390x and aarch64 with epel8).
Allow the test suite to be run with --full-sync so that release builds
can use the test suite while avoiding spurious failures.
Signed-off-by: Michael J Gruber
---
test/README | 8
test/test-lib-common.sh | 7 ++-
test/test-lib.sh| 2 +-
3 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/test/README b/test/README
index 10f127cb..1de79b78 100644
--- a/test/README
+++ b/test/README
@@ -110,6 +110,14 @@ printed on screen. This printing can be disabled by
setting the
NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
failures and skips are still printed.
+File system/mtime issues
+
+Normally, `notmuch new` uses directory modification times, and the test
+suite checks that this works. If you encounter spurious test failures
+where messages added by one subtest appear to show up in the next subtest
+only (indicating mtime/stat problems) you can set NOTMUCH_TEST_FULLSYNC
+so that the test suite uses the --full-sync option of notmuch new.
+
Skipping Tests
--
If, for any reason, you need to skip one or more tests, you can do so
diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
index ebbf4cdf..91039478 100644
--- a/test/test-lib-common.sh
+++ b/test/test-lib-common.sh
@@ -29,6 +29,11 @@ if [[ -z "$NOTMUCH_SRCDIR" ]] || [[ -z "$NOTMUCH_BUILDDIR"
]]; then
exit 1
fi
+if test -n "$NOTMUCH_TEST_FULLSCAN"
+then
+ NOTMUCH_NEW_OPTIONS="--full-scan"
+fi
+
backup_database () {
test_name=$(basename $0 .sh)
rm -rf $TMP_DIRECTORY/notmuch-dir-backup."$test_name"
@@ -226,7 +231,7 @@ EOF
# are also supported here, so see that function for details.
add_message () {
generate_message "$@" &&
-notmuch new > /dev/null
+notmuch new $NOTMUCH_NEW_OPTIONS > /dev/null
}
if test -n "$valgrind"
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 833bf5fe..0167b95f 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -521,7 +521,7 @@ test_json_nodes () {
}
NOTMUCH_NEW () {
-notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found
[0-9]* total file'
+notmuch new $NOTMUCH_NEW_OPTIONS "${@}" | grep -v -E -e '^Processed
[0-9]*( total)? file|Found [0-9]* total file'
}
NOTMUCH_DUMP_TAGS () {
--
2.35.1.476.gacce7e407c
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
> > > BTW, I have so far run the test suite 68 times in a row without failures > > > on a Debian s390x host. The file system is ext4, mounted relatime. It > > > would be interesting to know what file system is yielding the failures > > > Michael is seeing. > ... > So I'll try out a bit more, and if everything else fails, then running > the test suite with --full-scan is still better than not running it at > all. (I would have suggested a make variable, but it's really a corner > case. given the stat/mtime check). Last update for now: It's still happening sporadically only, and the archs where it does happen use xfs (relatime,seclabel,attr2) resp. tmpfs (nosuid,nodev,seclabel). It has to be something weird on our infra or specifically el8. So, either I implement `make test FULLSCAN=1` or patch in `--full-scan` or disable the tests on epel. Thanks for your patience :) Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
Tomi Ollila venit, vidit, dixit 2022-02-14 23:53:54: > On Mon, Feb 14 2022, David Bremner wrote: > > > Tomi Ollila writes: > > > >> > >> Looked notmuch-new.c -- time_t (seconds since epoch) is used as timestamp > >> comparisons (which would indicate the subsecond resolution most fs' provide > >> is not used)... > >> > >> ... and if so, I wonder why some of our tests are not failing all the time > >> for everyone...? > > > > Not claiming everything is fine, but there is code there targetted at > > the failure mode you mentioned: > > > > /* If the directory's mtime is the same as the wall-clock time > > * when we stat'ed the directory, we skip updating the mtime in > > * the database because a message could be delivered later in this > > * same second. This may lead to unnecessary re-scans, but it > > * avoids overlooking messages. */ > > if (fs_mtime != stat_time) > > _filename_list_add (state->directory_mtimes, path)->mtime = fs_mtime; > > This sure had to be tested... :D > > so I outcommented the line as // if (fs_mtime != stat_time) > > and then build and run test -- a lot of failures... > > also > > ./test/T750-gzip.sh 2>&1 | grep -e PASS -e FAIL > > PASS Single new gzipped message > PASS Single new gzipped message (full-scan) > FAIL Multiple new messages, one gzipped > FAIL Multiple new messages, one gzipped (full-scan) > FAIL Renamed (gzipped) message > PASS notmuch search with partially gzipped mail store > FAIL notmuch search --output=files with partially gzipped mail store > PASS show un-gzipped message > PASS show un-gzipped message (format mbox) > PASS show un-gzipped message (format raw) > FAIL show gzipped message > FAIL show gzipped message (mbox) > FAIL show gzipped message (raw) > PASS new doesn't run out of file descriptors with many gzipped files > > (above was "lucky" run, usually that 6th test, ...partially gzipped... > test also FAILed (I'd guess second happened to change there)). > > then restored the fs_mtime != stat_time line -- then all of 750 passed. > > (finally, run that 750-gzip in a loop (dropped last, slow test), hundreds > of times already -- no FAILures... (ecryptfs on ext4)) > > Tomi > > > BTW, I have so far run the test suite 68 times in a row without failures > > on a Debian s390x host. The file system is ext4, mounted relatime. It > > would be interesting to know what file system is yielding the failures > > Michael is seeing. Thanks for the detailed analysis. It convinces me that on notmuch's side everything is OK. This very much boils down to fs and stat issues. If I remember correctly, I've seen this isse once on a different release frpm epel-8 but only on epel-8 otherwise. This is on a build infrastructure where you may end up getting different hosts on each run, primed from the same "chroot". I'll try to find out more. Until now, we have been building notmuch release packages on Fedora without running the tests during the release build, because there used to be issues with the test suite (and before that that the separate test corpus download). With everything looking robust for a while, I'm switching tests on for Fedora release builds now, and at the same start packaging those extra packages for RedHat's enterprise platform. I want to avoid spurious release build failures, of course - otoh having those tests run is a good thing. So I'll try out a bit more, and if everything else fails, then running the test suite with --full-scan is still better than not running it at all. (I would have suggested a make variable, but it's really a corner case. given the stat/mtime check). Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
Am Sa., 12. Feb. 2022 um 22:10 Uhr schrieb David Bremner : > Tomi Ollila writes: > > > > > Does such a change hide "buggy" functionality ? > > We mostly don't use add_message, call notmuch new via NOTMUCH_NEW in > T050-new.sh. So I think it would mostly not hide bugs in notmuch > new. OTOH, I'm surprised the same issues with timestamps don't show up > there, if that is really the problem. > > And sure enough, I got a FAIL from NOTMUCH_NEW in T750-gzip just a few minutes ago. Trying with --full-scan there, too now. Alternatively, I'll just turn off tests again for package builds (like we had before)... ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
Am Sa., 12. Feb. 2022 um 21:45 Uhr schrieb Tomi Ollila : > On Sat, Feb 12 2022, David Bremner wrote: > > > Tomi Ollila writes: > > > >> On Sat, Feb 12 2022, Michael J. Gruber wrote: > >> > >> Only thing that came into mind are directory timestamps... if directory > >> (m)time is same as before notmuch will not scan it for files... > >> > >> ... following that if the granularity of directory timestamp were 1 > second, > >> then it could easily happen than first one new message is not seen, and > >> next time there is one extra message to be see... > > > > What do you think about adding --full-scan to the notmuch-new invocation > > in add_message? It doesn't make any tests fail and is about the same > > speed. I need to do a few more trials, but first time through it was > > actually faster (!), maybe because the cache is hot > > Does such a change hide "buggy" functionality ? > > Or do we consider notmuch new buggy if it does not notice all new messages > arrived every time ? > > The timestamping sounds like a perfect explanation of what I've been seeing. Unfortunately, I can't reproduce the issue "reliably" (with a certain probability), and so if everything succeeds with --full-scan 10 times it still does not mean much. As I understand, notmuch new without --full-sync may have issues when the time resolution is too low (or operations too fast) and will pick a message on the next run, so it's not really buggy - it uses a shortcut that may be too quick but does not loose messages in the long run. Michael > > ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
David Bremner venit, vidit, dixit 2022-02-12 01:03:00:
> Michael J Gruber writes:
>
> > Hi there,
> >
> > I'm trying to package notmuch for Redhat's enterprise linux and clones
> > (EPEL, extra packages for enterprise linux).
> >
> > This looks mostly fine, including the tests, except for intermittent
> > failures on epel-8-s390x. They look like the below, or in tests
> > following those, and apparantly all have to do with "db not synced yet"
> > or such, so that a message one subtest creates only shows up in the db
> > for the next subtest.
> >
> > I've also had completely fine test runs on epel-8-s390x, but I'm
> > starting to wonder whether I've just been lucky so far on other
> > platforms ... Could it be possible that generate_message(), even though
> > adding the message, still returns false and therefore add_message() does
> > not call "notmuch new"? One might want to drop the "&&" there in
> > test-lib-common, I dunno. This shouldn't be "intermittent".
>
> It's hard to see what can go wrong (maybe perl doesn't work?), but
> failing to generate a message should be a fatal error. Maybe try
> something like
>
Well, as you can see in both reports, the pattern is as follows:
The first subttest sees 1 less message than expected.
The second sees 1 more message than expected.
So in summary, they are generated but "notmuch new" is not run or does
not pick them up. Though it's a Heisenbug on a specific arch, I'll try
your suggestion and hammer our infra with it ...
> diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
> index ebbf4cdf..076777ca 100644
> --- a/test/test-lib-common.sh
> +++ b/test/test-lib-common.sh
> @@ -225,7 +225,7 @@ EOF
> # All of the arguments and return values supported by generate_message
> # are also supported here, so see that function for details.
> add_message () {
> -generate_message "$@" &&
> +generate_message "$@" || error "failed to generate message"
> notmuch new > /dev/null
> }
>
>
> to see if you get more information
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test suite timing issues?
Michael J Gruber venit, vidit, dixit 2022-02-11 14:55:32: > Hi there, > > I'm trying to package notmuch for Redhat's enterprise linux and clones > (EPEL, extra packages for enterprise linux). > > This looks mostly fine, including the tests, except for intermittent > failures on epel-8-s390x. They look like the below, or in tests > following those, and apparantly all have to do with "db not synced yet" > or such, so that a message one subtest creates only shows up in the db > for the next subtest. > > I've also had completely fine test runs on epel-8-s390x, but I'm > starting to wonder whether I've just been lucky so far on other > platforms ... Could it be possible that generate_message(), even though > adding the message, still returns false and therefore add_message() does > not call "notmuch new"? One might want to drop the "&&" there in > test-lib-common, I dunno. This shouldn't be "intermittent". > > Michael > > T210-raw: Testing notmuch show --format=raw > FAIL Attempt to show multiple raw messages > --- T210-raw.1.expected 2022-02-11 13:22:28.011556841 + > +++ T210-raw.1.output 2022-02-11 13:22:28.011556841 + > @@ -1 +1,7 @@ > -Error: search term did not match precisely one message (matched 2 > messages). > +From: Notmuch Test Suite > +To: Notmuch Test Suite > +Message-Id: > +Subject: Test message #1 > +Date: Fri, 05 Jan 2001 15:43:56 + > + > +This is just a test message (#1) > PASS Show a raw message > FAIL Show another raw message > --- T210-raw.3.expected 2022-02-11 13:22:28.031556841 + > +++ T210-raw.3.output 2022-02-11 13:22:28.031556841 + > @@ -1,7 +1 @@ > -From: Notmuch Test Suite > -To: Notmuch Test Suite > -Message-Id: > -Subject: Test message #2 > -Date: GENERATED_DATE > > -This is just a test message (#2) > Error: search term did not match precisely one message (matched 0 messages). Similar apparent timing issue on epel-8-aarch64: FAIL Adding message with long ID --- T290-long-id.2.expected 2022-02-11 13:55:17.028415326 + +++ T290-long-id.2.output 2022-02-11 13:55:17.028415326 + @@ -1 +1 @@ -Added 1 new message to the database. +No new mail. FAIL Referencing long ID after adding --- T290-long-id.3.expected 2022-02-11 13:55:17.078415753 + +++ T290-long-id.3.output 2022-02-11 13:55:17.078415753 + @@ -1 +1 @@ -Added 1 new message to the database. +Added 2 new messages to the database. Note how that message from the 1st subtest shows up only in the second. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Test suite timing issues?
Hi there, I'm trying to package notmuch for Redhat's enterprise linux and clones (EPEL, extra packages for enterprise linux). This looks mostly fine, including the tests, except for intermittent failures on epel-8-s390x. They look like the below, or in tests following those, and apparantly all have to do with "db not synced yet" or such, so that a message one subtest creates only shows up in the db for the next subtest. I've also had completely fine test runs on epel-8-s390x, but I'm starting to wonder whether I've just been lucky so far on other platforms ... Could it be possible that generate_message(), even though adding the message, still returns false and therefore add_message() does not call "notmuch new"? One might want to drop the "&&" there in test-lib-common, I dunno. This shouldn't be "intermittent". Michael T210-raw: Testing notmuch show --format=raw FAIL Attempt to show multiple raw messages --- T210-raw.1.expected 2022-02-11 13:22:28.011556841 + +++ T210-raw.1.output 2022-02-11 13:22:28.011556841 + @@ -1 +1,7 @@ -Error: search term did not match precisely one message (matched 2 messages). +From: Notmuch Test Suite +To: Notmuch Test Suite +Message-Id: +Subject: Test message #1 +Date: Fri, 05 Jan 2001 15:43:56 + + +This is just a test message (#1) PASS Show a raw message FAIL Show another raw message --- T210-raw.3.expected 2022-02-11 13:22:28.031556841 + +++ T210-raw.3.output 2022-02-11 13:22:28.031556841 + @@ -1,7 +1 @@ -From: Notmuch Test Suite -To: Notmuch Test Suite -Message-Id: -Subject: Test message #2 -Date: GENERATED_DATE -This is just a test message (#2) Error: search term did not match precisely one message (matched 0 messages). ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Python binding SIGABRT/SIGSEGV
Austin Lund venit, vidit, dixit 2022-02-10 23:21:58:
> On Thu, Feb 10, 2022 at 01:12:47PM +0100, Michael J Gruber wrote:
> > Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
> > > I'm clearly doing this python code wrong by not using the iterator
> > > correctly:
> > >
> > > > import notmuch2
> > > >
> > > > d = notmuch2.Database()
> > > > m = list(d.messages("since:today"))
> > > > p = m[0].path
> > > > print(p)
> > >
> > > But I seem to be getting a SIGABRT instead of a python stack trace. Is
> > > this the expected behaviour?
> >
> > You didn't expect it :)
> >
> > And this can be confusing. d.messages() returns an iterator through
> > Message objects whose lifetime depends on the iterator. In contrast,
> > thread.get_messages() returns on iterator through OwnedMessage objects
> > whose lifetime depends on the thread.
>
> I guess I didn't say it explicitly, but I would 'expect' the python
> interpreter to raise an exception rather than having an unhandled
> exception terminate the program. Perhaps raising a MemoryError or
> ReferenceError or some other exception would be better than an unhandled
> SIGABRT.
>
In fact you did. Sorry for overlooking this. (I still find db.messages()
vs thread.get_messages() confusing.)
The way memory handling works, one could even expect a more specific
notmuch2.ObjectDestroyedError.
As for the SIGABRT: I guess this is what "python -X faulthandler" is
for, especially in the context of C extension modules:
LANG=C python -X faulthandler t.py
Fatal Python error: Aborted
Current thread 0x7f5bf667e740 (most recent call first):
File "/usr/lib64/python3.10/site-packages/notmuch2/_message.py", line
131 in path
File "/tmp/t.py", line 5 in
Extension modules: _cffi_backend (total: 1)
Abgebrochen (Speicherabzug geschrieben)
(Yes, it disrespects LANG)
Line 131 is
ret = capi.lib.notmuch_message_get_filename(self._msg_p)
in the path method, and since _msg_p is a base.MemoryPointer() I would
have hoped (now) to get a notmuch2.ObjectDestroyedError. Maybe _msg_p is
destroyed but the Message object is not? Fishing in the dark here ...
Michael
P.S.: Sorry for the repeated message. I had problems with list bounces
in the past, then remembered the original alias wrong, then replied to
the reply to the wrong (only off-list-delivered) post. g...@grubix.eu is
the one subscribed to the list. I'll remember now.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Python binding SIGABRT/SIGSEGV
Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
> I'm clearly doing this python code wrong by not using the iterator correctly:
>
> > import notmuch2
> >
> > d = notmuch2.Database()
> > m = list(d.messages("since:today"))
> > p = m[0].path
> > print(p)
>
> But I seem to be getting a SIGABRT instead of a python stack trace. Is
> this the expected behaviour?
You didn't expect it :)
And this can be confusing. d.messages() returns an iterator through
Message objects whose lifetime depends on the iterator. In contrast,
thread.get_messages() returns on iterator through OwnedMessage objects
whose lifetime depends on the thread.
As soon as the iterator is depleted, the returned objects are (possibly)
gone. (Well, because it's return by reference in Python, and ...)
If you're interested in m[0] only you can "cheat" by not depleting the
iterator:
mm = next(d.messages("since:today"))
p = mm.path
This never frees the object (I think).
My attempts with notmuch2._message.OwnedMessage (and db as parent)
failed. There must be a better way, possibly using a context manager or
search.
I guess usually people just use the iterator in a for loop and do
something with the message inside the loop (while the iterator is not
depleted), such as converting it into a proper email.Message object
(i.e. instantiating a new object from it).
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/4] test: due not pass T380.1 for the wrong reasons
David Bremner venit, vidit, dixit 2022-02-09 23:12:24: > Michael J Gruber writes: > > > If gdb is missing then some files are never written to so that the > > comparisons of non-existing files succeeds for the wrong reason, > > claiming that `notmch new` is idempotent when it was in fact never run. > > > > Catch this and (for lack of a better spot) set up the files with a > > reason for the FAIL. > > > > I'm a bit confused by this. For me if gdb is missing I get > > missing prerequisites: gdb(1) > SKIP all tests in T380-atomicity > > ... > > All 1842 tests behaved as expected (8 expected failures). > All tests in 1 file skipped. > > Do I misunderstand something, or is prerequisite detection not working > for you? It does now, my understanding has been mislead by a few things. The issues were: On one of Fedora's arches which is emulated in the COPR test environment, gdb was present (just like in all of them) but apparantly failed to set any break points. T380 spew out something like cat: outcount: No such file or directory ./T380-atomicity.sh: line 79: ((: i < : syntax error: operand expected (error token is "< ") followed by wrong PASS/FAIL etc. When analysing this, I was confused by the way test_require_external_prereq works and the "if" in T380 (as opposed to how test_require_external_prereq is used in other tests). Over at git.git, we have test setup code in functions which don't get executed if prerequisites fail. I guess the "if" emulates that, but then the actual tests in T380 are outside the if block and use files and variables which are created in the if block. So, this is something to fix anyways. In order to try things out locally, I changed if test_require_external_prereq gdb; then into if test_require_external_prereq gggdb; then I know now that `test_require_external_prereq binary` does not test for the presence of binary, nor does it abort the test script. It merely checks among the set of "predefined external prerequisites", and this would be much clearer if "binary" wasn't an argument to that function but a suffix: `test_require_external_prereq_gdb` immediately says "I'm something predefined". Until now, I wondered how `test_require_external_prereq` could skip the PASS/FAIL outout if it could not skip the previous part of the test script. This is not clear from its implementation. I know now that indeed text_expect-success etc. call test_check_missing_external_prereqs_ for the current subtest and skip the output. Add to this the fact that the tests needing sfsexp or asan (and probably others) do things yet differently and call "test_done" immediately, so that no SKIP appears. And those were the only ones skipped at all here ... In effect, I knoew SKIP only from manually skipping tests (NOTMUCH_SKIP_TESTS). I'm not complaining, on the other hand I don't fail too bad about myself for being confused by this ;) So, in the long run I think test_setup() is worth thinking about. In the short run, initialising variables and files which are used is still a good thing, but I would have to rewrite some commit messages. I'll wait until it's clear how to handle style, though: switch to printf from echo whenever I touch those lines (leading to mixed use) or keeing style and leaving the style change for another series. Cheers, Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 4/4] test: set up the outcount file for T380.1
Tomi Ollila venit, vidit, dixit 2022-02-09 21:50:35:
> On Wed, Feb 09 2022, Michael J. Gruber wrote:
>
> > If gdb is present but for some reason `atomicity.py` fails to write to
> > the output file then the test fails with some ugly bash errors in the
> > wrong places (because the outcount variable is empty).
> >
> > Therefore, set up the outcount file with `0` to get the test script to
> > rund and the test to fail fpr a clearer reason.
> >
> > Background: We noticed this with arch armhfp emulated on x86_64 in
> > Fedora's COPR test build environment.
> >
> > Signed-off-by: Michael J Gruber
> > ---
> > test/T380-atomicity.sh | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh
> > index 49df5c38..caac28a3 100755
> > --- a/test/T380-atomicity.sh
> > +++ b/test/T380-atomicity.sh
> > @@ -64,6 +64,7 @@ if test_require_external_prereq gdb; then
> > # -tty /dev/null works around a conflict between the 'timeout' wrapper
> > # and gdb's attempt to control the TTY.
> > export MAIL_DIR
> > +echo -n 0 > outcount
>
> printf 0 > outcount (is my suggestion)
Would by mine, too :)
Again, this is not the style of T380 which uses echo exclusively so far.
I do think that printf is more stable across different systems and
shells, so a mass replace would be in order (but requires careful nl
checking).
Overally in the test suite, [f]printf wins by 8:3 roughly.
>
> > ${TEST_GDB} -tty /dev/null -batch -x $NOTMUCH_SRCDIR/test/atomicity.py
> > notmuch 1>gdb.out 2>&1
> >
> > # Get the final, golden output
> > --
> > 2.35.1.306.ga00bde9711
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/4] test: due not pass T380.1 for the wrong reasons
Tomi Ollila venit, vidit, dixit 2022-02-09 21:49:18: > On Wed, Feb 09 2022, Michael J. Gruber wrote: > > > If gdb is missing then some files are never written to so that the > > comparisons of non-existing files succeeds for the wrong reason, > > claiming that `notmch new` is idempotent when it was in fact never run. > > > > Catch this and (for lack of a better spot) set up the files with a > > reason for the FAIL. > > > > Signed-off-by: Michael J Gruber > > --- > > test/T380-atomicity.sh | 4 > > 1 file changed, 4 insertions(+) > > > > diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh > > index a6f1e037..7f618062 100755 > > --- a/test/T380-atomicity.sh > > +++ b/test/T380-atomicity.sh > > @@ -90,6 +90,10 @@ if test_require_external_prereq gdb; then > > i=$(expr $end - 1) > > fi > > done > > +else > > +echo -n "Test fails due to missing gdb." > searchall > > +echo -n > expectall > > I am not much of a fan of 'echo -n' (I remember seeing -n (and newline > echoed...), therefore first to use printf and second : > expectall > (unless printf '' > expectall) I'm in favour of using printf - usually I don't impose my favours but follow surrounding style, which is exactly what I did here. In the if-block before the else, the files are writen to using "echo -n". I would be weird to do it differently in both blocks. > > > +outcount=0 > > fi > > > > test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts' > > -- > > 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/4] test: correct comparison order in T380
Tomi Ollila venit, vidit, dixit 2022-02-09 21:47:46: > On Wed, Feb 09 2022, Michael J. Gruber wrote: > > > Specifying test comparisons as "expected actual" gives a better readable > > diff since the "-" indicates missing, "+" additional items compared to > > the expectations. > > > > Signed-off-by: Michael J Gruber > > --- > > test/T380-atomicity.sh | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh > > index afe49d93..a6f1e037 100755 > > --- a/test/T380-atomicity.sh > > +++ b/test/T380-atomicity.sh > > @@ -93,7 +93,7 @@ if test_require_external_prereq gdb; then > > fi > > > > test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts' > > -test_expect_equal_file searchall expectall > > +test_expect_equal_file expectall searchall > > Good in princible, but if things are like decade ago, we still have > hundreds of these in wrong order -- if so, all should be changed for > consistency. If I am wrong and all / many of those are already changed, > then this is OK. I haven't checked, but the order add to the confusion created by "detected >10 abort points" which is why I changed it here, and in order to (mis-)use these files in the following commits. > > > > test_begin_subtest "detected $outcount>10 abort points" > > test_expect_success "test $outcount -gt 10" > > -- > > 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 1/4] test: correct comparison order in T380
Specifying test comparisons as "expected actual" gives a better readable diff since the "-" indicates missing, "+" additional items compared to the expectations. Signed-off-by: Michael J Gruber --- test/T380-atomicity.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh index afe49d93..a6f1e037 100755 --- a/test/T380-atomicity.sh +++ b/test/T380-atomicity.sh @@ -93,7 +93,7 @@ if test_require_external_prereq gdb; then fi test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts' -test_expect_equal_file searchall expectall +test_expect_equal_file expectall searchall test_begin_subtest "detected $outcount>10 abort points" test_expect_success "test $outcount -gt 10" -- 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 3/4] test: reword T380.2 to be clearer
T380.2 gives a test description which depends on the actual test output, rather than the expected outcome or actual test which is performed. So, when the test fails due missing abort points, the test describes itself as `detected 0>10 abort points` so that it's not clear which part or which number is the expectation. (Also, `0>10` is no number ...) When the test is not run for some reason and fails because of that, the test describes itself as `detected >10 abort points`, which arguably is better or worse. Reword it to say `detected more than 10 abort points`, which is the actual expectation and what we test for. The failing test line still gives the actual number. Signed-off-by: Michael J Gruber --- test/T380-atomicity.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh index 7f618062..49df5c38 100755 --- a/test/T380-atomicity.sh +++ b/test/T380-atomicity.sh @@ -99,7 +99,7 @@ fi test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts' test_expect_equal_file expectall searchall -test_begin_subtest "detected $outcount>10 abort points" +test_begin_subtest "detected more than 10 abort points" test_expect_success "test $outcount -gt 10" test_done -- 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 2/4] test: due not pass T380.1 for the wrong reasons
If gdb is missing then some files are never written to so that the comparisons of non-existing files succeeds for the wrong reason, claiming that `notmch new` is idempotent when it was in fact never run. Catch this and (for lack of a better spot) set up the files with a reason for the FAIL. Signed-off-by: Michael J Gruber --- test/T380-atomicity.sh | 4 1 file changed, 4 insertions(+) diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh index a6f1e037..7f618062 100755 --- a/test/T380-atomicity.sh +++ b/test/T380-atomicity.sh @@ -90,6 +90,10 @@ if test_require_external_prereq gdb; then i=$(expr $end - 1) fi done +else +echo -n "Test fails due to missing gdb." > searchall +echo -n > expectall +outcount=0 fi test_begin_subtest '"notmuch new" is idempotent under arbitrary aborts' -- 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 4/4] test: set up the outcount file for T380.1
If gdb is present but for some reason `atomicity.py` fails to write to
the output file then the test fails with some ugly bash errors in the
wrong places (because the outcount variable is empty).
Therefore, set up the outcount file with `0` to get the test script to
rund and the test to fail fpr a clearer reason.
Background: We noticed this with arch armhfp emulated on x86_64 in
Fedora's COPR test build environment.
Signed-off-by: Michael J Gruber
---
test/T380-atomicity.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/test/T380-atomicity.sh b/test/T380-atomicity.sh
index 49df5c38..caac28a3 100755
--- a/test/T380-atomicity.sh
+++ b/test/T380-atomicity.sh
@@ -64,6 +64,7 @@ if test_require_external_prereq gdb; then
# -tty /dev/null works around a conflict between the 'timeout' wrapper
# and gdb's attempt to control the TTY.
export MAIL_DIR
+echo -n 0 > outcount
${TEST_GDB} -tty /dev/null -batch -x $NOTMUCH_SRCDIR/test/atomicity.py
notmuch 1>gdb.out 2>&1
# Get the final, golden output
--
2.35.1.306.ga00bde9711
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 0/4] test: T380 rework
I've been testing with tests now (duh), i.e. enabling the tests for regular Fedora distribution builds. This seems to be mostly OK, no flaky fails so far. asan works locally (on x86_64) but we don't release build with asan; and we don't have sfsexp in Fedora yet (working on it). All other tests run. In this context I noticed a few pecularities with T380. They all have to do with the cases when either gdb is not available or fails to set breakpoints, and this series tries to improve test output (and make tests fail for the right reason). The series is kind of micro-granular with commit messages longer than the patch diff (in good old git.git fashion), to make it readable and also partially-pickable if you prefer. Michael J Gruber (4): test: correct comparison order in T380 test: due not pass T380.1 for the wrong reasons test: reword T380.2 to be clearer test: set up the outcount file for T380.1 test/T380-atomicity.sh | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) -- 2.35.1.306.ga00bde9711 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Statistics about senders with notmuch possible?
signmeup1...@cock.li venit, vidit, dixit 2021-12-27 23:57:18: > I would like to create some kind of statistics to see from which sender > I've got the most emails, having something like a TOP 10 list. > > In the end of every year I delete all my emails, which are not marked as > archive and this year I got over 4.000 and most of them are just > notifications (like ebay, if someone buys something). But I really would > like to figure out where else the most mails are coming from to avoid > them in the next year. > > Is something like this possible with notmuch? `notmuch address` should provide everything you want (bar the sorting): `notmuch address --output=count --deduplicate=address date:2021 | sort -nr | head -n 10` Leave out the dedup option if you want to distinguish senders by the full mailbox address. And, of course, amend the query any way you want. That's the beauty of notmuch's modular approach. Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [WIP PATCH 0/1] autodoc failed to import notmuch2
David Bremner venit, vidit, dixit 2021-10-21 14:20:11: > Michael J Gruber writes: > > > Hi there > > > > During Fedora package builds I noticed that autodoc fails to import the > > notmuch2 module: > > > > ``` > > WARNING: autodoc: failed to import module 'notmuch2'; the following > > exception was raised: > > No module named 'notmuch2' > > ``` > > > > You never notice this on your own box where notmuch2 is installed, > > because then the build finds the previously installed module. But this > > is wrong, of course, because autodoc is supposed to document tthe module > > being build (in-source module). > > > > I also see this error when doing builds in a clean environment for > Debian. I guess I'm currently not using the docs from those builds, so I > don't know how bad the damage is. In that case: Which docs do you use, and which docs do you want us (packagers) to use? The usual fedora policy is to (re)build docs if the doc source is shipped. Disclaimer: I have no clue about autodoc ;) Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[WIP PATCH 0/1] autodoc failed to import notmuch2
Hi there During Fedora package builds I noticed that autodoc fails to import the notmuch2 module: ``` WARNING: autodoc: failed to import module 'notmuch2'; the following exception was raised: No module named 'notmuch2' ``` You never notice this on your own box where notmuch2 is installed, because then the build finds the previously installed module. But this is wrong, of course, because autodoc is supposed to document tthe module being build (in-source module). There are actually several problems leading to this: 1. search path The module residing in `bindings/python-cffi/notmuch2/__init__.py` needs a search path `bindings/python-cffi/`, not `bindings/python-cffi/notmuch2`. If we fix just that we get: ``` WARNING: autodoc: failed to import module 'notmuch2'; the following exception was raised: cannot import name '_capi' from partially initialized module 'notmuch2' (most likely due to a circular import) (/builddir/build/BUILD/notmuch-0.34/doc/../bindings/python-cffi/notmuch2/__init__.py) ``` 2. search path, again The module is built in `bindings/python-cffi/build/stage`, that is where `_capi` is to be found, and PATCH 1/1 fixes the path in `doc/conf.py`. 3. build order The patch from 2. actually helps only if we build in the following order (make targets): notmuch notmuch-shared ruby-bindings python-cffi-bindings all In particular, the bindings need to be built before the docs I can do this using two make calls, but a proper solution would specify the dependencies in the makefile. 4. library search path When you do 1. through 3. then autodoc almost manages to import notmuch2: it finds the proper module, _capi is found, but then libnotmuch.so is not found because LD_LIBRARY_PATH would need to be amended to find the freshly build library. Now, maybe I should simply ignore the autodoc warning. Or I should change my packaging to build twice (make && make install && make clean && make && make install). At least I wanted to raise attention to this. This problem (if it is one) is not specific to 0.34, it was the same at least with 0.33 but - being just a warning - was unnoticed. Cheers Michael Michael J Gruber (1): autodoc: fix search path for notmuch2 module doc/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.33.0.1146.g7ac282475c ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[WIP PATCH 1/1] autodoc: fix search path for notmuch2 module
a05da455 ("doc: add new python bindings to main documentation tree.",
2020-07-11)
added the path "bindings/python-cffi/notmuch2" to the search path for
autodoc. But the notmuch2 module cannot be imported from there - the dir
"is" the module, and it resides in "bindings/python-cffi" and in
"bindings/python-cffi/build/stage", with only the latter location having
the compiled capi part.
This goes unnoticed as long as you have notmuch2 in your standard search
path, but it means autodocs finds the wrong module, not the one one we
are building the doc against.
Fix the path so that the build tree module comes first.
Signed-off-by: Michael J Gruber
---
doc/conf.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/conf.py b/doc/conf.py
index 1fbd102b..c7fd8f5a 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -25,7 +25,7 @@ for pathdir in ['.', '..']:
version=infile.read().replace('\n','')
# for autodoc
-sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi',
'notmuch2'))
+sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi',
'build', 'stage'))
# read generated config
for pathdir in ['.', '..']:
--
2.33.0.1146.g7ac282475c
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [RFC 0/2] add --new-tags= to notmuch-new(1)
andr...@rammhold.de venit, vidit, dixit 2021-09-16 14:24:14:
> On 14:03 16.09.21, Michael J Gruber wrote:
> > I very much sympathize with your setup. But I think the real solution
> > would be one of these options:
> >
> > - use a lock file to prevent your scripts from running concurrently OR
>
> That would work. I'd love if that could be moved into notmuch such
> that running `notmuch new` will keep a (global) lock until all of the
> messages have been processed and all of the hooks have been executed.
>
> This is probably tricky to get right consistently across the notmuch
> code base. At least that was one of the comments I recall from a
> conversation on IRC when that topic was brought up.
I don't think it's notmuch's job to do that. But if you call notmuch for
a specific folder you can instead call a wrapper script which keeps a
folder-specific lock until that notmuch returns, and can you pass the
folder to the hooks via environment.
> > - match "tag:new and folder:UNIQUEFOLDER" (or "path:FOO/**")
> > This should be a perfect substitute for your new-UNIQUE tag.
>
> How would the files end up in the UNIQUEFOLDER? What I probably forgot
> to mention is that I do not sync only on the Mailbox boundary. I have an
> IAMP IDLE instance for "Inbox" on all of the accounts and then more for
> other folders in those accounts. I could probably script that (wouldn't
> take more than an evening I am sure) but requires a lot more additional
> code.
I assumed that each of your different notmuch calls (from different IDLE
instances) act on different folders or sets of folders, so that you can
distinguish the messages either way.
Depending on your setup, there may still be issues when a second sync
interferes so that the "first" notmuch new tags messages from the
"second" sync. That's exactly what a lock can prevent if you do the sync
in a pre-new hook.
>
> > As for the implementation you suggest: Basically, you implement
> > overriding the "new.tags" config, and I'm wondering whether it would be
> > worthwhile to implement "notmuch --config-value" instead:
> >
> > --config-value=SECTION.ITEM=VALUE
> > Override the config setting for SECTION.ITEM with the VALUE for
> > this invocation. This takes precedence over any setting in the
> > config file.
>
> That is an interesting idea. How would you implement that? Right now it
> looks like all changes in configuration are written to disk when you
> change them. That would require keeping a copy of all the settings in
> RAM (which is probably already the case) and only mutating those without
> writing to disk.
... and notmuch config get called from a hook should return that value,
ideally (I don't know how).
We already have at least these:
notmuch {insert,new,reindex,reply,show} --decrypt overrides index.decrypt
NOTMUCH_DATABASE overrides database.path
and the env vars for config and profile interacting with --config. So I
was hoping for a systematic solution. But I just love "git -c" too much :)
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [RFC 0/2] add --new-tags= to notmuch-new(1)
andr...@rammhold.de venit, vidit, dixit 2021-09-16 12:25:15: > I've recently became a little annoyed with a race condition in my current > notmuch setup that originates from only having a single set of new tags for > all notmuch-new(1) invocations. In the past I've mentioned this a couple of > times in the IRC channel and now I got around to implement a basic version of > this. > > I rougly process new messages like this: > > 1. new messages get the "new" tag through notmuch-new(1), > 2. the post-new hook calls a series of scripts ... > There are right now 8 mailboxes that I am retrieving mails from. I have a > scheduled job that updates all my local Maildir's every couple of minutes. > That one doesn't cause any issues on its own. > > But I also use IMAP IDLE to selectively update Maildir's as soon as a new mail > arrives. > > If I receive mails on multiple Maildir's within a short period of time the > above process is running into race-conditions since there is no way to > distinguish mails that have just been marked new. All of them carry the same > tags. > > In the worst case one of the last steps (2c/2d) would pick up the new mail > before any of the actual classification has been executed. This "leaks" mails > into my inbox which consequently can be overwhelming to look at. > > With this series I am able to give each notmuch-new(1) invocation a unique tag > (think: new-$(uuidgen)). This doesn't (on its own) solve the entire story but > is a first step in the "right direction" IMHO. I still have to wrap the entire I very much sympathize with your setup. But I think the real solution would be one of these options: - use a lock file to prevent your scripts from running concurrently OR - match "tag:new and folder:UNIQUEFOLDER" (or "path:FOO/**") This should be a perfect substitute for your new-UNIQUE tag. As for the implementation you suggest: Basically, you implement overriding the "new.tags" config, and I'm wondering whether it would be worthwhile to implement "notmuch --config-value" instead: --config-value=SECTION.ITEM=VALUE Override the config setting for SECTION.ITEM with the VALUE for this invocation. This takes precedence over any setting in the config file. I know this raises the question which config any hooks will see, but that is the same for your implementation. You can override the complete config by specifying a different file already, of course, so you could script that. Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: configure checks for python3-notmuch2 module
David Bremner venit, vidit, dixit 2021-07-11 23:29:57: > > It turns out we need pyconfig.h to build the (new) python bindings. I > don't think this is adequately checked for by the configure script, > since I had a Fedora user report a build failure (during make install, > which I found odd) missing pyconfig.h > > On Debian this is on libpython3.x-dev, which is not a dependency of the > python3-cffi package. You need python3-cffi to run python code which uses cffi bindings - think of it as the "lib" part. You need the devel package to generate a python module using cffi (i.e. binding for a specific c library). So, it makes sense for the lib package not to depend on the devel package (which is python3-devel on Fedora). I can't help with adapting configure to the header requirement, though. Michael > log follows. > -- > > cd bindings/python-cffi && \ > python3 setup.py build --build-lib build/stage && \ > mkdir -p build/stage/tests && cp tests/*.py build/stage/tests > running build > running build_py > running build_ext > generating cffi module 'build/temp.linux-x86_64-3.9/notmuch2._capi.c' > creating build/temp.linux-x86_64-3.9 > building 'notmuch2._capi' extension > creating build/temp.linux-x86_64-3.9/build > creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9 > gcc -pthread -Wno-unused-result -Wsign-compare > -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g > -grecord-gcc-switches -pipe -Wall -Werror=format-security > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong > -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection > -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g > -grecord-gcc-switches -pipe -Wall -Werror=format-security > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong > -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection > -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g > -grecord-gcc-switches -pipe -Wall -Werror=format-security > -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong > -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection > -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../lib > -I/usr/include/python3.9 -c build/temp.linux-x86_64-3.9/notmuch2._capi.c -o b ui > ld/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/notmuch2._capi.o > build/temp.linux-x86_64-3.9/notmuch2._capi.c:50:14: fatal error: pyconfig.h: > No such file or directory >50 | #include > | ^~~~ > compilation terminated. > error: command '/usr/bin/gcc' failed with exit code 1 > ___ > notmuch mailing list -- notmuch@notmuchmail.org > To unsubscribe send an email to notmuch-le...@notmuchmail.org ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH] test: deduplicate T590
Test numbers are a concise way to communicate about tests and to remeber
them. Currently, there is one pait of duplicates:
T590-libconfig.sh
T590-thread-breakage.sh
Renumber the latter one to 592 since this keeps the alphabetic order and
leaves room in between.
Signed-off-by: Michael J Gruber
---
test/{T590-thread-breakage.sh => T592-thread-breakage.sh} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename test/{T590-thread-breakage.sh => T592-thread-breakage.sh} (100%)
diff --git a/test/T590-thread-breakage.sh b/test/T592-thread-breakage.sh
similarity index 100%
rename from test/T590-thread-breakage.sh
rename to test/T592-thread-breakage.sh
--
2.32.0.96.gc6d41f9332
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] lib: consider all instances of Delivered-To header
Hannu Hartikainen venit, vidit, dixit 2021-06-23 12:29:06:
> When using notmuch-reply and guessing the From: address from
> Delivered-To headers, I had the wrong address chosen today. This was
> because the messages from the notmuch list contain these headers in this
> order:
>
> Delivered-To: hannu.hartikai...@gmail.com
> ...
> Delivered-To: ha...@hrtk.in
>
> In my .notmuch-config I have the following configuration:
>
> primary_email=ha...@hrtk.in
> other_email=hannu.hartikai...@gmail.com;...
>
> Before this change, notmuch-reply would guess From: @gmail.com because
> that is the first Delivered-To header present. After the change, the
> primary address is chosen as I would expect.
> ---
> lib/message-file.c | 12
> 1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/lib/message-file.c b/lib/message-file.c
> index 647ccf3a..7e8ea09c 100644
> --- a/lib/message-file.c
> +++ b/lib/message-file.c
> @@ -291,11 +291,15 @@ _notmuch_message_file_get_header
> (notmuch_message_file_t *message,
> if (value)
> return value;
>
> -if (strcasecmp (header, "received") == 0) {
> +if (strcasecmp (header, "received") == 0 ||
> +strcasecmp (header, "delivered-to") == 0) {
> /*
> -* The Received: header is special. We concatenate all
> -* instances of the header as we use this when analyzing the
> -* path the mail has taken from sender to recipient.
> +* The Received: header is special. We concatenate all instances of
> the
> +* header as we use this when analyzing the path the mail has taken
> +* from sender to recipient.
> +*
> +* Similarly, multiple instances of Delivered-To may be present. We
> +* concatenate them so the one with highest priority may be picked.
> */
> decoded = _notmuch_message_file_get_combined_header (message, header);
> } else {
> --
> 2.32.0
Without looking at the surrounding code I'm wondering:
Is an address from a received header (still) preferred over one from a
delivered-to, or does the order of headers in the mail envelope play a
role?
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test failures with notmuch 0.32 and 0.32.1 on openSUSE
Dan Čermák venit, vidit, dixit 2021-05-20 16:30:40: > Michael J Gruber writes: > > > > > So, I ran the container using buildah on Fedora 33 (without the make > > corpus_download). Again, I'm not a container guy, so I don't know what > > the influence of the host is. But: > > > > What ist the shell in that container? > > It should be bash in the final container, but it could be that buildah > is not using bash to run the actual build. > > > > I don't see T356 failing, but: > > I'll be dammed. That test was failing yesterday in the container build > and today it is succeeding. > > Unfortunately, it is still failing in the Open Build Service :-( With each fresh look it's getting more mysterious ... In https://build.opensuse.org/public/build/home:dancermak:branches:devel:libraries:c_c++/openSUSE_Factory/x86_64/notmuch/_log I see that tests are run with gnu parallel and not sequentially. The only FAILs are the sig_uid ones in T356. So, apparantly, the local buildah run does something different from OBS. Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test failures with notmuch 0.32 and 0.32.1 on openSUSE
David Bremner venit, vidit, dixit 2021-05-20 11:59:50: > Dan Čermák writes: > > > > > Right, I mistakenly thought this would grab the test database. Is that > > one no longer necessary? > > > > Ah, that's a different make target, which is indeed no longer necessary > (we don't do the relevant tests any more). > > >> 1) can you duplicate the problem without running the tests in parallel? > > > > They don't run in parallel, as gnu parallel is not installed inside the > > container. So: yes, I can. > > OK, it was a long shot, but gpgsm has some known issues with parallel tests. > > >> 2) What are the versions of the dependencies you are building with? > > > > emacs-el-27.2-1.2.noarch > > [snip...] > > Thanks. What about python and gnupg versions? > So, I ran the container using buildah on Fedora 33 (without the make corpus_download). Again, I'm not a container guy, so I don't know what the influence of the host is. But: What ist the shell in that container? I don't see T356 failing, but: T050-new.36 (Xapian exception: read only files) can read all files, so either chmod u-w fails or xapian can write the db anyway. T050-new.37 and the T060-count.14 are gdb related and too much for me to wrap my head around :| T150 "Xapian exception: read only files" is again chmod u-w Do they fail to fail (fail to throw) because the tests run as root (or wheel) inside the container? T380 clearly fails because the shell does not know arithmetic expansion. The rest passes as expected, except for ruby tests (skipped), known broken tests and the two USERNAME@FQDN issues which were fixed recently. 1486/1497 tests passed. 4 broken tests failed as expected. 7 tests failed. All tests in 2 files skipped. Maybe someone switched their default shell ;) Michael P.S.: I assume this during build is buildah-related: dbus-daemon[427]: [session uid=0 pid=427] org.freedesktop.DBus.Error.AccessDenied: Failed to set fd limit to 65536: Operation not permitted ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Test failures with notmuch 0.32 and 0.32.1 on openSUSE
Dan Čermák venit, vidit, dixit 2021-05-19 10:16:39:
> Hi list,
>
> notmuch 0.32 recently stopped building on openSUSE Leap and Tumbleweed
> due to failures in the test suite. It now consistently fails in
> T356-protected-headers:
>
> FAIL verify signed PKCS#7 subject (multipart-signed) signer User ID
> sig_uid: object not found:
> data[0][0][0]["crypto"]["signed"]["status"][0]["userid"]
> PASS verify signed PKCS#7 subject (onepart-signed)
> FAIL verify signed PKCS#7 subject (onepart-signed) signer User ID
> sig_uid: object not found:
> data[0][0][0]["crypto"]["signed"]["status"][0]["userid"]
> PASS confirm signed and encrypted PKCS#7 subject (sign+enc)
> FAIL confirm signed and encrypted PKCS#7 subject (sign+enc) signer User ID
> sig_uid: object not found:
> data[0][0][0]["crypto"]["signed"]["status"][0]["userid"]
> PASS confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp)
> FAIL confirm signed and encrypted PKCS#7 subject (sign+enc+legacy-disp)
> signer User ID
> sig_uid: object not found:
> data[0][0][0]["crypto"]["signed"]["status"][0]["userid"]
>
>
> Unfortunately, I don't really know what the issue at hand is and how to
> debug it. If anyone could help me out here, I'd be very grateful. I have
> attached a Dockerfile that you can use to reproduce this issue (simply
> build it using your favorite container build tool, e.g.
> `buildah bud --layers --build-arg NOTMUCH_VERSION=0.32`). Note that in
> the container build additional tests fail in contrast to building the
> rpm.
I'm no docker guy, sorry. But the failures indicate that the output of
signature verification changed (unless local variables stopped being
local;) ). Can you check whether
notmuch show --decrypt=true --format=json
"id:smime-${variant}@protected-headers.example"
for one of the variants (such as sign+enc) gives proper output but
misses sig_uid? You would have to that with
GNUPGHOME="${TEST_TMPDIR}/gnupg" etc.
Maybe gpgsm's output changed?
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] test: add known broken test for duplicate thread-id terms
David Bremner venit, vidit, dixit 2021-05-15 15:05:07: > According to my bijection, this bug has been present since commit > 411675a6ce in 2017. It is apparently harmless for regular use, but > does make notmuch crash when compiled with -DDEBUG_DATABASE_SANITY. If a message "belongs" to 2 threads, doesn't it mean that it "belongs" to at least 1 that it doesn't really belong to? The recent problem with "db update in pre-new hook" surfaced as extra thread assignments for the wrong messages (making them show up the wrong thread, too). I meanwhile noticed that they are more common, but am wondering whether they are really harmless? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] lib/notmuch_database_reopen: reload some database metada
David Bremner venit, vidit, dixit 2021-05-12 14:07:33: > David Bremner writes: > > > David Bremner writes: > > > >> In some uses of reopen, new documents and threads maybe have been > >> added, and e.g. compaction may have changed the uuid. > > > > It's quite possible there is more cached data that should be discarded, > > but I'll have to think about that a bit more. > > > > The following are at least potential problems, although I think none of > them are as urgent as the doc_id / thread_id issue. > > exception_reportedshould probably be reset to false > > atomic_nesting, atomic_dirty. Re-opening while in an open atomic > section is currently undefined > behaviour, should be checked. > > features Theoretically if the user runs notmuch-new in a pre-new hook, > the database could be upgraded. For now, I suggest not doing > that. This should be fixabable, but I prefer to take more time. Out of curiosity: What does reopening gain compared to a full close/open cycle? Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Is there a reason why the trashed flag is not synced?
Reto venit, vidit, dixit 2021-05-12 07:10:13: > On Tue, May 11, 2021 at 08:48:45AM +0200, Michael J Gruber wrote: > > If you sync gmail labels to exact copies in different folders then you're > > not > > holding it right, sorry ;) > > > Gmail is not an IMAP service; it has an IMAP API which exposes labels as > > folders, with all the caveats which this implies. That's why there are > > better ways to sync Gmail with a notmuch mail store (gmailieer). > > Better is always a relative term. I prefer a standardized protocol over some > homebrew stuff. ??? The question is whether you talk Gmail API to Gmail or IMAP to Gmail. What is homebrew here? > > I don't think notmuch should break its design principle (1 message id, 1 > > message) just to work around a problem caused by a wrong sync procedure: > > that "delete instruction" is a result (merely: artifact) of speaking IMAP > > to Gmail. > > It's not only gmail, really any IMAP server. > If you have a message in multiple folders at the same time (which is entirely > reasonable, think "todo" + where it actually should go after processing). > They all have the same message ID. > > Only because you delete the copy from todo doesn't mean that you want it gone > from other places as well. > > Sure, you could move instead of copying... > But that means that you may now need to hunt through multiple folders instead > of > being able to grab the message from the actual mbox it should be in. > > IMAP doesn't impose a workflow, just because yours happens to be a certain way > doesn't make it the gold standard ;) Exactly, those arguments are turning into the ridiculous. The point is that folders are neither labels nor tags. You use an IMAP-sync tool to bolt on the tag concept onto a folder structure. This creates problems which only the sync tool can solve, not the indexer (notmuch), because it is the sync tool which translates the IMAP folder structure (and IMAP flags) into a maildir store (and maildir flags). Indeed, IMAP even has extended flags but support on the IMAP client side is typically suboptimal, so people don't use them at all or not to a great extent; they would be a good match for notmuch tags. Now, the only point at which notmuch is involved is indexing the maildir. This includes synchronizing the maildir flags to notmuch tags - and currently notmuch fails to synchronize one flag from the maildir standard. Felipe and I suggest to reinstate that. Note that since notmuch identifies messages by mid and one mid can correspond to multiple files (e.g. due to folder copies, list copies, sent/cc) it has to decide which file to index. Currently, notmuch assumes the content to be identical (it indexes the content of one file) but takes a union of reference headers and flags from all files. "Union of [boolean] flags" is the real issue here: AND or OR? Currently it's OR for all flags, and this happens to work for most workflows because the tag "unread" does not correspond to a maildir flag - rather, maildir has "S" (seen) so that OR of seen maildir flags decides whether notmuch does not tag the message "unread". Notice how this imposes a "gold standard workflow" on how you use read/unread on multiple copies of the same message? On the IMAP side, marking 1 copy of your message read or unread does not change the other one (much to the dismay of users who want to use automatic filters for sorting). Sync that (mbsync or such) two your maildir store and you one message file with "S", the other one without. Index with notmuch and *the* message will not be "unread" (even though one copy is not "Seen" per maildir flag). Toggle the "unread" tag with notmuch and all copies or none will have "S"; syncing back to the IMAP side will correspondingly mark all copies read or all unread. So, unless you force notmuch to sync tags back to maildir flags (e.g. by toggling them) it does not even mark all copies S even though the tag (absence of "unread") would warrant that. But whether this is the correct representation of "S" as "not unread" really depends on your workflow, that is your usage of copies und your usage of that flag. And yet, notmuch does sync it rather than bailing out because it requires a decision. In the very same way as marking one copy read marks all read (on the notmuch side as opposed to the IMAP side), why shouldn't the same be true for deleted? Because users with an IMAP mindset will complain about one but not the other, even though they are analogous... That's why I suggest the softer approach of treating the union of "T" as and AND: tag a message as "trashed" if all its copies have maildir flag "T". That way, users with multiple copies
Re: [PATCH] test: change database from within pre-new hook
David Bremner venit, vidit, dixit 2021-05-12 00:34:33: > Michael J Gruber writes: > > > Due to the change in the config system, notmuch keeps a notmuch database > > open when it would not do so before. Consequently, it can miss changes > > to the database which are done from a hook (while notmuch holds the > > databse in read only mode). When notmuch itself writes to the database > > after that it uses wrong assumptions about the last used doc id etc. > > > > Demonstrate this by triggering an assertion. (This new test succeeds > > with notmuch 0.31.4.) > > > > Signed-off-by: Michael J Gruber > > --- > > test/T400-hooks.sh | 21 + > > 1 file changed, 21 insertions(+) > > > +notmuch search '*' > change.expected > > That line looks like leftover debugging to me, so I deleted > it. Otherwise it works fine, thanks for tracking down the bug and > providing a test. Thanks for the fix! I didn't put it in the commit message (because that commit might go before for your patch), but the test succeeds with your fix. Sorry for the debug line - typical case of re-editing without re-format-patching... I was wondering if it's worthwhile to test also: - that the change from pre hook is persistent (test for that msg) - test the same for post (even though the db is closed before) Cheers Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH] test: change database from within pre-new hook
Due to the change in the config system, notmuch keeps a notmuch database
open when it would not do so before. Consequently, it can miss changes
to the database which are done from a hook (while notmuch holds the
databse in read only mode). When notmuch itself writes to the database
after that it uses wrong assumptions about the last used doc id etc.
Demonstrate this by triggering an assertion. (This new test succeeds
with notmuch 0.31.4.)
Signed-off-by: Michael J Gruber
---
test/T400-hooks.sh | 21 +
1 file changed, 21 insertions(+)
diff --git a/test/T400-hooks.sh b/test/T400-hooks.sh
index 00c99337..58e2b1dd 100755
--- a/test/T400-hooks.sh
+++ b/test/T400-hooks.sh
@@ -28,6 +28,16 @@ EOF
echo "${TOKEN}" > ${2}
}
+create_change_hook () {
+mkdir -p ${HOOK_DIR}
+cat <"${HOOK_DIR}/${1}"
+#!/bin/sh
+notmuch insert --no-hooks < ${2} > /dev/null
+rm -f ${2}
+EOF
+chmod +x "${HOOK_DIR}/${1}"
+}
+
create_failing_hook () {
local HOOK_DIR=${2}
mkdir -p ${HOOK_DIR}
@@ -176,6 +186,17 @@ EOF
NOTMUCH_NEW
test_expect_equal_file write.expected write.output
+test_begin_subtest "pre-new with change in database [${config}]"
+test_subtest_known_broken
+rm -rf ${HOOK_DIR}
+notmuch search '*' > change.expected
+generate_message '[subject]="Inserted"'
+create_change_hook "pre-new" $gen_msg_filename $HOOK_DIR
+generate_message
+NOTMUCH_NEW
+output=$(notmuch search id:$gen_msg_id | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX 2001-01-05 [1/1] Notmuch Test
Suite; pre-new with change in database [${config}] (inbox unread)"
+
rm -rf ${HOOK_DIR}
done
test_done
--
2.31.1.708.gc9a0ac0934
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Possible threading issues in nm 0.32
... just a guess: Could it be that
a9f74aee ("CLI/new: drop the write lock to run the pre-new hook.", 2021-03-18)
was not enough?
notmuch_database_reopen() only reopens the xapian db but does not update
other members in notmuch_database_t *notmuch, such as the last doc id
and thread id.
If a pre-merge hook changes the database then values in that struct will
be out of date.
Before the config changes in nm 0.32, there was no such struct to begin
with. After that, notmuch holds the struct just to be able to run the
hook (from the proper dir).
So I would think that reopen()ing to MODE_READ_ONLY is not a problem.
But after the hook run, are full close() then open() to
MODE_READ_WRITE() is necessary so that the values in the struct are
correct (or change reopen() to do that).
Indeed, if you open in MODE_READ_ONLY and don't hold a write lock you
cannot trust any cached values such as those stored in the struct, can
you?
Michael
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
