Re: [PATCH] ruby: query: fix get sort
Felipe Contreras writes: > The order was wrong, right now `query.sort` doesn't return a number. > applied to master d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: git.notmuchmail.org changes
David Bremner writes: > The notmuch git repo is now hosted in gitolite [1]. This will allow more > fine-grained access control. There should not be any visible change to > https access, but let me know if something broke. Michael Gruber mentioned offlist that git:// cloning was broken. While I encourage people to migrate to https:// if possible, I also think I fixed git:// cloning. For the moment we still need to run the git daemon for wiki updates. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
git.notmuchmail.org changes
The notmuch git repo is now hosted in gitolite [1]. This will allow more fine-grained access control. There should not be any visible change to https access, but let me know if something broke. d [1] a front end for git, see https://gitolite.com signature.asc Description: PGP signature ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Michael J Gruber writes: > oh well, attachments ... > Can you encrypt to the key 6D92612D94E46381 interactively using an approriately simplified version of that command? ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: T350-crypto T357-index-decryption: possible race condition?
Michael J Gruber writes: > Building notmuch 0.37 with my usual spec-file as a rawhide-mock build > (a local chroot for the development "version" of Fedora which will > become Fedora 39) I see: > ``` > T350-crypto: Testing PGP/MIME signature verification and decryption > PASS emacs delivery of signed message via fcc > PASS emacs delivery of signed message via fcc and smtp > PASS signed part content-type indexing > PASS signature verification > PASS detection of modified signed contents > PASS corrupted pgp/mime signature > PASS signature verification without full user ID validity > PASS signature verification with signer key unavailable > ``` > There the suite "hangs" for about 2 minutes, followed by This sounds suspiciously like "timeout" kicking in and killing a test that is taking too long. You can set NOTMUCH_TEST_TIMEOUT in the environment to some smaller/larger number to test this. The first subtest in T357 is also sending an encrypted message, so it looks like some bad interaction between gpg and emacs. Maybe you can try sending an encrypted message from emacs interactively in your chroot environment. 0) you will first need to run gpg --no-tty --import ./test/openpgp4-secret-key.asc this will create a keyring etc... in the chroot 1) You can use the script ./devel/try-emacs-mua -q 2) Copy the following into *scratch* and run with C-j after the last closing paren (this is just copied from the test suite internals) (let ((message-send-mail-function (lambda () t)) (mail-host-address "example.com")) (notmuch-mua-mail) (message-goto-to) (insert "test_su...@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -") (message-goto-subject) (insert "My Subject") (message-goto-body) (insert "a body") (mml-secure-message-encrypt) (let ((mml-secure-smime-sign-with-sender t) (mml-secure-openpgp-sign-with-sender t)) (notmuch-mua-send-and-exit))) You will probably need to answer "c" to create the sent-mail folder at the prompt. > ``` > In the end, the suite complains: > ``` > '/builddir/build/BUILD/notmuch-0.37/test/test-results/T350-crypto' > does not exist! > '/builddir/build/BUILD/notmuch-0.37/test/test-results/T357-index-decryption' > does not exist! > ``` > At least for T350 this is strange because several subtests ran and > passed! This indicates a race or a wrong signal trap. Note that those files are summaries created by test_done, so it's not that surprising that they are not there, since test_done is not reached in those files. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 0/2] vim: doc: trivial fixes
Felipe Contreras writes: > These are some trivial patches long overdue from upstream. > > They have been already sent in some form long time ago > id:20210418224851.88240-3-felipe.contre...@gmail.com, but hopefully in a > smaller batch they'll get merged. > applied to master. Of course I still have no real way to review the doc patches, but I guess won't e.g. break the build. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 3/3] test/emacs: Add test cases for notmuch-search-hide-excluded
mohk...@kisara.moe writes: > From: Mohsin Kaleem > thanks for writing such an extensive set of tests for the new feature. All I'd ask is a bit more detail in the commit message. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 2/3] emacs: Allow notmuch-saved-searches to hide excluded messages
mohk...@kisara.moe writes: > From: Mohsin Kaleem > Apologies for the many messages. I think this commit message falls a bit short of our usual standards (as well as a one line subject, we like to have some kind of "why"). If you figure out the right answer to toggling saved searches with :excluded properties, it might be worth explaining your reasoning in the commit message. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 1/3] emacs: Add new option notmuch-search-hide-excluded
mohk...@kisara.moe writes: > Lastly I've amended some calls to notmuch-tree and notmuch-unthreaded > which didn't pass through the buffer local value of > notmuch-search-oldest-first (and now notmuch-search-exclude). > Examples of where I've done this > + include notmuch-jump-search if respinning, move 'include' to previous line > > If there was a reasoning behind these not persisting the value of these > variables then we should revert it before merging and discuss whether > it's worth persisting notmuch-search-exclude. > I think the fixes are fine, and make sense to include since you are touching that code anyway. This sentence probably doesn't belong in the commit message, so if respinning the series, I would drop it. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 2/3] emacs: Allow notmuch-saved-searches to hide excluded messages
mohk...@kisara.moe writes: > + :excludedWhether to show mail with excluded tags in the > + search. Possible values are `hide', `show', > + or nil. Nil means use the default value of > + `notmuch-search-hide-excluded'. So if I understand correctly, toggling excludes will not affect saved searches with a :excluded key. Is this what we want? I guess the alternative is to AND notmuch-search-hide-excluded with the value derived from the saved-search? ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 1/3] emacs: Add new option notmuch-search-hide-excluded
Mohsin Kaleem writes: > We could workaround this by changing the semantics of the variable. Make > it notmuch-search-show-excluded and default to nil so nothing has to be > changed in existing functions that call notmuch-search. Or we could > switch to using cl-defun and have the default value substituted in the > parameter list. I've never used cl-defun with interactive functions but > long term I think that's the saner solution. Thanks for the explanation. I think we can live with it as is for now, and look into the cl-defun "upgrade" later. I suspect it will be a bit confusing if we invert the sense of the variable relative to the CLI argument. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 1/3] emacs: Add new option notmuch-search-hide-excluded
> diff --git a/test/T310-emacs.sh b/test/T310-emacs.sh > index 9d0df187..b89a100e 100755 > --- a/test/T310-emacs.sh > +++ b/test/T310-emacs.sh > @@ -1030,7 +1030,7 @@ End of search results. > YYY/notmuch_fail exited with status 1 (see *Notmuch errors* for more details) > === ERROR === > YYY/notmuch_fail exited with status 1 > -command: YYY/notmuch_fail search --format\=sexp --format-version\=5 > --sort\=newest-first tag\:inbox > +command: YYY/notmuch_fail search --format\=sexp --format-version\=5 > --sort\=newest-first --exclude\=false tag\:inbox > exit status: 1" > I don't understand why --exclude is defaulting to 'false' here. Isn't this a change in behaviour? d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: JSON Output Documentation
David Bremner writes:
> "Tim Culverhouse" writes:
>
>> Hey everyone -
>>
>> I've scoured the manpages and website for some documentation on the JSON
>> output format, but haven't found what I'm looking for. For the most
>> part, the commands are easy to figure out, but `notmuch show` has me
>> confused on the array nesting. Does anyone know if there is
>> documentation on this structure?
>>
>> It seems to be something like:
>>
>> [ (Array of results)
>> [ (Array of thread)
>> [ (Another array of thread?)
>> {Thread parent object},
>> [ (array of thread children objects)]
>> ]
>> ]
>> ]
>
> We have had some discussions on the pros and cons of more publically
> documenting this interface, but for now you can find the documentation
> in the source, in devel/schemata.
See attached.
schemata
Description: Binary data
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: JSON Output Documentation
"Tim Culverhouse" writes:
> Hey everyone -
>
> I've scoured the manpages and website for some documentation on the JSON
> output format, but haven't found what I'm looking for. For the most
> part, the commands are easy to figure out, but `notmuch show` has me
> confused on the array nesting. Does anyone know if there is
> documentation on this structure?
>
> It seems to be something like:
>
> [ (Array of results)
> [ (Array of thread)
> [ (Another array of thread?)
> {Thread parent object},
> [ (array of thread children objects)]
> ]
> ]
> ]
We have had some discussions on the pros and cons of more publically
documenting this interface, but for now you can find the documentation
in the source, in devel/schemata.
At this point I'm open to people who either want to convert that
document to rst and integrate it with the published docs, or convince me
it's a bad idea. I think the idea was that we wanted to avoid commiting
to that interface, but since we already version it like a C library, I'm
a bit fuzzy on why that's a problem.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Correcting message references
David Bremner writes:
> Al Haji-Ali writes:
>
>> So it does seem to be a lingering ghost message, but I am sure that there
>> are no messages in the database referring to this ID (except messages in
>> this current thread which have the ID in the message body).
>> I don't know why this particular ID is associated to messages in another
>> seemingly unrelated thread as you in the pdf.
>>
>> Is there a way to remove this ghost message record somehow to test it? Or is
>> there a better way of figuring this out.
>
> It turns out notmuch does not remove ghost messages until all the other
> messages in the thread are deleted. I guess if you temporarily move
> the other messages in the thread out of the way and run notmuch new, the
> ghost message should be deleted.
>
> I don't know how often this lazy deletion is a problem. Deleting
> messages is already a bottleneck in notmuch-new so I am a bit hesitant
> to make it more complicated. It is possible to "garbage collect"
> unreferenced ghost messages. I'll have to think about how big a
> performance hit it would be to add this to notmuch new.
>
> d
Here is a prototype standalone program to find lingering unreferenced
ghosts. I find 33 (out of about 60k total ghost messages) in about 0.3s
on this laptop. Currently it does not modify the database, but the next
step would be to delete the documents rather than just printing them
out.
If you have libxapian-dev (or equivalent) installed you can build it
with
$ c++ ggc.cc -o ggc -lxapian
and then run it
$ ./ggc ~/.local/share/notmuch/default/xapian
I would be interested if it finds your problematic ghost message (and
how long it takes).
#include
#include
int main(int argc, char **argv){
if (argc != 2) {
fprintf (stderr, "usage: ggc xapian-database\n");
exit (1);
}
Xapian::Database db(argv[1]);
Xapian::Enquire enquire(db);
enquire.set_query(Xapian::Query("Tghost"));
auto mset = enquire.get_mset (0,db.get_doccount ());
for (auto iter=mset.begin (); iter != mset.end(); iter++){
std::string mid;
auto doc = iter.get_document ();
auto term_iter = doc.termlist_begin ();
term_iter.skip_to ("Q");
mid=(*term_iter).substr(1);
std::string ref_term = "XREFERENCE" + mid;
auto ref_count = db.get_termfreq (ref_term);
std::string reply_term = "XREPLYTO" + mid;
auto reply_count = db.get_termfreq (reply_term);
if (ref_count+reply_count == 0){
std::cout << "docid=" << *iter;
std::cout << " mid=" << mid;
std::cout << std::endl;
}
}
}
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Correcting message references
> > As long as some message refers to that ID, notmuch will create a "ghost > message", used for threading. You can look for a specific ghost message with something like $ quest -btype:T -b id:Q -d .local/share/notmuch/default/xapian \ "type:ghost and id:jwvczk7opm8.fsf-monnier+em...@gnu.org" quest is also part of xapian-tools. Unfortunately I don't think quest understands the way notmuch uses multiletter prefixes (without a :), so to find references you still need to use xapian-delve. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Correcting message references
Al Haji-Ali writes: > is completely unconnected the other 4 messages in the thread. Note > that if I change the "In-Reply-To" field in this message to anything > else, notmuch no longer groups these 5 messages into a single thread. > Yes, that's puzzling. I did not think about "ghost messages" (see below) when writing that script, so maybe that's the issue. > I tried searching for "jwvczk7opm8.fsf-monnier+em...@gnu.org" using > `xapian-delve` but got > > , > | term 'jwvczk7opm8.fsf-monnier+em...@gnu.org' not in database > ` You need to give the appropriate term prefix. Q for message id, or XREPLYTO or XREFERENCE as in my last message. > The first one is the draft. The second hit is the reason I thought the > only place left for notmuch to associate these messages is in the > xapian database. Note that if I delete the draft and reindex, only the > postlist.glass hit stubbornly remains and there seems to be no way to > make notmuch forget about this ID. As long as some message refers to that ID, notmuch will create a "ghost message", used for threading. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Correcting message references
Al Haji-Ali writes: > I changed the message, removed "B" from "References" and deleted the > files of all old (and intermediate) drafts that have "B" in > "References". But no matter what I do, I have "B" grouped with "D" > and any other messages which I create with "In-Reply-To" being "A". How did you find the files to delete? One trap to watch out for is that if using notmuch, you should use notmuch search --exlude=false, to make sure messages are not being hidden because of their tags. > I suspect that somewhere in the database the IDs of "A" and "B" are > linked now. Is there a way (short of deleting the database and > re-indexing) to correct this and remove this connection? The database does not store relationships explicitely, only via messages with references to other messages. At a high level you can try the attached script to get a picture of the corresponding thread. If you can't run the script, or it doesn't help, you can interrogate the database directly without going through notmuch. if the message-id of B is 'f...@example.org' you can search with for replies with xapian-delve (in xapian-tools on Debian and derivatives). xapian-delve -d .local/share/notmuch/default/xapian \ -t 'xreplyto...@example.org' and for references xapian-delve -d .local/share/notmuch/default/xapian \ -t 'xreference...@example.org' That will give you Xapian record numbers, and you can turn those into files with something like xapian-delve -d .local/share/notmuch/default/xapian -r 801793 -1 | \ perl -ne 's/XF(D|O).*?:// && print' For records with multiple files, you will have to figure out with file goes with which directory (or just find the file names, which supposed ot be unique). draw-thread Description: Binary data ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: API docs: notmuch_database_open_with_config with NULL config
James Cook writes: > It is possible that a short note indicating that detailed descriptions > will follow, placed at the top of the man page and/or at the top of the > "Functions" section, would have saved me from my impatience. I think I > do tend to read the first sentence or two of a man page before just > searching for what I'm looking for. But absent such a note, maybe I can > be excused for paging through a few screenfuls of API summary and > assuming that's all there was. I'm a bit reluctant to invest much effort in the current Doxygen based API docs (I'm hoping we eventually replace them with something more compatible with the rest of the sphinx-doc based documentation). On the other hand if someone else out there is a better Doxygen wrangler than I and the fix is relatively small, I don't see why we wouldn't apply that change. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: API docs: notmuch_database_open_with_config with NULL config
James Cook writes: > Hi list, > > If I'm not mistaken, calling notmuch_database_open_with_config with > config_path = NULL causes notmuch to try to find the config file > automatically. > > It would be nice if that were documented in the notmuch(3) man page. I > can try drafting a patch if that would help. > > (This message brought to you by trying to understand neomutt's notmuch > code...) Here is what i see in the notmuch(3) man page: config_path Path to config file. Config file is key-value, with mandatory sections. See notmuch-config(5) for more information. The key-value pair overrides the corresponding configuration data stored in the database (see notmuch_database_get_config) If config_path is NULL use the path specified • in environment variable NOTMUCH_CONFIG, if non-empty • by XDG_CONFIG_HOME/notmuch/ where XDG_CONFIG_HOME defaults to '$HOME/.config'. • by $HOME/.notmuch-config If config_path is '' (empty string) then do not open any configuration file. profile Name of profile (configuration/database variant). If non-NULL, append to the directory / file path determined for config_path and database_path. If NULL then use • environment variable NOTMUCH_PROFILE if defined, • otherwise 'default' for directories and '' (empty string) for paths. I'm not claiming the documentation is perfect, but it seems to be documented? Is there maybe some version skew between your man pages and library? d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] doc: fix typos
Jakub Wilk writes: > --- > doc/man1/notmuch-config.rst | 2 +- > doc/man1/notmuch-git.rst | 4 ++-- > doc/man7/notmuch-sexp-queries.rst | 8 > doc/notmuch-emacs.rst | 6 +++--- > 4 files changed, 10 insertions(+), 10 deletions(-) applied to master, thanks d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 3/3] ruby: remove FileNames object
Felipe Contreras writes: > Not used anymore now that we return an array of strings directly. series applied to master, with one whitespace tweak in patch 2/3 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] notmuch-mutt: replace extra command with notmuch-native thread search feature
Paul Wise writes: > that changes the meaning of the query but I couldn't find any way to > prevent an arbitrary Message-Id from inserting parentheses into the > query. Also notmuch converts Message-Id (test)@hostname to just > @hostname in the Xapian database. I think that right now I am not sure > that re-implementing all the idiosyncrasies of notmuch queries and > Message-Id munging within notmuch-mutt is the way to go, especially > since Message-Id fields that intersect with notmuch features are rare. The code that does removal of spaces and () delited sequences goes back to Carl's 2009 implementation of manual header parsing (which we have mostly, but not entirely replaced with calls to gmime). It would probably be reasonable to disable that code (or maybe migrate to gmime parsing of message-ids?), but the cost/benefit analysis is not too clear to me. I had to look this up, but apparently parens mark a comment in RFC822 / RFC5322. If I understand RFC5322 correctly then comments are only permitted in the obsolete syntax, but I guess it makes sense to accept those. I don't know how many messages in the wild use this syntax; of the .01% in in my mail store, most don't put the comment inside the actual <> delimited message-id id, and those that do look like errors of various kinds - some unbalanced parens - some (null) indicating printf-ing null pointers ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] notmuch-mutt: replace extra command with notmuch-native thread search feature
Paul Wise writes: > So I think that it would be good if there were a library for notmuch > query construction that would take care of all the different quoting > levels, Message-Id munging etc. This could then get language bindings > so that notmuch based tools and MUAs could use them for interactive > graphical query construction. Also maybe Xapian needs or has a way to > construct queries without serialising them to a text string and the > notmuch query construction library I propose could base itself on that. You might be interested in the s-expression query parser. Part of the goal is to be less dwim/quirky than the native Xapian query parser, and to make it easier / safer to construct notmuch queries via programs. It doesn't do the message-id munging you mention (yet?), but quoting is IMHO saner. It does require an extra dependency (sfsexpr). ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v3 5/5] debian: add autopkgtests
We generate output to stderr for BROKEN tests, which are not failures, so tell the test runner not to fail because of output on stderr. --- debian/tests/control | 17 + 1 file changed, 17 insertions(+) create mode 100644 debian/tests/control diff --git a/debian/tests/control b/debian/tests/control new file mode 100644 index ..87e00dc8 --- /dev/null +++ b/debian/tests/control @@ -0,0 +1,17 @@ +Test-command: env NOTMUCH_TEST_INSTALLED=1 TERM=dumb +NOTMUCH_HAVE_MAN=1 NOTMUCH_HAVE_SFSEXP=1 NOTMUCH_HAVE_XAPIAN_DB_RETRY_LOCK=1 +NOTMUCH_HAVE_PYTHON3_CFFI=1 NOTMUCH_HAVE_PYTHON3_PYTEST=1 +NOTMUCH_HAVE_ASAN=1 NOTMUCH_HAVE_TSAN=1 +./test/notmuch-test +Restrictions: allow-stderr +Depends: @, + build-essential, + dtach, + emacs-nox, + gdb, + git, + gnupg, + gpgsm, + libtalloc-dev, + man, + xapian-tools -- 2.39.2 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v3 1/5] test: use bash specific test for feature tests
It is desirable to have the tests consider these variables being
undefined as equivalent to the feature not being present, and in
particular for the tests not to generate errors.
We know the test suite is tied to bash anyway, so this is a simple
fix.
---
test/T060-count.sh | 2 +-
test/T081-sexpr-search.sh| 2 +-
test/T150-tagging.sh | 2 +-
test/T160-json.sh| 2 +-
test/T220-reply.sh | 2 +-
test/T240-dump-restore.sh| 2 +-
test/T391-python-cffi.sh | 3 +--
test/T392-python-cffi-notmuch.sh | 2 +-
test/T520-show.sh| 2 +-
test/T570-revision-tracking.sh | 2 +-
test/T700-reindex.sh | 2 +-
test/T800-asan.sh| 2 +-
test/T810-tsan.sh| 2 +-
test/T850-git.sh | 2 +-
14 files changed, 14 insertions(+), 15 deletions(-)
diff --git a/test/T060-count.sh b/test/T060-count.sh
index 48146706..4499b4cb 100755
--- a/test/T060-count.sh
+++ b/test/T060-count.sh
@@ -157,7 +157,7 @@ print("4: {} messages".format(query.count_messages()))
EOF
test_expect_equal_file EXPECTED OUTPUT
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [[ $NOTMUCH_HAVE_SFSEXP -eq 1 ]]; then
test_begin_subtest "and of exact terms (query=sexp)"
output=$(notmuch count --query=sexp '(and "wonderful" "wizard")')
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 0c7db9c2..17b02e06 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -2,7 +2,7 @@
test_description='"notmuch search" in several variations'
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_SFSEXP -ne 1 ]; then
+if [[ $NOTMUCH_HAVE_SFSEXP -ne 1 ]]; then
printf "Skipping due to missing sfsexp library\n"
test_done
fi
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index ac3f2539..af0b8448 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -328,7 +328,7 @@ test_expect_equal "$output" "A Xapian exception occurred
opening database"
add_email_corpus
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [[ $NOTMUCH_HAVE_SFSEXP -eq 1 ]]; then
test_query_syntax '(and "wonderful" "wizard")' 'wonderful and wizard'
test_query_syntax '(or "php" "wizard")' 'php or wizard'
diff --git a/test/T160-json.sh b/test/T160-json.sh
index 4a797f6a..6f637a21 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -66,7 +66,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"unread\"]}]"
test_begin_subtest "Search message: json, 64-bit timestamp"
-if [ $NOTMUCH_HAVE_64BIT_TIME_T -ne 1 ]; then
+if [[ $NOTMUCH_HAVE_64BIT_TIME_T -ne 1 ]]; then
test_subtest_known_broken
fi
add_message "[subject]=\"json-search-64bit-timestamp-subject\"" "[date]=\"Tue,
01 Jan 2999 12:00:00 -\"" "[body]=\"json-search-64bit-timestamp-message\""
diff --git a/test/T220-reply.sh b/test/T220-reply.sh
index 207f5788..a5a9a1df 100755
--- a/test/T220-reply.sh
+++ b/test/T220-reply.sh
@@ -24,7 +24,7 @@ test_begin_subtest "Basic reply"
notmuch reply id:${gen_msg_id} >OUTPUT 2>&1 && echo OK >> OUTPUT
test_expect_equal_file basic.expected OUTPUT
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [[ $NOTMUCH_HAVE_SFSEXP -eq 1 ]]; then
test_begin_subtest "Basic reply (query=sexp)"
notmuch reply --query=sexp "(id ${gen_msg_id})" >OUTPUT 2>&1 && echo OK >>
OUTPUT
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index a86f0fb7..b99a303e 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -118,7 +118,7 @@ notmuch dump -- from:cworth > dump-dash-cworth.actual
test_expect_equal_file dump-cworth.expected dump-dash-cworth.actual
-if [ $NOTMUCH_HAVE_SFSEXP -eq 1 ]; then
+if [[ $NOTMUCH_HAVE_SFSEXP -eq 1 ]]; then
test_begin_subtest "dump --query=sexp -- '(from cworth)'"
notmuch dump --query=sexp -- '(from cworth)' > dump-dash-cworth.actual2
diff --git a/test/T391-python-cffi.sh b/test/T391-python-cffi.sh
index 30872af0..59ad087f 100755
--- a/test/T391-python-cffi.sh
+++ b/test/T391-python-cffi.sh
@@ -2,11 +2,10 @@
test_description="python bindings (pytest)"
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ];
then
+if [[ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 || $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0
]]; then
test_done
fi
-
test_begin_subtest "python cffi tests (NOTMUCH_CONFIG set)"
pytest_dir=$NOTMUCH_BUILDDIR/bindings/python-cffi/build/stage
printf "[pytest]\nminversion = 3.0\naddopts = -ra\n" > $pytest_dir/pytest.ini
diff --git a/test/T392-python-cffi-notmuch.sh b/test/T392-python-cffi-notmuch.sh
index 15c8fc6b..c97da9b5 100755
--- a/test/T392-python-cffi-notmuch.sh
+++ b/test/T392-python-cffi-notmuch.sh
@@ -2,7 +2,7 @@
test_description="python bindings (notmuch test suite)"
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_PYTHON3_CFFI -eq 0 -o $NOTMUCH_HAVE_PYTHON3_PYTEST -eq 0 ];
then
+if [[
[PATCH v3 4/5] test: support testing notmuch as installed
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.
When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.
Some of the tests marked as broken when running against installed
notmuch are probably fixable.
---
test/README| 17 +
test/T000-basic.sh | 1 +
test/T010-help-test.sh | 5 -
test/T160-json.sh | 2 ++
test/T240-dump-restore.sh | 2 ++
test/T310-emacs.sh | 2 ++
test/T350-crypto.sh| 1 +
test/T360-symbol-hiding.sh | 4
test/T390-python.sh| 4
test/T391-python-cffi.sh | 6 +-
test/T395-ruby.sh | 8 ++--
test/T410-argument-parsing.sh | 4
test/T480-hex-escaping.sh | 4
test/T490-parse-time-string.sh | 4
test/T550-db-features.sh | 4
test/T566-lib-message.sh | 4
test/T592-thread-breakage.sh | 4
test/T710-message-id.sh| 4
test/T800-asan.sh | 5 +
test/T850-git.sh | 6 ++
test/export-dirs.sh| 2 +-
test/notmuch-test | 12 ++--
test/test-lib-common.sh| 13 ++---
test/test-lib-emacs.sh | 14 +++---
test/test-lib.sh | 27 +--
25 files changed, 140 insertions(+), 19 deletions(-)
diff --git a/test/README b/test/README
index 10f127cb..a81808b1 100644
--- a/test/README
+++ b/test/README
@@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may
treat tests
unable to run because of missing prerequisites, but not explicitly
skipped by the user, as failures.
+Testing installed notmuch
+-
+
+Systems integrators (e.g. Linux distros) may wish to test an installed
+version of notmuch. This can be done be running
+
+ $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
+
+In this scenario the test suite does not assume a built tree, and in
+particular cannot rely on the output of 'configure'. You may want to
+set certain feature environment variables ('NOTMUCH_HAVE_*') directly
+if you know those apply to your installed notmuch). Consider also
+setting TERM=dumb if the value of TERM cannot be used (e.g. in a
+chroot with missing terminfo). Note that having a built tree may cause
+surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
+cleaning first.
+
Writing Tests
-
The test script is written as a shell script. It is to be named as
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index a2f4d93f..642f918d 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an
existing file'
test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
test_begin_subtest 'PATH is set to build directory'
+test_subtest_broken_for_installed
test_expect_equal \
"$(dirname ${TEST_DIRECTORY})" \
"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
index da45d3ae..ae84c22d 100755
--- a/test/T010-help-test.sh
+++ b/test/T010-help-test.sh
@@ -12,13 +12,16 @@ test_expect_success 'notmuch help'
test_begin_subtest 'notmuch --version'
test_expect_success 'notmuch --version'
-if [ $NOTMUCH_HAVE_MAN -eq 1 ]; then
+if [[ $NOTMUCH_HAVE_MAN -eq 1 ]]; then
test_begin_subtest 'notmuch --help tag'
test_expect_success 'notmuch --help tag'
test_begin_subtest 'notmuch help tag'
test_expect_success 'notmuch help tag'
else
+if [[ -n $NOTMUCH_TEST_INSTALLED ]]; then
+ test_done
+fi
test_begin_subtest 'notmuch --help tag (man pages not available)'
test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
diff --git a/test/T160-json.sh b/test/T160-json.sh
index 6f637a21..b19058d7 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"tags\": [\"inbox\",
\"unread\"]}]"
+if [[ -z NOTMUCH_TEST_INSTALLED ]]; then
test_begin_subtest "Search message: json, 64-bit timestamp"
if [[ $NOTMUCH_HAVE_64BIT_TIME_T -ne 1 ]]; then
test_subtest_known_broken
@@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"query\": [\"id:$gen_msg_id\", null],
\"tags\": [\"inbox\",
\"unread\"]}]"
+fi
test_begin_subtest "Format version: too low"
test_expect_code 20 "notmuch search --format-version=0 \\*"
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index b99a303e..9072f636 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -139,6 +139,7 @@ notmuch dump --output=dump-outfile-dash-inbox.actual --
from:cworth
test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
[PATCH v3 2/5] test: check for empty/missing files in test_expect_equal_message_body
Messages can have empty bodies, but empty files are not messages.
---
test/test-lib.sh | 8
1 file changed, 8 insertions(+)
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 1a6525df..b5aa94dd 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -388,6 +388,14 @@ test_expect_equal_message_body () {
test "$#" = 2 ||
error "bug in the test script: not 2 parameters to
test_expect_equal_file"
+for file in "$1" "$2"; do
+ if [ ! -s "$file" ]; then
+ test_failure_ "Missing or zero length file: $file"
+ inside_subtest=
+ return $?
+ fi
+done
+
expected=$(sed '1,/^$/d' "$1")
output=$(sed '1,/^$/d' "$2")
test_expect_equal "$expected" "$output"
--
2.39.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v3 3/5] test: Guess a value for NOTMUCH_PYTHON
python3 will work for many people, and reduce the friction to running
the tests without running configure first.
---
test/test-lib-common.sh | 4
1 file changed, 4 insertions(+)
diff --git a/test/test-lib-common.sh b/test/test-lib-common.sh
index 18fa29c0..4d14e0b3 100644
--- a/test/test-lib-common.sh
+++ b/test/test-lib-common.sh
@@ -323,6 +323,10 @@ rm -rf "$TMP_DIRECTORY" || {
exit 1
}
+# Provide a guess at a usable Python, to support running tests without
+# running configure first.
+NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
+
# A temporary home directory is needed by at least:
# - emacs/"Sending a message via (fake) SMTP"
# - emacs/"Reply within emacs"
--
2.39.2
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
v3 test notmuch as installed
Compared to the previous version [1], the first patch is now applied, and the setup for running without configure now relies on the various NOTMUCH_HAVE_* environment variables working OK when unset. This allows eliminating the "defaults.sh". There is also some added documentation on how to use the new feature (in patch 4/5). [1]: id:20230104124145.2882983-1-da...@tethera.net ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Data loss
Carl Worth writes: > > What it could also do is create some placeholder file like > .NOTMUCH-MUTT-OUTPUT-DIR when it writes its output. And then it could > check for that file's existence before removing a directory. > Sounds reasonable to me. Would someone like to make a patch for notmuch-mutt? d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] notmuch-mutt: replace extra command with notmuch-native thread search feature
Paul Wise writes:
> +search($results_dir, $remove_dups, qq{thread:"{id:$mid}"});
did you test this with a message-id with spaces in it? the quoting is a
delicate.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes: > On Tue, Apr 4, 2023 at 12:54 PM David Bremner wrote: >> >> This sounds right. Can we use the detection of missing messages in >> wr_export to reset the appropriate counters? It looks like yes, given >> the call to store_lastmod. [snip] > I would rather go for a solution that is less hacky, and has less > chance of leaving the user in an unrecoverable state. fair enough. Certainly notmuch-git has too many accumulated performance hacks. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes: > On Mon, Apr 3, 2023 at 6:37 PM David Bremner wrote: > Or we could say that after jumping a certain threshold of lastmod we > delete all the messages and start from scratch, perhaps every 1000 > revisions. > > Or maybe the query could generate a virtual tag if a message was > deleted since the previous lastmod (e.g. "nm::deleted"). Then it would > be trivial for the remote helper to tell that to git. A complication here is that tags be attached to mail message documents in the database, so we would need to generate a so called "ghost message", and clean those up somehow. > I lean towards the threshold, because that way the user doesn't need > to do anything, and there's no modifications needed in libnotmuch. This sounds right. Can we use the detection of missing messages in wr_export to reset the appropriate counters? It looks like yes, given the call to store_lastmod. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes: > > I'm not familiar with git-annex, I would need to see an example of > such merging happening. I was confused, git-annex is using the builtin merge strategy "union", which is not eliminating duplicates or sorting, so probably not applicable here. I still have to try some merges between different machines to see what kind of conflicts can arise. > One advantage of using the fast-import format is that it's easy to > change it, or support multiple formats. > > In fact, the format could be specified in the URL, like > `nm::1:$HOME/mail` for the current notmuch-git format, and > `nm::2:$HOME/mail` for the new. This might also be a way to handle the "prefix" setting that nmbug / notmuch-git needs to only sync certain (e.g. notmuch::*) tags ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes:
>
> That should work to update existing tags, but how are we going to
> detect if a message has disappeared? Or is that not a thing?
Indeed the same thought had occurred to me not long ago. I remembered
(belately) that I'd been through some similar thought process with nmbug.
Messages can and do disappear. So for I guess that optimization not OK,
at least not without some complications.
> Does "lastmod:0.." get all the revisions? If so, it might make sense
> to set $lastmod to 0 initially.
>
> Then we could unconditionally do:
>
> $db.query('lastmod:%d..' % $lastmod, sort: Notmuch::SORT_UNSORTED)
That would work, but as you point out, we'd need to deal with deletions
somehow. It occurs to me that wr_export also needs to be able to handle
disappearing message-ids. I suppose like notmuch-restore it can just
complain and skip any missing ones. It's tempting to try to do some kind
of lazy cleanup at that point, but I don't really see how that fits with
the remote-helper protocol.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
David Bremner writes:
> Indeed that speeds up the initial clone on this machine from 39 minutes
> (I switched machines) to 30s. I will play with it a bit more, and report
> back.
It's not a showstopper, but "git pull" takes about 1/2 the wall time
(about 2/3 of the CPU time) of the original clone, even if there is only
one tag changed.
Two potential improvements I can think of.
- notmuch-dump.c calls notmuch_query_set_sort (query,
NOTMUCH_SORT_UNSORTED). I think I managed to do this (diff below),
but performance gain was negligible.
- Since you cache the lastmod value, you should be able to use it in a
query. This does make a big difference in my experiments. I had to
remove the 'deleteall' (otherwise only the changed messages are left
in the git repo). I'm not 100% this is correct, hopefully you see
quicker than I. In any case the lastmod query is what notmuch-git
uses.
diff --git a/git-remote-nm b/git-remote-nm
index c668b38..cabea26 100755
--- a/git-remote-nm
+++ b/git-remote-nm
@@ -148,9 +148,11 @@ def wr_import(ref)
wr_data("lastmod: %d\n" % ($lastmod || 0))
wr_l 'from refs/notmuch/master^0' if $lastmod
- wr_l 'deleteall'
+# wr_l 'deleteall'
- $db.query('').search_messages.each do |msg|
+ $query=$db.query("lastmod:%d.." % ($lastmod || 0) )
+ $query.sort=Notmuch::SORT_UNSORTED
+ $query.search_messages.each do |msg|
hash = Blake2b.hex(msg.message_id, Blake2b::Key.none, 2)
dir1, dir2 = hash[..1], hash[2..]
wr_l 'M 644 inline %s/%s/%s/tags' % [dir1, dir2,
encode_filename(msg.message_id)]
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes: > By distributing the files in multiple directories like notmuch-git > does using BLAKE2b, the operation is much faster. > > I've pushed the changes, now there's a dependency, but you can just > `gem install blake2b`. > > I'm able to clone the database of the performance corpus in 5 seconds: > > % git clone --bare nm::$PWD/mail mail.git Indeed that speeds up the initial clone on this machine from 39 minutes (I switched machines) to 30s. I will play with it a bit more, and report back. I had just finished a pretty graph showing nonlinear growth of the old version, but I guess nobody cares now ;) d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Data loss
Fulvio Pizzigoni writes: > > After this my .mutt directory (~ 4 GB di mail-boxess) appears so: > fulvio@linux:~$ ll .mutt > totale 12 > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp Hi Fulvio; I don't understand your situation yet, but I have a preliminary question. Is the date on your computer more or less correct? You mention the directories cur, new, and tmp being created by running notmuch setup; on the other hand your listing above shows they already existed in February. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Reimagining notmuch-git/nmbug
David Bremner writes: > > I'm intrigued (and indeed I hadn't really thought about the degree to > which we were re-inventing git-fast-import and friends); however so far > my experiments did not get far enough to say anything conclusive. > I did manage to finish, about 70 minutes elapsed. Although you'r probably right that a file of tags is the right representation (it is what git-annex uses also), I think we'd need to define a custom merge driver to take unions of lists in the same way that git-annex does. Otherwise merging will be less automagic than it is now. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH] CLI/git: add reset command
Sometimes merging is not what we want with tags; in particular it
tends to keep tags in the local repo that have been removed elsewhere.
This commit provides a new reset command; the reset itself is trivial,
but the work is to provide a safety check that uses the existing
--force and git.safe_fraction machinery.
---
notmuch-git.py | 37 +++--
test/T850-git.sh | 46 +-
2 files changed, 80 insertions(+), 3 deletions(-)
diff --git a/notmuch-git.py b/notmuch-git.py
index 57098aae..80787814 100644
--- a/notmuch-git.py
+++ b/notmuch-git.py
@@ -369,7 +369,7 @@ class CachedIndex:
_git(args=['read-tree', self.current_treeish], wait=True)
-def check_safe_fraction(status):
+def _check_fraction(change):
safe = 0.1
conf = _notmuch_config_get ('git.safe_fraction')
if conf and conf != '':
@@ -380,7 +380,7 @@ def check_safe_fraction(status):
_LOG.error('No existing tags with given prefix,
stopping.'.format(safe))
_LOG.error('Use --force to override.')
exit(1)
-change = len(status['added'])+len(status['deleted'])
+
fraction = change/total
_LOG.debug('total messages {:d}, change: {:d}, fraction:
{:f}'.format(total,change,fraction))
if fraction > safe:
@@ -388,6 +388,25 @@ def check_safe_fraction(status):
_LOG.error('Use --force to override or reconfigure git.safe_fraction.')
exit(1)
+def check_safe_fraction(status):
+
+change = len(status['added'])+len(status['deleted'])
+_check_fraction(change)
+
+def check_diff_fraction():
+
+# check number of directories (i.e. messages) changed.
+change_set = set()
+
+with _git(args=['diff', '--name-only', 'HEAD', '@{upstream}'],
+ stdout=_subprocess.PIPE) as git:
+for path in git.stdout:
+change_set.add(_os.path.dirname(path))
+
+change=len(change_set)
+_check_fraction(change)
+
+
def commit(treeish='HEAD', message=None, force=False):
"""
Commit prefix-matching tags from the notmuch database to Git.
@@ -620,6 +639,15 @@ def push(repository=None, refspecs=None):
_git(args=args, wait=True)
+def reset(force=False):
+"""
+reset the local git branch to match the remote one
+"""
+if not force:
+check_diff_fraction()
+
+_git(args=["reset","--soft","origin/master"],wait=True)
+
def status():
"""
Show pending updates in notmuch or git repo.
@@ -1048,6 +1076,7 @@ if __name__ == '__main__':
'merge',
'pull',
'push',
+'reset',
'status',
]:
func = locals()[command]
@@ -1142,6 +1171,10 @@ if __name__ == '__main__':
'Refspec (usually a branch name) to push. See '
'the entry in the OPTIONS section of '
'git-push(1) for other possibilities.'))
+elif command == 'reset':
+subparser.add_argument(
+'-f', '--force', action='store_true',
+help='reset a large fraction of tags.')
args = parser.parse_args()
diff --git a/test/T850-git.sh b/test/T850-git.sh
index 55cec78a..ae6e7a03 100755
--- a/test/T850-git.sh
+++ b/test/T850-git.sh
@@ -213,6 +213,51 @@ cat < EXPECTED
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "reset"
+notmuch git -C reset.git -p '' clone remote.git
+notmuch git -C reset.git checkout --force
+notmuch tag +test4 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C remote.git commit --force
+notmuch tag -test4 id:20091117190054.gu3...@dottiness.seas.harvard.edu
+notmuch git -C reset.git fetch
+notmuch git -C reset.git reset
+notmuch git -C reset.git checkout --force
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu | grep -v
'^#' > OUTPUT
+cat < EXPECTED
++inbox +signed +test2 +test3 +test4 +unread --
id:20091117190054.gu3...@dottiness.seas.harvard.edu
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reset (require force for large change)"
+notmuch git -C reset2.git -p '' clone remote.git
+notmuch git -C reset2.git checkout --force
+notmuch tag +test5 '*'
+notmuch git -C remote.git commit --force
+notmuch tag -test5 '*'
+notmuch git -C reset2.git fetch
+test_expect_code 1 "notmuch git -C reset2.git -l debug reset"
+
+test_begin_subtest "reset (don't require force for large change to one
message)"
+notmuch git -C reset3.git -p '' clone remote.git
+notmuch git -C reset3.git checkout --force
+notmuch dump id:20091117190054.gu3...@dottiness.seas.harvard.edu > BEFORE
+for tag in $(seq 1 100); do
+notmuch tag +$tag id:20091117190054.gu3...@dottiness.seas.harvard.edu
+done
+notmuch git -C remote.git commit --force
+notmuch restore < BEFORE
+notmuch git -C reset3.git fetch
+test_expect_code 0 "notmuch git -C reset3.git -l debug reset"
+
+test_begin_subtest "reset --force"
+notmuch git -C reset4.git -p '' clone remote.git
Re: Reimagining notmuch-git/nmbug
Felipe Contreras writes: > Hi, > > I noticed you promoted notmuch-git as a user tool to toy around with it. > > Very quickly I realized that most of what it does is something I've > been working on for at least 10 years: making git work with other > tools. > > I presume you haven't heard of git remote-helpers [1], because they do > precisely what notmuch-git is trying to do. > > As a proof of concept I created a remote helper for notmuch [2]. If > you have this script (`git-remote-nm`) anywhere in your path, git will > interpret URLs prefixed with "nm::" as notmuch transports, and you can > do: > > git clone nm::$HOME/mail I'm intrigued (and indeed I hadn't really thought about the degree to which we were re-inventing git-fast-import and friends); however so far my experiments did not get far enough to say anything conclusive. I tried your script with the bindings from master (554690) but it does not seem to like my split configuration, where the database lives in ~/.local/share/share/notmuch/default/xapian. $ git clone nm::/home/bremner/Maildir Cloning into 'Maildir'... /home/bremner/.config/scripts/git-remote-nm:164:in `initialize': failed to read/write file (Notmuch::FileError) from /home/bremner/.config/scripts/git-remote-nm:164:in `new' from /home/bremner/.config/scripts/git-remote-nm:164:in `' If I make a fake .notmuch directory, then it seems to work. I'm not sure if this is an issue with the bindings or with the script. Conceptually there is also the question of how to handle split configurations as a URL. Performance-wise the initial clone seems pretty slow. For my 600k messages I have been waiting a while now. htop tells me that git-fast-import has about 45 minutes of CPU time at this point. This machine is not that fast, but for comparison an initial (i.e. fresh repo, no caching) "notmuch git commit" takes about 15-20s. If you need a larger corpus of messages to play with, the notmuch performance suite includes about 400k messages, and running T00-new.sh will build a notmuch database that you can clone. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: v2 Index some attachements as text
David Bremner writes: > This obsoletes the series starting with > > id:20220903232839.1473915-2-da...@tethera.net > > Compared to that series this is rebased against master, it has some > more tests (including both positive and negative tests) and it > documents the non-anchoredness of the involved regex search. > I have applied this series to master. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 0/3] ruby: get rid of Tags object
Felipe Contreras writes: > We don't need a Tags enumerable object only for a small number of strings, we > can just get them directly. > > This fixes an interaction problem where we might request two tags iterables > from the same message: > > tags_0 = notmuch_message_get_tags(message); I have applied this series to master. By the way I noticed that the formatting of the old bindings code does not match the output of uncrustify -c ./devel/uncrustify/cfg $file. This is not a serious problem, but it does make the review process a bit noisy (since my clumsy script reformats every file touched by a given commit). Do you have any objection to my just going through and reformatting the bindings code with uncrustify at some point? It can be done as patches also, but it's a bit silly because there can be many small diffs. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/1] lib: replace some uses of Query::MatchAll with a thread-safe alternative
Kevin Boulain writes: > This replaces two instances of Xapian::Query::MatchAll with the > equivalent but thread-safe alternative Xapian::Query(std::string()). > Xapian::Query::MatchAll maintains an internal pointer to a refcounted > Xapian::Internal::QueryTerm. > Applied to master, thanks, and sorry for the delay. By the way I see sfsexp has merged your related PR last week. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 0/2] ruby: database open improvements
Felipe Contreras writes: > Essentially we want to do notmuch_database_open_with_config() and load the > default database. > Applied to master. Thanks for this change, both are things I'd wanted to fix. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 5/5] test: add test for notmuch_message_remove_all_properties_with_prefix
Kevin Boulain writes: > It wasn't covered, though it shares most of its implementation with > notmuch_message_remove_all_properties. I have applied this series to master, with one commit added in the middle commit 336334996750240608d5f29ed5dd8e40a69c4d79 Author: David Bremner Date: Thu Mar 30 07:56:17 2023 -0300 test: reveal notmuch_message_remove_all_properties as broken Close and re-open the database to show that the removal is not committed to the database. I also had to manually deal with a merge conflict for the last patch, so you might want to double check that. In my next life, I will make more liberal use of backup_database / restore_database, to make the tests in this file less order dependent. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v3 1/2] test: uncaught exception when editing properties of a removed message
Kevin Boulain writes: > These two functions don't fail gracefully when editing a removed > message: > BROKEN edit property on removed message without uncaught exception > --- T610-message-property.20.EXPECTED 2023-02-27 11:33:25.792764376 > + > +++ T610-message-property.20.OUTPUT 2023-02-27 11:33:25.793764381 > + > @@ -1,2 +1,3 @@ > == stdout == > == stderr == > +terminate called after throwing an instance of > 'Xapian::DocNotFoundError' > Series applied to master, thanks! d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/2] lib/message-property: sync removed properties to the database
Kevin Boulain writes: > > Yeah, talloc's documentation confirms that when the context is free'd > every child is also free'd. Not quite sure what style is preferred > (explicit or implicit, implicit sure leads to nicer code here). In general implicit de-allocation is fine. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/2] lib/message-property: sync removed properties to the database
Kevin Boulain writes: > On 2023-03-03 at 00:39 +02, Tomi Ollila wrote: >> Somehow testkey1 = testvalue1 disappeared from the test code (which is >> probably expected -- perhaps the commit message of the *change* 1/2 >> tried to point to that ;D) > > Yes, that proves notmuch_message_remove_all_properties is broken without > the patch. The eponymous test gave the impression the property had been > removed (stdout is empty) but I believe this is only due to > _notmuch_message_invalidate_metadata. Now the "dump message properties" > test doesn't list it anymore, which is what I expect. It would be nice to structure this in terms of a known broken test (perhaps modify the existing one to reopen the database and dump the properties) that is then fixed by patch adding the sync. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v2 1/2] test: uncaught exception when editing properties of a removed message
Kevin Boulain writes: > +== stderr == > +A Xapian exception occurred at message-property.cc:XXX: No termlist for > document 54 > +EOF > +test_expect_equal_file EXPECTED OUTPUT > + > +add_email_corpus Hi Kevin; Thanks for the speedy update. I hate to do this but I think the message is now _too_ specific. Goldilocks and the three tests I guess. In particular, is there some reason to think that the number 54 is stable here? I'm worried about the test being flaky if there is some Xapian related change. Maybe just sed the number away? d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 0/3] ruby: get rid of FileNames object
Felipe Contreras writes: > We don't need a FileNames enumerable object only for a small number of > strings, > we can just get them directly. > > This iterator is meant to be transient and works only once, so we better just > iterate it once. > > This is the same approach I took with the Tags object, I was waiting for > feedback on that approach but since there isn't any and there's no reason this > shouldn't work, here's the same for Filenames. Hi Felipe; I still haven't had a chance to look at the proposed changes, but I did wonder what your plan was as far as migration. Usually with the library itself we try to provide fairly smooth upgrades. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 2/2] lib/message-property: catch xapian exceptions
Kevin Boulain writes:
> Since libnotmuch exposes a C interface there's no way for clients to
> diff --git a/test/T610-message-property.sh b/test/T610-message-property.sh
> index 944e1810..f7cabe4d 100755
> --- a/test/T610-message-property.sh
> +++ b/test/T610-message-property.sh
> @@ -363,7 +363,6 @@ EOF
Overall this looks good, but I think the tests are a little cryptic as written.
> test_expect_equal_file /dev/null OUTPUT
>
> test_begin_subtest "edit property on removed message without uncaught
> exception"
> -test_subtest_known_broken
> cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
> EXPECT0(notmuch_database_remove_message (db, notmuch_message_get_filename
> (message)));
> EXPECT0(notmuch_message_remove_property (message, "example", "example"));
> @@ -371,11 +370,11 @@ EOF
> cat <<'EOF' >EXPECTED
> == stdout ==
> == stderr ==
> +line 30: 3
> EOF
> test_expect_equal_file EXPECTED OUTPUT
In general I think it's better if the output of a test does not change
when marked non-broken. More importantly, in this case it means that the
call to notmuch_message_remove_property is not actually returning 0 any
more. So that output is actually the EXPECT0 assertion failing. I agree
it should print something more helpful (and I understand these
assumptions are not documented). So you should probably test that the
status is not success, and ideally print the message. You can see some
examples in T560-lib-error.sh, in particular in the created file c_tail
which handles checking the error code and printing the message.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Help in Emacs config
Atanas Janackovski writes: > I have justed strated using `notmuch' in Emacs (doom configuration), and I > love > it! However, there are a couple of things I'd like help with. > > 1. I have a number of accounts that I use. How do I configure my config to > save >draft messages based on the "From" header? There currently isn't any automated filing rules in the way that notmuch-fcc-dirs works. It could be done, if there was sufficient interest (and somebody did it!), > 2. From time to time, I want to view `html' content in my browser. Does anyone >have any tips on how I would do this? I have tried the below but this did > not Not sure if this is what you are after, but you can choose ". o" and give the name of a browser. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: crash on incoming email
Antoine Beaupré writes: > > Still, one has to wonder: if emacs can just dump core if someone messes > with my eln-cache, that's... bad, no? Yes, it is bad, but the native compilation functionality is new, so hopefully things will get better over time. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/1] lib/notmuch: update example
Kevin Boulain writes: > Likely missed in 86cbd215e, when notmuch_query_search_messages_st was > renamed to notmuch_query_search_messages. > --- Applied to master, thanks. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [DRAFT 1/1] lib: replace some uses of Query::MatchAll with a thread-safe alternative
Kevin Boulain writes:
> This is what can be expected from the tests when they fail:
>== stderr ==
> +==
> +WARNING: ThreadSanitizer: data race (pid=207931)
> + Read of size 1 at 0x7b1001a0 by thread T2:
> +#0 memcpy (libtsan.so.2+0x62506)
> +#1 void std::__cxx11::basic_string,
> std::allocator >::_M_construct(char*, char*,
> std::forward_iterator_tag) [clone .isra.0] (libxapian.so.30+0x872b3)
> +
> + Previous write of size 8 at 0x7b1001a0 by thread T1:
> +#0 operator new(unsigned long) (libtsan.so.2+0x8ba83)
> +#1 Xapian::Query::Query(std::__cxx11::basic_string std::char_traits, std::allocator > const&, unsigned int, unsigned
> int) (libxapian.so.30+0x855cd)
> ...
I don't know what the difference between our environments is, but these
tests are failing in glib for me.
+==
+WARNING: ThreadSanitizer: data race (pid=392122)
+ Atomic read of size 1 at 0x7b1025c0 by thread T2:
+#0 pthread_rwlock_rdlock
../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1406
(libtsan.so.2+0x447f1)
+#1 g_rw_lock_reader_lock (libglib-2.0.so.0+0xa8b84)
+
+ Previous write of size 8 at 0x7b1025c0 by thread T1:
+#0 malloc
../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:647
(libtsan.so.2+0x3ebb8)
+#1 (libglib-2.0.so.0+0xa880a)
+
...
FWIW I have libglib2.0-0 version 2.74.5-1 on Debian.
> +printf "C compiler supports address sanitizer... "
s/address/thread/
> +test_cmdline="${CC} ${CFLAGS} ${CPPFLAGS} -fsanitize=thread minimal.c
> ${LDFLAGS} -o minimal"
> +if ${test_cmdline} >/dev/null 2>&1 && ./minimal
> @@ -186,7 +186,7 @@ _notmuch_query_string_to_xapian_query (notmuch_database_t
> *notmuch,
> {
> try {
> if (query_string == "" || query_string == "*") {
> - output = Xapian::Query::MatchAll;
> + output = Xapian::Query(std::string());
I think this probably deserves a brief comment like "use thread-safe
alternative to Query::MatchAll.
That is why I was thinking a preprocessor macro / inline-function might
be appropriate, to only need that comment in one place.
> +test_begin_subtest "create"
> +test_C ${MAIL_DIR} ${MAIL_DIR}-2 < +#include
> +#include
> +#include
> +#include
I guess we will find out if any changes are needed to test harness to
portably support pthreads. BSD and macOS are the usual places that have
slightly different expectations with respect to include files and
libraries; I don't know if that applies here.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [DRAFT 0/1] lib: replace some uses of Query::MatchAll with a thread-safe alternative
Kevin Boulain writes: > > Thoughts? Should clients using the library gate every function call to > Notmuch behind a lock instead? Ideally that should not be necessary. On the other hand, it may take some time to fix issues with concurrent access. > I can also start a discussion in sfsexp's issue tracker. Yes please. I guess in worst case we can implment come kind of concurrency control to sfsexp (wrap calls in libnotmuch inside some mutex), although that sounds pretty far from ideal. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/1] lib/notmuch: update example
Kevin Boulain writes:
> Likely missed in 86cbd215e, when notmuch_query_search_messages_st was
> renamed to notmuch_query_search_messages.
> ---
> lib/notmuch.h | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index ce375c04..73e246e0 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -1172,7 +1172,8 @@ notmuch_query_search_threads_st (notmuch_query_t
> *query, notmuch_threads_t **out
> *
> * query = notmuch_query_create (database, query_string);
> *
> - * for (messages = notmuch_query_search_messages (query);
> + * notmuch_query_search_messages (query, );
> + * for (;
> * notmuch_messages_valid (messages);
> * notmuch_messages_move_to_next (messages))
> * {
Thanks for the documentation update, but I don't think we should
encourage people to ignore the status code. Perhaps something like
if (notmuch_query_search_messages (query, ))
return EXIT_FAILURE;
for (;
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: notmuch-emacs inline image display broken in emacs-29
David Bremner writes: > Alex Murray writes: > > I tried replacing indent-rigidly with a simplified version the and that > leaves the image inserted, but unfortunately doesn't indent it > properly. If you want to play with it, my half working patch is > attached. > Actually, the current implementation using indent-rigidly doesn't indent inlined images properly either, so maybe that is a way forward. I remember from last time I considered globally replacing our use of indent-rigidly [1] there were a few issues to be dealt with, but maybe this is a reasonable approach. [1]: https://nmbug.notmuchmail.org/nmweb/show/20211214121726.2631714-1-da...@tethera.net ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: notmuch-emacs inline image display broken in emacs-29
Alex Murray writes: > Hi folks, > > I have noticed that inline image display doesn't work anymore in > emacs-29 / git master and have tracked it down to a change to the way > mm-inline-image inserts an image into the buffer. > > In > https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=d2186160a9e978960c0f96bc3b4fc65b5affc170 > mm-inline-image was changed to use insert-image instead of > put-image. This inserts the image into the buffer using a single space > as the string contents with a display property set as the image contents > so that the single space is not shown itself but instead the image is > shown in its place. > > I had thought the culprit then was the use of > notmuch-wash-elide-blank-lines within the default value of > notmuch-show-insert-text/plain-hook - assuming that since the line > containing the image is now just a single space, it then gets deleted > from the buffer and the image is never shown. However, even after > setting notmuch-show-insert-text/plain-hook to nil, images are still not > shown inline by notmuch. > > However, if I locally redefine mm-inline-image to use say a period "." > as the string argument to insert-image then the image appears as > expected. > > Any thoughts on what may be happening here and how best to resolve this? > The problem seems to be the call to indent-rigidly in notmuch-show-lazy-part. The image is actually inserted OK, but then apparently deleted by indent-rigidly (probably because it looks like whitespace). I tried replacing indent-rigidly with a simplified version the and that leaves the image inserted, but unfortunately doesn't indent it properly. If you want to play with it, my half working patch is attached. >From 484f78e11d7f520b76b30b7d92aec15ff71f215f Mon Sep 17 00:00:00 2001 From: David Bremner Date: Sat, 25 Feb 2023 09:12:57 -0400 Subject: [PATCH] WIP: replace use of indent-rigidly in notmuch-show-lazy --- emacs/notmuch-show.el | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el index 36cce619..522ba1fd 100644 --- a/emacs/notmuch-show.el +++ b/emacs/notmuch-show.el @@ -324,6 +324,25 @@ position of the message in the thread." ;;; Utilities +;; force all indentation to be spaces at BOL +;; Based on apply-on-rectangle, from rect.el +(defun notmuch--indent-rigidly (start end count) + (cond + ((zerop count) t) + ((< count 0) (indent-rigidly start end count)) + (t +(save-excursion + (let ((startpt (progn (goto-char start) (line-beginning-position))) + (endpt (progn (goto-char end) (line-end-position))) + (spaces (spaces-string count))) + (goto-char startpt) + (while + (progn + (insert spaces) + (cl-incf endpt count) + (and (zerop (forward-line 1)) (bolp) + (<= (point) endpt) + (defmacro with-current-notmuch-show-message ( body) "Evaluate body with current buffer set to the text of current message." `(save-excursion @@ -1024,7 +1043,7 @@ will return nil if the CID is unknown or cannot be retrieved." (narrow-to-region part-beg part-end) (delete-region part-beg part-end) (apply #'notmuch-show-insert-bodypart-internal part-args) - (indent-rigidly part-beg + (notmuch--indent-rigidly part-beg part-end (* notmuch-show-indent-messages-width depth))) (goto-char part-end) -- 2.39.1 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: crash on incoming email
Antoine Beaupré writes: > When I have the attached email in my `inbox` and hit ENTER on the email > (notmuch-search-show-thread here), Emacs crashes completely with: > > fév 22 11:57:10 angela emacs[112721]: > /home/anarcat/.emacs.d/eln-cache/28.2-8532cd27/tab-line-e55f541b-b1fd08d4.eln(F7461622d6c696e652d6175746f2d687363726f6c6c_tab_li> > fév 22 11:57:10 angela emacs[112721]: > /usr/bin/emacs(+0x1b811b)[0x55d1080b011b] > fév 22 11:57:10 angela emacs[112721]: > /home/anarcat/.emacs.d/eln-cache/28.2-8532cd27/tab-line-e55f541b-b1fd08d4.eln(F7461622d6c696e652d666f726d61742d74656d706c617465_> > fév 22 11:57:10 angela emacs[112721]: > /usr/bin/emacs(+0x1b811b)[0x55d1080b011b] > fév 22 11:57:10 angela emacs[112721]: > /home/anarcat/.emacs.d/eln-cache/28.2-8532cd27/tab-line-e55f541b-b1fd08d4.eln(F7461622d6c696e652d666f726d6174_tab_line_format_0+> I can't duplicate this here (at least not with emacs -q). I suspect we have pretty similar Debian environments, but maybe try moving your .emacs.d/eln-cache out of the way. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Thread safety?
Kevin Boulain writes: > The documentation of MatchNothing says it's thread-safe. > > If you'd like, I'm happy to submit a patch (likely very small, given the > limited number of occurrences) after I review a bit more the > documentation and the code (I'm just starting with Notmuch and Xapian so > that's probably the extent of what I can do). Sounds good. With apologies for the imposing table of contents, please also have a quick look at https://notmuchmail.org/contributing/ Probably the relevant bit is https://notmuchmail.org/contributing/#index5h2 I suspect a (deterministic) regression test is not really feasible. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs/tree: use two argument form of setq-local
David Bremner writes: > Apparently the macro setq-local only takes two arguments in Emacs 26.1 > --- > emacs/notmuch-tree.el | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el > index 14775d59..b58fa6a6 100644 > --- a/emacs/notmuch-tree.el > +++ b/emacs/notmuch-tree.el > @@ -1451,11 +1451,11 @@ notmuch-tree buffers, just set >(unless (derived-mode-p 'notmuch-tree-mode) > (user-error "notmuch-tree-outline-mode is only meaningful for notmuch > trees!")) >(if notmuch-tree-outline-mode > - (progn (setq-local outline-regexp "^[^\n]+" > - outline-level #'notmuch-tree-outline--level) > + (progn (setq-local outline-regexp "^[^\n]+") > + (setq-local outline-level #'notmuch-tree-outline--level) >(notmuch-tree-outline--set-visibility)) > -(setq-local outline-regexp (default-value 'outline-regexp) > - outline-level (default-value 'outline-level > +(setq-local outline-regexp (default-value 'outline-regexp)) > +(setq-local outline-level (default-value 'outline-level Applied to master. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Thread safety?
Kevin Boulain writes: > So, is Notmuch expected to be thread safe? There are some indications it > should be (e.g.: lib/init.cc has locking) but all instances of > Xapian::Query::MatchAll would have to be replaced. > Xapian states it's thread safe: > https://github.com/xapian/xapian-docsprint/blob/master/concepts/concurrency.rst > Hi Kevin; Thanks for tracking this down. I did try some experiments a while ago with multi-threaded indexing (which is where the locking you mentioned arose), but overall it isn't well tested (by me). It seems reasonable to try to provide the same level of thread safety as Xapian does. So I guess we should go ahead and replace all of the MatchAll objects. I'm guessing this might apply to Xapian::Query::MatchNothing as well. Probably preprocessor macros wrapping Xapian::Query(std::string()) and whatever the equivalent for MatchNothing is would make this easier to read. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH] emacs/tree: use two argument form of setq-local
Apparently the macro setq-local only takes two arguments in Emacs 26.1 --- emacs/notmuch-tree.el | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/emacs/notmuch-tree.el b/emacs/notmuch-tree.el index 14775d59..b58fa6a6 100644 --- a/emacs/notmuch-tree.el +++ b/emacs/notmuch-tree.el @@ -1451,11 +1451,11 @@ notmuch-tree buffers, just set (unless (derived-mode-p 'notmuch-tree-mode) (user-error "notmuch-tree-outline-mode is only meaningful for notmuch trees!")) (if notmuch-tree-outline-mode - (progn (setq-local outline-regexp "^[^\n]+" -outline-level #'notmuch-tree-outline--level) + (progn (setq-local outline-regexp "^[^\n]+") +(setq-local outline-level #'notmuch-tree-outline--level) (notmuch-tree-outline--set-visibility)) -(setq-local outline-regexp (default-value 'outline-regexp) - outline-level (default-value 'outline-level +(setq-local outline-regexp (default-value 'outline-regexp)) +(setq-localoutline-level (default-value 'outline-level ;;; _ -- 2.39.1 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: v2 of long filename diagnostics
David Bremner writes: > This one use a more reasonable error code. As far as showing up every > time, it does if I use --full-scan. That is the same (for me) as > non-email files. series applied to master. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v9] emacs: notmuch-tree-outline-mode
"Jose A. Ortega Ruiz" writes: > On Mon, Dec 26 2022, David Bremner wrote: > >> Amended by db: Copy docstring to manual and edit for presentation. Add >> two tests. > > Thanks for the grunt work, David. Just a comment on a sloppy comment of > mine below: > >> --- a/emacs/notmuch-tree.el >> +++ b/emacs/notmuch-tree.el >> @@ -1014,7 +1014,10 @@ unchanged ADDRESS if parsing fails." >> A message tree is another name for a single sub-thread: i.e., a >> message together with all its descendents." >>(let ((msg (car tree)) >> -(replies (cadr tree))) >> +(replies (cadr tree)) >> +;; outline level, computed from the message's depth and >> +;; wether or not it's the first message in the tree. > > typo: whether Applied to master, with that fix. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: emacs/notmuch hangs opening email
Jon Fineman writes: > When I try and open one specific email, I get a message of > "Fontifying..." and then the mouse pointer turns into a watch. > > If I C-G it breaks out and shows the email message with a red warning > at the top of "[some mark read tag changes may have failed]", but > since reading the message was interrupted some/many key mappings > aren't set, etc. > Is the message by chance SMIME signed? That can be fixed on the gpgsm level. Otherwise, the only thing I can think of is to M-x edebug-defun on notmuch-show--build-buffer, and step through it to see where it gets stuck. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
David Bremner writes: > Michael J Gruber writes: > >> >> Yes, the extra ones all are ghosts, and I slowly remember that they >> scared me in the past already ... >> >> These ghosts appear to be pretty common. It happens all the time that >> I am joined to an existing discussion thread where I do not have all >> references. > > I have about 8% ghost messages in my 730k messages. I don't think I have > any situation as extreme as you do with hundreds of ghost messages for a > small number of actual messages in thread. That turns out to be a lie, as I wrote below. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Michael J Gruber writes: > > Yes, the extra ones all are ghosts, and I slowly remember that they > scared me in the past already ... > > These ghosts appear to be pretty common. It happens all the time that > I am joined to an existing discussion thread where I do not have all > references. I have about 8% ghost messages in my 730k messages. I don't think I have any situation as extreme as you do with hundreds of ghost messages for a small number of actual messages in thread. If you would like to calculate the ratio for your mail store, you can run % xapian-delve -v -A Tghost ~/.local/share/notmuch/default/xapian % xapian-delve -v -A Tmail ~/.local/share/notmuch/default/xapian > I'd go as far as to say that counting ghosts as thread > members makes this useless for me. On the other hand, notmuch's own > count gets this right. And getting different counts is even more > confusing. The count shown in e.g. notmuch search is calculated after the query has been run, so it isn't easily usable as part of a query. Maybe there is a way to trade off some performance for less false positives. In principle we could do a query for each thread found by the current technique to postprocess the results. I can see that getting pretty slow if there are many results though. At least for the original motivation of looking for messages without replies counting ghost messages makes some sense. In general it also makes sense for finding large threads. I did the query '(thread (count 200 *))' on my mail store and most matches are genuinely large threads. A few are false positive like the one you describe. In my case it is easy to see where the ghosts come from, as the (spam) messages have hundreds of (presumably fictional) references. > >> 2) Do they have more than one G term? That suggests a bug somewhere. We >> actually have a test in the test suite [1] for that, but of course that is >> with a simple artificial database. > > No, they all have one. But their sheer number looks suspicious: those > 5 "real" e-mails have maybe 20 reference headers in total, and some of > them refer to some of those 5. Grepping the account store for those > references gives me around that number. Where do the 110 ghosts (90 > extra) come from which this thread points to? Still scared by them ... > we need ghost busters! The only information attached to a ghost message is the thread-id and the message-id. You can get a visual picture of the thread with the attached script. But that will probably just confirm what you did with grep. To see what is in the database, you can run % quest -btype:T -bthread:G -d mail/.notmuch/xapian "type:ghost and thread:0002" That gives you record numbers, that you can examine with xapian-delve -r. draw-thread Description: Binary data ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 5/6] WIP/test: (count ...) tests
The most interesting case is probably tags with a small number of
uses, since these could be typos or similar errors.
---
test/T083-sexpr-count.sh | 8
1 file changed, 8 insertions(+)
diff --git a/test/T083-sexpr-count.sh b/test/T083-sexpr-count.sh
index 858aa8bf..1be3f62d 100755
--- a/test/T083-sexpr-count.sh
+++ b/test/T083-sexpr-count.sh
@@ -133,4 +133,12 @@ id:87iqd9rn3l.fsf@vertex.dottedmag
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "messages with tags used by 4 messages"
+output=$(notmuch count --output=messages --query=sexp '(tag (count 4))')
+test_expect_equal "${output}" "4"
+
+test_begin_subtest "no tag is used less than 4 times"
+output=$(notmuch count --output=messages --query=sexp '(tag (count * 3))')
+test_expect_equal "${output}" "0"
+
test_done
--
2.39.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 4/6] WIP/test: pathname related tests
---
test/T083-sexpr-count.sh | 21 +
1 file changed, 21 insertions(+)
diff --git a/test/T083-sexpr-count.sh b/test/T083-sexpr-count.sh
index f3010d11..858aa8bf 100755
--- a/test/T083-sexpr-count.sh
+++ b/test/T083-sexpr-count.sh
@@ -112,4 +112,25 @@ notmuch@notmuchmail.org
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "attachment filenames with unique words"
+notmuch show --entire-thread=false --query=sexp '(attachment (count 1))' | \
+sed -n -e 's/, Content-type:.*$//' -e 's/.*Filename: //p' | sort > OUTPUT
+cat < EXPECTED
+0001-Deal-with-situation-where-sysconf-_SC_GETPW_R_SIZE_M.patch
+0001-Error-out-if-no-query-is-supplied-to-search-instead-.patch
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "messages in folders with several other messages"
+output=$(notmuch count --output=messages --query=sexp '(folder (count 28 *))')
+test_expect_equal "${output}" "28"
+
+test_begin_subtest "messages alone in a directory"
+notmuch search --output=messages --query=sexp '(path (count 1))' > OUTPUT
+cat < EXPECTED
+id:87lji4lx9v@yoom.home.cworth.org
+id:87iqd9rn3l.fsf@vertex.dottedmag
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.39.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 2/6] WIP/lib: support count modifier in sexp queries
In this initial commit, support all term based fields, but only
document/test the thread size feature.
---
lib/parse-sexp.cc| 65 ++--
test/T083-sexpr-count.sh | 30 +++
2 files changed, 79 insertions(+), 16 deletions(-)
create mode 100755 test/T083-sexpr-count.sh
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 9cadbc13..efe564c7 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -34,6 +34,8 @@ typedef enum {
SEXP_FLAG_ORPHAN = 1 << 8,
SEXP_FLAG_RANGE= 1 << 9,
SEXP_FLAG_PATHNAME = 1 << 10,
+SEXP_FLAG_COUNT= 1 << 11,
+SEXP_FLAG_MODIFIER = 1 << 12,
} _sexp_flag_t;
/*
@@ -65,24 +67,28 @@ static _sexp_prefix_t prefixes[] =
{ "and",Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_NONE },
{ "attachment", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_EXPAND |
SEXP_FLAG_COUNT},
{ "body", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_FIELD },
{ "date", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
- SEXP_FLAG_RANGE },
+ SEXP_FLAG_FIELD | SEXP_FLAG_RANGE },
+{ "count", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
+ SEXP_FLAG_MODIFIER | SEXP_FLAG_RANGE },
{ "from", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEX |
SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND
+ | SEXP_FLAG_COUNT },
{ "folder", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND |
- SEXP_FLAG_PATHNAME },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX
+ | SEXP_FLAG_EXPAND | SEXP_FLAG_PATHNAME | SEXP_FLAG_COUNT },
{ "id", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX },
{ "infix", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
SEXP_FLAG_SINGLE | SEXP_FLAG_ORPHAN },
{ "is", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
+ SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND | SEXP_FLAG_COUNT },
{ "lastmod", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
- SEXP_FLAG_RANGE },
+ SEXP_FLAG_FIELD | SEXP_FLAG_RANGE },
{ "matching", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_DO_EXPAND },
{ "mid",Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
@@ -97,9 +103,10 @@ static _sexp_prefix_t prefixes[] =
SEXP_FLAG_NONE },
{ "path", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX |
- SEXP_FLAG_PATHNAME },
+ SEXP_FLAG_PATHNAME | SEXP_FLAG_COUNT},
{ "property", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
+ SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND | SEXP_FLAG_COUNT },
{ "query", Xapian::Query::OP_INVALID,
Xapian::Query::MatchNothing,
SEXP_FLAG_SINGLE | SEXP_FLAG_ORPHAN },
{ "regex", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
@@ -109,13 +116,16 @@ static _sexp_prefix_t prefixes[] =
{ "starts-with",Xapian::Query::OP_WILDCARD,
Xapian::Query::MatchAll,
SEXP_FLAG_SINGLE },
{ "subject",Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEX |
SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND
+ | SEXP_FLAG_COUNT },
{ "tag",Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX
+ | SEXP_FLAG_EXPAND | SEXP_FLAG_COUNT},
{ "thread", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX |
[PATCH 3/6] WIP/test: (count ...) tests for to / from
--- test/T083-sexpr-count.sh | 85 1 file changed, 85 insertions(+) diff --git a/test/T083-sexpr-count.sh b/test/T083-sexpr-count.sh index e825ef3d..f3010d11 100755 --- a/test/T083-sexpr-count.sh +++ b/test/T083-sexpr-count.sh @@ -27,4 +27,89 @@ catOUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat OUTPUT +cat
[PATCH 6/6] WIP/tests: (count ...) tests for subject
---
test/T083-sexpr-count.sh | 8
1 file changed, 8 insertions(+)
diff --git a/test/T083-sexpr-count.sh b/test/T083-sexpr-count.sh
index 1be3f62d..b1c0a3ac 100755
--- a/test/T083-sexpr-count.sh
+++ b/test/T083-sexpr-count.sh
@@ -141,4 +141,12 @@ test_begin_subtest "no tag is used less than 4 times"
output=$(notmuch count --output=messages --query=sexp '(tag (count * 3))')
test_expect_equal "${output}" "0"
+test_begin_subtest "subjects with unique words"
+notmuch search --query=sexp '(and (from gusarov) (subject (count 1)))' |
notmuch_search_sanitize > OUTPUT
+cat
WIP: add a (count ...) modifier for sexp-queries
This updates and obsoletes the series at [1]. The backend that constructs queries is unmodified from that series, but the parser now allows the use of count modifier in several more places. Basically there is not much more code to maintain to do it for any field based on terms (notably not date), but the utility is a bit unclear in some cases. In (probably) decreasing order of compelling use case 1) (and (from bob) (thread (count 1))) # find messages from bob nobody replied to yet. 2) (and (subject init-systems) (thread (count 200 *))) # find me a mega thread on some topic 3) (and (to bremner) (from (count (2 * # find people that sent me at least 2 messages. 4) (tag (count 1)) # find tags used only once 5) (path (count 1)) # find messages alone in a directory 6) (subject (count 1)) # find words used only once in subjects [1]: id:20230213122631.2088558-1-da...@tethera.net ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 1/6] WIP/lib: add count query backend
---
lib/Makefile.local | 3 +-
lib/count-query.cc | 62 ++
lib/database-private.h | 6
3 files changed, 70 insertions(+), 1 deletion(-)
create mode 100644 lib/count-query.cc
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 4e766305..cc646946 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -66,7 +66,8 @@ libnotmuch_cxx_srcs = \
$(dir)/init.cc \
$(dir)/parse-sexp.cc\
$(dir)/sexp-fp.cc \
- $(dir)/lastmod-fp.cc
+ $(dir)/lastmod-fp.cc\
+ $(dir)/count-query.cc
libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/count-query.cc b/lib/count-query.cc
new file mode 100644
index ..5d258880
--- /dev/null
+++ b/lib/count-query.cc
@@ -0,0 +1,62 @@
+/* count-query.cc - generate queries for terms on few / many messages.
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2023 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see https://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner
+ */
+
+#include "database-private.h"
+
+notmuch_status_t
+_notmuch_count_strings_to_query (notmuch_database_t *notmuch, std::string
field,
+const std::string , const std::string ,
+Xapian::Query , std::string )
+{
+
+long from_idx = 0, to_idx = LONG_MAX;
+std::string term_prefix = _find_prefix (field.c_str ());
+std::vector terms;
+
+if (! from.empty ()) {
+ try {
+ from_idx = std::stol(from);
+ } catch (std::logic_error ) {
+ msg = "bad 'from' count: '" + from + "'";
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+}
+
+if (! to.empty ()) {
+ try {
+ to_idx = std::stod(to);
+ } catch (std::logic_error ) {
+ msg = "bad 'to' count: '" + to + "'";
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+}
+
+for (Xapian::TermIterator it = notmuch->xapian_db->allterms_begin
(term_prefix);
+it != notmuch->xapian_db->allterms_end (); ++it) {
+ Xapian::doccount freq = it.get_termfreq();
+ if (from_idx <= freq && freq <= to_idx)
+ terms.push_back (*it);
+}
+
+output = Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end
());
+return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/database-private.h b/lib/database-private.h
index b9be4e22..ba96a93c 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -387,5 +387,11 @@ notmuch_status_t
_notmuch_lastmod_strings_to_query (notmuch_database_t *notmuch,
const std::string , const std::string
,
Xapian::Query , std::string );
+
+/* count-query.cc */
+notmuch_status_t
+_notmuch_count_strings_to_query (notmuch_database_t *notmuch, std::string
field,
+const std::string , const std::string ,
+Xapian::Query , std::string );
#endif
#endif
--
2.39.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Michael J Gruber writes: > That is really weird: > ``` > xapian-delve -t G00021229 . > Posting List for term 'G00021229' (termfreq 115, collfreq 0, > wdf_max 0): 146259 ... > ``` > with 115 record numbers, all different. > Doing `xapian-delve -1r` for each of them and grepping for the G-lines > gives 115 times that correct thread id. > Grepping for the Q-lines and notmuch-searching for the message ids > gives only 5 results (the expected ones). Apparantly, there are bogus > mail records which that thread points to. 1) Do those "bogus" records have a "Tghost" term? That would be for messages that are known via references, but not actually in the local database. This is a bug / feature of the current implementation, it counts all messages known, whether or not local copies exist. 2) Do they have more than one G term? That suggests a bug somewhere. We actually have a test in the test suite [1] for that, but of course that is with a simple artificial database. [1]: in T670-duplicate-mid.sh: db=$HOME/.local/share/notmuch/default/xapian for doc in $(xapian-delve -1 -t '' "$db" | grep '^[1-9]'); do xapian-delve -1 -r "$doc" "$db" | grep -c '^G' done > OUTPUT.raw sort -u < OUTPUT.raw > OUTPUT ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Michael J Gruber writes: > > It has 5, as confirmed by the search output and that of `notmuch > count`. But it is matched by `count 115`. > `xapian-check` is happy. (There used to be some issue with additional > thread entries at some point.) > > Michael A simple test to try is % xapian-delve -t G00021229 \ ~/.local/share/notmuch/default/xapian adjusting your database path as needed. If that says "termfreq 115", then something is broken (or at least confusing) about your database (possibly related to the previous issues with threading). In that case I'm curious if there are 115 distinct record numbers. You can find all of the thread-ids attached to a given message with % xapian-delve -1r 267585 ~/.local/share/notmuch/default/xapian | grep ^G where 267585 is an example record number in my database. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Proof of concept for counting messages in thread
Michael J Gruber writes: > I am getting a few surprising matches, e.g. > ``` > notmuch search --query=sexp '(thread (count 115)))' > thread:00021229 2021-05-17 [5/5] Michael J Gruber ... redacted > notmuch count --exclude=false thread:00021229 > 5 > ``` > It could be some database issues, of course. Or me misunderstanding something > :) Hmm. I don't see any strange matches for that particular query, just a thread that actually has 115 messages. But there could also be bugs of course. Does xapin-check complain about your database? > > Patch 1/2 is crlf garbled, by the way. Applies cleanly after removing > the extra ^Ms. Hmm. Probably because of Content-Transfer-Encoding: 8bit I have a direct mailed copy that didn't go through mailman, and that looks OK. > > Michael ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 1/2] WIP/lib: add count query backend
---
lib/Makefile.local | 3 +-
lib/count-query.cc | 62 ++
lib/database-private.h | 6
3 files changed, 70 insertions(+), 1 deletion(-)
create mode 100644 lib/count-query.cc
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 4e766305..cc646946 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -66,7 +66,8 @@ libnotmuch_cxx_srcs = \
$(dir)/init.cc \
$(dir)/parse-sexp.cc\
$(dir)/sexp-fp.cc \
- $(dir)/lastmod-fp.cc
+ $(dir)/lastmod-fp.cc\
+ $(dir)/count-query.cc
libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/count-query.cc b/lib/count-query.cc
new file mode 100644
index ..5d258880
--- /dev/null
+++ b/lib/count-query.cc
@@ -0,0 +1,62 @@
+/* count-query.cc - generate queries for terms on few / many messages.
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2023 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see https://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner
+ */
+
+#include "database-private.h"
+
+notmuch_status_t
+_notmuch_count_strings_to_query (notmuch_database_t *notmuch, std::string
field,
+const std::string , const std::string ,
+Xapian::Query , std::string )
+{
+
+long from_idx = 0, to_idx = LONG_MAX;
+std::string term_prefix = _find_prefix (field.c_str ());
+std::vector terms;
+
+if (! from.empty ()) {
+ try {
+ from_idx = std::stol(from);
+ } catch (std::logic_error ) {
+ msg = "bad 'from' count: '" + from + "'";
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+}
+
+if (! to.empty ()) {
+ try {
+ to_idx = std::stod(to);
+ } catch (std::logic_error ) {
+ msg = "bad 'to' count: '" + to + "'";
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+}
+
+for (Xapian::TermIterator it = notmuch->xapian_db->allterms_begin
(term_prefix);
+it != notmuch->xapian_db->allterms_end (); ++it) {
+ Xapian::doccount freq = it.get_termfreq();
+ if (from_idx <= freq && freq <= to_idx)
+ terms.push_back (*it);
+}
+
+output = Xapian::Query (Xapian::Query::OP_OR, terms.begin (), terms.end
());
+return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/database-private.h b/lib/database-private.h
index b9be4e22..ba96a93c 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -387,5 +387,11 @@ notmuch_status_t
_notmuch_lastmod_strings_to_query (notmuch_database_t *notmuch,
const std::string , const std::string
,
Xapian::Query , std::string );
+
+/* count-query.cc */
+notmuch_status_t
+_notmuch_count_strings_to_query (notmuch_database_t *notmuch, std::string
field,
+const std::string , const std::string ,
+Xapian::Query , std::string );
#endif
#endif
--
2.39.1
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Proof of concept for counting messages in thread
So for this only supports counting messages in threads, and the sexp based query parser. It seems useful to expand it to other fields (from, e.g.). I'm not sure how motivated I am to shim this into the infix query parser, but we will see how it goes. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 2/2] WIP: support thread count queries
---
lib/parse-sexp.cc | 35 ---
test/T081-sexpr-search.sh | 6 ++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc
index 9cadbc13..1faa9023 100644
--- a/lib/parse-sexp.cc
+++ b/lib/parse-sexp.cc
@@ -34,6 +34,8 @@ typedef enum {
SEXP_FLAG_ORPHAN = 1 << 8,
SEXP_FLAG_RANGE= 1 << 9,
SEXP_FLAG_PATHNAME = 1 << 10,
+SEXP_FLAG_COUNT= 1 << 11,
+SEXP_FLAG_MODIFIER = 1 << 12,
} _sexp_flag_t;
/*
@@ -70,6 +72,8 @@ static _sexp_prefix_t prefixes[] =
SEXP_FLAG_FIELD },
{ "date", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
SEXP_FLAG_RANGE },
+{ "count", Xapian::Query::OP_INVALID,
Xapian::Query::MatchAll,
+ SEXP_FLAG_RANGE | SEXP_FLAG_MODIFIER },
{ "from", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_REGEX |
SEXP_FLAG_EXPAND },
{ "folder", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
@@ -113,7 +117,8 @@ static _sexp_prefix_t prefixes[] =
{ "tag",Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
{ "thread", Xapian::Query::OP_OR,
Xapian::Query::MatchNothing,
- SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX | SEXP_FLAG_EXPAND },
+ SEXP_FLAG_FIELD | SEXP_FLAG_BOOLEAN | SEXP_FLAG_WILDCARD |
SEXP_FLAG_REGEX |
+ SEXP_FLAG_EXPAND | SEXP_FLAG_COUNT },
{ "to", Xapian::Query::OP_AND,
Xapian::Query::MatchAll,
SEXP_FLAG_FIELD | SEXP_FLAG_WILDCARD | SEXP_FLAG_EXPAND },
{ }
@@ -513,6 +518,7 @@ _sexp_expand_param (notmuch_database_t *notmuch, const
_sexp_prefix_t *parent,
static notmuch_status_t
_sexp_parse_range (notmuch_database_t *notmuch, const _sexp_prefix_t *prefix,
+ const _sexp_prefix_t *parent,
const sexp_t *sx, Xapian::Query )
{
const char *from, *to;
@@ -552,6 +558,27 @@ _sexp_parse_range (notmuch_database_t *notmuch, const
_sexp_prefix_t *prefix,
to = "";
}
+if (strcmp (prefix->name, "count") == 0) {
+ notmuch_status_t status;
+ if (! parent) {
+ _notmuch_database_log (notmuch, "illegal '%s' outside field\n",
+ prefix->name);
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+ if (! (parent->flags & SEXP_FLAG_COUNT)) {
+ _notmuch_database_log (notmuch, "'%s' not supported in field
'%s'\n",
+ prefix->name, parent->name);
+ return NOTMUCH_STATUS_BAD_QUERY_SYNTAX;
+ }
+
+ status = _notmuch_count_strings_to_query (notmuch, parent->name, from,
to, output, msg);
+ if (status) {
+ if (! msg.empty ())
+ _notmuch_database_log (notmuch, "%s\n", msg.c_str ());
+ }
+ return status;
+}
+
if (strcmp (prefix->name, "date") == 0) {
notmuch_status_t status;
status = _notmuch_date_strings_to_query (NOTMUCH_VALUE_TIMESTAMP, from,
to, output, msg);
@@ -654,7 +681,9 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const
_sexp_prefix_t *parent
for (_sexp_prefix_t *prefix = prefixes; prefix && prefix->name; prefix++) {
if (strcmp (prefix->name, sx->list->val) == 0) {
- if (prefix->flags & (SEXP_FLAG_FIELD | SEXP_FLAG_RANGE)) {
+ if ((prefix->flags & (SEXP_FLAG_FIELD)) ||
+ ((prefix->flags & SEXP_FLAG_RANGE) &&
+! (prefix->flags & SEXP_FLAG_MODIFIER))) {
if (parent) {
_notmuch_database_log (notmuch, "nested field: '%s' inside
'%s'\n",
prefix->name, parent->name);
@@ -677,7 +706,7 @@ _sexp_to_xapian_query (notmuch_database_t *notmuch, const
_sexp_prefix_t *parent
}
if (prefix->flags & SEXP_FLAG_RANGE)
- return _sexp_parse_range (notmuch, prefix, sx->list->next,
output);
+ return _sexp_parse_range (notmuch, prefix, parent,
sx->list->next, output);
if (strcmp (prefix->name, "infix") == 0) {
return _sexp_parse_infix (notmuch, sx->list->next, output);
diff --git a/test/T081-sexpr-search.sh b/test/T081-sexpr-search.sh
index 0c7db9c2..2013fa5c 100755
--- a/test/T081-sexpr-search.sh
+++ b/test/T081-sexpr-search.sh
@@ -1318,5 +1318,11 @@ notmuch search subject:notmuch or List:notmuch |
notmuch_search_sanitize > EXPEC
notmuch search --query=sexp '(About notmuch)' | notmuch_search_sanitize >
OUTPUT
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "threads with one message"
+notmuch search --query=sexp '(and (from gusarov) (thread (count 1)))' |
notmuch_search_sanitize > OUTPUT
+cat
Re: Cannot find notmuch.el file
Vishal Chourasia writes: > Thanks! Installing the emacs-notmuch package from epel 8 worked. > Great, thanks for letting us know. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Cannot find notmuch.el file
Vishal Chourasia writes: > On 2/7/23 19:53, David Bremner wrote: >> Which distribution? At least on Debian based distributions, the emacs >> interface is packaged seperately as elpa-notmuch > RHEL Fedora has a separate emacs-notmuch package. Perhaps that also exists in RHEL? The page https://packages.fedoraproject.org/pkgs/notmuch/emacs-notmuch/ mentions EPEL 8 and EPEL 9 d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: Cannot find notmuch.el file
Vishal Chourasia writes: > Greetings, > > > where can I find the notmuch.el file when notmuch was installed from the > distribution's package repository? > > # notmuch --version > notmuch 0.35 Which distribution? At least on Debian based distributions, the emacs interface is packaged seperately as elpa-notmuch ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: emacs: keybindings in notmuch-show for copying the target of a link for a text/html part? (M-x shr-copy-url)
Daniel Kahn Gillmor writes: > it appears that by positioning the point somewhere within the link and > doing "M-x shr-copy-url" i can copy the target of the link to the kill > ring. > I think it can be added to the mode-map in emacs/notmuch-show.el. does > anyone have a preference about what keybinding to use for it? Or is > there some reason to not provide a default keybinding to this > functionality in notmuch-show mode at all? Having a key-binding seems reasonable enough. What about "c u" ? It's not exactly like the other "c" bindings of course, because it depends on the cursor position, but maybe that is less important than general mnemonic of 'c' for copy. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: ## error when running notmuch from emacsclient, but not from standalone emacs instance
ricardomart...@riseup.net writes: > Ok, so I tested this. I tried stopping the systemd unit of the emacs > server, opened an emacs instance, and started the server from within > emacs (M-x server-start ). Connected to this server in a new frame > ($ emacsclient -nc) and tried executing notmuch again (M-x notmuch > ). This time, it worked like it should. So, definitely an issue > with the systemd unit method of initialization there, although it only > executes /usr/bin/emacs --fg-daemon, so I'm not sure exactly is going on > here that's causing the problem. I don't have personal experience with running emacs from systemd, but I guess it could be worth trying M-: (executable-find "notmuch") That will verify that you are not being tripped up by multiple notmuch executables. Another thing to try is M-x notmuch-search tag:inbox This may give a more helpful error message (or it may work fine, indicating the problem is really with notmuch count, but the --batch option is almost 10 years old now, so I think that error checking probably needs to be updated. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: ## error when running notmuch from emacsclient, but not from standalone emacs instance
ricardomart...@riseup.net writes: > > When I try to run notmuch from the emacsclient instance (M-x notmuch > ), I get an error > "notmuch count --batch failed > Please check that the notmuch CLI is new enough to support `count > --batch'. In general we recommend running matching versions of > the CLI and emacs interface." > I agree it is strange strange to have problems only affecting emacsclient. The only thing I can think of off hand is that starting emacsclient does not reload the emacs initialization file, so make sure you run both experiments with a fresh restart of emacs. What is the output of M-x notmuch-version for both emacsclient and emacs? ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
ANNOUNCE: muchsync 7 released
I've just released version 7 of muchsync, a tool for pairwise synchronization of maildirs and notmuch tags across machines. There aren't any new features. However, I discovered that some of the sqlite queries were generating full table scans rather than simple lookups. The new version should be an order of magnitude faster when deleting large numbers of mail messages or reorganizing your mail directory structure. You can download muchsync from the usual place: https://www.muchsync.org/ Enjoy, David ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] configure: use pkg-config for gpgme
Daniel Kahn Gillmor writes: > Just a nudge on this patch: > > any chance we can have it applied in the notmuch tree? > > --dkg > Applied to master. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH] emacs: add global tag history
inwit writes: > Save a list of every tag change in the new variable notmuch-tag-history. > --- > Storing the full history of tags can prove useful for a) repeated tag > changes as in [0] and b) eventually logging and undoing tag changes. > > This is my first commit in elisp. I expect turbulences ahead. :) > > [0] > https://nmbug.notmuchmail.org/nmweb/show/CCSQ9HR3M748.2IRNNTHYR4A2M%40bisio > > emacs/notmuch-tag.el | 4 > 1 file changed, 4 insertions(+) Although this patch won't apply anymore, thanks for inspiring the work that lead to supporting undo. Unfortunately I stole the name "notmuch-tag-history" for a buffer local variable, so I guess someone interested in a global history would need to choose a different variable name. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH 1/4] test: mark some tests as broken when run as root.
Tomi Ollila writes: > > Good progress -- bash has: > > $ bash -c 'set' | grep UID > EUID=1001 > UID=1001 I made that change. > another question is whether test_subtest_broken_for_root > is good name. perhaps it is tolerable enough I couldn't think of anything better, but am happy to search and replace if someome comes up with something. Amended version applied to master. d ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v2 2/3] lib: parse index.as_text
We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path. As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.
---
lib/database-private.h | 4
lib/open.cc| 53 ++
2 files changed, 57 insertions(+)
diff --git a/lib/database-private.h b/lib/database-private.h
index b9be4e22..61232f1a 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -291,6 +291,10 @@ struct _notmuch_database {
/* Track what parameters were specified when opening */
notmuch_open_param_t params;
+
+/* list of regular expressions to check for text indexing */
+regex_t *index_as_text;
+size_t index_as_text_length;
};
/* Prior to database version 3, features were implied by the database
diff --git a/lib/open.cc b/lib/open.cc
index 67ff868c..54d1faf3 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -320,6 +320,8 @@ _alloc_notmuch (const char *database_path, const char
*config_path, const char *
notmuch->transaction_count = 0;
notmuch->transaction_threshold = 0;
notmuch->view = 1;
+notmuch->index_as_text = NULL;
+notmuch->index_as_text_length = 0;
notmuch->params = NOTMUCH_PARAM_NONE;
if (database_path)
@@ -427,6 +429,53 @@ _load_database_state (notmuch_database_t *notmuch)
notmuch, notmuch->xapian_db->get_uuid ().c_str ());
}
+/* XXX This should really be done lazily, but the error reporting path in the
indexing code
+ * would need to be redone to report any errors.
+ */
+notmuch_status_t
+_ensure_index_as_text (notmuch_database_t *notmuch, char **message)
+{
+int nregex = 0;
+regex_t *regexv = NULL;
+
+if (notmuch->index_as_text)
+ return NOTMUCH_STATUS_SUCCESS;
+
+for (notmuch_config_values_t *list = notmuch_config_get_values (notmuch,
+
NOTMUCH_CONFIG_INDEX_AS_TEXT);
+notmuch_config_values_valid (list);
+notmuch_config_values_move_to_next (list)) {
+ regex_t *new_regex;
+ int rerr;
+ const char *str = notmuch_config_values_get (list);
+ size_t len = strlen (str);
+
+ /* str must be non-empty, because n_c_get_values skips empty
+* strings */
+ assert (len > 0);
+
+ regexv = talloc_realloc (notmuch, regexv, regex_t, nregex + 1);
+ new_regex = [nregex];
+
+ rerr = regcomp (new_regex, str, REG_EXTENDED | REG_NOSUB);
+ if (rerr) {
+ size_t error_size = regerror (rerr, new_regex, NULL, 0);
+ char *error = (char *) talloc_size (str, error_size);
+
+ regerror (rerr, new_regex, error, error_size);
+ IGNORE_RESULT (asprintf (message, "Error in index.as_text: %s:
%s\n", error, str));
+
+ return NOTMUCH_STATUS_ILLEGAL_ARGUMENT;
+ }
+ nregex++;
+}
+
+notmuch->index_as_text = regexv;
+notmuch->index_as_text_length = nregex;
+
+return NOTMUCH_STATUS_SUCCESS;
+}
+
static notmuch_status_t
_finish_open (notmuch_database_t *notmuch,
const char *profile,
@@ -531,6 +580,10 @@ _finish_open (notmuch_database_t *notmuch,
if (status)
goto DONE;
+ status = _ensure_index_as_text (notmuch, );
+ if (status)
+ goto DONE;
+
autocommit_str = notmuch_config_get (notmuch,
NOTMUCH_CONFIG_AUTOCOMMIT);
if (unlikely (! autocommit_str)) {
INTERNAL_ERROR ("missing configuration for autocommit");
--
2.39.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v2 1/3] lib: add config key INDEX_AS_TEXT
Higher level processing as a list of regular expressions and
documentation will follow.
---
lib/config.cc| 3 +++
lib/notmuch.h| 1 +
test/T030-config.sh | 1 +
test/T055-path-config.sh | 1 +
test/T590-libconfig.sh | 5 +
5 files changed, 11 insertions(+)
diff --git a/lib/config.cc b/lib/config.cc
index 503a0c8b..2323860d 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -599,6 +599,8 @@ _notmuch_config_key_to_string (notmuch_config_key_t key)
return "database.autocommit";
case NOTMUCH_CONFIG_EXTRA_HEADERS:
return "show.extra_headers";
+case NOTMUCH_CONFIG_INDEX_AS_TEXT:
+ return "index.as_text";
default:
return NULL;
}
@@ -642,6 +644,7 @@ _notmuch_config_default (notmuch_database_t *notmuch,
notmuch_config_key_t key)
else
email = _get_email_from_passwd_file (notmuch);
return email;
+case NOTMUCH_CONFIG_INDEX_AS_TEXT:
case NOTMUCH_CONFIG_NEW_IGNORE:
return "";
case NOTMUCH_CONFIG_AUTOCOMMIT:
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 0b0540b1..935a8d59 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -2558,6 +2558,7 @@ typedef enum {
NOTMUCH_CONFIG_USER_NAME,
NOTMUCH_CONFIG_AUTOCOMMIT,
NOTMUCH_CONFIG_EXTRA_HEADERS,
+NOTMUCH_CONFIG_INDEX_AS_TEXT,
NOTMUCH_CONFIG_LAST
} notmuch_config_key_t;
diff --git a/test/T030-config.sh b/test/T030-config.sh
index 43bbce31..ea0b4012 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -57,6 +57,7 @@ database.mail_root=MAIL_DIR
database.path=MAIL_DIR
foo.list=this;is another;list value;
foo.string=this is another string value
+index.as_text=
maildir.synchronize_flags=true
new.ignore=
new.tags=unread;inbox
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index fe295324..efc79e8b 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -299,6 +299,7 @@ database.backup_dir
database.hook_dir
database.mail_root=MAIL_DIR
database.path
+index.as_text=
maildir.synchronize_flags=true
new.ignore=
new.tags=unread;inbox
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 26a1f033..9326ba3e 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -440,6 +440,7 @@ cat <<'EOF' >EXPECTED
10: 'USER_FULL_NAME'
11: '8000'
12: 'NULL'
+13: ''
== stderr ==
EOF
unset MAILDIR
@@ -725,6 +726,7 @@ test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "list by keys (ndlc)"
notmuch config set search.exclude_tags "foo;bar;fub"
notmuch config set new.ignore "sekrit_junk"
+notmuch config set index.as_text "text/"
cat c_head2 - c_tail <<'EOF' | test_C ${MAIL_DIR} %NULL% %NULL%
{
notmuch_config_key_t key;
@@ -751,6 +753,7 @@ cat <<'EOF' >EXPECTED
10: 'Notmuch Test Suite'
11: '8000'
12: 'NULL'
+13: 'text/'
== stderr ==
EOF
test_expect_equal_file EXPECTED OUTPUT
@@ -785,6 +788,7 @@ cat <<'EOF' >EXPECTED
10: 'USER_FULL_NAME'
11: '8000'
12: 'NULL'
+13: ''
== stderr ==
EOF
test_expect_equal_file EXPECTED OUTPUT.clean
@@ -856,6 +860,7 @@ database.backup_dir MAIL_DIR/.notmuch/backups
database.hook_dir MAIL_DIR/.notmuch/hooks
database.mail_root MAIL_DIR
database.path MAIL_DIR
+index.as_text text/
key with spaces value, with, spaces!
maildir.synchronize_flags true
new.ignore sekrit_junk
--
2.39.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH v2 3/3] lib: index attachments with mime types matching index.as_text
e "$0")/test-lib.sh || exit 1 + +add_email_corpus indexing +test_begin_subtest "empty as_text; skip text/x-diff" +messages=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain) +count=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz) +test_expect_equal "$messages,$count" "1,0" + +notmuch config set index.as_text "^text/" +add_email_corpus indexing + +test_begin_subtest "as_index is text/; find text/x-diff" +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "reindex with empty as_text, skips text/x-diff" +notmuch config set index.as_text +notmuch reindex '*' +messages=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain) +count=$(notmuch count id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz) +test_expect_equal "$messages,$count" "1,0" + +test_begin_subtest "reindex with empty as_text; skips application/pdf" +notmuch config set index.as_text +notmuch reindex '*' +gmessages=$(notmuch count id:871qo9p4tf@tethera.net) +count=$(notmuch count id:871qo9p4tf@tethera.net and body:not-really-PDF) +test_expect_equal "$messages,$count" "1,0" + +test_begin_subtest "reindex with as_text as text/; finds text/x-diff" +notmuch config set index.as_text "^text/" +notmuch reindex '*' +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "reindex with as_text as text/; skips application/pdf" +notmuch config set index.as_text "^text/" +notmuch config set index.as_text +notmuch reindex '*' +messages=$(notmuch count id:871qo9p4tf@tethera.net) +count=$(notmuch count id:871qo9p4tf@tethera.net and body:not-really-PDF) +test_expect_equal "$messages,$count" "1,0" + +test_begin_subtest "as_text has multiple regexes" +notmuch config set index.as_text "blahblah;^text/" +notmuch reindex '*' +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "as_text is non-anchored regex" +notmuch config set index.as_text "e.t/" +notmuch reindex '*' +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain > EXPECTED +notmuch search id:20200930101213.2m2pt3jrspvcrxfx@localhost.localdomain and ersatz > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "as_text is 'application/pdf'" +notmuch config set index.as_text "^application/pdf$" +notmuch reindex '*' +notmuch search id:871qo9p4tf@tethera.net > EXPECTED +notmuch search id:871qo9p4tf@tethera.net and '"not really PDF"' > OUTPUT +test_expect_equal_file_nonempty EXPECTED OUTPUT + +test_begin_subtest "as_text is bad regex" +notmuch config set index.as_text '[' +notmuch reindex '*' >& OUTPUT +cat< EXPECTED +Error in index.as_text: Invalid regular expression: [ +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_done diff --git a/test/corpora/indexing/fake-pdf:2,S b/test/corpora/indexing/fake-pdf:2,S new file mode 100644 index ..60a7a47f --- /dev/null +++ b/test/corpora/indexing/fake-pdf:2,S @@ -0,0 +1,11 @@ +From: David Bremner +To: exam...@example.com +Subject: attachment content type +Date: Thu, 05 Jan 2023 08:02:36 -0400 +Message-ID: <871qo9p4tf@tethera.net> +MIME-Version: 1.0 +Content-Type: application/pdf +Content-Disposition: attachment; filename=fake.pdf +Content-Transfer-Encoding: base64 + +dGhpcyBpcyBub3QgcmVhbGx5IFBERgo= \ No newline at end of file -- 2.39.0 ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
v2 Index some attachements as text
This obsoletes the series starting with id:20220903232839.1473915-2-da...@tethera.net Compared to that series this is rebased against master, it has some more tests (including both positive and negative tests) and it documents the non-anchoredness of the involved regex search. ___ notmuch mailing list -- notmuch@notmuchmail.org To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 1/4] test: mark some tests as broken when run as root.
File permission errors e.g., are hard to trigger as root.
---
test/T050-new.sh | 1 +
test/T150-tagging.sh | 1 +
test/test-lib.sh | 6 ++
3 files changed, 8 insertions(+)
diff --git a/test/T050-new.sh b/test/T050-new.sh
index cb67889c..240c1810 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -384,6 +384,7 @@ EOF
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "Xapian exception: read only files"
+test_subtest_broken_for_root
chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
output=$(NOTMUCH_NEW --debug 2>&1 | sed 's/: .*$//' )
chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 1a2fd77e..ac3f2539 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -320,6 +320,7 @@ test_begin_subtest "Tag name beginning with -"
test_expect_code 1 'notmuch tag +- One'
test_begin_subtest "Xapian exception: read only files"
+test_subtest_broken_for_root
chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
output=$(notmuch tag +something '*' 2>&1 | sed 's/: .*$//' )
chmod u+w ${MAIL_DIR}/.notmuch/xapian/*.*
diff --git a/test/test-lib.sh b/test/test-lib.sh
index eec5c5b4..c2b8cd8d 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -740,6 +740,12 @@ test_subtest_known_broken () {
test_subtest_known_broken_=t
}
+test_subtest_broken_for_root () {
+ if [ "$(id -u)" = "0" ]; then
+ test_subtest_known_broken_=t
+fi
+}
+
test_expect_success () {
exec 1>&6 2>&7 # Restore stdout and stderr
if [ -z "$inside_subtest" ]; then
--
2.39.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
[PATCH 4/4] debian: add autopkgtests
We generate output to stderr for BROKEN tests, which are not failures,
so tell the test runner not to fail because of output on stderr.
---
debian/tests/control | 13 +
test/T800-asan.sh| 8 ++--
test/defaults.sh | 2 +-
3 files changed, 20 insertions(+), 3 deletions(-)
create mode 100644 debian/tests/control
diff --git a/debian/tests/control b/debian/tests/control
new file mode 100644
index ..c8dc9ff7
--- /dev/null
+++ b/debian/tests/control
@@ -0,0 +1,13 @@
+Test-command: env NOTMUCH_TEST_INSTALLED=yes TERM=dumb ./test/notmuch-test
+Restrictions: allow-stderr
+Depends: @,
+ build-essential,
+ dtach,
+ emacs-nox,
+ gdb,
+ git,
+ gnupg,
+ gpgsm,
+ libtalloc-dev,
+ man,
+ xapian-tools
diff --git a/test/T800-asan.sh b/test/T800-asan.sh
index 1ddff0fe..658b88f3 100755
--- a/test/T800-asan.sh
+++ b/test/T800-asan.sh
@@ -2,17 +2,21 @@
test_description='run code with ASAN enabled against the library'
. $(dirname "$0")/test-lib.sh || exit 1
-if [ $NOTMUCH_HAVE_ASAN -ne 1 ]; then
+if [ "$NOTMUCH_HAVE_ASAN" -ne 1 ]; then
printf "Skipping due to missing ASAN support\n"
test_done
fi
+if [ -n "$LD_PRELOAD" ]; then
+printf "Skipping due to non-empty LD_PRELOAD\n"
+test_done
+fi
+
add_email_corpus
TEST_CFLAGS="-fsanitize=address"
test_begin_subtest "open and destroy"
-test_subtest_broken_for_installed
test_C ${MAIL_DIR} ${NOTMUCH_CONFIG} <
#include
diff --git a/test/defaults.sh b/test/defaults.sh
index 6327e43b..ee4ded83 100644
--- a/test/defaults.sh
+++ b/test/defaults.sh
@@ -2,4 +2,4 @@
NOTMUCH_PYTHON=${NOTMUCH_PYTHON-python3}
NOTMUCH_HAVE_MAN=${NOTMUCH_HAVE_MAN-1}
NOTMUCH_HAVE_SFSEXP=${NOTMUCH_HAVE_SFSEXP-1}
-NOTMUCH_HAVE_ASAN=${NOTMUCH_HAVE_ASAN-0}
+NOTMUCH_HAVE_ASAN=${NOTMUCH_HAVE_ASAN-1}
--
2.39.0
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
