[PATCH 2/2] n_m_remove_indexed_terms: reduce number of Xapian API calls.
Previously this functioned scanned every term attached to a given
Xapian document. It turns out we know how to read only the terms we
need to preserve (and we might have already done so). This commit
replaces many calls to Xapian::Document::remove_term with one call to
::clear_terms, and a (typically much smaller) number of calls to
::add_term. Roughly speaking this is based on the assumption that most
messages have more text than they have tags.
According to the performance test suite, this yields a roughly 40%
speedup on "notmuch reindex '*'"
---
lib/message.cc | 66 +-
1 file changed, 38 insertions(+), 28 deletions(-)
diff --git a/lib/message.cc b/lib/message.cc
index 6f2f6345..3e33d8b8 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -716,6 +716,8 @@ _notmuch_message_remove_terms (notmuch_message_t *message,
const char *prefix)
/* Remove all terms generated by indexing, i.e. not tags or
* properties, along with any automatic tags*/
+/* According to Xapian API docs, none of these calls throw
+ * exceptions */
notmuch_private_status_t
_notmuch_message_remove_indexed_terms (notmuch_message_t *message)
{
@@ -727,45 +729,53 @@ _notmuch_message_remove_indexed_terms (notmuch_message_t
*message)
tag_prefix = _find_prefix ("tag"),
type_prefix = _find_prefix ("type");
-for (i = message->doc.termlist_begin ();
-i != message->doc.termlist_end (); i++) {
+/* Make sure we have the data to restore to Xapian*/
+_notmuch_message_ensure_metadata (message,NULL);
- const std::string term = *i;
+/* Empirically, it turns out to be faster to remove all the terms,
+ * and add back the ones we want. */
+message->doc.clear_terms ();
+message->modified = true;
- if (term.compare (0, type_prefix.size (), type_prefix) == 0)
- continue;
+/* still a mail message */
+message->doc.add_term (type_prefix + "mail");
- if (term.compare (0, id_prefix.size (), id_prefix) == 0)
- continue;
+/* Put back message-id */
+message->doc.add_term (id_prefix + message->message_id);
- if (term.compare (0, property_prefix.size (), property_prefix) == 0)
- continue;
+/* Put back non-automatic tags */
+for (notmuch_tags_t *tags = notmuch_message_get_tags (message);
+notmuch_tags_valid (tags);
+notmuch_tags_move_to_next (tags)) {
- if (term.compare (0, tag_prefix.size (), tag_prefix) == 0 &&
- term.compare (1, strlen("encrypted"), "encrypted") != 0 &&
- term.compare (1, strlen("signed"), "signed") != 0 &&
- term.compare (1, strlen("attachment"), "attachment") != 0)
- continue;
+ const char *tag = notmuch_tags_get (tags);
- try {
- message->doc.remove_term ((*i));
- message->modified = true;
- } catch (const Xapian::InvalidArgumentError) {
- /* Ignore failure to remove non-existent term. */
- } catch (const Xapian::Error &error) {
- notmuch_database_t *notmuch = message->notmuch;
-
- if (!notmuch->exception_reported) {
- _notmuch_database_log(notmuch_message_get_database (message),
"A Xapian exception occurred creating message: %s\n",
- error.get_msg().c_str());
- notmuch->exception_reported = true;
- }
- return NOTMUCH_PRIVATE_STATUS_XAPIAN_EXCEPTION;
+ if (STRNCMP_LITERAL (tag, "encrypted") != 0 &&
+ STRNCMP_LITERAL (tag, "signed") != 0 &&
+ STRNCMP_LITERAL (tag, "attachment") != 0) {
+ std::string term = tag_prefix + tag;
+ message->doc.add_term(term);
}
}
+
+/* Put back properties */
+notmuch_message_properties_t *list;
+
+for (list = notmuch_message_get_properties (message, "", false);
+notmuch_message_properties_valid (list);
notmuch_message_properties_move_to_next (list)) {
+ std::string term = property_prefix +
+ notmuch_message_properties_key(list) + "=" +
+ notmuch_message_properties_value(list);
+
+ message->doc.add_term(term);
+}
+
+notmuch_message_properties_destroy (list);
+
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
}
+
/* Return true if p points at "new" or "cur". */
static bool is_maildir (const char *p)
{
--
2.20.1
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/2] CLI/reindex: fix memory leak
Since message is owned by messages, it was held for the entire run of the program. This in turn means that the Xapian::Document objects are not freed, and thus one ends up with (effectively) a copy of one's entire mailstore in memory when running notmuch reindex '*' Thanks to Olly Betts for the patient help debugging, and the suggestion of a fix. --- notmuch-reindex.c | 1 + 1 file changed, 1 insertion(+) diff --git a/notmuch-reindex.c b/notmuch-reindex.c index d8589120..fefa7c63 100644 --- a/notmuch-reindex.c +++ b/notmuch-reindex.c @@ -71,6 +71,7 @@ reindex_query (notmuch_database_t *notmuch, const char *query_string, ret = notmuch_message_reindex(message, indexopts); if (ret != NOTMUCH_STATUS_SUCCESS) break; + notmuch_message_destroy (message); } if (!ret) -- 2.20.1 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
reindex improvements
With the upgrade path for the body: prefix being "notmuch reindex '*'", it's important that that is usable. It turns out that due to a blunder on my part it wasn't really unless your whole mail store fits comfortably in RAM. [PATCH 1/2] CLI/reindex: fix memory leak This is an outright bugfix. [PATCH 2/2] n_m_remove_indexed_terms: reduce number of Xapian API This is optional, but the performance payoff is impressive. I suspect it might be even more impressive on spinning rust, but I didn't test it. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: Add missing type and group to defcustom variable
Obviously good. On Monday, 2019-04-15 at 15:35:19 +02, Örjan Ekeberg wrote: > Added the type 'regexp and group 'notmuch-send as properties to the > customizable variable notmuch-mua-attachment-regexp. > --- > emacs/notmuch-mua.el | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el > index 94fa19d7..7fdd76bc 100644 > --- a/emacs/notmuch-mua.el > +++ b/emacs/notmuch-mua.el > @@ -121,7 +121,9 @@ multiple parts get a header." >"Message body text indicating that an attachment is expected. > > This is not used unless `notmuch-mua-attachment-check' is added > -to `notmuch-mua-send-hook'.") > +to `notmuch-mua-send-hook'." > + :type 'regexp > + :group 'notmuch-send) > > ;; > > -- > 2.20.1 > > ___ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch dme. -- Please forgive me if I act a little strange, for I know not what I do. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] emacs: Add missing type and group to defcustom variable
Added the type 'regexp and group 'notmuch-send as properties to the customizable variable notmuch-mua-attachment-regexp. --- emacs/notmuch-mua.el | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/emacs/notmuch-mua.el b/emacs/notmuch-mua.el index 94fa19d7..7fdd76bc 100644 --- a/emacs/notmuch-mua.el +++ b/emacs/notmuch-mua.el @@ -121,7 +121,9 @@ multiple parts get a header." "Message body text indicating that an attachment is expected. This is not used unless `notmuch-mua-attachment-check' is added -to `notmuch-mua-send-hook'.") +to `notmuch-mua-send-hook'." + :type 'regexp + :group 'notmuch-send) ;; -- 2.20.1 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Drop support for pre-1.4 Xapian, prepare for 1.5.x
David Bremner writes: > The current development release of Xapian drops several deprecated > APIs. This hampers testing of notmuch against Xapian master (i.e. to > see if a problem has been fixed). This series fixes most of the > incompatibilities. > > The one thing this series doesn't handle is test/T530-upgrade.sh, > which mostly fails because Xapian 1.5 / master drops support for > Chert. I think that we should probably deprecate support for notmuch > v1 databases. These were replaced by version 2 in 2010. I think more > thought is probably needed for this migration so I left those failing > tests for now. This isn't quite as scary as it sounds, because we > really only support stable releases of Xapian, so the chert removal > effectively won't happen until 1.6. > I forgot, there's one other failing test with xapian 1.5 T650-regexp-query: Testing regular expression searches FAIL null from: search --- T650-regexp-query.7.EXPECTED2019-04-15 11:03:10.128070898 + +++ T650-regexp-query.7.OUTPUT 2019-04-15 11:03:10.128070898 + @@ -1 +0,0 @@ -thread:XXX 2001-01-05 [1/1] -; empty from (inbox unread) ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
