date:20170317

[PATCH 1/4] test: add known broken test for empty from: and subject: query

2017-03-17 Thread David Bremner

These queries currently fail with field processors enabled because the
code expects a non-empty string.
---
 test/T650-regexp-query.sh | 9 +
 1 file changed, 9 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index df48ab82..b1d84439 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -11,6 +11,15 @@ fi
 
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
+test_begin_subtest "empty from: search"
+test_subtest_known_broken
+notmuch search --output=messages 'from:""' and from:cworth > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "empty subject: search"
+test_subtest_known_broken
+notmuch search --output=messages 'subject:""' and from:cworth > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
 test_begin_subtest "regexp from search, case sensitive"
 notmuch search --output=messages from:/carl/ > OUTPUT
 test_expect_equal_file /dev/null OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 4/4] lib: only trigger phrase processing for regexp fields when needed

2017-03-17 Thread David Bremner

The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
---
 lib/regexp-fields.cc  | 10 --
 test/T650-regexp-query.sh |  2 --
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 42239a66..08c6ccb5 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -161,8 +161,14 @@ RegexpFieldProcessor::operator() (const std::string & str)
 } else {
/* TODO replace this with a nicer API level triggering of
 * phrase parsing, when possible */
-   std::string quoted='"' + str + '"';
-   return parser.parse_query (quoted, NOTMUCH_QUERY_PARSER_FLAGS, 
term_prefix);
+   std::string query_str;
+
+   if (str.find (' ') != std::string::npos)
+   query_str = '"' + str + '"';
+   else
+   query_str = str;
+
+   return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS, 
term_prefix);
 }
 }
 #endif
diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index ba4a64e0..c2b99d1b 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -20,12 +20,10 @@ notmuch search --output=messages 'subject:""' and 
from:cworth > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
 
 test_begin_subtest "xapian wildcard search for from:"
-test_subtest_known_broken
 notmuch search --output=messages 'from:cwo*' > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
 
 test_begin_subtest "xapian wildcard search for subject:"
-test_subtest_known_broken
 test_expect_equal $(notmuch count 'subject:count*') 1
 
 test_begin_subtest "regexp from search, case sensitive"
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/4] lib: handle empty string in regexp field processors

2017-03-17 Thread David Bremner

The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so decided to match any message.
---
 lib/regexp-fields.cc  | 3 +++
 test/T650-regexp-query.sh | 2 --
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 8e740a81..42239a66 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -148,6 +148,9 @@ RegexpFieldProcessor::RegexpFieldProcessor (std::string 
prefix, Xapian::QueryPar
 Xapian::Query
 RegexpFieldProcessor::operator() (const std::string & str)
 {
+if (str.size () == 0)
+   return Xapian::Query::MatchAll;
+
 if (str.at (0) == '/') {
if (str.at (str.size () - 1) == '/'){
RegexpPostingSource *postings = new RegexpPostingSource (slot, 
str.substr(1,str.size () - 2));
diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index b1d84439..049477b4 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -12,12 +12,10 @@ fi
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
 test_begin_subtest "empty from: search"
-test_subtest_known_broken
 notmuch search --output=messages 'from:""' and from:cworth > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
 
 test_begin_subtest "empty subject: search"
-test_subtest_known_broken
 notmuch search --output=messages 'subject:""' and from:cworth > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
 test_begin_subtest "regexp from search, case sensitive"
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

fixes for regexp field processors

2017-03-17 Thread David Bremner

This series fixes two bugs introduces in notmuch 0.24

1) queries of the form  'from:foo*' and 'subject:bar*' no longer work (the * is 
just silently dropped)
2) queries of the form 'from:""' and 'subject:""' crashes with an uncaught 
exception.

Unless there objections, I'll probably roll these patches (along with
perhaps the two memory fixes recently posted) into a bugfix release
0.24.1. Tentative release date early next week.

d


___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/4] test: add known broken tests wildcard search in from and subject

2017-03-17 Thread David Bremner

This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
because of quoting issues.
---
 test/T650-regexp-query.sh | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 049477b4..ba4a64e0 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -18,6 +18,16 @@ test_expect_equal_file cworth.msg-ids OUTPUT
 test_begin_subtest "empty subject: search"
 notmuch search --output=messages 'subject:""' and from:cworth > OUTPUT
 test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "xapian wildcard search for from:"
+test_subtest_known_broken
+notmuch search --output=messages 'from:cwo*' > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
+test_begin_subtest "xapian wildcard search for subject:"
+test_subtest_known_broken
+test_expect_equal $(notmuch count 'subject:count*') 1
+
 test_begin_subtest "regexp from search, case sensitive"
 notmuch search --output=messages from:/carl/ > OUTPUT
 test_expect_equal_file /dev/null OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] lib/message.cc: fix Coverity finding (use after free)

2017-03-17 Thread Tomi Ollila

The object where pointer to `data` was received was deleted before
it was used in _notmuch_string_list_append().

Relevant Coverity messages follow:

3: extract
Assigning: data = std::__cxx11::string(message->doc.()).c_str(),
which extracts wrapped state from temporary of type std::__cxx11::string.

4: dtor_free
The internal representation of temporary of type std::__cxx11::string
is freed by its destructor.

5: use after free:
Wrapper object use after free (WRAPPER_ESCAPE)
Using internal representation of destroyed object local data.
---

There were 30+ other 'defects' found, but none of the other seems
dangerous (and very few are kinda false). I forked github repo
of notmuch and tried this free oss coverity service -- they were
surprisingly quick to grant me access to the defects...

 lib/message.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index e08659e5d96a..cb313326270e 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -870,9 +870,9 @@ _notmuch_message_ensure_filename_list (notmuch_message_t 
*message)
 *
 * It would be nice to do the upgrade of the document directly
 * here, but the database is likely open in read-only mode. */
-   const char *data;
 
-   data = message->doc.get_data ().c_str ();
+   std::string datastr = message->doc.get_data ();
+   const char *data = datastr.c_str ();
 
if (data == NULL)
INTERNAL_ERROR ("message with no filename");
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] fix memory leaks in notmuch-show.c:format_headers_sprinter()

2017-03-17 Thread Tomi Ollila

On Thu, Mar 16 2017, Jeffrey Stedfast  wrote:

> Hey guys,
>
> Was just grepping through notmuch sources and discovered what I think are 
> memory leaks in notmuch-show.c’s format_headers_sprinter() code.
>
> Internet_address_list_to_string() and g_mime_message_get_date_as_string()
> return allocated string buffers and not const, so from what I can tell
> from taking a look at the sprinter-sexp.c’s sexp_string() function, the
> code leaks the recipients_string as well as the date string.

Change looks good, tests pass (it even applied with git am).

The commit message could be amended like so, that subject line and the
paragraph above this were left...

Tomi


>
> Attached is a patch which fixes these leaks.
>
> Hope this helps,

>
> Jeff
>
>
> diff --git a/notmuch-show.c b/notmuch-show.c
> index aff93803..095595e2 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -202,8 +202,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
> *message,
>   * reflected in the file devel/schemata. */
>  
>  InternetAddressList *recipients;
> -const char *recipients_string;
> +char *recipients_string;
>  const char *reply_to_string;
> +char *date_string;
>  
>  sp->begin_map (sp);
>  
> @@ -218,6 +219,7 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
> *message,
>  if (recipients_string) {
>   sp->map_key (sp, "To");
>   sp->string (sp, recipients_string);
> + g_free (recipients_string);
>  }
>  
>  recipients = g_mime_message_get_recipients (message, 
> GMIME_RECIPIENT_TYPE_CC);
> @@ -225,6 +227,7 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
> *message,
>  if (recipients_string) {
>   sp->map_key (sp, "Cc");
>   sp->string (sp, recipients_string);
> + g_free (recipients_string);
>  }
>  
>  recipients = g_mime_message_get_recipients (message, 
> GMIME_RECIPIENT_TYPE_BCC);
> @@ -232,6 +235,7 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
> *message,
>  if (recipients_string) {
>   sp->map_key (sp, "Bcc");
>   sp->string (sp, recipients_string);
> + g_free (recipients_string);
>  }
>  
>  reply_to_string = g_mime_message_get_reply_to (message);
> @@ -248,7 +252,9 @@ format_headers_sprinter (sprinter_t *sp, GMimeMessage 
> *message,
>   sp->string (sp, g_mime_object_get_header (GMIME_OBJECT (message), 
> "References"));
>  } else {
>   sp->map_key (sp, "Date");
> - sp->string (sp, g_mime_message_get_date_as_string (message));
> + date_string = g_mime_message_get_date_as_string (message);
> + sp->string (sp, date_string);
> + g_free (date_string);
>  }
>  
>  sp->end (sp);
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] rename libutil.a to libnotmuch_util.a

2017-03-17 Thread Tomi Ollila

On Tue, Mar 14 2017, David Bremner  wrote:

> Apparently some systems (MacOS?) have a system library called libutil
> and the name conflict causes problems. Since this library is quite
> notmuch specific, rename it to something less generic.
> ---

LGTM

>  Makefile.global |  2 +-
>  Makefile.local  |  2 +-
>  lib/Makefile.local  |  4 ++--
>  test/Makefile.local |  6 +++---
>  util/Makefile.local | 10 +-
>  5 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/Makefile.global b/Makefile.global
> index 7a78e9b5..cae4c7d1 100644
> --- a/Makefile.global
> +++ b/Makefile.global
> @@ -52,7 +52,7 @@ PV_FILE=bindings/python/notmuch/version.py
>  STD_CFLAGS := -std=gnu99
>  FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CPPFLAGS) $(STD_CFLAGS) 
> $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CONFIGURE_CFLAGS)
>  FINAL_CXXFLAGS = $(CPPFLAGS) $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) 
> $(extra_cxxflags) $(CONFIGURE_CXXFLAGS)
> -FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lutil -Llib -lnotmuch
> +FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lnotmuch_util -Llib -lnotmuch
>  ifeq ($(LIBDIR_IN_LDCONFIG),0)
>  FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
>  endif
> diff --git a/Makefile.local b/Makefile.local
> index e75b6eae..03eafaaa 100644
> --- a/Makefile.local
> +++ b/Makefile.local
> @@ -241,7 +241,7 @@ notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
>  
>  notmuch.o: version.stamp
>  
> -notmuch: $(notmuch_client_modules) lib/libnotmuch.a util/libutil.a 
> parse-time-string/libparse-time-string.a
> +notmuch: $(notmuch_client_modules) lib/libnotmuch.a util/libnotmuch_util.a 
> parse-time-string/libparse-time-string.a
>   $(call quiet,CXX $(CFLAGS)) $^ $(FINAL_LIBNOTMUCH_LDFLAGS) -o $@
>  
>  notmuch-shared: $(notmuch_client_modules) lib/$(LINKER_NAME)
> diff --git a/lib/Makefile.local b/lib/Makefile.local
> index cd92fc79..d36fd5a0 100644
> --- a/lib/Makefile.local
> +++ b/lib/Makefile.local
> @@ -60,8 +60,8 @@ libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) 
> $(libnotmuch_cxx_srcs:.cc=.o)
>  $(dir)/libnotmuch.a: $(libnotmuch_modules)
>   $(call quiet,AR) rcs $@ $^
>  
> -$(dir)/$(LIBNAME): $(libnotmuch_modules) notmuch.sym util/libutil.a 
> parse-time-string/libparse-time-string.a
> - $(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) 
> $(FINAL_LIBNOTMUCH_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@ util/libutil.a 
> parse-time-string/libparse-time-string.a
> +$(dir)/$(LIBNAME): $(libnotmuch_modules) notmuch.sym util/libnotmuch_util.a 
> parse-time-string/libparse-time-string.a
> + $(call quiet,CXX $(CXXFLAGS)) $(libnotmuch_modules) 
> $(FINAL_LIBNOTMUCH_LDFLAGS) $(LIBRARY_LINK_FLAG) -o $@ util/libnotmuch_util.a 
> parse-time-string/libparse-time-string.a
>  
>  notmuch.sym: $(srcdir)/$(dir)/notmuch.h $(libnotmuch_modules)
>   sh $(srcdir)/$(lib)/gen-version-script.sh $< $(libnotmuch_modules) > $@
> diff --git a/test/Makefile.local b/test/Makefile.local
> index 46805972..0df72c92 100644
> --- a/test/Makefile.local
> +++ b/test/Makefile.local
> @@ -12,15 +12,15 @@ smtp_dummy_srcs = \
>  
>  smtp_dummy_modules = $(smtp_dummy_srcs:.c=.o)
>  
> -$(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o util/libutil.a
> +$(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o 
> util/libnotmuch_util.a
>   $(call quiet,CC) $^ -o $@ $(LDFLAGS)
>  
> -$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
> +$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o 
> util/libnotmuch_util.a
>   $(call quiet,CC) $^ -o $@ $(LDFLAGS) $(TALLOC_LDFLAGS)
>  
>  random_corpus_deps =  $(dir)/random-corpus.o  $(dir)/database-test.o \
>   notmuch-config.o status.o command-line-arguments.o \
> - lib/libnotmuch.a util/libutil.a \
> + lib/libnotmuch.a util/libnotmuch_util.a \
>   parse-time-string/libparse-time-string.a
>  
>  $(dir)/random-corpus: $(random_corpus_deps)
> diff --git a/util/Makefile.local b/util/Makefile.local
> index 905f2376..a6962d49 100644
> --- a/util/Makefile.local
> +++ b/util/Makefile.local
> @@ -3,14 +3,14 @@
>  dir := util
>  extra_cflags += -I$(srcdir)/$(dir)
>  
> -libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
> +libnotmuch_util_c_srcs := $(dir)/xutil.c $(dir)/error_util.c 
> $(dir)/hex-escape.c \
> $(dir)/string-util.c $(dir)/talloc-extra.c 
> $(dir)/zlib-extra.c \
>   $(dir)/util.c
>  
> -libutil_modules := $(libutil_c_srcs:.c=.o)
> +libnotmuch_util_modules := $(libnotmuch_util_c_srcs:.c=.o)
>  
> -$(dir)/libutil.a: $(libutil_modules)
> +$(dir)/libnotmuch_util.a: $(libnotmuch_util_modules)
>   $(call quiet,AR) rcs $@ $^
>  
> -SRCS := $(SRCS) $(libutil_c_srcs)
> -CLEAN := $(CLEAN) $(libutil_modules) $(dir)/libutil.a
> +SRCS := $(SRCS) $(libnotmuch_util_c_srcs)
> +CLEAN := $(CLEAN) $(libnotmuch_util_modules) $(dir)/libnotmuch_util.a
> -- 
> 2.11.0
>
>

Re: [RFC patch 2/2] lib: index message files with duplicate message-ids

2017-03-17 Thread Daniel Kahn Gillmor

On Thu 2017-03-16 20:34:22 -0400, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>>  0) what happens when one of the files gets deleted from the message
>> store? do the terms it contributes get removed from the index?
>
> That's a good guestion, and an issue I hadn't thought about.
> Currently there's no way to do this short of deleting all the terms (for
> all the files (excepting tags and properties, presumably) and
> reindexing. This will require some more thought, I think.

i didn't mean to raise the concern to drag this work down, i just want
to make sure the problem is on the table.  dropping all terms on
deletion and re-indexing remaining files with the same message ID isn't
terribly efficient, but i don't think it's going to be terribly costly
either.  we're not talking about hundreds of files per message-id in
most normal cases; usually only two (sent-to-self,
recvd-from-mailing-list), and maybe a half-dozen at most (messages sent
to multiple mailboxes that all forward to me).

of course, if multiple files are deleted concurrently, and notmuch
notices that one of them is missing, then re-indexing the other will
depend on whether it was also deleted in that same batch.

>>  1) when a message is displayed to the user as a result of a match, it
>> gets pulled from one of the files, not both.  if it's pulled from
>> the file that didn't have the term the user searched for, that's
>> likely to be confusing.  do you have a way to avoid that confusion?
>
> I was looking for an incremental improvement, so I imagined something
> like various output flagging "yes, there are duplicate files for this
> message", and letting users dig those out using something like the
> --duplicate= option.

This kind of output flagging would be worthwhile in its own right, and
maybe is an even less controversial place to start for the incremental
improvement.

>> It also occurs to me that one of the things i'd love to have is
>> well-indexed notes about any given e-mail.  So if this was adopted, i
>> could presumably just write a file that has the same Message-Id as the
>> message, put my notes in it, and index it.  that's a little weird,
>> though.  would there be a better way to do such a thing?
>
> One option would be to use a note=foo mesage property. That's not
> immediately searchable though, although we could kludge together
> something like the subject regexp search which would be slower.

right, i think i'd want the notes to be searchable, if possible.

Now i'm thinking about attack scenarios for this multi-indexed scheme,
though.  If i know that you've already gotten an e-mail with message-id
X, then i can go ahead and remotely, silently add search terms to that
message by sending you new messages that have the same message-id.  That
seems troubling :/ The status quo at least requires the attacker to win
a race to get their message indexed first, obscuring the real message.
in the proposed new scenario, the attacker doesn't need to win any race.
they can't prevent the true message from being indexed, but they can
associate it with whatever toxicity (e.g. "viagra", or "From:
killfiled-user") they want which might be useful in suppressing the
message in a post-processing run.

ugh, mail,

  --dkg

signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Announcing Astroid v0.8

2017-03-17 Thread Gaute Hope


Greetings,

Astroid v0.8 is now ready!

 Astroid is a lightweight and fast graphical threads-with-tags email
 client for notmuch. Written in C++ using GTK+, WebKit and gmime.


Astroid can be found at:

$ git clone https://github.com/astroidmail/astroid.git


## Usage and instructions 


   https://github.com/astroidmail/astroid#astroid-mua

 once you get Astroid running press '?' to get a list of keybindings
 for the current context.

 and The Tour:

   https://github.com/astroidmail/astroid/wiki

## Changes since v0.7

* New logo and icons by Elio Qoshi ( http://patreon.com/ura )

* Run hook for thread-view (Hugo Roy)

* Session-persistent folder history for attaching files and saving
 attachments. default is configurable.

* Remove duplicates when reply-all'ing.

* Handle mailto when astroid is already running.

* `scons --propagate-environment` uses environment from build environment.

* Show pane-focus with blue bar at top, swap focus with M-space. The
 keybinding is now located at 'pane.swap_focus'.

* Add optional drop-down terminal ('|') which has session-persistent
 working directory. Will silently be disabled if VTE is missing.

* Delay sending of message with configured time, in order to be able to
 cancel wrongly sent messages.

* New --start-polling, --stop-polling and --refresh LASTMOD options which
 makes external polling mechanisms work better.

* All notmuch tags are added to message CSS with 'nm-' prefix and some 
sanitation to allow thread-view customizing based on tag.

* Reply-To mailing list.

* Fix crypto and performance bugs among many others.


Regards, Gaute



pgpuEeKFeN_lG.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Re: Inconsistency between tag database and notmuch search

2017-03-17 Thread David Bremner

Navin Kabra  writes:

>
> Specifically, I have a few threads/messages that don't have any tags,
> but still show up in notmuch search for those tags. And using notmuch
> tag to remove (or add) any tags on these threads has no effect. Doing
> a `notmuch compact` does not fix this issue.

0) As a first step, run xapian-check, something like

% xapian-check ~/Maildir/.notmuch/xapian

If your database is corrupted, there's not much notmuch can do about it
(except maybe make things worse).  I would think that compact would have
detected any corruption, but you never know.

>
> ~$ notmuch search thread:00058ca0 and tag:inbox
> thread:00058ca0  February 22 [1/1] Mandar Joshi; 
> Announcing TiECon Pune 2017 - Pune's largest, most energizing 
> startup event! ()

1) Can you also see the problem by looking at individual message-ids?,
   e.g. using notmuch show

2) In particular Does notmuch dump generate the correct output? If so,
   then dump your tags, rebuild your database, and reload the tags is
   one option.

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] test: add known broken test for xapian wildcard search in from:

2017-03-17 Thread David Bremner

This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
perhaps because of quoting issues.
---
There should be a similar test for subject: but I wanted to report
this bug now.

 test/T650-regexp-query.sh | 5 +
 1 file changed, 5 insertions(+)

diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index df48ab82..4c3a562f 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -11,6 +11,11 @@ fi
 
 notmuch search --output=messages from:cworth > cworth.msg-ids
 
+test_begin_subtest "xapian wildcard search for from:"
+test_subtest_known_broken
+notmuch search --output=messages 'from:cwo*' > OUTPUT
+test_expect_equal_file cworth.msg-ids OUTPUT
+
 test_begin_subtest "regexp from search, case sensitive"
 notmuch search --output=messages from:/carl/ > OUTPUT
 test_expect_equal_file /dev/null OUTPUT
-- 
2.11.0

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Inconsistency between tag database and notmuch search

2017-03-17 Thread Navin Kabra



I've managed to get my notmuch tag database in an inconsistent 
state and
can't figure out how to fix it. Because of this I have "unread" 
messages sitting in my inbox permanently and I can't get rid of 
them.


Specifically, I have a few threads/messages that don't have any 
tags,

but still show up in notmuch search for those tags. And using
notmuch tag to remove (or add) any tags on these threads has no 
effect. Doing a `notmuch compact` does not fix this issue.


(I believe I managed to do this by running some notmuch commands 
while a

`xapian-compact` was running in the background.)

Here are some commands that demonstrate the problem:

   ~$ notmuch --version
   notmuch 0.23.5

   ~$ notmuch search thread:00058ca0
   thread:00058ca0  February 22 [1/1] Mandar Joshi; 
   Announcing TiECon Pune 2017 - Pune's largest, most energizing 
   startup event! ()


   ~$ notmuch search thread:00058ca0 and tag:inbox
   thread:00058ca0  February 22 [1/1] Mandar Joshi; 
   Announcing TiECon Pune 2017 - Pune's largest, most energizing 
   startup event! ()


   ~$ notmuch tag -inbox thread:00058ca0 

   ~$ notmuch search thread:00058ca0 
   thread:00058ca0  February 22 [1/1] Mandar Joshi; 
   Announcing TiECon Pune 2017 - Pune's largest, most energizing 
   startup event! ()


   ~$ notmuch tag +inbox thread:00058ca0 

   ~$ notmuch search thread:00058ca0 
   thread:00058ca0  February 22 [1/1] Mandar Joshi; 
   Announcing TiECon Pune 2017 - Pune's largest, most energizing 
   startup event! ()



As you can see from the `()` at the end of the notmuch search 
output, this thread has no tags associated with it. And still, it 
is showing up in the notmuch search output.


What should I do to get rid of these messages from the notmuch 
search output?

___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/4] test: add known broken test for empty from: and subject: query

[PATCH 4/4] lib: only trigger phrase processing for regexp fields when needed

[PATCH 2/4] lib: handle empty string in regexp field processors

fixes for regexp field processors

[PATCH 3/4] test: add known broken tests wildcard search in from and subject

[PATCH] lib/message.cc: fix Coverity finding (use after free)

Re: [PATCH] fix memory leaks in notmuch-show.c:format_headers_sprinter()

Re: [PATCH] rename libutil.a to libnotmuch_util.a

Re: [RFC patch 2/2] lib: index message files with duplicate message-ids

Announcing Astroid v0.8

Re: Inconsistency between tag database and notmuch search

[PATCH] test: add known broken test for xapian wildcard search in from:

Inconsistency between tag database and notmuch search

13 matches

Site Navigation

Mail list logo

Footer information