Re: xapian parser bug?
Olly Betts writes:
> On Sun, Sep 30, 2018 at 09:05:25AM -0300, David Bremner wrote:
>> if (str.find (' ') != std::string::npos)
>> query_str = '"' + str + '"';
>> else
>> query_str = str;
>>
>> return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS,
>> term_prefix);
>
> I wouldn't recommend trying to generate strings to feed to QueryParser
> like this code seems to be doing. QueryParser aims to parse input from
> humans not machines.
str is the parameter to the FieldProcessor () operator. The field
processor needs a way to approximate the standard probabilistic prefix
parsing in the fallback case. The addition of quotes is to force the
generation of a phrase query, otherwise e.g. subject:"christmas party"
doesn't work out well.
I tried using OP_PHRASE as a the default operators, but it doesn't
handle some cases I need.
% quest -o phrase 'bob jones '
UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries
If I don't recursively call parse_query, then I guess I need to generate
terms in a compatible way before turning them into a phrase query. Maybe
that's not as hard as I orginally thought, since being in phrase turns
off the stemmer anyway iiuc. Is there a Xapian API I can use to extract
"bob", "jones", "bob", "example", "com" from the example above? I guess
I guess I could use a throwaway Xapian::Document and a TermGenerator
(basically aping xapian_core/tests/api_termgen.cc).
d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: unused/total message counts in hello screen?
Jeff Templon writes: > An alternative would be if I could toggle the entire page to show just > the unread tags (so any tags for which all messages were read would > just not show up). Maybe even preferable, that last one. This can be done setting "notmuch-hello-tag-list-make-query" to "tag:unread" I have mine set to "tag:inbox" so that I only see tag counts on the home screen that haven't been archived yet. Cheers, Will ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: xapian parser bug?
On Sun, Sep 30, 2018 at 09:05:25AM -0300, David Bremner wrote:
> if (str.find (' ') != std::string::npos)
> query_str = '"' + str + '"';
> else
> query_str = str;
>
> return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS,
> term_prefix);
I wouldn't recommend trying to generate strings to feed to QueryParser
like this code seems to be doing. QueryParser aims to parse input from
humans not machines.
As well as the case where str is an operation name, the code above looks
like it will mishandle cases where str contains a tab or double quotes.
There are likely other problem cases too.
Cheers,
Olly
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[astroid] Announcing Astroid v0.14
Greetings, Astroid v0.14 has been released! Astroid is a lightweight and fast graphical threads-with-tags email client for notmuch. Written in C++ using GTK+, WebKit2 and GMime. Astroid can be acquired, along with instructions for use, at: https://github.com/astroidmail/astroid once you get Astroid running press '?' to get a list of keybindings for the current context. For 'The Tour' check out: https://github.com/astroidmail/astroid/wiki ## Changes since v0.13 * Use cmark by default as markdown processor. * Respect $NOTMUCH_CONFIG and ignore configuration option if set. Otherwise fall back to option from configuration file. * We now have a man page! Generated either by scdoc or ronn. * Number of bugfixes. Gaute Hope (24): log: re-introduce log-level command line option to overrdide config git: only ignore build/ dirs, not build* tv: remove deactived syntax code readme: update demo image cmake: webkit2gtk >= 2.20 is required w2: fix jump to top / bottom readme: fix typos fix #535: close correct tab if Mode::close is called on non-active tab cmake: libhypocycloid should always be static test: bump plugins to GIR v0.2 fix #541: use 'cmark' rather than 'marked' as default markdown processor cmake: generate and install man pages using either scdoc or ronn" man: mention $NOTMUCH_CONFIG (93ef250b) history: man page fix #539: reference to LICENSE.md. fix #542 cf: do not show default startup queries when intentionally cleared. force saved_searches if no startup queries. cmake, man: add option to disable manpage generation -DDISABLE_DOCS=ON fix #548: correctly clear messages and focus state when loading new thread mw: remove modes when closing window tests: make gpg setup more verbose tests: quote cp src dir fix #550: use srcdoc attribute when setting message content in iframe tv, iframe: delay set iframe src in order to let body element be rendered first Release v0.14 Iain Buclaw (1): fix #501: Add hook to signal_delete_event, ensure proper destruction of window Johannes Löthberg (1): Use NOTMUCH_CONFIG if set, otherwise expand astroid.notmuch_config Timothée Floure (4): Add initial manual page astroid(1) Add the --log-level option to the manpage Rephrase the 'NOTE' (on software used with astroid) section of the manpage Mention the possibility to customize the thread-view in the manpage Tristan Cacqueray (1): cmake: fix scalable icon install path Yurii Rashkovskii (1): Problem: tests/run-test.sh might fail pgpOPKrLWHqWu.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: xapian parser bug?
David Bremner writes:
> Olly Betts writes:
>
>>
>> FWIW, I also couldn't reproduce this (I tried with quest and 1.4.7):
>>
>> $ quest -psubject:S -fdefault,boolean_any_case 'subject:"and"'
>> Parsed Query: Query(Sand@1)
>>
>
> Ah, OK, it must have something to do with the way that notmuch is using
> field processors. And I see now that the following code (from
> lib/regexp-fields.cc) is probably related (at least it explains
> subject:" not" works)
>
> if (str.find (' ') != std::string::npos)
> query_str = '"' + str + '"';
> else
> query_str = str;
>
> return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS,
> term_prefix);
For the record, I have proposed a fix for notmuch (str is known to be
non-empty there). This will phrase quote by default, unless the string
looks like a wildcard query (without spaces).
diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 084bc8c0..52f30d82 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -194,7 +194,7 @@ RegexpFieldProcessor::operator() (const std::string & str)
* phrase parsing, when possible */
std::string query_str;
- if (str.find (' ') != std::string::npos)
+ if (*str.rbegin () != '*' || str.find (' ') != std::string::npos)
query_str = '"' + str + '"';
else
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/2] lib: use phrase search for anything not ending in '*'
Anything that does not look like a wildcard should be safe to
quote. This should fix the problem searching for xapian keywords.
---
lib/regexp-fields.cc | 2 +-
test/T650-regexp-query.sh | 2 --
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/regexp-fields.cc b/lib/regexp-fields.cc
index 084bc8c0..52f30d82 100644
--- a/lib/regexp-fields.cc
+++ b/lib/regexp-fields.cc
@@ -194,7 +194,7 @@ RegexpFieldProcessor::operator() (const std::string & str)
* phrase parsing, when possible */
std::string query_str;
- if (str.find (' ') != std::string::npos)
+ if (*str.rbegin () != '*' || str.find (' ') != std::string::npos)
query_str = '"' + str + '"';
else
query_str = str;
diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh
index 7448c024..181e3856 100755
--- a/test/T650-regexp-query.sh
+++ b/test/T650-regexp-query.sh
@@ -81,12 +81,10 @@ add_message '[from]="and"' '[subject]="and-and-and"'
printf "id:$gen_msg_id\n" > EXPECTED
test_begin_subtest "quoted xapian keyword search for from:"
-test_subtest_known_broken
notmuch search --output=messages 'from:"and"' > OUTPUT
test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "quoted xapian keyword search for subject:"
-test_subtest_known_broken
notmuch search --output=messages 'subject:"and-and-and"' > OUTPUT
test_expect_equal_file EXPECTED OUTPUT
--
2.19.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/2] test: add two known broken tests searching for xapian keywords
Thanks to plujon for pointing out this problem on IRC. The underlying issue is that the quotes are stripped before the field processors get the query string, and the heuristic for putting them back is not quite right. --- test/T650-regexp-query.sh | 13 + 1 file changed, 13 insertions(+) diff --git a/test/T650-regexp-query.sh b/test/T650-regexp-query.sh index 4085340f..7448c024 100755 --- a/test/T650-regexp-query.sh +++ b/test/T650-regexp-query.sh @@ -77,6 +77,19 @@ test_expect_equal_file cworth.msg-ids OUTPUT test_begin_subtest "xapian wildcard search for subject:" test_expect_equal $(notmuch count 'subject:count*') 1 +add_message '[from]="and"' '[subject]="and-and-and"' +printf "id:$gen_msg_id\n" > EXPECTED + +test_begin_subtest "quoted xapian keyword search for from:" +test_subtest_known_broken +notmuch search --output=messages 'from:"and"' > OUTPUT +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest "quoted xapian keyword search for subject:" +test_subtest_known_broken +notmuch search --output=messages 'subject:"and-and-and"' > OUTPUT +test_expect_equal_file EXPECTED OUTPUT + test_begin_subtest "regexp from search, case sensitive" notmuch search --output=messages from:/carl/ > OUTPUT test_expect_equal_file /dev/null OUTPUT -- 2.19.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: xapian parser bug?
Olly Betts writes:
>
> FWIW, I also couldn't reproduce this (I tried with quest and 1.4.7):
>
> $ quest -psubject:S -fdefault,boolean_any_case 'subject:"and"'
> Parsed Query: Query(Sand@1)
>
Ah, OK, it must have something to do with the way that notmuch is using
field processors. And I see now that the following code (from
lib/regexp-fields.cc) is probably related (at least it explains
subject:" not" works)
if (str.find (' ') != std::string::npos)
query_str = '"' + str + '"';
else
query_str = str;
return parser.parse_query (query_str, NOTMUCH_QUERY_PARSER_FLAGS,
term_prefix);
The motivation for not always triggering phrase processing is that it
breaks/disables wildcards. In particular this change was to fix the
query 'subject:foo*'. The difficulty here is that the field processor
doesn't know if its string argument was originally quoted.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 2/4] emacs: Minor refactoring of crypto code
David Edmondson writes: > - (setq help-msg (concat "Click to retrieve key ID " keyid " from > keyserver and redisplay." > + (setq label (concat "Unknown key ID " keyid " or unsupported > algorithm") > + button-action 'notmuch-crypto-sigstatus-error-callback > + help-msg (concat "Click to retrieve key ID " keyid > + " from keyserver and redisplay."))) > + I wonder if we should still claim we're going to redisplay here, since I have the feeling most of the time that won't happen? ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 3/4] emacs: Add notmuch-crypto-gpg-program and use it
David Edmondson writes: > Allow the user to specify the gpg program to use when retriving keys, > etc., defaulting to the value of `epg-gpg-program'. typo in retriving. More interestingly, would you mind documenting the use case in the commit message? I'm unclear on a couple points - why one needs a different gpg for key retrieval - whether the new variable is just for key retrieval, or as the name notmuch-crypto-gpg-program suggests, all crypto operations from notmuch. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: xapian parser bug?
David — this isn't the behaviour I see what QueryParser alone, unless you're
driving it in a way I don't expect. In python:
>>> import xapian
>>> qp = xapian.QueryParser()
>>> qp.add_prefix('subject', 'S')
>>> str(qp.parse_query('subject:"and"',
>>> qp.FLAG_DEFAULT|qp.FLAG_BOOLEAN_ANY_CASE))
'Query(Sand@1)'
>>> str(qp.parse_query('subject:"or"',
>>> qp.FLAG_DEFAULT|qp.FLAG_BOOLEAN_ANY_CASE))
'Query(Sor@1)'
>>> str(qp.parse_query('subject:"not"',
>>> qp.FLAG_DEFAULT|qp.FLAG_BOOLEAN_ANY_CASE))
'Query(Snot@1)'
>>> str(qp.parse_query('subject:" not "',
>>> qp.FLAG_DEFAULT|qp.FLAG_BOOLEAN_ANY_CASE))
'Query(Snot@1)'
Note that I'm using 1.4.7, and from your output I believe you're not (the * in
the query description I believe doesn't happen in those situations any more).
J
> On 29 Sep 2018, at 23:09, David Bremner wrote:
>
>
> Today we noticed that keywords can't be searched as prefixed terms. Or
> that's what it looks like anyway. I tested and, or, and not.
>
> ╰─% NOTMUCH_DEBUG_QUERY=y notmuch search 'subject:"and"'
> Query string is:
> subject:"and"
> notmuch search: A Xapian exception occurred
> A Xapian exception occurred parsing query: Syntax: AND
>
> Query string was: subject:"and"
>
> ╰─% NOTMUCH_DEBUG_QUERY=y notmuch search 'subject:"or"'
> Query string is:
> subject:"or"
> notmuch search: A Xapian exception occurred
> A Xapian exception occurred parsing query: Syntax: OR
>
> Query string was: subject:"or"
>
> ╰─% NOTMUCH_DEBUG_QUERY=y notmuch search 'subject:"not"'
> Query string is:
> subject:"not"
> notmuch search: A Xapian exception occurred
> A Xapian exception occurred parsing query: Syntax: NOT
>
> Query string was: subject:"not"
>
> Interestingly, putting space around the operator seems to be a
> workaround. Something about turning on phrase parsing maybe?
>
> ╰─% NOTMUCH_DEBUG_QUERY=y notmuch count 'subject:" not "'
> Query string is:
> subject:" not "
> Exclude query is:
> QueryKspam OR Kdeleted) OR Kmuted) OR Kbad-address))
> Final query is:
> Query(((Tmail AND 0 * XSUBJECTnot@1) AND_NOT (((Kspam OR Kdeleted) OR Kmuted)
> OR Kbad-address)))
> 9927
>
--
James Aylett
devfort.com — spacelog.org — tartarus.org/james/
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: xapian parser bug?
On Sun, Sep 30, 2018 at 09:50:30AM +0100, James Aylett wrote: > Note that I'm using 1.4.7, and from your output I believe you're not > (the * in the query description I believe doesn't happen in those > situations any more). 1.4.4 and later eliminate redundant 0 scaling factors, but this one isn't actually redundant: > > Query(((Tmail AND 0 * XSUBJECTnot@1) AND_NOT (((Kspam OR Kdeleted) OR > > Kmuted) OR Kbad-address))) If it was on the right-hand side of AND_NOT it would be eliminated (because the right-hand side doesn't contribute any weight anyway). FWIW, I also couldn't reproduce this (I tried with quest and 1.4.7): $ quest -psubject:S -fdefault,boolean_any_case 'subject:"and"' Parsed Query: Query(Sand@1) Cheers, Olly ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
