[PATCH v3 6/9] lib: add date range query support

[Michal Sojka]

On Thu, Sep 13 2012, Michal Nazarewicz wrote:
> On Wed, Sep 12 2012, Jani Nikula wrote:
>> Add a custom value range processor to enable date and time searches of
>> the form date:since..until, where "since" and "until" are expressions
>> understood by the previously added date/time parser, to restrict the
>> results to messages within a particular time range (based on the Date:
>> header).
>>
>> If "since" or "until" describes date/time at an accuracy of days or
>> less, the values are rounded according to the accuracy, towards past
>> for "since" and towards future for "until". For example,
>> date:november..yesterday would match from the beginning of November
>> until the end of yesterday. Expressions such as date:today..today
>> means since the beginning of today until the end of today.
>
> IMO this is totally unintuitive and not how the range should work.
> date:foo..bar should return messages whose date >= foo and < bar.  So
> for instance date:november..yesterday should return messages whose date
> is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
> yesterdays messages one would do: date:yesterday..today.

For me, date:monday..wednesday means all messages received on monday,
tuseday or wednesday. If I say Wednesday, I'm really interested in
Wednesday and not the day before Wednesday.

I'd also like to allow syntax like date:yesterday with the meaning all
messages sent yesterday. My idea how to implement this was described in
id:"87bovryqp0.fsf at steelpick.2x.cz". Unfortunately, I have no time to
implement it myself.

-Michal

Re: [PATCH v3 6/9] lib: add date range query support

[Michal Sojka]

On Thu, Sep 13 2012, Michal Nazarewicz wrote:
 On Wed, Sep 12 2012, Jani Nikula wrote:
 Add a custom value range processor to enable date and time searches of
 the form date:since..until, where since and until are expressions
 understood by the previously added date/time parser, to restrict the
 results to messages within a particular time range (based on the Date:
 header).

 If since or until describes date/time at an accuracy of days or
 less, the values are rounded according to the accuracy, towards past
 for since and towards future for until. For example,
 date:november..yesterday would match from the beginning of November
 until the end of yesterday. Expressions such as date:today..today
 means since the beginning of today until the end of today.

 IMO this is totally unintuitive and not how the range should work.
 date:foo..bar should return messages whose date = foo and  bar.  So
 for instance date:november..yesterday should return messages whose date
 is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
 yesterdays messages one would do: date:yesterday..today.

For me, date:monday..wednesday means all messages received on monday,
tuseday or wednesday. If I say Wednesday, I'm really interested in
Wednesday and not the day before Wednesday.

I'd also like to allow syntax like date:yesterday with the meaning all
messages sent yesterday. My idea how to implement this was described in
id:87bovryqp0@steelpick.2x.cz. Unfortunately, I have no time to
implement it myself.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

On Thu, Sep 13 2012, Jani Nikula wrote:
> I find "since" rounding towards past and "until" rounding towards future
> a very simple rule. But YMMV.

To implement rounding, each date needs to have a period of time to align
to.  I call that a duration.  But if you have such a duration than
I propose a solution where you don't need any kind of rounding.

If ?yesterday? has a duration of one day, than ?date:yesterday? would be
equivalent to ?date:yesterday..yesterday + 1 day? and this works
perfectly well with ranges open on the right side.  So to implement date
specifications with a single date, no additional code is really
required.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Micha? ?mina86? Nazarewicz(o o)
ooo +--ooO--(_)--Ooo--
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

[PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

> Michal Nazarewicz  writes:
>> IMO this is totally unintuitive and not how the range should work.
>> date:foo..bar should return messages whose date >= foo and < bar.  So
>> for instance date:november..yesterday should return messages whose date
>> is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
>> yesterdays messages one would do: date:yesterday..today.

On Thu, Sep 13 2012, David Bremner wrote:
> I don't find ranges being half-open by default to be very
> intuitive. Perhaps I don't program in python enough.

Perhaps C than: ?for (i = 0; i < 10; ++i)? is the standard idiom and the
end range is open.

Let's take a look at:

date:2012/01/01..2012/01/01 + 1 day

in my opinion, that should give results from the first of January only,
since ?+ 1 day? indicates in a way how long user want the period to be.

I think it's also easier to pragmatically create ranges.  For instance,
let's say you want to create ranges for each week, you'd end up with:

date:2012/01/02..2012/01/09 ## 2012w01
date:2012/01/09..2012/01/16 ## 2012w02
date:2012/01/16..2012/01/23 ## 2012w03

Notice how the opening date of a range matches the closing date of
the previous date.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Micha? ?mina86? Nazarewicz(o o)
ooo +--ooO--(_)--Ooo--
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

Re: [PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

 Michal Nazarewicz min...@mina86.com writes:
 IMO this is totally unintuitive and not how the range should work.
 date:foo..bar should return messages whose date = foo and  bar.  So
 for instance date:november..yesterday should return messages whose date
 is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
 yesterdays messages one would do: date:yesterday..today.

On Thu, Sep 13 2012, David Bremner wrote:
 I don't find ranges being half-open by default to be very
 intuitive. Perhaps I don't program in python enough.

Perhaps C than: “for (i = 0; i  10; ++i)” is the standard idiom and the
end range is open.

Let's take a look at:

date:2012/01/01..2012/01/01 + 1 day

in my opinion, that should give results from the first of January only,
since “+ 1 day” indicates in a way how long user want the period to be.

I think it's also easier to pragmatically create ranges.  For instance,
let's say you want to create ranges for each week, you'd end up with:

date:2012/01/02..2012/01/09 ## 2012w01
date:2012/01/09..2012/01/16 ## 2012w02
date:2012/01/16..2012/01/23 ## 2012w03

Notice how the opening date of a range matches the closing date of
the previous date.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz(o o)
ooo +email/xmpp: m...@google.com--ooO--(_)--Ooo--

pgpT8vbANXDEM.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

On Thu, Sep 13 2012, Jani Nikula wrote:
 I find since rounding towards past and until rounding towards future
 a very simple rule. But YMMV.

To implement rounding, each date needs to have a period of time to align
to.  I call that a duration.  But if you have such a duration than
I propose a solution where you don't need any kind of rounding.

If “yesterday” has a duration of one day, than “date:yesterday” would be
equivalent to “date:yesterday..yesterday + 1 day” and this works
perfectly well with ranges open on the right side.  So to implement date
specifications with a single date, no additional code is really
required.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz(o o)
ooo +email/xmpp: m...@google.com--ooO--(_)--Ooo--

pgpZJqNBZKt06.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 6/9] lib: add date range query support

[Jani Nikula]

On Thu, 13 Sep 2012, Tomi Ollila  wrote:
> On Thu, Sep 13 2012, Michal Nazarewicz  wrote:
>
>> On Wed, Sep 12 2012, Jani Nikula wrote:
>>> Add a custom value range processor to enable date and time searches of
>>> the form date:since..until, where "since" and "until" are expressions
>>> understood by the previously added date/time parser, to restrict the
>>> results to messages within a particular time range (based on the Date:
>>> header).
>>>
>>> If "since" or "until" describes date/time at an accuracy of days or
>>> less, the values are rounded according to the accuracy, towards past
>>> for "since" and towards future for "until". For example,
>>> date:november..yesterday would match from the beginning of November
>>> until the end of yesterday. Expressions such as date:today..today
>>> means since the beginning of today until the end of today.
>>
>> IMO this is totally unintuitive and not how the range should work.
>> date:foo..bar should return messages whose date >= foo and < bar.  So
>> for instance date:november..yesterday should return messages whose date
>> is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
>> yesterdays messages one would do: date:yesterday..today.
>
>
> I find yesterday..yesterday to return whole yesterday's messages more
> intuitive than that returning zero messages and requiring yesterday..today
> to see messages sent yesterday. However, I've noticed that range
> described as -1day..-1day (if that syntax is/were supported)
> that would be a bit confusing (In yesterday's case I think the length
> of 'yesterday' is 24h, but in '-1day' the lenght is one second
> (or something))

"yesterday" equals "1 day", so you can use date:yesterday..yesterday and
date:1d..1d interchangeably.

> Anyway, this just emphasizes that this is confusing matter; we need
> a good idiom to comprehend this issue...

I find "since" rounding towards past and "until" rounding towards future
a very simple rule. But YMMV.

One technical aspect is preparing for handling date:expr *without*
range, for example date:yesterday, in the future (this is currently not
supported by xapian). Intuitively that should mean all messages received
yesterday. Because the date parser does not see the range (or lack of
it) at all (and this is very much by design), the glue layer in notmuch
lib between the parser and xapian should handle it gracefully, with no
understanding of expr itself. The obvious and simple way to handle that
is to just duplicate expr on both sides of the range, and date:yesterday
would equal date:yesterday..yesterday, in a way that is very simple to
implement and explain to users.


BR,
Jani.

[PATCH v3 6/9] lib: add date range query support

[Tomi Ollila]

On Thu, Sep 13 2012, Michal Nazarewicz  wrote:

> On Wed, Sep 12 2012, Jani Nikula wrote:
>> Add a custom value range processor to enable date and time searches of
>> the form date:since..until, where "since" and "until" are expressions
>> understood by the previously added date/time parser, to restrict the
>> results to messages within a particular time range (based on the Date:
>> header).
>>
>> If "since" or "until" describes date/time at an accuracy of days or
>> less, the values are rounded according to the accuracy, towards past
>> for "since" and towards future for "until". For example,
>> date:november..yesterday would match from the beginning of November
>> until the end of yesterday. Expressions such as date:today..today
>> means since the beginning of today until the end of today.
>
> IMO this is totally unintuitive and not how the range should work.
> date:foo..bar should return messages whose date >= foo and < bar.  So
> for instance date:november..yesterday should return messages whose date
> is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
> yesterdays messages one would do: date:yesterday..today.


I find yesterday..yesterday to return whole yesterday's messages more
intuitive than that returning zero messages and requiring yesterday..today
to see messages sent yesterday. However, I've noticed that range
described as -1day..-1day (if that syntax is/were supported)
that would be a bit confusing (In yesterday's case I think the length
of 'yesterday' is 24h, but in '-1day' the lenght is one second
(or something))

Anyway, this just emphasizes that this is confusing matter; we need
a good idiom to comprehend this issue...

Tomi


>
>> Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
>> specify date:..until or date:since.. to not limit the start or end
>> date, respectively.
>>
>> CAVEATS:
>>
>> Xapian does not support spaces in range expressions. You can replace
>> the spaces with '_', or (in most cases) '-', or (in some cases) leave
>> the spaces out altogether.
>
> -- 
> Best regards, _ _
> .o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
> ..o | Computer Science,  Micha? ?mina86? Nazarewicz(o o)
> ooo + google.com>--ooO--(_)--Ooo--___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

On Wed, Sep 12 2012, Jani Nikula wrote:
> Add a custom value range processor to enable date and time searches of
> the form date:since..until, where "since" and "until" are expressions
> understood by the previously added date/time parser, to restrict the
> results to messages within a particular time range (based on the Date:
> header).
>
> If "since" or "until" describes date/time at an accuracy of days or
> less, the values are rounded according to the accuracy, towards past
> for "since" and towards future for "until". For example,
> date:november..yesterday would match from the beginning of November
> until the end of yesterday. Expressions such as date:today..today
> means since the beginning of today until the end of today.

IMO this is totally unintuitive and not how the range should work.
date:foo..bar should return messages whose date >= foo and < bar.  So
for instance date:november..yesterday should return messages whose date
is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
yesterdays messages one would do: date:yesterday..today.

> Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
> specify date:..until or date:since.. to not limit the start or end
> date, respectively.
>
> CAVEATS:
>
> Xapian does not support spaces in range expressions. You can replace
> the spaces with '_', or (in most cases) '-', or (in some cases) leave
> the spaces out altogether.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Micha? ?mina86? Nazarewicz(o o)
ooo +--ooO--(_)--Ooo--
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 

[PATCH v3 6/9] lib: add date range query support

[David Bremner]

Michal Nazarewicz  writes:

> IMO this is totally unintuitive and not how the range should work.
> date:foo..bar should return messages whose date >= foo and < bar.  So
> for instance date:november..yesterday should return messages whose date
> is > 2012/11/01 00:00:00 and < 2012/09/12 00:00:00.  So to get
> yesterdays messages one would do: date:yesterday..today.

I don't find ranges being half-open by default to be very
intuitive. Perhaps I don't program in python enough.

d

[PATCH v3 6/9] lib: add date range query support

[Jani Nikula]

Add a custom value range processor to enable date and time searches of
the form date:since..until, where "since" and "until" are expressions
understood by the previously added date/time parser, to restrict the
results to messages within a particular time range (based on the Date:
header).

If "since" or "until" describes date/time at an accuracy of days or
less, the values are rounded according to the accuracy, towards past
for "since" and towards future for "until". For example,
date:november..yesterday would match from the beginning of November
until the end of yesterday. Expressions such as date:today..today
means since the beginning of today until the end of today.

Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
specify date:..until or date:since.. to not limit the start or end
date, respectively.

CAVEATS:

Xapian does not support spaces in range expressions. You can replace
the spaces with '_', or (in most cases) '-', or (in some cases) leave
the spaces out altogether.

Entering date:expr without ".." (for example date:yesterday) will not
work as you might expect. You can achieve the expected result by
duplicating the expr both sides of ".." (for example
date:yesterday..yesterday).

Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
produce an error either.

Signed-off-by: Jani Nikula 
---
 lib/Makefile.local |1 +
 lib/database-private.h |1 +
 lib/database.cc|5 +
 lib/parse-time-vrp.cc  |   40 
 lib/parse-time-vrp.h   |   19 +++
 5 files changed, 66 insertions(+)
 create mode 100644 lib/parse-time-vrp.cc
 create mode 100644 lib/parse-time-vrp.h

diff --git a/lib/Makefile.local b/lib/Makefile.local
index d1635cf..6c0f42f 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -58,6 +58,7 @@ libnotmuch_c_srcs =   \

 libnotmuch_cxx_srcs =  \
$(dir)/database.cc  \
+   $(dir)/parse-time-vrp.cc\
$(dir)/directory.cc \
$(dir)/index.cc \
$(dir)/message.cc   \
diff --git a/lib/database-private.h b/lib/database-private.h
index 88532d5..d3e65fd 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -52,6 +52,7 @@ struct _notmuch_database {
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;
+Xapian::ValueRangeProcessor *date_range_processor;
 };

 /* Return the list of terms from the given iterator matching a prefix.
diff --git a/lib/database.cc b/lib/database.cc
index 761dc1a..4df3217 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -19,6 +19,7 @@
  */

 #include "database-private.h"
+#include "parse-time-vrp.h"

 #include 

@@ -710,12 +711,14 @@ notmuch_database_open (const char *path,
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);
+   notmuch->date_range_processor = new ParseTimeValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);

notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
notmuch->query_parser->set_database (*notmuch->xapian_db);
notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
notmuch->query_parser->set_stemming_strategy 
(Xapian::QueryParser::STEM_SOME);
notmuch->query_parser->add_valuerangeprocessor 
(notmuch->value_range_processor);
+   notmuch->query_parser->add_valuerangeprocessor 
(notmuch->date_range_processor);

for (i = 0; i < ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
prefix_t *prefix = _PREFIX_EXTERNAL[i];
@@ -778,6 +781,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch->xapian_db = NULL;
 delete notmuch->value_range_processor;
 notmuch->value_range_processor = NULL;
+delete notmuch->date_range_processor;
+notmuch->date_range_processor = NULL;
 }

 void
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
new file mode 100644
index 000..148c117
--- /dev/null
+++ b/lib/parse-time-vrp.cc
@@ -0,0 +1,40 @@
+
+#include "database-private.h"
+#include "parse-time-vrp.h"
+#include "parse-time-string.h"
+
+#define PREFIX "date:"
+
+/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
+Xapian::valueno
+ParseTimeValueRangeProcessor::operator() (std::string , std::string )
+{
+time_t t, now;
+
+/* Require date: prefix in start of the range... */
+if (STRNCMP_LITERAL (begin.c_str (), PREFIX))
+   return Xapian::BAD_VALUENO;
+
+/* ...and remove it. */
+begin.erase (0, sizeof (PREFIX) - 1);
+
+/* Use the same 'now' for begin and end. */
+if (time () == (time_t) -1)
+   return Xapian::BAD_VALUENO;
+
+if (!begin.empty ()) {
+   if (parse_time_string (begin.c_str (), , , PARSE_TIME_ROUND_DOWN))
+   return 

Re: [PATCH v3 6/9] lib: add date range query support

[Michal Nazarewicz]

On Wed, Sep 12 2012, Jani Nikula wrote:
 Add a custom value range processor to enable date and time searches of
 the form date:since..until, where since and until are expressions
 understood by the previously added date/time parser, to restrict the
 results to messages within a particular time range (based on the Date:
 header).

 If since or until describes date/time at an accuracy of days or
 less, the values are rounded according to the accuracy, towards past
 for since and towards future for until. For example,
 date:november..yesterday would match from the beginning of November
 until the end of yesterday. Expressions such as date:today..today
 means since the beginning of today until the end of today.

IMO this is totally unintuitive and not how the range should work.
date:foo..bar should return messages whose date = foo and  bar.  So
for instance date:november..yesterday should return messages whose date
is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
yesterdays messages one would do: date:yesterday..today.

 Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
 specify date:..until or date:since.. to not limit the start or end
 date, respectively.

 CAVEATS:

 Xapian does not support spaces in range expressions. You can replace
 the spaces with '_', or (in most cases) '-', or (in some cases) leave
 the spaces out altogether.

-- 
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz(o o)
ooo +email/xmpp: m...@google.com--ooO--(_)--Ooo--

pgpDvsWRcl4Af.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 6/9] lib: add date range query support

[Tomi Ollila]

On Thu, Sep 13 2012, Michal Nazarewicz min...@mina86.com wrote:

 On Wed, Sep 12 2012, Jani Nikula wrote:
 Add a custom value range processor to enable date and time searches of
 the form date:since..until, where since and until are expressions
 understood by the previously added date/time parser, to restrict the
 results to messages within a particular time range (based on the Date:
 header).

 If since or until describes date/time at an accuracy of days or
 less, the values are rounded according to the accuracy, towards past
 for since and towards future for until. For example,
 date:november..yesterday would match from the beginning of November
 until the end of yesterday. Expressions such as date:today..today
 means since the beginning of today until the end of today.

 IMO this is totally unintuitive and not how the range should work.
 date:foo..bar should return messages whose date = foo and  bar.  So
 for instance date:november..yesterday should return messages whose date
 is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
 yesterdays messages one would do: date:yesterday..today.


I find yesterday..yesterday to return whole yesterday's messages more
intuitive than that returning zero messages and requiring yesterday..today
to see messages sent yesterday. However, I've noticed that range
described as -1day..-1day (if that syntax is/were supported)
that would be a bit confusing (In yesterday's case I think the length
of 'yesterday' is 24h, but in '-1day' the lenght is one second
(or something))

Anyway, this just emphasizes that this is confusing matter; we need
a good idiom to comprehend this issue...

Tomi



 Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
 specify date:..until or date:since.. to not limit the start or end
 date, respectively.

 CAVEATS:

 Xapian does not support spaces in range expressions. You can replace
 the spaces with '_', or (in most cases) '-', or (in some cases) leave
 the spaces out altogether.

 -- 
 Best regards, _ _
 .o. | Liege of Serenely Enlightened Majesty of  o' \,=./ `o
 ..o | Computer Science,  Michał “mina86” Nazarewicz(o o)
 ooo +email/xmpp: 
 m...@google.com--ooO--(_)--Ooo--___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 6/9] lib: add date range query support

[David Bremner]

Michal Nazarewicz min...@mina86.com writes:

 IMO this is totally unintuitive and not how the range should work.
 date:foo..bar should return messages whose date = foo and  bar.  So
 for instance date:november..yesterday should return messages whose date
 is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
 yesterdays messages one would do: date:yesterday..today.

I don't find ranges being half-open by default to be very
intuitive. Perhaps I don't program in python enough.

d

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 6/9] lib: add date range query support

[Jani Nikula]

On Thu, 13 Sep 2012, Tomi Ollila tomi.oll...@iki.fi wrote:
 On Thu, Sep 13 2012, Michal Nazarewicz min...@mina86.com wrote:

 On Wed, Sep 12 2012, Jani Nikula wrote:
 Add a custom value range processor to enable date and time searches of
 the form date:since..until, where since and until are expressions
 understood by the previously added date/time parser, to restrict the
 results to messages within a particular time range (based on the Date:
 header).

 If since or until describes date/time at an accuracy of days or
 less, the values are rounded according to the accuracy, towards past
 for since and towards future for until. For example,
 date:november..yesterday would match from the beginning of November
 until the end of yesterday. Expressions such as date:today..today
 means since the beginning of today until the end of today.

 IMO this is totally unintuitive and not how the range should work.
 date:foo..bar should return messages whose date = foo and  bar.  So
 for instance date:november..yesterday should return messages whose date
 is  2012/11/01 00:00:00 and  2012/09/12 00:00:00.  So to get
 yesterdays messages one would do: date:yesterday..today.


 I find yesterday..yesterday to return whole yesterday's messages more
 intuitive than that returning zero messages and requiring yesterday..today
 to see messages sent yesterday. However, I've noticed that range
 described as -1day..-1day (if that syntax is/were supported)
 that would be a bit confusing (In yesterday's case I think the length
 of 'yesterday' is 24h, but in '-1day' the lenght is one second
 (or something))

yesterday equals 1 day, so you can use date:yesterday..yesterday and
date:1d..1d interchangeably.

 Anyway, this just emphasizes that this is confusing matter; we need
 a good idiom to comprehend this issue...

I find since rounding towards past and until rounding towards future
a very simple rule. But YMMV.

One technical aspect is preparing for handling date:expr *without*
range, for example date:yesterday, in the future (this is currently not
supported by xapian). Intuitively that should mean all messages received
yesterday. Because the date parser does not see the range (or lack of
it) at all (and this is very much by design), the glue layer in notmuch
lib between the parser and xapian should handle it gracefully, with no
understanding of expr itself. The obvious and simple way to handle that
is to just duplicate expr on both sides of the range, and date:yesterday
would equal date:yesterday..yesterday, in a way that is very simple to
implement and explain to users.


BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 6/9] lib: add date range query support

[Jani Nikula]

Add a custom value range processor to enable date and time searches of
the form date:since..until, where since and until are expressions
understood by the previously added date/time parser, to restrict the
results to messages within a particular time range (based on the Date:
header).

If since or until describes date/time at an accuracy of days or
less, the values are rounded according to the accuracy, towards past
for since and towards future for until. For example,
date:november..yesterday would match from the beginning of November
until the end of yesterday. Expressions such as date:today..today
means since the beginning of today until the end of today.

Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
specify date:..until or date:since.. to not limit the start or end
date, respectively.

CAVEATS:

Xapian does not support spaces in range expressions. You can replace
the spaces with '_', or (in most cases) '-', or (in some cases) leave
the spaces out altogether.

Entering date:expr without .. (for example date:yesterday) will not
work as you might expect. You can achieve the expected result by
duplicating the expr both sides of .. (for example
date:yesterday..yesterday).

Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
produce an error either.

Signed-off-by: Jani Nikula j...@nikula.org
---
 lib/Makefile.local |1 +
 lib/database-private.h |1 +
 lib/database.cc|5 +
 lib/parse-time-vrp.cc  |   40 
 lib/parse-time-vrp.h   |   19 +++
 5 files changed, 66 insertions(+)
 create mode 100644 lib/parse-time-vrp.cc
 create mode 100644 lib/parse-time-vrp.h

diff --git a/lib/Makefile.local b/lib/Makefile.local
index d1635cf..6c0f42f 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -58,6 +58,7 @@ libnotmuch_c_srcs =   \
 
 libnotmuch_cxx_srcs =  \
$(dir)/database.cc  \
+   $(dir)/parse-time-vrp.cc\
$(dir)/directory.cc \
$(dir)/index.cc \
$(dir)/message.cc   \
diff --git a/lib/database-private.h b/lib/database-private.h
index 88532d5..d3e65fd 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -52,6 +52,7 @@ struct _notmuch_database {
 Xapian::QueryParser *query_parser;
 Xapian::TermGenerator *term_gen;
 Xapian::ValueRangeProcessor *value_range_processor;
+Xapian::ValueRangeProcessor *date_range_processor;
 };
 
 /* Return the list of terms from the given iterator matching a prefix.
diff --git a/lib/database.cc b/lib/database.cc
index 761dc1a..4df3217 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -19,6 +19,7 @@
  */
 
 #include database-private.h
+#include parse-time-vrp.h
 
 #include iostream
 
@@ -710,12 +711,14 @@ notmuch_database_open (const char *path,
notmuch-term_gen = new Xapian::TermGenerator;
notmuch-term_gen-set_stemmer (Xapian::Stem (english));
notmuch-value_range_processor = new Xapian::NumberValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);
+   notmuch-date_range_processor = new ParseTimeValueRangeProcessor 
(NOTMUCH_VALUE_TIMESTAMP);
 
notmuch-query_parser-set_default_op (Xapian::Query::OP_AND);
notmuch-query_parser-set_database (*notmuch-xapian_db);
notmuch-query_parser-set_stemmer (Xapian::Stem (english));
notmuch-query_parser-set_stemming_strategy 
(Xapian::QueryParser::STEM_SOME);
notmuch-query_parser-add_valuerangeprocessor 
(notmuch-value_range_processor);
+   notmuch-query_parser-add_valuerangeprocessor 
(notmuch-date_range_processor);
 
for (i = 0; i  ARRAY_SIZE (BOOLEAN_PREFIX_EXTERNAL); i++) {
prefix_t *prefix = BOOLEAN_PREFIX_EXTERNAL[i];
@@ -778,6 +781,8 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch-xapian_db = NULL;
 delete notmuch-value_range_processor;
 notmuch-value_range_processor = NULL;
+delete notmuch-date_range_processor;
+notmuch-date_range_processor = NULL;
 }
 
 void
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
new file mode 100644
index 000..148c117
--- /dev/null
+++ b/lib/parse-time-vrp.cc
@@ -0,0 +1,40 @@
+
+#include database-private.h
+#include parse-time-vrp.h
+#include parse-time-string.h
+
+#define PREFIX date:
+
+/* See *ValueRangeProcessor in xapian-core/api/valuerangeproc.cc */
+Xapian::valueno
+ParseTimeValueRangeProcessor::operator() (std::string begin, std::string end)
+{
+time_t t, now;
+
+/* Require date: prefix in start of the range... */
+if (STRNCMP_LITERAL (begin.c_str (), PREFIX))
+   return Xapian::BAD_VALUENO;
+
+/* ...and remove it. */
+begin.erase (0, sizeof (PREFIX) - 1);
+
+/* Use the same 'now' for begin and end. */
+if (time (now) == (time_t) -1)
+   return Xapian::BAD_VALUENO;
+
+if (!begin.empty ()) {
+   if (parse_time_string (begin.c_str (), t, now, PARSE_TIME_ROUND_DOWN))
+   return Xapian::BAD_VALUENO;