folder and path completely broken in HEAD?

[Jameson Graef Rollins]

On Sat, May 03 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> First, are there people out there who do not use a collection of maildir
> directories, with all mail in cur and new?

o/ I completely abandoned the usage of separate mail "folders" since I
started using notmuch.  All my mail now just goes into a single maildir.
I consider this one of notmuch's awesome features.  It's honestly been a
bit perplexing to me that folks have put so much work into
parsing/indexing paths and folders given that it seems to me that
notmuch has utterly obviated the need to worry about such things (thank
god!).

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: 

Re: folder and path completely broken in HEAD?

[Jameson Graef Rollins]

On Sat, May 03 2014, dm-list-email-notm...@scs.stanford.edu wrote:
 First, are there people out there who do not use a collection of maildir
 directories, with all mail in cur and new?

o/ I completely abandoned the usage of separate mail folders since I
started using notmuch.  All my mail now just goes into a single maildir.
I consider this one of notmuch's awesome features.  It's honestly been a
bit perplexing to me that folks have put so much work into
parsing/indexing paths and folders given that it seems to me that
notmuch has utterly obviated the need to worry about such things (thank
god!).

jamie.


pgpRJXuRxDyWc.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[Mark Walters]

> The trick here is that it's easy to miss people who are happy with
> current functionality. Adding functionality to address newly-identified
> use cases makes a lot of sense. But removing functionality runs the risk
> of only discovering that people were relying on it after the fact,
> (Which seems to have happened here).

Yes indeed. I think one thing that I, at least, hadn't realised is that
people have lots of strange mail layouts often to work around problems
in mail agents (notmuch or others) or filesystem limitations etc.

>>> The idea of path: is that it's the exact filesystem directory, relative
>>> from the maildir root, with an rsync like ** syntax for recursive
>>> matching.
>
> This definition of "path:" seems good. It covers a lot of cases that the
> original "folder:" did not and allows precise, predictable control.
>
>>> Turns out people want folder: to hide maildir implementation
>>> details like cur and new.
>
> Which is something that "folder:" always did.
>
>>> These are not compatible, or you need to add a syntax that's not easy
>>> or discoverable.
>
> OK. So I won't argue that we don't need two different syntaxes here. But
> I will continue to discuss a use case that was addressed before and is
> no longer.
>
>> I think many of us would agree, but there were users who wanted to
>> distinguish new and cur, and in at least one case, the toplevel as
>> well.
>
> So now, "path:" allows for that, right?
>
> My concern is not so much that "path:" was added to address new things,
> but more than "folder:" was modified in a way that dropped useful
> functionality, (beyond just fixing bugs in "folder:" such as the
> accidental support of stemming).

One possibly perverse remark: it wouldn't surprise me if someone
actually used the stemming. I have used folders called old older and
oldest and that probably can be excluded by stemming.

All I am really saying is that any change we made was going to break
some people's setups.

>> I think it is unfortunate that we need two variants, but I think they do
>> do different things.
>
> I'll accept that.
>
> But, while I have heard a good definition of "path:", (see above), I
> haven't heard a good one for "folder:" yet. Can you provide that?

I think folder:foo/bar means "In the maildir exactly foo/bar"

>> Also, I think any single user will find one matches their setup and
>> use that one essentially exclusively:
>
> The current discussion is evidence against that. We have a user of
> folder: that can no longer get at the desired functionality, (that used
> to exist).

Sorry I wasn't trying to suggest that the current setup does everything
people want (clearly not!): just answering your question about the
differences being confusing. since each user will probably only use one
of them they probably become accustomed to its use.

Indeed, I think of the choice as being analogous to allowing the user to
choose which path scheme they like (so sort of like a customisation):
maildir folder based or filesystem path based.

>> Indeed, it may be that a third option of roughly a maildir++: search term
>> might solve David's use case.
>
> Or just making "folder:" index each term of a filesytem path like it
> used to do. And if that doesn't give sufficient control to some users,
> then "path:" is available.

We could do this. It might break things for user who are using the new
syntax  Maybe we could make an initial / match the root of the
maildir store. But I think some people will dislike any of the options.

>
> I've already lost what I would have preferred, (a single syntax for all
> use cases---which was lost not to a design problem, but simply the
> implementation difficulty of requiring a custom query parser). I really
> would not like to see things continue down to have a *third* syntax.

So this third syntax would fit with my view of it being a customisation
like thing.

Best wishes

Mark

Re: folder and path completely broken in HEAD?

[Mark Walters]

 The trick here is that it's easy to miss people who are happy with
 current functionality. Adding functionality to address newly-identified
 use cases makes a lot of sense. But removing functionality runs the risk
 of only discovering that people were relying on it after the fact,
 (Which seems to have happened here).

Yes indeed. I think one thing that I, at least, hadn't realised is that
people have lots of strange mail layouts often to work around problems
in mail agents (notmuch or others) or filesystem limitations etc.

 The idea of path: is that it's the exact filesystem directory, relative
 from the maildir root, with an rsync like ** syntax for recursive
 matching.

 This definition of path: seems good. It covers a lot of cases that the
 original folder: did not and allows precise, predictable control.

 Turns out people want folder: to hide maildir implementation
 details like cur and new.

 Which is something that folder: always did.

 These are not compatible, or you need to add a syntax that's not easy
 or discoverable.

 OK. So I won't argue that we don't need two different syntaxes here. But
 I will continue to discuss a use case that was addressed before and is
 no longer.

 I think many of us would agree, but there were users who wanted to
 distinguish new and cur, and in at least one case, the toplevel as
 well.

 So now, path: allows for that, right?

 My concern is not so much that path: was added to address new things,
 but more than folder: was modified in a way that dropped useful
 functionality, (beyond just fixing bugs in folder: such as the
 accidental support of stemming).

One possibly perverse remark: it wouldn't surprise me if someone
actually used the stemming. I have used folders called old older and
oldest and that probably can be excluded by stemming.

All I am really saying is that any change we made was going to break
some people's setups.

 I think it is unfortunate that we need two variants, but I think they do
 do different things.

 I'll accept that.

 But, while I have heard a good definition of path:, (see above), I
 haven't heard a good one for folder: yet. Can you provide that?

I think folder:foo/bar means In the maildir exactly foo/bar

 Also, I think any single user will find one matches their setup and
 use that one essentially exclusively:

 The current discussion is evidence against that. We have a user of
 folder: that can no longer get at the desired functionality, (that used
 to exist).

Sorry I wasn't trying to suggest that the current setup does everything
people want (clearly not!): just answering your question about the
differences being confusing. since each user will probably only use one
of them they probably become accustomed to its use.

Indeed, I think of the choice as being analogous to allowing the user to
choose which path scheme they like (so sort of like a customisation):
maildir folder based or filesystem path based.

 Indeed, it may be that a third option of roughly a maildir++: search term
 might solve David's use case.

 Or just making folder: index each term of a filesytem path like it
 used to do. And if that doesn't give sufficient control to some users,
 then path: is available.

We could do this. It might break things for user who are using the new
syntax  Maybe we could make an initial / match the root of the
maildir store. But I think some people will dislike any of the options.


 I've already lost what I would have preferred, (a single syntax for all
 use cases---which was lost not to a design problem, but simply the
 implementation difficulty of requiring a custom query parser). I really
 would not like to see things continue down to have a *third* syntax.

So this third syntax would fit with my view of it being a customisation
like thing.

Best wishes

Mark

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[Mark Walters]

On Mon, 05 May 2014, Jani Nikula  wrote:
> Hi Carl -
>
> On Tue, 06 May 2014, Carl Worth  wrote:
>> dm-list-email-notmuch at scs.stanford.edu writes:
>>> However, currently it seems strange that there are *two* different
>>> search terms (folder and path), and that neither one lets you search for
>>> a portion of your folder name.
>>
>> For what it's worth, I totally agree. I'm guilty as far as sitting out
>> of the detailed design discussions, (I don't use any sort of
>> folder-based searching, so I don't care too much). I was aware of the
>> problems of the original "folder:" code I wrote, and I was happy to hear
>> that people were addressing those problems.
>>
>> But it's terribly strange to me that notmuch now has two different
>> search terms that overlap so much in functionality. I know that I will
>> never trust myself to be able to distinguish/describe the folder:
>> vs. path: semantics without consulting the documentation. And that's
>> discouraging.
>
> The discussions about this were lengthy and tedious, and I was glad we
> eventually reached some consensus about what we wanted. It's always
> disappointing to find out one hasn't found the solution to satisfy
> everyone, but in the end I think I'm happier we were able to reach a
> decision, do something about the real issues people were facing with
> folder: and move on, rather than just grind to a halt. I think we were
> pretty close to everyone just dropping the ball and letting the folder:
> prefix be, warts and all.

I would just like to second what Jani said. There was a lot of
discussion and, at the time, the outcome covered all use cases that
anyone showed any sign of wanting. And these included things like
distinguishing between messages in cur or new or the toplevel of a maildir,
wanting to search all subdirectories of a particular path (if eg the
main mail folder is split into 256 subsirectories 00..ff).

> The idea of path: is that it's the exact filesystem directory, relative
> from the maildir root, with an rsync like ** syntax for recursive
> matching. Turns out people want folder: to hide maildir implementation
> details like cur and new. These are not compatible, or you need to add a
> syntax that's not easy or discoverable.
>
> path: is now pretty much complete, and allows one to do robust scripting
> that won't break in surprising ways. folder: is something we can still
> add new functionality into, for example fancier interpretations of
> maildir++, or anchoring if we ever get the custom query parser.
>
>> I think the original "folder:" shortcomings could have been addressed
>> without adding two terms, and also without losing some functionality,
>> (as shown in David's use case).
>>
>> I would have liked to have seen some explicit syntax for anchoring the
>> beginning and end of the directory name, (which could have then been
>> re-used for anchoring subject: or even some future header: prefix).
>
> As I understood it, that would have required writing a custom query
> parser, a significant effort. At least nobody came up with a scheme to
> do the anchoring without the parser while addressing the other issues
> with the old folder: prefix.
>
>> I've always thought that the "cur" and "new" directories were somewhere
>> on the spectrum between pointless and annoying. The idea with the
>> original "folder:" indexing was to implicitly ignore these, (when both
>> existed).

I think many of us would agree, but there were users who wanted to
distinguish new and cur, and in at least one case, the toplevel as well.


>>
>> It seems that the new "folder:" continues this idea, while the new
>> "path:" does not.
>
> Correct.
>
>> Meanwhile, the new "folder:" anchors the search to the beginning of
>> the directory, while "path:" does not.
>
> Incorrect (or I don't understand you).
>
>> And finally, "path:" adds a magic syntax to do hierarchical matching
>> while "folder:" does not.
>
> Correct.
>
>> That's an odd hodgepodge of non-orthogonal distinction in
>> functionality.

>
> I'm sorry to hear you think that way. One is verbatim filesystem path,
> the other hides mail store folder implementation details as a
> convenience.

I think it is unfortunate that we need two variants, but I think they do
do different things. Also, I think any single user will find one matches
their setup and use that one essentially exclusively: if everything is
in maildirs then you probably want folder:, if you want exact control
then use path:.

Indeed, it may be that a third option of roughly a maildir++: search term
might solve David's use case.

Best wishes

Mark

folder and path completely broken in HEAD?

[Jani Nikula]

Hi Carl -

On Tue, 06 May 2014, Carl Worth  wrote:
> dm-list-email-notmuch at scs.stanford.edu writes:
>> However, currently it seems strange that there are *two* different
>> search terms (folder and path), and that neither one lets you search for
>> a portion of your folder name.
>
> For what it's worth, I totally agree. I'm guilty as far as sitting out
> of the detailed design discussions, (I don't use any sort of
> folder-based searching, so I don't care too much). I was aware of the
> problems of the original "folder:" code I wrote, and I was happy to hear
> that people were addressing those problems.
>
> But it's terribly strange to me that notmuch now has two different
> search terms that overlap so much in functionality. I know that I will
> never trust myself to be able to distinguish/describe the folder:
> vs. path: semantics without consulting the documentation. And that's
> discouraging.

The discussions about this were lengthy and tedious, and I was glad we
eventually reached some consensus about what we wanted. It's always
disappointing to find out one hasn't found the solution to satisfy
everyone, but in the end I think I'm happier we were able to reach a
decision, do something about the real issues people were facing with
folder: and move on, rather than just grind to a halt. I think we were
pretty close to everyone just dropping the ball and letting the folder:
prefix be, warts and all.

The idea of path: is that it's the exact filesystem directory, relative
from the maildir root, with an rsync like ** syntax for recursive
matching. Turns out people want folder: to hide maildir implementation
details like cur and new. These are not compatible, or you need to add a
syntax that's not easy or discoverable.

path: is now pretty much complete, and allows one to do robust scripting
that won't break in surprising ways. folder: is something we can still
add new functionality into, for example fancier interpretations of
maildir++, or anchoring if we ever get the custom query parser.

> I think the original "folder:" shortcomings could have been addressed
> without adding two terms, and also without losing some functionality,
> (as shown in David's use case).
>
> I would have liked to have seen some explicit syntax for anchoring the
> beginning and end of the directory name, (which could have then been
> re-used for anchoring subject: or even some future header: prefix).

As I understood it, that would have required writing a custom query
parser, a significant effort. At least nobody came up with a scheme to
do the anchoring without the parser while addressing the other issues
with the old folder: prefix.

> I've always thought that the "cur" and "new" directories were somewhere
> on the spectrum between pointless and annoying. The idea with the
> original "folder:" indexing was to implicitly ignore these, (when both
> existed).
>
> It seems that the new "folder:" continues this idea, while the new
> "path:" does not.

Correct.

> Meanwhile, the new "folder:" anchors the search to the beginning of
> the directory, while "path:" does not.

Incorrect (or I don't understand you).

> And finally, "path:" adds a magic syntax to do hierarchical matching
> while "folder:" does not.

Correct.

> That's an odd hodgepodge of non-orthogonal distinction in
> functionality.

I'm sorry to hear you think that way. One is verbatim filesystem path,
the other hides mail store folder implementation details as a
convenience.


BR,
Jani.

Re: folder and path completely broken in HEAD?

[Mark Walters]

On Mon, 05 May 2014, Jani Nikula j...@nikula.org wrote:
 Hi Carl -

 On Tue, 06 May 2014, Carl Worth cwo...@cworth.org wrote:
 dm-list-email-notm...@scs.stanford.edu writes:
 However, currently it seems strange that there are *two* different
 search terms (folder and path), and that neither one lets you search for
 a portion of your folder name.

 For what it's worth, I totally agree. I'm guilty as far as sitting out
 of the detailed design discussions, (I don't use any sort of
 folder-based searching, so I don't care too much). I was aware of the
 problems of the original folder: code I wrote, and I was happy to hear
 that people were addressing those problems.

 But it's terribly strange to me that notmuch now has two different
 search terms that overlap so much in functionality. I know that I will
 never trust myself to be able to distinguish/describe the folder:
 vs. path: semantics without consulting the documentation. And that's
 discouraging.

 The discussions about this were lengthy and tedious, and I was glad we
 eventually reached some consensus about what we wanted. It's always
 disappointing to find out one hasn't found the solution to satisfy
 everyone, but in the end I think I'm happier we were able to reach a
 decision, do something about the real issues people were facing with
 folder: and move on, rather than just grind to a halt. I think we were
 pretty close to everyone just dropping the ball and letting the folder:
 prefix be, warts and all.

I would just like to second what Jani said. There was a lot of
discussion and, at the time, the outcome covered all use cases that
anyone showed any sign of wanting. And these included things like
distinguishing between messages in cur or new or the toplevel of a maildir,
wanting to search all subdirectories of a particular path (if eg the
main mail folder is split into 256 subsirectories 00..ff).

 The idea of path: is that it's the exact filesystem directory, relative
 from the maildir root, with an rsync like ** syntax for recursive
 matching. Turns out people want folder: to hide maildir implementation
 details like cur and new. These are not compatible, or you need to add a
 syntax that's not easy or discoverable.

 path: is now pretty much complete, and allows one to do robust scripting
 that won't break in surprising ways. folder: is something we can still
 add new functionality into, for example fancier interpretations of
 maildir++, or anchoring if we ever get the custom query parser.

 I think the original folder: shortcomings could have been addressed
 without adding two terms, and also without losing some functionality,
 (as shown in David's use case).

 I would have liked to have seen some explicit syntax for anchoring the
 beginning and end of the directory name, (which could have then been
 re-used for anchoring subject: or even some future header: prefix).

 As I understood it, that would have required writing a custom query
 parser, a significant effort. At least nobody came up with a scheme to
 do the anchoring without the parser while addressing the other issues
 with the old folder: prefix.

 I've always thought that the cur and new directories were somewhere
 on the spectrum between pointless and annoying. The idea with the
 original folder: indexing was to implicitly ignore these, (when both
 existed).

I think many of us would agree, but there were users who wanted to
distinguish new and cur, and in at least one case, the toplevel as well.



 It seems that the new folder: continues this idea, while the new
 path: does not.

 Correct.

 Meanwhile, the new folder: anchors the search to the beginning of
 the directory, while path: does not.

 Incorrect (or I don't understand you).

 And finally, path: adds a magic syntax to do hierarchical matching
 while folder: does not.

 Correct.

 That's an odd hodgepodge of non-orthogonal distinction in
 functionality.


 I'm sorry to hear you think that way. One is verbatim filesystem path,
 the other hides mail store folder implementation details as a
 convenience.

I think it is unfortunate that we need two variants, but I think they do
do different things. Also, I think any single user will find one matches
their setup and use that one essentially exclusively: if everything is
in maildirs then you probably want folder:, if you want exact control
then use path:.

Indeed, it may be that a third option of roughly a maildir++: search term
might solve David's use case.

Best wishes

Mark


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Carl Worth]

dm-list-email-notm...@scs.stanford.edu writes:
 First, thanks for the response.  The responsiveness and friendliness of
 the notmuch mailing list goes a long way towards compensating for any
 missing features / customizability one might want.

I'm delighted to hear that the notmuch community maintains such a
helpful and friendly atmosphere.

 However, currently it seems strange that there are *two* different
 search terms (folder and path), and that neither one lets you search for
 a portion of your folder name.

For what it's worth, I totally agree. I'm guilty as far as sitting out
of the detailed design discussions, (I don't use any sort of
folder-based searching, so I don't care too much). I was aware of the
problems of the original folder: code I wrote, and I was happy to hear
that people were addressing those problems.

But it's terribly strange to me that notmuch now has two different
search terms that overlap so much in functionality. I know that I will
never trust myself to be able to distinguish/describe the folder:
vs. path: semantics without consulting the documentation. And that's
discouraging.

I think the original folder: shortcomings could have been addressed
without adding two terms, and also without losing some functionality,
(as shown in David's use case).

I would have liked to have seen some explicit syntax for anchoring the
beginning and end of the directory name, (which could have then been
re-used for anchoring subject: or even some future header: prefix).

 First, are there people out there who do not use a collection of maildir
 directories, with all mail in cur and new?

When I started notmuch, my entire mail archive was in non-maildir
directories. Since then, the bulk of my mail has been converted to
maildir (by accident more than anything). But I still do create
non-maildir directories for mail occasionally.

One use case for this is when someone forwards me an email (or a thread
of emails) to demonstrate a bug in notmuch. In a case like this, I will
do something like:

mkdir ~/mail/dm-folder-bug-2014-05

and then copy the mail files into that directory. I see no benefit nor
point to creating cur and new directories in a case like this.

 If not, why does notmuch try to find mail in non-mail-directories, and
 why do you need search terms differentiating new and cur?

I've always thought that the cur and new directories were somewhere
on the spectrum between pointless and annoying. The idea with the
original folder: indexing was to implicitly ignore these, (when both
existed).

It seems that the new folder: continues this idea, while the new
path: does not. Meanwhile, the new folder: anchors the search to the
beginning of the directory, while path: does not. And finally, path:
adds a magic syntax to do hierarchical matching while folder: does
not.

That's an odd hodgepodge of non-orthogonal distinction in functionality.

 Conversely, I find it particularly weird that there's no convenient
 way to say stop trying to index stuff that isn't in a maildir (cur or
 new).  You can do the inverse and blacklist files, but then I end up
 with stuff like this in my .notmuch-config:

Like I said, I started with an assumption of a hierarchy of directories
containing nothing but mail files, (that's exactly what I had at the
time).

 
 ignore=dovecot-keywords;dovecot-uidlist;dovecot-uidvalidity;dovecot.index;dovecot.index.cache;dovecot.index.log;dovecot.index.log.2;dovecot.index.search;dovecot.index.search.uids;maildirfolder;

I really do like explicitly describing the things you want to
ignore. One of the worst things that notmuch could do is to silently
fail to index legitimate emails. So I would not be in favor of features
that made that easier.

So the above looks fairly reasonable to me. Though it would also be
reasonable to allow for some wildcards there so that you could have
simply:

 ignore=dovecot*;maildirfolder;

 Second, does anyone out there have a collection with more than a few
 thousand maildirs?

I certainly don't. I have just short of one million messages, but only
841 directories in my mail store. And since that's counting .,
./tmp, ./cur, and ./new for every maildir, I probably only have
about 200 maildirs.

-Carl

-- 
carl.d.wo...@intel.com


pgpVzdGIUyAKY.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Jani Nikula]

Hi Carl -

On Tue, 06 May 2014, Carl Worth cwo...@cworth.org wrote:
 dm-list-email-notm...@scs.stanford.edu writes:
 However, currently it seems strange that there are *two* different
 search terms (folder and path), and that neither one lets you search for
 a portion of your folder name.

 For what it's worth, I totally agree. I'm guilty as far as sitting out
 of the detailed design discussions, (I don't use any sort of
 folder-based searching, so I don't care too much). I was aware of the
 problems of the original folder: code I wrote, and I was happy to hear
 that people were addressing those problems.

 But it's terribly strange to me that notmuch now has two different
 search terms that overlap so much in functionality. I know that I will
 never trust myself to be able to distinguish/describe the folder:
 vs. path: semantics without consulting the documentation. And that's
 discouraging.

The discussions about this were lengthy and tedious, and I was glad we
eventually reached some consensus about what we wanted. It's always
disappointing to find out one hasn't found the solution to satisfy
everyone, but in the end I think I'm happier we were able to reach a
decision, do something about the real issues people were facing with
folder: and move on, rather than just grind to a halt. I think we were
pretty close to everyone just dropping the ball and letting the folder:
prefix be, warts and all.

The idea of path: is that it's the exact filesystem directory, relative
from the maildir root, with an rsync like ** syntax for recursive
matching. Turns out people want folder: to hide maildir implementation
details like cur and new. These are not compatible, or you need to add a
syntax that's not easy or discoverable.

path: is now pretty much complete, and allows one to do robust scripting
that won't break in surprising ways. folder: is something we can still
add new functionality into, for example fancier interpretations of
maildir++, or anchoring if we ever get the custom query parser.

 I think the original folder: shortcomings could have been addressed
 without adding two terms, and also without losing some functionality,
 (as shown in David's use case).

 I would have liked to have seen some explicit syntax for anchoring the
 beginning and end of the directory name, (which could have then been
 re-used for anchoring subject: or even some future header: prefix).

As I understood it, that would have required writing a custom query
parser, a significant effort. At least nobody came up with a scheme to
do the anchoring without the parser while addressing the other issues
with the old folder: prefix.

 I've always thought that the cur and new directories were somewhere
 on the spectrum between pointless and annoying. The idea with the
 original folder: indexing was to implicitly ignore these, (when both
 existed).

 It seems that the new folder: continues this idea, while the new
 path: does not.

Correct.

 Meanwhile, the new folder: anchors the search to the beginning of
 the directory, while path: does not.

Incorrect (or I don't understand you).

 And finally, path: adds a magic syntax to do hierarchical matching
 while folder: does not.

Correct.

 That's an odd hodgepodge of non-orthogonal distinction in
 functionality.

I'm sorry to hear you think that way. One is verbatim filesystem path,
the other hides mail store folder implementation details as a
convenience.


BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Carl Worth]

Jani Nikula j...@nikula.org writes:
 The discussions about this were lengthy and tedious, and I was glad we
 eventually reached some consensus about what we wanted.

This much I do understand. And I apologize if I seem to be raising anew
issues that should finally be behind us.

 Meanwhile, the new folder: anchors the search to the beginning of
 the directory, while path: does not.

 Incorrect (or I don't understand you).
...
 That's an odd hodgepodge of non-orthogonal distinction in
 functionality.

 I'm sorry to hear you think that way. One is verbatim filesystem path,
 the other hides mail store folder implementation details as a
 convenience.

It may be that I simply misunderstood one aspect of path:. If it's
effectively ''anchored'' the same way as folder: then perhaps these
are more similar than I thought. In which case, I apologize for the
noise.

-Carl


pgpepWFsMp21N.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[Jani Nikula]

On Sat, 03 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> Mark Walters  writes:
>
>>> All the way back.  Now you are saying there will be no convenient way to
>>> match just the "mail.class" part without the year?  How very
>>> distressing.  Ugh.
>>
>> Hi
>>
>> I am not quite sure what you are meaning by hierarchically group
>> messages. Searching for path:dir/foo/bar/** should give all messages in
>> all directories beneath dir/foo/bar. 
>
> The problem is that the maildir++ spec disallows such pathnames.  If I
> need compatibility with maildir++ (for instance for an imap server), at
> least on a per-year basis, then my maildirs have to have names like:
>
>2013/.foo.foo
>2013/.foo.bar
>2013/.foo.baz
>2014/.foo.foo
>2014/.foo.bar
>2014/.foo.baz
>
> So if I want a way to aggregate all my foo mail in a single search,
> right now I just ask for folder:foo.  Will there be any equivalent in
> the new notmuch?

Given that we're in feature freeze and ready for an imminent release of
0.18, I think it's pretty safe to say, no.

---

It's not going to help you, but I'll mention a few of the issues the old
folder: search had, which we also had complaints about, and which would
have been quite hard to fix while preserving the behaviour you want. In
short, we considered the old folder: search broken.

Given layout:

Foo/{cur,new}
foo/{cur,new}
fooing/{cur,new}
bar/foo/{cur,new}
cur
new

It was impossible to refer to the top level folder.

It was impossible to refer to foo without also referring to Foo, fooing,
and bar/foo.

In your layout, if you also had 2013/.bar.foo, folder:foo would match
that as well. To not match that, you would have to include each
folder:.foo.xxx in the search.


BR,
Jani.

folder and path completely broken in HEAD?

[Mark Walters]

Hi

On Sat, 03 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> Mark Walters  writes:
>
>>> All the way back.  Now you are saying there will be no convenient way to
>>> match just the "mail.class" part without the year?  How very
>>> distressing.  Ugh.
>>
>> Hi
>>
>> I am not quite sure what you are meaning by hierarchically group
>> messages. Searching for path:dir/foo/bar/** should give all messages in
>> all directories beneath dir/foo/bar. 
>
> The problem is that the maildir++ spec disallows such pathnames.  If I
> need compatibility with maildir++ (for instance for an imap server), at
> least on a per-year basis, then my maildirs have to have names like:
>
>2013/.foo.foo
>2013/.foo.bar
>2013/.foo.baz
>2014/.foo.foo
>2014/.foo.bar
>2014/.foo.baz
>
> So if I want a way to aggregate all my foo mail in a single search,
> right now I just ask for folder:foo.  Will there be any equivalent in
> the new notmuch?

I agree with Jani that for 0.18 this won't be possible. However, there
is probably no urgent reason to upgrade (given you know it will cause
you some problems) so the question is can we add some functionality to
cover this use case. Given the way we do the folder matching I think it
unlikely we would want to add globbing/wildcards to the start of a path
(*).

However, we might be able to support maildir++ as its own prefix. If you
could so searches of the form maildir++:.foo.bar.** which would match
the maildir .foo.bar and all submaildirs++ (eg .foo.bar.a.b etc) would
that be sufficient?

This wouldn't deal with 2013/.foo.bar but would allow .foo.bar.2013

Would this be enough help to make it worth considering?

Best wishes

Mark

(*) if we get a custom parser then it may all be manageable.

folder and path completely broken in HEAD?

[dm-list-email-notm...@scs.stanford.edu]

Jani Nikula  writes:

> It's not going to help you, but I'll mention a few of the issues the old
> folder: search had, which we also had complaints about, and which would
> have been quite hard to fix while preserving the behaviour you want. In
> short, we considered the old folder: search broken.
>
> Given layout:
>
>   Foo/{cur,new}
>   foo/{cur,new}
>   fooing/{cur,new}
>   bar/foo/{cur,new}
>   cur
>   new
>
> It was impossible to refer to the top level folder.
>
> It was impossible to refer to foo without also referring to Foo, fooing,
> and bar/foo.
>
> In your layout, if you also had 2013/.bar.foo, folder:foo would match
> that as well. To not match that, you would have to include each
> folder:.foo.xxx in the search.

First, thanks for the response.  The responsiveness and friendliness of
the notmuch mailing list goes a long way towards compensating for any
missing features / customizability one might want.

I was already aware of the issues you raise, and had worked around them
by just renaming all my mail folders.  I agree that searching for a
particular folder is crucial functionality, and found it weird that I
had to abandon my main top-level mailbox (which I just renamed
.INBOX.Main).

However, currently it seems strange that there are *two* different
search terms (folder and path), and that neither one lets you search for
a portion of your folder name.  Admittedly the old folder code was one
of the parts of the notmuch source that didn't make sense to me (and now
I'm starting to understand why--e.g., the fact that it used stemming,
for instance, was just weird and maybe accidental).

I may be able to hack around this problem in the emacs lisp part or with
a wrapper script.  I'm already having to defadvice around
notmuch-call-notmuch-sexp to implement features I'm missing from
wanderlust and gnus (e.g., the inability to specify regexps matching all
my email addresses).

But to help me understand the current design, can you answer a couple of
questions?


First, are there people out there who do not use a collection of maildir
directories, with all mail in cur and new?  If not, why does notmuch try
to find mail in non-mail-directories, and why do you need search terms
differentiating new and cur?  Conversely, I find it particularly weird
that there's no convenient way to say "stop trying to index stuff that
isn't in a maildir (cur or new)."  You can do the inverse and blacklist
files, but then I end up with stuff like this in my .notmuch-config:


ignore=dovecot-keywords;dovecot-uidlist;dovecot-uidvalidity;dovecot.index;dovecot.index.cache;dovecot.index.log;dovecot.index.log.2;dovecot.index.search;dovecot.index.search.uids;maildirfolder;

(And still I think there are other junk files that get left around by my
imap server or other software, so notmuch new still repeatedly tries to
index junk whenever I run it.)


Second, does anyone out there have a collection with more than a few
thousand maildirs?  I understand some people index millions of messages,
but even with tiny maildirs of a few hundred messages each, that's a
mere 10,000 maildirs.  So for that, who needs any kind of Xapian
indexing fanciness or dedicated XFOLDER prefix?  Scanning the entire set
of XDIRECTORY terms to apply an arbitrary predicate should take
negligible time.  So all you really need is a boolean search term
containing the directory docid (or with the existing schema the ability
to do a prefix search on XFDIRENTRYn:*).

Thanks,
David

folder and path completely broken in HEAD?

[Mark Walters]

On Sat, 03 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> Jani Nikula  writes:
>
>> On Fri, 02 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
>>>
>>> I'm using a pretty standard maildir++ layout.  For example, underneath
>>> my database.path I have a bunch of mail in directories such as:
>>>
>>> .INBOX.Main/{new,cur}
>>> .mail.class/{new,cur}
>>> .mail.voicemail/{new,cur}
>>> ...
>> Here's additional commentary on the specific queries.
>>
>>> linux7$ ./notmuch count folder:mail
>>> 0
>>> linux8$ ./notmuch count folder:.mail
>>> 0
>
> Oh, man.  That's a serious bummer.
>
> Is there any mechanism left that would let me hierarchically group
> messages?  I've got a ton of mail.* folders, and create new ones
> dynamically.  I really want a mechanism to group them hierarchically, so
> I can have a search that matches all current and future mail
> directories.  I organized my whole mail setup around folders because a)
> tags do not provide this kind of hierarchical control, and b) there
> doesn't seem to be a convenient way to apply tags 100% reliably on
> message delivery, whereas I *can* control the folder 100% reliably.
>
> Worse, because of my poor performance, I was hoping to segregate
> messages by year.  So it would be:
>
>   2013/.mail.class
>   2013/.mail.voicemail
>   2014/.mail.class
>   2014/.mail.voicemail
>
> All the way back.  Now you are saying there will be no convenient way to
> match just the "mail.class" part without the year?  How very
> distressing.  Ugh.

Hi

I am not quite sure what you are meaning by hierarchically group
messages. Searching for path:dir/foo/bar/** should give all messages in
all directories beneath dir/foo/bar. 

So for the year example you give you could have directories
mail/class/2013 and mail/class/2014/ etc. 

Anyway if you can describe your use more fully we may be able to
help. And if it can't do what you want it would also give us some idea
of what would need to be changed.

Finally, I am not quite sure what this means

> there doesn't seem to be a convenient way to apply tags 100% reliably
> on message delivery, whereas I *can* control the folder 100% reliably.

Are you wanting to tag messages based on some properties when doing
notmuch new while avoiding races etc? Or are you worried about duplicate
message-ids? Or something else?

Best wishes

Mark




>
> David
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[Suvayu Ali]

On Fri, May 02, 2014 at 09:57:49PM -0700, dm-list-email-notmuch at 
scs.stanford.edu wrote:
> 
> Worse, because of my poor performance, I was hoping to segregate
> messages by year.  So it would be:
> 
>   2013/.mail.class
>   2013/.mail.voicemail
>   2014/.mail.class
>   2014/.mail.voicemail
> 
> All the way back.  Now you are saying there will be no convenient way to
> match just the "mail.class" part without the year?  How very
> distressing.  Ugh.

If you are using shell commands, you can always use:

  $ notmuch count -- folder:201{3,4}/.mail.class

Not sure how to achieve the same from the Emacs UI (or other UIs).

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

folder and path completely broken in HEAD?

[Suvayu Ali]

Hello David,

On Fri, May 02, 2014 at 02:16:07PM -0700, David Mazieres expires 2014-07-31 PDT 
wrote:
> Mark Walters  writes:
> >
> > Before checking other things: have you run notmuch new? That's needed to
> > update the database. It is an irreversible update so notmuch-0.17 will
> > not work with the updated database.
> 
> No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
> The discussion of the NEWS was a bit confusing, so I wanted to check it
> out.  I knew something had to be very wrong.
> 
> Weill the new primitives still allow me to group my mail hierarchically
> in searches?  The new man page is not totally clear on what is being
> matched.

Here are some examples:

  $ notmuch config get database.path
  /home/user/Mail
  $ notmuch count -- folder:Gmail/Tux
  5113
  $ notmuch count -- 'path:Gmail/**'
  87807

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

folder and path completely broken in HEAD?

[Jani Nikula]

On Fri, 02 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> Hey, I'm playing around with the head of the git repository
> (bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
> new folder: prefixes work, as folders are a critical part of how I
> organize my mail.  (Since tags are not hierarchical, folders are the
> best way for me to group mail to a bunch of related addresses, while
> retaining the ability to separate out any mailboxes that become high
> traffic.)
>
> I'm using a pretty standard maildir++ layout.  For example, underneath
> my database.path I have a bunch of mail in directories such as:
>
> .INBOX.Main/{new,cur}
> .mail.class/{new,cur}
> .mail.voicemail/{new,cur}
>
> It used to be the case that if I wanted to read all of my "mail" mail, I
> could search for folder:mail, while to look at just voicemail, I could
> say folder:mail.voicemail, etc.  Now, I can't get anything to match a
> folder predicate period.  For example, using notmuch as notmuch-0.17 and
> ./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

First, read Mark's reply about database upgrade, and heed the notice
about upgrade being automatic but downgrade requiring a reindex of all
your mails.

Here's additional commentary on the specific queries.

> linux7$ ./notmuch count folder:mail
> 0
> linux8$ ./notmuch count folder:.mail
> 0

These two will not match anything.

> linux9$ ./notmuch count folder:.mail.voicemail
> 0

After the upgrade, this should work for .mail.voicemail/{new,cur}.

> linux10$ ./notmuch count folder:.mail.voicemail/cur
> 0

This won't match anything. To specifically search for messages in cur,
you can use the new path prefix, path:.mail.voicemail/cur

> linux 11$ ./notmuch count path:.mail.voicemail
> 0

This will match messages in .mail.voicemail only, and nothing in the cur
or new subdirectories.

> linux 12$ ./notmuch count path:.mail.voicemail/'**'
> 0

I'm not sure your quoting will work, but if you don't have any other
subfolders than cur and new, this should be equal to
folder:.mail.voicemail

> linux 13$ ./notmuch count path:.mail.voicemail/cur 
> 0

This should match messages in the cur subdirectory.

> linux 14$ ./notmuch count folder:mail.voicemail
> 0

This won't match anything.


BR,
Jani.

Re: folder and path completely broken in HEAD?

[Mark Walters]

On Sat, 03 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:
 Jani Nikula j...@nikula.org writes:

 On Fri, 02 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:

 I'm using a pretty standard maildir++ layout.  For example, underneath
 my database.path I have a bunch of mail in directories such as:

 .INBOX.Main/{new,cur}
 .mail.class/{new,cur}
 .mail.voicemail/{new,cur}
 ...
 Here's additional commentary on the specific queries.

 linux7$ ./notmuch count folder:mail
 0
 linux8$ ./notmuch count folder:.mail
 0

 Oh, man.  That's a serious bummer.

 Is there any mechanism left that would let me hierarchically group
 messages?  I've got a ton of mail.* folders, and create new ones
 dynamically.  I really want a mechanism to group them hierarchically, so
 I can have a search that matches all current and future mail
 directories.  I organized my whole mail setup around folders because a)
 tags do not provide this kind of hierarchical control, and b) there
 doesn't seem to be a convenient way to apply tags 100% reliably on
 message delivery, whereas I *can* control the folder 100% reliably.

 Worse, because of my poor performance, I was hoping to segregate
 messages by year.  So it would be:

   2013/.mail.class
   2013/.mail.voicemail
   2014/.mail.class
   2014/.mail.voicemail

 All the way back.  Now you are saying there will be no convenient way to
 match just the mail.class part without the year?  How very
 distressing.  Ugh.

Hi

I am not quite sure what you are meaning by hierarchically group
messages. Searching for path:dir/foo/bar/** should give all messages in
all directories beneath dir/foo/bar. 

So for the year example you give you could have directories
mail/class/2013 and mail/class/2014/ etc. 

Anyway if you can describe your use more fully we may be able to
help. And if it can't do what you want it would also give us some idea
of what would need to be changed.

Finally, I am not quite sure what this means

 there doesn't seem to be a convenient way to apply tags 100% reliably
 on message delivery, whereas I *can* control the folder 100% reliably.

Are you wanting to tag messages based on some properties when doing
notmuch new while avoiding races etc? Or are you worried about duplicate
message-ids? Or something else?

Best wishes

Mark





 David
 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Mark Walters markwalters1...@gmail.com writes:

 All the way back.  Now you are saying there will be no convenient way to
 match just the mail.class part without the year?  How very
 distressing.  Ugh.

 Hi

 I am not quite sure what you are meaning by hierarchically group
 messages. Searching for path:dir/foo/bar/** should give all messages in
 all directories beneath dir/foo/bar. 

The problem is that the maildir++ spec disallows such pathnames.  If I
need compatibility with maildir++ (for instance for an imap server), at
least on a per-year basis, then my maildirs have to have names like:

   2013/.foo.foo
   2013/.foo.bar
   2013/.foo.baz
   2014/.foo.foo
   2014/.foo.bar
   2014/.foo.baz

So if I want a way to aggregate all my foo mail in a single search,
right now I just ask for folder:foo.  Will there be any equivalent in
the new notmuch?

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Jani Nikula]

On Sat, 03 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:
 Mark Walters markwalters1...@gmail.com writes:

 All the way back.  Now you are saying there will be no convenient way to
 match just the mail.class part without the year?  How very
 distressing.  Ugh.

 Hi

 I am not quite sure what you are meaning by hierarchically group
 messages. Searching for path:dir/foo/bar/** should give all messages in
 all directories beneath dir/foo/bar. 

 The problem is that the maildir++ spec disallows such pathnames.  If I
 need compatibility with maildir++ (for instance for an imap server), at
 least on a per-year basis, then my maildirs have to have names like:

2013/.foo.foo
2013/.foo.bar
2013/.foo.baz
2014/.foo.foo
2014/.foo.bar
2014/.foo.baz

 So if I want a way to aggregate all my foo mail in a single search,
 right now I just ask for folder:foo.  Will there be any equivalent in
 the new notmuch?

Given that we're in feature freeze and ready for an imminent release of
0.18, I think it's pretty safe to say, no.

---

It's not going to help you, but I'll mention a few of the issues the old
folder: search had, which we also had complaints about, and which would
have been quite hard to fix while preserving the behaviour you want. In
short, we considered the old folder: search broken.

Given layout:

Foo/{cur,new}
foo/{cur,new}
fooing/{cur,new}
bar/foo/{cur,new}
cur
new

It was impossible to refer to the top level folder.

It was impossible to refer to foo without also referring to Foo, fooing,
and bar/foo.

In your layout, if you also had 2013/.bar.foo, folder:foo would match
that as well. To not match that, you would have to include each
folder:.foo.xxx in the search.


BR,
Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Mark Walters]

Hi

On Sat, 03 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:
 Mark Walters markwalters1...@gmail.com writes:

 All the way back.  Now you are saying there will be no convenient way to
 match just the mail.class part without the year?  How very
 distressing.  Ugh.

 Hi

 I am not quite sure what you are meaning by hierarchically group
 messages. Searching for path:dir/foo/bar/** should give all messages in
 all directories beneath dir/foo/bar. 

 The problem is that the maildir++ spec disallows such pathnames.  If I
 need compatibility with maildir++ (for instance for an imap server), at
 least on a per-year basis, then my maildirs have to have names like:

2013/.foo.foo
2013/.foo.bar
2013/.foo.baz
2014/.foo.foo
2014/.foo.bar
2014/.foo.baz

 So if I want a way to aggregate all my foo mail in a single search,
 right now I just ask for folder:foo.  Will there be any equivalent in
 the new notmuch?

I agree with Jani that for 0.18 this won't be possible. However, there
is probably no urgent reason to upgrade (given you know it will cause
you some problems) so the question is can we add some functionality to
cover this use case. Given the way we do the folder matching I think it
unlikely we would want to add globbing/wildcards to the start of a path
(*).

However, we might be able to support maildir++ as its own prefix. If you
could so searches of the form maildir++:.foo.bar.** which would match
the maildir .foo.bar and all submaildirs++ (eg .foo.bar.a.b etc) would
that be sufficient?

This wouldn't deal with 2013/.foo.bar but would allow .foo.bar.2013

Would this be enough help to make it worth considering?

Best wishes

Mark

(*) if we get a custom parser then it may all be manageable.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Jani Nikula j...@nikula.org writes:

 It's not going to help you, but I'll mention a few of the issues the old
 folder: search had, which we also had complaints about, and which would
 have been quite hard to fix while preserving the behaviour you want. In
 short, we considered the old folder: search broken.

 Given layout:

   Foo/{cur,new}
   foo/{cur,new}
   fooing/{cur,new}
   bar/foo/{cur,new}
   cur
   new

 It was impossible to refer to the top level folder.

 It was impossible to refer to foo without also referring to Foo, fooing,
 and bar/foo.

 In your layout, if you also had 2013/.bar.foo, folder:foo would match
 that as well. To not match that, you would have to include each
 folder:.foo.xxx in the search.

First, thanks for the response.  The responsiveness and friendliness of
the notmuch mailing list goes a long way towards compensating for any
missing features / customizability one might want.

I was already aware of the issues you raise, and had worked around them
by just renaming all my mail folders.  I agree that searching for a
particular folder is crucial functionality, and found it weird that I
had to abandon my main top-level mailbox (which I just renamed
.INBOX.Main).

However, currently it seems strange that there are *two* different
search terms (folder and path), and that neither one lets you search for
a portion of your folder name.  Admittedly the old folder code was one
of the parts of the notmuch source that didn't make sense to me (and now
I'm starting to understand why--e.g., the fact that it used stemming,
for instance, was just weird and maybe accidental).

I may be able to hack around this problem in the emacs lisp part or with
a wrapper script.  I'm already having to defadvice around
notmuch-call-notmuch-sexp to implement features I'm missing from
wanderlust and gnus (e.g., the inability to specify regexps matching all
my email addresses).

But to help me understand the current design, can you answer a couple of
questions?


First, are there people out there who do not use a collection of maildir
directories, with all mail in cur and new?  If not, why does notmuch try
to find mail in non-mail-directories, and why do you need search terms
differentiating new and cur?  Conversely, I find it particularly weird
that there's no convenient way to say stop trying to index stuff that
isn't in a maildir (cur or new).  You can do the inverse and blacklist
files, but then I end up with stuff like this in my .notmuch-config:


ignore=dovecot-keywords;dovecot-uidlist;dovecot-uidvalidity;dovecot.index;dovecot.index.cache;dovecot.index.log;dovecot.index.log.2;dovecot.index.search;dovecot.index.search.uids;maildirfolder;

(And still I think there are other junk files that get left around by my
imap server or other software, so notmuch new still repeatedly tries to
index junk whenever I run it.)


Second, does anyone out there have a collection with more than a few
thousand maildirs?  I understand some people index millions of messages,
but even with tiny maildirs of a few hundred messages each, that's a
mere 10,000 maildirs.  So for that, who needs any kind of Xapian
indexing fanciness or dedicated XFOLDER prefix?  Scanning the entire set
of XDIRECTORY terms to apply an arbitrary predicate should take
negligible time.  So all you really need is a boolean search term
containing the directory docid (or with the existing schema the ability
to do a prefix search on XFDIRENTRYn:*).

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[dm-list-email-notm...@scs.stanford.edu]

Jani Nikula  writes:

> On Fri, 02 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
>>
>> I'm using a pretty standard maildir++ layout.  For example, underneath
>> my database.path I have a bunch of mail in directories such as:
>>
>> .INBOX.Main/{new,cur}
>> .mail.class/{new,cur}
>> .mail.voicemail/{new,cur}
>> ...
> Here's additional commentary on the specific queries.
>
>> linux7$ ./notmuch count folder:mail
>> 0
>> linux8$ ./notmuch count folder:.mail
>> 0

Oh, man.  That's a serious bummer.

Is there any mechanism left that would let me hierarchically group
messages?  I've got a ton of mail.* folders, and create new ones
dynamically.  I really want a mechanism to group them hierarchically, so
I can have a search that matches all current and future mail
directories.  I organized my whole mail setup around folders because a)
tags do not provide this kind of hierarchical control, and b) there
doesn't seem to be a convenient way to apply tags 100% reliably on
message delivery, whereas I *can* control the folder 100% reliably.

Worse, because of my poor performance, I was hoping to segregate
messages by year.  So it would be:

  2013/.mail.class
  2013/.mail.voicemail
  2014/.mail.class
  2014/.mail.voicemail

All the way back.  Now you are saying there will be no convenient way to
match just the "mail.class" part without the year?  How very
distressing.  Ugh.

David

folder and path completely broken in HEAD?

[Mark Walters]

Hi

Before checking other things: have you run notmuch new? That's needed to
update the database. It is an irreversible update so notmuch-0.17 will
not work with the updated database.

The update should save a dump of the tags before updating but you might
want to do that manually too (and maybe even keep a complete copy of the
database). The update might take quite a long time (but it can be
interrupted and restarted)

Best wishes

Mark


On Fri, 02 May 2014, dm-list-email-notmuch at scs.stanford.edu wrote:
> Hey, I'm playing around with the head of the git repository
> (bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
> new folder: prefixes work, as folders are a critical part of how I
> organize my mail.  (Since tags are not hierarchical, folders are the
> best way for me to group mail to a bunch of related addresses, while
> retaining the ability to separate out any mailboxes that become high
> traffic.)
>
> I'm using a pretty standard maildir++ layout.  For example, underneath
> my database.path I have a bunch of mail in directories such as:
>
> .INBOX.Main/{new,cur}
> .mail.class/{new,cur}
> .mail.voicemail/{new,cur}
>
> It used to be the case that if I wanted to read all of my "mail" mail, I
> could search for folder:mail, while to look at just voicemail, I could
> say folder:mail.voicemail, etc.  Now, I can't get anything to match a
> folder predicate period.  For example, using notmuch as notmuch-0.17 and
> ./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:
>
> linux2$ notmuch count folder:mail
> 16257
> linux3$ notmuch count folder:mail.class
> 1896
> linux4$ notmuch count folder:mail.voicemail
> 34
> linux5$ notmuch count folder:mail.voicemail/cur
> 34
> linux6$ notmuch count folder:.mail.voicemail/cur
> 34
> linux7$ ./notmuch count folder:mail
> 0
> linux8$ ./notmuch count folder:.mail
> 0
> linux9$ ./notmuch count folder:.mail.voicemail
> 0
> linux10$ ./notmuch count folder:.mail.voicemail/cur
> 0
> linux 11$ ./notmuch count path:.mail.voicemail
> 0
> linux 12$ ./notmuch count path:.mail.voicemail/'**'
> 0
> linux 13$ ./notmuch count path:.mail.voicemail/cur 
> 0
> linux 14$ ./notmuch count folder:mail.voicemail
> 0
>
> What gives?  Are the path and folder predicates completely broken, or is
> something very important missing from the new notmuch-search-terms
> manual page?  How can I make this work?
>
> Thanks,
> David
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

folder and path completely broken in HEAD?

[David Mazieres expires 2014-07-31 PDT]

Mark Walters  writes:

> Hi
>
> Before checking other things: have you run notmuch new? That's needed to
> update the database. It is an irreversible update so notmuch-0.17 will
> not work with the updated database.

No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
The discussion of the NEWS was a bit confusing, so I wanted to check it
out.  I knew something had to be very wrong.

Weill the new primitives still allow me to group my mail hierarchically
in searches?  The new man page is not totally clear on what is being
matched.

Thanks,
David

folder and path completely broken in HEAD?

[dm-list-email-notm...@scs.stanford.edu]

Hey, I'm playing around with the head of the git repository
(bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
new folder: prefixes work, as folders are a critical part of how I
organize my mail.  (Since tags are not hierarchical, folders are the
best way for me to group mail to a bunch of related addresses, while
retaining the ability to separate out any mailboxes that become high
traffic.)

I'm using a pretty standard maildir++ layout.  For example, underneath
my database.path I have a bunch of mail in directories such as:

.INBOX.Main/{new,cur}
.mail.class/{new,cur}
.mail.voicemail/{new,cur}

It used to be the case that if I wanted to read all of my "mail" mail, I
could search for folder:mail, while to look at just voicemail, I could
say folder:mail.voicemail, etc.  Now, I can't get anything to match a
folder predicate period.  For example, using notmuch as notmuch-0.17 and
./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

linux2$ notmuch count folder:mail
16257
linux3$ notmuch count folder:mail.class
1896
linux4$ notmuch count folder:mail.voicemail
34
linux5$ notmuch count folder:mail.voicemail/cur
34
linux6$ notmuch count folder:.mail.voicemail/cur
34
linux7$ ./notmuch count folder:mail
0
linux8$ ./notmuch count folder:.mail
0
linux9$ ./notmuch count folder:.mail.voicemail
0
linux10$ ./notmuch count folder:.mail.voicemail/cur
0
linux 11$ ./notmuch count path:.mail.voicemail
0
linux 12$ ./notmuch count path:.mail.voicemail/'**'
0
linux 13$ ./notmuch count path:.mail.voicemail/cur 
0
linux 14$ ./notmuch count folder:mail.voicemail
0

What gives?  Are the path and folder predicates completely broken, or is
something very important missing from the new notmuch-search-terms
manual page?  How can I make this work?

Thanks,
David

folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Hey, I'm playing around with the head of the git repository
(bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
new folder: prefixes work, as folders are a critical part of how I
organize my mail.  (Since tags are not hierarchical, folders are the
best way for me to group mail to a bunch of related addresses, while
retaining the ability to separate out any mailboxes that become high
traffic.)

I'm using a pretty standard maildir++ layout.  For example, underneath
my database.path I have a bunch of mail in directories such as:

.INBOX.Main/{new,cur}
.mail.class/{new,cur}
.mail.voicemail/{new,cur}

It used to be the case that if I wanted to read all of my mail mail, I
could search for folder:mail, while to look at just voicemail, I could
say folder:mail.voicemail, etc.  Now, I can't get anything to match a
folder predicate period.  For example, using notmuch as notmuch-0.17 and
./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

linux2$ notmuch count folder:mail
16257
linux3$ notmuch count folder:mail.class
1896
linux4$ notmuch count folder:mail.voicemail
34
linux5$ notmuch count folder:mail.voicemail/cur
34
linux6$ notmuch count folder:.mail.voicemail/cur
34
linux7$ ./notmuch count folder:mail
0
linux8$ ./notmuch count folder:.mail
0
linux9$ ./notmuch count folder:.mail.voicemail
0
linux10$ ./notmuch count folder:.mail.voicemail/cur
0
linux 11$ ./notmuch count path:.mail.voicemail
0
linux 12$ ./notmuch count path:.mail.voicemail/'**'
0
linux 13$ ./notmuch count path:.mail.voicemail/cur 
0
linux 14$ ./notmuch count folder:mail.voicemail
0

What gives?  Are the path and folder predicates completely broken, or is
something very important missing from the new notmuch-search-terms
manual page?  How can I make this work?

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Jani Nikula]

On Fri, 02 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:
 Hey, I'm playing around with the head of the git repository
 (bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
 new folder: prefixes work, as folders are a critical part of how I
 organize my mail.  (Since tags are not hierarchical, folders are the
 best way for me to group mail to a bunch of related addresses, while
 retaining the ability to separate out any mailboxes that become high
 traffic.)

 I'm using a pretty standard maildir++ layout.  For example, underneath
 my database.path I have a bunch of mail in directories such as:

 .INBOX.Main/{new,cur}
 .mail.class/{new,cur}
 .mail.voicemail/{new,cur}

 It used to be the case that if I wanted to read all of my mail mail, I
 could search for folder:mail, while to look at just voicemail, I could
 say folder:mail.voicemail, etc.  Now, I can't get anything to match a
 folder predicate period.  For example, using notmuch as notmuch-0.17 and
 ./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

First, read Mark's reply about database upgrade, and heed the notice
about upgrade being automatic but downgrade requiring a reindex of all
your mails.

Here's additional commentary on the specific queries.

 linux7$ ./notmuch count folder:mail
 0
 linux8$ ./notmuch count folder:.mail
 0

These two will not match anything.

 linux9$ ./notmuch count folder:.mail.voicemail
 0

After the upgrade, this should work for .mail.voicemail/{new,cur}.

 linux10$ ./notmuch count folder:.mail.voicemail/cur
 0

This won't match anything. To specifically search for messages in cur,
you can use the new path prefix, path:.mail.voicemail/cur

 linux 11$ ./notmuch count path:.mail.voicemail
 0

This will match messages in .mail.voicemail only, and nothing in the cur
or new subdirectories.

 linux 12$ ./notmuch count path:.mail.voicemail/'**'
 0

I'm not sure your quoting will work, but if you don't have any other
subfolders than cur and new, this should be equal to
folder:.mail.voicemail

 linux 13$ ./notmuch count path:.mail.voicemail/cur 
 0

This should match messages in the cur subdirectory.

 linux 14$ ./notmuch count folder:mail.voicemail
 0

This won't match anything.


BR,
Jani.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[David Mazieres expires 2014-07-31 PDT]

Mark Walters markwalters1...@gmail.com writes:

 Hi

 Before checking other things: have you run notmuch new? That's needed to
 update the database. It is an irreversible update so notmuch-0.17 will
 not work with the updated database.

No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
The discussion of the NEWS was a bit confusing, so I wanted to check it
out.  I knew something had to be very wrong.

Weill the new primitives still allow me to group my mail hierarchically
in searches?  The new man page is not totally clear on what is being
matched.

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Jani Nikula j...@nikula.org writes:

 On Fri, 02 May 2014, dm-list-email-notm...@scs.stanford.edu wrote:

 I'm using a pretty standard maildir++ layout.  For example, underneath
 my database.path I have a bunch of mail in directories such as:

 .INBOX.Main/{new,cur}
 .mail.class/{new,cur}
 .mail.voicemail/{new,cur}
 ...
 Here's additional commentary on the specific queries.

 linux7$ ./notmuch count folder:mail
 0
 linux8$ ./notmuch count folder:.mail
 0

Oh, man.  That's a serious bummer.

Is there any mechanism left that would let me hierarchically group
messages?  I've got a ton of mail.* folders, and create new ones
dynamically.  I really want a mechanism to group them hierarchically, so
I can have a search that matches all current and future mail
directories.  I organized my whole mail setup around folders because a)
tags do not provide this kind of hierarchical control, and b) there
doesn't seem to be a convenient way to apply tags 100% reliably on
message delivery, whereas I *can* control the folder 100% reliably.

Worse, because of my poor performance, I was hoping to segregate
messages by year.  So it would be:

  2013/.mail.class
  2013/.mail.voicemail
  2014/.mail.class
  2014/.mail.voicemail

All the way back.  Now you are saying there will be no convenient way to
match just the mail.class part without the year?  How very
distressing.  Ugh.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Suvayu Ali]

Hello David,

On Fri, May 02, 2014 at 02:16:07PM -0700, David Mazieres expires 2014-07-31 PDT 
wrote:
 Mark Walters markwalters1...@gmail.com writes:
 
  Before checking other things: have you run notmuch new? That's needed to
  update the database. It is an irreversible update so notmuch-0.17 will
  not work with the updated database.
 
 No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
 The discussion of the NEWS was a bit confusing, so I wanted to check it
 out.  I knew something had to be very wrong.
 
 Weill the new primitives still allow me to group my mail hierarchically
 in searches?  The new man page is not totally clear on what is being
 matched.

Here are some examples:

  $ notmuch config get database.path
  /home/user/Mail
  $ notmuch count -- folder:Gmail/Tux
  5113
  $ notmuch count -- 'path:Gmail/**'
  87807

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Suvayu Ali]

On Fri, May 02, 2014 at 09:57:49PM -0700, 
dm-list-email-notm...@scs.stanford.edu wrote:
 
 Worse, because of my poor performance, I was hoping to segregate
 messages by year.  So it would be:
 
   2013/.mail.class
   2013/.mail.voicemail
   2014/.mail.class
   2014/.mail.voicemail
 
 All the way back.  Now you are saying there will be no convenient way to
 match just the mail.class part without the year?  How very
 distressing.  Ugh.

If you are using shell commands, you can always use:

  $ notmuch count -- folder:201{3,4}/.mail.class

Not sure how to achieve the same from the Emacs UI (or other UIs).

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch