[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Thu, 26 Nov 2009 13:14:22 -0800, Carl Worth  wrote:

> Then I'm still being really dense here. The non-propagation of mtime is
> the with actual directories. And the code is trying to do the right
> thing for that.

The stat() is correct, it's the check for the d_type field coming out of 
scandir().

20:08 [   jedbrown  ] cworth: I think there are two problems with symlinks.
20:08 [   jedbrown  ] cworth: First, notmuch-new.c:152 checks path_mtime which 
is mtime for the current directory.
20:09 [   jedbrown  ] cworth: Additionally, won't entry->d_type == DT_LNK for a 
symlink?
20:14 [ aneesh  ] jedbrown,  i already have a patch posted for that
20:15 [ aneesh  ] 1259125104-18785-1-git-send-email-aneesh.kumar at 
linux.vnet.ibm.com


Jed

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Thu, 26 Nov 2009 10:34:12 -0800, Carl Worth  wrote:
> I'm a little confused here. Notmuch only uses stat, so it should be
> looking at the target's mtime already. It actually takes special effort
> (via lstat) to get at the mtime of the link itself.
> 
> So why aren't things just working?

Because mtime doesn't change on the directory *above* the symlinks.

I think.

Jed

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Carl Worth]

On Thu, 26 Nov 2009 19:39:01 +0100, Jed Brown  wrote:
> On Thu, 26 Nov 2009 10:34:12 -0800, Carl Worth  wrote:
> > I'm a little confused here. Notmuch only uses stat, so it should be
> > looking at the target's mtime already. It actually takes special effort
> > (via lstat) to get at the mtime of the link itself.
> > 
> > So why aren't things just working?
> 
> Because mtime doesn't change on the directory *above* the symlinks.
> 
> I think.

Then I'm still being really dense here. The non-propagation of mtime is
the with actual directories. And the code is trying to do the right
thing for that.

So I still don't see what's being different with the symlinks. I should
probably stop exposing ignorance here on the list, and just sit down
with symlinks in a mail store and see what's happening.

-Carl

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Carl Worth]

On Mon, 23 Nov 2009 16:03:00 +0100, Jed Brown  wrote:
> > But still no luck.  Finally, I deleted the symlinks and created them
> > anew, and then it indexed the 12 new mails that arrived in the
> > meantime.

OK. So that's definitely a bug we need to fix.

> The real solution is for notmuch to check mtime of whatever the
> symlink's target.

I'm a little confused here. Notmuch only uses stat, so it should be
looking at the target's mtime already. It actually takes special effort
(via lstat) to get at the mtime of the link itself.

So why aren't things just working?

-Carl

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

Twas brillig at 16:03:00 23.11.2009 UTC+01 when jed at 59A2.org did gyre and 
gimble:

 JB> The real solution is for notmuch to check mtime of whatever the
 JB> symlink's target.

It does exactly this AFACT, stat() everywhere, not lstat().

-- 
  http://fossarchy.blogspot.com/
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: 

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

Twas brillig at 16:01:41 23.11.2009 UTC+01 when tassilo at member.fsf.org did 
gyre and gimble:

 >>> Probably mail does not get indexed due to mtime checks. Please try
 >>> whether touch'ing directory with mailboxes makes it work.
 >> No, it seems that doesn't help either.
 TH> Ah, I'm stupid!  I don't have to touch the symlinks or the
 TH> directories inside the locations the symlinks point to, but instead
 TH> I have to touch the top-level directory where the symlinks are
 TH> contained in.  Then it works as expected, AFAICT.

Really odd.

-- 
  http://fossarchy.blogspot.com/
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: 

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

Twas brillig at 14:55:40 23.11.2009 UTC+01 when tassilo at member.fsf.org did 
gyre and gimble:

 TH> Whenever I delete those symlinks and created them anew, the new
 TH> mails get indexed with the next "notmuch new".  Of course, I could
 TH> create a script that does exactly that, but there should be a
 TH> better way, right?

Probably mail does not get indexed due to mtime checks. Please try
whether touch'ing directory with mailboxes makes it work.

-- 
  http://fossarchy.blogspot.com/
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: 

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Mon, 23 Nov 2009 16:01:41 +0100, Tassilo Horn  
wrote:
> Tassilo Horn  writes:

> I don't have to touch the symlinks or the directories inside the
> locations the symlinks point to, but instead I have to touch the
> top-level directory where the symlinks are contained in.

Ah, it's slightly more subtle.  Notmuch correctly uses stat() instead of
lstat() to check whether the link target changed.  However, mtime for
your mail root directory (containing the symlinks) does not get updated
when the target of the symlinks is updated.  I think the only way to fix
this is to replace the current search (which skips a directory as soon
as it's mtime is older than the database) with one that enters all
directories so that symlinks are actually followed.

Jed

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Mon, 23 Nov 2009 15:43:41 +0100, Tassilo Horn  
wrote:
> First, I only touched the two symlinks.

Unfortunately, this actually touched the file pointed to by the symlink,
if you stat the symlink you will see that mtime did not change.

> This didn't help.  Then I used
> "find . -type d | xargs touch" to touch all directories inside the
> directories the symlinks point to.

Actually, this would not have followed the symlinks so it does the same
thing as before.  I think it is actually hard (or not possible) to
change mtime on symlinks under Linux.

> But still no luck.  Finally, I deleted the symlinks and created them
> anew, and then it indexed the 12 new mails that arrived in the
> meantime.

If /var is on the same filesystem, you could use hard links instead of
symlinks.  Otherwise I would just add the appropriate ln -sf in the hook
before notmuch new.

The real solution is for notmuch to check mtime of whatever the
symlink's target.

Jed

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Tassilo Horn]

Tassilo Horn  writes:

Hi Mikhail,

>>  TH> Whenever I delete those symlinks and created them anew, the new
>>  TH> mails get indexed with the next "notmuch new".  Of course, I could
>>  TH> create a script that does exactly that, but there should be a
>>  TH> better way, right?
>>
>> Probably mail does not get indexed due to mtime checks. Please try
>> whether touch'ing directory with mailboxes makes it work.
>
> No, it seems that doesn't help either.

Ah, I'm stupid!  I don't have to touch the symlinks or the directories
inside the locations the symlinks point to, but instead I have to touch
the top-level directory where the symlinks are contained in.  Then it
works as expected, AFAICT.

Thanks a lot,
Tassilo

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks

[Tassilo Horn]

Mikhail Gusarov  writes:

Hi Mikhail,

>  TH> Whenever I delete those symlinks and created them anew, the new
>  TH> mails get indexed with the next "notmuch new".  Of course, I could
>  TH> create a script that does exactly that, but there should be a
>  TH> better way, right?
>
> Probably mail does not get indexed due to mtime checks. Please try
> whether touch'ing directory with mailboxes makes it work.

No, it seems that doesn't help either.

First, I only touched the two symlinks.  This didn't help.  Then I used
"find . -type d | xargs touch" to touch all directories inside the
directories the symlinks point to.  But still no luck.  Finally, I
deleted the symlinks and created them anew, and then it indexed the 12
new mails that arrived in the meantime.

Bye,
Tassilo

[notmuch] Notmuch doesn't index new mails when mail location contains symlinks (was: Notmuch doesn't index new mails)

[Tassilo Horn]

Tassilo Horn  writes:

Hi all,

I've investigated a bit further.

> [notmuch doesn't index new mails although all directories and files
> are readable and writable.]

In my config, I have:

--8<---cut here---start->8---
[database]
path=/home/horn/Mail/Dovecot
--8<---cut here---end--->8---

In that directory, there are two symlinks pointing to the real mail
location in /var/spool/mail:

--8<---cut here---start->8---
[horn at localhost][~/Mail/Dovecot][0][5213]
[:)] % ll
total 0
lrwxrwxrwx 1 horn horn 34 Nov 23 14:49 fastmail -> 
/var/spool/mail/fastmail/mailboxes
lrwxrwxrwx 1 horn horn 29 Nov 23 14:49 uni -> /var/spool/mail/uni/mailboxes
--8<---cut here---end--->8---

Whenever I delete those symlinks and created them anew, the new mails
get indexed with the next "notmuch new".  Of course, I could create a
script that does exactly that, but there should be a better way, right?

Bye,
Tassilo