Re: muchsync files renames

[dm-list-email-notmuch]

Amadeusz Żołnowski  writes:

>> So... based on all the evidence so fare the culprit seems to be that
>> something is moving mail files into your Spam folder on the client.
>> If that rings any bells and solves the problem, great.  If not, here
>> is what we need to do to track it down further.
>
> I have followed you hints to track down the issue.  All of these
> messages are spam. What I suspect follows.
>
> All of these files have been placed to new/ subdir by maildrop and
> during posthook (afew) have been stripped of any tags besides 'spam'
> tag, in particular 'unread' tag has been removed, but files still remain
> in new/ subdir.  So... what had to happen is that during muchsync these
> messages have been discovered as already read, so they don't belong to
> new/ but must be moved to cur/.  And this is what happened on client
> side.  During next muchsync these changes had to be pushed to server,
> i.e. move from new/ to cur/.

Right.  Muchsync checks to see if maildir.synchronize_flags is true on
the client.  If it is, then muchsync calls
notmuch_message_tags_to_maildir_flags after setting the flags (which is
the same as what would happen if you set the tags manually with the
"notmuch tag" command).

A maildir file in the new/ directory can't have any tags (except the
implicit unread flag, which is indicated by the absence of "S" in the
end of the filename).  So the notmuch_message_tags_to_maildir_flags()
function is renaming the file to the cur subdirectory, and then
propagating this rename back to the server.

The one thing I'm still unclear on is whether afew is running on the
client of the server.  If you are running it on the client, then this
makes sense.  If you are running it on the server, then somehow afew
must not be respecting the maildir.synchronize_flags setting.
Otherwise, the file should already be moved to the cur directory after
having the unread tag stripped off on the server.  I guess the other
option is that your maildir.synchronize_flags false on the server and
true on the client.

> So if my assumptions are correct, actually there is no issue!  I would
> just have to adjust afew filtering to prevent this behaviour.

Right.  You could have afew preserve the unread flag on spam.
Alternatively, you could just disable maildir.synchronize_flags on both
the client and server.  Finally, you could just accept the performance
penalty, as one would hope that this is a one-time thing and that
usually you don't have 5000 new spam messages every time you synchronize
your mail.

David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/2] emacs: add function to resend message to new recipients

[Tomi Ollila]

On Mon, Aug 31 2015, David Bremner  wrote:

> Tomi Ollila  writes:
>
>>
>>   emacs -q -L $PWD/emacs -l emacs/notmuch.el -f notmuch --eval '(progn (setq 
>> notmuch-address-command "nottoomuch-addresses.sh") 
>> (notmuch-address-message-insinuate))'
>>

> Ah, I missed notmuch-address-message-insinuate; it does work if I run
> that. I wonder if this can be simplified at all now that
> notmuch-message-mode exists. In general I'm leery of running code when we
> load notmuch*.el, as this has a history of causing problems for people
> using notmuch e.g. as a search tool for gnus.

Currently as both setting notmuch-address-command and
notmuch-address-message-insinunate us required I would not attempt to do any
"magic" there. Whenever we get built-in address completion (i.e built-in
is used when notmuch-address-command is kept nil we could consider to add
some defcustom which can be used to activate address completion -- but 
as there are these *-insinunate -commands in many elisp packages it is
probably best to investigate whether this really can be "automated".
Perhaps some lazy initialization could be in place, then.

> The completion interface takes a little getting used to, but it's
> definitely usable. And of course much better than what we have now ;).
>
> One thing I noticed which I _think_ is a bug, is that after calling
> notmuch-show-resend-message, notmuch-view-raw-message doesn't work
> anymore, complaining about a read-only buffer.

That is interesting in a sense that I could not reproduce. I think I know
exactly why this happens: notmuch-view-raw-message creates read-only buffer
which is not removed after message is resent: re-running
notmuch-view-raw-message on same message will `get-buffer-create' with
same name and pick the old read-only buffer; inserting data to that buffer
fails -- just that in my tests (V C-x b RET V) I could not get the same
outcome.

The simplest change is to change (bury-buffer) with (kill-buffer). The raw
message buffer that was used to send should not be interesting; more
interesting is the current (same) message in notmuch-show buffer -- and 'V'
can be used to re-view the raw message. 
What could be added is printing the message id of the message resent to
*Messages* buffer. This way re-sender can verify what was resent if being
unsure. Just that in the case I've used this I have not looked back...

So, probably I make applicable RFC patch of this feature...


>
> d

Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[Amadeusz Żołnowski]

Hi David,

dm-list-email-notm...@scs.stanford.edu writes:
> The one thing I'm still unclear on is whether afew is running on the
> client of the server.

It is run as a post-hook, i.e. after "notmuch new", so it's on the
server.

> I guess the other option is that your maildir.synchronize_flags false
> on the server and true on the client.

Both have this set to true.

> If you are running it on the server, then somehow afew must not be
> respecting the maildir.synchronize_flags setting.  Otherwise, the file
> should already be moved to the cur directory after having the unread
> tag stripped off on the server.

Not necessarily. The recommended setup of notmuch for afew is that
"notmuch new" tags messages with "new" tag only. Then afew processes all
messages with "new" tag. So if it is a spam, then it gets "new" removed
and "spam" added. A spam message at any time doesn't have "unread" tag
assigned which should explain this behaviour.  So the problem is to be
fixed on the afew side.

Once again thank you for helping with tracking down the issue.

-- 
Amadeusz Żołnowski


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/2] emacs: add function to resend message to new recipients

[David Bremner]

Tomi Ollila  writes:

> That is interesting in a sense that I could not reproduce. I think I know
> exactly why this happens: notmuch-view-raw-message creates read-only buffer
> which is not removed after message is resent: re-running
> notmuch-view-raw-message on same message will `get-buffer-create' with
> same name and pick the old read-only buffer; inserting data to that buffer
> fails -- just that in my tests (V C-x b RET V) I could not get the same
> outcome.

This test actually reproduces the problem for me. Maybe it's emacs
version specific? I mainly 24.5.1, but I can't duplicate the problem in
emacs23.

d
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski  writes:

> Not necessarily. The recommended setup of notmuch for afew is that
> "notmuch new" tags messages with "new" tag only. Then afew processes all
> messages with "new" tag. So if it is a spam, then it gets "new" removed
> and "spam" added. A spam message at any time doesn't have "unread" tag
> assigned which should explain this behaviour.  So the problem is to be
> fixed on the afew side.

Let's just make sure I understand:  Your mail starts out like this:

Path:  spam/new/nnn.MnnnPnnnQnRn.machine
Tags:  new

Then you run afew, and afew runs

notmuch tag -new +spam 

You are saying that that even though maildir.synchronize_tags is true,
you end up with:

Path:  spam/new/nnn.MnnnPnnnQnRn.machine
Tags:  spam

That's a little surprising, because the next time you run "notmuch new,"
I would have expected it to add the unread flag based on the pathname.
But, I suppose it might make sense for notmuch to special-case that
flag.  In other words, if notmuch new finds a file called:

spam/new/nnn.MnnnPnnnQnRn.machine:2,

Then it will add the unread tag to the Xapian database.  But maybe if it
finds a file in the new folder it doesn't add the unread flag.

But why does notmuch_message_tags_to_maildir_flag() then feel the need
to rename the file when muchsync calls it.  Muchsync should ideally
behave exactly the same as the notmuch tag command.  Specifically, when
muchsync receives a new file from the server, it does the following:

 1. create file in same directory as the server (presumably spam/new)

 2. Call the following functions on this file:
  notmuch_database_add_message()
  notmuch_message_freeze()
  notmuch_message_remove_all_tags()
  notmuch_message_add_tag() for each tag in new.tags
  if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
  notmuch_message_thaw()

 3. get the current tags of the message from the server (presumably just
spam)

 4. Call the following functions on the Message-ID:
  notmuch_message_freeze()
  notmuch_message_remove_all_tags()
  notmuch_message_add_tag() for each tag sent *by the server*
  if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
  notmuch_message_thaw()

So what I'm wondering is how this is any different from what is already
happening on the server.  "notmuch new" should be doing what muchsync
does in step 2, and afew (via "notmuch tag") should be doing what
muchsync does in step 4.

Yet somehow you are saying that on the server the file stays in
spam/new/, while on the client muchsync's actions cause the file to move
to spam/cur/?  So that means there's still something I don't really
understand--possibly the series of notmuch library calls happening
server side (which I should then maybe emulate client side).

None of this is super serious, beyond a one-time extra cost, but I like
to understand things thoroughly, particularly when writing software that
manipulates critical data like mail...

It there any possibility that new.tags has a different setting on your
client and server machines?

Thanks,
David
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [RFC PATCH 5/5] cli: add support for deduplicating based on case insensitive address

[Tomi Ollila]

On Sun, Aug 30 2015, David Bremner  wrote:

> Jani Nikula  writes:
>
>> Consider all variants of an email address as one, and print the most
>> common variant.
>
> Other than the quibbles already mentioned, the series looks ok to
> me. For production it should have one or two tests I guess. Oh, and man
> page updates. But you knew that I guess.
>
> I can live with the current argument syntax, but since a certain a mount
> of bikeshedding is expected, here is an alternative suggestion
>
> --deduplication={none|mailbox|address}

(is s/deduplicate/deduplication/ intended or accidental change?)

Is this complete alternative to --deduplicate={no|yes|address}, respectively?

If it is it looks somewhat better to me (provided that it is accurate)
anyway, the user command line interface looks good to me.



Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [RFC PATCH 5/5] cli: add support for deduplicating based on case insensitive address

[David Bremner]

Tomi Ollila  writes:

>> I can live with the current argument syntax, but since a certain a mount
>> of bikeshedding is expected, here is an alternative suggestion
>>
>> --deduplication={none|mailbox|address}
>
> (is s/deduplicate/deduplication/ intended or accidental change?)

accidental. hmm. well,

--deduplicate={no|mailbox|address}

is a few characters shorter.

> Is this complete alternative to --deduplicate={no|yes|address},
> respectively?

Yes

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

ANNOUNCE: muchsync 2 released on Gentoo

[Amadeusz Żołnowski]

Just a short note, I have added net-mail/muchsync-2 to Gentoo tree.

Enjoy,

-- 
Amadeusz Żołnowski


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: muchsync files renames

[Amadeusz Żołnowski]

Hi David,

First of all thank you a lot for support.  I am Cc'ing ml because the
last paragraph may be useful hint for other users.


David Mazieres writes:
> So to be clear, you are getting tons of lines that start "[SERVER]
> [notmuch]" and contain the string "Ignoring non-mail file"?  Is the
> "##...##" literal, or is that an ellipsis?

I have just cut off few directories on the path. :-) All of these files
are invalid spam mail, indeed.  I have removed them.  One problem less.


> Also, those file names were not generated were not generated by
> muchsync.  Any mail file created by muchsync will have a file name of
> the form:
>
> nnn.MnnnPnnnQnRn.machine
> nnn.MnnnPnnnQnRn.machine:2,

Just to makes things clear (once again? :-)), these file names are
generated only on client side.  Muchsync is not gonna ever to sync file
names to server, is it?


> When you run "notmuch new" on the server, without muchsync, does it
> take forever and print all these message while scanning non mail
> files?

No. Notmuch doesn't print these messages when I just run "notmuch new"
myself.  Anyway there was only around 100 of invalid mail files.


> Okay, this is the interesting part.  It appears that 5775 out of your
> 115877 messages have been moved to a different directory on the
> client.  I notice that the one message you include above has been
> moved to the Spam maildir.

> Is it possible that A) you have some spam filtering on the client that
> is moving things to the Spam folder,

I have a mailfilter rule which moves mails with "X-Spam-Status: Yes" to
Spam directory, but this happens on delivery before notmuch indexing.


> or B) that one of your two machines is using a case-independent file
> system that is causing confusion between "Spam" and "spam"?

I am testing it on single GNU/Linux host between different users.  It is
ext4 fs.


> So... based on all the evidence so fare the culprit seems to be that
> something is moving mail files into your Spam folder on the client.
> If that rings any bells and solves the problem, great.  If not, here
> is what we need to do to track it down further.

I have followed you hints to track down the issue.  All of these
messages are spam. What I suspect follows.

All of these files have been placed to new/ subdir by maildrop and
during posthook (afew) have been stripped of any tags besides 'spam'
tag, in particular 'unread' tag has been removed, but files still remain
in new/ subdir.  So... what had to happen is that during muchsync these
messages have been discovered as already read, so they don't belong to
new/ but must be moved to cur/.  And this is what happened on client
side.  During next muchsync these changes had to be pushed to server,
i.e. move from new/ to cur/.

So if my assumptions are correct, actually there is no issue!  I would
just have to adjust afew filtering to prevent this behaviour.


Thank you,

-- 
Amadeusz Żołnowski


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch