Re: Solr, Dovecot & macOS / iOS

[@lbutlr via dovecot]

On 13 Aug 19, at 05:58 , James Brown  wrote:
> 
> b) does Mail.app and other mail clients on Macs or iOS devices perform 
> searches on their local copy of mail or does it just send a search request to 
> the server?

Mail.app uses spotlight on the local data, so if your users are all Mac then 
solar is pointless.

Re: Dovecot Sieve doesn't move spam to Junk folder in maildir

[Nagy Tibor via dovecot]

You're right, dovecot was not configured as LDA, I did it, and really I see the 
following line in log/dovecot.info:


2019-08-13 15:49:39 lda(tibor.n...@softxs.ch): Info: sieve: 
msgid=<5b78f784-9bc4-aa94-219d-25ccb0d11...@hu.inter.net>: stored mail into 
mailbox 'INBOX'



I've tried also with a test spam mail and I see the following line in 
log/dovecot.info:


2019-08-13 15:52:15 lda(tibor.n...@softxs.ch): Info: sieve: 
msgid=<904f3c82-6db1-8b5c-d465-ef140c2f5...@hu.inter.net>: stored mail into 
mailbox 'Junk'



Thanks.


-- Only for information  if somebody is interested for 
the solution -


The appropriate configuration lines in postfix/main.cf:


### Configuration for virtual mailboxes
home_mailbox = Maildir/
virtual_mailbox_domains = /etc/postfix/vhosts
virtual_mailbox_base = /home/vmail
virtual_mailbox_maps = hash:/etc/postfix/vmaps
virtual_minimum_uid = 1000
virtual_uid_maps = static:1002
virtual_gid_maps = static:1002

### Configuration for dovecot as LDA (New 2019-08-13)
virtual_transport = dovecot
spamassassin_destination_recipient_limit = 1
dovecot_destination_recipient_limit = 1


The appropriate configuration lines in postfix/master.cf


smtp  inet  n   -   -   -   -   smtpd
  -o content_filter=spamassassin
[...]

### Configuration for dovecot as LDA (New 2019-08-13)
# spamassassin unix - n   n   -   -   pipe
#  user=spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} 
${original_recipient}

spamassassin unix - n   n   -   -   pipe
  flags=DROhu user=vmail:vmail argv=/usr/bin/spamc -f -e
  /usr/lib/dovecot/deliver -f ${sender} -d ${user}@${nexthop}


The appropriate configuration lines in dovecot/dovecot.conf:

plugin {
  sieve = /etc/dovecot/sieve/default.sieve
}
protocol lda {
  mail_plugins = $mail_plugins sieve
}


The appropriate mail.log lines:

Aug 13 15:49:37 zg-3 postfix/smtpd[16296]: connect from 
smarthost06.digicable.hu[94.21.128.16]
Aug 13 15:49:37 zg-3 postfix/smtpd[16296]: C1835A3C9F: 
client=smarthost06.digicable.hu[94.21.128.16]
Aug 13 15:49:37 zg-3 postfix/cleanup[16304]: C1835A3C9F: 
message-id=<5b78f784-9bc4-aa94-219d-25ccb0d11...@hu.inter.net>
Aug 13 15:49:37 zg-3 postfix/qmgr[16267]: C1835A3C9F: from=, 
size=1139, nrcpt=1 (queue active)
Aug 13 15:49:37 zg-3 postfix/smtpd[16296]: disconnect from 
smarthost06.digicable.hu[94.21.128.16] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 
quit=1 commands=7
Aug 13 15:49:39 zg-3 postfix/pipe[16305]: C1835A3C9F: to=, 
relay=spamassassin, delay=1.9, delays=0.1/0.01/0/1.8, dsn=2.0.0, status=sent 
(delivered via spamassassin service)

Aug 13 15:49:39 zg-3 postfix/qmgr[16267]: C1835A3C9F: removed
[...]
Aug 13 15:52:15 zg-3 postfix/smtpd[16296]: connect from 
smtp4.enternet.hu[62.112.192.37]
Aug 13 15:52:15 zg-3 postfix/smtpd[16296]: 6C04CA3C9F: 
client=smtp4.enternet.hu[62.112.192.37]
Aug 13 15:52:15 zg-3 postfix/cleanup[16304]: 6C04CA3C9F: 
message-id=<904f3c82-6db1-8b5c-d465-ef140c2f5...@hu.inter.net>
Aug 13 15:52:15 zg-3 postfix/qmgr[16267]: 6C04CA3C9F: from=, 
size=917, nrcpt=1 (queue active)
Aug 13 15:52:15 zg-3 postfix/smtpd[16296]: disconnect from 
smtp4.enternet.hu[62.112.192.37] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 
commands=7
Aug 13 15:52:15 zg-3 postfix/pipe[16305]: 6C04CA3C9F: to=, 
relay=spamassassin, delay=0.23, delays=0.04/0/0/0.19, dsn=2.0.0, status=sent 
(delivered via spamassassin service)

Aug 13 15:52:15 zg-3 postfix/qmgr[16267]: 6C04CA3C9F: removed


2019. 08. 06. 15:21 keltezéssel, Alexander Dalloz via dovecot írta:

Am 2019-08-06 15:04, schrieb Nagy Tibor via dovecot:

I have an *Ubuntu 16.04* mail server with
postfix/spamassassin/*dovecot-2.2.2**2* with virtual mailboxes in
maildir format. The whole process is working excellently since a year.
Spams are marked with "*SPAM*" in subject and with
"X-Spam-Flag: YES" in email header.

I tried now to add sieve to dovecot to move the SPAM messages into the
Junk folder without any success, I don't see any sign that sieve does
something at all. The output of "dovecot -n":


[ ... ]


Jul 30 20:05:13 zg-3 postfix/cleanup[4450]: 5F7FEA3CBC:
message-id=
Jul 30 20:05:13 zg-3 postfix/qmgr[1557]: 5F7FEA3CBC:
from=, size=3304, nrcpt=1 (queue active)
Jul 30 20:05:13 zg-3 postfix/virtual[4455]: 5F7FEA3CBC:
to=, relay=virtual, delay=0.02, delays=0.01/0.01/0/0,
dsn=2.0.0, status=sent (delivered to maildir)


Your Postfix does not deliver via dovecot's LDA, thus sieve never gets called.


Jul 30 20:05:13 zg-3 postfix/qmgr[1557]: 5F7FEA3CBC: removed

What can be wrong?



Alexander

Re: Solr, Dovecot & macOS / iOS

[Jean-Daniel via dovecot]

> Le 13 août 2019 à 14:53, Sami Ketola  a écrit :
> 
> 
> 
>> On 13 Aug 2019, at 15.37, Jean-Daniel via dovecot > > wrote:
>> 
>> 
>> 
>>> Le 13 août 2019 à 14:16, Sami Ketola via dovecot >> > a écrit :
>>> 
>>> 
>>> 
 On 13 Aug 2019, at 14.58, James Brown via dovecot >>> > wrote:
 
 I’m thinking of getting Solr working with my Dovecot server. Server is new 
 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of 
 thousands of messages.
 
 But I’m not sure if:
 
 a) it will make enough of a difference and
>>> 
>>> Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 
>>> 100s of thousands of small files in same directory. If so, then use mdbox 
>>> with periodic purges.
>>> 
 
 b) does Mail.app and other mail clients on Macs or iOS devices perform 
 searches on their local copy of mail or does it just send a search request 
 to the server?
>>> 
>>> None of the apple devices use IMAP SEARCH. They ALL maintain and use their 
>>> own local search database on the device. Also they seem to refresh the 
>>> database every now and then redownloading all emails.
>> 
>> Do you have a source for that. My experience is that without server search 
>> support, iOS is very slow at returning result. Moreover, it keep only latest 
>> messages and never download message until you read them.
> 
> I'm a apple device user myself. I have couple of iPhones, couple of iPads, 
> couple if MacBooks and Mail.app on any of them is not using IMAP SEARCH.
> And I cannot find any configuration option to enable it. Only spotlight index 
> is used. On Mac OS Mail.App seems to store the indexed data to:
> 
> samik@samikworkmac:~>ls -1 Library/Mail/V6/MailData/Envelope\ Index*
> Library/Mail/V6/MailData/Envelope Index
> Library/Mail/V6/MailData/Envelope Index-shm
> Library/Mail/V6/MailData/Envelope Index-wal
> 
> if those files are removed or spotlight search for mails is disabled Mail.App 
> can't find anything anymore. It does not fall back to IMAP SEARCH.
> 

My question was more about iOS. I know that macOS Mail does not rely on any way 
on remote indexing and has it’s own local index, but as it also store all 
messages locally, it’s an easy requirement. For iOS that only download messages 
meta-data by default, I was not so sure. 

I’m accessing my mail server using Apple devices only, and see some imap SEARCH 
requests in dovecot stats, but can’t figure out where they came from though. So 
you may be right.

Re: Solr, Dovecot & macOS / iOS

[Sami Ketola via dovecot]

> On 13 Aug 2019, at 15.37, Jean-Daniel via dovecot  wrote:
> 
> 
> 
>> Le 13 août 2019 à 14:16, Sami Ketola via dovecot  a 
>> écrit :
>> 
>> 
>> 
>>> On 13 Aug 2019, at 14.58, James Brown via dovecot  
>>> wrote:
>>> 
>>> I’m thinking of getting Solr working with my Dovecot server. Server is new 
>>> 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of 
>>> thousands of messages.
>>> 
>>> But I’m not sure if:
>>> 
>>> a) it will make enough of a difference and
>> 
>> Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 
>> 100s of thousands of small files in same directory. If so, then use mdbox 
>> with periodic purges.
>> 
>>> 
>>> b) does Mail.app and other mail clients on Macs or iOS devices perform 
>>> searches on their local copy of mail or does it just send a search request 
>>> to the server?
>> 
>> None of the apple devices use IMAP SEARCH. They ALL maintain and use their 
>> own local search database on the device. Also they seem to refresh the 
>> database every now and then redownloading all emails.
> 
> Do you have a source for that. My experience is that without server search 
> support, iOS is very slow at returning result. Moreover, it keep only latest 
> messages and never download message until you read them.

I'm a apple device user myself. I have couple of iPhones, couple of iPads, 
couple if MacBooks and Mail.app on any of them is not using IMAP SEARCH.
And I cannot find any configuration option to enable it. Only spotlight index 
is used. On Mac OS Mail.App seems to store the indexed data to:

samik@samikworkmac:~>ls -1 Library/Mail/V6/MailData/Envelope\ Index*
Library/Mail/V6/MailData/Envelope Index
Library/Mail/V6/MailData/Envelope Index-shm
Library/Mail/V6/MailData/Envelope Index-wal

if those files are removed or spotlight search for mails is disabled Mail.App 
can't find anything anymore. It does not fall back to IMAP SEARCH.

Sami

Re: Solr, Dovecot & macOS / iOS

[Jean-Daniel via dovecot]

> Le 13 août 2019 à 14:16, Sami Ketola via dovecot  a 
> écrit :
> 
> 
> 
>> On 13 Aug 2019, at 14.58, James Brown via dovecot  
>> wrote:
>> 
>> I’m thinking of getting Solr working with my Dovecot server. Server is new 
>> 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands 
>> of messages.
>> 
>> But I’m not sure if:
>> 
>> a) it will make enough of a difference and
> 
> Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 
> 100s of thousands of small files in same directory. If so, then use mdbox 
> with periodic purges.
> 
>> 
>> b) does Mail.app and other mail clients on Macs or iOS devices perform 
>> searches on their local copy of mail or does it just send a search request 
>> to the server?
> 
> None of the apple devices use IMAP SEARCH. They ALL maintain and use their 
> own local search database on the device. Also they seem to refresh the 
> database every now and then redownloading all emails.

Do you have a source for that. My experience is that without server search 
support, iOS is very slow at returning result. Moreover, it keep only latest 
messages and never download message until you read them.

Re: Solr, Dovecot & macOS / iOS

[Sami Ketola via dovecot]

> On 13 Aug 2019, at 14.58, James Brown via dovecot  wrote:
> 
> I’m thinking of getting Solr working with my Dovecot server. Server is new 
> 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands 
> of messages.
> 
> But I’m not sure if:
> 
> a) it will make enough of a difference and

Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 
100s of thousands of small files in same directory. If so, then use mdbox with 
periodic purges.

> 
> b) does Mail.app and other mail clients on Macs or iOS devices perform 
> searches on their local copy of mail or does it just send a search request to 
> the server?

None of the apple devices use IMAP SEARCH. They ALL maintain and use their own 
local search database on the device. Also they seem to refresh the database 
every now and then redownloading all emails.

> 
> I’m guessing the searches are done locally so no point in Solr?

No point if only apple client devices.

Sami

Solr, Dovecot & macOS / iOS

[James Brown via dovecot]

I’m thinking of getting Solr working with my Dovecot server. Server is new 
6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of 
messages.

But I’m not sure if:

a) it will make enough of a difference and

b) does Mail.app and other mail clients on Macs or iOS devices perform searches 
on their local copy of mail or does it just send a search request to the server?

I’m guessing the searches are done locally so no point in Solr?

Does anyone know? 

Thanks, James.


smime.p7s
Description: S/MIME cryptographic signature