Re: Dict quota calculation errors "remote disconnected"/"broken pipe" on 2.22.

2017-02-12 Thread ygrishin-lists 

On 2017-02-08 00:10, Steffen Kaiser wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sun, 5 Feb 2017, ygrishin-li...@mail2.ca wrote:


service dict {
 unix_listener dict {
 mode = 0660
 user = Debian-exim
 group = Debian-exim
 }
}

dovecot-lda-erros.log:
**
Feb 04 14:23:33 lda(testuser@XXX): Error: read(/var/run/dovecot/dict) 
failed: Remote disconnected


dovecot.log:

Feb 04 13:57:06 imap(YYY@XXX): Error: write(/var/run/dovecot/dict) 
failed: Broken pipe

...

dovecot-debug.log:
**
Feb 04 13:18:12 lda(YYY@XXX): Error: read(/var/run/dovecot/dict) 
failed: Remote disconnected
Feb 04 13:18:12 lda(YYY@XXX): Error: dict quota: Quota update failed, 
it's now desynced
Feb 04 13:57:07 lda(testuser@XXX): Error: write(/var/run/dovecot/dict) 
failed: Broken pipe


Does a process listens on /var/run/dovecot/dict ?


It certainly does:

#lsof /var/run/dovecot/dict
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
dovecot 1140 root   42u  unix 0xc6fe2300  0t0 15861 
/var/run/dovecot/dict type=STREAM


$ ls -l /var/run/dovecot/dict
srw-rw 1 Debian-exim Debian-exim 0 Feb 12 03:53 
/var/run/dovecot/dict



The socket is accessable by Debian-exim:Debian-exim only (0660). As
what user and group does the LDA and imap service run as?


LDA works as dovecot:Debian-exim:
*
lda:
driver = pipe
...
group = Debian-exim
...

(without specifying the user explicitly).

Yuriy

Re: dovecot config for 1500 simultaneous connection

2017-02-12 Thread Christian Balzer 

Hello,

On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote:

> Thanks for the info. I do have one further question for you. On your servers 
> that are currently handling 50k IMAP sessions, how many users does that 
> correspond to? Since many users will have multiple IMAP sessions on multiple 
> devices, I’d like to hear about some real-world numbers that could be used 
> for budgeting a new project like mine.
>

Those servers would actually be the wrong ones to look at, as they are
primarily accessed by the aforementioned broken client, so the numbers are
somewhat skewed.
However looking at other servers with a more "normal" user spread things
aren't actually too different.

The average number of sessions per user tends to be 2.
The clear majority (over 50%) only has one session open (people with well
behaved and configured clients watching the INBOX mostly).
Another 30% has 2 sessions open, again the typical state of this would be
clients watching another mailbox besides INBOX, typically SENT.
The rest has 3 and more sessions open.

The number of sessions could of course be drastically reduced if any
client would support IMAP NOTIFY, alas none that I'm aware of do.

Lastly no more than 60% of the session seem to be in IDLE at any given
time, so my comments about RAM and IMAP hibernation effectiveness stand.

> Also, do you use Dovecot IMAP proxies in front of your backend servers? If 
> so, how many IMAP sessions can one proxy server handle (assuming the proxy 
> does authorization using MySQL running on a separate server)? And, could the 
> proxy server be tuned to help in optimizing mostly IDLE backend sessions?
> 

Yes to Dovecot Proxying, of course.

No idea about MySQL, with (Open)LDAP nothing is breaking a sweat at an
average of 140 logins per second (IMAP and POP) on the 2 proxy servers.
If you can fit your entire dataset into RAM it should be fine, my LDAP
servers fall into that category and take about 10% of a slow (1.2GHz, 34%,
power-save mode) core only to handle the that load (also 2 servers).
And the rate of logins/s is what you need to worry about most and optimize
for. 

The proxies will of course have to do the shuffling of data and SSL
en/de-coding, but again they're not particular busy with that.

The number of sessions comes into play when looking at the number of login
processes on the proxies and their memory footprint.
An IMAP login process on the proxies in performance mode with a client
limit of 1000 will consume about 55MB at most. 
So assume at least 55KB RAM per session.

Read Timo's mail I linked to about IMAP hibernation, AFAIK nothing has
happened to make proxies more supportive for this though. 

Christian
> > On Feb 12, 2017, at 1:58 AM, Christian Balzer  wrote:
> > 
> > 
> > Hello,
> > 
> > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote:
> >   
> >>> 1. 256GB of real RAM, swap is for chums.
> >> 
> >> Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, 
> >> especially with fast SSD storage (which is a lot cheaper than RAM)?
> >>   
> > 
> > I'm sure about tax and death, not much else.
> > 
> > But as a rule of thumb I'd avoid swapping out stuff on production servers,
> > even if it were to SSDs.
> > Incidentally the servers I'm talking about here have their OS and swap on
> > Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC
> > S3610s.
> > 
> > Relying on the kernel to make swap decisions is likely to result in much
> > reduced performance even with fast SWAP when you're overcommitting things
> > on that scale.
> > 
> > 
> > But read on.
> >   
> >> Seems that these IMAP processes are long lived processes (idling most of 
> >> the time) that don’t need that much of the contents of real memory 
> >> available for much of the life of the process. I use a database proxy in 
> >> front of MySQL (for my web apps) so that there can be a large number of 
> >> TCP connections to the proxy where the frontend requests are queued for 
> >> execution using a small number of backend connections.
> >> 
> >> Could Dovecot IMAP be re-written to be more efficient so it works more 
> >> like MySQL (or other scalable data servers) that could handle a million or 
> >> more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP 
> >> sessions aren’t doing much most of the time and shouldn’t really average 
> >> 2MB of active data per session that needs to be resident in main memory at 
> >> all times.
> >>   
> > See IMAP hibernation:
> > https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html 
> > 
> > 
> > I'm going to deploy/test this in production in about 2 months from now,
> > but if you look at the link and the consequent changelog entries you'll see
> > that it has certain shortcomings and bug fixes in pretty much each release
> > after it was introduced.
> > 
> > But this is the correct way to tackle things, not SWAP.
> > 
> > Alas I'm not

Re: Replacement for antispam plugin

2017-02-12 Thread Håkon Alstadheim 


Den 12. feb. 2017 23:56, skrev Marcus Rueckert:
> On 2017-02-12 23:00:49 +0100, Håkon Alstadheim wrote:
>> Just follow the wiki and replace sa-learn scripts with calling dspam.
>> Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
>> line-endings (\r).
I had a typo here, (\r\n) gets replaced by just (\n).

> 
> tbh ... what do you do about mails which just use \r as separator?
> 

You are mistaken.

Firstly: s/\r$// will remove \r from the end of any line, do nothing if
there is no \r.
Secondly: All line-endings as seen by transfer-agents is transferred
with the same line endings (MTA-MTA: \r\n; locally: usually the same but
may be different). "Line ending" here pertains to after each header, and
between message-parts. If there are naked (\n) characters in the message
body, that is of no concern. Point of filter is to make Dspam-signature
parseable for dspam. If there is an (\r) at the end of the header, Dspam
will not find a match, and be unable to reclassify the mail.

> s|\r\n|\n| is safer.
> 
> darix
> 
>> like so:
>>
>> - learn-spam.sh -
>> #!/bin/sh
>> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
>>
>> - learn-ham.sh --
>> #!/bin/sh
>> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent
>>
>> --
>>
>> Theese will be invoked with the owner of the mailbox as current uid, so
>> that is all.
>

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/12/2017 om 11:00 PM schreef Håkon Alstadheim:
>
> Den 12. feb. 2017 21:44, skrev ebr...@whitehorsetc.com:
>>
>> Any opinion on dspam's interoperability with this?
>>
>>
> Just follow the wiki and replace sa-learn scripts with calling dspam.
> Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
> line-endings (\r).
>
> like so:
>
> - learn-spam.sh -
> #!/bin/sh
> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
>
> - learn-ham.sh --
> #!/bin/sh
> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent
>
> --
>
> Theese will be invoked with the owner of the mailbox as current uid, so
> that is all.

Actually, Pigeonhole should be able to do that too:

https://github.com/dovecot/pigeonhole/blob/master/doc/plugins/sieve_extprograms.txt#L112

Yes, I need to update the wiki.


Regards,

Stephan.

Re: Replacement for antispam plugin

2017-02-12 Thread Marcus Rueckert 
On 2017-02-12 23:00:49 +0100, Håkon Alstadheim wrote:
> Just follow the wiki and replace sa-learn scripts with calling dspam.
> Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
> line-endings (\r).

tbh ... what do you do about mails which just use \r as separator?

s|\r\n|\n| is safer.

darix

> like so:
> 
> - learn-spam.sh -
> #!/bin/sh
> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
> 
> - learn-ham.sh --
> #!/bin/sh
> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent
> 
> --
> 
> Theese will be invoked with the owner of the mailbox as current uid, so
> that is all.

-- 
   openSUSE - SUSE Linux is my linux
   openSUSE is good for you
   www.opensuse.org

Re: Replacement for antispam plugin

2017-02-12 Thread Håkon Alstadheim 


Den 12. feb. 2017 21:44, skrev ebr...@whitehorsetc.com:
> 
> 
> Any opinion on dspam's interoperability with this?
> 
> 

Just follow the wiki and replace sa-learn scripts with calling dspam.
Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
line-endings (\r).

like so:

- learn-spam.sh -
#!/bin/sh
sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam

- learn-ham.sh --
#!/bin/sh
sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent

--

Theese will be invoked with the owner of the mailbox as current uid, so
that is all.

Re: Replacement for antispam plugin

2017-02-12 Thread Håkon Alstadheim 


Den 12. feb. 2017 15:12, skrev Ralph Seichter:
> On 12.02.2017 13:25, Stephan Bosch wrote:
> 
>> The "imap.mailbox" environment is the empty string in this case. Why?
>> Well, the Sieve interpreter does not know about it, since the
>> "imapsieve" extension is not activated in the require line.
> 
> Now there's a facepalm moment. ;-) Thank you, with a modified 'require'
> statement things are working for me.
> 
> I see that https://wiki.dovecot.org/HowTo/AntispamWithSieve has already
> been updated, that's nice.

Tried to add a small variation for dSpam, but there is some anti-spam
functionality that I don't understand.

Mind putting this in after the sa-learn-ham.sh: ?
---
Or, if you are using dspam, (dropping 'sa-' as that would be misleading)

learn-spam.sh

{{{
#!/bin/sh

sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
}}}

learn-ham.sh

{{{
#!/bin/sh

sed -e 's/\r$//' | /usr/bin/dspam --debug --source=error --class=innocent
}}}

-

> 
>> You could debug this with the non-standard "vnd.dovecot.debug"
>> extension.
> 
> Thanks again, I will keep this in mind for future debugging.
> 
> -Ralph
>

Re: Replacement for antispam plugin

2017-02-12 Thread ebroch 


Any opinion on dspam's interoperability with this?




On Fri, Feb 10, 2017 at 1:07 AM -0700, "Aki Tuomi"  wrote:










Hi!
Since antispam plugin is deprecated and we would really prefer people
not to use it, we wrote instructions on how to replace it with
IMAPSieve. Comments and suggestions are most welcome.

https://wiki.dovecot.org/HowTo/AntispamWithSieve

---
Aki Tuomi
Dovecot oy

Re: Replacement for antispam plugin

2017-02-12 Thread George Kontostanos 
On Sun, Feb 12, 2017 at 8:56 PM, Stephan Bosch  wrote:
> Op 2/12/2017 om 5:36 PM schreef George Kontostanos:
>> On Sun, Feb 12, 2017 at 3:52 PM, Aki Tuomi  wrote:
>>
 On February 10, 2017 at 10:06 AM Aki Tuomi  wrote:


 Hi!
 Since antispam plugin is deprecated and we would really prefer people
 not to use it, we wrote instructions on how to replace it with
 IMAPSieve. Comments and suggestions are most welcome.

 https://wiki.dovecot.org/HowTo/AntispamWithSieve

 ---
 Aki Tuomi
 Dovecot oy
>>> Hi everyone,
>>>
>>> thank you all for your feedback, questions and comments. We have upgraded
>>> the documentation based on this, including information how to exclude Trash
>>> folder in ham script.
>>>
>>> Aki
>>>
>>
>> Thank you all very much. I am now running into a very weird issue. Whenever
>> an Junk email is “seen” I get the following error:
>>
>> Debug: sieve: vnd.dovecot.execute extension: no bin or socket directory
>> specified; extension is unconfigured (both sieve_execute_bin_dir and
>> sieve_execute_socket_dir are not set)
>> Feb 12 18:02:54 imap(user@domain): Debug: imapsieve: Static mailbox rule
>> [1]: mailbox=`Junk' from=`*' causes=(COPY) =>
>> before=`file:/usr/local/lib/dovecot/sieve/report-spam.sieve' after=(none)
>> Feb 12 18:02:54 imap(user@dmain): Debug: imapsieve: Static mailbox rule
>> [2]: mailbox=`*' from=`Junk' causes=(COPY) =>
>> before=`file:/usr/local/lib/dovecot/sieve/report-ham.sieve' after=(none)
>
> That is not an error. That is merely a debug message indicating that the
> vnd.dovecot.execute extension is unconfigured.
>
> Don't enable that extension if you're not using the "execute"
> command/test. Yes, it is in the example, but it has no function either.
> Only the vnd.dovecot.pipe extension is actually used ("pipe" command).
>
>> Also it automatically creates a .spamassassin/ folder in the user:
>>
>> -rw---  1 vmail  vmail136 Feb 10 17:33 .dovecot.lda-dupes
>> lrwx--  1 vmail  vmail 17 Mar 13  2016 .dovecot.sieve@ ->
>> managesieve.sieve
>> -rw---  1 vmail  vmail322 Feb 11 03:02 .dovecot.svbin
>> drwx--  2 vmail  vmail512 Feb 12 18:04 .spamassassin/
>> drwx--  5 vmail  vmail512 Feb 10 17:32 Drafts/
>> drwx--  5 vmail  vmail512 Feb 12 18:09 Junk/
>> drwx--  5 vmail  vmail512 Feb 11 17:36 Saved/
>> drwx--  5 vmail  vmail512 Jan 16 11:55 Sent/
>> drwx--  5 vmail  vmail512 Feb 12 18:09 Trash/
>> drwx--  2 vmail  vmail   1024 Feb 12 18:07 cur/
>> -rw---  1 vmail  vmail 21 Mar 13  2016 dovecot-acl-list
>> -rw---  1 vmail  vmail 68 Oct 17 13:19 dovecot-keywords
>> -rw---  1 vmail  vmail245 Feb 12 18:07 dovecot-uidlist
>> -rw---  1 vmail  vmail  8 Jun  9  2016 dovecot-uidvalidity
>> -r--r--r--  1 vmail  vmail  0 Mar 12  2016 dovecot-uidvalidity.56e48129
>> -rw---  1 vmail  vmail640 Feb 11 22:24 dovecot.index
>> -rw---  1 vmail  vmail  27788 Feb 12 18:07 dovecot.index.cache
>> -rw---  1 vmail  vmail  25996 Feb 12 18:07 dovecot.index.log
>> -rw---  1 vmail  vmail192 Feb 12 16:04 dovecot.mailbox.log
>> -rw---  1 vmail  vmail181 Feb 10 17:34 managesieve.sieve
>> drwx--  2 vmail  vmail512 Feb 12 17:46 new/
>> -rw---  1 vmail  vmail 39 Jun  9  2016 subscriptions
>> drwx--  2 vmail  vmail512 Feb 12 18:04 tmp/
>>
>> Any ideas ? I am attaching my config.
>
> The likely scenario is that the sa-learn tool is creating a hidden
> directory inside the $HOME directory of the user for user-specific state
> information. According to your mail_home and mail_location
> configuration, that is the same directory as the INBOX mailbox. That is
> why this may be interpreted as a mailbox by the maildir format. It is
> generally a bad idea to have those equal; you should put the mail
> location in a sub-directory of the home directory to prevent problems
> like this.
>
> https://wiki.dovecot.org/VirtualUsers/Home
>
> Regards,
>
> Stephan.
>
>
>
That makes perfect sense. I might need to find a way to migrate now to
the correct structure.

Thanks


-- 
George Kontostanos
---

Re: Replacement for antispam plugin

2017-02-12 Thread Ralph Seichter 
On 12.02.17 19:05, George Kontostanos wrote:

> Actually I think that sa-learn is invoked as user vmail. But of course
> I might be wrong.

It might depend on system configuration. On my servers, Sieve scripts are
definitely executed as the OS user that matches the current IMAP user.

> Do you have any suggestions as per the way sa-learn should be executed?

Instead of calling sa-learn directly, I use a script to store the piped
raw message in the file system, using separate directories for ham and
spam. This way, there are no delays. A periodic Cron job, running as my
global SpamAssassin user, later collects the files and invokes sa-learn,
ensuring that sync operations only happen once per run.

-Ralph

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/12/2017 om 5:36 PM schreef George Kontostanos:
> On Sun, Feb 12, 2017 at 3:52 PM, Aki Tuomi  wrote:
>
>>> On February 10, 2017 at 10:06 AM Aki Tuomi  wrote:
>>>
>>>
>>> Hi!
>>> Since antispam plugin is deprecated and we would really prefer people
>>> not to use it, we wrote instructions on how to replace it with
>>> IMAPSieve. Comments and suggestions are most welcome.
>>>
>>> https://wiki.dovecot.org/HowTo/AntispamWithSieve
>>>
>>> ---
>>> Aki Tuomi
>>> Dovecot oy
>> Hi everyone,
>>
>> thank you all for your feedback, questions and comments. We have upgraded
>> the documentation based on this, including information how to exclude Trash
>> folder in ham script.
>>
>> Aki
>>
>
> Thank you all very much. I am now running into a very weird issue. Whenever
> an Junk email is “seen” I get the following error:
>
> Debug: sieve: vnd.dovecot.execute extension: no bin or socket directory
> specified; extension is unconfigured (both sieve_execute_bin_dir and
> sieve_execute_socket_dir are not set)
> Feb 12 18:02:54 imap(user@domain): Debug: imapsieve: Static mailbox rule
> [1]: mailbox=`Junk' from=`*' causes=(COPY) =>
> before=`file:/usr/local/lib/dovecot/sieve/report-spam.sieve' after=(none)
> Feb 12 18:02:54 imap(user@dmain): Debug: imapsieve: Static mailbox rule
> [2]: mailbox=`*' from=`Junk' causes=(COPY) =>
> before=`file:/usr/local/lib/dovecot/sieve/report-ham.sieve' after=(none)

That is not an error. That is merely a debug message indicating that the
vnd.dovecot.execute extension is unconfigured.

Don't enable that extension if you're not using the "execute"
command/test. Yes, it is in the example, but it has no function either.
Only the vnd.dovecot.pipe extension is actually used ("pipe" command).

> Also it automatically creates a .spamassassin/ folder in the user:
>
> -rw---  1 vmail  vmail136 Feb 10 17:33 .dovecot.lda-dupes
> lrwx--  1 vmail  vmail 17 Mar 13  2016 .dovecot.sieve@ ->
> managesieve.sieve
> -rw---  1 vmail  vmail322 Feb 11 03:02 .dovecot.svbin
> drwx--  2 vmail  vmail512 Feb 12 18:04 .spamassassin/
> drwx--  5 vmail  vmail512 Feb 10 17:32 Drafts/
> drwx--  5 vmail  vmail512 Feb 12 18:09 Junk/
> drwx--  5 vmail  vmail512 Feb 11 17:36 Saved/
> drwx--  5 vmail  vmail512 Jan 16 11:55 Sent/
> drwx--  5 vmail  vmail512 Feb 12 18:09 Trash/
> drwx--  2 vmail  vmail   1024 Feb 12 18:07 cur/
> -rw---  1 vmail  vmail 21 Mar 13  2016 dovecot-acl-list
> -rw---  1 vmail  vmail 68 Oct 17 13:19 dovecot-keywords
> -rw---  1 vmail  vmail245 Feb 12 18:07 dovecot-uidlist
> -rw---  1 vmail  vmail  8 Jun  9  2016 dovecot-uidvalidity
> -r--r--r--  1 vmail  vmail  0 Mar 12  2016 dovecot-uidvalidity.56e48129
> -rw---  1 vmail  vmail640 Feb 11 22:24 dovecot.index
> -rw---  1 vmail  vmail  27788 Feb 12 18:07 dovecot.index.cache
> -rw---  1 vmail  vmail  25996 Feb 12 18:07 dovecot.index.log
> -rw---  1 vmail  vmail192 Feb 12 16:04 dovecot.mailbox.log
> -rw---  1 vmail  vmail181 Feb 10 17:34 managesieve.sieve
> drwx--  2 vmail  vmail512 Feb 12 17:46 new/
> -rw---  1 vmail  vmail 39 Jun  9  2016 subscriptions
> drwx--  2 vmail  vmail512 Feb 12 18:04 tmp/
>
> Any ideas ? I am attaching my config.

The likely scenario is that the sa-learn tool is creating a hidden
directory inside the $HOME directory of the user for user-specific state
information. According to your mail_home and mail_location
configuration, that is the same directory as the INBOX mailbox. That is
why this may be interpreted as a mailbox by the maildir format. It is
generally a bad idea to have those equal; you should put the mail
location in a sub-directory of the home directory to prevent problems
like this.

https://wiki.dovecot.org/VirtualUsers/Home

Regards,

Stephan.

Re: Replacement for antispam plugin

2017-02-12 Thread George Kontostanos 
On Sun, Feb 12, 2017 at 7:52 PM, Ralph Seichter  wrote:
> On 12.02.2017 17:36, George Kontostanos wrote:
>
>> it automatically creates a .spamassassin/ folder in the user
>
> That happens because sa-learn is invoked as the user who is logged into
> IMAP. If you want all users to contribute to a global SpamAssassin
> database (like I do), you'll need to create your own learning mechanism
> instead of calling sa-learn directly.
>
> -Ralph

Actually I think that sa-learn is invoked as user vmail. But of course
I might be wrong.

sa-learn-ham.sh

LOG='/var/log/sa-learn.log'
exec /usr/local/bin/sa-learn --ham -D >> $LOG 2>&1

I had to give ownership to vmail to sa-learn.log otherwise it would
refuse to run.

Do you have any suggestions as per the way sa-learn should be executed?

Thanks for your help.

-- 
George Kontostanos
---

Re: Replacement for antispam plugin

2017-02-12 Thread Ralph Seichter 
On 12.02.2017 17:36, George Kontostanos wrote:

> it automatically creates a .spamassassin/ folder in the user

That happens because sa-learn is invoked as the user who is logged into
IMAP. If you want all users to contribute to a global SpamAssassin
database (like I do), you'll need to create your own learning mechanism
instead of calling sa-learn directly.

-Ralph

Re: Replacement for antispam plugin

2017-02-12 Thread George Kontostanos 
On Sun, Feb 12, 2017 at 3:52 PM, Aki Tuomi  wrote:

>
> > On February 10, 2017 at 10:06 AM Aki Tuomi  wrote:
> >
> >
> > Hi!
> > Since antispam plugin is deprecated and we would really prefer people
> > not to use it, we wrote instructions on how to replace it with
> > IMAPSieve. Comments and suggestions are most welcome.
> >
> > https://wiki.dovecot.org/HowTo/AntispamWithSieve
> >
> > ---
> > Aki Tuomi
> > Dovecot oy
>
> Hi everyone,
>
> thank you all for your feedback, questions and comments. We have upgraded
> the documentation based on this, including information how to exclude Trash
> folder in ham script.
>
> Aki
>


Thank you all very much. I am now running into a very weird issue. Whenever
an Junk email is “seen” I get the following error:

Debug: sieve: vnd.dovecot.execute extension: no bin or socket directory
specified; extension is unconfigured (both sieve_execute_bin_dir and
sieve_execute_socket_dir are not set)
Feb 12 18:02:54 imap(user@domain): Debug: imapsieve: Static mailbox rule
[1]: mailbox=`Junk' from=`*' causes=(COPY) =>
before=`file:/usr/local/lib/dovecot/sieve/report-spam.sieve' after=(none)
Feb 12 18:02:54 imap(user@dmain): Debug: imapsieve: Static mailbox rule
[2]: mailbox=`*' from=`Junk' causes=(COPY) =>
before=`file:/usr/local/lib/dovecot/sieve/report-ham.sieve' after=(none)

Also it automatically creates a .spamassassin/ folder in the user:

-rw---  1 vmail  vmail136 Feb 10 17:33 .dovecot.lda-dupes
lrwx--  1 vmail  vmail 17 Mar 13  2016 .dovecot.sieve@ ->
managesieve.sieve
-rw---  1 vmail  vmail322 Feb 11 03:02 .dovecot.svbin
drwx--  2 vmail  vmail512 Feb 12 18:04 .spamassassin/
drwx--  5 vmail  vmail512 Feb 10 17:32 Drafts/
drwx--  5 vmail  vmail512 Feb 12 18:09 Junk/
drwx--  5 vmail  vmail512 Feb 11 17:36 Saved/
drwx--  5 vmail  vmail512 Jan 16 11:55 Sent/
drwx--  5 vmail  vmail512 Feb 12 18:09 Trash/
drwx--  2 vmail  vmail   1024 Feb 12 18:07 cur/
-rw---  1 vmail  vmail 21 Mar 13  2016 dovecot-acl-list
-rw---  1 vmail  vmail 68 Oct 17 13:19 dovecot-keywords
-rw---  1 vmail  vmail245 Feb 12 18:07 dovecot-uidlist
-rw---  1 vmail  vmail  8 Jun  9  2016 dovecot-uidvalidity
-r--r--r--  1 vmail  vmail  0 Mar 12  2016 dovecot-uidvalidity.56e48129
-rw---  1 vmail  vmail640 Feb 11 22:24 dovecot.index
-rw---  1 vmail  vmail  27788 Feb 12 18:07 dovecot.index.cache
-rw---  1 vmail  vmail  25996 Feb 12 18:07 dovecot.index.log
-rw---  1 vmail  vmail192 Feb 12 16:04 dovecot.mailbox.log
-rw---  1 vmail  vmail181 Feb 10 17:34 managesieve.sieve
drwx--  2 vmail  vmail512 Feb 12 17:46 new/
-rw---  1 vmail  vmail 39 Jun  9  2016 subscriptions
drwx--  2 vmail  vmail512 Feb 12 18:04 tmp/

Any ideas ? I am attaching my config.

Thanks

# 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: FreeBSD 10.3-RELEASE-p11 amd64  ufs
auth_mechanisms = plain login
auth_verbose = yes
default_client_limit = 2560
default_process_limit = 512
dict {
  acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
  quota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
}
log_path = /var/log/dovecot.log
mail_home = /usr/local/vhosts/mail/%d/%n
mail_location = maildir:/usr/local/vhosts/mail/%d/%n:LAYOUT=fs
mail_max_userip_connections = 20
mail_plugins = quota acl
mail_privileged_group = vmail
mail_shared_explicit_inbox = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date index ihave
duplicate mime foreverypart extracttext imapsieve
mbox_write_locks = fcntl
namespace {
  inbox = no
  list = children
  location =
maildir:/usr/local/vhosts/mail/%%d/%%n:LAYOUT=fs:INDEX=/usr/local/vhosts/indexes/%d/%n/shared/%%u:INDEXPVT=/usr/local/vhosts/indexes/%d/%n/shared/%%u
  prefix = shared/%%d/%%n/
  separator = /
  subscriptions = no
  type = shared
}
namespace inbox {
  inbox = yes
  list = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Junk {
auto = subscribe
special_use = \Junk
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
  type = private
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  acl = vfile
  acl_shared_dict = proxy::acl
  imapsieve_mailbox1_before =
file:/usr/local/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before =
file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *

Re: Replacement for antispam plugin

2017-02-12 Thread Ralph Seichter 
On 12.02.2017 13:25, Stephan Bosch wrote:

> The "imap.mailbox" environment is the empty string in this case. Why?
> Well, the Sieve interpreter does not know about it, since the
> "imapsieve" extension is not activated in the require line.

Now there's a facepalm moment. ;-) Thank you, with a modified 'require'
statement things are working for me.

I see that https://wiki.dovecot.org/HowTo/AntispamWithSieve has already
been updated, that's nice.

> You could debug this with the non-standard "vnd.dovecot.debug"
> extension.

Thanks again, I will keep this in mind for future debugging.

-Ralph

Re: Replacement for antispam plugin

2017-02-12 Thread Aki Tuomi 

> On February 10, 2017 at 10:06 AM Aki Tuomi  wrote:
> 
> 
> Hi!
> Since antispam plugin is deprecated and we would really prefer people
> not to use it, we wrote instructions on how to replace it with
> IMAPSieve. Comments and suggestions are most welcome.
> 
> https://wiki.dovecot.org/HowTo/AntispamWithSieve
> 
> ---
> Aki Tuomi
> Dovecot oy

Hi everyone,

thank you all for your feedback, questions and comments. We have upgraded the 
documentation based on this, including information how to exclude Trash folder 
in ham script.

Aki

Re: dovecot config for 1500 simultaneous connection

2017-02-12 Thread KT Walrus 
Thanks for the info. I do have one further question for you. On your servers 
that are currently handling 50k IMAP sessions, how many users does that 
correspond to? Since many users will have multiple IMAP sessions on multiple 
devices, I’d like to hear about some real-world numbers that could be used for 
budgeting a new project like mine.

Also, do you use Dovecot IMAP proxies in front of your backend servers? If so, 
how many IMAP sessions can one proxy server handle (assuming the proxy does 
authorization using MySQL running on a separate server)? And, could the proxy 
server be tuned to help in optimizing mostly IDLE backend sessions?

> On Feb 12, 2017, at 1:58 AM, Christian Balzer  wrote:
> 
> 
> Hello,
> 
> On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote:
> 
>>> 1. 256GB of real RAM, swap is for chums.  
>> 
>> Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, 
>> especially with fast SSD storage (which is a lot cheaper than RAM)?
>> 
> 
> I'm sure about tax and death, not much else.
> 
> But as a rule of thumb I'd avoid swapping out stuff on production servers,
> even if it were to SSDs.
> Incidentally the servers I'm talking about here have their OS and swap on
> Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC
> S3610s.
> 
> Relying on the kernel to make swap decisions is likely to result in much
> reduced performance even with fast SWAP when you're overcommitting things
> on that scale.
> 
> 
> But read on.
> 
>> Seems that these IMAP processes are long lived processes (idling most of the 
>> time) that don’t need that much of the contents of real memory available for 
>> much of the life of the process. I use a database proxy in front of MySQL 
>> (for my web apps) so that there can be a large number of TCP connections to 
>> the proxy where the frontend requests are queued for execution using a small 
>> number of backend connections.
>> 
>> Could Dovecot IMAP be re-written to be more efficient so it works more like 
>> MySQL (or other scalable data servers) that could handle a million or more 
>> IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions 
>> aren’t doing much most of the time and shouldn’t really average 2MB of 
>> active data per session that needs to be resident in main memory at all 
>> times.
>> 
> See IMAP hibernation:
> https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html 
> 
> 
> I'm going to deploy/test this in production in about 2 months from now,
> but if you look at the link and the consequent changelog entries you'll see
> that it has certain shortcomings and bug fixes in pretty much each release
> after it was introduced.
> 
> But this is the correct way to tackle things, not SWAP.
> 
> Alas I'm not expecting miracles and if more than 20% of the IMAP sessions
> here will be hibernated at any given time I'd be pleasantly surprised. 
> 
> Because between:
> 
> 1. Finding a sensible imap_hibernate_timeout. 
> 
> 2. Having well behaved clients that keep idling instead of restarting the
> sequence (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/ 
> )
> 
> 3. Having lots of mobile clients who either get disconnected (invisible to
> Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts
> (a large mobile carrier here times out idle TCP sessions after 2 minutes,
> forcing people to use 1 minute IDLE renewals, making 1. up there a
> nightmare).
> 
> 4. Having really broken clients (don't ask, I can't tell) which open IMAP
> sessions, don't put them into IDLE and thus having them expire after 30
> minutes.
> 
> the pool of eligible IDLE sessions isn't as big as it could be, in my case
> at least.
> 
>> My mail server isn’t that large yet as I haven’t fully deployed Dovecot 
>> outside my own small group yet, but it would be nice if scaling Dovecot IMAP 
>> to millions of users wasn’t limited to 50,000 IMAP sessions on a server...
>> 
> 
> Scaling up is nice and desirable from a cost (rack space, HW) perspective,
> but the scalability of things OTHER than Dovecot as I pointed out plus
> that little detail of failure domains (do you really want half of your
> eggs in one basket?) argue for scaling out after a certain density. 
> 
> I'm feeling my way there at this time, but expect more than 100k sessions
> per server to be tricky.
> 
> Lastly, when I asked about 500k sessions per server here not so long ago,
> ( http://www.dovecot.org/list/dovecot/2016-November/106284.html 
>  )
> Timo mentioned that he's not aware of anybody doing more than 50k per
> server, something I got licked already and definitely will go to 100k
> eventually.
> 
> Regards,
> 
> Christian
>>> On Feb 10, 2017, at 11:07 AM, Christian Balzer  wrote:
>>> 
>>> On Fri, 10 Feb 2017

Re: Maildirsize not updated

2017-02-12 Thread chaouche yacine 
Now this is interesting : du, doveadm quota get and maildirsize have three 
different values for this particular user : 


Max Quota is : 1G

du   :883M  (86%)
maildirsize  :  1048M (102%)
doveadm : 34402?  (32%)



Trace
-


root@messagerie[10.10.10.19] ~ # cd /var/vmail/domain.tld/m.stefan/
root@messagerie[10.10.10.19] ~ # alias dush
alias dush='du -h --max-depth=1 | sort -h'
root@messagerie[10.10.10.19] /var/vmail/domain.tld/m.stefan # dush 

4.0K./courierimapkeywords 
4.0K./new
4.0K./tmp
8.0K./courierimaphieracl
28K ./.INBOX.Altert
28K ./.INBOX.notifs
36K ./.BOITE_RECEPTION
36K ./.PIGE
80K ./dovecot
428K./.Drafts
728K./.Alert Process
1020K   ./.Trash
1.3M./.Junk
3.4M./cur
3.6M./.Alert BLANC
7.5M./.Alert PUB
9.7M./.IT SUPPORT
9.9M./.GLPI
12M ./.REGIE
15M ./.LTO
27M ./.VDN
29M ./.Sent
43M ./.HD SUPPORT
105M./.MAM
105M./.PRTG
514M./.INBOX
883M.
root@messagerie[10.10.10.19] /var/vmail/domain.tld/m.stefan # cat maildirsize 
1073741824S
1098898439 9380
-382366814 -439
382 1
27713 1
-383309268 -489
27433 1
4274 1
27740 1
31452 1
-665709 -153
5722 1
5797 1
27713 1
158840 1
27203 1
27744 1
26032 1
27717 1
1807 1
26989 1
27152 1
26239 1
3066 1
30846 1
4272 1
26020 1
27713 1
30040 1
26753 1
27152 1
27125 1
27744 1
26559 1
26166 1
29845 1
27121 1
27740 1
88624 1
32080 1
26672 1
27121 1
27152 1
26343 1
31456 1
27717 1
27152 1
26343 1
26032 1
29847 1
28309 1
29709 1
26241 1
26343 1
27152 1
31360 1
26016 1
27717 1
30857 1
27148 1
26343 1
28309 1
27152 1
26343 1
26168 1
28305 1
27822 1
27152 1
47273 1
26343 1
32056 1
26460 1
30523 1
28141 1
28152 1
28309 1
27148 1
26343 1
3999 1
1688 1
27990 1
28305 1
27152 1
26343 1
26032 1
27928 1
27479 1
90009 1
27366 1
27264 1
69705 1
27261 1
48383 1
28309 1
26241 1
27152 1
32079 1
30474 1
26016 1
30146 1
27713 1
29818 1
27152 1
30769 1
29194 1
27717 1
27740 1
1508 1
1536 1
1999 1
26146 1
1550 1
1587 1
27125 1
47289 1
27496 1
27744 1
32080 1
26464 1
32048 1
27713 1
27744 1
29845 1
27121 1
27744 1
26032 1
30146 1
27121 1
45454 1
26241 1
28332 1
32103 1
3859 1
26016 1
27121 1
28336 1
4272 1
29709 1
29688 1
27125 1
28336 1
1757 1
4631 1
54951 1
26170 1
1757 1
1975 1
29765 1
28882 1
1757 1
25683 1
71184 1
28332 1
32080 1
55040 1
26464 1
1757 1
4631 1
30706 1
4322 1
49607 1
1757 1
1757 1
46535 1
47378 1
27717 1
27744 1
21723 1
1759 1
1757 1
20218 1
21737 1
21724 1
539443 1
1892 1
27713 1
27744 1
26036 1
27713 1
27740 1
32079 1
12815596 1
26523 1
54511 1
26020 1
27125 1
29847 1
28336 1
30423 1
27009 1
27065 1
26510 1
27121 1
324880 1
27740 1
21128 1
213811 1
20186 1
20239 1
26146 1
21360 1
29481 1
25668 1
27826 1
28128 1
27125 1
31333 1
1820 1
28332 1
26488 1
27778 1
31127 1
4290 1
4290 1
4309 1
27121 1
27744 1
4290 1
8871 1
8871 1
4290 1
1773 1
30578 1
47705 1
10764 1
10764 1
30433 1
31066 1
4290 1
30461 1
31094 1
26680 1
29135 1
29141 1
26032 1
30146 1
30109 1
31054 1
29226 1
31483 1
26016 1
29949 1
30890 1
30095 1
29226 1
5880 1
29173 1
30422 1
root@messagerie[10.10.10.19] /var/vmail/domain.tld/m.stefan # doveadm quota get 
-u m.ste...@domain.tld
Quota name Type Value   Limit   %
User quota STORAGE 344402 1048576  32
User quota MESSAGE   8542   -   0
root@messagerie[10.10.10.19] /var/vmail/domain.tld/m.stefan # 





On Sunday, February 12, 2017 2:18 PM, chaouche yacine 
 wrote:
I am using dovecot lmtp

root@messagerie[10.10.10.19] ~ # grep virtual_transport /etc/postfix/main.cf
# transport_maps = hash:/var/lib/mailman/data/transport-mailman, 
proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
# virtual_transport = maildrop
virtual_transport = lmtp:unix:private/dovecot-lmtp
root@messagerie[10.10.10.19] ~ # 






On Thursday, February 9, 2017 7:54 PM, WJCarpenter  
wrote:



Who delievers incoming mail, dovecot LDA or something else?

This is what caused a similar problem for me: 
https://dovecot.org/list/dovecot/2016-April/104091.html

Re: Maildirsize not updated

2017-02-12 Thread chaouche yacine 
I am using dovecot lmtp

root@messagerie[10.10.10.19] ~ # grep virtual_transport /etc/postfix/main.cf
# transport_maps = hash:/var/lib/mailman/data/transport-mailman, 
proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
# virtual_transport = maildrop
virtual_transport = lmtp:unix:private/dovecot-lmtp
root@messagerie[10.10.10.19] ~ # 





On Thursday, February 9, 2017 7:54 PM, WJCarpenter  
wrote:



Who delievers incoming mail, dovecot LDA or something else?

This is what caused a similar problem for me: 
https://dovecot.org/list/dovecot/2016-April/104091.html

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/10/2017 om 10:15 PM schreef George Kontostanos:
> On Fri, Feb 10, 2017 at 9:59 PM, Ralph Seichter 
> wrote:
>
>
> Same problem here. As a workaround I tried the following:
>
> # From Spam folder to Inbox
>   imapsieve_mailbox2_name = Inbox
>   imapsieve_mailbox2_from = Spam
>   imapsieve_mailbox2_causes = COPY
>   imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve
>
> In theory this should trigger the  report-ham script, only if a mail is
> moved from Spam to Inbox. However, it does not seem to work either.


That is because you found a rather stupid, yet minor bug. The "INBOX"
mailbox is rather special, since it is the only mailbox name that is
treated case-insensitively. The normal form is "INBOX". Due to the bug,
your "Inbox" will not match "INBOX". You can work around this for now by
specifying "INBOX" instead. Well, you will not need this workaround for
a workaround anyway if the fix I proposed in my earlier message also
works for you. But for posterity...


Regards,

Stephan.

Re: Plugin sieve vs. imap_sieve

2017-02-12 Thread Stephan Bosch 
Op 2/12/2017 om 1:15 PM schreef Matthias Fechner:
> Dear all,
>
>
> I just saw that a "new" plugin imap_sieve is available:
> https://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve
>
> I configured everything using the plugin sieve.
>
> Was the plugin sieve just renamed to imap_sieve or is it completely
> different?

It is completely different. The sieve plugin is a plugin for LDA and
LMTP, which perform final delivery of a message. The imap_sieve plugin
is for IMAP and invokes Sieve scripts for messages that are already stored.

> Thanks a lot for explaining what had been changed here, I cannot find
> anything in the wiki.

For one, the wiki page you referenced should explain that pretty well.
You could also read the referenced http://tools.ietf.org/html/rfc6785 to
find out what features this plugin adds in detail.

Regards,

Stephan.

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/10/2017 om 8:59 PM schreef Ralph Seichter:
> On 10.02.17 20:34, Michael Slusarz wrote:
>
>> When you move a message to a new mailbox, that is a "new message"
>> event (a new UID in the target mailbox is created; the message count
>> increases). So imap.mailbox is set to the name of the *target* mailbox.
> My tests seem to indicate otherwise. Deleting a message currently
> located in the Junk folder causes the report-ham.sieve script to be
> invoked, which in my case contains the following:
>
>   require ["vnd.dovecot.pipe", "copy", "environment"];
>   if environment "imap.mailbox" "Trash" {
> stop;
>   } elsif environment "imap.mailbox" "Junk" {
> pipe :copy "debug-junk";
>   } else {
> pipe :copy "learn-ham";
>   }
>
> I can see that "learn-ham" is always invoked when a message is deleted
> from or moved out of the Junk folder, so my guess is that imap.mailbox
> is neither "Trash" nor "Junk" ("debug-junk" is never called, according
> to the Dovecot logs). Unfortunately I don't know how to debug this
> further.
>
> I don't want "learn-ham" to be run when a message located in the Junk
> folder is manually deleted or moved to Trash, and right now I don't know
> how to accomplish this.

The "imap.mailbox" environment is the empty string in this case. Why?
Well, the Sieve interpreter does not know about it, since the
"imapsieve" extension is not activated in the require line.

You could debug this with the non-standard "vnd.dovecot.debug"
extension. I debugged it like this:

require ["imapsieve", "environment", "variables", "vnd.dovecot.debug"];

if environment :matches "imap.mailbox" "*" {
set "mailbox" "${1}";
}

debug_log "imap.mailbox = ${mailbox}";

if string "${mailbox}" "Trash" {
stop;
}

debug_log "REPORT HAM";

This will log one or two info messages, depending on what the target
mailbox is.

I tested this with the example configuration and it works fine. I've
verified this by invoking the imap service directly from the command
line, so that it is immediately logged-in and spews al debug messages
directly to stderr:

$ sudo /usr/lib/dovecot/imap -u harrie
imap(harrie)<>: Debug: Loading modules from directory:
/usr/lib/dovecot/modules
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib15_notify_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_fts_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_mail_log_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_replication_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib20_virtual_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
imap(harrie)<>: Debug: Module loaded:
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so
imap(harrie)<>: Debug: auth USER input: harrie uid=1000 gid=124
home=/ext/test/home/test/harrie
Debug: Effective uid=1000, gid=124, home=/ext/test/home/test/harrie
Debug: replication: No mail_replica setting - replication disabled
Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes,
hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir
Debug: maildir++: root=/ext/test/home/test/harrie/Maildir, index=,
indexpvt=, control=, inbox=/ext/test/home/test/harrie/Maildir, alt=
Debug: Namespace : type=private, prefix=virtual/, sep=/, inbox=no,
hidden=no, list=yes, subscriptions=yes location=virtual:~/Maildir/virtual
Debug: fs: root=/ext/test/home/test/harrie/Maildir/virtual, index=,
indexpvt=, control=, inbox=, alt=
* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE LITERAL+ SEARCH=FUZZY
NOTIFY URLAUTH URLAUTH=BINARY METADATA SPECIAL-USE] Logged in as harrie
23423 SELECT Spam
imap(harrie): Debug: Namespace : Using
permissions from /ext/test/home/test/harrie/Maildir: mode=0700 gid=default
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
Flags permitted.
* 1 EXISTS
* 1 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1485122806] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
23423 OK [READ-WRITE] Select completed (0.000 + 0.000 secs).

This is what happens for moving from Spam to a non-Trash mailbox:

2343 MOVE 1 "Frop"
imap(harrie): Debug: imapsieve: mailbox Frop:
MOVE event
imap(harrie): Debug: sieve: Pigeonhole version
0.5.devel (462a535) initializing
imap(harrie): Debug: sieve: include:
sieve_global is not set; it is currently not possible to include
`:global' scripts.
imap(harrie): Debug: sieve: Pigeonhole Sieve PGP
Encrypt plugin version 0.2.devel loaded
imap(harrie): Debug: sieve: Sieve imapsieve
plugin for Pigeonhole version 0.5.devel (462a535) loaded
imap(harrie): Debug:

Re: antispam plugin pipe backend error when moving multiple emails

2017-02-12 Thread Sergey Urushkin 

Hello,
wondering why it's still an issue with current git, while this old 
working solution exists (tested with dovecot 2.22 and rspamd 1.4) :


https://www.dovecot.org/list/dovecot/2013-November/093810.html

---
Best regards,
Sergey Urushkin


Stéphane Cottin писал 2016-04-18 15:44:

Hello,

I'm bumping this because it still occurs with dovecot 2.2.22.

my dovecot-antispam plugin configuration :

  antispam_allow_append_to_spam = no
  antispam_backend = pipe
  antispam_pipe_program = /usr/bin/rspamc
  antispam_pipe_program_args = -h;127.0.0.1:11334;-P;
  antispam_pipe_program_notspam_arg = learn_ham
  antispam_pipe_program_spam_arg = learn_spam
  antispam_pipe_tmpdir = /var/tmp
  antispam_spam = Junk
  antispam_trash = trash;Trash;Deleted Items;Deleted Messages

zlib enabled:
  zlib_save = gz
  zlib_save_level = 9


When moving 2 or more messages from inbox to the Junk folder:

"J47 NO [CANNOT] Failed to copy to temporary file (0.000 + 0.000
secs).”. Command attempted: “J47 UID MOVE 106318:106319 Junk"

or sometimes

"J123 NO [CANNOT] Failed to read mail beginning (0.000 + 0.000
secs).”. Command attempted: “J123 UID MOVE 170789:170790 Junk"

and still have the "Cached message size smaller..." in dovecot logs.

It occurs at least when header lines of an email contains Non-ASCII
Text (rfc1342).
Batches of full ascii emails are not affected.

I can easily reproduce this from/to the Junk folder, but had
unconfirmed reports of similar errors when batch moving mails across
regular folders.

Stéphane


On 8 Nov 2015, at 11:50, Stéphane Cottin wrote:


Hi,

I've got some trouble with the dovecot antispam plugin and the pipe 
backend.


I'm using dovecot 2.2.18 with maildirs and zlib compression enabled.

When moving 2 or more emails at once from the Junk folder to another 
one, I always have the following error : "Failed to copy to temporary 
file"


In the server logs :

imap(v...@vvv.vvv): Error: 
read(zlib(/data/Maildir/.test/tmp/1446974366.M123890P936.vvv)) failed: 
Cached message size smaller than expected (13553 < 13562, box=test, 
UID=0)


The same operation with one email at a time, on the same emails, works 
as expected.


Stéphane

Plugin sieve vs. imap_sieve

2017-02-12 Thread Matthias Fechner 
Dear all,


I just saw that a "new" plugin imap_sieve is available:
https://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/IMAPSieve

I configured everything using the plugin sieve.

Was the plugin sieve just renamed to imap_sieve or is it completely
different?

Thanks a lot for explaining what had been changed here, I cannot find
anything in the wiki.


Gruß
Matthias

-- 

"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the universe trying to
produce bigger and better idiots. So far, the universe is winning." --
Rich Cook

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/10/2017 om 8:46 PM schreef David Mehler:
> Hello,
>
> Chiming in on this with a question, and will be getting to it over the
> weekend or later this evening time permitting.
>
> Does retraining a message as either spam or ham alter message headers
> for example x-spam or the spamassassin-modified subject header?
>
> If not is it possible to do so after processing? For example, I have a
> message inadvertently tagged as spam, from Spamassassin it gets an
> x-spam header added as well as a modified subject. Retraining that
> message as ham moving it to say any other folder but spam i'd like for
> that x-spam header to be set to as it is not spam, and the
> spamassassin subject to be removed. Is this doable?

I think you could use the "editheader" extension in the report-ham.sieve
script to do that. You'll need "variables" as well to modify the subject.

Regards,

Stephan.

Re: Replacement for antispam plugin

2017-02-12 Thread Aki Tuomi 

> On February 12, 2017 at 1:32 PM Stephan Bosch  wrote:
> 
> 
> Op 2/10/2017 om 2:20 PM schreef Alessio Cecchi:
> > Il 10/02/2017 09:06, Aki Tuomi ha scritto:
> >> Hi!
> >> Since antispam plugin is deprecated and we would really prefer people
> >> not to use it, we wrote instructions on how to replace it with
> >> IMAPSieve. Comments and suggestions are most welcome.
> >>
> >> https://wiki.dovecot.org/HowTo/AntispamWithSieve
> > Hi,
> >
> > imap_stats plugin is required?
> 
> No, that is just part of the example.
> 
> Regards,
> 
> Stephan.

And removed from the example too, thank you for pointing this out.

Aki

Re: Replacement for antispam plugin

2017-02-12 Thread Stephan Bosch 
Op 2/10/2017 om 2:20 PM schreef Alessio Cecchi:
> Il 10/02/2017 09:06, Aki Tuomi ha scritto:
>> Hi!
>> Since antispam plugin is deprecated and we would really prefer people
>> not to use it, we wrote instructions on how to replace it with
>> IMAPSieve. Comments and suggestions are most welcome.
>>
>> https://wiki.dovecot.org/HowTo/AntispamWithSieve
> Hi,
>
> imap_stats plugin is required?

No, that is just part of the example.

Regards,

Stephan.

Re: Managesieve cannot access script store

2017-02-12 Thread Stephan Bosch 
Op 2/11/2017 om 3:24 PM schreef dovelist:
> OK, I've figured it out:
>
> In the dovecot profile for apparmor the sieve directory is not
> confgured. I solved it this way:
>
> To configure only one directory in the apparmor profile, I placed the
> active-script link inside the .sieve  directory. Keeping the scripts
> separate in a store subdirectory, like this:
> In /etc/dovecot/conf.d/90-sieve.conf :
>
>sieve = file:~/.sieve/store;active=~/.sieve/active.sieve
>
> Then dovecot is granted access by adding the .sieve directory in the
> apparmor profile. The dovecot file in the tunables directory seems to
> be a neat way to that:
> In /etc/apparmor.d/tunables/dovecot :
>
>@{DOVECOT_MAILSTORE}=@{HOME}/Maildir/ /var/spool/mail/ @{HOME}/.sieve/
>
> Ofcourse the .sieve directory is not really a MAILSTORE. But this way,
> the configuration stays close to the defaults. I didn't find something
> like DOVECOT_SIEVESTORE, which would be more appropriate.
>
> After restart of apparmor and dovecot, it works!
>
> @Stephan: thanks for the advice - it did help to pinpoint the problem! 

I have no experience with AppArmor. I assume these profile configuration
files are created by the packagers for your distribution. You could talk
to them to get this fixed in general.

Regards,

Stephan.

Re: Managesieve cannot access script store

2017-02-12 Thread Stephan Bosch 
Op 2/10/2017 om 5:05 PM schreef dovelist:
> Hi Stephan,
>
>> Normally, Dovecot permission errors are more helpful than that. So, this
>> error message in itself is a bit of a bug:
>
> I'm glad to h've been able to help with this beta-test ;-)
>
>
>> About the cause of this error: keep in mind that the whole directory
>> path needs read/execute permission, not only the leaf directory.
>
> Have checked. They are...
>
>
>> You could try a command other than LISTSCRIPTS in your manual debugging
>> efforts. That should take a different code path that provides a more
>> detailed error.
>
> I tried:
>
> PUTSCRIPT "hutsefluts" {6+}
> keep;
>
> Gives the same result:
>
> Feb 10 15:43:26 p150 dovecot[2042]: managesieve(rogier): Error: sieve:
> file storage: save:
> open(/home/rogier/sieve/tmp/hutsefluts_1486737806.M728733P6414.p150.sieve)
> failed: Permission denied

Ah, so there's more. Fixed that too:

https://github.com/dovecot/pigeonhole/commit/34d44f7ad9e872dec6ffa62de2642cb91ad5f6fc

Regards,

Stephan.