Re: Dovecot sieve filters

[Håkon Alstadheim]

Den 19.09.2021 16:37, skrev Gedalya:

On 9/19/21 21:24, j.emerlik wrote:

Error: sieve: report-ham: line 1: the envelope extension cannot be used in this 
context (needs access to message envelope)


My guess would be that the envelope is not available because this is sieve 
running in IMAP, not during delivery.

If the From: header is also good, maybe try if address :is "from" "*" { .. }

Sounds right. There /is/ no envelope in  the on-disk copy of the mail. 
The "from" line at the very top of the mail is _usually_ tere, and is a 
copy of the envelope from, but it is not strictly defined to be 
anything  in particular.  I see you are trying to down-case the 
address-part of the from-line. Why not do that in the shell-script 
"sa-learn-ham.sh" ?

Re: Antispam plugin

[Håkon Alstadheim]

Den 17.01.2021 02:15, skrev Toni Mueller:

Hi,

On Tue, Sep 22, 2020 at 02:01:21PM +0300, Aki Tuomi wrote:

You don't need antispam plugin:

https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/

I also don't see any docs about the plugin on the wiki.
Does it mean that this plugin is deprecated?

The link describes how the functionality of the antispam plugin is no 
longer needed, you can set up antispam with built-in functionality of 
dovecot.

Re: Mail slowed down to a crawl...

[Håkon Alstadheim]

Den 15.11.2018 21:00, skrev StarionTech (IMAP):

Been moving along just fine for a couple years now, then in the last two days, 
email has slowed to a crawl.

Retrieving email via IMAP is very slow, progress bar in the mail client shows 
that it is downloading messages almost constantly, like it never closes the 
connection, or it’s getting the messages very slowly.  The server that 
dovecot/postfix is on has also bogged down to a crawl.

The postfix queue is sitting with hundreds of queued up good customer email.

It’s almost acting like dovecot is not getting the messages from postfix, or 
they’re being delayed for some reason.

How to approach fixing this?


Check server logs for errors. Both dovecot and postfix struggling 
suggest some common underlying cause.


Do measurements  on Cpu, disk, cpu-wait. Which process is hogging the 
server ?  Spam-ass acting up? Drives filling up?


Gut instinct says one or more of your drives are failing, or you have 
file or file-system corruption, so if you are not running smartd, first 
priority is installing smartd/smartmontools and see what falls out.

Re: after reboot listen(*, 995) failed: Address already in use/listen(*, 993) failed: Address already in use

[Håkon Alstadheim]

Den 08.11.2018 17:52, skrev Robert Kudyba:
This is still happening after a reboot, Fedora 28. Restarting dovecot 
fixes the problem. Does anyone know if it could be related to this bug 
report? *https://bugzilla.redhat.com/show_bug.cgi?id=103401#c130 * and 
suggested work around to add ports to 
/proc/sys/net/ipv4/ip_local_reserved_ports?

Yes.

Nov  8 12:21:41 ourdomain dovecot[1386]: Error: service(pop3-login): listen(*, 
995) failed: Address already in use
Nov  8 12:21:41 ourdomain dovecot[1386]: Error: service(pop3-login): listen(::, 
995) failed: Address already in use
Nov  8 12:21:41 ourdomain dovecot[1386]: Error: service(imap-login): listen(*, 
993) failed: Address already in use
Nov  8 12:21:41 ourdomain dovecot[1386]: Error: service(imap-login): listen(::, 
993) failed: Address already in use


If you have nfs clients running, they often step on ports in that 
range.  Alternate work-around is setting sunrpc.min_resvport = 994 in 
/etc/sysctl.d/99-whatever.conf


Better would be to move away from port-based "security" (not actually 
secure at all) . Either switch to samba or nfs v4



dovecot --version
2.2.36 (1f10bfa63)

dovecot -n
# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# OS: Linux 4.18.16-200.fc28.x86_64 x86_64 Fedora release 28 (Twenty 
Eight)

# Hostname: ourdomain.com 
auth_debug = yes
debug_log_path = /var/log/dovecot-debug.log
mail_debug = yes
mbox_write_locks = fcntl
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  driver = pam
}
ssl = required
ssl_cert = 

ssl_cipher_list = PROFILE=SYSTEM
ssl_key =  # hidden, use -P to show it
userdb {
  driver = passwd
}
protocol imap {
  mail_location = mbox:~/mail:INBOX=/var/mail/%u

Re: Really slow IMAP performance

[Håkon Alstadheim]

Den 24. feb. 2018 15:04, skrev Neil Jerram:
> Aki Tuomi  writes:
> 
>>> On 24 February 2018 at 15:47 Neil Jerram  wrote:
> 
> [...]
  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning:
 Transaction log file
 /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked
 for 98 seconds (Mailbox was synchronized)
> [...]
> 
>> You are using mbox format. This is ... bit slow. =)
>>
>> When you move mails between mbox files, it has to rewrite the entire mbox 
>> file every time. You should probably start using maildir or sdbox instead.
> 
> Ah, right, thanks.
> 
> My INBOX file has 22990 messages.  Is the slowness that I am seeing
> definitely expected for an mbox of that size?  (It may also be relevant
> that the HDD it's stored on is pretty old now, and has been known to
> report SMART errors...)
> 
> If so, I'll start looking at how to migrate, given that my system is
> Postfix + Dovecot.  If you have any particular recommendations or
> migration pointers for a system like that, I'd appreciate them.
> 

I'd go with whatever tools you are familiar with. If you don't know
where to start, formail(1) can read an mbox and do whatever for each
mail contained therein. This, together with procmail, used to be the
go-to tools in the days before IMAP.

Theese days you'd probably want to involve your local delivery agent on
the output from formail. The lda would invoke sieve instead of procmail
if that is your thing.

Whatever you do, try to set up so you can do some tests before you blast
22000 mails to somewhere you do not want them :-)

Re: Mark message as read when moved to Trash

[Håkon Alstadheim]

Den 25. nov. 2017 11:36, skrev Peter Chiochetti:
> Am 2017-11-25 um 10:12 schrieb Felix Rubio Dalmau:
>> Hi all,
>>
>> When I send a message to trash, without previously reading it
>> (just with the subject is enough to say I do not want to read it), it
>> remains as "unread". Then, clients (I am using Kmail) report there are
>> unread message, when all of them are in Trash. The question, then is:
>> Is there any way to automatically mark a message as read, when that
>> message is moved to Trash?
>>
>> Thank you!
>> Felix
>>
> 
> Hello Felix, the Kmail list might be a better fit for that question,
> dont you think so?
> 

Maybe not. Look into imapsieve, it should have all the functionality you
need and more.

Re: Replacement for antispam plugin

[Håkon Alstadheim]

Den 12. feb. 2017 23:56, skrev Marcus Rueckert:
> On 2017-02-12 23:00:49 +0100, Håkon Alstadheim wrote:
>> Just follow the wiki and replace sa-learn scripts with calling dspam.
>> Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
>> line-endings (\r).
I had a typo here, (\r\n) gets replaced by just (\n).

> 
> tbh ... what do you do about mails which just use \r as separator?
> 

You are mistaken.

Firstly: s/\r$// will remove \r from the end of any line, do nothing if
there is no \r.
Secondly: All line-endings as seen by transfer-agents is transferred
with the same line endings (MTA-MTA: \r\n; locally: usually the same but
may be different). "Line ending" here pertains to after each header, and
between message-parts. If there are naked (\n) characters in the message
body, that is of no concern. Point of filter is to make Dspam-signature
parseable for dspam. If there is an (\r) at the end of the header, Dspam
will not find a match, and be unable to reclassify the mail.

> s|\r\n|\n| is safer.
> 
> darix
> 
>> like so:
>>
>> - learn-spam.sh -
>> #!/bin/sh
>> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
>>
>> - learn-ham.sh --
>> #!/bin/sh
>> sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent
>>
>> --
>>
>> Theese will be invoked with the owner of the mailbox as current uid, so
>> that is all.
> 

Re: Replacement for antispam plugin

[Håkon Alstadheim]

Den 12. feb. 2017 21:44, skrev ebr...@whitehorsetc.com:
> 
> 
> Any opinion on dspam's interoperability with this?
> 
> 

Just follow the wiki and replace sa-learn scripts with calling dspam.
Dspam direct pipe needs mail-line-endings (\r\n) translated into unix
line-endings (\r).

like so:

- learn-spam.sh -
#!/bin/sh
sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam

- learn-ham.sh --
#!/bin/sh
sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=innocent

--

Theese will be invoked with the owner of the mailbox as current uid, so
that is all.

Re: Replacement for antispam plugin

[Håkon Alstadheim]

Den 12. feb. 2017 15:12, skrev Ralph Seichter:
> On 12.02.2017 13:25, Stephan Bosch wrote:
> 
>> The "imap.mailbox" environment is the empty string in this case. Why?
>> Well, the Sieve interpreter does not know about it, since the
>> "imapsieve" extension is not activated in the require line.
> 
> Now there's a facepalm moment. ;-) Thank you, with a modified 'require'
> statement things are working for me.
> 
> I see that https://wiki.dovecot.org/HowTo/AntispamWithSieve has already
> been updated, that's nice.

Tried to add a small variation for dSpam, but there is some anti-spam
functionality that I don't understand.

Mind putting this in after the sa-learn-ham.sh: ?
---
Or, if you are using dspam, (dropping 'sa-' as that would be misleading)

learn-spam.sh

{{{
#!/bin/sh

sed -e 's/\r$//' | /usr/bin/dspam --source=error --class=spam
}}}

learn-ham.sh

{{{
#!/bin/sh

sed -e 's/\r$//' | /usr/bin/dspam --debug --source=error --class=innocent
}}}

-

> 
>> You could debug this with the non-standard "vnd.dovecot.debug"
>> extension.
> 
> Thanks again, I will keep this in mind for future debugging.
> 
> -Ralph
> 

Re: Replacement for antispam plugin

[Håkon Alstadheim]

Nice. Finally got around to switching. Had to add filter to my dspam
pipe (another one I'm overdue for replacing) . Antispam plugin seems to
have automatically converted line endings from "mail-type" '\r\n' into
linux plain '\n' .

Like so:
sed -e 's/\r$//' | dspam ...


Den 10. feb. 2017 09:06, skrev Aki Tuomi:
> Hi!
> Since antispam plugin is deprecated and we would really prefer people
> not to use it, we wrote instructions on how to replace it with
> IMAPSieve. Comments and suggestions are most welcome.
> 
> https://wiki.dovecot.org/HowTo/AntispamWithSieve
> 
> ---
> Aki Tuomi
> Dovecot oy
> 

Re: CPU for Dovecot

[Håkon Alstadheim]

Den 25. nov. 2016 14:29, skrev Miloslav Hůla:
> Hi,
>
> we are planning to change hardware for our standalone Dovecot instance
> handling ~5800 IMAP users with 1TB mailboxes on local RAID. Is there
> some recommendation about CPU?
>
> We can choose from:
>  - Intel Xeon E5-2620v4 - 2,1GHz@8,0GT 20MB cache, 8core, HT, 85W,
> LGA2011
>  - Intel Xeon E5-2623v4 - 2,6GHz@8,0GT 10MB cache, 4core, HT, 85W,
> LGA2011
>
> The difference is about more cores vs. hi frequency.
As far as I can see, high frequency costs more than more cores, so the
2623 is more geared towards interactive use, or single-threaded servers
(like a single java thread). 50% reduction in number of cores gets you
only a few percent increase in frequency. Go with 2620 for mail server.

Re: deb repo with dovecot-antispam ?

[Håkon Alstadheim]

Den 02. mai 2016 02:39, skrev Stephan Bosch:
> Op 5/2/2016 om 12:38 AM schreef Håkon Alstadheim:
>> There are .debs available for version 2.2 both from backports and
>> ix.dovecot.fi, but neither seem to have the antispam plugin built for
>> version 2.2. Anybody know why, and (if there is no particular reason)
>> how to get dovecot-antispam in a proper .deb package for dovecot-2.2.* ?
> Including it in the current Xi build process isn't trivial. I haven't
> needed it so far, so it never got included.
>
> I played with this a bit. The attached crude shell script will create a
> ./build directory in which a fresh debian package is built. It needs at
> least git and devscripts packages installed. Adjust the script as needed.
>
> Clear the build directory manually if you need to start over.
>
> You can also use this shell script as a step-by-step guide on how to
> make a fresh Debian package for it:
>
> - It clones git repositories for dovecot-antispam and its debian package.
> - From dovecot-antispam it creates the required orig.tar.gz file and
> unpacks it.
> - The debian directory is copied from the official Debian package. This
> will only work if the latest dovecot-antispam is still compatible.
> Currently, it is.
> - Its changelog is updated for the new version.
> - The package is built.
>
> At my end it builds successfully, but I haven't actually tried the
> resulting package.
>
> Regards,
>
> Stephan.
>
>
Thanks! Don't have time to try it now, but I have a script which will
unhold dovecot, update and then build dovecot-antispam iff there was a
new version of dovecot, and then hold dovectot. I'll try plugging your
script in there, and see how It goes.

deb repo with dovecot-antispam ?

[Håkon Alstadheim]

There are .debs available for version 2.2 both from backports and
ix.dovecot.fi, but neither seem to have the antispam plugin built for
version 2.2. Anybody know why, and (if there is no particular reason)
how to get dovecot-antispam in a proper .deb package for dovecot-2.2.* ?

Re: ot: data consumption IMAP vs POP

[Håkon Alstadheim]

Den 27. jan. 2016 07:30, skrev Steffen Kaiser:
> On Wed, 27 Jan 2016, voy...@sbt.net.au wrote:
>
> > I have a domain with about 50 mailboxes, server is located here in
> AUS but
> > most of the users are on a LAN is SEAsia location. They were using POP
> > (with Thunderbird), I suggested then can use IMAP instead, so they did.
>
> > now they are asking;
>
> > "Looks like Imap is adding a lot to our internet bandwidth"
>
> > I guess they have some bandwidth limitation on their link
>
> which limit(s)?
>
> > I think I can understand that IMAP would increase bandwidth requirement,
> > didn't expect it to cause 'problems'
>
> > is there any optimization or changes I can make to reduce that ?
> > the b/w limitation are at the client LAN link
>
> > any other suggestions ?
>
> Do they have problems more while sending or more while reading or more
> when doing "flagging, moving, deleting"?
>
> Sending bandwitdh can be reduced by using BCC instead of the IMAP
> append to the sent mailbox.
> Reading bandwidth should not change, unless they watch really many
> mailboxes.
>
> -- Steffen Kaiser
... but of course re-configuring mail-client will cause all mail that is
still on the server to be re-down-loaded. Did you specifically ask if
bandwidth problems persisted after the first connections ? Might take
quite a while if there is a lot of mail. Might be "on demand" when
entering a mail-box for the first time. If mail comes pre-sorted into
mail-boxes, watching several mail-boxes for new mail might be a
long-term "pessimization" (opposite of optimization) .

Re: Resubmission after N days

[Håkon Alstadheim]

On 04. feb. 2015 07:01, Thomas Güttler wrote:

Hi,


I would like to implement a 43Folder system[1] with dovecot and a mail user 
agent.

Use case:

- I have a new mail in my inbox. I read it and see that I can't handle it now. 
I want to handle this mail in 5 days.
- Now I want to have some sort of resubmission: the mail should be moved to a 
different location for these 5 days.
- After 5 days the mail should be moved to my inbox again.

How can this be done with Dovecot/IMAP?

I have programming experience, but I am new to Dovecot/IMAP.

I would like solve this with Dovecot only (without a database or web server).

Does the IMAP protocol over methods which could help?


[1] 43Folder https://en.wikipedia.org/wiki/Tickler_file




I would abandon this tack. Inspiration for a different angle: When I 
right-click a mail in the Icedove (Thunderbird) I am offered to convert 
the mail into an event or a task. A possible work flow would be to have 
a single postponed folder (also known as mailbox) , and move stuff 
into there while at the same time converting them to tasks/events.


This functionality has been fragile in the past, in that certain 
contents in the mail would render the event/task invalid. Combination of 
bugs in the conversion from regular email and the reading of calendar 
info in my case (davical networked calendar).


I think that any effort on your part would be better spent testing and 
polishing a solution involving some kind of calendar backend.

Re: Resubmission after N days

[Håkon Alstadheim]

On 05. feb. 2015 22:29, Thomas Güttler wrote:

Am 05.02.2015 um 12:18 schrieb Steffen Kaiser:

On Thu, 5 Feb 2015, Thomas Güttler wrote:

What specific action shall be done via IMAP? The move?
Then use any IMAP client library, use the LIST command to get the mailboxes, 
FETCH to get the messages and COPY/EXPUNGE
to move them.

You are right moving the mail should be simple.
I ask myself it is possible to store the date of the resubmission in the mail 
itself.

How shall the date be visible to the user?

Remember that altering the message may invalidate DKIM and other security / 
verification stuff.

Instead of a simple mv you can prefix the message with X-Date:  header, so 
you mark the message, but it is not visible to the user.

If your goal is to have the message up according the client side message 
sorting by date, you would need to alter the Date: header and possibly break message 
verification.

What's your goal in order to  store the date of the resubmission ?


My goal is to use IMAP only, not external database. But I guess this is not 
possible.

I could alter the subject of the mail. What kind of message verification could 
break?

Regards,
   Thomas Güttler



(code samples are for bash. untested.)

Putting data in the mail is hard without violating mail integrity.

If you really want this, an easy way would be to have doveadm create a 
code mailbox resubmit/code folder, and subfolders under there with 
names like date --iso gives you, eg with code mailbox 
resubmit.$(date -d next week +%F)/code . Note: next week , 
with the quotes, literally is a valid value for date -d .


Then have a cron job moving from codemailbox resubmit.$(date 
--iso)/code to inbox. Remember to remove the \Seen flag from the mail.


If you do not want to see the mails before next week, just do not 
subscribe to those folders (see -s flag to doveadm mailbox create )


Now, pick a scripting-language and start coding :-) .

--
Håkon Alstadheim / N-7510 Skatval / email: ha...@alstadheim.priv.no
tlf: 74 82 60 27 mob: 47 35 39 38
http://alstadheim.priv.no/hakon/

Re: Resubmission after N days

[Håkon Alstadheim]

On 05. feb. 2015 08:59, Steffen Kaiser wrote:


On Wed, 4 Feb 2015, Thomas Güttler wrote:

Am 04.02.2015 um 07:49 schrieb Steffen Kaiser:

On Wed, 4 Feb 2015, Thomas Güttler wrote:

I would like to implement a 43Folder system[1] with dovecot and a 
mail user agent.



Use case:


- I have a new mail in my inbox. I read it and see that I can't 
handle it now. I want to handle this mail in 5 days.
- Now I want to have some sort of resubmission: the mail should be 
moved to a different location for these 5 days.

- After 5 days the mail should be moved to my inbox again.


What mail storage are you using? With Maildir all messages are plain 
files, you could move those messages into a specific folder and do:


Yes, I use Maildir



find /path/to/folder -mtime +$days -print0 | \
 xargs -r0 mv -t /path/to/INBOX/new


Yes, this should work. But a solution which uses IMAP would be 
preferred.


What specific action shall be done via IMAP? The move?
Then use any IMAP client library, use the LIST command to get the 
mailboxes, FETCH to get the messages and COPY/EXPUNGE to move them. 

No need to use the IMAP server directly.

Read the manual pages for doveadm(1) and the pages it refers to like 
doveadm-move(1) and doveadm-flags(1) , check out the \Seen flag. Then 
sit down with your favourite scripting language, and make some cron-jobs.


On principle you do NOT want to manipulate the the dates of the mails, 
but the user could sort by order received, to get the recently moved 
mails displayed at the top of the mailbox listing.

Re: Best practise for deleting with Thunderbird

[Håkon Alstadheim]

On 08. nov. 2014 19:15, Reindl Harald wrote:


move in case of thunderbird is always copy+delete


that's how it works

Really? Sure? Icedove gives me THREE choices, move, mark as deleted, or 
delete immediately. I thought the codebase was the same for thunderbird 
and icedove?


With the middle choice (Just mark as deleted) I get a red cross over 
deleted messages, and they stay around until I compact the mailbox. 
The just mark as deleted might depend on dovecot announcing that 
capability, I have not checked. It IS available with dovecot using 
Maildir storage, under all icedove versions since I can remember.


Check your mail-client and see. Under account settings - server 
settings.

Re: [Dovecot] IPhone wont delete IMAP messages from dovecot mailbox

[Håkon Alstadheim]

On 13. nov. 2012 15:44, ti...@tibby.hu wrote:

Hello!
I've got dovecot running with postfix. Have virtual users under vmail 
folder.

When I connect with outlook, I can read, delete messages.
When I sign on with webmail, also can delete messages,
Mutt is working as well on the mail server, can delete messages.

Once the message is deleted it moves  to the trash. Works as it 
supposed to.


But on Iphone, when I add a mail user, got the mailbox synced with 
IMAP I can see the INBOX, but when I delete a message, get this error:
Unable to Move Message The message could not be moved to the mailbox 
Trash.


http://www.netchimp.co.uk/webdesign/wp-content/uploads/2011/12/iphone_unable_to_move_message.png 



What causing this problem?
Please Help.


I've seen this. Fixed it for my device by changing the settings on the 
Ipad, and then changing them back. Seems the local trash folder is not 
created automatically on iOS.

Re: [Dovecot] Advanced dovecot tricks - spam review/release

[Håkon Alstadheim]

On 03. okt. 2012 21:48, Marc Perkel wrote:

Hi,

I'm looking for some advice to do a really advanced trick with 
Dovecot. I'm not sure if this can be done. I need to describe first.


I have a spam filtering company that does front end spam filtering. 
(Junk Email Filter) I want to add a system where I store a copy of 
spam on a server and make it available to the customer to review and 
maybe resent on false positives.


I know I could do something simple where I deliver all spam to a 
domain account and make it available to an administrator. Then if it's 
a false positive they would drag the message to a resend folder. 
I'll have something the checks the folder one a minute to pick up and 
resend.


However

What would be very cool is delivering the spam to individual accounts. 
So a user who logs in individually can see their own spam. But the 
admin for the domain would be able to see all users. Maybe the users 
would appear as folders?


Then a master account (me) would be able to log in and see all the 
domains as folders and the users as folders inside the domains?


One thing I can do is deliver the spam to 3 different places so it's 
visible on all levels.


I'm just wondering if anyone out there has any ideas about that. And 
I'll need an authentication system.





Check out the dovecot sieve plugin. I use the following default 
pre-filter for all users:

---
require [regex, fileinto, imap4flags];

# Catch mail tagged as Spam, except Spam retrained and delivered to the 
mailbox

if allof (header :regex X-DSPAM-Result ^(Spam|Virus|Bl[ao]cklisted)$,
  not header :contains X-DSPAM-Reclassified Innocent,
  not header :contains Received-SPF pass .securityfocus.com) {

  # Mark as read
  #setflag \\Seen;
  addflag $junk;
  # Move into the Junk folder
  fileinto INBOX.Junk;

  # Stop processing here
  stop;
}
-
Together with the dovecot antispam plugin this makes the beginnings of a 
very intuitive
system. I just click to remove the junk flag on any false positive, and 
it gets re-delivered to

me.

The dovecot lda also supports a switch to deliver to a specific folder I 
believe. This would
be an alternative if you get the spam delivered through a separate 
channel anyway.

---
The other part of your requirements could be met by using dovecot public 
folders, which
I have never used myself. Maybe set up so admins can subscribe to the 
junk-folder of
any user they want ? Refiling false positives might get messy for an 
admin though.


Regards, Håkon.