RE: Which DKIM application for postfix 3.9.0
On Tue, Apr 23, 2024 at 7:33 AM wrote: > > I am upgrading to postfix 3.9.0. I have not used DKIM in previous postfix > > installs, but I >> would like to start now with the new google rules. I have done some research >> and opendkim >> is the most recommended, however, other research states the opendkim has >> been >> abandoned by it's maintainers. So I am looking for a good alternative dkim >> software >> that will work with postfix that I can compile myself. I do not run on any >> linux >> version, so therefore I can not just apt-get a new dkim application. >> I run Solaris and therefore need to compile my applications, postfix and >> dkim. >> Any good suggestions will be appreciated. I just rolled out a locally compiled opendkim on my mail server. It works, but there are a few gotchas. Although it seems like a moribund project, there is a late beta version that includes some important patches, most notably the "Header:\n LongHeaderValue" bug that needs fixing. You can look at https://sourceforge.net/p/opendkim/patches/ to find that patch, as well as others you deem important. As DKIM standards are not going to change soon, having end-of-line software is not as bad as it seems unless you need particular enhancements to make it work better in your circumstances. Once you get your setup dialed, you can probably set it and forget it. Most of the headaches have actually been internal: local mail injection via sendmail would skip miltering, From header canonicalization by the MTA would not be seen by the opendkim milter thereby creating messages with missing or invalid signatures, and mailing list/auto reply/forwarder software mangling messages. I think Postfix does a better job in this regard, so these issues may not present itself. (I did a Postfix/opendkim milter on an Ubuntu system and it was much less hassle.) You should look at *lots* of DMARC RUA reports. People are doing crazy batsh*t stuff with your mail domain. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Dovecot somehow creating new local e-mails from a compromised account
Greg Earle writes: > Obviously I've changed the account password but I would really like to > know how they were able to create e-mails on my system when ostensibly I > would have assumed they could only read the account's e-mails via IMAP. The INBOX is not read-only. There's nothing really special about this mailbox versus other mailboxes. IMAP is used to create messages all the time. For example, if you have an outgoing "Sent" folder, once your mail reader gives it to your mail server, it then puts a copy into the outgoing folder via IMAP. Once someone gain controls over your IMAP account, they can create messages with a format totally different than what your mail server can make. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Users with enough rope to hang themselves
Rupert Gallagher writes: > I keep finding myself in a corner with a user. He uses mail extensively, which > is fine, he has a huge archive of own professional correspondence, which is > fine, but he uses mail folders as if they were regular system folders, with > very long paths, and keeps renaming them and moving them around, daily, > breaking the mail index Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate indices)? > and ultimately wasting his own time looking around for > lost mail. His Inbox holds a gargantuan of subfolders, causing both the client > and the server to overwork each time he opens the mail. His Archive is a maze > of subfolders with repeating names. I advised him almost daily across 20 year > on how to stay organised, but he keeps abusing the service. Semantically, he may be inept/disorganized/unappreciative, but I wouldn't raise this to abuse. However, the damages are often the same. Maybe the fix is not technical but social by making it clear you're done trying to fix his mistakes and he's on his own. Just sayin'. > I want to help him by limiting what he can do with folders. This is the > agenda: > 1. the Archive is the only place where he can create folders; I'm guessing https://doc.dovecot.org/configuration_manual/acl/ > 2. folder names have a maximum length of 20 characters. No clue here: maybe artful remapping of namespaces? https://doc.dovecot.org/configuration_manual/mail_location/#custom-namespace-location Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Fwd: Dovecot, Load Balancing and SSL
l...@relay.gb.net writes: > I wonder if someone would provide me with some advice. I've been setting > up a couple of Postfix servers just for fun. I've got two Postfix > servers m1.domain.com and m2.domain.com. I can send and receive mail via > both of them. Ive also got Postfixadmin and RoundCube on them and I'm > replicating the database over both servers. > > I introduced a load balancer. Postfixadmin and Roundcube work perfectly. > However. When I send mail from Thunderbird. M1 reports that the > certificate does not match. It's expecting a certificate for > mail.domain.com. Complaints when you're sending mail? This is not Dovecot's problem, but Postfix's. If you're setting your SMTP outgoing to M1, then the certificate M1 uses have M1 as a subject. If you're setting the outgoing mail server as "mail.domain.com" load balanced to M1/M2, then both your Postifx servers need to use the same certificate with "mail.domain.com" as a subject. Simple as that. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: [EXT] Re: How to get a memory pointer in the core process
From: Joan Moreau > I am trying to avoid closing/ reopening a file pointer to the exact same file > between each call to the plugin Can't you do an end-run around all this by having a persistent standalone process that holds all your persistent data, and the plugin collects session data and passes it to the persistent process via sockets? You'll have to have some initial handshake protocol to establish session context, but this seems the easiest way to accomplish what you want. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Dovecot installation and ssl certificates
Jerry Stuckle writes: > I'm starting with POP3 (because it's easy to handle from the CLI). I > have it working from localhost - I can telnet to port localhost 110 and > access emails (of course I can't do this from a remote system because it > requires SSL). Sure you can: (STARTTLS style) openssl s_client -starttls pop3 -connect your.pop.server:110 (SSL style) openssl s_client -connect your.pop.server:995 You can also use ncat (exercise left to the reader). "localhost", I believe, is a specific exclusion where SSL in *not* enforced during the session. > Trying to access this from a remote system with Thunderbird is not > working. /var/log/mail.log shows the following: > > 2024-03-03T22:18:54.887061-05:00 debian-server dovecot: pop3-login: > Disconnected: Connection closed: SSL_accept() failed: error:0A000412:SSL > routines::sslv3 alert bad certificate: SSL alert number 42 (no auth > attempts in 0 secs): user=<>, rip=206.223.85.12, lip=206.223.85.137, TLS > handshaking: SSL_accept() failed: error:0A000412:SSL routines::sslv3 > alert bad certificate: SSL alert number 42, session= This looks like an error message stating STARTTLS was expected but the client used a plaintext session. Try enabling STARTTLS on your Thunderbird. If that is not the cause ... Ref: https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/ The most important dovecot configurations are ssl_cert = I need help. First of all, an explanation as to how to configure the > dovecot-openssl.cnf file (an example with actual values - real or fake - > would be a real help). > > Second, where does this go? Maybe running doveconf will tell you where your installation expects the main configuration file to be. This file may include other config files. > Note that for right now I'm trying to just get one domain working but > eventually this will serve at least 4 domains. Once I get the first > domain working, thoughts about how to get multiple domains working would > also be appreciated. This depends on how you set up your filesystem and authentication and your security constraints. You'll have to be more specific on your setup. Confining my reply to just SSL setup, you can obtain a SSL certificate with multiple domains named listed, which makes multi-domain SSL support easier. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Bug/Warning not sure which
> From: Richard Shetron > We run our own dns for sgeinc.com. > I've always used mail.sgeinc.com as my incoming and outgoing server. At > various times mail has been an alias for another machine. It's > currently on the same address as sge.sgeinc.com. On the update forced > on us on 2/22/24 or 2/23/24 it stopped working. It still works as an > outgoing server but incoming POP3 it stopped working. It started > working when I changed my incoming server to sge.sgeinc.com. Maybe your SSL cert? $ openssl s_client -connect mail.sgeinc.com:995 < /dev/null |& openssl x509 -noout -text | grep DNS: DNS:sge.sgeinc.com, DNS:sgeinc.com, DNS:www.sgeinc.com "mail.sgeinc.com" is not in your list of alternate names, hence your mail clients started rejecting the SSL certificate as invalid. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: "Connection reset by peer" errors with Outlook
From: Steve Dondley > I have no idea what is triggering it for so many different users from legit > email addresses. Still investigating. But this appears to be a fail2ban > problem, not a dovecot problem. My logs are filled with failed authentication from Outlook clients. The clients seem to be trying different usernames (with/without domains), and maybe SSL/TLS flavours. My guess is Outlook is doing some autodiscovery/autoconfiguration thing, and occasionally hits the right combo and successfully authenticates. I'm not sure I would characterise this as a fail2ban problem. Fail2ban is doing what it says on the side of the tin: looking for repeated authentication failures, and blocking those that fail too many times. The real problem is Outlook fumble around for the correct settings, and mimicking a brute force attack. I've had great difficulty getting some Outlook clients to configure exactly the settings it should have (like excluding domain names from usernames). Try running his command line using Windows-R (not from cmd.exe). outlook.exe /PIM NoEmail This will avoid the auto-setup process that railroads you into frustration. MacOSX Mail app tries the same stuff, but at least you can turn that behaviour off and stop it from second guessing your settings. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Migrate Dovecot 2.0.16 (mbox) to Dovecot 2.3.16 (MaiDir) and preserve POP UIDs
From: "Barbara M."
> I am obviouly interested to resync the INBOX with somethig that give to
> clients the old UIDs so they don't re-download all messages in the inbox
> as duplicates into their mailbox.
When I mitigated away from qpopper, I configured
protocol pop3 {
...
pop3_reuse_xuidl = yes
}
Maybe that's of use to you?
Joseph Tam
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: What is the difference between BEFORE, SENTBEFORE, and SAVEDBEFORE?
On Thu, Jan 18, 2024 at 6:42 PM Joseph Tam wrote: > If you dump the above values e.g. > > doveadm fetch -ftab -A 'mailbox date.received' mailbox Trash BEFORE 90d Correction: if what I suspect is true, this won't show you anything as all your messages will be younger than 90d. Instead, remove the "BEFORE 90d" condition and dump all values. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: What is the difference between BEFORE, SENTBEFORE, and SAVEDBEFORE?
From: Paul Pace > BEFORE date specification > Matches messages with an internal date before date > specification. > > SENTBEFORE date specification > Matches messages with a Date: header before date specification. > > SAVEDBEFORE date specification > Matches messages, which were saved before date specification. > > I am creating a cron job to purge old messages in Trash or Spam folders, > but I discovered using doveadm search queries using savedbefore that > somehow no message in any of the folders shows up with queries starting > at 5d, even though there are much older messages. I don't know exactly the answer to your query, but the semantics of each term may not be the problem. I had this exact problem a decade ago with my own trash purging script. If you dump the above values e.g. doveadm fetch -ftab -A 'mailbox date.received' mailbox Trash BEFORE 90d you may find that many of them have the same value, and what's more, the time coincides with the first time you queried for that value (i.e. when you ran the above doveadm fetch). I think these fields may not have cached values before you ask for it, then it gets instantiated the current timestamp when you do. If you do a fetch every day, you'll eventually reach 90d, and it will work forever more (+/- 1 day). Perhaps adding those fields into these settings is a more direct and better solution: https://doc.dovecot.org/configuration_manual/mail_cache_settings/ Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Anyone Watching Actvity from this network? Attempting Dovecot Buffer Overflows?
On Wed, 15 Nov 2023, 23:25 Michael Peddemors, wrote: > Not sure yet if it is Dovecot, or the SSL libraries they are > attempting > to break, but using a variety of SSL/TLS methods and connections... > > They are not interested in dovecot per se. They scan for TLS vulnerabilities, > mostly. They're running comprehensive port scans, so they're targeting more than just SSL services. > OrgName:Academy of Internet Research Limited Liability > Company > OrgId: AIRLL > Address:#A1- 5436 > Address:1110 Nuuanu Ave > City: Honolulu > StateProv: HI > PostalCode: 96817 > Country:US Out of business virtual offices, naturally. AIRLL also operating out of 195.96.137.0/24. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: How to reduce the number of UNIX sockets?
From: Steve Litt > Recently I've been getting the dreaded X error "Maximum number of > clients exceeded", so I performed an lsof -U to find who was using > UNIX sockets. This uncovered a buggy looping program I wrote using up > gobs of UNIX sockets, along with some browsers, dbus-daemon, electron > (used by vscode), smbd, and Dovecot, which uses 36 UNIX sockets. Only 36? The number of sockets seems to scale proportionally with the number of users. The login process of my service easily consumes several hundred sockets and could on occasions run into the thousands. (See also my previous posts on socket starvation.) Each client connection will consume a few sockets -- it uses maybe a few more than some applications do as they separate privileged daemons from worker daemons, and pass data around via sockets, but socket use if within reason. > I can kill dbus-daemon and the buggy program I wrote, minimize use of > browsers, disable smbd except when I (rarely) need it, but can you > think of things I can do to reduce the number of UNIX sockets used by > Dovecot on my machine? You can limit the number of clients per IP (e.g. mail_max_userip_connections) to lower the number of concurrent mailboxes that are open. Or use the idle process facility to park inactive connections (not sure if that frees sockets). Or cap the number of clients (process_limit, service_count, etc.), but if you're bumping up against those limits, you've either underprovisioned your service or you have something misconfigured. (Again, see my previous post on service_limit.) Maybe you need to *increase* those limits if you're hitting some client limit in regular use. >From what you describe, you resolved the cause of your socket starvation whose cause was not dovecot. I'm not sure what you hope to gain by saving a few sockets that dovecot uses just to make headroom for a buggy script. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: expunge & sh error
From: "Данила Колесников" > I would like to expunge Inbox mailbox every 60 days > This mailbox is not in the config any I try to use "doveadm expunge". > But my sh configs doesn't works: Fatal: Invalid search date parameter: 60d > If I enter the contents of the config directly into the console - > everything works correctly > the same code in sh - ends with an error. This sounds like a shell parsing error rather than a dovecot bug. What is the exact command you are using? Are you trying to empty your INBOX every 60d, or rather, remove messages older than 60days? If the former, you can probably just delete the entire INBOX folder or mailbox via filesystem commands as an alternative. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
On Mon, Jul 17, 2023 at 11:27 PM Aki Tuomi wrote:
Aki Tuomi wrote:
> Did you check the
> https://doc.dovecot.org/configuration_manual/service_configuration/#service-limits
> to see if it is documented? A pull request would be appreciated if it's
> still wrong.
Thanks for the updates. It does mention the problem in point 3, which
I quote here
3. Services that have no blocking operations (e.g. imap-login,
pop3-login):
For best performance (but a bit less safety), these should have
process_limit and process_min_avail set to the number of CPU cores, so
each CPU will be busy serving the process but without unnecessary
context switches. Then client_limit needs to be set high enough to be
able to serve all the needed connections (max connections=process_limit
* client_limit). service_count is commonly set to unlimited (0) for
these services. Otherwise when the service_count is beginning to be
reached, the total number of available connections will shrink. With
very bad luck that could mean that all the processes are simply waiting
for the existing connections to die away before the process can die and
a new one can be created. Although this could be made less likely by
setting process_limit higher than process_min_avail, but that's still
not a guarantee since each process could get a very long running
connection and the process_limit would be eventually reached.
It's not wrong, but I think it can be worded simpler for beginners
trying to wrap their head around how to properly size these limits.
The number of times I helped people out with this suggest it's not
well understood.
My experience would suggest it's more common than "very bad luck".
I discovered it as soon as I used service_limit, then having to double
and re-double process_limit just to keep ahead of process starvation.
For service_limit>0, process_limit values should falls between these
2 extremes
{max_connection}/{service_limit}: optimistically assumes
all clients exit expediently, but this will likely
cause lock ups in real life use; and
{max_connection}: guarantees an available process but makes
process_limit redundant.
Setting an "optimal" process_limit/service_limit combo requires
empirically monitoring the number processes running, finding peak usage,
then adding a safety factor. A beginner may be better off setting
process_limit={max_connection} and be done with it.
It would be interesting to ask a busy site admin using service_limit=1 to
offer real-life stats of how mail clients actually behave by examining
age distribution e.g. 'ps -ef | grep -F imap-login'.
The other issue is, given the behaviour of lingering clients, whether
service_limit>1 is useful at all. If a large number lingering clients
prevent imap-login from restarting, memory is being wasted here, rather
than with memory leaks. If lingering clients can be forced to exit,
or their resources transferred to another new process, this can
be avoided.
I'm not sure I can skillfully convey the above wordy explanation
without blowing out the man page, but here's an attempt
3. Services that have no blocking operations (e.g. imap-login,
pop3-login):
For maximum performance with slight loss in security, set
process_limit and process_min_avail to available CPU cores to
minimize context switching. Adjust client_limit so that
process_limit*client_limit serves your maximum expected client
connections {max connections}.
Setting service_limit=0 improves performance, allowing server
processes to live indefinitely (unlimited connections), but may
potentially suffer from memory leaks. Setting service_limit=1
offers maximum security as each process serves only one client
connection; set process_limit={max connections} if using
this value.
Larger values of service_limit will cap the client connections a
process can serve before restarting. However, long lived clients
can delay the process from exiting indefinitely; this may result
in a large number of lingering processes waiting to exit, causing
problems if process_limit is set too low preventing new processes
being spawned to serve new connections. You can conservatively
set process_limit to a large fraction of {max connections},
then adjust downwards based on observation.
...
service_count
...
See note 3. above.
--------
Better?
Joseph Tam
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Tons of imap-login processes despite client_limit very high
https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html
From: D D
> We're seeing a ton of imap-login processes running even when using high
> performance mode
> (https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode).
> According to the docs:
>
> "process_min_avail should be set to be at least the number of CPU cores in
> the system, so that all of them will be used. Otherwise new processes are
> created only once an existing one’s connection count reaches client_limit"
>
> We have process_min_avail=4, client_limit=0 and default_client_limit=20.
> So we'd expect seeing only 4 imap-login processes serving a ton of
> connections each. Yet, we see thousands of imap-login processes (more than
> half of all the imap processes):
> ...
>
> Is having so many imap-login processes normal with our config? Did we
> misunderstand the docs or is there something wrong here?
>
>
> default_client_limit = 1048576
> default_process_limit = 20
>
> service imap-login {
> # client_limit = 0 # default is 0
> # process_limit = 0 # default is 0
> service_count = 100
This service limit might be your culprit.
I wrote about the strange interaction between service_count and
process_limit here:
https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html
This gotcha should really be documented.
Joseph Tam
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Postfix: running a script on authentication failure
On Thu, 22 Jun 2023, Michael Peddemors wrote: > * Use services like RATS-AUTH to block IPs that can safely be blocked as > known hackers.. Cool. Are there other DNSRBLs (apart from bl.blocklist.de) that list BFD attack IPs? > * Use services like RATS-NULL (or SpamHaus DROP lists) right in the firewall > level. There are SOME networks > that should simply be 'unplugged' Can't find it in https://spamrats.com/. Is it an DNSRBL or downloadable file? > * Turn off port 110 (well, all plain text authentication) 90% less email > compromise reports when you do.. That will disable STARTTLS though. Even though it's not plaintext, maybe that is a good thing as it avoids MITM banner stripping attacks. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Submission behaviour
From: "André Rodier"
> chain input {
>
> # Limit new imap connections ala fail2ban
> meta nfproto ipv4 tcp dport imaps ct state new,untracked \
> limit rate over 10/minute add @banned_imap_ipv4 { ip saddr }
I'm don't know all the subttlties of this rule, but there are some
mail clients (MacOSX
Mail comes to mind) that will bombard your IMAP server with new connections when
it does a global search. It will open a new connection for each
mailbox, then do
a search. When your connection limit is reached, it will then close
all the open
connections and do another round.
This may be interpreted as a BFD attack, and you'll lock out a legitimate user.
Joseph Tam
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Dovecot reposting inactivity as auth failed
From: pe...@netsecpt.pt
> Hi , i am having an issue with dovecot , in log files of imap inactivity
> lines have the word included "auth failed" , witch is not true , what happens
> next is that fail2ban is looking for that word too in log file of dovecot
> ,and when it finds it it bans my public ip address .
> Is there any change to change this behavior in dovecot , what i mean is to
> insert "auth failed" when in fact it is an authentication failed , and not
> use it as general for every thing in log file .
Putting aside the semantics that not supplying credentials before the
timeout *is* an auth failure,
I would think the best way to handle this is to change the pattern
fail2ban triggers to be more specific about what it considers an auth
failure. If this is a typical log entry you want to avoid an
automatic ban
dovecot: imap-login: Disconnected: Inactivity (auth failed, 1
attempts in 180 secs): user= ...
I would modify /etc/fail2ban/filter.d/dovecot.conf to limit it to
0-99sec like so
failregex = ...( in \d{1,2} secs)...
Some BFD attempts will leak through but it avoids triggering on any
inactivity >99s.
Joseph Tam
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Incorrect saved dates in mailboxes
> OK, that does make sense so far. Today I have checked, and the correct > dates seem to be showing up now after the initial date (ever since that > initial run of the "dovecot expunge" command was run on all mailboxes). > So in our case I have the expunge set to delete older than 30 days. I > assume if I wait 30 days from now, it will start working? Yup. If you run your script every day (and thus, run "doveadm fetch ... date.saved" as well), that will make sure any new mail put into your Trash folder will have date.saved within 24h of the "real" value. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Incorrect saved dates in mailboxes
Chris Szilagyi writes: > Recently, I noticed that our expunge script is not working, and I don't > think it ever has on this server. On further inspection, it looks like > the saved date for emails in the folders we want to expunge is set to a > recent date (yesterday) for almost all messages. For example, when I > run the command "doveadm -f tab fetch -u username date.saved mailbox > Trash", the date for almost all of the messages is yesterday at the > exact same time. I tried this on other users and they have this exact > same timestamp shown for most of their messages, too. > > I tried checking for "date.received" and that shows correct, it is > "date.saved" that is not working. > > Any idea of how or why this would not be showing the correct date? I've > used this setup in the past on other servers with dovecot and it has > worked great, no idea why we would be seeing this issue now. I recall having the same problem. I think the "date.saved" is not instantiated in the cache until you query for it. So the value you were shown is when you last dumped its value if it wasn't previously set -- your run of same values coincided when you ran "doveadm fetch". My expunge script just uses date.received instead -- it seems to work. Joseph Tam ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: SSL error
Ruben Safir wrote:
> > This got nothing to with LE or own CA. Bottom line is, you need to
> > add your own CA to the cert tore (ideally)
>
> what is a cert tore?
Someone has probably already replied to this, but it's a typo:
the OP wanted to say "store".
The certificate you created was used to sign itself ("self signed")
and thus, asserts its own validity. If you need *other* people to
trust your SSL service, you should sign your certificate using a third
party authority (e.g. LetsEncrypt) to sign it. Most internet users
will have these third party signing authority's certificates in their
certificate store to validate your service certificate. If this is
for your own personal use (i.e. you don't care about trust since you
know it's your own certificate), you have to add your self-signed
certificate into *your* system's certificate authority store so that
your mail reader does not complain about an untrusted certificate.
Clear?
Joseph Tam
Re: ot: how to t/s TBird problems ?
Voytek Eymont) wrote: > yesterday it was > --- > I'm still experiencing a 40 second delay to retrieve emails for > xxx If *this* is the problem you saw (and not the 2 hour delay mentioned further in the thread), you can get a hint where the problem lies if you see a 40s gap in the session logs: it will tell you who was doing what when the pause happened (e.g. during authentication? During LIST fetch? During message fetch?) For example, if dovecot was busy mulching through a large INBOX rebuilding indices, I can see how it can chew up 40s under some circumstances. Joseph Tam
Re: ot: how to t/s TBird problems ?
Voytek Eymont writes: > I've enabled logging as per your suggestion: > > -rw--- 1 vmail vmail 127 Oct 16 21:38 20221016-213738.25640.1.in > -rw--- 1 vmail vmail 8546603 Oct 16 21:38 20221016-213738.25640.1.out > -rw--- 1 vmail vmail 96 Oct 16 21:58 20221016-215757.26075.1.in > -rw--- 1 vmail vmail 8343463 Oct 16 21:58 20221016-215757.26075.1.out > > # cat 20221016-213738.25640.1.in > 1665916659.491025 STAT > 1665916659.550829 LIST > 1665916676.430794 UIDL > 1665916693.761281 RETR 114437 > 1665916694.440965 QUIT > # cat 20221016-215757.26075.1.in > 1665917878.786953 STAT > 1665917878.863136 LIST > 1665917905.610805 UIDL > 1665917924.491198 QUIT > # > > what should I look in the .out file ? > > some of the file is like: > > 1665916661.234807 114436 70097 > 1665916661.234814 114437 154498 > 1665916661.234821 . > 1665916676.430870 +OK > 1665916676.981415 1 24b95283283a > 1665916676.981459 2 24ba5283283a > > > 1665916679.434297 114436 00033fcf5283283a > 1665916679.434327 114437 00033fd05283283a > 1665916679.434349 . > 1665916694.048139 +OK 154498 octets > 1665916694.048199 Return-Path: > I haven't seen anyone else replying, but there doesn't seem anything anomalous with the output. The session commands-repliesd is is more or less what I expect, although to make sense of this, you'll have to splice the input and output files together using timestamps to see the sequential flow of data. I forget what the symptoms you originally reported, but theoretically, you could simulate either client or server by feeding in the above data and see how the other end behaves. If dovecot is serving out the correct data, then TB is somehow misinterpreting it. > on an uneducated guess, the mailbox is just 'too large' ? > POP has difficulty handling so many files ? Typically, if some resource limit is hit, one side or the other will create a log or notification. Your INBOX is large, but not outrageous. You can test it directly by creating smaller subsets of the INBOX messages and see if the problem goes away. Joseph Tam
Re: ot: how to t/s TBird problems ?
> I recently upgraded my Thunderbird email client and have experienced
> problems since.
> It appears that when Tbird polls for new messages it gets held up
> waiting for a response from the server
> I'm using POP port 995.
> Any ideas as to why I'm having a problem ?
> ---
>
> how to investigate such issue ?
I suspect you'll need to do session logging e.g.
protocol pop3 {
...
rawlog_dir = /writable/logdir/%u
}
then
mkdir /writable/logdir/user
chmod 0777 /writable/logdir/user
to obtain session transcripts of what server/client are doing.
I don't see any obvious errors from the logs that indicate any failure.
I do see the INBOX is rather large so maybe a timeout is involved.
Joseph Tam
Re: Dead links at https://wiki.dovecot.org/Migration/MailFormat
> These links: > > http://dag.wieers.com/rpm/packages/pine/pine-4.64-3.el4.rf.i386.rpm > http://staff.washington.edu/chappa/pine/info/maildir.html > > on this page > > https://wiki.dovecot.org/Migration/MailFormat > > at the Dovecot wiki do not lead to useful results. I lost the context of this thread, but if you're looking for mailutil or the older pine stuff, the project has forked inti alpine and you can find the source tarball at https://alpineapp.email/ Joseph Tam
Re: dovecot/config processes open, and consuming all memory
From: Aki Tuomi
> The *default* configuration for service config is usually just fine.
> Is there some reason you decided to modify it in first place?
Are you asking me, or the OP?
I guess the blunt answer is ignorance on my part. However, I pointed
out that the docs about this setting is somewhat misleading -- it's used
to limit any potential memory leaks for long-lived processes by ensuring
it terminates periodically.
I read this and thought to myself "this is a good thing to do", without
realizing that it would have the opposite effect as lingering clients
could delay service termination indefinitely. The side effect is many
new processes are spawned to handle new clients, and eventually
the maximum process limit is reached, and chaos ensues.
So for services like imap-login and others that can have lingering
clients, the only sensible values for service_limit is {0,1}. If you
set service_limit>1, the asymptotic behaviour is like service_limit=1,
and process_limit would have to be adjusted accordingly.
The docs can explain this rather wordy and subtle explanation of
service_limit, or service_limit can be constrained to values {0,1} so that
others don't blunder along the same path I did.
Joseph Tam
Re: dovecot/config processes open, and consuming all memory
> I'm having strange behavior in dovecot 2.3.16.
> It's opening dozens of dovecot/config process and consuming all server
> memory. Normally each process consumes between 700Mb and 1Gb of ram.
>
> Would anyone have an idea about this?
>
> service config {
>vsz_limit = 2048M
>idle_kill = 60s
>service_count = 1024
> }
Not sure it's related, but if you have service_count not 0 or 1, there
is a strange interaction
with other limits that could cause processes to hang around. My
description of problem
https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html
Your situation is slgihtly different (service not imap_login, and
idle_kill timeout should reduce
lingering processes that caused my problem), but try setting
service_limit to either 0 or
1 and see if your problem goes away, or gets worse.
You can also see how many file descriptors are being held by the
config process, and
see the behaviour over time (e.g. monitor /proc/{pid}/fd/*); maybe
that will give you a clue
as to what the config process is doing.
Joseph Tam
Re: dovecot Digest, Vol 231, Issue 31
> > doveadm -fjson mailbox status -u user unseen "*" > > Very nice Aki! I can pass that JSON to a Python program I make to parse > JSON, and then just report the ones not having "unseen":"0" . Thank Or use format "-ftab" and grep non-zero entries. Simpler than parsing JSON. Joseph Tam
Re: Deleting "folders only" folder doesn't actually delete
> I'm using dovecot-2.3.17.1-1.fc34.x86_64 on fedora34 and Thunderbird 91 on > fedora35. When creating a folder of type "folders only" in Thunderbird, then > trying to delete it, it doesn't actually delete it from the filesystem. Is > this a permissions problem? Or perhaps a Thunderbird problem? Are you using MBOX backend? I think I ran across this problem and it's some confusion as to whether the target is a file or directory. I was finally able to delete it by appending '/' to the name of the folder in my mail reader, but some mail readers do not allow you to type the mailbox name to delete. I believe Tbunderbird has some IMAP server setting that will give it a hint. Joseph Tam
Re: dovecot Digest, Vol 230, Issue 21
On Thu, 9 Jun 2022, Richard wrote:
> Rather than simply upping the limit I think a reasonable question to
> ask is why/how they are managing to do that. That's a lot of open
> folders.
If this mail client behaves anything like Apple mail clients, these
connection storms can come about when doing global searches. The mail
clients will march through each mailbox (opening a connection for each
mailbox) looking for a pattern.
If you can narrow this scenario for one specific user (e.g. $user), you
can deep dive what's going on by enabling IMAP session logs for this user
protocol imap {
...
rawlog_dir = /log/dir/%u
}
then
(Make sure this user has write permissions into this directory)
mkdir /log/dir/$user
After you're done, you can disable logging,
rm -rf /log/dir/$user
Joseph Tam
Re: Force TCP socket disconnect on imap login failure?
On Wed, 25 May 2022, Hippo Man wrote: iptables (linux) & pf firewall (freebsd) do drop the packets immediately as the tables are updated. In my case, that is not occurring. After issuing the iptables DROP command, the client can continue to send more and more login attempts. Only when the client disconnects does the block of the socket seem to work for that IP address. I continue to see numerous instances of this behavior. I'm running debian 8. Perhaps the iptables on this nearly obsolete version of linux do not behave in the way that you have experienced. Many firewall keep a side cache of estalished connection. Either implicitly or explicitly, an established TCP session will do an end-run around your rules. conntrack seems to be the iptables utility you need to flush a connection cache: https://www.systutorials.com/docs/linux/man/8-conntrack/ e.g. conntrack -D -s x.x.x.x However, even this may not be enough as dovecot may send an outgoing packet (being oblivious to firewall rules), which could re-establish a connection as firewall rules typically allow free egress, and can automatically create missing state entries. I'm not sure how this is typically handled -- maybe an outbound block rule is required to handle this niche case to finally drive a stake through a BFD connection's heart. (more stuff: https://unix.stackexchange.com/questions/646663/iptables-how-kill-established-connection-except-for-an-ip). Joseph Tam
Re: Force TCP socket disconnect on imap login failure?
On Tue, 24 May 2022, Hippo Man wrote: Late to this party. * Hacker makes numerous login attempts one after the other with various passwords, and without disconnecting in between attempts. I've seen 10 and more of these repeated attempts rapidly during a single imap or pop3 connection. Maybe this settings helps? auth_failure_delay = 5 secs I get lots of BFD, and although they have no chance of guessing a password this way, it produces an annoying amount of rubbish in my logs. This slow them down to either reduce the volume of attempts (and logs), but also gives you ample time to enact a countermeasure. I will get the latest dovecot source code and modify it so that dovecot will disconnect after "N" failed imap or pop3 login attacks, where "N" is some sort of configuration variable (with a default of zero, meaning do not disconnect). I will then use this personal version of dovecot with "N" set to a fairly low value (probably 1!). 1, in my opinion, is really too low. This can lockout a legitimate user with a simple typo, or network hiccough. It would be better to externalize this, rather than bake it into dovecot. Have you considered https://doc.dovecot.org/configuration_manual/authentication/auth_policy/ Furthermore, I will continue to automatically monitor the logs and perform the same iptables DROP actions for the failed login attempts. The combination of these two actions will give me the behavior that I desire. You can also preempt many BFD runs without resorting to one-strike-you're-out policy 1) Look up connecting host in RBL and do a prememptive block e.g. bl.websitewelcome.com, bl.blocklist.de, dnsbl.darklist.de are some examples of brute force DNSRBLs. You'll find many of attacking IPs are represented on one of these lists. 2) Triggerimmediate block against authentication attempts that can not possibly be real (e.g. "mysql", "testuser", "nagios", etc.) Joseph Tam
Re: Message attachments, relocated with Tbird in Dovecot maildir store, not openable; reversible by moving BACK to inbox?
On Wed, 18 May 2022, PGNet Dev wrote: checking ls -altr /tmp/pid-59993/SomeFile.pdf -r+ 1 pgnd pgnd 27 May 18 07:46 'SomeFile.pdf' This may or may not get you closer to the solution, but out of curiosity, what's in the 27 bytes worth of data? And are those quotes really there? Joseph Tam
Re: how to setup IMAPs with letsencrypt
On Sun, 24 Apr 2022, ??? (alice) wrote: [Actually, I wrote] otherwise you'll have to use DNS challenge method to support multiple hostnames on the same certificate. do you know how to implement this? Others have pointed out resources, but at a very basic level, you'll need a scriptable way to add TXT records for your domain. Plenty of ACMEbots supply plugins for various cloud provider APIs, but if you're running your own DNS server like I am, you may have roll your own plugin. If you don't have this level of control over your DNS zone, you'll have to bodge it with HTTP challenge and a stub web servers. the original certificates were issued for domain: sample.com. But this certs can be used for any.sample.com too? For wildcarded certs (valid for *.sample.com), your only recourse is use DNS challenges. Joseph Tam
Re: how to setup IMAPs with letsencrypt
I have setup website using letsencrypt for certification. how can I setup IMAP to use this certs as well? Make entries in /etc/dovecot/conf.d/10-ssl.conf ssl = required ssl_cert = Keep in mind the subject name (CN or SAN AltNames) of your certificate must match your IMAP server name e.g. if your certificate is made for "www.mydomain.com", you'll have to configure your IMAP clients to also use "www.mydomain.com" as the IMAP server name. This typically means the web and IMAP server must reside on the same server, otherwise you'll have to use DNS challenge method to support multiple hostnames on the same certificate. Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Tue, 19 Apr 2022, Sami Ketola wrote: In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms. They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared. This is more about gmail ?spam protection?. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted. While Gmail's spam classifcation is frustratingly opaque (as with many other providers), there's one other weirdness that can make messages dissappear: when you send from Gmail through a forwarder back to the same Gmail account. The mail will be accepted, but not appear in your INBOX. This is either some labelling weirdness (cannot be labelled as both INBOX and Sent) or maybe an anti-loop protection. This causes confusion when Gmail users test their mail forwarding I set up for them. However, we're truly off-topic: my point was that Gmail users get a distorted sense of how most mail systems work (mailbox operations scale with message count), as they get brainwashed into the "Gmail" way of doing things. A relevant question is whether you can use Dovecot's virtual mailbox feature to define a catch-all virtual mailbox to placate these users which won't bring an imap process to its knees. Joseph Tam
Re: Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote: As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly. I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for. Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX. Joseph Tam
Setting imap-login process_limit when service_count>1
More of my users are accessing mail remotely, which probably explained
why I started seeing
Mar 18 05:50:34 dovecot: master: Warning: service(imap-login):
process_limit (2) reached, client connections are being dropped
I played around with increasing limits, but made the mistake of setting
service_count to a value other than 0 and 1.
service imap-login {
client_limit = 1024
service_count = 10240
process_min_avail = 2
process_limit = 8
}
Doubling and redoubling process_limit and increasing service_count didn't
prevent IMAP from eventually grinding to a halt because process_limit
was reached.
Mar 18 12:36:12 dovecot: master: Warning: service(imap-login):
process_limit (4) reached, client connections are being dropped
Mar 18 20:39:48 viol dovecot: master: Warning: service(imap-login):
process_limit (8) reached, client connections are being dropped
Observing the way imap-login processes spawn and retain file descriptors,
I finally understood the subtlety of [1], which explains that imap-login
will not exit despite reaching service_count if one SSL connection is
still open.
With many long-lived client connections, the asymptotic behaviour is that
the total #clients plateaus, but get spread out over many imap-login
processes, with many lingering on to hold a few SSL connections.
For service_count>1, process_limit should be set to a large fraction of
peak simultaneous clients (i.e. the same value used when service_count=1),
otherwise there is a high likelihood of running into process_limit and
game over.
Given this behaviour, there doesn't seem much sense in setting
service_count to anything but 0 (unlimited => performance mode) or 1
(security mode). Setting to other values supposedly limit memory leaks,
but if a single persistent SSL client can hold up an imap-login process
from exiting and releasing memory, it seems to negate this purpose.
Anyways, maybe [1] can mention this so others don't fall into the same
pit I did.
References
[1] https://doc.dovecot.org/admin_manual/login_processes/
Joseph Tam
Re: log failed plaintext password for specific user only
On Wed, 23 Mar 2022, mj wrote: We are currently observing a high number of failed authentications for a specific user, coming from *many* diffirent IPs across the globe, with most IPs only trying once or twice, making this difficult to block. The number of failed authentications cause this account to regularly become blocked in AD. We would like to know if they are trying older actual passwords from the user, or if it's just dictionary attack. Rather than messing around with dovecot configuration, I think you can process trace (strace?) the auth process and intercept read/write buffers to a few key low numbered sockets and extract username/plaintext passwords from them, filtering out those you don't need. Sort of hacky, buy avoid messing about with dovecot, or even restarting it. You can possibly extend this by taking the auth information, and triggering a block if you recongize it as a dictionary attack, but it may be too late as your AD will see it by that point. Joseph Tam
Re: Dupliate-ish email search?
On Wed, 2 Mar 2022, @lbutlr wrote: I'm mulling over writing some code to find emails in a maildir that are duplicates, ish. That is to say that sometimes the same message doesn't quite show up as an exact match. Like some ad company may send you three identical messages, except they aren't actually EXACTLY identical, the message-IDs are different, and may the to address quoted part is different, so normal duplicate finders fail to find them. Before I start, is this a solved problem? Not perfectly, and maybe impossible in the general sense. If you've ever had to anonymize mail by comparing samples sent by a mailing list provider to 2 different recipients, you can see various hashes and identifiers that show up in tracking headers and URLs. Adding customized name labels e.g. "Dear Alfred P. Sloan" or individual specific information, and this becomes a complex question how different is different. If you make some simplifying assumptions (e.g. exact same message body, same header for From/Sending network or IP/time-range/Subject, you can do a fairly good job. Joseph Tam
Re: Apple Mail behaviour: can not create sub-folders
On Tue, 1 Mar 2022, Jan Bramkamp wrote: One of my Apple Mail users recently complained his mail reader couldn't create sub-folders -- he could only create top-level folders. Playing around with this, I discovered that I could create folders ( as opposed to mialboxes) *if* I specified mailbox name with a trailing slash. Has anyone come across this? Is this related to https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfolders/ Which path separators did were used? '/' (maps directly to filesystem pathname). The user stated that it behaved "normally" before where folders and mailboxes were handled without fuss. Thunderbird also has a setting which hints the client as to whether whether Maildir or MBOX was being used by forbidding creating mailboxes that also contained other mailboxes. I guess I'm asking whether the Apple mail client now has a similar control or it's up to the user to figure it all out. Joseph Tam
Apple Mail behaviour: can not create sub-folders
One of my Apple Mail users recently complained his mail reader couldn't create sub-folders -- he could only create top-level folders. Playing around with this, I discovered that I could create folders ( as opposed to mialboxes) *if* I specified mailbox name with a trailing slash. Has anyone come across this? Is this related to https://doc.dovecot.org/configuration_manual/mail_location/mbox/mboxchildfolders/ ? Joseph Tam
Re: Is Diffie-Hellman needed?
dove...@ptld.com writes: Is Diffie-Hellman needed on a modern new dovecot setup? Needed? Strictly speaking, probably not. Most clients will fall back to RSA, although some security hardened clients might refuse if you don't have non-RSA ciphers. However i see Diffie-Hellman related warnings in logs: dovecot[1073]: imap-login: Error: Diffie-Hellman key exchange requested, but no DH parameters provided. Set ssl_dh= It just something you have to do to start using DH, analogous to having to generate 2 primes when using RSA. If you don't set it up, DH can't be used. And follow up question; The docs say you are encouraged to disable non-ECC DH algorithms completely. However i didn't see anything on that same page explaining how to go about doing that. Can someone point me to something explaining what that means and how to go about doing it? You have to specify your own ciphersuite via the ssl_cipher_list configuration like this (Remove all the ciphers with "RSA" although I can't say whether this is a good idea, especially if you need compatibiluty with older clients.) https://gist.github.com/keithws/d073c6f825e02fc823a7c32d406acada justina colmena follows up with I want better explanations of the maths. If RSA and DSA algorithms based on standard arithmetic exponentiation modulo the product of two large primes are "deprecated" -- that means that there have been or are expected to be major mathematical and algorithmic advances in factoring large integers. Disclaimer: I'm not an expert on this. I don't think RSA is in any immenent danger of being broken, although there are some theoretetical factoring methods if quantum comoputing becomes a thing. There are other reasons to avoid RSA key echanges. - EC keys are smaller and stronger for the same key size. Not a hugely compelling reason as key exchanges are only a miniscule part of overall processing. - perfect forward secrecy: the disclosure of a private key will not compromise past traffic. This is probably the more compelling reason. The maths are easy for those algorithms, whereas the ECC algorithms are based on very advanced maths which aren't being explained satisfactorily to the general public, To be fair, just because people understand what primes are and will just take a mathematician's word for it that factoring is hard, neither topics can be satisfactorily explained to a layperson so that their strength is apparent. There is an argument to be made (and people have made them) that EC is hard to do right, so curve parameters should be chosen to minimize the chance of implemenatation mistakes and side-channel leaks. Also, some NIST curves parameters were chosen in less than fully transparent way -- that's not to say they're broken, but it open them up for suspicion. You can YouTube various terms and get tutorials at various levels of technical complexity, but the mathematics can get pretty hairy for both key exchange methods. Joseph Tam
Re: Non-user logins?
On Fri, 7 Jan 2022, Ken Wright wrote: On Fri, 2022-01-07 at 23:27 -0500, Dave McGuire wrote: On 1/7/22 11:24 PM, Ken Wright wrote: So, if anyone can tell me what's going on with all these logins, I'd be much obliged! I see them all the time on the mail servers I run. Typical kids trying to mess with other peoples' stuff. I run fail2ban to catch those log entries and block the source IP address for a month on the first failed login. At any one time I have between 12,000 and 15,000 addresses in my blocked list for IMAP. Dave, that's exactly the kind of answer I was looking for. Fail2ban, huh? I'll have to check that out. Thanks again! Yup, these SMTP AUTH BFD attempts come in thick and heavy. Another resource to preempt these attacks is Spamhaus's XBL blacklist. At my site, there was a 99.2% hit rate and very low false positives. Even those FPs led to some useful discoveries that the client had a malware they didn't know about. http://www.blocklist.de/en/index.html also run a DBS RBL list and I've had zero FPs after years of use. I think you can even get Fail2ban report to your attackers to this site to add to the crowdsourcing. Joseph Tam
Re: Doveadm auth test fails
On Wed, 5 Jan 2022, Ken Wright wrote: Jan 5 22:09:30 grace dovecot: auth: Debug: client passdb out: FAIL#0111#011user=m...@mydomain.com Just a wild ass guess, but does your password backend expect "me", or "m...@mydomain.com" (which is what it was given). Joseph Tam
Can dovecot be leveraged to exploit Solr/Log4shell?
I'm surprised I haven't seen this mentioned yet. An internet red alert went out Friday on a new zero-day exploit. It is an input validation problem where Java's Log4j module can be instructed via a specially crafted string to fetch and execute code from a remote LDAP server. It has been designated the Log4shell exploit (CVE-2021-44228). Although I don't use it, I immediately thought of Solr, which provides some dovecot installations with search indexing. Can dovecot be made to pass on arbitrary loggable strings to affected versions of Solr (7.4.0-7.7.3, 8.0.0-8.11.0)? Those running Solr to implement Dovecot FTS should look at https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228 Joseph Tam
RE: Mailbox connection fails: Connection closed (No commands sent) Help please
On Wed, 8 Dec 2021, post...@aecperformance.com wrote: Thunderbird says: Wrong Site The certificate belongs to a different site, which could mean that someone is trying to impersonate this site. $ openssl s_client -connect aecperformance.com:993 < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -F -A1 'X509v3 Subject Alternative Name:' X509v3 Subject Alternative Name: DNS:aecperformance.com, DNS:deanhh.com, DNS:dev.aecperformance.com, DNS:sizzelicks.com, DNS:softlinksys.com, DNS:www.aecperformance.com, DNS:www.deanhh.com, DNS:www.sizzelicks.com, DNS:www.softlinksys.com Is your Thunderbird set up to use one of the above server names, and not, for example, imap.aecperformance.com. The server name has to match one of the above. Joseph Tam
Re: ZFS storage and backup
On Fri, 19 Nov 2021, James wrote: On 15/11/2021 16:18, infoomatic wrote: Regarding storage I tend to use sdbox, from what I have read it seems to be the better option when using a COW filesystem compared to mdbox. One more https://doc.dovecot.org/admin_manual/mailbox_formats/ sdbox single-dbox, one message per file. mdbox multi-dbox, multiple messages per file. so I guess sdbox is better with ZFS. I could test each but I think I will find the IO used by dovecot is low for each. I have one user with 32,164 emails in INBOX and IO is not a problem. It depends on what aspect of performance you're talking about and how it is implemented, but as I understand it, ZFS snapshots are done at the block level, and just as long as mdbox leaves message blocks in situ (by manipulating indices instead?) and doesn't shuffle them around, unchanged messages won't bloat snapshot storage, unlike MBOx where a one message insertion/deletion at the beginning will cause the entire mailbox to end up in snapshot storage. question is: compression at file system level or in dovecot storage? This relates to my comment -- if the compression is done at the message level rather than the whole MDBOX, the above is not applicable as any change to a byte will affect all subsequent bytes. I think MDBOX is a compromise in data granularity that tries to strike a balance between various aspects of I/O performance. Joseph Tam
Re: Strategies for protecting IMAP (e.g. MFA)
On Sun, 14 Nov 2021, Michael Peddemors wrote: And there are RBL's now for know IP(s) used by IMAP hackers, including SpamRats RATS-AUTH that can assist in reducing those attacks. These guys also lists brute forcers: http://www.blocklist.de/en/rbldns.html I don't know how well they catch IMAP hackers, but they list 95%+ of our ssh brute forcing attacks. Joseph Tam
Re: Doveadm fetch slow and 100%CPU with a specific message-id
On Mon, 25 Oct 2021, Ron Garret wrote: Note that message-ids are not guaranteed to be unique. During my test I found groups of as many as 20 different messages with the same message ID. (Turns out this makes quite a reliable spam signal!) It's by far not a rare situation: duplicate message-ids happen whenever the sender names more than one local recipient during SMTP. It's a wholly unreliable way to indicates spaminess. However, if a high proportion of those recipients do not exist, ... I think you may have misunderstood. What you say isn?t wrong, but in the case of multiple local SMTP recipients, all of the duplicate messages will have the same content. What I have found is the same message ID in messages with (very) *different* content (and often sent to the same user). All of that has been spam (and it is hard to imagine any situation in which it would not be). Ah, that is a different situation. It could happen if the same message tooks different paths to your user e.g. via mailing list processor, but that is less common and would probably break DKIM. Joseph Tam
Re: Doveadm fetch slow and 100%CPU with a specific message-id
On Mon, 25 Oct 2021, Ron Garret wrote: Note that message-ids are not guaranteed to be unique. During my test I found groups of as many as 20 different messages with the same message ID. (Turns out this makes quite a reliable spam signal!) It's by far not a rare situation: duplicate message-ids happen whenever the sender names more than one local recipient during SMTP. It's a wholly unreliable way to indicates spaminess. However, if a high proportion of those recipients do not exist, ... Joseph Tam
Re: Fwd: Fwd: folders and subfolders
On Tue, 20 Jul 2021, Stephane Magnier wrote:
I found this page
http://etutorials.org/server+administration/sendmail/part+i+build+and+install/chapter+4.+configure+sendmail.cf+with+m4/featurelocal_lmtp/
Where they explained just to add this
FEATURE(`local_lmtp')
MAILER(`local')
I can also see :
FEATURE(`local_lmtp', `/usr/sbin/mail.local')
MAILER(`local')
So, having Dovecot : 10-master.conf
service lmtp {
unix_listener /var/run/lmtp {
mode = 0660
user = mail
group = mail
}
}
I wrote FEATURE(`localhost_lmtp',`/var/run/lmtp')
No, no. What you're stipulating here is that /var/run/lmtp is an
executable that communicates LMTP via stdin, whereas dovecot is configured
to communicates LMTP via a socket connection to /var/run/lmtp. You need
to configure sendmail
FEATURE(`local_lmtp',`[IPC]',`FILE /var/run/lmtp')
If you don't need LMTP exposed to the internet (i.e. your front-end MTA
is on the same host as your LMTP), socket connection is probably simpler
and safer than TCP connections.
Joseph Tam
Re: folders management
On Tue, 13 Jul 2021, Stephane Magnier wrote: On my folders' architecture, I can see all the folders underneeth each other fodler1 folder2 etc I would like to have the possibility to have _folders INTO folders_, like Folder1 Folder2 ...folder 21 ..folder 22 Folders 3 Apparently, this is link to this d?claration :__*_mbox_:/var/spool2/mail/%u. *Is that correct ? is that the "mbox" declaration? or Am I complety wrong ? Not directly. The LAYOUT=fs parameter is probably what you need to change. https://doc.dovecot.org/configuration_manual/namespace/ Section: Hierarchy separators If I have need to change...I have to change it for what ? What are your current settings? * Can I use this new structure with NFS shares ? Yes. * What is the backside of it ? mbox format has lots of performance drawbacks. You can also have nested folders in Maildir, so if you have a choice, use that format. Now, if I make this modification.. on an existing dovecot system... , Will it make a true disaster ? or I can change the declaration easily? If you currently use maildir and change to mbox, that will go badly. Aagin, there's not much that can be without knowing your current settings. Have you tried creating nested folder structure with your mail clients? Joseph Tam
Re: good options for Multiple users on a common email account
On Wed, 23 Jun 2021, Pat G wrote: i ve a mail server to manage with some email accounts but with multiples users (+50) using a common email. it indicates sometimes that it can't connect cause too many connections. what are the good options to allow a lot of users for a specific account ? i modifyied these options : auth_worker_max_count = 60 mail_max_userip_connections = 60 is it sufficient ? Probably not, but it depends on the mail client they are using. Some mail clients chew up 3 or 5 concurrent connections per session. MacOSX Mail.app will consume all the connections while doing global pattern searching (i.e. if you allow 200, it will open up 200 connections at a time before closing them). If your users use POP3, then the limits have a good chance of working. If too many concurrent connections becomes a problem, maybe you can use mailbox sharing rather than single account access. Joseph Tam
Re: Dovecot v2.3.13 reporting (very) incorrect vsize for some maildir folders
On Thu, 20 May 2021, Eirik Rye wrote: I noticed that `ls -s` reported a completely different size to `du`, but similar to what dovecot reports: # ls -s | head -1 total 14099016 # du 7050436 . I assume there are some sparseness or block size related shenanigans going on here instead, causing differences in reported physical usage by `du` (syscall `newfstatat()`) compared to `ls` (syscall `lstat()`) and dovecot. You'll note the ratio between then is almost exactly 2. Some utilities report space usage in 512-byte block, some in K. I would hazard a guess that 'ls -s' is reporting in blocks, not K. The man page for my OS 's'ls' states exactly that -- counts are in blocks. Joseph Tam
Re: connection closes every 10 minutes
On Mon, 26 Apr 2021, Marco Fioretti wrote: 3) a few days ago I received a new modem from my ISP, as part of their network upgrade operations 4) more or less in the same moment the problem I reported here disappeared. Now mutt stays connected even 24 hours without losing connection. I am NOT 100% sure that the problem disappeared AFTER the change of modem. That happened during a few chaotic days, both work- and family-wise, so I did not take notes. And modems may have nothing to do at all with the disconnections. But now the problem is not there anymore, I have no clue what may have happened, and if anybody can guess... thanks in advance. Does this modem also have an integrated router? These units tend to act as NAT gateways/firewalls that keep track of "active" sessions by tracking external/interface NAT address mappings. Cheap or older one could have TTL on these entries i.e. if no traffic is detected within a time window, it is discarded, and appearing as if the endpoints had disconnected. I guess it could also happen if the state tracking tables has limited memory and your internal network is busy, like a family member opening up a P2P application. Just a hypothesis. Apr 12 16:12:49 SERVERNAME dovecot: imap(ACCOUNTNAME): Logged out in=164 out=757 However, my hypothesis wouldn't produce this. This is a active logout. Joseph Tam
Re: Mass Stripping Attachments by Directory, Age, Size
On Thu, 18 Mar 2021, Plutocrat wrote:
I've been looking around for a solution to this problem. I want to prune down
the attachments on a server before a migration. Some of the emails are 7
years old and have 40Mb attachments, so this seems like a good opportunity to
rationalize things. So perhaps I'd like to "Remove all attachments from
emails older than 2 years, in the .Sent directory", or "Attachments over 10Mb
anywhere in the mail tree"
I've found the strip_attachments.pl script here
<https://fossies.org/linux/Mail-Box/examples/strip-attachments.pl> which
works fine on mbox (as tested on my local Thunderbird mboxes), but not on
maildir which is on the dovecot server. My Perl isn't strong enough to
re-purpose it.
It you have anything that works on mbox, it will probably work on Maildir
as each file can be considered a single message mbox. You can combine
the script with
find ~user/MailDir -type f ... -exec /path/to/mbox-strip {} \;
The ... can be replaced with more file tests (like minimum size or age
or only within */cur/) to cut down on processing.
I wrote a gawk script to slim down a multi-Gb Outlook mbox
for a user, but it wasn't really complicated, just matching for
/^Content-Transfer-Encoding:.*base64/i header (virtually all bulky data
will be encoded this way), buffering the base64 data part, then outputting
it if it was small, or deleting/replacing/extracting it otherwise.
It was a one-off discarded tool but I can hunt for it if you're hard up.
I've looked at ripmime and mpack/munpack, and although they seem like useful
tools to do the job of deconstructing the mail into its constituent parts, it
doesn't seem to help in re-building the email. I think they could be used
with a bit of study into mail MIME structure, and used with a helper script.
So before I take a deep dive into scripting my own solution, I just wanted to
check if anyone else on the list has been through this and has some resources
or pointers they can share, or maybe even someone to tell me "Duh, you can do
it with doveadm of course".
MIMEDefang may help.
Joseph Tam
Re: bug: some table header(?) output goes to stderr instead of stdout
On 2021-03-18, Marc wrote: [@ sbin]# doveadm -f table -o mail_location=mdbox_deleted:/home/popusers/testtest/mdbox:INDEX=/home/popindex/testtest/index fetch -u testtest 'guid' mailbox INBOX 2> /dev/null 3c967f33b8aea671f3551db1ea8e33e9 6fa01ccc103a7009c7b940657dbcd72c ba955a6d6218950f42e5b0ee0a33a916 Strange -- my version (2.3.10) dumps headers to stdout, not stderr # doveadm -f table fetch -u $user guid mailbox INBOX 2>/dev/null | cat -n | head -3 1 guid 2 8104226179c70d7cc248c9924cabdb8c 3 0813554a7ed4cf1e113f42a4cc8bc477 From a strictly design point of view, this seems more correct as the header is part of the data, not "out of band" output like errors. If headers are pumped to stderr, you can always employ shell hacks (as a followup poster did) to unify them to stdout # doveadm -f table fetch ... 2>&1 Maybe it's better to add another formatter to avoid tricky parsing or shell hacks e.g. # doveadm -f tab-nohdr ... Joseph Tam
Re: Mailbox configuration questions
On Sat, 6 Mar 2021, justina colmena ~biz wrote: I am having subtle problems with IMAP mailbox configuration on certain clients such as KMail and Thunderbird, whereas the previous setup was working on K9Mail (mobile) and Trojit? (desktop). I was using Maildir folders, which were mostly working before, but for some reason I had to create explicitly named namespaces for the flatfile (mbox) Inbox and the Maildir "Home" folders. I also specified an INDEX directory for the inboxes, which I made world-writable and sticky, because of permission problems creating subdirectories in it. Ordinarily, mail readers using a remote mail protocol are not concerned with the underlying storage; IMAP servers deal with those details and provide abstractions to the client such as namespaces, mailboxes, messages, etc., although it does manifest itself in some ways (e.g. Maildir allows maiboxes to contain both messages and other mailboxes). The abtstractions provided by POP and IMAP are quite different, though. I can't quite tell from your statement whether you're using the same server (and configurations) for both sets of clients. A dovecot configurations dump would be useful. KMail always seems to put sent mail into a local "sent-mail" folder, rather than the IMAP Sent folder associated with the sending account. (KMail and Thunderbird have a more POP-oriented architecture for the desktop, whereas Trojit? is exclusively IMAP.) (What do you mean by "POP-oriented"? One mailbox (INBOX)? Store and forward operation? I wouldn't agree with either of these 2 assertions.) Outgoing mailbox name is a mail reader setting. Some default to "sent-mail", some to "Sent", some to others. There are various ways you can try unifying them to a single mailbox in IMAP: - mailbox aliasing: various ways such as filesystem symlink, or dovecot aliasing (https://wiki2.dovecot.org/Plugins/MailboxAlias). - IMAP SPECIAL-USE (RFC6514) which hints to the mail reader which mailbox to use for a specific purpose. Not all readers implement this. - publish a standard configuration for your users: this delegates control to your users, rather than enforcing it using the server. Is there an easier better way to organize some of this stuff? Or how is it "usually" done? I'm not sure what you mean by "organizing": making users' mail more consistent across different mail readers, despite their differences? Most are taken care of by using IMAP, and there are special niche settings for the mail reader features you're trying to address. Joseph Tam
Re: t/s expired cert error
On Wed, 3 Mar 2021, Yassine Chaouche wrote: Le 3/2/21 ? 9:02 PM, Matthias Kneer a ?crit : # echo | openssl s_client -connect emu.sbt.net.au:110 2>/dev/null | openssl x509 -noout -enddate I am intrigued about the function of echo in that command line ? It just a dummy input so that openssl s_client does wait for data from stdin. The OP could have also done openssl s_client ...
Re: Can Dovecot honor Outlook's "leave mail on server for X days" setting?
On Tue, 2 Mar 2021, Steve Dondley wrote: I've got a linux box running dovecot/postfix using maildir format. I was surprised to learn that a client that had many GBs of email was running POP3, not IMAP. It turns out they had a setting to delete POP3 mail after X days turned on but it just went ignored. I know this is not how POP3 is supposed to work, but is there a way to get dovecot to honor the user's settings in Outlook? Or should I just tell the client to turn this off and use a proper IMAP account? It's not dovecot (or any POP3) server's job to implement this setting, it's the client's. Typically, the mail reader downloads a list of messages, then issues "DELE" commands to remove messages based on whatever criteria the user had set. Dovecot has no idea what the user's setting is. Other people seem to have the same problem: https://www.hmailserver.com/forum/viewtopic.php?t=29988 https://answers.microsoft.com/en-us/msoffice/forum/msoffice_outlook-mso_other-mso_2010/outlook-2010-not-removing-email-from-pop3-server/f952f469-a95b-4a5a-a805-a304354c2c1a https://portal.smartertools.com/community/a90038/outlook-pop-leave-mail-on-server-for-days-not-deleting-mail.aspx You may have to create a POP3 session log to diagnose what POP3 commands you're client is issuing. Joseph Tam
Re: Obtaining the IMAP GUID from a sieve script
On Fri, 15 Jan 2021, Ron Garret wrote: Why not simply use the message-id? Because not every email has one. RFC5322 doesn?t require them. Doesn't your MTA then insert one if it's missing? Joseph Tam
Re: ulimit -n vs client_limit vs process_limit
On Thu, 17 Dec 2020, Aki Tuomi wrote: What should I set "ulimit -n" relative to client_limit? Or perhaps I've roofed You need to adjust LimitNOFiles (or ulimit -n). Dovecot needs more file descriptors than just the ones used by imap-login process, so it is a good idea to consider setting it to at least 3x time the value. Thanks, good to know. Out of curiosity, I did a snapshot of FD usage for dovecot's supervisory processes: # for p in imap-login config log auth pop3-login stats dovecot anvil; do # echo "$p \c"; ls /proc/`pgrep -f /$p`/fd | wc -l # done imap-login 357 config 14 log 38 auth 22 pop3-login 20 stats 193 dovecot 221 anvil 18 I guess "dovecot" or "stat" are the particular processes that needs to have FD limits set larger than to the sum of client_limits. Joseph Tam
ulimit -n vs client_limit vs process_limit
As many of my users are accessing their mail remotely, I've seen
service loads increase and came across this log messages which I
haven't seen before
imap-login: Error: socketpair() failed: Too many open files: user=<*>,
...
I gather I have to increase file descriptor limits, which is currently
set to match
default_client_limit = 1000
What should I set "ulimit -n" relative to client_limit? Or perhaps I've roofed
service imap-login {
process_limit = 2
...
}
and should adjust that?
Joseph Tam
Re: Disallow acces via imap, but keep lmtp running
On Wed, 16 Dec 2020, Plutocrat wrote: On 16/12/2020 06.16, Julian Kippels wrote: what is the best way to temporarily disable access to a mailbox via imap, but keep it possible to deliver to the mailbox via lmtp? Block IMAP ports on the firewall? passdb with "deny=yes"? Or if IMAP is the only authenticated service, munge their password hash. Joseph Tam
Re: Putting UIDL value into X-UIDL: header
On Mon, 14 Dec 2020, S?leyman D?zdaban wrote: Thank you for your answer. I'm aware of pop3 migration plugin, but I want to know if there is any way to put UIDL value into messages. It's been a while since I've migrated from Qpopper, but I thought there was a config setting to reuse UIDLs. Maybe pop3_reuse_xuidl = yes Joseph Tam
Re: important message
On Thu, 10 Dec 2020, Aki Tuomi wrote: Hi everyone, sorry about this, this email was accidentically approved. We will be more careful next time. If you're wondering what this is all about, I believe spammers have lately found a way to subvert a Google Forms feature and have been hammering it to piggyback spam: https://security.stackexchange.com/questions/241263/how-is-it-possible-that-this-spam-mail-came-from-google-forms-without-revealing Blocking mail from @trix.bounces.google.com will squelch them, but may also biock legitimate response receipts. Joseph Tam
Re: Recommended Protocols?
On Mon, 9 Nov 2020, Raymond Herrera wrote: I am preparing a new server, with Dovecot 2.2.36 and would like to know the currently recommended protocols. Should I stick to what I have? I would prefer to start with the easiest configuration possible, which I will revise later. This is the command that I have been using to verify the server's functionality: % openssl s_client -connect localhost:imaps Implicit SSL (SSL/TLS) has the slight advantage over STARTTLS as a MITM cannot strip the STARTTLS server banner during the session handshake and downgrade the client to plaintext. However the most important security consideration are - set SSL version to at least TLS 1.2 to avoid known weakness in older versions. - set cipher list to avoid weak ciphers. One of many guides https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices - (client) enforce SSL connection (i.e. refuse plaintext sessions). Joseph Tam
Re: SV: Looking for a guide to collect all e-mail from the ISP mail server
On Tue, 27 Oct 2020, Sebastian Nielsen wrote:
Kind of stupid that there doesn't exist some common standard for 2FA that
works in email clients.
You can bodge it for HOTP/TOTP hardware token generators. Dovecot allows
custom plugins to check passwords. The plugin can take passwords of
the form {password}+{2fa-token}, then split each part to check against
authentication systems to check validity.
Joseph Tam
Re: Users unable to login
On Wed, 21 Oct 2020, David Pottage wrote: I think there is a limit on the number of concurrent IMAP sessions that each user can have open at once, and the default is 5 or something fairly low. Modern smartphone IMAP clients can easily have that many sessions open, if the user is monitoring many folders for new mail, so when the user attempts to log in with another device they cannot because they have hit the limit. Global searches using Apple mail readers will open as many concurrent mailboxes as your settings allow, even hundreds. However, they're closed in batches as well, so a graph of user mailbox connections will show sawtooth patterns. Joseph Tam
Re: Auro expunge
On Wed, 14 Oct 2020, Maciej Milaszewski wrote: On 14.10.2020 16:28, Adrian Minta wrote: $DOVEADM expunge -A mailbox Trash savedbefore 30d $DOVEADM expunge -A mailbox Spam savedbefore 30d You might as we save yourself an invocation by doing $DOVEADM expunge -A \( mailbox Trash OR mailbox Spam \) savedbefore 30d But if you have more users (200K) that is a problem with that scripts Why would an access triggered expungement be more problematic vs a periodic cron job? If it creates intense I/O loads, you can do it at off-peak times and/or only do userbase subsets at any one time. Joseph Tam
Re: Feature request.
On Fri, 9 Oct 2020, David Morsberger wrote: Both the renew hook and post hook are good candidates for our reload script. Each has a downside however. The post hook will be run after every renewal attempt, regardless of if anything was actually renewed or not. This will result in the services being reloaded many times for no reason. An alternative to using certbot hooks is to use an inotify based tool (available for most Linux based OS). A certificate update triggers a restart script. For example, https://linux.die.net/man/5/incrontab The renew hook only runs if a certificate was successfully renewed, but it will be run once for each certificate. This could mean reloading services multiple times if you have multiple certificates. If you only have a single certificate however it'll work great. For this case, I think you need a periodic (cron) process, restart rather than a synchronous process, that will check certs and restart/reload once per day/week/whatever. This is the method I use as my LE certificates are obtained via DNS challenges on a different host. Joseph Tam
Re: debugging TLS with wireshark and a custom application ?
On Wed, 30 Sep 2020, Kurt Jaeger wrote: My question is: can dovecot be used to debug/decrypt TLS sessions ? The reason I'm asking: A custom application wants to speak IMAP with TLS with a dovecot instance. If it's happening during the handshake (i.e. IMAP/POP hasn't even started) you can try debugging the interaction by using "openssl s_server" on an alternate port with the same SSL parameters used by your dovecot. It's not the full-fledged environment you're trying to test but may expose the problem. Joseph Tam
Re: Apple Mail Since upgrade to dovecot 2.3.x unable to connect
On Mon, 17 Aug 2020, Johannes Rohr wrote: You need to set ssl_min_protocol = TLSv1.2 # or TLSv1 Thanks, tried both, but unsuccessfully. Don't give up too easily/early on this. I said this before, but MacOSX Mail behaves weirdly. I've more than once changed a server setting, without apparent effect, only to have MacOSX Mail mysteriously start working again after some time. Maybe it caches settings. Also, disable "Automatic manage connection" as failure to establish a successful session will cause your client to do some auto-wandering to discover settings, which could really do your head in. Joseph Tam
Re: Migration issue
On Tue, 4 Aug 2020, Kishore Potnuru wrote: So, both password files (master and regular user credentials) have the same contents in this scenario, correct? No. Master users are administrative users you allow to authenticate as another account without having to know their password. If the master account is "master", and the user account is "xyz", then an administrator can access xyz's mail by authenticating as username: xyz*master password: password for master This would allow, for example, to migrate all users via the IMAPC mechanism without having to know all their passwords. By making both master and passdb's the same, you allow anyone to access anybody else's account e.g. "xyz" can access account for "abc" by using their password with user "abc*xyz". Joseph Tam
Re: Migration issue
On Mon, 3 Aug 2020, Kishore Potnuru wrote:
===
Jul 28 11:14:23 auth: Fatal: Master passdb can't have pass=yes if there are
no passdbs
Jul 28 11:14:23 master: Error: service(auth): command startup failed,
throttling
===
after the above error, I have commented "pass=yes" in production1 (old
server) server, then I see the below error.
Jul 28 11:17:10 auth: Fatal: No passdbs specified in configuration file.
PLAIN mechanism needs one
Jul 28 11:17:10 master: Error: service(auth): command startup failed,
throttling
===
=
My old server dovecot.conf (production1):
=
passdb {
args = /etc/dovecot/passwd
driver = passwd-file
master = yes
pass = yes
}
My interpretation of your error messages is you need 2 sets of credentials:
regular users and master users. You've only supplied master passwords.
You'll need
# Contains master users credentials
passdb {
args = /etc/dovecot/master-passwd
driver = passwd-file
master = yes
pass = yes
}
# Contains regular user credentials
passdb {
args = /etc/dovecot/passwd
driver = passwd-file
}
Joseph Tam
Re: Massive alias / bulk delivery problem
On Tue, 14 Jul 2020, gnd wrote: anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know. It depends on what you mean by "optimize". Arguably, letting postfix/dovecot hammer its brains out for a few minutes might have the lowest overall performance/complexity cost, but may also be DoS'ing your mail system for a small time window. Configuring postfix to concurrently send message to a single set-UID LMTP will probaby help with minimizing process overhead. It might be especially useful if you also need to de-duping large attachments. If your current setup can cope with this mail load, just let it. If it ain't broke, why fix it? If the intensity is causing problems, you could offload mail delivery to an auxilliary process outside your mail system by aliasing to a handler script ( e.g. |remail.sh), which accepts the message, then sends it to your 20k+ recipients in small batches with small delays. Joseph Tam
Re: How to use dovecot only as POP3 server / prevent it from creating .imap directories?
On Wed, 17 Jun 2020, Josef 'Jeff' Sipek wrote: On Tue, Jun 09, 2020 at 12:32:18 +0200, Binarus wrote: ... ./mail/inbox ./mail/.imap ./mail/.imap/dovecot-uidvalidity ./mail/.imap/dovecot.list.index.log ./mail/.imap/dovecot-uidvalidity.5edce848 ./mail/.imap/INBOX ./mail/.imap/INBOX/dovecot.index.log ./mail/.imap/INBOX/dovecot.index ./mail/.imap/INBOX/dovecot.index.log.2 ./mail/.imap/INBOX/dovecot.index.cache The .imap directory isn't really about IMAP. It is sort of a generic directory that just happens to have "imap" in the name. The index files you see are required for various features inside dovecot to work properly. Many of them are related to performance rather than a specific protocol. Spot on. If you don't want the index files to live inside the users' home directories, take a look at the INDEX and CONTROL keys in the mail_location docs: https://doc.dovecot.org/configuration_manual/mail_location/#format While that still creates the files, you can move them off to a location that the users do not see. Or you can create in-memory indices, but that is a solution to the wrong problem. By specifically stating in mail_location that ~user holds mailboxes, you are telling Dovecot an untruth. It's better to tell Dovecot user mailboxes (other than INBOX) don't exist, rather than to push all the indices under the carpet. Joseph Tam
Re: handling spam from gmail.
On Fri, 12 Jun 2020, Andreas Born wrote: Maybe, and I really hope so, this problem no longer exists. I will immediately reconfigure my mail system, if rejecting mails after DATA will be safe and reliable nowadays. In particular, bots don't hang around for the DATA response. Any MTA that ignores SMTP responses for the DATA step would also ignore common conditions like full mailbox. Such brokeness and failure to follow RFC is by itself grounds to reject the mail until the MTA software is fixed. One blacklist operator actually uses this as a criteria for blacklisting (Section: Tracking use of QUIT) http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists I issue post-DATA return codes, and I have yet, in decades of use, had problems with legitimate senders. Joseph Tam
Re: SV: handling spam from gmail.
On Thu, 11 Jun 2020, lists wrote: I get two or three of these a day. They are not from Gmail but have a "reply to" address that is a Gmail account. The messages cone from an email account that passes SPF and DKIM. So the sender and reply domains differ, but that isn't unique. I have email that I need that arrives like that. This entire thread belongs on an anti-spam forum, but you might want to check out http://msbl.org/ebl.html Joseph Tam
Re: How to use dovecot only as POP3 server / prevent it from creating .imap directories?
On Sun, 7 Jun 2020, Binarus wrote: So how exactly do I have to alter the configuration to implement your suggestion, i.e. to make dovecot look only at the mbox file and to prevent the creation of unnecessary directories? Maybe try mail_location = mbox:/empty/dir:INBOX=~/inbox Not sure whether owner=rootZ:root, mode=555 will work, but those permissions would be the safest. Joseph Tam
Re: Simple backup of maildir folder
On Sun, 31 May 2020, Laura Smith wrote:
A couple of notes on this quite useful script:
My mktemp does not support -p (FreeBSD 12.1) is I had to change the script to:
In my scripts I tend to create a tempdir and then tempfiles within that. It
makes the cleanup routine neater, e.g. at the top of my scripts :
TEMP_DIR=$(mktemp -qd || { doLog "Failed to make temp dir !"; exit 1; })
rmTmpFiles() { rm -rf "${TEMP_DIR}"; }
createTempFile() { local MYTEMP=$(mktemp -qp "${TEMP_DIR}" || doLog "Failed to
create temp file"; exit 1); echo $MYTEMP; }
Also my backup scripts have locking procedures built-in so as to avoid race
conditions.
You might also want a trap handler that does a cleanup in case something
goes sideways in the middle of processing e.g.
trap rmTmpFiles 0
Joseph Tam
Re: Running doveadm without config file?
Sami Ketola writes: I tried this with /dev/null and /tmp/empty.conf but ran into another wall doveadm(root): Fatal: execv(/usr/bin/doveconf) failed: No such file or directory How did you install dovecot on that system since /usr/bin/doveconf is part of the base package? Manual install: not using a package manager. (I've edited the doveconf location, but you've outed me.) I was hoping to get "doveadm pw" working on non-dovecot servers without having to provide seemingly irrelevant dependencies, but it's probably more bother than its worth. Thanks, anyways. Joseph Tam
Re: identify 143 vs 993 clients
On Sun, 31 May 2020, Jean-Daniel wrote: So yes the safest way to go is to just use port 993, but as long as the client is not set to a "TLS if available" option then port 143 is also safe. I don?t think you can call an option safe if it relies on the users to properly configure their client. We all know that users are usually bad at following instructions ;-) I think Peter nailed it, but let's put it this way: the server policy is irrelevant to client side policy. *If* the client has been not been configured to disable plaintext password, a malicious party can coax a password out of a client, despite what the server policy is, or even whether the server is available. Only allowing implicit SSL will guarantee insecurely configured clients will fail (and maybe not even that if it autoconfigures), but it doesn't prevent them from being exploited. Joseph Tam
Re: Running doveadm without config file?
On Fri, 29 May 2020, Sami Ketola wrote: # echo plaintextpass | doveadm pw -s BLF-CRYPT doveadm(user): Fatal: Error reading configuration: stat(/etc/dovecot/dovecot.conf) failed: No such file or directory Is there a way to circumvent the need for a configuration file? I don't think so. But you can specify the location of the config file with -c /path/to/file like echo -n plaintextpass | doveadm -c /root/emptyfile.conf pw -s SHA1 Thanks, Sami. I tried this with /dev/null and /tmp/empty.conf but ran into another wall doveadm(root): Fatal: execv(/usr/bin/doveconf) failed: No such file or directory Joseph Tam
Running doveadm without config file?
It would be useful to run the "doveadm" utility on a non-dovecot server e.g. generating password hashes: # echo plaintextpass | doveadm pw -s BLF-CRYPT doveadm(user): Fatal: Error reading configuration: stat(/etc/dovecot/dovecot.conf) failed: No such file or directory Is there a way to circumvent the need for a configuration file? Joseph Tam
Re: identify 143 vs 993 clients
On Tue, 26 May 2020, mj wrote: On 25/05/2020 23:04, Voytek wrote: jumping here with a question, if I use 143 with STARTTLS, and, force TLS/SSL in configuration, that's equivalent from security POV, isn't it? and, same for 110 STARTTLS? Or am I missing something? There's an important clause here that often becomes overlooked: "force TLS/SSL in [client] configuration". If you don't fulfil this condition, STARTTLS can fall prey to downgrade attacks. This has been done, and not by small players: https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks Some mail readers, like macOX Mail, will happily reconfigure your mail reader server settings to use plaintext unless you disable it. Interesting point, after some googling, I think you are right, and as long as we have set "disable_plaintext_auth = yes" (and we have that) we should be fine keeping 143 open. Right? Yes, provided the above condition is met. However, unless you control all endpoints, that's hard to enforce. One doubt I had: "disable_plaintext_auth = yes" sounds as if only the authentication part is secured, and the rest is kept plain text, whereas with 993/SSL, *everything* would be encrypted? Once STARTTLS negotiations are over, it is equivalent to SSL: all data is encrypted. However, I see your point: the configuration label suggests it's limited to authentication data, as opposed to all data. Something like "ssl_forbid_decline" or "ssl_not_optional" might have been clearer. Joseph Tam
Re: fail2ban setup centos 7 not picking auth fail?
On Fri, 22 May 2020, Jerry wrote: On Thu, 21 May 2020 23:22:04 -0700, lists stated: I use SSHGuard on well ssh (doh!), but supposedly you can use it for postfix and dovecot also. I can tell you it is well supported. I am on Centos 7 using firewalld. SSHGuard works fairly well with Postfix; however, it is virtually useless with Dovecot. It never picks up on "auth fail" and a few others. I have submitted documentation and requests to SSHGuard, but they have never acted upon them, other than to say that they will look into it. That's the beauty of open source -- if you got time and skillz, you can roll up your sleeves and do it yourself. I peeked at the source, and it requires some Lex/Yacc coding. Even if you don't have those codng skills, you can probably make a good guess by looking at the .l/.y files. The authors can make it a lot easier to extend if they externalize the patterns into runtime configuration like fail2ban does, rather than baking them into executables. Joseph Tam
Re: What's a Reasonable Inbox Size?
On Fri, 8 May 2020, Joseph Tam wrote: It depends on what you consider reasonable. Whoops. Editing error. What I wanted to send. On Fri, 8 May 2020, a...@globalchangemusic.org wrote: So, generally speaking, you don't want to have inboxes that just sync all day long, due to massive amounts of small files in the inbox. I don't know enough about what is involved when your client tries to sync to comment on your particular situation. If the exchange of information involves only delta changes (e.g. list datum that have been added/removed since the last sync), and if this information is readily available in Dovecot's caches, then this operation might be optimized to take minimal time. If however, it involves exchanging entire lists of many messages IDs, or worse, involves Dovecot accessing each message, it will result in large amounts of time spent in I/O (network, disk or both). With Maildir (many small message in a folder), this causes seeking all over the disk. Some filesystems (XFS?) may be better at this than others. The description of your problem seems to suggest the latter, so breaking up gigantic mailboxes into manageable volumes will help. If you really want to see what's going on when a client syncs, you can network trace, process trace, or use Dovecot's rawlog feature https://wiki.dovecot.org/Debugging/Rawlog to directly observe the iteraction between a server and client. This may be OK in the case of a rarely accessed archive folder, but not good for regularly accessed inboxes, etc.? This is not really so much technical advice as a rule of thumb: there's not a lot of payoff to optimizing rare operations. Joseph Tam
Re: What's a Reasonable Inbox Size?
On Fri, 8 May 2020, a...@globalchangemusic.org wrote: It depends on what you consider reasonable. The processing time of file operation that iterates through a mailbox will generally go up proportinately with size. If you do a text search without some indexing system like Solr, it will take a very long time. If the mailbox is just some archive that you pile up and forget about it except for once in a blue moon retrieval, then it might be reasonable. If it's an active mailbox, it will be a pain to navigate, in the same way a single folder with 100K files or a file cabinet with huge stacks of envelopes. I would guess some partioning of the large mailboxes into smaller mailboxes would help with active mailboxes. Most people spend most of their time on new/recent messages, so making time or size or subject based volmes wouldn't be a bad idea. If the bulk of the size are redundant copies of attachments, then Dovecot's *dbox support de-duping which would aso help. So, generally speaking, you don't want to have inboxes that just sync all day long, due to massive amounts of small files in the inbox. This may be OK in the case of a rarely accessed archive folder, but not good for regularly accessed inboxes, etc.? Joseph Tam
Re: What's a Reasonable Inbox Size?
On Thu, 7 May 2020, Asai wrote: I have several users who have inboxes that are over 20 GB. As email admins, how do you handle inboxes that are so large? Do you use mailbox types that have better performance like dbox? We're using maildir. What's a reasonable inbox size? Is 20+ GB reasonable and nothing to worry about? It depends on what you consider reasonable. The processing time of file operation that iterates through a mailbox will generally go up proportinately with size. If you do a text search without some indexing system like Solr, it will take a very long time. If the mailbox is just some archive that you pile up and forget about it except for once in a blue moon retrieval, then it might be reasonable. If it's an active mailbox, it will be a pain to navigate, in the same way a single folder with 100K files or a file cabinet with huge stacks of envelopes. I would guess some partioning of the large mailboxes into smaller mailboxes would help with active mailboxes. Most people spend most of their time on new/recent messages, so making time or size or subject based volmes wouldn't be a bad idea. If the bulk of the size are redundant copies of attachments, then Dovecot's *dbox support de-duping which would aso help. Joseph Tam
Re: Dovecot IMAPS : Thunderbird SSL cert issue / Evolution OK
On Thu, 30 Apr 2020, hanas...@gmail.com wrote: Apr 8 18:10:18 hh dovecot: imap-login: Debug: SSL error: SSL_accept() failed: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate: SSL alert number 42 According to this https://serverfault.com/questions/806141/is-the-alert-ssl3-read-bytessslv3-alert-bad-certificate-indicating-that-the-s this error comes about when you specify the client must authenticate with their own certificate. If your Dveocot setup is working with Evolution, have you ported the client certificate to the Thunderbird setup? Joseph Tam
Re: Recommendations on intrusion prevention/detection?
On Wed, 22 Apr 2020, Johannes Rohr wrote: It is a pity that the IMAP protocol does not support 2 factor authentication, which seems to stop 90% of intrusion attempts in their tracks. You could use VPN, which can enforce 2FA. You can hack 2FA into IMAP or any protocol where you can control the backend authenticator. It's easier with time-based OTP (TOTP) token generators. Authenticate using the usual username and the concatenation of (user-password)(otp-token), then invalidate the opt-token to foil replay-attacks. The backend will have to split the credentials into individual factors that can be checked separately. Is there a reasonable way of detecting and preventing logins from unusual IP ranges? Or are there other strategies you would recommend? Start by defining "unusual". Once you have a characterization of unusual, implement the detection. For example, - more than failures? - attempt to authenticate to non-existent generic accounts e.g. "root"? - weird time of day? - authentication from implausible geographic regions? (e.g. Chad)? - logins from mutiple geolocation in short time frames? As the saying goes regarding the value of prevention vs cure, enforce good security habits for your users: password strength, endpoint malware protection, skepticism, etc. Joseph Tam
Re: Dovecot Failed to initialize SSL server context
Adam Raszkiewicz writes: I'm trying to configure TLS for Dovecot 2.3 but after setting all things up I'm not able to start Dovecot: Apr 16 20:56:25 imap-login: Error: Failed to initialize SSL server context: Can't load SSL private key: Key is for a different cert than ssl_cert: user=<>, rip=::1, lip=::1, secured, session= This is your problem -- you have a mismatch between key and certificate. Check again using doveconf your certificate and key file ocations they correspond to what you expect. I have created a brand new key pair, csr and the cert but still it complains to start. Any thoughts on that? You normally don't use the CSR, but you can check all the files to make sure the modulus matches and they belong to each other openssl x509 -in file.crt -noout -modulus openssl rsa -in file.key -noout -modulus openssl req -in file.csr -noout -modulus Joseph Tam
Re: got a listener on 993
On Tue, 14 Apr 2020, Ivo wrote: Maybe this thread can help you with your first question : https://dovecot.org/pipermail/dovecot/2014-August/097488.html I was more or less going to say the same thing. Further to this, it's more important to make sure your clients enforce SSL/STARTTLS use by disabling auto-discovery, and if you're ultra-conservative, certificate pinning. Joseph Tam
Re: doveadm backup from gmail with imapc
On Thu, 9 Apr 2020, Plutocrat wrote:
I could never figure out how to get the dovecot indexes correct, so
that when the client connected to the new server via POP, it didn't
re-download all the messages. IMAP was OK though.
It's been a while since I migrated from the qpopper, but I believe this
configuration may be the answer:
(On new server)
protocol pop3 {
...
pop3_reuse_xuidl = yes
}
Joseph Tam
Re: At rest encryption (with protected crypto keys)
On Tue, 24 Mar 2020, Kees de Jong wrote: As stated on the Dovecot documentation, at rest encryption is possible [1]. However, these keys are present on the system itself and are unprotected. Therefore, if a system is compromised, the attacker has access to the encrypted mail and the keys. There is no security benefit in that situation, except for hoping that the attacker doesn't understand that this is happening and how. Nextcloud does this a bit better. A key is used to encrypt user data as well [2]. However, that key is protected with the user's password. When the user logs in and requests data, the user's password unlocks the key and data can be read and written safely. This also takes into account password changes. Files don't need to be encrypted again, the encryption key is simply re-encrypted with the new user's password. How does the Dovecot community see this? The answer depends on how much security you want, and what you assume an eavesdropper has access to. The protection described in the second paragraph is merely an extension of the first, where secrecy is implemented on the server side. If the system is compromised, it only takes several strategic placement of code to intercept the secret parts and unravel the entire workings. It may require expertise, but in theory, it's falls prey to the dishonest administrator or skilled attacker. A stronger form is client-side encryption: the key and encryption is done on the client side, then only the encrypted data is transferred to the server. The Nextcloud (or Dropbop) example is to have a encrypted FS on the client side (e.g. VeraCrypt) and the whole container is sync'd on the storage side (the server). At no point does the server side ever get to see keys. Joseph Tam
Re: How does dovecot determine users from /etc/passwd?
On Wed, 19 Feb 2020, Philip Colmer wrote: /usr/bin/doveadm flags add -A '\Deleted' SEEN SENTBEFORE 12w && /usr/bin/doveadm expunge -A DELETED MAILBOX '*' This may have been overtaken by Sami autoexpunge solution, but you can roll two I/O intensive operations into one -- there's no point setting flags on a message you'll expunge. /usr/bin/doveadm expunge -A DELETED OR \( SEEN SENTBEFORE 12w \) Joseph Tam
Solaris crash again (was v2.3.9 released)
On Wed, 4 Dec 2019, Aki Tuomi wrote: We are pleased to release v2.3.9 of Dovecot. Please find it from locations below I compiled this and ran into the same crash problem I reported as Issue 3 (v2.3.6) in https://dovecot.org/pipermail/dovecot/2019-July/116413.html The last 2 messages in the thread contains Timo's analysis of the problem, and my contributed patch which fixed it. However, this patch was not applied to subsequent versions. https://dovecot.org/pipermail/dovecot/2019-July/116619.html Is this problem peculiar to Solaris (requiring me to re-apply this patch for all future versions) or should this patch be applied in all cases? Joseph Tam
