error getting messages from mdbox_deleted
I was trying to list messages deleted with doveadm -f flow -o mail_location=mdbox_deleted:/home/mail/xxx /mdbox fetch -u xxx "mailbox date.saved guid" ALL And getting this error. What is this about? doveadm(xxx): Panic: file mdbox-map.c: line 1494 (mdbox_map_get_uid_validity): assertion failed: (map->view != NULL) doveadm(xxx): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xa192e) [0x7f854625592e] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f854625599a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f85461e5257] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x61830) [0x7f854654c830] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_update_header+0xa0) [0x7f8546551110] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x5d698) [0x7f8546548698] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x5d7e0) [0x7f85465487e0] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) [0x7f85465372d4] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7f8546537377] -> doveadm(doveadm_mail_iter_init+0x98) [0x55b4cd34c118] -> doveadm(+0x305b1) [0x55b4cd34a5b1] -> doveadm(+0x2bdcc) [0x55b4cd345dcc] -> doveadm(+0x2c9ca) [0x55b4cd3469ca] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x233) [0x55b4cd347823] -> doveadm(doveadm_cmd_run_ver2+0x50c) [0x55b4cd357fac] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55b4cd358047] -> doveadm(main+0x1e4) [0x55b4cd336214] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8545e08555] -> doveadm(+0x1c605) [0x55b4cd336605] Aborted
RE: Processing incoming mail efficiently
> -Original Message- > From: dovecot On Behalf Of Ron Garret > Sent: 30 January 2021 17:49 > To: Dovecot > Subject: Processing incoming mail efficiently > > I’ve asked a related question on this list before but I now have a much > better handle on what I’m doing and I realize that I still don’t know > the answer, so I’m going to ask this again in a slightly different form. > > I’m writing a spam filter, so obviously I need to feed incoming mail to > it somehow. The “obvious” way to do this is with a sieve script using > the pipe extension. There are two problems with this: No, that is not obvious, this would imply a dependency on sieve. > 1. This will always pipe the entire file no matter how big it is. The > filter will often not need to process the body of the message, Yes because your starting point is wrong. Using mailfromd you can process a specific milter state, see envfrom envrcpt etc. https://puszcza.gnu.org.ua/software/mailfromd/manual/mailfromd.html#handler-names only the > headers, or only the first part of a multipart MIME message. Is there > any way to allow my filter to open the file in which the message is > stored rather than piping it a copy of the message? > > 2. Once the filter has processed the message and decided if it’s spam > it still needs to move the message to the appropriate folder (INBOX or > Junk). To do this it needs to somehow correlate the *content* of the > message that was piped to it with the UID of the message that needs to > be moved. One way to do this is to pull out the message-id header and > then use doveadm No, in what ever milter state you are processing. You can add a message header 'This is spam'. And you make just one sieve rule that moves messages on the existance of that specific header. > to find the file containing the message with that > message-id, but there are two problems with this. First, not all > messages have message-ids. I can work around this by adding my own First you have crawl, before walking. So learn how to crawl. It does not make sense trying to make something, if you do not know specifics. > message-id to messages that don’t already have them, but this just feel > wrong. And second, unless dovecot keeps an index of message-ids (does > it?) then this will be horribly inefficient because it will have to > essentially grep for the message id every time I want to move a message. > So it seems like there has to be a better way, but I can’t think of what > that would be. Start playing with mailfromd. It has scripting language to configure it and all tools(funtions) are available to do whatever you can think of. https://puszcza.gnu.org.ua/software/mailfromd/manual/mailfromd.html#Filter-Script-Example > I figure this has to be a solved problem because I am obviously not the > first person to write a spam filter for dovecot. What is the Right Way > to do this? > As written above
mail event
> > Using mail_log plugin is always very recommended, > > https://doc.dovecot.org/configuration_manual/plugins/mail_event_logging/ > I am not sure if I am correct, but I think this logs every deleted message not? I was testing with this a while ago. Is there also an option that only logs something like '300 messages deleted'?
RE: mbox to pst advice
> > > On 2021-01-17 18:43, Odhiambo Washington wrote: > > > Personally, I would not bother looking for a script or even > asking > > anyone if they knew how to convert > > mbox2pst. > > there is always alternatives :-) > > aid4mail > > You have good experience with this (and are not in any way affiliated with this company/product)? Because there are so many of these 'our super-hero-mail-converter tools' are the best.
RE: mbox to pst advice
> > Anyone an idea how to convert mbox ot pst on linux? > > > > > > Just run let Outlook fetch the mail and it will automatically create a > PST. > > There is no need for wasting CPU cycles and time converting :-) Scripting this on linux would be much faster for me than doing it manually with outlook.
mbox to pst advice
Anyone an idea how to convert mbox ot pst on linux?
RE: CVE-2020-24386: IMAP hibernation allows accessing other peoples mail
I guess redhat will backport it. -Original Message- Sent: 04 January 2021 14:02 To: dovecot@dovecot.org Subject: Re: CVE-2020-24386: IMAP hibernation allows accessing other peoples mail On 2021-01-04 13:03, Aki Tuomi wrote: > Vulnerable version: 2.2.26-2.3.11.3 > Fixed version: 2.3.13 No fix for 2.2.36? -- BR/Mvh. Dan Malm, Systems Engineer, One.com
RE: CVE-2020-24386: IMAP hibernation allows accessing other peoples mail
This also applies when you have users seperated at os level? -Original Message- Sent: 04 January 2021 13:03 To: dovecot-n...@dovecot.org; dovecot@dovecot.org Subject: CVE-2020-24386: IMAP hibernation allows accessing other peoples mail Open-Xchange Security Advisory 2021-01-04 Product: Dovecot Vendor: OX Software GmbH Internal reference: DOP-2009 (Bug ID) Vulnerability type: CWE-150: Improper Neutralization of Escape, Meta, or Control Sequences Vulnerable version: 2.2.26-2.3.11.3 Vulnerable component: imap Report confidence: Confirmed Solution status: Fixed by Vendor Fixed version: 2.3.13 Vendor notification: 2020-08-17 Solution date: 2020-08-27 Public disclosure: 2021-01-04 CVE reference: CVE-2020-24386 CVSS: 8.2 (CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:N) Vulnerability Details: When imap hibernation is active, an attacker can cause Dovecot to discover file system directory structure and access other users' emails using specially crafted command. The attacker must have valid credentials to access the mail server. Risk: Attacker can access other users' emails and filesystem information. Workaround: Operators can choose to disable IMAP hibernation. IMAP hibernation is not on by default. To ensure imap hibernation is disabled, make sure imap_hibernate_timeout is set to 0 or unset. Solution: Operators should update to 2.3.13 or later version.
RE: Imap client with ~7500 imap cmds ~250/~500 read() syscall
>
>
>
>>
>> Looks like this mail client has problems with the archive
>> namespace[1], I assume I have nothing strange in there because other
>> clients just issue around ~50 cmds. So I guess this is just bad
>> programming of some new flashy trendy tool someone has downloaded?
>>
>> The raw in log has
>> nr
>> 34212 LIST ""
>> 35190 SELECT Archive
>>
>> Like this:
>>
>
>> ..(more mailboxes)..
>>
>> mailbox Archive {
>> auto = create
>> autoexpunge = 0
>> autoexpunge_max_mails = 0
>> comment =
>> driver =
>> special_use = \Archive
>> }
>> mailbox "Archived mail" {
>> auto = no
>> autoexpunge = 0
>> autoexpunge_max_mails = 0
>> comment =
>> driver =
>> special_use = \Archive
>> }
>> mailbox "Archived messages" {
>> auto = no
>> autoexpunge = 0
>> autoexpunge_max_mails = 0
>> comment =
>> driver =
>> special_use = \Archive
>> }
>> ..(more mailboxes)..
>>
>> order = 0
>> prefix =
>> separator = /
>> subscriptions = yes
>> type = private
>> }
>
>Bit curious config, do you have an INBOX namespace somewhere as
>well? This looks like root namespace as it has no prefix.
>
Yes I have inbox namespace[1]. I think this config comes from in the
past wanting to have different mail locations and being bound to the
/var/spool/mail mbox files of sendmail.
[1]
namespace inbox {
disabled = no
hidden = no
ignore_on_failure = no
inbox = yes
list = yes
location =
mailbox Deleted {
auto = no
autoexpunge = 0
autoexpunge_max_mails = 0
comment =
driver =
special_use = \Trash
}
..
mailbox Trash {
auto = create
autoexpunge = 0
autoexpunge_max_mails = 0
comment =
driver =
special_use = \Trash
}
order = 0
prefix =
separator = /
subscriptions = yes
type = private
}
RE: Imap client with ~7500 imap cmds ~250/~500 read() syscall
Looks like this mail client has problems with the archive namespace[1],
I assume I have nothing strange in there because other clients just
issue around ~50 cmds. So I guess this is just bad programming of some
new flashy trendy tool someone has downloaded?
The raw in log has
nr
34212 LIST ""
35190 SELECT Archive
Like this:
350 SELECT Archive
2351 LIST "" "*"
2352 SELECT Archive
2353 LIST "" "*"
2354 SELECT Archive
2355 LIST "" "*"
2356 SELECT Archive
2357 LIST "" "*"
2358 SELECT Archive
2359 LIST "" "*"
2360 SELECT Archive
2361 LIST "" "*"
2362 SELECT Archive
2363 LIST "" "*"
2364 SELECT Archive
2365 LIST "" "*"
2366 SELECT Archive
2367 LIST "" "*"
2368 SELECT Archive
2369 LIST "" "*"
2370 SELECT Archive
2371 LIST "" "*"
2372 SELECT Archive
2373 LIST "" "*"
2374 SELECT Archive
2375 LIST "" "*"
2376 SELECT Archive
2377 LIST "" "*"
2378 SELECT Archive
2379 LIST "" "*"
2380 SELECT Archive
2381 LIST "" "*"
2382 SELECT Archive
2383 LIST "" "*"
2384 SELECT Archive
2385 LIST "" "*"
2386 SELECT Archive
2387 LIST "" "*"
2388 SELECT Archive
2389 LIST "" "*"
2390 SELECT Archive
2391 LIST "" "*"
2392 SELECT Archive
2393 LIST "" "*"
2394 SELECT Archive
2395 LIST "" "*"
2396 SELECT Archive
2397 LIST "" "*"
2398 SELECT Archive
2399 LIST "" "*"
2400 SELECT Archive
2401 LIST "" "*"
2402 SELECT Archive
2403 LIST "" "*"
2404 SELECT Archive
[1]
namespace 4archives {
disabled = no
hidden = no
ignore_on_failure = no
inbox = no
list = yes
location =
mdbox:/home/mail-archive/%u/Archive/:CONTROL=/home/mail-archive/%u/Archi
ve/control:INDEX=/home/archiveindex/%u/index
..(more mailboxes)..
mailbox Archive {
auto = create
autoexpunge = 0
autoexpunge_max_mails = 0
comment =
driver =
special_use = \Archive
}
mailbox "Archived mail" {
auto = no
autoexpunge = 0
autoexpunge_max_mails = 0
comment =
driver =
special_use = \Archive
}
mailbox "Archived messages" {
auto = no
autoexpunge = 0
autoexpunge_max_mails = 0
comment =
driver =
special_use = \Archive
}
..(more mailboxes)..
order = 0
prefix =
separator = /
subscriptions = yes
type = private
}
RE: Imap client with ~7500 imap cmds ~250/~500 read() syscall
>>
>> >
>> >>
>> >>
>> >> I am playing a bit with the monitoring, and I have noticed there
is
>> one
>> >> client of the chart with ~7500, while the others are between ~17
>> >> and
>> ~60
>> >> (dovecot_user_num_cmds).
>> >>
>> >> I assume this is related to the client, or is it possible
something
>> is
>> >> wrong on the server side? Is someone having also such numbers or
is
>> this
>> >> really strange. Is it possible to limit these cmd's?
>> >
>> >Maybe you should look into what command(s) are being executed first?
>> >
>> >event_exporter cmd_export {
>> > format = json
>> > format_args = time-rfc3339
>> > transport = log
>> >}
>> >
>> >metric imap_command {
>> > event_name = imap_command_finished
>> > filter {
>> > user = anomalous
>> > }
>> > exporter = cmd_export
>> >}
>> >
>>
>> Is there also a way to do this on dovecot 2.2?
>
>With rawlogs, I suppose.
>
If I get this raw log, I am getting huge amount of lists
cut -d ' ' -f2 20201230-181651-16624.out |sort | uniq -c | sort -n
..
17 FLAGS
20 0
328 NO
1284 )
1284
9021 OK
133350 LIST
RE: import_environment = $import_environment DEBUG=1 where should this be logged?
You know by any chance how to limit this only for one user?
-Original Message-
From: Odhiambo Washington [mailto:odhia...@gmail.com]
Sent: 30 December 2020 18:00
To: dovecot
Subject: Re: import_environment = $import_environment DEBUG=1 where
should this be logged?
Hi Marc,
If you read the link keenly - Debugging/Rawlog - Dovecot Wiki
<https://wiki.dovecot.org/Debugging/Rawlog> - you will realize that you need
to do some stuff.
I have done them and they produced the results.
Edit /path/to/dovecot/conf.d/10-master.conf and add the following:
Under this block:
add executable = imap postlogin to the block that has service imap {
...
}
Then create another block in the same file as follows:
service postlogin {
executable = script-login -d rawlog
unix_listener postlogin {
}
}
Identify the user for whom you want the rawlogs:
doveadm user -u techni...@mydomain.name
userdb: techni...@mydomain.name
user : techni...@mydomain.name
home : /var/spool/virtual/mydomain.name/technical
uid : 26
gid : 26
Now do the following:
cd /var/spool/virtual/mydomain.name/technical
mkdir dovecot.rawlog
chown -R 26:26 dovecot.rawlog
systemctl restart dovecot (or however you do it in CentOS)
Now login to imap as the user .. perform some operations.
Now look inside
/var/spool/virtual/mydomain.name/technical/dovecot.rawlog/
PS: Adapt as necessary. I tested this on FreeBSD with dovecot- 2.3.11.3
Ignore the /tmp/rawlog/%u portion of the HOWTO. I did it too, but there
was nothing in /tmp/rawlog/ even though I created the directory and did
chmod 1777 /tmp/rawlog.
Adios.
On Wed, 30 Dec 2020 at 19:30, Marc Roos
wrote:
You can also set DEBUG environment to have rawlog log an info
message
why it's not doing anything:
import_environment = $import_environment DEBUG=1
Where should this message appear, I also don't have this one.
[1]
https://wiki2.dovecot.org/Debugging/Rawlog
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)
RE: import_environment = $import_environment DEBUG=1 where should this be logged?
Yes thanks was missing the dovecot.rawlog dir.
-Original Message-
From: Odhiambo Washington [mailto:odhia...@gmail.com]
Sent: 30 December 2020 18:00
To: dovecot
Subject: Re: import_environment = $import_environment DEBUG=1 where
should this be logged?
Hi Marc,
If you read the link keenly - Debugging/Rawlog - Dovecot Wiki
<https://wiki.dovecot.org/Debugging/Rawlog> - you will realize that you need
to do some stuff.
I have done them and they produced the results.
Edit /path/to/dovecot/conf.d/10-master.conf and add the following:
Under this block:
add executable = imap postlogin to the block that has service imap {
...
}
Then create another block in the same file as follows:
service postlogin {
executable = script-login -d rawlog
unix_listener postlogin {
}
}
Identify the user for whom you want the rawlogs:
doveadm user -u techni...@mydomain.name
userdb: techni...@mydomain.name
user : techni...@mydomain.name
home : /var/spool/virtual/mydomain.name/technical
uid : 26
gid : 26
Now do the following:
cd /var/spool/virtual/mydomain.name/technical
mkdir dovecot.rawlog
chown -R 26:26 dovecot.rawlog
systemctl restart dovecot (or however you do it in CentOS)
Now login to imap as the user .. perform some operations.
Now look inside
/var/spool/virtual/mydomain.name/technical/dovecot.rawlog/
PS: Adapt as necessary. I tested this on FreeBSD with dovecot- 2.3.11.3
Ignore the /tmp/rawlog/%u portion of the HOWTO. I did it too, but there
was nothing in /tmp/rawlog/ even though I created the directory and did
chmod 1777 /tmp/rawlog.
Adios.
On Wed, 30 Dec 2020 at 19:30, Marc Roos
wrote:
You can also set DEBUG environment to have rawlog log an info
message
why it's not doing anything:
import_environment = $import_environment DEBUG=1
Where should this message appear, I also don't have this one.
[1]
https://wiki2.dovecot.org/Debugging/Rawlog
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)
import_environment = $import_environment DEBUG=1 where should this be logged?
You can also set DEBUG environment to have rawlog log an info message why it's not doing anything: import_environment = $import_environment DEBUG=1 Where should this message appear, I also don't have this one. [1] https://wiki2.dovecot.org/Debugging/Rawlog
RE: Getting rawlogging working with dovecot 2.2
Anyone using this with CentOS7 and dovecot-2.2.36-6.el7_8.1.x86_64?
RE: Getting rawlogging working with dovecot 2.2
>
>According to this manual[1] I just have to add these lines to my
config,
>not?
>
>protocol imap {
> rawlog_dir = /tmp/rawlog/%u
>}
>
>I should be seeing something in this /tmp/rawlog if I delete an message
>via imap not? I have already made this folder 777 and /tmp/rawlog/test
>777, but still nothing is there. The normal syslog of dovecot does not
>report any errors.
>
>[1]
>https://wiki2.dovecot.org/Debugging/Rawlog
>
If I add these changes for the rawlog binary but still nothing, also
having the home dir in userdb although the doveadm user -u test return
correct values.
service imap {
executable = imap postlogin
}
service postlogin {
executable = script-login -d rawlog
unix_listener postlogin {
}
}
Getting rawlogging working with dovecot 2.2
According to this manual[1] I just have to add these lines to my config,
not?
protocol imap {
rawlog_dir = /tmp/rawlog/%u
}
I should be seeing something in this /tmp/rawlog if I delete an message
via imap not? I have already made this folder 777 and /tmp/rawlog/test
777, but still nothing is there. The normal syslog of dovecot does not
report any errors.
[1]
https://wiki2.dovecot.org/Debugging/Rawlog
RE: read state not propagated
>Can you somehow confirm that the "read" status is not available
>for the 2nd user in dovecot indexes? Maybe use mail_log plugin to
> see when the flag changes occur?
Is it possible to configure this logging for only one users, maybe via
this special-userdb?
This would be sufficient not?
plugin {
mail_log_fields = flags
}
RE: Imap client with ~7500 imap cmds ~250/~500 read() syscall
>
>>
>>
>> I am playing a bit with the monitoring, and I have noticed there is
one
>> client of the chart with ~7500, while the others are between ~17 and
~60
>> (dovecot_user_num_cmds).
>>
>> I assume this is related to the client, or is it possible something
is
>> wrong on the server side? Is someone having also such numbers or is
this
>> really strange. Is it possible to limit these cmd's?
>
>Maybe you should look into what command(s) are being executed first?
>
>event_exporter cmd_export {
> format = json
> format_args = time-rfc3339
> transport = log
>}
>
>metric imap_command {
> event_name = imap_command_finished
> filter {
> user = anomalous
> }
> exporter = cmd_export
>}
>
Is there also a way to do this on dovecot 2.2?
Imap client with ~7500 imap cmds ~250/~500 read() syscall
I am playing a bit with the monitoring, and I have noticed there is one client of the chart with ~7500, while the others are between ~17 and ~60 (dovecot_user_num_cmds). I assume this is related to the client, or is it possible something is wrong on the server side? Is someone having also such numbers or is this really strange. Is it possible to limit these cmd's?
RE: Initial Replication
Hi David, Afaik you have to put the 'old' emails before replicating on both, because replicating only works on new arriving. I just migrated to a server on a different platform keeping same os and dovecot version. One namespace had mbox storage which on the new server became mdbox. For the mbox -> mdbox namespace I used this doveadm backup -f -n inbox -F /root/backup-accounts.txt tcp:mailXX.local:542 Test first with one account, depending on the permissions of the directories on the destination, dovecot will create all the files. I had a 2nd namespace that I needed to migrate that was already in mdbox. I decided the rsync that, because here it was 2 days vs 5 hours syncing. After the rsync I did again the backup on that namespace doveadm backup -f -n Archive -F /root/backup-accounts.txt tcp:mailXX.local:542 Then I ran a script counting messages in mailboxes of all users on both servers and a diff showed that everything was ok. I guess if you handle it like this you will not loose a message. ;) -Original Message- From: David Morsberger [mailto:da...@mmpcrofton.com] Sent: 29 December 2020 20:09 To: dovecot mailing list Subject: Initial Replication I searched the archives and do not see a question/answer for this. Should I perform a set of ordered steps after setting up replication? I have setup dovecot on both ends with identical config except for the remote hostname. Main concern is the existing server gets replicated to the new server. I want nothing from new server replicated to the old server. Risking "Trial and error" and "crossing fingers doesnt work with production emails Specifically: Do I need to setup the mail directories for the various virtual users on the remote? If so, do the directories need to empty? Should I initially do a manually one-way synch using doveadm from the existing server? If so, is there a way of turning off automatic replication while the manual one runs? Anything else? T David
RE: read state not propagated
> > >> >> I have migrated the inbox namespace from mbox to mdbox by doing a >> doveadm backup to the new server. Both servers are having CentOS7 + >> dovecot-2.2.36-6.el7_8.1.x86_64, both servers have identical >> configurations (new server has some different vsz_limits and stats >> enabled) >> >> Now a user complains that an imap mailbox he opens at the same time >> with a colleague does not propagate the 'read' state. Previously if >> his colleague read the message, it would show on his mailbox also 'read'. >> After the transfer it keeps being unread for him. >> >> Am I correct to assume that he should indeed get an updated to 'read'? >> Where should I look to fix this? What can be the cause of this? > >How are they accessing the mailbox? Same credentials? yes, mostly just 2nd account in apple mail client
read state not propagated
I have migrated the inbox namespace from mbox to mdbox by doing a doveadm backup to the new server. Both servers are having CentOS7 + dovecot-2.2.36-6.el7_8.1.x86_64, both servers have identical configurations (new server has some different vsz_limits and stats enabled) Now a user complains that an imap mailbox he opens at the same time with a colleague does not propagate the 'read' state. Previously if his colleague read the message, it would show on his mailbox also 'read'. After the transfer it keeps being unread for him. Am I correct to assume that he should indeed get an updated to 'read'? Where should I look to fix this? What can be the cause of this?
RE: doveadm backup -f -n inbox doing writes?
Hmmm, but I have made a few files with accounts to sync, if I use the list with the oldest least changing mailaccounts I get this spike during the doveadm backup. And if I issue the same command 30s later, the spike is back for the duration of the check. And since it was finished quite quickly, ran it for a third time, again the write spike. doveadm backup -n inbox -F /root/backup-old-60-accounts.txt tcp:mail04.local:542 [@mail]# rpm -qa | grep dovecot dovecot-pigeonhole-2.2.36-6.el7_8.1.x86_64 dovecot-2.2.36-6.el7_8.1.x86_64 -Original Message- From: Aki Tuomi [mailto:aki.tu...@open-xchange.com] Sent: 22 December 2020 09:52 To: Marc Roos; dovecot Subject: Re: doveadm backup -f -n inbox doing writes? > On 22/12/2020 00:12 Marc Roos wrote: > > > If I start a 'doveadm backup -f -n inbox' on the source server, why do > also disk writes increase on the source server (indexes mostly?) Probably because your indexes are not up to date? Aki
doveadm backup -f -n inbox doing writes?
If I start a 'doveadm backup -f -n inbox' on the source server, why do also disk writes increase on the source server (indexes mostly?)
RE: migration from 2.0.16
Oh interesting this imapc option. Is there any advantage of using that instead of what I am currently using doveadm backup -n inbox -F /root/backup-rest-2.txt tcp:mail04.local:542 Also any difference between pulling or pushing the messages? To check, I have made some script that outputs messages of mailboxes that I can compare between source and destination server. But this of course does not show if messages are 100% ok. [@ ~]# mailbox-ls.sh test size listing mailboxes of test: Archive messages=0 Archive/2011 messages=0 Archive/2012 messages=0 Archive/2013 messages=0 Archive/2014 messages=0 Archive/2015 messages=0 Archive/2016 messages=0 Archive/2017 messages=0 Archive/2018 messages=0 Archive/2019 messages=3500 Archive/Archive messages=1 Deleted Messages messages=28 Drafts messages=2 INBOX messages=1325 INBOX/test2 messages=3 Junk messages=2 Sent messages=1 Trash messages=132 -Original Message- From: Aki Tuomi [mailto:aki.tu...@open-xchange.com] Sent: 17 December 2020 10:16 To: Marc Roos; barbara; dovecot Subject: RE: migration from 2.0.16 I would recommend using dsync migration to get rid of mbox format. We no longer develop that format, and bugs are limited to reading mbox format. I would also recommend using master password / master user login with doveadm sync, and do the synchronization over imapc: to get the data safely migrated to your new system. You should use doveadm sync -u user backup -R imapc: on the new server to pull the data from old server. See https://wiki.dovecot.org/Migration/Dsync for more details. Aki > On 17/12/2020 11:08 Marc Roos wrote: > > > I would not choose centos 8 it has EOL < than centos7. IBM is pulling > the plug on the centos distribution, and makes it more or less a beta > for the rhel. Thus centos7 and then you have a few years to decide > what to choose. Enough to go to full containerized eg. ;) > > You do not need to rsync, dovecot can sync messages. I am just in the > process of migrating a server from a different network to a different > mailbox format. > > My approach was to create an 'archive' namespace on shared slower but > distributed storage so I do not have to move to much data. > > > > -Original Message- > From: Barbara M. [mailto:barb...@rfx.it] > Sent: 17 December 2020 01:47 > To: dovecot@dovecot.org > Subject: migration from 2.0.16 > > > I have an old server with CentOS 6.x and dovecot 2.0.16 (postfix-2.6.6 > and roundcube), that was an update from a 1.x many years ago ... > Users in /etc/passwd > Mailbox format mbox > some filtering via procmail > About 5.000 users, 1 TB data (/var/mail + /home/users) > > Obviously I am searching for a smooth upgrade path (with no or minimal > downtime and users problems) ;-) > > Until few days ago my idea was a CentOS 8.x new box with the standard > default packages (dovecot-2.3.8, postfix-3.3, ...). > Now this can be reconsidered. I suppose there will be a RockyLinux or > something equivalent but if there is a good reason I can consider > Debian or other OS if they have a decent EOL or some advantages. > > Anyway, the more relevant problem at the moment is collect info for > the best approch to have a smooth dovecot upgrade. > > My dream is the possibility to configure a new server and rsync the > data (/var/mail + /home/users), and, when the tests are satisfiable do > the final sync and swap the IP, but I suppose deleting the .imap > folders isn't a simple complete solution to compatibility problems. > > Any hints, links, experiences are appreciated. > > Thanks, B.
RE: migration from 2.0.16
I would not choose centos 8 it has EOL < than centos7. IBM is pulling the plug on the centos distribution, and makes it more or less a beta for the rhel. Thus centos7 and then you have a few years to decide what to choose. Enough to go to full containerized eg. ;) You do not need to rsync, dovecot can sync messages. I am just in the process of migrating a server from a different network to a different mailbox format. My approach was to create an 'archive' namespace on shared slower but distributed storage so I do not have to move to much data. -Original Message- From: Barbara M. [mailto:barb...@rfx.it] Sent: 17 December 2020 01:47 To: dovecot@dovecot.org Subject: migration from 2.0.16 I have an old server with CentOS 6.x and dovecot 2.0.16 (postfix-2.6.6 and roundcube), that was an update from a 1.x many years ago ... Users in /etc/passwd Mailbox format mbox some filtering via procmail About 5.000 users, 1 TB data (/var/mail + /home/users) Obviously I am searching for a smooth upgrade path (with no or minimal downtime and users problems) ;-) Until few days ago my idea was a CentOS 8.x new box with the standard default packages (dovecot-2.3.8, postfix-3.3, ...). Now this can be reconsidered. I suppose there will be a RockyLinux or something equivalent but if there is a good reason I can consider Debian or other OS if they have a decent EOL or some advantages. Anyway, the more relevant problem at the moment is collect info for the best approch to have a smooth dovecot upgrade. My dream is the possibility to configure a new server and rsync the data (/var/mail + /home/users), and, when the tests are satisfiable do the final sync and swap the IP, but I suppose deleting the .imap folders isn't a simple complete solution to compatibility problems. Any hints, links, experiences are appreciated. Thanks, B.
RE: Recovering expunged but not purged e-mails from mdbox with zlib compression
Could one use this "mail_location=mdbox_deleted:/path/to/mdbox" for lets say a 'recovery' namespace. Where users can copy their deleted messages from? -Original Message- From: Aki Tuomi [mailto:aki.tu...@open-xchange.com] Sent: Tuesday, November 10, 2020 7:16 PM To: Daniel Schütze; dovecot@dovecot.org Subject: Re: Recovering expunged but not purged e-mails from mdbox with zlib compression > On 10/11/2020 20:07 Daniel Schütze wrote: > > > Dear Aki > Thank you. Unfortunately I'm struggling to get the right syntax for > this as it looks like someone else was here too > https://dovecot.org/pipermail/dovecot/2018-July/112441.html > The location in my dovecot.conf is > > mdbox:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/% > %n This syntax is for accessing shared folders. You probably should try doveadm -Dv -o mail_location=mdbox_deleted:/path/to/mdbox fetch -u victim text ALL > I'm trying to fetch a message (for testing purposes now) based on it's guid as that is available from doveadm dump. > So my syntax is (based on the previous person who didn't get it to work). > > doveadm -o > "mail_location=mdbox_deleted:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:IN > DEXPVT=~/mdbox/shared/%%n" fetch "body" guid (msg.guid from doveadm > dump) > Doveadm does not support var expand, so this will not work. See above for syntax. > But that's getting no response. I've tried putting in the hard path in case there is any trouble with the %%h etc but that doesn't help either. > I've also tried the fetch with the subject of a known deleted message and also adding mailbox Drafts (as I know that's where it is or rather was) but to no avail. > Given doveadm dump gives msg.uid and no subject I'd prefer to use that in any case. > There are no error messages, I do with doveadm was a little more verbose to help over these bumps! > > Any chance of a helping hand? > > > > Aki > > > > Daniel Schütze > Director > > -- > > CWA International Ltd > 5th Floor, 42 Trinity Square > London > EC3N 4DJ > > (t) + 44 (0)20 7242 8444 > (e) d...@cwa.uk.com > (w) http://www.cwa.international/ > On 10/11/20 10:53, Aki Tuomi wrote: > > > You can use mdbox_deleted driver to access mails with refcount 0. > > See https://wiki2.dovecot.org/MailboxFormat/dbox > > > > Aki > > > > On 10.11.2020 12.42, Daniel Schütze wrote: > > > > > Yesterday I had to recover an e-mail which a user had deleted. If > > > I understand this correctly the message was expunged but not > > > purged (doveadm purge had not been run). > > > > > > This e-mail was clearly still in the mdbox stored with zlib > > > compression as I could tell using the doveadm dump command > > > (doveadm dump -t dbox filename). > > > > > > I could however not reveal the e-mail with the normal doveadm > > > fetch -u username "body" guid (guid from dump output) > > > > > > In the end I was able to recover the e-mail by cutting it out from > > > the mdbox and running gunzip over it, but this method was very > > > fiddly and would not have worked for a bulk job. I appreciate I > > > could have just deleted the index files and gotten all the deleted > > > messages back by having the index rebuilt but that would have been > > > the proverbial "hammer to crack a nut". > > > > > > Fortunately this e-mail had no attachment for me to worry about, > > > as those are also detached for sis by dovecot. > > > > > > Can anyone tell me of a quick and easy way to recover one or > > > multiple e-mails marked as expunged but not purged which are > > > stored in a mdbox with zlib compression? I'm sure I'll have to do > > > this again the future and my method wouldn't work with a folder! > > > > > > Clearly if I was not using zlib compression I could just have read > > > the contents of the mdbox without any complication. > > > > > > > > > -- > > > Daniel Schütze > > > Director > > > > > > -- > > > > > > CWA International Ltd > > > 5th Floor, 42 Trinity Square > > > London > > > EC3N 4DJ > > > > > > (t) + 44 (0)20 7242 8444 > > > (e) d...@cwa.uk.com > > > (w) http://www.cwa.international/ > > >
RE: Looking for a guide to collect all e-mail from the ISP mail server
> Btw., why is an open port 25 evil if the MTA is configured correctly? > Can you elaborate, please? He does not know, that is why he assumes this. He first needs to aquire some basic principles and learn, as he wrote.
RE: Looking for a guide to collect all e-mail from the ISP mail server
> That's nonsense. I will give one example: Airbus, the European aerospace corporation, uses Google Workspace. What do they store there? That is the question, maybe some irrelevant data, I doubt if they store CAD drawings online or data that is protected by GDPR legislation. And even when, are you going to burn books, when Airbus is going to burn books?
RE: Looking for a guide to collect all e-mail from the ISP mail server
> It's hard to imagine anyone being that dumb, but then this society has been surprising me a lot in recent years. If I tell some woman in the store that she is about to buy an energy drink promoted by/having a picture of a convicted rapist. They look at me weird and the most stupid response I got was 'but I am not buying it for myself'.
RE: Looking for a guide to collect all e-mail from the ISP mail server
> Yes, you all want me to open ports. I'm sorry guys, but I won't budge: > 1) Opening a port means reconfiguring the firewall. You may find it funny, but some non-profits have no firewall, just a standard ADSL router. The ones that the telecom company provides often has no IP filtering abilities. Read your router manual, you can easily only port forward from a single or multiple ips to your local > 2) I will not expose an SMTP server to the outside word. I will not install in, or advise to, a small business a piece of software that craves for attention The problem is your knowledge is limited, and therefore draw incorrect conclusions. So maybe try and find someone that has more knowledge in your group, or ask around in your charity. > 3) Of course I can ask the current ISP. And they may comply. But how about the next one? What next one? You should stick with your ISP for years, I have. > 4) Of course I can filter my provider's IP in some Linux firewall. But then the provider will change its setup and won't tell me. Or I will not have time to modify the configuration. Or the next person will not have time just this week. These things do not change. I did not change my mail ip's the last 10 years or so. I guess only 'hillbillies' that hop around from supplier to supplier to cut a few dollars a month do this.
RE: Looking for a guide to collect all e-mail from the ISP mail server
> I too would strongly advise you to use Google Workspace (the recent new name for G Suite, previously known as Google Apps). > It's cheap, very reliable, and has all features you can dream of, including an autoresponder. > It's unrealistic to think that it's possible to beat a service that costs a mere USD 6 / user / month (and is free for nonprofits!). I would not advice any company that is continuously being fined for breaking the law.
RE: Looking for a guide to collect all e-mail from the ISP mail server
> When you are a small business or a volunteer-run club or charity, you don't ask your provider. > You have no leverage. You may not even be able to change provider so easily. Just ask, I will bet they do it. They do not need to configure that much even I think. By default smtp servers are queueing mail for down hosts.
RE: SV: SV: Looking for a guide to collect all e-mail from the ISP mail server
I know. I am not stating this. -Original Message- From: lists [mailto:li...@lazygranch.com] Sent: Monday, October 26, 2020 3:17 PM To: dovecot Subject: Re: SV: SV: Looking for a guide to collect all e-mail from the ISP mail server As I previously stated the reverse pointer does not have to match your domain. Suppose you ran a hosting company called host.com. Suppose you had clients client1.com and client2.com. This requires virtual mailboxes. That is one domain, host.com provides email services for client1.com and client2.com. Most servers would just have a reverse pointer to host.com. Original Message From: m.r...@f1-outsourcing.eu Sent: October 26, 2020 7:04 AM To: build+dove...@de-korte.org; dovecot@dovecot.org Subject: RE: SV: SV: Looking for a guide to collect all e-mail from the ISP mail server > and forward- and reverse DNS records for your mailserver match. do even googles ips confirm to this standard?
RE: Looking for a guide to collect all e-mail from the ISP mail server
> Besides, the way you suggest means opening a SMTP port to the outside world. A security risk and more work at the firewall etc. You can just allow some ip addresses of your provider to connect, not? Nothing outside world.
RE: SV: SV: Looking for a guide to collect all e-mail from the ISP mail server
> and forward- and reverse DNS records for your mailserver match. do even googles ips confirm to this standard?
RE: Looking for a guide to collect all e-mail from the ISP mail server
> email does not get silently dropped or moved to spam when working with gmail. Gmail is dropping email on purpose?
RE: Looking for a guide to collect all e-mail from the ISP mail server
you should ask your ip provider to set a proper reverse lookup for you. If I would get a lot of spam from upcloud.host ips, I would also consider blocking upcloud.host reverse dns lookups. If it is your ip, it is an easy request to have it changed. -Original Message- From: Sami Ketola [mailto:s...@ketola.io] Sent: Monday, October 26, 2020 11:22 AM To: lists Cc: Arjen de Korte; Dovecot Mailing List Subject: Re: Looking for a guide to collect all e-mail from the ISP mail server On 26. Oct 2020, at 11.36, lists wrote: Actually the reverse pointer doesn't have to match. In fact this is impossible if you are setting up virtual accounts on one server for different domains. You just need to have a reverse pointer. Most email servers look to seen if the reverse pointer has a "dyn" in it and blocks those. Also your own email server is not behaving nicely: : host lazygranch.com[198.199.119.111] said: 500 5.7.1 <83-136-254-93.uk-lon1.upcloud.host[83.136.254.93]>: Client host rejected: eat a bag of dicks (in reply to RCPT TO command) and for that reason I have blacklisted you from any help requests. You may do the same whatever you are telling me to do. Sami
RE: Looking for a guide to collect all e-mail from the ISP mail server
There was just a recent discussion on the spamassassin mailing list where also amazon was mentioned, and from what I can recollect and my experience, outgoing amazon mail has a bad reputation. So if you want host your vm somewhere, choose something that is not cheap and not big. Spammers more most likely to choose cheap. -Original Message- Cc: N; dovecot@dovecot.org Subject: Re: Looking for a guide to collect all e-mail from the ISP mail server 26. 10. 2020 v 12:15, R. Diez : > > I would be happy to take a pre-packaged mail server solution like iRedMail which includes RoundCube or whatever. Have a look at Mailcow too, it comes with almost everything. Ive been running it for a year now, after many years of using a self-assembled stack, and its a bliss. I have it coupled with Amazon SES for some domains that run mailing lists. Thats a cheap option if you want to offload the sender reputation problem to someone else. -F
RE: SV: Looking for a guide to collect all e-mail from the ISP mail server
> and also the problem is that gmail imposes heavy spam filters and "reputation blocks" > meaning smaller providers with low email volumes, are put in the spam folder, even if > they never send spam, just because their email volume is so low (ergo, they must > prove they don't spam before getting out of ispam folder) How do you know that?
RE: SV: Looking for a guide to collect all e-mail from the ISP mail server
Amen to that! -Original Message- From: lists [mailto:li...@lazygranch.com] Sent: Monday, October 26, 2020 7:09 AM To: Dovecot Mailing List Subject: Re: SV: Looking for a guide to collect all e-mail from the ISP mail server Good luck with all that coding. I have four years now of running my own email server. Zero hacks. I keep the attack surface to a minimum. Less is more. One thing you don't want to do is write your own code. This stuff is always way harder than you think. Worse yet you run alpha generation code because you are the only one using it. All software has bugs. What you need is a mass of users flogging the code and finding the bugs. Now if you do use a browser, you have to deal with leaks, bugs, possible process interaction if more than one tab is open, and possibly browser extensions hacks if extensions are used. Count me out. And did you miss the part where I was hacked via RoundCube?
RE: Looking for a guide to collect all e-mail from the ISP mail server
Maybe get something like Zimbra, such solutions also have support that
you can buy when you need it or don't have time (I guess).
-Original Message-
From: R. Diez [mailto:rdiezmail-2...@yahoo.de]
Sent: Sunday, October 25, 2020 6:57 PM
To: dovecot@dovecot.org
Subject: Looking for a guide to collect all e-mail from the ISP mail
server
Hi all:
I am evaluating mail server solutions for a small business. The trouble
is, I am only a part-time admin and a newbie to mail servers.
Most guides I have seen are rather unrealistic: they encourage you to
expose your e-mail server to the Internet, and hope that you have the
resources
to keep it patched up.
I would rather have an internal mail server that collects e-mails from a
standard ISP mail server. It is like the old "POP3 Connector" that came
with
Microsoft Exchange. Sometimes, there is a mailbox per user on the ISP,
and a corresponding one on the local server. Other times, there is a
single
"catch all" or "multidrop" mailbox on the ISP.
Users can still access their internal mailboxes from outside through an
OpenVPN connection. The goal is that only VPN, and perhaps SSH, are
accessible from the outside. We do not need to arrange any special SMTP
configuration with the ISP either.
This kind of mail server setup is rather different to the standard
configuration. You do not normally need you own antivirus and spam
filter, and you
do not need to configure SSL certificates, MX or SPF DNS records. Most
ISP handle that correctly and economically. Internal e-mail does not
leave
your LAN, and your internal SMTP server is just a relay for the external
ISP SMTP server.
Furthermore, most guides do not explain how to setup an autoresponder
("I am on holiday until xxx") so that users can enable theirs with the
mouse.
Editing configuration files over SSH is not really an option for normal
users. This detail is important because it could be the only thing I
need
above standard e-mail. Further groupware features can be seen as nice
but ultimately unnecessary luxury, and a basic shared calendar can be
accomplished with a separate server like https://radicale.org/ and a
calendar client like one built into Thunderbird. Hopefully, that is all
I would
need for a small business.
Can anyone point me to the kind of guide I need? Failing that, I would
need information or examples about using fetchmail, getmail or similar
software
with Dovecot. Good or bad experiences from you guys would also help.
Each of those tools has a detailed man page, but there are many options
and ways with different advantages and disadvantages. I would need a
simpler
guide to get started.
I am aware that there are pre-packaged mail server solutions that would
perhaps bring an easy-to-use autoresponder, but I haven't seen one yet
that
where you could tick a box like "this server is only internal and
collects mail from the ISP server" during installation. Nor have I seen
instructions
about reconfiguring the mail server for my ISP mail scenario.
I am prepared to learn more and write my own Perl scripts and/or
installation guide, but it would be stupid to waste time if something
easy already
exists. After all, the setup I am describing (external ISP mail server
+ internal mail server) is not so weird.
Thanks in advance,
rdiez
RE: forwarding email with sieve of spf domains
However as far as I can determine from my forward test is that, the from header is currently from the original message and not from the forwarder. -Original Message- From: Scott Q. [mailto:qm...@top-consulting.net] Sent: Sunday, October 25, 2020 5:01 PM To: Marc Roos; dovecot Subject: Re: forwarding email with sieve of spf domains There's no ambiguity here, if you send a message, you are the sender. The envelope from should be yours. On Sunday, 25/10/2020 at 11:48 Marc Roos wrote: Say someone has setup spf for his domain and sends an email to a user that has in roundcube enabled the sieve forward. If the message is forwarded without altering the message headers, this could result in a message being blocked or not relayed, because sending hosts ip, is not in the spf of the from: domain. Possible solutions are: - add option if enabled, it replaces the From: with that of the email address of the sieve user. (Maybe move the original sender to the Reply-To header? Maybe exception for 'internal' forward?) - Upon processing the message, check the spf records, if they are enforced, do the above, otherwise do nothing. https://tools.ietf.org/html/rfc5228#section-4.2
forwarding email with sieve of spf domains
Say someone has setup spf for his domain and sends an email to a user that has in roundcube enabled the sieve forward. If the message is forwarded without altering the message headers, this could result in a message being blocked or not relayed, because sending hosts ip, is not in the spf of the from: domain. Possible solutions are: - add option if enabled, it replaces the From: with that of the email address of the sieve user. (Maybe move the original sender to the Reply-To header? Maybe exception for 'internal' forward?) - Upon processing the message, check the spf records, if they are enforced, do the above, otherwise do nothing. https://tools.ietf.org/html/rfc5228#section-4.2
Testing with imaptest to non existing folder makes imap service crash
FWIIW I had this message still in drafts /usr/bin/imaptest - append=100,0 logout=0 host=xxx port=143 user=xxx pass=xxx seed=100 secs=1 clients=1 mbox=64kb.mbox box=inbox/test msgs=10 And inbox/test does not exist, results in: Fatal: master: service(imap): child 20282 killed with signal 11 (core dumped)
RE: How to move/reorganise existing e-mails to yearly subfolders
> >First of all, thanks for your answer. > >> What is the problem with having huge online mailboxes? Just choose a > good european provider that has encryption all the way through to their> storage platform. > >We already have a European ISP with a standard e-mail server. I wanted to keep our own mail server on premises, so that it is not exposed on >the Internet. The current server fetches (and removes) all e-mails from the ISP. That has many advantages, for example, internal e-mail >still works in case of an Internet outage. Use batched smtp, you will never miss an email, and just keep your on premises solution. You can do a lot with smtp configurations. You can even have email delivered on a 2nd location. >If I wanted to change the setup, I would have to start evaluating such an "encryption all the way through to their storage platform" >feature. I am not sure that it is worth the effort. In any case, that sounds like a limiting factor when choosing another ISP, in case the >current one starts making trouble. > > >> I had exactly the same idea about migrating. You have to think twice > about moving emails around of users. They do not like it ;) > > [...] > >I don't really want to do that, that's why I wrote "If I set a mailbox size limit, users will have to delete old mails by themselves". > >I do not know much about the legal aspects, but in case we need to keep all e-mails for legal data retention requirements, I would like to >store those e-mails separately, so that if a user deletes it, the original e-mail is still archived somewhere else. > >That is why I mentioned the Postfix's BCC feature. The idea is that you have a separate mailbox where a separate copy of all e-mails to and >from all users land. That is the separate mailbox where I wanted to reorganise e-mails by date, in order to archive the e-mails in smaller >chunks on a yearly basis. Those e-mails do not need to be online after all. Chances are, they will never be needed anyway. That should be simple to realize, just to folders to archive! You do not need BCC to have copies delivered to 2nd account. You also have to think about outgoing mail. Duplicate those as well. And force spf, so users cannot send message via any other outgoing mailservers. > >> I have created an 'archive' environment on a distributed filesystem, and it >> takes me quite a lot of persuading to have people (or allow me) to move >> messages from common Sent and Inbox mailboxes to the Archive namespace > > [...] > >I am actually a newbie in mail service matters, but my guess is that there is no amount of persuasion that could possibly help. You have to >set a hard limit per mailbox and let the users deal with it, don't you? Otherwise, sooner or later the server will overload. Or I would need >to become a full-time e-mail server admin, which is not an option either! You have to explain to people the advantages, eg when adding a phone, it does not download sync a huge inbox or Sent folder. We are not having any limit's. With current day solutions, I would say there is no need to. You can also outsource the work on your on premisses vm ;) >I am actually a friend of having 2 backup disks that rotate, where one is always physically off premises, and offline. But I wonder how I >could keep the backups encrypted and synchronised with 2 rotating disks. Maybe Veracrypt + rsync. Sounds sufficient, luks encryption is also fine. >I am hoping that the amount of big attachments in all incoming and outgoing mail still fits in normal external USB 3.0 disks. Or at least a >few years' worth of it per disk. But I still would not want to have say 1 TB of mail data online. That would make the VM unmanageable for >part-time sysadmins like me. At this point I do not see a need why you need to have any data online. The online servers just need to be properly configured for your on-premisses servers. > >> [@~]# mailbox-ls.sh testtest size > > [...] >> I would not trust anyone else's programming with my >> users email, you should also not. > >I am not sure that I would trust my own e-mail server programming abilities either. 8-) > >If you have written such scripts, perhaps you could point me to some example scripts that I could use as a starting point for such e-mail >reorganisation tasks? > > > > [...] >> But when I migrate to mdbox this is not necessary anymore. > >I am not sure that I would trust a file format where the indexes cannot be rebuilt if they become corrupt. If I need an advanced format for >search performance reasons, I would probably consider an SQL-based backend then. Currently I have many inbox'es and other mbox files of >25GB that is not sustainable. maildir with lots of files is also not an option. > >> I do not like the sound of "Postfix BCC feature", I use sendmail and I >> can duplicate messages with that, without altering anything in them. >
RE: How to move/reorganise existing e-mails to yearly subfolders
What is the problem with having huge online mailboxes? Just choose a good european provider that has encryption all the way through to their storage platform. I had exactly the same idea about migrating. You have to think twice about moving emails around of users. They do not like it ;) I have created an 'archive' environment on a distributed filesystem, and it takes me quite a lot of persuading to have people (or allow me) to move messages from common Sent and Inbox mailboxes to the Archive namespace (was not able to use the alternative storage option that dbox allows). Side note is that if you do archive these emails, most users do not even notice you have done this. I ended up creating a script and a webmail plugin for users to enable autoarchiving, which creates something like this. [@~]# mailbox-ls.sh testtest size listing mailboxes of testtest: Archive messages=0 Archive/2011 messages=0 Archive/2012 messages=0 Archive/2013 messages=0 Archive/2014 messages=0 Archive/2015 messages=0 Archive/2016 messages=0 Archive/2017 messages=0 Archive/2018 messages=0 Archive/2019 messages=3500 Archive/Archive messages=1 Deleted Messages messages=16 Drafts messages=2 INBOX messages=1286 INBOX/test2 messages=11 Junk messages=2 Sent messages=0 Trash messages=132 A cron job checks then if the script has run for the user this year, if not it starts archiving in the down hours otherwise it runs again in next year's 2nd quarter. What ever you choose, move messages with "doveadm move". I would not trust anyone else's programming with my users email, you should also not. Read the man pages on tools that work via imap, if they change headers, users are going to download all their messages again. I was thinking of splitting up folders like eg inbox/sales to Archive/2016/sales, Archive/2017/sales. But when I migrate to mdbox this is not necessary anymore. I do not like the sound of "Postfix BCC feature", I use sendmail and I can duplicate messages with that, without altering anything in them. You do not want anything that changes your data. If your provider uses this mdbox format (maybe others support this also) then messages a user deletes, are not even removed from the server until "doveadm purge" is given ;) -Original Message- From: R. Diez [mailto:rdiezmail-2...@yahoo.de] Sent: Monday, October 19, 2020 3:49 PM To: dovecot@dovecot.org Subject: How to move/reorganise existing e-mails to yearly subfolders Hi all: I am new to e-mail servers and I am evaluating Dovecot. Not really the best combination. 8-) I am trying to find a balance between legal data retention requirements and online mailbox size. I do not want huge online mailboxes, as doing offline, rotating data backups could then take forever (among other reasons). I would rather avoid online (cloud) backups (data protection etc.). If I set a mailbox size limit, users will have to delete old mails by themselves. Or I could somehow script the deletion of attachments from old e-mails, as attachments are usually the main cause of huge mailboxes. Incidentally, can anyone point me to an easy way to achieve this? Preferably over with IMAP, otherwise with Dovecot tools. With regards to legal data retention (which I am no expert about either), I thought I could use some Postfix BCC feature I heard about in order to copy all incoming and outgoing e-mails to a single "data retention" mailbox. Or maybe several of them. I could then archive e-mails from that mailbox on a yearly basis. I would like to automatically organise e-mails inside that mailbox into subfolders like this: 2019/alice 2019/bob 2020/alice 2020/bob That is: [year]/[username] With such a folder structure, it is easier to see what is going on. Is there a tool that can reorganise existing e-mails into such a folder structure? I found some tools on the Internet to backup and export mails from IMAP to IMAP or maildir destinations. But I could not find a tool that just reorganises (moves) e-mails in such a manner inside an existing mailbox, maybe with a user-defined pattern for the destination folders. I guess moving e-mails around on the same mailbox would be much faster than exporting and reimporting them in some clever way. I could always write a Perl script, but that takes time. Such a tool may already exist. Or perhaps somebody could mention a similar, good-written script I can use as a starting point. I am sure there are many small gotchas to avoid. At the moment, I am only confident with Perl and Java. Maybe JavaScript. It would be best to reorganise the e-mails over IMAP. This way, I am independent of the e-mail server. But a Dovecot-specific solution would also be helpful. I could use such a reorganisation tool not just for archiving or data retention purposes, but to reorganise other mailboxes too, like my personal mailbox. I would rather have a script. Clicking around in Thunderbird does not scale. I have seen that you
RE: Using NFS to extend local email storage
What about moving the mailserver on-premises, buy a big UPS and do some batched smtp on the cloud, so if you are down, you will not lose incoming mails? -Original Message- From: Maciej Kokociski [mailto:maciej.kokocin...@hands.pl] Sent: Sunday, October 18, 2020 7:51 PM To: dovecot@dovecot.org Subject: Using NFS to extend local email storage We are running our IMAP mail server on a VPS in the cloud (dovecot+exim+horde webmail), and we are using our on-premises NAS for daily backups through custom rsync scripts (which work well with maildir). There is a lot of storage available on the NAS, while not so much on the VPS. Since, we often get downtime on-premises due to power outages we cannot move the mail server on-premises (though the outages do not interfere too much with the backups). Additional storage in the cloud is at a premium as we would like to avoid further increasing costs. Now, my idea is to set up an NFS volume on the NAS, and configure the cloud-based server as the client. I would like to store only the most recent emails on the main server, and keep the whole data set on the NAS. Naturally I have read the dovecot wiki concerning NFS, and I have browsed this mailing list's archive for answers, but I still have many concerns. Ideally, I would like to use as much storage as possible from the main server, so that only really old emails need to be fetched over the network if requested over IMAP. Also I would like the search function to work regardless whether the NAS is available or not. I wonder if anybody tried a similar configuration before. My concerns are: - should I use NFS volume directly and configure FS-Cache, or should I configure a cache volume separately using e.g. bcache or some other tool? - how to configure the cache to keep as many files and metadata locally? - is it possible to fine tune the cache so it always keeps all the newly created/modified files locally? - is it possible to tune the cache so it keeps at least n files for each user (each maildir)? - will it work fine if I put NFS traffic into an ssh tunnel for security? - will dovecot freeze/crash when the NFS volume disconnects or is temporarily unavailable, what will actually happen in such circumstances? - will the search function work when the NFS volume is unavailable? (I also plan to use solr in the future) - what happens when the main server crashes and then recovers from the NFS share? is the share guaranteed to be consistent? I hope that I have not asked too many questions for a single message... ;) I would really appreciate any help, because I am rather a newbie in dovecot administration, and it overwhelms me. Perhaps, what I am trying to achieve is not the way to go. I am open to suggestions. Best regards, Maciej
backup of namespace, is still looking at (touching?) other namespace?
When I am doing this: doveadm backup -f -n inbox -F /root/backup-accounts.txt tcp:mailxx.local:542 I am getting an error on the distributed storage, which I exactly did not wanted to be touched. doveadm(testaccount): Error: remote(mailxx.local:542): User initialization failed: Namespace 'Archive/': stat(/home/mail-archive/testaccount/Archive/mailboxes) failed: Permission denied (euid=xxx(testaccount) egid=xx(x) missing +x perm: /home/mail-archive, dir owned by 0:0 mode=0700) doveadm(testaccount): Error: remote(mailxx.local:542): dsync-server: User init failed doveadm(testaccount): Error: Failed to start remote dsync-server command: Remote exit_code=75
RE: Procmail with Dovecot
No need for user shell access. Before switching to sieve, I made some email interface where users could turn on 'services' by sending an email to themselves. -Original Message- To: dovecot@dovecot.org Subject: Re: Procmail with Dovecot On 13.10.20 11:08, Dan Egli wrote: > Hey folks, here's a question. I want to enable procmail for the users > so that they can have their incoming messages sorted info various > folders and such regardless of MUA access or web access. I know I can > set procmail to deliver to a maildir, but wouldn't that screw up > dovecot's indexes? If it would, what can I do to ensure that the indexes stay correct? Wouldn't it be better if you enabled Sieve and use that instead? procmail needs shell access (or a very sophisticated web-upload thingy to weed out any shell escapes from procmail) to work and a user can easily create loops or break their mail reception completely. Sieve at least makes sure the syntax is correct before installing a ruleset. Grüße, Sven.
RE: Preparing for replication: dsync-local(testaccount): Panic: file mbox-lock.c
Maybe a bug? I deleted the whole tree on the destination server, ran doveadm sync -D -1 -n inbox -u testacc tcp:mailxx.local:542 Only the directory structure of mdbox was created, no files. 2nd time I run: doveadm sync -D -1 -n inbox -u testacc tcp:mailxx.local:542 The files are created in storage. What makes the m.XXX reset? I thought this would start at m.001 after deleting the whole home dir [1] [@storage]# ls -cs1t 3928 m.737 4032 m.734 5868 m.735 5868 m.736 4100 m.733 4100 m.717 4100 m.718 4100 m.719 4100 m.720 4100 m.721 4100 m.722 4100 m.723 4100 m.724 4100 m.725 4100 m.726 4100 m.727 4100 m.728 4100 m.729 4100 m.730 4100 m.731 4100 m.732 4052 m.716 [2] [@]# ls -cs1t mdbox/storage/ total 93448 3928 m.759 4032 m.756 4100 m.741 4100 m.742 4100 m.743 4100 m.744 4100 m.745 4100 m.746 4100 m.747 4100 m.748 4100 m.749 4100 m.750 4100 m.751 4100 m.752 4100 m.753 4100 m.754 4100 m.755 5868 m.757 5868 m.758 4100 m.739 4100 m.740 4052 m.738 -Original Message- Subject: Re: Preparing for replication: dsync-local(testaccount): Panic: file mbox-lock.c Replication is not supported with mbox format. You can only do unidirectional sync out of mbox. Aki I am preparing a bit for setting up replication. However when I manually try to dsync an account, the first time I execute this[1] command it seems to be ok. The 2nd time I am getting this error[2]. If I add -1 (one way syncing) the error disappears. Does this mean I will have problems with setting up replication between these two servers? [1] [@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 [2] [@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 dsync-local(testaccount): Panic: file mbox-lock.c: line 799 (mbox_lock): assertion failed: (lock_type == F_RDLCK || mbox->mbox_lock_type != F_RDLCK) dsync-local(testaccount): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xa192e) [0x7fd4f827992e] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fd4f827999a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7fd4f8209257] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mbox_lock+0xef) [0x7fd4f858feef] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mbox_save_begin+0x658) [0x7fd4f8591958] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x83) [0x7fd4f855a573] -> doveadm(+0x4a6a3) [0x55e5ae8276a3] -> doveadm(dsync_mailbox_import_mail+0xca) [0x55e5ae82a9fa] -> doveadm(dsync_brain_sync_mails+0xb3) [0x55e5ae824963] -> doveadm(dsync_brain_run+0x551) [0x55e5ae820551] -> doveadm(+0x438e0) [0x55e5ae8208e0] -> doveadm(+0x5899f) [0x55e5ae83599f] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fd4f828f672] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x1 0f) [0x7fd4f8290d5f] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fd4f828f70c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fd4f828f8c8] -> doveadm(+0x297c9) [0x55e5ae8067c9] -> doveadm(+0x2bdcc) [0x55e5ae808dcc] -> doveadm(+0x2c9ca) [0x55e5ae8099ca] -> doveadm(doveadm_mail_try_run+0x215) [0x55e5ae80a325] -> doveadm(main+0x46a) [0x55e5ae7f949a] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd4f7e2c555] -> doveadm(+0x1c605) [0x55e5ae7f9605] Aborted CentOS Linux release 7.8.2003 (Core) dovecot-pigeonhole-2.2.36-6.el7_8.1.x86_64 dovecot-2.2.36-6.el7_8.1.x86_64
Doveadm-Sync for sieve / control
How to dsync sieve rules and maybe other necessary files like in control?
Preparing for replication: dsync-local(testaccount): Panic: file mbox-lock.c
I am preparing a bit for setting up replication. However when I manually try to dsync an account, the first time I execute this[1] command it seems to be ok. The 2nd time I am getting this error[2]. If I add -1 (one way syncing) the error disappears. Does this mean I will have problems with setting up replication between these two servers? [1] [@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 [2] [@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 dsync-local(testaccount): Panic: file mbox-lock.c: line 799 (mbox_lock): assertion failed: (lock_type == F_RDLCK || mbox->mbox_lock_type != F_RDLCK) dsync-local(testaccount): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xa192e) [0x7fd4f827992e] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fd4f827999a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7fd4f8209257] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mbox_lock+0xef) [0x7fd4f858feef] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mbox_save_begin+0x658) [0x7fd4f8591958] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x83) [0x7fd4f855a573] -> doveadm(+0x4a6a3) [0x55e5ae8276a3] -> doveadm(dsync_mailbox_import_mail+0xca) [0x55e5ae82a9fa] -> doveadm(dsync_brain_sync_mails+0xb3) [0x55e5ae824963] -> doveadm(dsync_brain_run+0x551) [0x55e5ae820551] -> doveadm(+0x438e0) [0x55e5ae8208e0] -> doveadm(+0x5899f) [0x55e5ae83599f] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7fd4f828f672] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) [0x7fd4f8290d5f] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7fd4f828f70c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7fd4f828f8c8] -> doveadm(+0x297c9) [0x55e5ae8067c9] -> doveadm(+0x2bdcc) [0x55e5ae808dcc] -> doveadm(+0x2c9ca) [0x55e5ae8099ca] -> doveadm(doveadm_mail_try_run+0x215) [0x55e5ae80a325] -> doveadm(main+0x46a) [0x55e5ae7f949a] -> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd4f7e2c555] -> doveadm(+0x1c605) [0x55e5ae7f9605] Aborted CentOS Linux release 7.8.2003 (Core) dovecot-pigeonhole-2.2.36-6.el7_8.1.x86_64 dovecot-2.2.36-6.el7_8.1.x86_64
RE: Feature request.
Does a dovecot reload not do that? For a webserver I just set a flag and a cron job. Whenever I put a new cert, the webserver reloads. -Original Message- To: Rogier Wolff; dovecot@dovecot.org Subject: Re: Feature request. > On 09/10/2020 11:16 Rogier Wolff wrote: > > > Hi, > > I get my Email from my own SMTP server on the internet using > "fetchmail". Some time ago I did the smart thing and configured > dovecot to use SSL and the letsencrypt certificate that automatically > renews. > > Wel. a few days ago my certificate expired and the fetchmail > deamon running in the background had nowhere to complain. So I didn't > notice. > > It turns out that dovecot had been running uninterrupted since august > 13th, the certificate was renewed on september 7th and I suspect it > expired on october 7th. > > So Feature request: check the expiry date on the SSL certificate > as it is being loaded and check for a new certificate if it HAS > expired. > > If you worry about performance, this could be done where: > > TLS handshaking: SSL_accept() failed: error:14094415:SSL > routines:ssl3_read_bytes:sslv3 alert certificate expired: SSL alert > number 45 > > is reported. That would mean that ONE client will once get the error > before dovecot fixes it. My personal fix is to restart dovecot once a > week from now on. > > I might be running an older version: > > # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version > 0.4.21 (92477967) # OS: Linux 4.15.0-34-generic x86_64 Ubuntu 18.04.5 > LTS > > if it has already been fixed, please accept my apologies. > > Roger. > That is indeed old version, but no, there is no automatic certificate reloading in Dovecot yet. This has been suggested before, and we have it in our internal issue tracker, but unfortunately I can't promise any date when it will be done. Aki
RE: Providers running dovecot?
>Le dimanche 27 septembre 2020 à 16:30 +0200, Olivier Cailloux a écrit : >> Dear list, >> >> I am looking for providers of free e-mail addresses known to run >> Dovecot (or a variant thereof) for IMAP access. I need only a few MB >> storage space and no particular features beyond SMTP and IMAP. >> >> The reason I ask is that Dovecot is known to implement the IMAP spec >> quite respectfully, and I am writing a software which uses IMAP >> search >> (so I would suggest my users to register an e-mail to a provider >> implementing correctly the IMAP Search specifications, to reduce the >> probability of bugs). (More details here: >> https://www.webhostingtalk.com/showthread.php?t=1821627 >> .) >> > > > >These private offers are very kind, but my question was more about >finding a provider who offers this access as a normal service, not as >some special favor to me. That’s because I want to recommend this >provider to the users of a software I am developing. And I do not >expect my users will agree to pay some fee (even a low fee) to register There is no such thing as free. If you do not pay anything, you know you are the product. >for an e-mail address just to use my software, so I’d recommend only a >provider who gives starter plans for free. (Of course these users in >turn would perhaps then upgrade their plan if they want to.) I don't think providers would be very willing (understatement) to install your software on their platform that services their other clients. I assume this is server side, since you enquire about dovecot. >So far I didn’t find a service provider providing free e-mail accounts >(similar to GMail, Yahoo, …) and using Dovecot, which I find very >surprising, as I thought some of these big names, or at least some >smaller ones that I do not know, would use Dovecot. > T-mobile uses dovecot, find t-mobile users ;)
RE: SV: How to Modify Message and add more Attachments
> >Thats because in your example the data is sent outside the facility to a third party (in this case, wetransfer/outlook) And wetransfer/outlook is operated in third countries, which can cause GDPR problems as the legal protection for the data disappears. > That is just a part. We had to sign such agreement between companies in the same country, city even. Data is not even leaving the country. Putting personal data at a third party requires a processing agreement. >The OP were asking about a solution which modifies email which have already been received in a local, secure facility to add the voice mail to locally stored messages. >Thats not prohibited. That has not been questioned, sending that data to google is being questioned. >Imagine if the OP has a SIP server and email server inside the same physical machine. Do you really think it would be prohibited to move a file from "asterisk/vm" to "var/spool/mail/"? No because it belongs to the expected necessary processing activities of a voip provider. This voip provider cannot just send these files to facebook that is easy to understand. So you can not send these files to google as well. Does not matter if they have some fancy AD processing api. >The security for the data is the same regardless of which format is used. Obviously
RE: How to Modify Message and add more Attachments
I have clients that process personal data and they even need to have 'special' processing agreements with companies like wetransfer and outlook.com. I had to sign also such agreement and prepare a vm for hot/cold data encryption for processing personal data. If someone leaves a voice mail message, he does not expect that this is going to be send to a third party. I think this expectation causes the gdpr 'by default' highest privacy/security of personal data protection to be applicable. Lots of companies are being fined currently for breaching gdpr, small, large, international even nation governmental organisations. Better check this. -Original Message- Subject: RE: How to Modify Message and add more Attachments Can you elaborate on the concern? -Original Message- From: Marc Roos Sent: Tuesday, October 6, 2020 4:17 PM To: dovecot ; Mrinal Sharma Subject: RE: How to Modify Message and add more Attachments CAUTION - EXTERNAL EMAIL This email originated from outside of Smith Micro Software. Do not click links or open attachments unless you recognize the sender and know the content is safe. If are processing Europeans voice mail you have to check if that is even allowed, could be a problem with GDPR legislation. -Original Message- Subject: RE: How to Modify Message and add more Attachments Thanks, am planning to use Google's Speech-to-Text. -Original Message- Sent: Tuesday, October 6, 2020 3:39 PM To: dovecot ; Mrinal Sharma Subject: RE: How to Modify Message and add more Attachments CAUTION - EXTERNAL EMAIL This email originated from outside of Smith Micro Software. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hmmm, that does not sound nice storing files as email. Maybe use document database? Look at this[1], see if it is possible to use the rados plugin to store files directly as objects? What are you using for speech to text? [1] https://github.com/ceph-dovecot/dovecot-ceph-plugin -Original Message- To: dovecot@dovecot.org Subject: How to Modify Message and add more Attachments Hello Everyone, I am working on a product in which we are planning to store voice messages in Dovecot sent by a user to another user. The message would be stored as an email with .wav attachment. Once the Voice message is received, it may get Transcribed. The message can be further be processed and more information can be added to the message later. The original plan was to Modify the email and add New information as attachments to same message. As I understood, messages stored in Dovecot are immutable. What is the best option to achieve this functionality? Thanks, Mrinal
RE: How to Modify Message and add more Attachments
If are processing Europeans voice mail you have to check if that is even allowed, could be a problem with GDPR legislation. -Original Message- Subject: RE: How to Modify Message and add more Attachments Thanks, am planning to use Google's Speech-to-Text. -Original Message- Sent: Tuesday, October 6, 2020 3:39 PM To: dovecot ; Mrinal Sharma Subject: RE: How to Modify Message and add more Attachments CAUTION - EXTERNAL EMAIL This email originated from outside of Smith Micro Software. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hmmm, that does not sound nice storing files as email. Maybe use document database? Look at this[1], see if it is possible to use the rados plugin to store files directly as objects? What are you using for speech to text? [1] https://github.com/ceph-dovecot/dovecot-ceph-plugin -Original Message- To: dovecot@dovecot.org Subject: How to Modify Message and add more Attachments Hello Everyone, I am working on a product in which we are planning to store voice messages in Dovecot sent by a user to another user. The message would be stored as an email with .wav attachment. Once the Voice message is received, it may get Transcribed. The message can be further be processed and more information can be added to the message later. The original plan was to Modify the email and add New information as attachments to same message. As I understood, messages stored in Dovecot are immutable. What is the best option to achieve this functionality? Thanks, Mrinal
RE: How to Modify Message and add more Attachments
Hmmm, that does not sound nice storing files as email. Maybe use document database? Look at this[1], see if it is possible to use the rados plugin to store files directly as objects? What are you using for speech to text? [1] https://github.com/ceph-dovecot/dovecot-ceph-plugin -Original Message- To: dovecot@dovecot.org Subject: How to Modify Message and add more Attachments Hello Everyone, I am working on a product in which we are planning to store voice messages in Dovecot sent by a user to another user. The message would be stored as an email with .wav attachment. Once the Voice message is received, it may get Transcribed. The message can be further be processed and more information can be added to the message later. The original plan was to Modify the email and add New information as attachments to same message. As I understood, messages stored in Dovecot are immutable. What is the best option to achieve this functionality? Thanks, Mrinal
RE: Providers running dovecot?
>> I am looking for providers of free e-mail addresses known to run >> Dovecot (or a variant thereof) for IMAP access. > >Possibly Posteo. Not free IIRC, but very inexpensive (~1EUR/month). I already offered him a free account to test with, and some GB's of testing mail. But him seem to have disappeared already ;)
RE: Vacation sieve explained: how does it work?
You can do whatever you like, as long as the result is this 'text' file.
I have also bash file that modifies this file for users. You can make a
5 min cron job that detects changes in ldap and then creates the sieve
rule.
-Original Message-
Cc: dovecot
Subject: Re: Vacation sieve explained: how does it work?
> It is just a sieve rule.
>
> 1.
> On disk default file .dovecot.sieve
>
> 2. I am using roundcube webmail for that.
>
> 3. with if false # true
> # rule:[Out of Office]
> if true
> {
> vacation :days 1 :subject "Test" "test test"; }
Is there a way to inject variables from the userdb into the above?
For example, if I was to pass the vacation message attribute through the
userdb lookup, could I then test if the variable was present, and then
embed that variable in the message?
Regards,
Graham
—
RE: Vacation sieve explained: how does it work?
It is just a sieve rule.
1.
On disk default file .dovecot.sieve
2. I am using roundcube webmail for that.
3. with if false # true
# rule:[Out of Office]
if true
{
vacation :days 1 :subject "Test" "test test";
}
-Original Message-
From: Graham Leggett [mailto:minf...@sharp.fm]
Sent: woensdag 23 september 2020 14:03
To: dovecot@dovecot.org
Subject: Vacation sieve explained: how does it work?
Hi all,
I have a legacy system that uses gnarwl to handle vacation messages,
most specifically the vacation messages are stored in LDAP. We have a
web based tool that allows people to update their vacation message, all
they’re doing is modifying LDAP.
I am trying to find out if dovecot’s vacation sieve can do the same
thing. I am struggling however to find something that describes to me
how dovecot vacation sieve is configured. Is there a howto or
walkthrough anywhere?
Specific burning questions:
- If I was to configure the dovecot vacation sieve, where is the body of
each vacation message stored? File on disk? In a store like an LDAP
store? Somewhere else?
- How is the body of vacation message modified? Does this happen
through the IMAP protocol, a command line tool, something else?
- How is the vacation message switched on and off? Does this happen
through the IMAP protocol, a command line tool, or something else?
The docs at
https://wiki2.dovecot.org/Pigeonhole/Sieve/Extensions/Vacation describe
how to control how often the vacation messages are sent, but very little
else. I am very confused :(
Regards,
Graham
—
Imaptest using the 'DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA'
I just noticed that the imaptest program is using this default first message 'DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA' of the mbox file to test with. I think this should be execluded, as one does not expect this behaviour.
RE: Is it possible to only replicate indexes and not the mail data
>> Is it possible to only replicate indexes and not the mail data? >> (Because the data is already on distributed storage) > Does replication re-copy data that is already on the replication server? > I don't think it does. Hmmm, maybe I do not understand fully. But if your emails are stored on distributed storage. Then regardless if an email is received on srv1 or srv2, it is stored on this distributed storage, and thus 'available' on both servers. However only on 1 server the index will be updated, the server that receives this email. Can't this get messy with how replication is described on this page[1]. Only the index needs to be updated on the other server. [1] https://wiki.dovecot.org/Replication "The replication is done by looking at Dovecot index files (not what exists in filesystem)"
ltmp delivery with ldap and prefetch
I am able to authorize and connect via imap. But I am not able to
deliver a message via lmtp
I am getting this error message.
passdb didn't return userdb entries, trying the next userdb
[1] dovecot-ldap.conf.ext
pass_attrs =
uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uidNu
mber=userdb_uid,gidNumber=userdb_gid
user_attrs =
uid=%{ldap:uidNumber},gid=%{ldap:gidNumber},home=%{ldap:homeDirectory}
[2] pass / user db's
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
auth_verbose = default
default_fields =
deny = no
driver = ldap
master = no
mechanisms =
name =
override_fields =
pass = no
result_failure = continue
result_internalfail = continue
result_success = return-ok
skip = never
username_filter =
}
userdb {
args = /etc/dovecot/special-userdb
auth_verbose = default
default_fields =
driver = passwd-file
name =
override_fields =
result_failure = continue
result_internalfail = continue
result_success = return-ok
skip = never
}
userdb {
args =
auth_verbose = default
default_fields =
driver = prefetch
name =
override_fields =
result_failure = continue
result_internalfail = continue
result_success = return-ok
skip = never
}
RE: using %d as a variable in the ldap search base
If you already tested %d in the ou of the auth user bind, and it is not
working. I guess you are just left with options like
1. 3 different vm's
2. auth bind = no base, scope subtree that is parent of those 3 ou's and
then apply filters something like
pass_filter = (&(objectClass=posixAccount)(uid=%n)(ou:dn:=%d))
3. investigate if nslcd(/?) has an option to use multiple queries
for passwd and have dovecot authenticate against the system.
-Original Message-
Cc: dovecot; luke-dovecot
Subject: Re: using %d as a variable in the ldap search base
I'm trying to use auth bind to avoid having a plain text password in a
config file. With %u instead of %n, the @domain part of the login ends
up in the uid field of the search filter. As I said, my OUs have
overlapping users, so I have configured things such that the users are
logging in with user@domain, and need to get the domain component into
the search base as you cannot filter on an Organizational Unit in an
ldap filter. If I have a single search base, it will return multiple
users for those %ns that overlap. If I use %u, it will return no users
for any account, because the uids do not have the @domain in them. I
tried both ways.
Luke
On Mon, Aug 31, 2020 at 09:45:17PM +0200, Marc Roos wrote:
> You have two ways of authenticating against ldap. I decided to use the
> method where a single account has access to the user credentials.
> (Advantage of this method, you can limit ldap lookups eg do not have
> to do 2nd for the userdb)
>
> debug_level = 1
> uris = ldaps://ldap.local:8443
> dn = cn=,cn=b,ou=c,dc=,dc=,dc=local
> dnpass =
> base = ou=asdfadsfa,ou=,ou=ggg,dc=f,dc=,dc=local
> scope = subtree
>
> user_filter = (&(objectClass=posixAccount)(uid=%u)
> pass_attrs =
> uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uid
> Nu
> mber=userdb_uid,gidNumber=userdb_gid
>
> However I did not have any issues authenticating against ldap with
> yours also. But using the %u
>
>
>
>
>
> -Original Message-
> From: Luke Schierer [mailto:luke-dove...@schierer.org]
> Sent: maandag 31 augustus 2020 21:32
> To: dovecot@dovecot.org
> Subject: using %d as a variable in the ldap search base
>
> Hi,
>
> I'm trying to configure dovecot to use LDAP authentication directly,
> and I'm having a bit of trouble. I have a somewhat unusual setup, in
> that I have an LDAP directory that has 3 OUs each of which have their
> own set of users, some of which overlap. As I was trying to figure
> things out, I was setting the search base in my
> /etc/dovecot/dovecot-ldap.conf.ext
> file which is referenced by /etc/dovecot/conf.d/auth-ldap.conf.ext
> file to
>
> base = ou=%d,dc=thecrazyguys,dc=net
>
> however, the resulting searches against the directory endup just
> dropping the %d, resulting in
>
> ou=,dc=thecrazyguys,dc=net
>
> which is invalid. On a whim, I tried a search base of
>
> base = dc=%d,dc=thecrazyguys,dc=net
>
> and found that it did correctly substitute in the variable, which
> would be correct, except that my ldap tree is set up with OUs and not
> an extra DC segment.
>
> for whatever reason, it will do variable substitution for dc=%d, but
> not for ou=%d. this is certainly not documented, and seems like wrong
> behavior, since having an ou in a search base is valid.
>
> I'm including configuration information below. Please let me know if
> I've missed including information that is required.
>
> Thanks!
> Luke
>
>
> luke@schierer@littera001:/etc/dovecot$ lsb_release -rd
> Description: Ubuntu 18.04.5 LTS
> Release: 18.04
> luke@schierer@littera001:/etc/dovecot$
>
> luke@schierer@littera001:/etc/dovecot$ dpkg -l | grep -i dovecot ii
> dovecot-core 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-imapd
> 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-ldap 1:2.2.33.2-1ubuntu4.6
> amd64 ii dovecot-pop3d 1:2.2.33.2-1ubuntu4.6 amd64
> luke@schierer@littera001:/etc/dovecot$
>
> root@littera001:/etc/dovecot# dovecot -n # 2.2.33.2 (d6601f4ec):
> /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS:
> Linux 4.15.0-112-generic x86_64 Ubuntu 18.04.5 LTS auth_verbose = yes
> first_valid_uid = 1001 imap_hibernate_timeout = 100 secs lock_method =
> dotlock mail_cache_min_mail_count = 15 mail_fsync = always
> mail_location = maildir:~/Maildir:LAYOUT=fs:INDEX=MEMORY
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mail_privileged_group = mail
> mailbox_list_index = yes
> maildir_broken_filename_sizes = yes
> maildir_very_dirty_syncs = yes
> mbox_min_index_size = 10 B
> mmap_disable = yes
> namespace inbox {
> inbox = yes
> location =
> mailbox Draf
RE: using %d as a variable in the ldap search base
You have two ways of authenticating against ldap. I decided to use the
method where a single account has access to the user credentials.
(Advantage of this method, you can limit ldap lookups eg do not have to
do 2nd for the userdb)
debug_level = 1
uris = ldaps://ldap.local:8443
dn = cn=,cn=b,ou=c,dc=,dc=,dc=local
dnpass =
base = ou=asdfadsfa,ou=,ou=ggg,dc=f,dc=,dc=local
scope = subtree
user_filter = (&(objectClass=posixAccount)(uid=%u)
pass_attrs =
uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uidNu
mber=userdb_uid,gidNumber=userdb_gid
However I did not have any issues authenticating against ldap with yours
also. But using the %u
-Original Message-
From: Luke Schierer [mailto:luke-dove...@schierer.org]
Sent: maandag 31 augustus 2020 21:32
To: dovecot@dovecot.org
Subject: using %d as a variable in the ldap search base
Hi,
I'm trying to configure dovecot to use LDAP authentication directly, and
I'm having a bit of trouble. I have a somewhat unusual setup, in that I
have an LDAP directory that has 3 OUs each of which have their own set
of users, some of which overlap. As I was trying to figure things out,
I was setting the search base in my /etc/dovecot/dovecot-ldap.conf.ext
file which is referenced by /etc/dovecot/conf.d/auth-ldap.conf.ext file
to
base = ou=%d,dc=thecrazyguys,dc=net
however, the resulting searches against the directory endup just
dropping the %d, resulting in
ou=,dc=thecrazyguys,dc=net
which is invalid. On a whim, I tried a search base of
base = dc=%d,dc=thecrazyguys,dc=net
and found that it did correctly substitute in the variable, which would
be correct, except that my ldap tree is set up with OUs and not an extra
DC segment.
for whatever reason, it will do variable substitution for dc=%d, but not
for ou=%d. this is certainly not documented, and seems like wrong
behavior, since having an ou in a search base is valid.
I'm including configuration information below. Please let me know if
I've missed including information that is required.
Thanks!
Luke
luke@schierer@littera001:/etc/dovecot$ lsb_release -rd
Description: Ubuntu 18.04.5 LTS
Release: 18.04
luke@schierer@littera001:/etc/dovecot$
luke@schierer@littera001:/etc/dovecot$ dpkg -l | grep -i dovecot ii
dovecot-core 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-imapd
1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-ldap 1:2.2.33.2-1ubuntu4.6 amd64
ii dovecot-pop3d 1:2.2.33.2-1ubuntu4.6 amd64
luke@schierer@littera001:/etc/dovecot$
root@littera001:/etc/dovecot# dovecot -n # 2.2.33.2 (d6601f4ec):
/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS:
Linux 4.15.0-112-generic x86_64 Ubuntu 18.04.5 LTS auth_verbose = yes
first_valid_uid = 1001 imap_hibernate_timeout = 100 secs lock_method =
dotlock mail_cache_min_mail_count = 15 mail_fsync = always mail_location
= maildir:~/Maildir:LAYOUT=fs:INDEX=MEMORY
mail_nfs_index = yes
mail_nfs_storage = yes
mail_privileged_group = mail
mailbox_list_index = yes
maildir_broken_filename_sizes = yes
maildir_very_dirty_syncs = yes
mbox_min_index_size = 10 B
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocols = " imap pop3"
ssl_cert =
Imaptest stalls when removing msgs argument
I have had this imaptest[1] stall. I compared this command with what you have on the website and I was a bit surprised that what is on the website is working. What I have noticed is that if you remove this msgs argument, the imaptest stalls. I am not sure if this is according to design, but one might expect that specifying a secs, makes specifying msgs redundant? [1] imaptest - append=100,0 logout=0 host=xx.local port=143 user=xxx pass=xxx seed=100 secs=240 clients=1 mbox=64kb.mbox box=inbox/test. [2] https://doc.dovecot.org/admin_manual/sample_tests/ Using: dovecot-imaptest-2.3.10-2.x86_64
Rsync the dovecot repo
Is it possible to rsync some repo that has centos7 2.3.11? http://repo.dovecot.org/ce-2.3-latest/centos/7/RPMS/x86_64/2.3.11.3-3_ce/
Is it possible to only replicate indexes and not the mail data
Is it possible to only replicate indexes and not the mail data? (Because the data is already on distributed storage)
RE: Ldap userdb prefetch not working
Was related to auth_bind = yes
-Original Message-
To: dovecot
Subject: Ldap userdb prefetch not working
I am having some issues with getting the prefetch working.
In /etc/dovecot/dovecot-ldap.conf.ext I am having:
pass_attrs =
uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uidNu
mber=userdb_uid,gidNumber=userdb_gid
If I do an ldap search from this location I am able to get these
properties.
ldapsearch -LLL -W -s sub -D
"uid=test,ou=xx,ou=xx,ou=xx,dc=xx,dc=xx,dc=xx" -b
"uid=test,ou=xx,ou=xx,ou=xx,dc=xx,dc=xx,dc=xx" -H
ldaps://ldap.local:8443 '(&(objectClass=posixAccount)(uid=test))'
Yet I am still having this
Aug 24 19:51:35 auth: Debug:
prefetch(test,127.0.0.1,<0Qf36aStdJV/AAAB>): passdb didn't return userdb
entries, trying the next userdb Aug 24 19:51:35 auth: Debug:
prefetch(test,127.0.0.1,<0Qf36aStdJV/AAAB>): Finished userdb lookup Aug
24 19:51:35 auth: Error: plain(test,127.0.0.1,<0Qf36aStdJV/AAAB>):
user not found from any userdbs
This is my auth-ldap.conf.ext
passdb {
driver = ldap
# Path for LDAP configuration file, see
example-config/dovecot-ldap.conf.ext
args = /etc/dovecot/dovecot-ldap.conf.ext
}
# "prefetch" user database means that the passdb already provided the #
needed information and there's no need to do a separate userdb lookup.
#
userdb {
driver = prefetch
}
RE: Expuning & Purging doesn't fully remove emails?
But is that not adviced in the manual[1]? I am also in the process of setting up dovecot behind a haproxy. Still testing with this. I was wondering why one would choose haproxy and not eg dovecot proxy (with director) [1] https://wiki2.dovecot.org/HAProxy -Original Message- From: Zelic Bojan [mailto:bojan.ze...@kudelskisecurity.com] Sent: woensdag 26 augustus 2020 23:39 To: dovecot@dovecot.org Subject: Re: Expuning & Purging doesn't fully remove emails? I managed to figure this out. Just wanted to follow up if anyone in the future encounters the same issue. I am using HAProxy along with dovecot replication. When an email comes in... it is round robin'd to each of the 2 dovecot/postfix servers. I have postfix running on each server and it uses the local dovecot LMTP service for storing the mails. We sometimes get surges of emails(hundreds or thousands in a couple of seconds). Each alternative request hits a different mail server and then should be replicated. The replication works and we don't encounter missing or duplicated emails. However, when I purge expunged emails, somehow this does not fully remove them when it's setup in this fashion. I ran 2 tests: 1) I send 1000 emails to the loadbalancer(round-robin), expunge on both servers & purge them on both servers, recreate the indexes... ~200 emails come back. 2) I send 1000 emails to one of the mail servers, expunge & purge them, recreate the indexes... 0 emails come back. My fix is to remove the round robin load balancing and use sticky tables in HAproxy. From: Zelic Bojan Sent: Thursday, August 20, 2020 1:42 PM To: dovecot@dovecot.org Subject: Expuning & Purging doesn't fully remove emails? Hello, I'm facing an issue where deleted emails keep re-appearing after my mailbox index gets recreated. I'm running version 2.2.36 of dovecot, but I tested the same scenario under 2.3.10. I'm also using mdbox, autoexpunge, and using dovecot replication. I've had several instances now where some expunged emails show up again in a mailbox. I noticed this error: doveadm: Error: Corrupted dbox file /var/mail/virtual/mail...@domain.com/mdbox/storage/m.3228 (around offset=1988744): msg header has bad magic value which caused the index to get rebuilt... however several times now, the indexes got rebuilt even though there doesnt seem like there was any error... so im not sure why that is. lmtp(13910): Warning: fscking index file /var/mail/virtual/mailbox@domain/mdbox/storage/dovecot.map.index lmtp(13910): Warning: fscking index file /var/mail/virtual/mailbox@domain/mdbox/storage/dovecot.map.index lmtp(13910): Warning: mdbox /var/mail/virtual/mailbox@domain/mdbox/storage: rebuilding indexes I'm not sure why these mails keep coming back though... or if there's anything that I can do to limit the number of emails that get restored. I want to make sure expunged & purged emails stay expunged & purged. If I run a purge and then force index recreation... why would expunged emails come back? Shouldn't I expect them all to be deleted & purged? Does expunge not expunge all emails? (In production, I'm running autoexpunge but this test below shows what happens when I attempt to expunge everything) doveadm search -u email@domain all | wc -l # output 22096 doveadm expunge -u email@domain mailbox '*' all doveadm search -u email@domain all | wc -l # output: 0 doveadm purge -u email@domain doveadm dump /var/mail/virtual/email@domain/mdbox/storage/ | grep -c 'ref.*\b0\b' # output: 0 doveadm force-resync -u email@domain Inbox # output: # doveadm(email@domain): Warning: fscking index file /var/mail/virtual/email@domain/mdbox/storage/dovecot.map.index # doveadm(email@domain): Warning: mdbox /var/mail/virtual/email@domain/mdbox/storage: rebuilding indexes # doveadm(email@domain): Warning: fscking index file /var/mail/virtual/email@domain/mdbox/storage/dovecot.map.index doveadm search -u email@domain all | wc -l # output: 843 I would expect the output to be 0. Theoretically I deleted all emails and purged all emails. Nothing should be left on the disk? However... I can see there are still m.* files in mdbox/storage for the mailbox. Overall, I'm not sure why the index got recreated... but I'm trying to limit the impact of mailbox corruption so that deleted emails do not come back if the index is somehow recreated again. If I were to re-run expunge, purge, and force-resync a 2nd time... it does get emptied out, but I'm not looking to run force-resync intentially since it causes dataloss with mdbox, and re-running only expunge & purge doesn't seem to do anything. Bojan Zelic Sr. IT Infrastructure Engineer
Environment variable in config
I am getting errors addressing environment variables. Are they supposed
to work in such sections. Not really clear from this[1] page.
service health-check {
# example health-check.
executable = script -p /bin/health-check.sh
inet_listener health-check {
port = env:PORT0
}
}
[1]
https://doc.dovecot.org/configuration_manual/config_file/config_variables/
Ldap userdb prefetch not working
I am having some issues with getting the prefetch working.
In /etc/dovecot/dovecot-ldap.conf.ext I am having:
pass_attrs =
uid=user,userPassword=password,host=host,homeDirectory=userdb_home,uidNu
mber=userdb_uid,gidNumber=userdb_gid
If I do an ldap search from this location I am able to get these
properties.
ldapsearch -LLL -W -s sub -D
"uid=test,ou=xx,ou=xx,ou=xx,dc=xx,dc=xx,dc=xx" -b
"uid=test,ou=xx,ou=xx,ou=xx,dc=xx,dc=xx,dc=xx" -H
ldaps://ldap.local:8443 '(&(objectClass=posixAccount)(uid=test))'
Yet I am still having this
Aug 24 19:51:35 auth: Debug:
prefetch(test,127.0.0.1,<0Qf36aStdJV/AAAB>): passdb didn't return userdb
entries, trying the next userdb
Aug 24 19:51:35 auth: Debug:
prefetch(test,127.0.0.1,<0Qf36aStdJV/AAAB>): Finished userdb lookup
Aug 24 19:51:35 auth: Error: plain(test,127.0.0.1,<0Qf36aStdJV/AAAB>):
user not found from any userdbs
This is my auth-ldap.conf.ext
passdb {
driver = ldap
# Path for LDAP configuration file, see
example-config/dovecot-ldap.conf.ext
args = /etc/dovecot/dovecot-ldap.conf.ext
}
# "prefetch" user database means that the passdb already provided the
# needed information and there's no need to do a separate userdb lookup.
#
userdb {
driver = prefetch
}
when are these conf.ext files loaded
Are these .conf.ext always loaded? Or only when they are configured somewhere else?
RE: submission message quota
mailfromd should also be able to do this. Going to apply this soon myself -Original Message- To: dovecot@dovecot.org Subject: Re: submission message quota Am 20.08.2020 um 08:29 schrieb Gerry: > Hello, > > I am trying to come up with a way to have individual quotas per user > for the submission service. Similar to what I could achieve with > Postfix and policyd. > > More specifically, the quota I am most interested in, is limiting the > number of messages a single account can send within a given timeframe. > Ideally, I'd also like to limit the number of total recipients within > a given timeframe, to mitigate the loophole of adding multiple > recipients to a single message. > > Example: account Y is allowed to send 500 messages per 60 minutes, > with a maximum of 2000 recipients overall. > > What would be the best path to take? > > Thanks! > > Gerry If postfix is handling your submission service you can have a look at postfwd https://www.postfwd.org/ I use it to limit sending of emails, recipients, etc, etc Greetings Becki
/usr/include/dovecot/str.h
FYI, I am building on alpine 3.10 and 3.11 I am getting this build[1] error. I think these header files of dovecot are not correct. Or are they? I changed line 35 in /usr/include/dovecot/str.h from str_append_max(str, cstr, max_len); to str_append_max(str, (const char *)cstr, max_len); [1] /usr/include/dovecot/str.h: In function 'void str_append_n(string_t*, const void*, size_t)': /usr/include/dovecot/str.h:35:22: error: invalid conversion from 'const void*' to 'const char*' [-fpermissive] 35 | str_append_max(str, cstr, max_len); | ^~~~ | | | const void* /usr/include/dovecot/str.h:31:48: note: initializing argument 2 of 'void str_append_max(string_t*, const char*, size_t)' 31 | void str_append_max(string_t *str, const char *cstr, size_t max_len); |^~~~
RE: Please Help me
If you plan on using distributed storage, enable this on the old environment so you can move messages to the distributed storage (I have archive mailboxes there). This way you are only spending time copying recent email. -Original Message- From: Kishore Potnuru [mailto:kishore.reac...@gmail.com] Sent: zondag 12 juli 2020 22:18 To: Dovecot Mailing List Subject: Please Help me Hi All, I need to migrate the complete data & setup/environment from an old dovecot environment to a new dovecot environment. Old environment details: Linux OS- Red Hat Enterprise Linux Server release 6.5 (Santiago) Dovecot version - 2.0.9 Postfix version - 2.6.6 New Environment details: ( 2 systems) Linux OS- Red Hat Enterprise Linux Server release 7.7 (Maipo) Dovecot version - 2.2.36 (1f10bfa63) Postfix version - 2.10.1 In the new environment, I have 2 servers. I have got 2 of the above servers with above configuration and need to create HA/Resilience. I know there is advanced versions in Redhat and Dovecot. As per my organisation's existing support/policy, that is what maximum supported RHEL and official version of dovecot as per RHEL 7.7. Now my questions are: 1. I will stop the dovecot/postfix services on the server in old environment. Is it possible to move/copy the complete configuration along with data (directories/sub-directories/users/groups/existing email in new or cur folders/complete email domains along with complete structure) to the other 2 servers in new env? Could you please suggest me the commands or steps follow that? 2. Now, With the above 2 servers, I would like to create the HA/Resilience. I know there is a Replication options which are there. Could you please provide me the steps/video, how it needs to be done? I have got the systems and I need to move/copy the environment to a new environment and create the HA/Resilience. I am not sure, how to proceed and what needs to be done clearly? I am stuck here. Please help me in this step by step procedure. Thanks, Kishore Potnuru
RE: Urgent Help required
>> Am I able to install (2.3 version) on RHEL 6.10 or RHEL 7.7 versions? Will there be any issues? >> I know RHEL 6.10 is out of support in November. But I want to understand and try in my test environment. Why do you ask? If you open te repo link you can see there is 2.3-latest there not? Of course you will have issues installing these if you type as root cd / && rm -R -f * You say you are testing, so do the testing.
RE: Urgent Help required
>> IBM is too big, too blue, and too politically correct. Something is a little bit off. I would pick IBM over Microsoft or Google any time. Totally fan of OpenPOWER initiative and power8/9 cpu's. I hope your wrong and IBM - RedHat is going to be a great combination.
RE: Urgent Help required
>>The other side of the question is, Why is the software always so "vulnerable" and "broken" in the first >>place as to be unsuitable for Long Term Support? >> >>If the software code worked when it was released some number of years ago, then why doesn't it still work >>the same way today as it it did when it was released? Whenever I hear people complain about computers and/or software. I always suggest them to use something like the abacus of 2000 bc. You should be glad for what they do for you ;) And since humans operate them, you will find annoyances as with doctors amputating the wrong limb.
RE: Urgent Help required
>>> with broken or vulnerable software is there really a benefit? >> >> LTS distributions back port necessary patches >Then the OP should be able to update to a dovecot that doesn't have the issue, right? I have no idea what his issue is, and why he is stuck even in specific releases. I have been running dovecot on el6 and el7 for years and years without issues.
RE: Urgent Help required
> with broken or vulnerable software is there really a benefit? LTS distributions back port necessary patches
RE: handling spam from gmail.
> Wrong mailing list. You need to ask on the list for the MTA you are > using (Sendmail, Postfix, ). Yes will ask soon at sendmail. > Actually, this sounds like a job for a custom milter, which would look > at the domain name of the sending system, and reject the mail with your > message. Dunno if there is one that works exactly like this. > I think also, I should have some contact coding milters. I think this could be ok for things like this spamhaus bad tld list or so.
RE: handling spam from gmail.
>> Yes tell that to the people that create rhel6, rhel7 and rhel8 and give >> lts support. > >as said that has nothing to do with your wrong training and RHEL has >always the same problem: you can't package the latest and greatest shit >over 10 years because it requrires newer versions of dependencies > >so what you get with your lazyness by using a LTS distribution is a >"never change a running system, just fix the worst bugs and don't touch >anything else" > >upstream developers don't hold development for 10 years > >written from a Fedora workstation with kernel 5.6.18-200.fc31.x86_64 >from last night realyed over a datacenter firewall and a mailserver >using the same kernel > >> Unless google pays you to train your software to mark their messages as >> spam, you might want to consider yourself not to smart as well ;) > >unless i make good money from customers paying for a nearly 100% >hitrate of spam combine with a zero-false-positive policy i am likely >smarter than you I would argue it is quite difficult to identify intelligence. I am pretty sure I would not start with your reasoning. I have a favourite German saying I like to quote in matters like these "gegen Dummheit kämpfen Götter selbst vergebens" >> My solution would solve the problem others create (see the other mail). >> Your solution wastes your time and will always be carrying water to the >> sea. I think if 50% of providers in the world would do this, it would >> quickly be end of story for the spam originating from the networks like >> google and amazon. > >they won't give a shit and when i get such idiotic mails as you propose >i take the phone, call the sender and suggest to fire his mailadmin >better sooner than later > >> If there is a McDonalds build next to your home, and their clients throw >> waste into your garden. You hold McDonalds liable for cleaning this up >> not? Or are you also going to cleanup their mess indefinitely. > >what a nonsense
RE: handling spam from gmail.
Yes thanks, I know, however the criteria for putting emails into this procedure is a different subject. Just wondered what people are doing. -Original Message- To: dovecot@dovecot.org Subject: Re: handling spam from gmail. On Thu, Jun 11, 2020 at 10:19:50AM +0200, Marc Roos wrote: > > > I am sick of this gmail spam. Does anyone know a solution where I can > do something like this: > > 1. received email from adcpni...@gmail.com 2. system recognizes this > email address has been 'whitelisted', continue with 7. > 3. system recognizes as this email never been seen before 4. auto > reply with something like (maybe with a wait time of x hours): >Your message did not receive the final recipient. You are sending > from a known spam provider >network that is why we blocked your message. Please confirm that: >- you are not a spammer and >- you have permission to use the mail adress you send your message to >- you and your provider agree to uphold GDPR legislation >- you and your provider are liable for damages when breaching any > of the above. > > >Click link to confirm and you agree with the above >https://www.domainwithoutletsencryptcertificate.com/asdfasdfadsfaf > > 5. sender clicks confirm url > 6. email address is added to some white list. > 7. email is delivered to recipient. If you do this rgularly enough, sending these messages to what are likely forged return addresses, you might just end up being classified as a spam sender yourself. -- hendrik > > > > >
RE: handling spam from gmail.
Yes tell that to the people that create rhel6, rhel7 and rhel8 and give lts support. Unless google pays you to train your software to mark their messages as spam, you might want to consider yourself not to smart as well ;) My solution would solve the problem others create (see the other mail). Your solution wastes your time and will always be carrying water to the sea. I think if 50% of providers in the world would do this, it would quickly be end of story for the spam originating from the networks like google and amazon. If there is a McDonalds build next to your home, and their clients throw waste into your garden. You hold McDonalds liable for cleaning this up not? Or are you also going to cleanup their mess indefinitely. -Original Message- From: Reindl Harald [mailto:h.rei...@thelounge.net] Sent: donderdag 11 juni 2020 11:09 To: Marc Roos; dominic; dovecot; lists; users Subject: Re: handling spam from gmail. Am 11.06.20 um 11:04 schrieb Marc Roos: > I have got lots of shit coming from *.google.com like these: > > X-Spam-Status: No, score=2.1 required=3.0 tests=BAYES_00 because you are too dumb to train your bayes give me one such message and i am pretty sure it will fire BAYES_80 or BAYES_99 which will burn it with fire and lead to a milter reject here SpamAssassin 3.3.1 is also not very smart in 2020 so instead ask for dumb solutions which making you part of a bigger problem better do your homework
RE: handling spam from gmail.
Your logics sucks. There is a difference between how email works and how spamassassin works. You are assuming that everyone in the world is using spamassassin by including it in 'how email works'. Maybe you like to post a link to your bayes files Am 11.06.20 um 11:13 schrieb Marc Roos: > You do not understand how mail works. Google mail is only getting > through when spf checks and the likes are being passed. unless you don't manage to get rid of BAYES_00 in case of clear spam messages don't tell me you understand how email works
RE: handling spam from gmail.
You do not understand how mail works. Google mail is only getting through when spf checks and the likes are being passed. I am not creating any problems with this, I am just bouncing them back. Google has enough billions to handle these issues. If everyone would apply this procedures, people with legitimate email accounts would move from a spam network to some other provider. People joining these providers are the problem, because it allows these networks to mix spam with legitimate email. When clients start moving out, spam networks are becoming easier to hard block and these providers start thinking about their infrastructure and their bussines model. If everyone would be doing this, it is solving the spam problem. My below procedure should be applicable for any network generating a lot of spam. -Original Message- From: Reindl Harald [mailto:h.rei...@thelounge.net] Sent: donderdag 11 juni 2020 10:25 To: Marc Roos; dovecot; users Subject: Re: handling spam from gmail. Am 11.06.20 um 10:19 schrieb Marc Roos: > I am sick of this gmail spam. Does anyone know a solution where I can > do something like this: > > 1. received email from adcpni...@gmail.com 2. system recognizes this > email address has been 'whitelisted', continue with 7. > 3. system recognizes as this email never been seen before 4. auto > reply with something like (maybe with a wait time of x hours): >Your message did not receive the final recipient. You are sending > from a known spam provider >network that is why we blocked your message. Please confirm that: >- you are not a spammer and >- you have permission to use the mail adress you send your message to >- you and your provider agree to uphold GDPR legislation >- you and your provider are liable for damages when breaching any > of the above. > > >Click link to confirm and you agree with the above >https://www.domainwithoutletsencryptcertificate.com/asdfasdfadsfaf > > 5. sender clicks confirm url > 6. email address is added to some white list. > 7. email is delivered to recipient. and i am sick of people not understanding how email works! you don't send unasked mail to a in the most cases forged sender unless you want to be part of the problem backscatters and brainless autoreplies have to be burnt with fire
RE: SV: handling spam from gmail.
I know it is not dovecot who should fix this. But anyone using dovecot is using an MTA, and receiving spam ;) I know how to look at email headers. Spf and dkim is not solving anything here. -Original Message- From: Sebastian Nielsen [mailto:sebast...@sebbe.eu] Sent: donderdag 11 juni 2020 10:23 To: Marc Roos; 'dovecot'; 'users' Subject: SV: handling spam from gmail. This is not a job for dovecot. You should look into whatever is your MTA (exim, postfix etc) and implement the solution there. But my initial suggestion is to check SPF and DKIM of the email. Because I know that gmail does terminate spammers quick, but if you don't validate SPF or DKIM, you might be a victim of spoofed Gmail email. Best regards, Sebastian Nielsen -Ursprungligt meddelande- Från: dovecot-boun...@dovecot.org För Marc Roos Skickat: den 11 juni 2020 10:21 Till: dovecot ; users Ämne: handling spam from gmail. I am sick of this gmail spam. Does anyone know a solution where I can do something like this: 1. received email from adcpni...@gmail.com 2. system recognizes this email address has been 'whitelisted', continue with 7. 3. system recognizes as this email never been seen before 4. auto reply with something like (maybe with a wait time of x hours): Your message did not receive the final recipient. You are sending from a known spam provider network that is why we blocked your message. Please confirm that: - you are not a spammer and - you have permission to use the mail adress you send your message to - you and your provider agree to uphold GDPR legislation - you and your provider are liable for damages when breaching any of the above. Click link to confirm and you agree with the above https://www.domainwithoutletsencryptcertificate.com/asdfasdfadsfaf 5. sender clicks confirm url 6. email address is added to some white list. 7. email is delivered to recipient.
handling spam from gmail.
I am sick of this gmail spam. Does anyone know a solution where I can do something like this: 1. received email from adcpni...@gmail.com 2. system recognizes this email address has been 'whitelisted', continue with 7. 3. system recognizes as this email never been seen before 4. auto reply with something like (maybe with a wait time of x hours): Your message did not receive the final recipient. You are sending from a known spam provider network that is why we blocked your message. Please confirm that: - you are not a spammer and - you have permission to use the mail adress you send your message to - you and your provider agree to uphold GDPR legislation - you and your provider are liable for damages when breaching any of the above. Click link to confirm and you agree with the above https://www.domainwithoutletsencryptcertificate.com/asdfasdfadsfaf 5. sender clicks confirm url 6. email address is added to some white list. 7. email is delivered to recipient.
RE: Pigeonhole-sieve auto-reply
A vaction message does not need to be sending text about a leave of absense. It is just a rule with criteria being executed. Change the rule to whatever you want, get to know the sieve 'language'. I asked once here something about executing rules on dragging messages to mailboxes/folders, they refered me to imapsieve or sieveimap. Maybe this could help you also. -Original Message- From: @lbutlr [mailto:krem...@kreme.com] Sent: donderdag 11 juni 2020 6:46 To: dovecot mailing list Subject: Pigeonhole-sieve auto-reply Is it possible to have a sieve script reply with a press message to certain emails (and only certain emails) based on sieve matches? I see a lot on vacation replies, but I want something more specific. Something along the lines of procmails formail command? Everything I’ve searched for is about vacation r filing replies into the same folder as the original message.
RE: Ms Exchange vs dovecot
How did you decide for the Baikal? I have been testing a long time ago with this apple ccs calendarserver, but it did/does not feel right. I think there were tasks also in this one, but I am not sure anymore. I had argument with the developers that they should not enforce openssl building, they did not get that they are developers and not sysadmins and should just do development. Their approach made it cumbersome to update the ccs server, exactly the opposite of their goal. I even saw some issues recently that meeting invitation requests were not compatible. Which is an issue with lots of caldav implementations. I have everything in ldap, so that support is necessary. I don't think people use that many exchange features, like with office 80%-90% can just do fine with libre/open office. Granting someone access to folders is mostly it I guess. Ccs was supporting this also if I remember correctly. -Original Message- From: (Michael Hirmke) [mailto:m...@mike.franken.de] Sent: 09 May 2020 13:36 To: dovecot@dovecot.org Subject: Re: Ms Exchange vs dovecot Hi Marc, >I have recently been working/testing with exchange 2016 and started >thinking if I should even migrate to this platform. I assume more >people here have experience with exchange and this idea. I was an Exchange admin for years and even had an Exchange server at home for about 20 years - just for fun and for testing purposes. Three months ago I migrated to dovecot and baikal - and dropped Exchange completely. This worked flawless, so *I* don't miss Exchange at all. But: You can't compare dovecot with Exchange, because dovecot is a mail server, Exchange is a groupware server. This is why I added a baikal server to my infrastructure. Baikal is a Cal- and CardDAV server, that can replace the calendar und contact parts of Exchange. Nevertheless you loose many features of an Exchange server after migrating to such a setup, so if your users got used to these feature, it wouldn't be possible to drop Exchange. It is only feasable for small environments with few people or in a new environment, where nobody has used an Exchange Server until now. IMHO. This was not your question, it is meant as background information, if you wouldn't already know that. For your environment I can't tell if it is possible to migrate to Exchange, because you didn't write, if you already have an Active Directory in place, which is necessary for Exchange on premise. If you want to use Microsoft's Azure AD and the Exchange cloud services on top, you have to migrate your users to Azure AD. In any case you need an Active Directory for Exchange server. >I was wondering if this is possible with a dovecot setup > 1. public folder can be implemented with a public mailbox? Yes, but public folders in Exchange are dying for years. They still exist, but are only supported so so. Public mailboxes in dovecot are supported full fledged. > 2. authorize users via groups access to mailboxes/folders of the >public folder/mailbox. I think I saw ACL's with dovecot, does this >compare to 'folder permissions' Not really, but I'm not an expert for permissions on public mailboxes. > 3. is it possible with sieve to apply a rule on any mailbox/folder? >Thus if I 'drag' a message to a folder, the sieve rule is activated? You can configure a folder to act on incoming mail in the folder properties. I never tested, though, if "incoming" also applies when copying to a folder. Bye. Michael. -- Michael Hirmke
RE: Ms Exchange vs dovecot
> > >Thank you, Michael, for an intelligent and reasoned response. The last thing this forum needs are the rantings of some anarchist with dreams of socialism. > Yes indeed, it seems to be very difficult to stay on topic. >In any event, I question why the OP is interested in Exchange 2016? It has already been surpassed by MS Exchange 2019. I would seriously question the wisdom of using any outdated software, especially if it happens to be in a 'mission-critical' position. Perhaps this URL might be of interest to the OP. > You never heard of LTS etc? You are such idiot that is waiting in front of the apple store, when a new version out? > >I do agree that DOVECOT != MS EXCHANGE. They are two very different animals. Yes, nobody has questioned this. Do you also agree that a pigeon is not a snake? We can continue for quite a while like this. >I have never liked having to use multiple applications to achieve the same results I can with an 'all-in-one,' This is the future. Everything is going to be microservices and distributed (that scales ;)). This all-in-one is nice for people who click next-next-next. All these companies that try to do everything are not speciliazed in any specific feature. Example how microsoft fucks up in this area with outlook. Save send message in folder that is not inbox: works on imap folders, not on public folder not on shared mailbox (wtf) Categories: not working on imap. public folder contacts: do not sync to mobile. The whole exchange server looks like a mess, with all this trace logging on. For the majority you cannot even specify logging levels, let alone do remote logging. Why all proprietary shit? Just offer out of the box card/caldav access. 10 years from now exchange is dead.
RE: Marking all emails in "Trash" as opened, and also prohibiting email clients from creating new mails
Someone just told me about imapsieve. Sieve rules for folders. I assume that could solve your issue. https://wiki.dovecot.org/HowTo/AntispamWithSieve -Original Message- Sent: 09 May 2020 17:32 To: dovecot@dovecot.org Subject: Marking all emails in "Trash" as opened, and also prohibiting email clients from creating new mails Dovecot version: 2.3.7.2 (3c910f64b) (pkg shipped by: Ubuntu-Desktop 20.04) I want to accomplish 2 things in dovecot: 1: I want to force all mails inside Trash to have an "opened"/"read" flag and "Non-Recent" flag. Basically Status: RO This regardless how the flag appears, either by copying/moving the mail into trash, creating a new mail in trash, flagging email in Trash or whatever. Basically, no email in Trash should ever be able to have a recent or unread flag. I tried with a static mail filter and sieve filter to add \\seen to the email upon COPY (as mentioned here: https://dovecot.org/pipermail/dovecot/2017-November/110122.html ), but regardless how I do it, it doesn't work when Samsung Email client trashes an unread email, AND/OR also, it causes weird issues like duplicate email in the trash folder sometimes. Best would be some event filter that executes for every mail that somehow end up in Trash, that checks if \\seen is present, if not, then it will add it, on all emails in trash? But how I do to prevent the duplicate copy that appears sometimes? 2: I want to prohibit email clients from ever creating a new mail in Sent folder. If its possible to allow MOVE and/or COPY, it should be allowed, only new mail should be prohibited. (also note that external processes must be able to create new mail in Sent) HOWEVER - this prohibition must be silent - ergo the newly created email is simply discarded. No error message or error codes should be returned to IMAP client. (The reason I want this, is because I have configured my outgoing SMTP server to populate Sent, and some email clients doesn't have the option to "Don't store a copy of the email in Sent folder" resulting in duplicates) Best regards, Sebastian Nielsen
RE: Ms Exchange vs dovecot
I was wondering about the sieve rules, because I thought they were executed during mail delivery in the lmtp process. You can also 'guess' this a bit from syntax of the rules or the single file they are stored in. Thus if you 'drag' messages between folders, they are not executed. Off topic: I know Exchange is a different solution. What I think is stupid, is that they store mail in a database still. Making it difficult to scale. (I wonder if they have such solution in their cloud) Better would be per user of course. I also do not like that they try and push users to their cloud with all this 365 advertising in the on premises solution. Sooner or later on premises will be gone. Public folder is not removed, they were thinking of it, and community complained (afaik), so they kept it, still there in 2019. (although changed) -Original Message- From: MIhai Badici [mailto:mi...@badici.ro] Sent: 09 May 2020 12:32 To: dovecot@dovecot.org Subject: Re: Ms Exchange vs dovecot First of all, Exchange is a complete solution. Dovecot is a imap/pop3 server ( a good one, sure... ) So replacing exchange means to find an integrated solution. about the questions: public folder was removed in exchange. IMHO they made the right choice :) There is a different thing, need different tools and different client app. You can create shared mailboxes ( i think it's a little demand for that, but yes) Sieve rules should work ( never tried actually) I can see a need for an integration with folders only when you work with webmail (like roundcube) . In this scenario ( a liitle bit like gmail) is good to see the folders and attach them ( or save) I use the kolab plugins for roundcube and there is a sort of integration between the chwala ( files plugin) and any webdav capable file server ( I use owncloud/nextcloud). So you can share files, edit etc using owncloud but also attach them and save them from webmail. You can find a lot of plugins in owncloud to deal with files, even editing with onlyoffice . IMHO, that's the way, there is no need to create a client app to deal with all; maybe other people will not agree but... On 5/9/20 1:07 PM, Marc Roos wrote: > > My, my, did not expect this discussion. It is our own fault we are > stuck with google and microsoft monopolies. If small companies would > combine effort (resources and cash) and would not reinvent/create the > wheel constantly on our own little islands, we would have much better > products. So respect for the dovecot team. > > The reason I am asking is that, the public folder solution is not as > it was in 2000. Exchange 2016+ do not support CDO etc. Nobody > transitioned between the two? > > 1. public folder can be implemented with a public mailbox? > > 2. authorize users via groups access to mailboxes/folders of the > public folder/mailbox. I think I saw ACL's with dovecot, does this > compare to 'folder permissions' > > 3. is it possible with sieve to apply a rule on any mailbox/folder? > Thus if I 'drag' a message to a folder, the sieve rule is activated? > >
RE: Ms Exchange vs dovecot
My, my, did not expect this discussion. It is our own fault we are stuck with google and microsoft monopolies. If small companies would combine effort (resources and cash) and would not reinvent/create the wheel constantly on our own little islands, we would have much better products. So respect for the dovecot team. The reason I am asking is that, the public folder solution is not as it was in 2000. Exchange 2016+ do not support CDO etc. Nobody transitioned between the two? 1. public folder can be implemented with a public mailbox? 2. authorize users via groups access to mailboxes/folders of the public folder/mailbox. I think I saw ACL's with dovecot, does this compare to 'folder permissions' 3. is it possible with sieve to apply a rule on any mailbox/folder? Thus if I 'drag' a message to a folder, the sieve rule is activated?
Ms Exchange vs dovecot
I have recently been working/testing with exchange 2016 and started thinking if I should even migrate to this platform. I assume more people here have experience with exchange and this idea. I was wondering if this is possible with a dovecot setup 1. public folder can be implemented with a public mailbox? 2. authorize users via groups access to mailboxes/folders of the public folder/mailbox. I think I saw ACL's with dovecot, does this compare to 'folder permissions' 3. is it possible with sieve to apply a rule on any mailbox/folder? Thus if I 'drag' a message to a folder, the sieve rule is activated?
RE: Urgent - Help needed
? It is a remark about that server software is dramatically outdated, and you turn it into distribution discussion? I can understand the choice for paid support by Kishore. Redhat has biggest professional workforce and now it is under the umbrella of ibm, I like them even more. In production I would not run anything other than rhel/centos. -Original Message- To: dovecot@dovecot.org Subject: Re: Urgent - Help needed On 2020-05-05 20:15, Alexander Dalloz wrote: > Please inform yourself before posting such snarks comments. if i ever need precompiled problems i would use slackware if i need opensource i would choice freebsd or gentoo have a nice day https://www.kernel.org/ have kernel 5.6.10 stable now
