Doveadm backup...
Hi I've recently tried to use the Dovecadm backup to backup the emails.. with the following syntax doveadm -Dv backup -u userx maildir:/mnt/nfs-backup/userx Sounds to be OK with few emails... Some of them got a lot of emails and one f them got an error and stop ! dsync(userx): Debug: brain S: Import Trash: Import change type=expunge GUID=1725fa475d774ee19cb98dfb6737b4f1 UID=24891 hdr_hash= result=GUIDs match dsync(userx): Debug: brain S: Import Trash: Import change type=expunge GUID=916ed110b4b1522868be6194f1ae36ff UID=24892 hdr_hash= result=GUIDs match dsync(userx): Debug: brain S: Import Trash: Import change type=expunge GUID=eb8d75c530a7b02fc26b494d9006c91b UID=24893 hdr_hash= result=GUIDs match dsync(userx): Debug: brain S: Import Trash: Import change type=expunge GUID=aee9155875c34861fd6500f1f2f51a26 UID=24894 hdr_hash= result=GUIDs match dsync(userx): Debug: brain S: Import Trash: Last common UID=24894. Delayed expunges= dsync(userx): Debug: brain S: Import Trash: Saved UIDs: dsync(userx): Debug: brain S: Import Trash: Finish update: min_next_uid=24895 min_first_recent_uid=24895 min_highest_modseq=35344 min_highest_pvt_modseq=0 dsync(userx): Debug: /mnt/nfs-backup/userx/.Trash/dovecot.index.cache: Compressed, file_seq changed 1645803588 -> 1645803589, size=32, max_uid=24894 dsync(userx): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID 7e05c335174bf1608f0a02004eac7fb4 (maybe it was just deleted?) dsync(userx): Debug: auth-master: conn unix:/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10) I empty the trash... exactly the same problem... Any idea why this ?? Thanks and regards,
Re: email location - files or sql
On 2/25/22 12:51, Chris Bennett wrote: On Wed, Feb 23, 2022 at 08:48:11PM -0500, Robert Moskowitz wrote: I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users. Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB. I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits. What is the opinion on repairs to problems that occur? Easier to fix in a DB? That does give a simple set of fixed points and dates to make repairs simple. I backup into a tar.gz and that preserves a timepoint, but how in the heck could a fix a problem that occurs? That seems like a nightmare problem. Does anyone have any method to do that? I'd love to know that if it exists. I love Maildir, but the backups are very large. I don't have a huge set of emails, but downloading a copy to home instead of my other server is a big task. Since I use a single ISP and once had a company shut down all servers, which makes me nervous. I have used a simple rsync with --delete to maintain a backup of the maildir directories. rsync is run via cron. You stop postfix, wait a bit, rsync, then restart postfix. But recently, partly to do migration, one person pointed out that he uses Dovecot on the backup server to just use imap to replicate the mail store. I have to learn how to do this...
Re: email location - files or sql
On Wed, Feb 23, 2022 at 08:48:11PM -0500, Robert Moskowitz wrote: > > > I liked Maildir at the time and still do. Only a 1,000+ emails a day. My > wife keeps a lot (10K messages) on the server, I keep all of my various > boxes on the server small. So do my other users. > > > Personally, I would not use a relational database as a mail store unless > > specifically required by the mail system of your choice. Email is not > > organised in a way that benefits from a RDB. > > I was there for the beginning of RDB. Almost had NOMAD shoved down my > throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release > site of DB2, where we worked out how to do a UNION which was not supported > in the original design. I have seen email systems that stuff the messages > into RDB and really wonder if the hammer really fits. > What is the opinion on repairs to problems that occur? Easier to fix in a DB? That does give a simple set of fixed points and dates to make repairs simple. I backup into a tar.gz and that preserves a timepoint, but how in the heck could a fix a problem that occurs? That seems like a nightmare problem. Does anyone have any method to do that? I'd love to know that if it exists. I love Maildir, but the backups are very large. I don't have a huge set of emails, but downloading a copy to home instead of my other server is a big task. Since I use a single ISP and once had a company shut down all servers, which makes me nervous. -- Hmm, Chris Bennett
Re: Dovecot 2.3.18 and OpenMetrics-Prometheus problem
Am 25.02.2022 um 11:50 schrieb Christian Rößner :
>
> Hi Aki,
>
>> Am 25.02.2022 um 11:34 schrieb Aki Tuomi :
>>
>>>
>>> On 25/02/2022 12:23 Christian Rößner wrote:
>>>
>>>
>>> Hi,
>>>
>>> I have defined several metrics in Dovecot and activated the metrics
>>> service. After the I tried to include this in Prometheues, but I get
>>> strange errors. I gave up now after several hours in the hope that this
>>> might be a bug. So I ask for help here:
>>>
>>> Distribution: Rocky 8
>>> Official Dovecot Repo
>>> Version: 2.3.18
>>>
>>> Config for metrics, service and events:
>>>
>>
>> Can you include the full metrics output, please?
>
> Yes
>
> --
> support@mx ~ » curl "http://localhost:9900/metrics";
> # HELP process_start_time_seconds Timestamp of service start
> # TYPE process_start_time_seconds gauge
> process_start_time_seconds 1645743604
> # HELP dovecot_build Dovecot build information
> # TYPE dovecot_build info
> dovecot_build_info{version="2.3.18",revision="9dd8408c18"} 1
> # HELP dovecot_auth_success Total number of all events of this kind
> # TYPE dovecot_auth_success counter
> dovecot_auth_success_total 1843
> # HELP dovecot_auth_success_duration_seconds Total duration of all events of
> this kind
> # TYPE dovecot_auth_success_duration_seconds counter
> dovecot_auth_success_duration_seconds_total 1.424372
> # HELP dovecot_auth_failures Total number of all events of this kind
> # TYPE dovecot_auth_failures counter
> dovecot_auth_failures_total 1
> # HELP dovecot_auth_failures_duration_seconds Total duration of all events of
> this kind
> # TYPE dovecot_auth_failures_duration_seconds counter
> dovecot_auth_failures_duration_seconds_total 3.917099
> # HELP dovecot_auth_client_userdb Total number of all events of this kind
> # TYPE dovecot_auth_client_userdb counter
> dovecot_auth_client_userdb_total 30882
> # HELP dovecot_auth_client_userdb_duration_seconds Total duration of all
> events of this kind
> # TYPE dovecot_auth_client_userdb_duration_seconds counter
> dovecot_auth_client_userdb_duration_seconds_total 35.746910
> # HELP dovecot_auth_master Total number of all events of this kind
> # TYPE dovecot_auth_master counter
> dovecot_auth_master_total 1843
> # HELP dovecot_auth_master_duration_seconds Total duration of all events of
> this kind
> # TYPE dovecot_auth_master_duration_seconds counter
> dovecot_auth_master_duration_seconds_total 1.029899
> # HELP dovecot_client_connected Total number of all events of this kind
> # TYPE dovecot_client_connected counter
> dovecot_client_connected_total 10591
> # HELP dovecot_client_connected_duration_seconds Total duration of all events
> of this kind
> # TYPE dovecot_client_connected_duration_seconds counter
> dovecot_client_connected_duration_seconds_total 0.359288
> # HELP dovecot_client_disconnected Total number of all events of this kind
> # TYPE dovecot_client_disconnected counter
> dovecot_client_disconnected_total{reason="Connection closed"} 7173
> dovecot_client_disconnected_total{reason="Deinitializing"} 3418
> dovecot_client_disconnected_count 10591
> # HELP dovecot_client_disconnected_duration_seconds Total duration of all
> events of this kind
> # TYPE dovecot_client_disconnected_duration_seconds counter
> dovecot_client_disconnected_duration_seconds_total{reason="Connection
> closed"} 146.829269
> dovecot_client_disconnected_duration_seconds_total{reason="Deinitializing"}
> 72.953468
> dovecot_client_disconnected_duration_seconds_sum 219.782730
> # HELP dovecot_imap_command Total number of all events of this kind
> # TYPE dovecot_imap_command counter
> dovecot_imap_command_total{cmd_name="unknown"} 94
> dovecot_imap_command_total{cmd_name="unknown",tagged_reply_state="OK"} 75
> dovecot_imap_command_total{cmd_name="FETCH"} 720
> dovecot_imap_command_total{cmd_name="FETCH",tagged_reply_state="OK"} 720
> dovecot_imap_command_total{cmd_name="IDLE"} 5076
> dovecot_imap_command_total{cmd_name="IDLE",tagged_reply_state="OK"} 1555
> dovecot_imap_command_total{cmd_name="UID FETCH"} 2473
> dovecot_imap_command_total{cmd_name="UID FETCH",tagged_reply_state="OK"} 2473
> dovecot_imap_command_total{cmd_name="UID SEARCH"} 7402
> dovecot_imap_command_total{cmd_name="UID SEARCH",tagged_reply_state="OK"} 7400
> dovecot_imap_command_total{cmd_name="UID SEARCH",tagged_reply_state="BAD"} 2
> dovecot_imap_command_total{cmd_name="LOGOUT"} 1112
> dovecot_imap_command_total{cmd_name="LOGOUT",tagged_reply_state="OK"} 1112
> dovecot_imap_command_total{cmd_name="LIST"} 5942
> dovecot_imap_command_total{cmd_name="LIST",tagged_reply_state="OK"} 5942
> dovecot_imap_command_total{cmd_name="GETQUOTAROOT"} 88
> dovecot_imap_command_total{cmd_name="GETQUOTAROOT",tagged_reply_state="OK"} 88
> dovecot_imap_command_total{cmd_name="ID"} 600
> dovecot_imap_command_total{cmd_name="ID",tagged_reply_state="OK"} 600
> dovecot_imap_command_total{cmd_name="COMPRESS"} 591
> dovecot_imap_command_total{
Re: Dovecot 2.3.18 and OpenMetrics-Prometheus problem
Hi Aki,
> Am 25.02.2022 um 11:34 schrieb Aki Tuomi :
>
>>
>> On 25/02/2022 12:23 Christian Rößner wrote:
>>
>>
>> Hi,
>>
>> I have defined several metrics in Dovecot and activated the metrics service.
>> After the I tried to include this in Prometheues, but I get strange errors.
>> I gave up now after several hours in the hope that this might be a bug. So I
>> ask for help here:
>>
>> Distribution: Rocky 8
>> Official Dovecot Repo
>> Version: 2.3.18
>>
>> Config for metrics, service and events:
>>
>
> Can you include the full metrics output, please?
Yes
--
support@mx ~ » curl "http://localhost:9900/metrics";
# HELP process_start_time_seconds Timestamp of service start
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1645743604
# HELP dovecot_build Dovecot build information
# TYPE dovecot_build info
dovecot_build_info{version="2.3.18",revision="9dd8408c18"} 1
# HELP dovecot_auth_success Total number of all events of this kind
# TYPE dovecot_auth_success counter
dovecot_auth_success_total 1843
# HELP dovecot_auth_success_duration_seconds Total duration of all events of
this kind
# TYPE dovecot_auth_success_duration_seconds counter
dovecot_auth_success_duration_seconds_total 1.424372
# HELP dovecot_auth_failures Total number of all events of this kind
# TYPE dovecot_auth_failures counter
dovecot_auth_failures_total 1
# HELP dovecot_auth_failures_duration_seconds Total duration of all events of
this kind
# TYPE dovecot_auth_failures_duration_seconds counter
dovecot_auth_failures_duration_seconds_total 3.917099
# HELP dovecot_auth_client_userdb Total number of all events of this kind
# TYPE dovecot_auth_client_userdb counter
dovecot_auth_client_userdb_total 30882
# HELP dovecot_auth_client_userdb_duration_seconds Total duration of all events
of this kind
# TYPE dovecot_auth_client_userdb_duration_seconds counter
dovecot_auth_client_userdb_duration_seconds_total 35.746910
# HELP dovecot_auth_master Total number of all events of this kind
# TYPE dovecot_auth_master counter
dovecot_auth_master_total 1843
# HELP dovecot_auth_master_duration_seconds Total duration of all events of
this kind
# TYPE dovecot_auth_master_duration_seconds counter
dovecot_auth_master_duration_seconds_total 1.029899
# HELP dovecot_client_connected Total number of all events of this kind
# TYPE dovecot_client_connected counter
dovecot_client_connected_total 10591
# HELP dovecot_client_connected_duration_seconds Total duration of all events
of this kind
# TYPE dovecot_client_connected_duration_seconds counter
dovecot_client_connected_duration_seconds_total 0.359288
# HELP dovecot_client_disconnected Total number of all events of this kind
# TYPE dovecot_client_disconnected counter
dovecot_client_disconnected_total{reason="Connection closed"} 7173
dovecot_client_disconnected_total{reason="Deinitializing"} 3418
dovecot_client_disconnected_count 10591
# HELP dovecot_client_disconnected_duration_seconds Total duration of all
events of this kind
# TYPE dovecot_client_disconnected_duration_seconds counter
dovecot_client_disconnected_duration_seconds_total{reason="Connection closed"}
146.829269
dovecot_client_disconnected_duration_seconds_total{reason="Deinitializing"}
72.953468
dovecot_client_disconnected_duration_seconds_sum 219.782730
# HELP dovecot_imap_command Total number of all events of this kind
# TYPE dovecot_imap_command counter
dovecot_imap_command_total{cmd_name="unknown"} 94
dovecot_imap_command_total{cmd_name="unknown",tagged_reply_state="OK"} 75
dovecot_imap_command_total{cmd_name="FETCH"} 720
dovecot_imap_command_total{cmd_name="FETCH",tagged_reply_state="OK"} 720
dovecot_imap_command_total{cmd_name="IDLE"} 5076
dovecot_imap_command_total{cmd_name="IDLE",tagged_reply_state="OK"} 1555
dovecot_imap_command_total{cmd_name="UID FETCH"} 2473
dovecot_imap_command_total{cmd_name="UID FETCH",tagged_reply_state="OK"} 2473
dovecot_imap_command_total{cmd_name="UID SEARCH"} 7402
dovecot_imap_command_total{cmd_name="UID SEARCH",tagged_reply_state="OK"} 7400
dovecot_imap_command_total{cmd_name="UID SEARCH",tagged_reply_state="BAD"} 2
dovecot_imap_command_total{cmd_name="LOGOUT"} 1112
dovecot_imap_command_total{cmd_name="LOGOUT",tagged_reply_state="OK"} 1112
dovecot_imap_command_total{cmd_name="LIST"} 5942
dovecot_imap_command_total{cmd_name="LIST",tagged_reply_state="OK"} 5942
dovecot_imap_command_total{cmd_name="GETQUOTAROOT"} 88
dovecot_imap_command_total{cmd_name="GETQUOTAROOT",tagged_reply_state="OK"} 88
dovecot_imap_command_total{cmd_name="ID"} 600
dovecot_imap_command_total{cmd_name="ID",tagged_reply_state="OK"} 600
dovecot_imap_command_total{cmd_name="COMPRESS"} 591
dovecot_imap_command_total{cmd_name="COMPRESS",tagged_reply_state="OK"} 591
dovecot_imap_command_total{cmd_name="SELECT"} 3084
dovecot_imap_command_total{cmd_name="SELECT",tagged_reply_state="OK"} 3084
dovecot_imap_command_total{cmd_name="NOOP"} 225
dovecot_imap_comman
Re: Dovecot 2.3.18 and OpenMetrics-Prometheus problem
> On 25/02/2022 12:23 Christian Rößner wrote: > > > Hi, > > I have defined several metrics in Dovecot and activated the metrics service. > After the I tried to include this in Prometheues, but I get strange errors. I > gave up now after several hours in the hope that this might be a bug. So I > ask for help here: > > Distribution: Rocky 8 > Official Dovecot Repo > Version: 2.3.18 > > Config for metrics, service and events: > Can you include the full metrics output, please? Aki
Dovecot 2.3.18 and OpenMetrics-Prometheus problem
Hi,
I have defined several metrics in Dovecot and activated the metrics service.
After the I tried to include this in Prometheues, but I get strange errors. I
gave up now after several hours in the hope that this might be a bug. So I ask
for help here:
Distribution: Rocky 8
Official Dovecot Repo
Version: 2.3.18
Config for metrics, service and events:
-
metric auth_success {
exporter = log
filter = event=auth_request_finished AND success=yes
}
metric auth_failures {
exporter = log
filter = event=auth_request_finished AND NOT success=yes
}
metric auth_client_userdb {
filter = event=auth_client_userdb_lookup_finished
}
metric auth_master {
exporter = log
filter = event=auth_master_client_login_finished
}
metric client_connected {
filter = event=client_connection_connected
}
metric client_disconnected {
filter = event=client_connection_disconnected
group_by = reason
}
metric imap_command {
filter = event=imap_command_finished
group_by = cmd_name tagged_reply_state
}
metric lmtp_command {
filter = event=smtp_server_command_finished AND protocol=lmtp
group_by = cmd_name status_code duration:exponential:1:5:10
}
metric imap_hibernate {
filter = event=imap_client_hibernated
group_by = error
}
metric imap_unhibernate {
filter = event=imap_client_hibernated
group_by = reason error
}
metric mail_delivery {
filter = event=mail_delivery_finished
group_by = duration:exponential:1:5:10
}
metric sql_query {
filter = event=sql_query_finished
group_by = error_code
}
metric sieve_actions {
filter = event=sieve_action_finished
group_by = action_name error
}
metric managesieve {
filter = event=managesieve_command_finished
group_by = script_name
}
metric dict_lookups {
filter = event=dict_lookup_finished
group_by = driver error
}
service stats {
inet_listener http {
port = 9900
}
}
event_exporter log {
format = json
format_args = time-rfc3339
transport = log
}
-
Shortened curl output:
-
# HELP process_start_time_seconds Timestamp of service start
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1645743604
# HELP dovecot_build Dovecot build information
# TYPE dovecot_build info
dovecot_build_info{version="2.3.18",revision="9dd8408c18"} 1
# HELP dovecot_auth_success Total number of all events of this kind
# TYPE dovecot_auth_success counter
dovecot_auth_success_total 1669
# HELP dovecot_auth_success_duration_seconds Total duration of all events of
this kind
# TYPE dovecot_auth_success_duration_seconds counter
dovecot_auth_success_duration_seconds_total 1.317570
# HELP dovecot_auth_failures Total number of all events of this kind
# TYPE dovecot_auth_failures counter
dovecot_auth_failures_total 1
# HELP dovecot_auth_failures_duration_seconds Total duration of all events of
this kind
# TYPE dovecot_auth_failures_duration_seconds counter
...
# EOF
-
Prometheus yml config snippet:
-
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: dovecot_exporter
static_configs:
- targets:
- 172.30.10.1:9900
-
Prometheus (Docker stable latest) with debug level show these lines:
-
prometheus_1 | ts=2022-02-25T10:22:00.840Z caller=scrape.go:1307
level=debug component="scrape manager" scrape_pool=dovecot_exporter
target=http://172.30.10.1:9900/metrics msg="Append failed" err="expected label
name or left brace, got \"INVALID\""
-
Therefor the service is considered as being down.
I really have no ide, what I can do here.
Thanks for any help in advance
Christian Rößner
--
Rößner-Network-Solutions
Zertifizierter ITSiBe / CISO
Karl-Bröger-Str. 10, 36304 Alsfeld
Fax: +49 6631 78823409, Mobil: +49 171 9905345
USt-IdNr.: DE225643613, https://roessner.website
PGP fingerprint: 658D 1342 B762 F484 2DDF 1E88 38A5 4346 D727 94E5
