Hi all.
I'm migrating from Zabbix to Icinga2, using the Icinga2 Agent in most of my
servers. However, I am getting a lot of delayed checks and I do not know
what to do to fix it. I've posted my question at monitoring-portal.org [1]
and haven't got much help there, so decided to post it here as well.
Troubleshooting steps, which did not correct the problem:
- restarted all services on the master and on the monitored host
- “Force check” does not work; the service is not checked again after
the force; it falls back into Late status
My Environment:
icinga2-ido-pgsql
icingaweb2 2.6.2-1
icinga2 2.10.2-1
- Version used ( icinga2 --version ): version: r2.10.2-1
- Operating System and version: Ubuntu 18.04.1 LTS (Bionic Beaver)
- Enabled features ( icinga2 feature list ): Enabled features: api
checker command ido-pgsql mainlog notification perfdata
- Icinga Web 2 version and modules (System - About): 2.6.2
- Config validation ( icinga2 daemon -C ):
[2018-12-20 13:23:22 +1300] information/cli: Icinga application loader
(version: r2.10.2-1)
[2018-12-20 13:23:22 +1300] information/cli: Loading configuration file(s).
[2018-12-20 13:23:22 +1300] information/ConfigItem: Committing config
item(s).
[2018-12-20 13:23:22 +1300] information/ApiListener: My API identity:
icinga.datacentre.example.com
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule 'ping6' (in
/etc/icinga2/conf.d/services.conf: 34:1-34:21) for type 'Service' does not
match anywhere! [2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule 'ssh' (in
/etc/icinga2/conf.d/services.conf: 47:1-47:19) for type 'Service' does not
match anywhere! [2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule 'apt' (in
/etc/icinga2/zones.d/master/services.conf: 51:1-51:19) for type 'Service'
does not match anywhere! [2018-12-20 13:23:22 +1300] warning/ApplyRule:
Apply rule
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule '' (in
/etc/icinga2/zones.d/master/services.conf: 69:1-69:66) for type 'Service'
does not match anywhere! [2018-12-20 13:23:22 +1300] warning/ApplyRule:
Apply rule
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule '' (in
/etc/icinga2/zones.d/master/services.conf: 92:1-92:68) for type 'Service'
does not match anywhere! [2018-12-20 13:23:22 +1300] warning/ApplyRule:
Apply rule
[2018-12-20 13:23:22 +1300] warning/ApplyRule: Apply rule '' (in
/etc/icinga2/zones.d/master/services.conf: 191:1-191:86) for type 'Service'
does not match anywhere! [2018-12-20 13:23:22 +1300]
information/ConfigItem: Instantiated 1 ScheduledDowntime. [2018-12-20
13:23:22 +1300] information/ConfigItem: Instantiated 92 Services.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
IcingaApplication. [2018-12-20 13:23:22 +1300] information/ConfigItem:
Instantiated 9 Hosts. [2018-12-20 13:23:22 +1300] information/ConfigItem:
Instantiated 1 FileLogger. [2018-12-20 13:23:22 +1300]
information/ConfigItem: Instantiated 2 NotificationCommands. [2018-12-20
13:23:22 +1300] information/ConfigItem: Instantiated 176 Notifications.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
NotificationComponent. [2018-12-20 13:23:22 +1300] information/ConfigItem:
Instantiated 4 HostGroups. [2018-12-20 13:23:22 +1300]
information/ConfigItem: Instantiated 1 ApiListener. [2018-12-20 13:23:22
+1300] information/ConfigItem: Instantiated 1 Downtime. [2018-12-20
13:23:22 +1300] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1 Comment.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
CheckerComponent. [2018-12-20 13:23:22 +1300] information/ConfigItem:
Instantiated 11 Zones. [2018-12-20 13:23:22 +1300] information/ConfigItem:
Instantiated 1 ExternalCommandListener.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
ScheduledDowntime.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 92
Services.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
IcingaApplication.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 9 Hosts.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
FileLogger.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 2
NotificationCommands.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 176
Notifications.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
NotificationComponent.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 4
HostGroups.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
ApiListener.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1 Downtime.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
PerfdataWriter.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1 Comment.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
CheckerComponent.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 11 Zones.
[2018-12-20 13:23:22 +1300] information/ConfigItem: Instantiated 1
ExternalCommandListener.
zones.conf:
object Endpoint "icinga.datacentre.example.com" {
}
object Zone "master" {
endpoints = [ "icinga.datacentre.example.com" ]
}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}
services.conf (original file from /etc/icinga2/conf.d/services.conf):
// Ping Checkapply Service "Ping" {
check_command = "ping4"
assign where host.address // check is executed on the master node
}
// System Loadapply Service "System Load" {
check_command = "load"
command_endpoint = host.vars.client_endpoint // Check executed on client node
assign where host.vars.client_endpoint
}
// System Process Countapply Service "Process" {
check_command = "procs"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Logged in User Countapply Service "Users" {
check_command = "users"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Disk Usage Checkapply Service "Disk" {
check_command = "disk"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Disk Usage Check for Specific Partitionapply Service for (disk =>
config in host.vars.local_disks) {
check_command = "disk"
vars += config
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Icinga 2 Service Checkapply Service "Icinga2 Service" {
check_command = "icinga"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Apt service checkapply Service "apt" {
import "generic-service"
check_command = "apt"
display_name = "apt-get updates"
assign where host.vars.os == "ubuntu"
command_endpoint = host.vars.client_endpoint
}
// YUM service checkapply Service "yum" {
import "generic-service"
check_command = "yum"
display_name = "yum updates"
assign where host.vars.os == "centos"
command_endpoint = host.vars.client_endpoint
}
// TCP Port Checkapply Service for (tcp_port => config in
host.vars.local_tcp_port) {
check_command = "tcp"
vars += config
display_name = + vars.service_name + " - " + vars.port_number
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
//
// API check commands
//
// Apache VirtualHost Checkapply Service for (http_vhost => config in
host.vars.local_http_vhosts) {
check_command = "http"
vars += config
display_name = "API Status"
// command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
Checking the "retry interval value: icinga2 object list --type Service
Object 'ns2.datacentre.example.com!/' of type 'Service':
% declared in '/etc/icinga2/zones.d/master/services.conf', lines 41:1-41:59
* __name = "ns2.datacentre.example.com!/"
* action_url = ""
* check_command = "disk"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
43:3-43:24
* check_interval = 60
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 28:3-28:21
* check_period = ""
* check_timeout = null
* command_endpoint = "ns2.datacentre.example.com"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
45:3-45:46
* display_name = "/"
* enable_active_checks = true
* enable_event_handler = true
* enable_flapping = false
* enable_notifications = true
* enable_passive_checks = true
* enable_perfdata = true
* event_command = ""
* flapping_threshold = 0
* flapping_threshold_high = 30
* flapping_threshold_low = 25
* groups = [ ]
* host_name = "ns2.datacentre.example.com"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
41:1-41:59
* icon_image = ""
* icon_image_alt = ""
* max_check_attempts = 5
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 27:3-27:24
* name = "/"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
41:1-41:59
* notes = ""
* notes_url = ""
* package = "_etc"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
41:1-41:59
* retry_interval = 30
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 29:3-29:22
* source_location
* first_column = 1
* first_line = 41
* last_column = 59
* last_line = 41
* path = "/etc/icinga2/zones.d/master/services.conf"
* templates = [ "/", "generic-service" ]
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
41:1-41:59
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 26:1-26:34
* type = "Service"
* vars
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
44:3-44:16
* disk_partitions = "/"
* volatile = false
* zone = "master"
% = modified in '/etc/icinga2/zones.d/master/services.conf', lines
41:1-41:59
Why are those checks taking so long?
[1] https://monitoring-portal.org/t/icinga2-delayed-checks/5420/7
-
Kind regards,
Lucas Possamai
_______________________________________________
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
