Hi everyone,
I'm fairly new to monitoring and set up an incinga2 installation with a
master and a satellite. It's currently just running the default checks
With most of the default configuration. The only changes I did manually
were the removal of the downtimes config (for some reasons the config in
/var/lib... remained, I deleted it manually) and adding my mail address
to the users.conf.
The checks are working and I receive notifications if something goes
wrong, but everytime a notification is sent an icinga process consumes
100% of my master-node CPU until I manually restart the service.
I activated debuglog but still can't see a reason for this behavior.
Both nodes are Debian Jessie with the debmon repositories enabled:
Package icinga2:
i 2.4.7-1~debmon8+1 debmon-jessie
Is this a configuration mistake (checkconfig has no error output) or a
bug and worth to be reported?
Greetings
Thore
P.S.:
The output from icinga2 troubleshoot while Icinga was in "rage mode"
after a swap warning:
Started collection at 2016-04-25 15:54:00
====================================================
============== GENERAL INFORMATION ==============
Application version: r2.4.7-1
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Enabled features:
api checker command debuglog ido-mysql mainlog notification
Disabled features:
compatlog gelf graphite icingastatus livestatus opentsdb perfdata
statusdata syslog
============== OBJECT INFORMATION ==============
Checking object file from /var/cache/icinga2/icinga2.debug
Found the 132 objects:
Type : Count
ApiListener : 1
ApiUser : 1
CheckCommand : 54
CheckerComponent : 1
Downtime : 3
Endpoint : 2
ExternalCommandListener : 1
FileLogger : 2
Host : 2
HostGroup : 2
IcingaApplication : 1
IdoMysqlConnection : 1
Notification : 25
NotificationCommand : 2
NotificationComponent : 1
Service : 23
ServiceGroup : 3
TimePeriod : 3
User : 1
UserGroup : 1
Zone : 2
The objects origins are:
/etc/icinga2/conf.d/api-users.conf
/etc/icinga2/conf.d/app.conf
/etc/icinga2/conf.d/apt.conf
/etc/icinga2/conf.d/commands.conf
/etc/icinga2/conf.d/groups.conf
/etc/icinga2/conf.d/hosts.conf
/etc/icinga2/conf.d/notifications.conf
/etc/icinga2/conf.d/services.conf
/etc/icinga2/conf.d/timeperiods.conf
/etc/icinga2/conf.d/users.conf
/etc/icinga2/features-enabled/api.conf
/etc/icinga2/features-enabled/checker.conf
/etc/icinga2/features-enabled/command.conf
/etc/icinga2/features-enabled/debuglog.conf
/etc/icinga2/features-enabled/ido-mysql.conf
/etc/icinga2/features-enabled/mainlog.conf
/etc/icinga2/features-enabled/notification.conf
/etc/icinga2/repository.d/endpoints/mydomain.tld.conf
/etc/icinga2/repository.d/hosts/mydomain.tld.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/apt.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/disk %2F.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/disk.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/http.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/icinga.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/load.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/ping4.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/procs.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/ssh.conf
/etc/icinga2/repository.d/hosts/mydomain.tld/users.conf
/etc/icinga2/repository.d/zones/mydomain.tld.conf
/etc/icinga2/zones.conf
/usr/share/icinga2/include/command-icinga.conf
/usr/share/icinga2/include/command-plugins.conf
/var/lib/icinga2/api/packages/_api/mon0-1460855463-1/conf.d/downtimes/mon0.mydomain.tld!load!mon0-1461283252-4.conf
/var/lib/icinga2/api/packages/_api/mon0-1460855463-1/conf.d/downtimes/mon0.mydomain.tld!load!mon0-1461369629-0.conf
/var/lib/icinga2/api/packages/_api/mon0-1460855463-1/conf.d/downtimes/mon0.mydomain.tld!load!mon0-1461456050-0.conf
db_ido-itl.conf
============== LOGS AND CRASH REPORTS ==============
Getting the last 20 lines of 2 FileLogger objects.
Logger debug-file at path: /var/log/icinga2/debug.log
[begin: '/var/log/icinga2/debug.log' line: 249583]
# [2016-04-25 15:53:57 +0200] debug/IdoMysqlConnection: Query: COMMIT
# [2016-04-25 15:53:57 +0200] debug/IdoMysqlConnection: Query: BEGIN
# [2016-04-25 15:53:57 +0200] debug/CheckerComponent: Executing check
for 'mon0.mydomain.tld!swap'
# [2016-04-25 15:53:57 +0200] notice/ApiListener: Relaying
'event::SetNextCheck' message
# [2016-04-25 15:53:57 +0200] notice/Process: Running command
'/usr/lib/nagios/plugins/check_swap' '-c' '25%' '-w' '50%': PID 19729
# [2016-04-25 15:53:57 +0200] debug/CheckerComponent: Check finished
for object 'mon0.mydomain.tld!swap'
# [2016-04-25 15:53:57 +0200] notice/Process: PID 19729
('/usr/lib/nagios/plugins/check_swap' '-c' '25%' '-w' '50%') terminated
with exit code 1
# [2016-04-25 15:53:57 +0200] debug/DbEvents: add checkable check
history for 'mon0.mydomain.tld!swap'
# [2016-04-25 15:53:57 +0200] notice/ApiListener: Relaying
'event::CheckResult' message
# [2016-04-25 15:53:58 +0200] debug/IdoMysqlConnection: Query: UPDATE
icinga_servicestatus SET acknowledgement_type = '0',
active_checks_enabled = '1', check_command = 'swap', check_source =
'mon0.mydomain.tld', check_type = '0', current_check_attempt = '1',
current_notification_number = '1', current_state = '1',
endpoint_object_id = 1, event_handler = '', event_handler_enabled =
'1', execution_time = '0.0065529346466064453', flap_detection_enabled
= '0', has_been_checked = '1', instance_id = 1, is_flapping = '0',
is_reachable = '1', last_check = FROM_UNIXTIME(1461592437),
last_hard_state = '1', last_hard_state_change =
FROM_UNIXTIME(1461592407), last_notification =
FROM_UNIXTIME(1461592407), last_state_change =
FROM_UNIXTIME(1461592257), last_time_ok = FROM_UNIXTIME(1461592197),
last_time_warning = FROM_UNIXTIME(1461592437), latency = '0',
long_output = '', max_check_attempts = '5', next_check =
FROM_UNIXTIME(1461592497), next_notification = FROM_UNIXTIME(
1461593902), normal_check_interval = '1', notifications_enabled =
'1', original_attributes = 'null', output = 'SWAP WARNING - 48% free
(122 MB out of 256 MB) ', passive_checks_enabled = '1',
percent_state_change = '0', perfdata = 'swap=122MB;128;64;0;256',
problem_has_been_acknowledged = '0', process_performance_data = '1',
retry_check_interval = '0.5', scheduled_downtime_depth = '0',
service_object_id = 80, should_be_scheduled = '1', state_type = '1',
status_update_time = FROM_UNIXTIME(1461592437) WHERE service_object_id = 80
# [2016-04-25 15:53:58 +0200] debug/IdoMysqlConnection: Query: COMMIT
# [2016-04-25 15:53:58 +0200] debug/IdoMysqlConnection: Query: BEGIN
# [2016-04-25 15:53:58 +0200] debug/CheckerComponent: Executing check
for 'mon0.mydomain.tld!ping4'
# [2016-04-25 15:53:58 +0200] notice/ApiListener: Relaying
'event::SetNextCheck' message
# [2016-04-25 15:53:58 +0200] notice/Process: Running command
'/usr/lib/nagios/plugins/check_ping' '-4' '-H' '127.0.0.1' '-c'
'200,15%' '-w' '100,5%': PID 19730
# [2016-04-25 15:53:58 +0200] debug/CheckerComponent: Check finished
for object 'mon0.mydomain.tld!ping4'
# [2016-04-25 15:53:59 +0200] notice/JsonRpcConnection: Received
'event::SetNextCheck' message from 'mydomain.tld'
# [2016-04-25 15:53:59 +0200] notice/ApiListener: Relaying
'event::SetNextCheck' message
# [2016-04-25 15:53:59 +0200] debug/IdoMysqlConnection: Query: COMMIT
# [2016-04-25 15:53:59 +0200] debug/IdoMysqlConnection: Query: BEGIN
[end: '/var/log/icinga2/debug.log' line: 249603]
Logger main-log at path: /var/log/icinga2/icinga2.log
[begin: '/var/log/icinga2/icinga2.log' line: 2440]
# [2016-04-25 15:50:07 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2485/15min);
# [2016-04-25 15:50:22 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2520/15min);
# [2016-04-25 15:50:37 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2556/15min);
# [2016-04-25 15:50:52 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2594/15min);
# [2016-04-25 15:51:07 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2631/15min);
# [2016-04-25 15:51:22 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.43333/s (146/min 731/5min 2666/15min);
# [2016-04-25 15:51:37 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 732/5min 2703/15min);
# [2016-04-25 15:51:52 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 732/5min 2741/15min);
# [2016-04-25 15:52:07 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 732/5min 2778/15min);
# [2016-04-25 15:52:22 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 732/5min 2813/15min);
# [2016-04-25 15:52:37 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 733/5min 2850/15min);
# [2016-04-25 15:52:52 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 733/5min 2888/15min);
# [2016-04-25 15:53:07 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.45/s (147/min 733/5min 2925/15min);
# [2016-04-25 15:53:22 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.46667/s (148/min 733/5min 2961/15min);
# [2016-04-25 15:53:27 +0200] information/Checkable: Checking for
configured notifications for object 'mon0.mydomain.tld!swap'
# [2016-04-25 15:53:27 +0200] information/Notification: Sending
notification 'mon0.mydomain.tld!swap!mail-icingaadmin' for user
'icingaadmin'
# [2016-04-25 15:53:27 +0200] information/Notification: Completed
sending notification 'mon0.mydomain.tld!swap!mail-icingaadmin' for
checkable 'mon0.mydomain.tld!swap' and user 'icingaadmin'.
# [2016-04-25 15:53:37 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.48333/s (149/min 735/5min 2984/15min);
# [2016-04-25 15:53:37 +0200] information/ConfigObject: Dumping program
state to file '/var/lib/icinga2/icinga2.state'
# [2016-04-25 15:53:52 +0200] information/IdoMysqlConnection: Query
queue items: 0, query rate: 2.48333/s (149/min 735/5min 2197/15min);
[end: '/var/log/icinga2/icinga2.log' line: 2460]
No crash logs found in /var/log/icinga2/crash/
============== CONFIGURATION FILES ==============
A collection of important configuration files follows, please make sure
to remove any sensitive data such as credentials, internal company
names, etc
[begin: '/etc/icinga2/icinga2.conf']
# /**
# * Icinga 2 configuration file
# * - this is where you define settings for the Icinga application
including
# * which hosts/services to check.
# *
# * For an overview of all available configuration options please refer
# * to the documentation that is distributed as part of Icinga 2.
# */
#
# /**
# * The constants.conf defines global constants.
# */
# include "constants.conf"
#
# /**
# * The zones.conf defines zones for a cluster setup.
# * Not required for single instance setups.
# */
# include "zones.conf"
#
# /**
# * The Icinga Template Library (ITL) provides a number of useful
templates
# * and command definitions.
# * Common monitoring plugin command definitions are included separately.
# */
# include <itl>
# include <plugins>
# // include <plugins-contrib>
#
# /**
# * The features-available directory contains a number of configuration
# * files for features which can be enabled and disabled using the
# * icinga2 feature enable / icinga2 feature disable CLI commands.
# * These commands work by creating and removing symbolic links in
# * the features-enabled directory.
# */
# include "features-enabled/*.conf"
#
# /**
# * The repository.d directory contains all configuration objects
# * managed by the 'icinga2 repository' CLI commands.
# */
# include_recursive "repository.d"
#
# /**
# * Although in theory you could define all your objects in this file
# * the preferred way is to create separate directories and files in
the conf.d
# * directory. Each of these files must have the file extension ".conf".
# */
# include_recursive "conf.d"
[end: '/etc/icinga2/icinga2.conf']
[begin: '/etc/icinga2/zones.conf']
# /*
# * Generated by Icinga 2 node setup commands
# * on 2016-04-17 04:13:32 +0200
# */
#
# object Endpoint NodeName {
# }
#
# object Zone ZoneName {
# endpoints = [ NodeName ];
# }
#
#
[end: '/etc/icinga2/zones.conf']
====================================================
Finished collection at 2016-04-25 15:54:00
Took 0.15338993072509766 seconds
_______________________________________________
icinga-users mailing list
[email protected]
https://lists.icinga.org/mailman/listinfo/icinga-users
