Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Hi again,
On Mon, Nov 03, 2014 at 04:47:28PM +0100, Evgeni Golov wrote:
Trial-and-error says, removing the following lines helps...
# Graylog2
destination d_graylog2 { udp(192.0.2.91 port(514) spoof_source(yes)); };
log { source(remotesrc); filter(f_cust_prod); destination(d_cust_prod);
destination(d_graylog2); };
log { source(remotesrc); filter(f_mail); filter(f_cust_mail);
destination(d_cust_mail); destination(d_graylog2); };
Is this by any possibility some buffering issue when forwarding
messages?
It seems to be the d_graylog2 part, not sending traffic to a remote UDP
destination solves the issue for us.
Do you have any more clues how to fix this properly?
Greets
Evgeni
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#703639: [Syslog-ng-maintainers] Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Evgeni == Evgeni Golov evg...@debian.org writes: Evgeni Hi, Evgeni On Fri, Mar 22, 2013 at 03:59:11PM +0100, Gergely Nagy wrote: * Fixing the configuration and reloading gets things back in order, no matter how many times messages were duplicated before. Evgeni I have a heavily customized config, which does not throw any errors, Evgeni but triggers the issue on a wheezy box. Evgeni The config is for a central log-server, which gets syslog via UDP from Evgeni quite a few hosts and sorts these accordingly. Every day at logrotate Evgeni a SIGHUP is issued and my /var/log gets full. The ratio is about 1 real Evgeni message to 3000 (yes, three thousand!) duplicates :/ Evgeni A real restart solves the issue. Evgeni I hope this is helful for you to track down the issue. I didn't have time to reproduce the problem yet, but I'm going to forward it upstream. Are you using the wheezy version of syslog-ng? If so, can you try the 3.5.6 backport from wheezy-backports, and see if the problem persists? (A lot of things that may result in this kind of behaviour have been fixed between 3.3.5 in wheezy and 3.5.6 in backports.) -- |8] signature.asc Description: PGP signature
Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Hi,
On Fri, Mar 22, 2013 at 03:59:11PM +0100, Gergely Nagy wrote:
* Fixing the configuration and reloading gets things back in order, no
matter how many times messages were duplicated before.
I have a heavily customized config, which does not throw any errors,
but triggers the issue on a wheezy box.
The config is for a central log-server, which gets syslog via UDP from
quite a few hosts and sorts these accordingly. Every day at logrotate
a SIGHUP is issued and my /var/log gets full. The ratio is about 1 real
message to 3000 (yes, three thousand!) duplicates :/
A real restart solves the issue.
I hope this is helful for you to track down the issue.
@version: 3.3
# First, set some global options.
options {
create_dirs(yes);
dir_perm(0755);
chain_hostnames(off);
flush_lines(0);
keep_hostname(yes);
};
#
# This is the default behavior of sysklogd package
# Logs may come from unix stream, but not from another machine.
#
source localsrc { unix-dgram(/dev/log);
internal();
# Keep a log socket within the postfix chroot
unix-stream(/var/spool/postfix/dev/log);
};
#
# If you wish to get logs from remote machine you should uncomment
# this and comment the above source line.
#
source remotesrc { udp(); };
# After that set destinations.
# First some standard logfile
#
destination authlog { file(/var/log/auth.log owner(root) group(adm)
perm(0640)); };
destination syslogfile { file(/var/log/syslog owner(root) group(adm)
perm(0640)); };
destination cron { file(/var/log/cron.log owner(root) group(adm)
perm(0640)); };
destination daemon { file(/var/log/daemon.log owner(root) group(adm)
perm(0640)); };
destination kern { file(/var/log/kern.log owner(root) group(adm)
perm(0640)); };
destination lpr { file(/var/log/lpr.log owner(root) group(adm)
perm(0640)); };
destination mail { file(/var/log/mail.log owner(root) group(adm)
perm(0640)); };
destination user { file(/var/log/user.log owner(root) group(adm)
perm(0640)); };
destination uucp { file(/var/log/uucp.log owner(root) group(adm)
perm(0640)); };
# This files are the log come from the mail subsystem.
#
destination mailinfo { file(/var/log/mail.info owner(root) group(adm)
perm(0640)); };
destination mailwarn { file(/var/log/mail.warn owner(root) group(adm)
perm(0640)); };
destination mailerr { file(/var/log/mail.err owner(root) group(adm)
perm(0640)); };
# Logging for INN news system
#
destination newscrit { file(/var/log/news/news.crit owner(root)
group(adm) perm(0640)); };
destination newserr { file(/var/log/news/news.err owner(root) group(adm)
perm(0640)); };
destination newsnotice { file(/var/log/news/news.notice owner(root)
group(adm) perm(0640)); };
# Some `catch-all' logfiles.
#
destination debug { file(/var/log/debug owner(root) group(adm)
perm(0640)); };
destination messages { file(/var/log/messages owner(root) group(adm)
perm(0640)); };
# root's console.
#
destination console { usertty(root); };
# Virtual console.
#
destination console_all { file(/dev/tty8); };
# The named pipe /dev/xconsole is for the nsole' utility. To use it,
# you must invoke nsole' with the -file' option:
#
#$ xconsole -file /dev/xconsole [...]
#
# destination xconsole { pipe(/dev/xconsole); };
destination ppp { file(/var/log/ppp.log owner(root) group(adm)
perm(0640)); };
destination switches { file(/var/log/switches.log owner(root) group(adm)
perm(0640)); };
destination term { file(/var/log/term.log owner(root) group(adm)
perm(0640)); };
# normal single files
destination nt { file(/var/log/nt.log owner(root) group(adm) perm(0640));
};
destination cust_apache { file(/var/log/cust/apache.log owner(root)
group(adm) perm(0640)); };
destination cust_postgres { file(/var/log/cust/postgres.log owner(root)
group(adm) perm(0640)); };
# by-host log file
destination d_by_host { file(/var/log/hosts/$FULLHOST/syslog-$YEAR$MONTH$DAY
owner(root) group(adm) perm(0640)); };
##
## Remote Destinations
destination d_logger01 { udp(192.0.2.111 port(514)); };
destination d_logger02 { udp(192.0.2.112 port(514)); };
# Here's come the filter options. With this rules, we can set which
# message go where.
filter f_switches { facility(local5); };
filter f_term { host(^term[0-9]); };
filter f_esx { host(^esx[0-9]*); };
filter f_nt { host(^(windows|fenster)); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog {
not facility(auth, authpriv, news, mail)
and not level(debug)
and not host(^(windows|esx[0-9]*|term[0-9]))
and not (host(^(api|web)[ab][0-9][0-9][0-9]) and program(apache2))
and not (host(^dbsrv) and program(postgres));
};
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail)
and not (
message(connect from localhost) or
message(lost
Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Hi,
On Mon, Nov 03, 2014 at 12:14:02PM +0100, Evgeni Golov wrote:
I have a heavily customized config, which does not throw any errors,
but triggers the issue on a wheezy box.
Trial-and-error says, removing the following lines helps...
# Graylog2
destination d_graylog2 { udp(192.0.2.91 port(514) spoof_source(yes)); };
log { source(remotesrc); filter(f_cust_prod); destination(d_cust_prod);
destination(d_graylog2); };
log { source(remotesrc); filter(f_mail); filter(f_cust_mail);
destination(d_cust_mail); destination(d_graylog2); };
Is this by any possibility some buffering issue when forwarding
messages?
Greets
Evgeni
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Control: found -1 3.3.5-4
Control: found -1 3.3.6-2
Cye Stoner cye.sto...@echostar.com writes:
When syslog-ng is sent a SIGHUP with the previous configurationin place,
the following errors are thrown to /var/log/messages(in duplicate)
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error initializing new configuration, reverting to old config;
Error initializing new configuration, reverting to old config;
Thanks for the report, I reproduced it with the latest git master of
syslog-ng 3.3 too, so the problem applies to the latest 3.3 as well.
While I have not found a fix yet, I did notice two things:
* Fixing the configuration and reloading gets things back in order, no
matter how many times messages were duplicated before.
* Running in foreground with internal logs printed to stderr, the
problem does not appear.
I'll try to narrow this down, and provide a fix ASAP.
I'm using the following configuration to test with:
@version: 3.3
source s_internal {
internal();
};
destination d_all {
file(/tmp/all-messages.log);
};
log {
source(s_internal);
#source(s_does_not_exist);
destination(d_all);
};
--
|8]
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#703639: kill -HUP $(cat /var/run/syslog-ng.pid) can cause duplicate logging issues
Package: syslog-ng
Version: 3.3.5-2
When executing the following command to reload the syslog-ng
configuration file:
# kill -HUP $(cat /var/run/syslog-ng.pid)
Duplicate logging issues can show up if a source is listed in the
configuration file but is not defined previously. The following log
definition would cause the issue:
# ... Other logs defined above.
log
{
source(s_sys_does_not_exist);
destination(d_sys);
};
When syslog-ng is sent a SIGHUP with the previous configurationin place,
the following errors are thrown to /var/log/messages(in duplicate)
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error initializing new configuration, reverting to old config;
Error initializing new configuration, reverting to old config;
At this point, all of the previously configured syslog-ng logs will be
sent in duplicate as well.This does not apply to just /var/log/messages
but every custom log defined.
If syslog-ng is sent SIGHUP a 2nd time without fixing the configuration
file,everything is logged 4 times (all at the same timestamp).
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error in configuration, unresolved source reference;
source='s_sys_does_not_exist'
Error initializing new configuration, reverting to old config;
Error initializing new configuration, reverting to old config;
Error initializing new configuration, reverting to old config;
Error initializing new configuration, reverting to old config;
The original syntax error is not caught through the use of syslog-ng
-s but does prevent syslog-ng from starting if is stopped and then
restarted.
Expected behavior:
1. The error should not cause log messages to happen in duplicate.
Logging should continue identically to how it did before the SIGHUP was
issued.
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
