# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
### DEAL WITH REMOTE MESSAGES ########################################
$DirGroup secopsr
$FileGroup secopsr
$umask 0000
$DirCreateMode 0754
$FileCreateMode 0644
$template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
:inputname , isequal , "imudp" ?RemoteClient
:inputname , isequal , "imtcp" ?RemoteClient
:fromhost-ip , !isequal , "127.0.0.1" stop
########################################################################
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
### PUSH LOCAL MESSAGES TO REMOTE ####################################
kern.* @unixloghost
*.info;mail.none;authpriv.none;cron.none @unixloghost
authpriv.* @unixloghost
*.emerg @unixloghost
local7.* @unixloghost
########################################################################
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
-----Original Message-----
From: David Lang <da...@lang.hm>
Sent: Thursday, August 01, 2019 7:12 PM
To: Singh, Radesh <radesh_si...@csx.com>
Cc: David Lang <da...@lang.hm>; Singh, Radesh via rsyslog
<rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages
yep, each of those errors is probably a lost log in sysctl you can increase the
buffer size, but you probably also need to tune rsyslog to handle logs faster
if you can post your config it would help
enabling impstats would help understand what's going on inside rsyslog
(including showing the counts of the number of messages it sees)
David Lang
On Thu, 1 Aug 2019, Singh, Radesh wrote:
> Date: Thu, 1 Aug 2019 22:48:40 +0000
> From: "Singh, Radesh" <radesh_si...@csx.com>
> To: David Lang <da...@lang.hm>,
> "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I may have found more clues...
>
> The number of errors related to UDP are really high...
>
> netstat -su
> IcmpMsg:
> InType0: 6
> InType3: 980584
> InType8: 28959
> InType11: 688
> OutType0: 28959
> OutType3: 987041
> OutType8: 6
> Udp:
> 140571327944 packets received
> 332780 packets to unknown port received.
> 151716349245 packet receive errors
> 207450667309 packets sent
> 151716213631 receive buffer errors
> 287 send buffer errors
> InCsumErrors: 135614
> UdpLite:
> IpExt:
> InBcastPkts: 446852
> InOctets: 73907716648604
> OutOctets: 103321724708971
> InBcastOctets: 130548312
> InNoECTPkts: 356643654014
> InECT0Pkts: 810
>
> I compared this to another log host, and while the other log host has errors,
> the numbers are nowhere as large.
>
> Radesh
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Thursday, August 01, 2019 5:07 PM
> To: David Lang <da...@lang.hm>; Singh, Radesh via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> Thank you for your response.
>
> The firewall logs don't get rotated.
> The logs from the firewall get this rule applied to them:
> $template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
> :inputname , isequal , "imudp" ?RemoteClient :inputname , isequal ,
> "imtcp" ?RemoteClient
>
> So we usually have some messages from the previous day, but that isn't a
> concern to us.
>
> At first I suspected that logrotation might be the culprit, but the issue is
> occurring all throughout the day, and our logs usually rotate between 0000
> and 0500 each day.
> The only logrotation config file that interacts with rsyslog is our syslog
> config.
> In that config file (/etc/logrotate.d/syslog), I see the following config:
>
> syslog
> /var/log/cron
> /var/log/maillog
> /var/log/messages
> /var/log/secure
> /var/log/spooler
> /var/log/boot.log
> {
> missingok
> sharedscripts
> notifempty
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null ||
> true
> endscript
> }
>
> So, we are sending HUP, not using the copytruncate option.
>
> I wonder if we are having issue with UDP buffers.
> I haven't had to investigate that before.
>
> Could you provide any suggestions on how you'd go about that on RHEL7?
> A quick google return several results... one of which suggests watching
> /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run
> across this before, wondering if that is the best approach.
>
> Thanks,
>
> Radesh
>
>
> -----Original Message-----
> From: David Lang <da...@lang.hm>
> Sent: Thursday, August 01, 2019 4:45 PM
> To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <radesh_si...@csx.com>
> Subject: [E]Re: [rsyslog] Missing messages
>
> how are you rotating your logs? are you sending rsyslog a HUP or are you
> doing a /etc/init.d/reload (which is a full restart)?
>
> are you using copytruncate as you rotate your logs?
>
> these are the most common issues.
>
> if you are sending via UDP, check your OS UDP buffers, if they are filling
> up, the packets will be dropped before they get to rsyslog.
>
> 8.24 is pretty old (~2.5 years), but it's unlikely to be the root
> cause of the problem
>
> David Lang
>
>
> On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
>
>> Date: Thu, 1 Aug 2019 19:13:11 +0000
>> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: "Singh, Radesh" <radesh_si...@csx.com>
>> Subject: Re: [rsyslog] Missing messages
>>
>> Just to ensure that I don't too much confusion.
>> I made the following statement:
>>
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I should have said:
>> Messages that aren't getting logged leave no trace anywhere, EXCEPT the
>> packet capture.
>>
>> The missing messages are being captured in the tcpdump, so when my firewall
>> guy sends me a sample of what he sent, if I'm running a packet capture, I've
>> validated that the server has received the data, even if rsyslog isn't
>> writing it down.
>>
>> Thanks again.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Singh,
>> Radesh via rsyslog
>> Sent: Thursday, August 01, 2019 3:08 PM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Singh, Radesh <radesh_si...@csx.com>
>> Subject: [E][rsyslog] Missing messages
>>
>> List,
>>
>> Our firewall guys reported that they're missing messages.
>> They've got devices pointed at a dedicated rsyslog server ( they are the
>> only folks using it ) and they are seeing cases were they are sending
>> messages, I am receiving the messages, but the messages aren't getting
>> written to the logs.
>> The missing messages aren't from any particular source, the issue seems to
>> occur all throughout the day, and in the case of the messages, we expect to
>> see when sessions are built and torn down.
>> There are times the entire session is captured, times when part of the
>> session is missed, and times when the entire session is missed.
>>
>> I know the message is reaching the server, because I ran a packet capture
>> and see the data.
>> I turned on debug and captured a pretty healthy dump of data.
>> BTW, I set: RSYSLOG_DEBUG to "Debug".
>>
>> Messages that are getting logged, are visible in the debug logs.
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I'm running:
>> Name : rsyslog
>> Version : 8.24.0
>> Release : 16.el7
>> Architecture: x86_64
>>
>> On RHEL 7.5.
>>
>> Do you guys have any ideas of things I might try to get more info?
>>
>> Thank you,
>>
>> Radesh
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-M
>> O
>> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi
>> 4
>> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH
>> 2
>> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr
>> 5
>> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT0
>> 9
>> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%
>> 3 A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
>> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kk
>> l
>> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPob
>> E
>> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJ
>> s
>> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5
>> q
>> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNV
>> g
>> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26v
>> z
>> ubCvnjmiVvAjt-g/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>> THAT.
>> __________________________________________
>> E - EXTERNAL SENDER
>> Use discretion when clicking links, opening attachments, or replying.
>> __________________________________________
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKL
>> r
>> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaA
>> e
>> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPD
>> M
>> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS
>> 9
>> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JM
>> c
>> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1
>> Y
>> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2
>> F
>> rsyslog
>> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-r
>> n
>> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xj
>> m
>> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78lj
>> G
>> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aez
>> h
>> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVU
>> T
>> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiL
>> j
>> UUXUdy0ikBSRjDg/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>> THAT.
>>
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
