Here's a look at how much memory we've got:
total used free shared buff/cache available
Mem: 65687464 1753788 9512004 336 54421672 63298172
Swap: 4194300 85080 4109220
Based on what I've been reading online I bumped up the following sysctls.
sysctl net.core.rmem_max=8388608
sysctl net.core.wmem_max=8388608
sysctl net.ipv4.udp_mem='2051962 3077940 8388608'
sysctl net.core.netdev_max_backlog=5000
Our send errors, don't look bad, especially in comparison to the rx errors, so
guessing I don't need to mess with that one, but, not sure what to set rmem_max
to, and how much memory to allow for udp traffic.
And then of course, once those values are tuned, what changes to make in my
rsyslog.conf.
Radesh
-----Original Message-----
From: David Lang <da...@lang.hm>
Sent: Thursday, August 01, 2019 7:12 PM
To: Singh, Radesh <radesh_si...@csx.com>
Cc: David Lang <da...@lang.hm>; Singh, Radesh via rsyslog
<rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages
yep, each of those errors is probably a lost log in sysctl you can increase the
buffer size, but you probably also need to tune rsyslog to handle logs faster
if you can post your config it would help
enabling impstats would help understand what's going on inside rsyslog
(including showing the counts of the number of messages it sees)
David Lang
On Thu, 1 Aug 2019, Singh, Radesh wrote:
> Date: Thu, 1 Aug 2019 22:48:40 +0000
> From: "Singh, Radesh" <radesh_si...@csx.com>
> To: David Lang <da...@lang.hm>,
> "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I may have found more clues...
>
> The number of errors related to UDP are really high...
>
> netstat -su
> IcmpMsg:
> InType0: 6
> InType3: 980584
> InType8: 28959
> InType11: 688
> OutType0: 28959
> OutType3: 987041
> OutType8: 6
> Udp:
> 140571327944 packets received
> 332780 packets to unknown port received.
> 151716349245 packet receive errors
> 207450667309 packets sent
> 151716213631 receive buffer errors
> 287 send buffer errors
> InCsumErrors: 135614
> UdpLite:
> IpExt:
> InBcastPkts: 446852
> InOctets: 73907716648604
> OutOctets: 103321724708971
> InBcastOctets: 130548312
> InNoECTPkts: 356643654014
> InECT0Pkts: 810
>
> I compared this to another log host, and while the other log host has errors,
> the numbers are nowhere as large.
>
> Radesh
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Thursday, August 01, 2019 5:07 PM
> To: David Lang <da...@lang.hm>; Singh, Radesh via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> Thank you for your response.
>
> The firewall logs don't get rotated.
> The logs from the firewall get this rule applied to them:
> $template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
> :inputname , isequal , "imudp" ?RemoteClient :inputname , isequal ,
> "imtcp" ?RemoteClient
>
> So we usually have some messages from the previous day, but that isn't a
> concern to us.
>
> At first I suspected that logrotation might be the culprit, but the issue is
> occurring all throughout the day, and our logs usually rotate between 0000
> and 0500 each day.
> The only logrotation config file that interacts with rsyslog is our syslog
> config.
> In that config file (/etc/logrotate.d/syslog), I see the following config:
>
> syslog
> /var/log/cron
> /var/log/maillog
> /var/log/messages
> /var/log/secure
> /var/log/spooler
> /var/log/boot.log
> {
> missingok
> sharedscripts
> notifempty
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null ||
> true
> endscript
> }
>
> So, we are sending HUP, not using the copytruncate option.
>
> I wonder if we are having issue with UDP buffers.
> I haven't had to investigate that before.
>
> Could you provide any suggestions on how you'd go about that on RHEL7?
> A quick google return several results... one of which suggests watching
> /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run
> across this before, wondering if that is the best approach.
>
> Thanks,
>
> Radesh
>
>
> -----Original Message-----
> From: David Lang <da...@lang.hm>
> Sent: Thursday, August 01, 2019 4:45 PM
> To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <radesh_si...@csx.com>
> Subject: [E]Re: [rsyslog] Missing messages
>
> how are you rotating your logs? are you sending rsyslog a HUP or are you
> doing a /etc/init.d/reload (which is a full restart)?
>
> are you using copytruncate as you rotate your logs?
>
> these are the most common issues.
>
> if you are sending via UDP, check your OS UDP buffers, if they are filling
> up, the packets will be dropped before they get to rsyslog.
>
> 8.24 is pretty old (~2.5 years), but it's unlikely to be the root
> cause of the problem
>
> David Lang
>
>
> On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
>
>> Date: Thu, 1 Aug 2019 19:13:11 +0000
>> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: "Singh, Radesh" <radesh_si...@csx.com>
>> Subject: Re: [rsyslog] Missing messages
>>
>> Just to ensure that I don't too much confusion.
>> I made the following statement:
>>
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I should have said:
>> Messages that aren't getting logged leave no trace anywhere, EXCEPT the
>> packet capture.
>>
>> The missing messages are being captured in the tcpdump, so when my firewall
>> guy sends me a sample of what he sent, if I'm running a packet capture, I've
>> validated that the server has received the data, even if rsyslog isn't
>> writing it down.
>>
>> Thanks again.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Singh,
>> Radesh via rsyslog
>> Sent: Thursday, August 01, 2019 3:08 PM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Singh, Radesh <radesh_si...@csx.com>
>> Subject: [E][rsyslog] Missing messages
>>
>> List,
>>
>> Our firewall guys reported that they're missing messages.
>> They've got devices pointed at a dedicated rsyslog server ( they are the
>> only folks using it ) and they are seeing cases were they are sending
>> messages, I am receiving the messages, but the messages aren't getting
>> written to the logs.
>> The missing messages aren't from any particular source, the issue seems to
>> occur all throughout the day, and in the case of the messages, we expect to
>> see when sessions are built and torn down.
>> There are times the entire session is captured, times when part of the
>> session is missed, and times when the entire session is missed.
>>
>> I know the message is reaching the server, because I ran a packet capture
>> and see the data.
>> I turned on debug and captured a pretty healthy dump of data.
>> BTW, I set: RSYSLOG_DEBUG to "Debug".
>>
>> Messages that are getting logged, are visible in the debug logs.
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I'm running:
>> Name : rsyslog
>> Version : 8.24.0
>> Release : 16.el7
>> Architecture: x86_64
>>
>> On RHEL 7.5.
>>
>> Do you guys have any ideas of things I might try to get more info?
>>
>> Thank you,
>>
>> Radesh
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-M
>> O
>> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi
>> 4
>> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH
>> 2
>> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr
>> 5
>> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT0
>> 9
>> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%
>> 3 A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
>> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kk
>> l
>> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPob
>> E
>> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJ
>> s
>> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5
>> q
>> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNV
>> g
>> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26v
>> z
>> ubCvnjmiVvAjt-g/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>> THAT.
>> __________________________________________
>> E - EXTERNAL SENDER
>> Use discretion when clicking links, opening attachments, or replying.
>> __________________________________________
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKL
>> r
>> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaA
>> e
>> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPD
>> M
>> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS
>> 9
>> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JM
>> c
>> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1
>> Y
>> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2
>> F
>> rsyslog
>> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-r
>> n
>> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xj
>> m
>> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78lj
>> G
>> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aez
>> h
>> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVU
>> T
>> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiL
>> j
>> UUXUdy0ikBSRjDg/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>> THAT.
>>
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
