David,
I may have found more clues...
The number of errors related to UDP are really high...
netstat -su
IcmpMsg:
InType0: 6
InType3: 980584
InType8: 28959
InType11: 688
OutType0: 28959
OutType3: 987041
OutType8: 6
Udp:
140571327944 packets received
332780 packets to unknown port received.
151716349245 packet receive errors
207450667309 packets sent
151716213631 receive buffer errors
287 send buffer errors
InCsumErrors: 135614
UdpLite:
IpExt:
InBcastPkts: 446852
InOctets: 73907716648604
OutOctets: 103321724708971
InBcastOctets: 130548312
InNoECTPkts: 356643654014
InECT0Pkts: 810
I compared this to another log host, and while the other log host has errors,
the numbers are nowhere as large.
Radesh
-----Original Message-----
From: Singh, Radesh
Sent: Thursday, August 01, 2019 5:07 PM
To: David Lang <[email protected]>; Singh, Radesh via rsyslog
<[email protected]>
Subject: RE: [E]Re: [rsyslog] Missing messages
David,
Thank you for your response.
The firewall logs don't get rotated.
The logs from the firewall get this rule applied to them:
$template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
:inputname , isequal , "imudp" ?RemoteClient :inputname , isequal , "imtcp"
?RemoteClient
So we usually have some messages from the previous day, but that isn't a
concern to us.
At first I suspected that logrotation might be the culprit, but the issue is
occurring all throughout the day, and our logs usually rotate between 0000 and
0500 each day.
The only logrotation config file that interacts with rsyslog is our syslog
config.
In that config file (/etc/logrotate.d/syslog), I see the following config:
syslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/boot.log
{
missingok
sharedscripts
notifempty
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null ||
true
endscript
}
So, we are sending HUP, not using the copytruncate option.
I wonder if we are having issue with UDP buffers.
I haven't had to investigate that before.
Could you provide any suggestions on how you'd go about that on RHEL7?
A quick google return several results... one of which suggests watching
/proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run
across this before, wondering if that is the best approach.
Thanks,
Radesh
-----Original Message-----
From: David Lang <[email protected]>
Sent: Thursday, August 01, 2019 4:45 PM
To: Singh, Radesh via rsyslog <[email protected]>
Cc: Singh, Radesh <[email protected]>
Subject: [E]Re: [rsyslog] Missing messages
how are you rotating your logs? are you sending rsyslog a HUP or are you doing
a /etc/init.d/reload (which is a full restart)?
are you using copytruncate as you rotate your logs?
these are the most common issues.
if you are sending via UDP, check your OS UDP buffers, if they are filling up,
the packets will be dropped before they get to rsyslog.
8.24 is pretty old (~2.5 years), but it's unlikely to be the root cause of the
problem
David Lang
On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
> Date: Thu, 1 Aug 2019 19:13:11 +0000
> From: "Singh, Radesh via rsyslog" <[email protected]>
> To: rsyslog-users <[email protected]>
> Cc: "Singh, Radesh" <[email protected]>
> Subject: Re: [rsyslog] Missing messages
>
> Just to ensure that I don't too much confusion.
> I made the following statement:
>
> Messages that aren't getting logged leave no trace anywhere.
>
> I should have said:
> Messages that aren't getting logged leave no trace anywhere, EXCEPT the
> packet capture.
>
> The missing messages are being captured in the tcpdump, so when my firewall
> guy sends me a sample of what he sent, if I'm running a packet capture, I've
> validated that the server has received the data, even if rsyslog isn't
> writing it down.
>
> Thanks again.
>
> -----Original Message-----
> From: rsyslog <[email protected]> On Behalf Of Singh,
> Radesh via rsyslog
> Sent: Thursday, August 01, 2019 3:08 PM
> To: rsyslog-users <[email protected]>
> Cc: Singh, Radesh <[email protected]>
> Subject: [E][rsyslog] Missing messages
>
> List,
>
> Our firewall guys reported that they're missing messages.
> They've got devices pointed at a dedicated rsyslog server ( they are the only
> folks using it ) and they are seeing cases were they are sending messages, I
> am receiving the messages, but the messages aren't getting written to the
> logs.
> The missing messages aren't from any particular source, the issue seems to
> occur all throughout the day, and in the case of the messages, we expect to
> see when sessions are built and torn down.
> There are times the entire session is captured, times when part of the
> session is missed, and times when the entire session is missed.
>
> I know the message is reaching the server, because I ran a packet capture and
> see the data.
> I turned on debug and captured a pretty healthy dump of data.
> BTW, I set: RSYSLOG_DEBUG to "Debug".
>
> Messages that are getting logged, are visible in the debug logs.
> Messages that aren't getting logged leave no trace anywhere.
>
> I'm running:
> Name : rsyslog
> Version : 8.24.0
> Release : 16.el7
> Architecture: x86_64
>
> On RHEL 7.5.
>
> Do you guys have any ideas of things I might try to get more info?
>
> Thank you,
>
> Radesh
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO
> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4
> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2
> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5
> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09
> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3
> A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kkl
> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE
> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJs
> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q
> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVg
> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vz
> ubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKLr
> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaAe
> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPDM
> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS9
> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JMc
> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1Y
> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2F
> rsyslog
> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-rn
> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xjm
> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78ljG
> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aezh
> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVUT
> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiLj
> UUXUdy0ikBSRjDg/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
__________________________________________
E - EXTERNAL SENDER
Use discretion when clicking links, opening attachments, or replying.
__________________________________________
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
