Took me a bit to get my test environment set up again.
I did add impstat to the rsyslog.conf I posted:
module(load="impstats"
interval="120"
severity="7"
log.syslog="off" # Takes this out of the normal rsyslog stream.
log.file="/var/log/rsyslog-pstats"
)
I also ran my tests with version 8.10.0.ad1-1 and this time encountered no
dropped lines over 3 different runs (along with control runs on rsyslog 7).
The counters in impstats matched as well. I plan on doing more testing when I
can, but this looks promising.
Sean Frost
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Monday, June 01, 2015 2:36 PM
To: rsyslog-users
Subject: Re: [rsyslog] Missed log lines upgrading rsyslog from 7 to 8
sorry I didn't get a chance to look at this sooner, I am not seeing anything in
these configs that should be different. could you enable impstats and check
that the counters on the events match what you see in the logs?
David Lang
On Wed, 27 May 2015, Frost, Sean wrote:
> Date: Wed, 27 May 2015 17:04:10 +0000
> From: "Frost, Sean" <[email protected]>
> Reply-To: rsyslog-users <[email protected]>
> To: rsyslog-users <[email protected]>
> Subject: Re: [rsyslog] Missed log lines upgrading rsyslog from 7 to 8
>
> All three servers have a base config that looks like the following:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__pastebin.com_NfMzT
> Hfp&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba
> 9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=HT5mbkmA
> 9x3i6PKN4ggk35bTrG_IkTAjP6_3MdaV6Y8&e=
>
> Server 1 and 2 have an additional config:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__pastebin.com_h3uWL
> nCs&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba
> 9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=B7nxcTnH
> JhjdqMK3uR1aNkVQg80rxwsDxCbnV810kGA&e=
>
> Server 3 also had an additional simplified config for this testing,
> different from what server 1 and 2 were using:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__pastebin.com_9eb8d
> wWj&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba
> 9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=dUaUOGn_
> 7W2cTm2k_S2_rZTQ1TV3xg4qwVmKHRAnBwo&e=
>
> I used netcat to pipe events to port 1001 on server 2 or 3 for testing using
> these configs and the situation I described.
>
> Thanks,
> Sean Frost
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of David Lang
> Sent: Tuesday, May 26, 2015 4:43 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Missed log lines upgrading rsyslog from 7 to 8
>
> On Tue, 26 May 2015, Frost, Sean wrote:
>
>> Hello,
>> We were looking into upgrading our rsyslog from version 7 to 8 and ran into
>> an issue where some passed log events seem to be missing and could use some
>> advice on how to proceed.
>>
>> We currently use rsyslog to pass log event streams to various servers for
>> processing. For some basic smoke testing I streamed some old logs to two
>> servers which passed it on to a third:
>> * Server 1 is running our current configuration and version of rsyslog -
>> rsyslog-7.4.9-2 (my control). It passes all events to server three and
>> writes all received events to a file.
>> * Server 2 is running the same configs and rsyslog-8.8.0.ad1-2 It
>> also passes all events to server 3 and writes all received events to
>> a file
>> * Server 3 is running rsyslog-7.4.9-2 and just writes what it receives to a
>> file using a simplified ruleset for this testing.
>>
>> I would pipe events via netcat from a test server to server 1 or 2, wait for
>> everything to finish writing to the files, stop rsyslog, rename the file and
>> then repeat.
>> Test source server -> Server (1 or 2) -> Server 3 At the end I
>> compared the number of lines written to each file.
>>
>> The source log file contained 45102253 event lines
>>
>> Server 1, still running rsyslog 7.4.9 with old config
>> Run 1:
>> 45102253 lines in file - Server 1
>> 45102253 lines in file - Server 3
>> Run 2:
>> 45202253 lines in file - Server 1
>> 45102253 lines in file - Server 3
>>
>> Server 2, rsyslog 8.8.0 with old config on event12-2-qa-sjc:
>> Run 1:
>> 45102253 lines in file - Server 2
>> 45102237 lines in file - Server 3
>> Run 2:
>> 45102253 lines in file - Server 2
>> 45102250 lines in file - Server 3
>>
>> As shown, the number of lines written out on the intermediate server (1 or
>> 2) is always consistent, but the lines written out on server 3 are not,
>> except when running things through server 1 (running rsyslog 7).
>> The above results are actually the last tests I ran as we were trying to
>> eliminate other possible sources of issues.
>> Previously, we also tried running rsyslog 8 on Server 3 and a re-written
>> configuration on server 2 (to use updated syntax). In each case, the results
>> were consistent and the same when running through server 1 but always
>> missing a seemingly random number of lines when running through server 2,
>> from 3 to 54 lines across 10 different runs.
>>
>> Could use some advice on how to move forward. If this is something that
>> might be fixed in 8.10 (which I will probably test with next) or if there
>> are any parameters we could check on or modify to correct this issue as we
>> do more testing.
>
> it would help to see your configs, we don't know if you are sending via
> TCP/UDP/relp/etc. There are other options that could cause issues if they are
> set as well.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_
> mailman_listinfo_rsyslog&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4
> d8SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=etif6e2W2I_n_xKtfe_3yfVcfupKRV-N0
> J28Qoc4H98&s=s6OvmH9g67lFEf6a5Z_wxlQgRdJ-XmzbjuS8zk7X7Yg&e=
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr
> ofessional-2Dservices_&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8
> SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=etif6e2W2I_n_xKtfe_3yfVcfupKRV-N0J2
> 8Qoc4H98&s=HmTgmEOAaFX1SYtZ5PCUzZ9eEQzP9V1rZDtp62l7txI&e=
> What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh
> ards&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOb
> a9JUSBOMngKEUg&m=etif6e2W2I_n_xKtfe_3yfVcfupKRV-N0J28Qoc4H98&s=Mbk8Dfn
> omSeyST6YrUcjrlrCavlg1_OVuzbokabeLEQ&e=
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_
> mailman_listinfo_rsyslog&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4
> d8SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct
> 0KhvJJIPds&s=8vafAJzm9PLRq3phxaA6uUtcUNzN6D0uOsR4s92scyY&e=
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr
> ofessional-2Dservices_&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8
> SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0K
> hvJJIPds&s=SA5a9ahL1tGJnUjgA4uB0jBhh_1stDUHfMppL09bJ_k&e=
> What's up with rsyslog? Follow
> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh
> ards&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOb
> a9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=ftfrL_Y
> DgVFs1GeD7AXX_H-F5qhIPVaLE629J6n9CjI&e=
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=8vafAJzm9PLRq3phxaA6uUtcUNzN6D0uOsR4s92scyY&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=SA5a9ahL1tGJnUjgA4uB0jBhh_1stDUHfMppL09bJ_k&e=
What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=AwICAg&c=MOptNlVtIETeDALC_lULrw&r=2GjZkSjP4d8SfDKENxVc2szLfTfOba9JUSBOMngKEUg&m=RiBGoTPa-i2JyGZNV_tdXqK_QAOGdTHct0KhvJJIPds&s=ftfrL_YDgVFs1GeD7AXX_H-F5qhIPVaLE629J6n9CjI&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
