*Update*
In the log below you may notice suddenly relp queue grow to 35k events
(that's when the forwarding relay was restarted). And -at the end- it
drops to 0, as the relay stop forwarding.
Why rsyslog has't been able to reduce queue size from ~35k for quite a
long time? Is 10k the aprox. maximum ingest with our current config?
Wed Jan 18 09:47:57 2017: global: origin=dynstats
...
Wed Jan 18 10:32:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:32:57 2017: relp: origin=core.queue size=0 enqueued=1729
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:32:57 2017: main Q: origin=core.queue size=12 enqueued=129
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 10:37:57 2017: global: origin=dynstats
Wed Jan 18 10:37:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: omelasticsearch: origin=omelasticsearch
submitted=1889 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:37:57 2017: json: origin=core.action processed=1889
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: norm: origin=core.action processed=1889
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: elastic: origin=core.action processed=1889
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: imrelp[20514]: origin=imrelp submitted=1889
Wed Jan 18 10:37:57 2017: resource-usage: origin=impstats utime=1036000
stime=876000 maxrss=8092 minflt=1500 majflt=0 inblock=10 oublock=45
nvcsw=11377 nivcsw=129
Wed Jan 18 10:37:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:37:57 2017: relp: origin=core.queue size=0 enqueued=1889
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:37:57 2017: main Q: origin=core.queue size=12 enqueued=142
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 10:42:58 2017: global: origin=dynstats
Wed Jan 18 10:42:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: omelasticsearch: origin=omelasticsearch
submitted=70614 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:42:58 2017: json: origin=core.action processed=70901
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: norm: origin=core.action processed=70901
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: elastic: origin=core.action processed=70901
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: imrelp[20514]: origin=imrelp submitted=104878
Wed Jan 18 10:42:58 2017: resource-usage: origin=impstats utime=15748000
stime=11172000 maxrss=75992 minflt=18537 majflt=0 inblock=10 oublock=48
nvcsw=219548 nivcsw=7017
Wed Jan 18 10:42:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:42:58 2017: relp: origin=core.queue size=35001
enqueued=104879 full=0 discarded.full=0 discarded.nf=0 maxqsize=35003
Wed Jan 18 10:42:58 2017: main Q: origin=core.queue size=12 enqueued=155
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 10:47:58 2017: global: origin=dynstats
Wed Jan 18 10:47:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: omelasticsearch: origin=omelasticsearch
submitted=162118 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:47:58 2017: json: origin=core.action processed=163061
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: norm: origin=core.action processed=163061
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: elastic: origin=core.action processed=163061
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: imrelp[20514]: origin=imrelp submitted=196015
Wed Jan 18 10:47:58 2017: resource-usage: origin=impstats utime=37292000
stime=24932000 maxrss=104684 minflt=25667 majflt=0 inblock=10 oublock=51
nvcsw=496418 nivcsw=14202
Wed Jan 18 10:47:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:47:58 2017: relp: origin=core.queue size=33978
enqueued=196016 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 10:47:58 2017: main Q: origin=core.queue size=3 enqueued=168
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 10:52:58 2017: global: origin=dynstats
Wed Jan 18 10:52:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: omelasticsearch: origin=omelasticsearch
submitted=265391 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:52:58 2017: json: origin=core.action processed=265461
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: norm: origin=core.action processed=265461
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: elastic: origin=core.action processed=265461
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:52:58 2017: imrelp[20514]: origin=imrelp submitted=299439
Wed Jan 18 10:52:58 2017: resource-usage: origin=impstats utime=60212000
stime=40316000 maxrss=135732 minflt=33431 majflt=0 inblock=10 oublock=54
nvcsw=808532 nivcsw=22916
Wed Jan 18 10:52:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:52:58 2017: relp: origin=core.queue size=35002
enqueued=299440 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 10:52:58 2017: main Q: origin=core.queue size=2 enqueued=181
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 10:57:58 2017: global: origin=dynstats
Wed Jan 18 10:57:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: omelasticsearch: origin=omelasticsearch
submitted=370571 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:57:58 2017: json: origin=core.action processed=370933
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: norm: origin=core.action processed=370933
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: elastic: origin=core.action processed=370933
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:57:58 2017: imrelp[20514]: origin=imrelp submitted=404910
Wed Jan 18 10:57:58 2017: resource-usage: origin=impstats utime=83596000
stime=55228000 maxrss=167020 minflt=41286 majflt=0 inblock=10 oublock=57
nvcsw=1124817 nivcsw=33282
Wed Jan 18 10:57:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:57:58 2017: relp: origin=core.queue size=35001
enqueued=404911 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 10:57:58 2017: main Q: origin=core.queue size=12 enqueued=194
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:02:58 2017: global: origin=dynstats
Wed Jan 18 11:02:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: omelasticsearch: origin=omelasticsearch
submitted=471328 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:02:58 2017: json: origin=core.action processed=472309
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: norm: origin=core.action processed=472309
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: elastic: origin=core.action processed=472309
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:02:58 2017: imrelp[20514]: origin=imrelp submitted=505264
Wed Jan 18 11:02:58 2017: resource-usage: origin=impstats
utime=106828000 stime=69648000 maxrss=196836 minflt=48810 majflt=0
inblock=10 oublock=60 nvcsw=1429398 nivcsw=41644
Wed Jan 18 11:02:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:02:58 2017: relp: origin=core.queue size=33979
enqueued=505265 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:02:58 2017: main Q: origin=core.queue size=12 enqueued=207
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:07:58 2017: global: origin=dynstats
Wed Jan 18 11:07:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: omelasticsearch: origin=omelasticsearch
submitted=574676 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:07:58 2017: json: origin=core.action processed=574709
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: norm: origin=core.action processed=574709
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: elastic: origin=core.action processed=574709
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:07:58 2017: imrelp[20514]: origin=imrelp submitted=608687
Wed Jan 18 11:07:58 2017: resource-usage: origin=impstats
utime=130764000 stime=84716000 maxrss=227096 minflt=56388 majflt=0
inblock=10 oublock=63 nvcsw=1741614 nivcsw=50338
Wed Jan 18 11:07:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:07:58 2017: relp: origin=core.queue size=35002
enqueued=608688 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:07:58 2017: main Q: origin=core.queue size=12 enqueued=220
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:12:58 2017: global: origin=dynstats
Wed Jan 18 11:12:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: omelasticsearch: origin=omelasticsearch
submitted=676472 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:12:58 2017: json: origin=core.action processed=677109
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: norm: origin=core.action processed=677109
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: elastic: origin=core.action processed=677109
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:12:58 2017: imrelp[20514]: origin=imrelp submitted=711085
Wed Jan 18 11:12:58 2017: resource-usage: origin=impstats
utime=154780000 stime=99132000 maxrss=257392 minflt=63988 majflt=0
inblock=10 oublock=66 nvcsw=2049218 nivcsw=58542
Wed Jan 18 11:12:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:12:58 2017: relp: origin=core.queue size=35000
enqueued=711086 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:12:58 2017: main Q: origin=core.queue size=12 enqueued=233
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:17:58 2017: global: origin=dynstats
Wed Jan 18 11:17:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: omelasticsearch: origin=omelasticsearch
submitted=770669 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:17:58 2017: json: origin=core.action processed=771317
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: norm: origin=core.action processed=771317
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: elastic: origin=core.action processed=771317
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:17:58 2017: imrelp[20514]: origin=imrelp submitted=805293
Wed Jan 18 11:17:58 2017: resource-usage: origin=impstats
utime=177712000 stime=113216000 maxrss=285424 minflt=71000 majflt=0
inblock=10 oublock=69 nvcsw=2334000 nivcsw=65950
Wed Jan 18 11:17:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:17:58 2017: relp: origin=core.queue size=35000
enqueued=805294 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:17:58 2017: main Q: origin=core.queue size=12 enqueued=246
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:22:58 2017: global: origin=dynstats
Wed Jan 18 11:22:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: omelasticsearch: origin=omelasticsearch
submitted=873317 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:22:58 2017: json: origin=core.action processed=873717
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: norm: origin=core.action processed=873717
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: elastic: origin=core.action processed=873717
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:22:58 2017: imrelp[20514]: origin=imrelp submitted=907693
Wed Jan 18 11:22:58 2017: resource-usage: origin=impstats
utime=201500000 stime=128332000 maxrss=315576 minflt=78597 majflt=0
inblock=10 oublock=72 nvcsw=2643581 nivcsw=75008
Wed Jan 18 11:22:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:22:58 2017: relp: origin=core.queue size=35000
enqueued=907694 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:22:58 2017: main Q: origin=core.queue size=12 enqueued=259
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:27:58 2017: global: origin=dynstats
Wed Jan 18 11:27:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: omelasticsearch: origin=omelasticsearch
submitted=972862 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:27:58 2017: json: origin=core.action processed=973045
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: norm: origin=core.action processed=973045
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: elastic: origin=core.action processed=973045
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:27:58 2017: imrelp[20514]: origin=imrelp submitted=1007022
Wed Jan 18 11:27:58 2017: resource-usage: origin=impstats
utime=224592000 stime=142848000 maxrss=344840 minflt=85924 majflt=0
inblock=10 oublock=75 nvcsw=2943338 nivcsw=84266
Wed Jan 18 11:27:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:27:58 2017: relp: origin=core.queue size=35001
enqueued=1007023 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:27:58 2017: main Q: origin=core.queue size=12 enqueued=272
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
Wed Jan 18 11:32:58 2017: global: origin=dynstats
Wed Jan 18 11:32:58 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: omelasticsearch: origin=omelasticsearch
submitted=1053233 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 11:32:58 2017: json: origin=core.action processed=1053233
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: norm: origin=core.action processed=1053233
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: elastic: origin=core.action processed=1053233
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 11:32:58 2017: imrelp[20514]: origin=imrelp submitted=1053233
Wed Jan 18 11:32:58 2017: resource-usage: origin=impstats
utime=243280000 stime=154784000 maxrss=368376 minflt=91831 majflt=0
inblock=10 oublock=78 nvcsw=3186315 nivcsw=90199
Wed Jan 18 11:32:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 11:32:58 2017: relp: origin=core.queue size=0
enqueued=1053233 full=0 discarded.full=0 discarded.nf=0 maxqsize=35011
Wed Jan 18 11:32:58 2017: main Q: origin=core.queue size=12 enqueued=285
full=0 discarded.full=0 discarded.nf=0 maxqsize=13
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
