Hello,
I'm using rsyslog to send the log to syslog receivers (Logstash) via TLS with
mutual authentication on redhat 7.2 with rSyslog 8.26 installed from adiscon
rpm.
The problematic rsyslog server has 2 main roles:
1/ Relay the information from other rsyslog syslog-tls protocol and send it to
logtsash with syslog-tls:
Rsyslog2 --TLS--> Rsyslog1 --TLS--> LoadBalancer --TLS--> 2x Logstash.
2/ Receive the logs from devices using standard syslog protocol and send them
to logstash with syslog-tls:
Devices ----> Rsyslog1 --TLS--> LoadBalancer --TLS--> 2x Logstash.
The issue appears when there is more activitie, the first memory queue of
rsyslog1 is growing (role 1: TLS to TLS).
I don't understand why the queue is growing (maybe I misunderstand the
impstats), but even if the enqueued volumes is equal to the processed volume:
the queue is still growing (or sometimes decreasing):
IMPSTATS e.g.
Time parsedstats.origin parsedstats.name parsedstats.size
parsedstats.enqueued parsedstats.processed
May 22nd 2017, 13:57:15.814 core_queue RelayToLogStashFromTLS queue
5,049,201 58,643 -
May 22nd 2017, 13:57:15.814 core_queue RelayToLogStashFromTLS
queue[DA] 0 0 -
May 22nd 2017, 13:57:15.814 core_action RelayToLogStashFromTLS -
- 58,583
May 22nd 2017, 13:57:05.803 core_queue RelayToLogStashFromTLS queue
4,997,558 63,249 -
May 22nd 2017, 13:57:05.803 core_queue RelayToLogStashFromTLS
queue[DA] 0 0 -
May 22nd 2017, 13:57:05.803 core_action RelayToLogStashFromTLS -
- 63,249
May 22nd 2017, 13:56:55.793 core_queue RelayToLogStashFromTLS queue
4,941,309 57,56 -
May 22nd 2017, 13:56:55.793 core_queue RelayToLogStashFromTLS
queue[DA] 0 0 -
May 22nd 2017, 13:56:55.792 core_action RelayToLogStashFromTLS -
- 57,56
May 22nd 2017, 13:56:45.792 core_queue RelayToLogStashFromTLS queue
4,891,749 2,372 -
May 22nd 2017, 13:56:45.792 core_queue RelayToLogStashFromTLS
queue[DA] 0 0 -
May 22nd 2017, 13:56:45.791 core_action RelayToLogStashFromTLS -
- 2,372
May 22nd 2017, 13:56:35.788 core_queue RelayToLogStashFromTLS queue
4,896,377 2,258 -
May 22nd 2017, 13:56:35.788 core_queue RelayToLogStashFromTLS
queue[DA] 0 0 -
May 22nd 2017, 13:56:35.788 core_action RelayToLogStashFromTLS -
- 2,258
The load balancer is not using any ssl bridging, and the balance method is:
round robin.
The CPU usage of rsyslog1 doesn't go higher than 4-5%.
If I ran some tcp dump, I saw some TCP window full, but anyway it's part of tcp
(windows scaling).
The logstash is using only 15 % of the cpu, and if I disable one Logstash
server it's going at 30%.
I've tried to reproduce the issue in a minimal lab with debugging on, and
despite the fact that I'm not able to generate enough logs (with logger in a
loop). The problem doesn't happen, and Logstash is running at 100%.
Out of idea... What can I check next?
Rsyslog1 config:
/ ** CONFIG Start **/
global(
defaultNetstreamDriver="gtls"
#debug.gnutls="10"
#TCP TLS
defaultNetstreamDriverCAFile="/apps/rsyslog/cert/CA.crt"
defaultNetstreamDriverCertFile="/apps/rsyslog/cert/rsylog1.mydomain.local.crt"
defaultNetstreamDriverKeyFile="/apps/rsyslog/cert/rsylog1.mydomain.local.key"
)
module(load="imptcp")
module(
load="imtcp"
streamDriver.name="gtls"
streamDriver.mode="1"
streamDriver.authMode="x509/name"
permittedPeer=["rsyslog2-1.mydomain.local","rsyslog2-2.mydomain.local"]
)
module(
load="impstats"
interval="10"
resetCounters="on"
format="cee"
ruleset="remote_tcp_secure_6514" # send stats to the
remote_tcp_secure_6514 ruleset
)
template(name="SyslogSource" type="string"
string="<%pri%>%timestamp:::date-rfc3339% %hostname% %fromhost-ip%
%syslogtag% %msg%\n"
)
template(name="SyslogSourceRelay" type="string"
string="<%pri%>%timestamp:::date-rfc3339% %hostname% %syslogtag%
%msg%\n"
)
/* Input entries */
input(type="imptcp" port="514" ruleset="remote_tcp_secure_6514")
input(
type="imtcp"
port="6514"
name="imtcp_secure"
ruleset="remote_relay_tcp_secure_6514"
)
main_queue(
queue.size="1000000" # capacity of the main queue
queue.dequeuebatchsize="1000" # process messages in batches of 1000 and move
them to the action queues
queue.workerthreads="1" # 1 thread for the main queue
)
/* Remote Logging */
RuleSet (name="remote_tcp_secure_6514")
{
action(
type="omfwd"
name="SendToLogStashFrom514"
target="logstash.lb.mydomain.local"
port="6514"
protocol="tcp"
StreamDriverMode="1"
StreamDriver="gtls"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="*.lb.mydomain.local"
action.resumeRetryCount="-1"
action.resumeInterval="1"
queue.type="linkedList"
queue.filename="ToLogStash"
queue.spoolDirectory="/apps/rsyslog/spool"
queue.maxDiskSpace="298g"
queue.saveOnShutdown="on"
queue.dequeuebatchsize="200"
RebindInterval="1000"
queue.workerthreads="1"
queue.highwatermark="846039250"
queue.lowwatermark="423019625"
template="SyslogSource"
queue.size="10000000000"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
)
}
RuleSet (name="remote_relay_tcp_secure_6514")
{
action(
type="omfwd"
name="RelayToLogStashFromTLS"
target="logstash.lb.mydomain.local"
port="6514"
protocol="tcp"
StreamDriverMode="1"
StreamDriver="gtls"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="*.lb.mydomain.local"
action.resumeRetryCount="-1"
action.resumeInterval="1"
queue.type="linkedList"
queue.filename="RelayToLogStash"
queue.spoolDirectory="/apps/rsyslog/spool"
queue.maxDiskSpace="200g"
queue.saveOnShutdown="on"
queue.dequeuebatchsize="200"
RebindInterval="1000"
queue.workerthreads="1"
queue.highwatermark="846039250"
queue.lowwatermark="423019625"
template="SyslogSourceRelay"
queue.size="10000000000"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
)
}
if $programname == 'rsyslogd' and ($msg contains "peer did not provide a
certificate, not permitted to talk to it" or $msg contains "will be closed due
to error#012" or $msg contains "The TLS connection was non-properly
terminated.#012") then stop
/** CONFIG end **/
thanks!
Ludovic
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
