** Description changed:
+ [Impact]
+
+ On Ubuntu 22.04 LTS, the GnuTLS network stream driver in rsyslog can enter a
receive retry loop when a TLS connection is interrupted under poor network
conditions.
+ The affected input thread repeatedly receives EAGAIN, consumes one CPU core,
and delays TLS log delivery.
+
+ The receive retry path does not account for the GnuTLS read/write direction
and may keep retrying even when GnuTLS has no pending data.
+ The upstream fix checks the GnuTLS read/write direction and pending-data
state, and waits for socket readiness when GnuTLS cannot read more data.
+
+ [Test Plan]
+
+ Please use the reproducer scripts in [Other Info].
+
+ The test sets up an anonymous GnuTLS input and two TLS 1.2 sessions,
+ withholding the last byte of one record in each session.
+
+ 1. Install the required packages and configure the test input.
+
+ sudo apt install rsyslog rsyslog-gnutls python3
+ sudo install -m 644 lp2125768.conf /etc/rsyslog.d/99-lp2125768.conf
+ sudo systemctl restart rsyslog
+
+ 2. Start two TLS sessions and observe rsyslogd while session A remains
+ incomplete.
+
+ sudo truncate -s 0 /var/log/lp2125768.log
+ python3 tls-record-proxy.py 16515 12 >proxy-a.log 2>&1 &
+ python3 tls-record-proxy.py 16516 1 >proxy-b.log 2>&1 &
+ sleep 1
+ python3 tls-boundary-client.py 16515 session-a 14 >client-a.log 2>&1 &
+ sleep 1
+ python3 tls-boundary-client.py 16516 session-b 7 >client-b.log 2>&1 &
+ sleep 3
+ sudo grep -c session-b-second /var/log/lp2125768.log # before: 0, after: 1
+ top -b -d 5 -n 2 -p "$(pidof rsyslogd)" | tail -n 3
+ wait
+ sudo grep -c 'session-[ab]-second' /var/log/lp2125768.log # expected: 2
+
+ Before the fix, the first grep prints 0 and rsyslogd consumes
+ approximately one CPU core until session A releases its byte.
+
+ After installing the patched rsyslog and rsyslog-gnutls packages, restart
rsyslog and repeat step 2.
+ The first grep should print 1 while session A is still incomplete and
rsyslogd should remain near idle.
+ The final grep should print 2 both before and after the fix.
+
+
+ [Where problems could occur]
+
+ The change affects the GnuTLS receive and readiness paths used by encrypted
TCP inputs.
+ A regression in these paths could close a fragmented TLS connection too early
or leave buffered input waiting even though it can be processed.
+ The test checks that both TLS sessions complete and that session B is not
blocked while session A is incomplete.
+
+ [Other Info]
+
+ Reproducer files
+
+ lp2125768.conf
+
+ module(load="imtcp")
+
+ input(type="imtcp" port="16514"
+ StreamDriver.Name="gtls"
+ StreamDriver.Mode="1"
+ StreamDriver.AuthMode="anon")
+
+ *.* action(type="omfile" file="/var/log/lp2125768.log")
+
+ tls-record-proxy.py
+
+ #!/usr/bin/env python3
+
+ import socket, sys, threading, time
+
+
+ def relay(source, destination):
+ while data := source.recv(65536):
+ destination.sendall(data)
+
+
+ listen_port = int(sys.argv[1])
+ hold_seconds = int(sys.argv[2])
+
+ with socket.create_server(("127.0.0.1", listen_port)) as listener:
+ client, _ = listener.accept()
+ with client, socket.create_connection(("127.0.0.1", 16514)) as server:
+ threading.Thread(target=relay, args=(server, client),
daemon=True).start()
+ buffered = bytearray()
+ records = []
+
+ while len(records) < 2:
+ buffered.extend(client.recv(65536))
+ while len(buffered) >= 5:
+ size = 5 + int.from_bytes(buffered[3:5], "big")
+ if len(buffered) < size:
+ break
+ record = bytes(buffered[:size])
+ del buffered[:size]
+ if record[0] == 23:
+ records.append(record)
+ else:
+ server.sendall(record)
+
+ payload = b"".join(records)
+ server.sendall(payload[:-1])
+ print("TRIGGER", flush=True)
+ time.sleep(hold_seconds)
+ server.sendall(payload[-1:] + buffered)
+ print("RELEASE", flush=True)
+ relay(client, server)
+
+ tls-boundary-client.py
+
+ #!/usr/bin/env python3
+
+ import socket, ssl, sys, time
+
+
+ port = int(sys.argv[1])
+ label = sys.argv[2]
+ linger_seconds = int(sys.argv[3])
+
+ prefix = f"<134>Sep 5 12:20:00 client {label} ".encode()
+ first = prefix + b"B" * (16383 - len(prefix)) + b"\n"
+ second = f"<134>Sep 5 12:20:01 client {label}-second\n".encode()
+
+ context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+ context.check_hostname = False
+ context.verify_mode = ssl.CERT_NONE
+ context.minimum_version = ssl.TLSVersion.TLSv1_2
+ context.maximum_version = ssl.TLSVersion.TLSv1_2
+ context.set_ciphers("aNULL:@SECLEVEL=0")
+
+ with socket.create_connection(("127.0.0.1", port)) as raw:
+ with context.wrap_socket(raw) as tls:
+ tls.sendall(first)
+ tls.sendall(second)
+ time.sleep(linger_seconds)
+
+
+
+ [Original Description]
+
The rsyslog version 8.2112.0-2ubuntu2.2 in Ubuntu 22.04 has a spin-loop
bug, consuming 100% CPU in the GNU TLS module under certain bad network
conditions.
Apply this fix:
https://github.com/rsyslog/rsyslog/commit/aefcfa4d0f6e213c9fac814c3e6bd53970b7e90e
The change set as-is applies directly to the source.
See an old issue, with fresh comments here:
https://github.com/rsyslog/rsyslog/issues/4927#issuecomment-3338350615
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2125768
Title:
Rsyslog Include patch for 100% CPU spin-loop in version
8.2112.0-2ubuntu2.2
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2125768/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
