I got this expanded explanation from my chat with copilot on this subject, in case you find it of any value:
How logging is wired in Ubuntu 24.04 - *HAProxy package dependency*: The Debian/Ubuntu maintainers declare rsyslog as the logging dependency because HAProxy traditionally logs to syslog (/dev/log), and rsyslog is the daemon that consumes those messages and writes them to files. - *systemd‑journald in the middle*: Since Ubuntu 20.04, journald actually *owns* the /run/systemd/journal/syslog socket. Rsyslog doesn’t create /dev/log itself anymore — it connects to journald’s socket. Journald is the “first hop” for syslog messages, then forwards them to rsyslog if rsyslog is installed. - *So restarting journald helps*: When journald’s socket state or permissions get into a bad state (after an upgrade or race at boot), HAProxy can’t send messages. Restarting journald recreates /run/systemd/journal/syslog with the correct mode (srw-rw-rw-), and HAProxy can write again. Rsyslog is downstream, so HAProxy never talks to it directly — it talks to journald’s socket On Wed, Dec 3, 2025 at 8:37 AM Martín Córdova <[email protected]> wrote: > Thank you for the detailed response, I confirm that I have not altered the > default global log settings, I just added some directives to defaults: > > global > log /dev/log local0 > log /dev/log local1 notice > chroot /var/lib/haproxy > stats socket /run/haproxy/admin.sock mode 660 level admin > stats timeout 30s > user haproxy > group haproxy > daemon > > # Default SSL material locations > ca-base /etc/ssl/certs > crt-base /etc/ssl/private > > # See: > https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate > ssl-default-bind-ciphers > ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 > ssl-default-bind-ciphersuites > TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 > ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets > > defaults > log global > mode http > option httplog > option dontlognull > timeout connect 5000 > timeout client 50000 > timeout server 50000 > errorfile 400 /etc/haproxy/errors/400.http > errorfile 403 /etc/haproxy/errors/403.http > errorfile 408 /etc/haproxy/errors/408.http > errorfile 500 /etc/haproxy/errors/500.http > errorfile 502 /etc/haproxy/errors/502.http > errorfile 503 /etc/haproxy/errors/503.http > errorfile 504 /etc/haproxy/errors/504.http > unique-id-format %[uuid()] > unique-id-header X-Request-ID > log-format "${HAPROXY_HTTP_LOG_FMT} %ID" > > I understand from your explanation that this error is not related to > journald but to rsyslog. However, I can't workaround this error without > restarting journald; I have not restarted other services like rsyslog, just > journald. Maybe this one is forcing a restart of rsyslog too? > > On Wed, Dec 3, 2025 at 8:20 AM Jonas Jelten <[email protected]> > wrote: > >> a default haproxy install on noble is configured with: >> global >> log /dev/log local0 >> log /dev/log local1 notice >> [...] >> >> and this is a direct link to systemd-journald's socket. >> >> /dev/log -> /run/systemd/journal/dev-log >> >> which is managed by systemd-journald-dev-log.socket and does socket >> activation to start systemd-journald. so i think this is set up >> correctly (by dependencies & startup order). >> >> >> BUT, haproxy runs chrooted in /var/lib/haproxy with an separate /dev/log >> socket: >> >> rsyslog in /etc/rsyslog.d/49-haproxy.conf sets up a separate >> /var/lib/haproxy/dev/log socket to "Create an additional socket in >> haproxy's chroot in order to allow logging via /dev/log to chroot'ed >> HAProxy processes". rsyslog then filters out haproxy messages from that >> socket and logs them to /var/log/haproxy.log. >> >> so this should thus be independent of systemd-journald, and the error >> you're seeing should only be related to rsyslog. >> >> haproxy.service has after=rsyslog.service, but doesn't "want=" it. so >> rsyslog.service must be "enabled" (since haproxy doesn't start it) for >> the log socket to be available at all. >> >> since your output shows rsyslog.service is enabled, it seems there a race >> condition between rsyslog.service starting up (without being ready) and >> haproxy.service trying to use the socket before that? >> in any case I don't see how systemd-journald interferes here. >> >> >> I'd say the proper solution would be to get rid of rsyslog here, and just >> use journal logging + systemd service security confinements instead of the >> haproxy integrated chroot. >> >> -- >> You received this bug notification because you are subscribed to the bug >> report. >> https://bugs.launchpad.net/bugs/2133460 >> >> Title: >> haproxy for noble may start before systemd-journald >> >> Status in haproxy package in Ubuntu: >> Incomplete >> >> Bug description: >> This applies to the HAProxy 2.8 package installed from the Noble 24.04 >> repos: >> >> Distributor ID: Ubuntu >> Description: Ubuntu 24.04.3 LTS >> Release: 24.04 >> Codename: noble >> >> HAProxy version 2.8.15-0ubuntu0.24.04.1 2025/10/08 - >> https://haproxy.org/ >> >> When rebooting the OS, HAProxy may start before systemd-journald >> because its service configuration does not have a dependency under the >> [Unit] section on journald. This will cause a socket permission error >> when writing the HTTP access log: >> >> haproxy[44612]: [ALERT] (44612) : sendmsg()/writev() failed in logger >> #1: Permission denied (errno=13) >> >> As a workaround, a restart of journald and haproxy will solve the >> problem, but if journald is the default logger service in Ubuntu >> 24.04, I think this could be avoided by adding the proper dependencies >> under the Unit service configuration. >> >> [Unit] >> After=systemd-journald.service >> Requires=systemd-journald.service >> >> To manage notifications about this bug go to: >> >> https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/2133460/+subscriptions >> >> > > -- > API-Server++ - C++ low-code REST engine > https://github.com/cppservergit/apiserver-odbc > Martín Córdova y Asociados, C.A. > WhatsApp: +58-2686639 > www.linkedin.com/in/martincordova > -- API-Server++ - C++ low-code REST engine https://github.com/cppservergit/apiserver-odbc Martín Córdova y Asociados, C.A. WhatsApp: +58-2686639 www.linkedin.com/in/martincordova -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2133460 Title: haproxy for noble may start before systemd-journald To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/2133460/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
