There is one instance per connection; they get cleaned up after a successful stop, but accumulate on failures.
It seems that your service unit depends on firewalld, and firewalld is masked (i.e. forbidden to start), so the service breaks due to dependency failure. Not sure where that dependency comes from – maybe you have a drop-in adding this dependency, try "systemctl cat commandsocket_57813@.service". That will show all files systemd reads, which might be more than one. (Or maybe your distro patched it in for _all_ sockets? Wouldn't be surprised.) But in general, this doesn't seem a very reliable check. Do you care about specific services, like HTTP? Then monitor *those* services instead. Do you use this as theft detection? Power outage watch? On Fri, May 5, 2017, 11:30 Robert Pilja <robert.pi...@1und1.de> wrote: > Hello, > > i'm looking for help on debugging a systemd issue after upgrading our > systems from RHEL 7.2 to 7.3 (systemd: 219-19.el7_2.9.ppc64 -> > 219-30.el7_3.7.ppc64). > > > > We are using a custom socket unit which basically opens a tcp port and > returns /bin/true to established connections. > > This unit provides an healthcheck/heartbeat mechanism used by our > loadbalancer framework. > > For the last 1.5 years, this implementation worked flawless. Until now. > > > > Here is our config: > > > > /etc/systemd/system/commandsocket_57813@.service > > [Unit] > > Description=service for /bin/true ListenStream Port 57813 > host present check > > > > [Service] > > ExecStart=/bin/true > > StandardOutput=socket > > > > /etc/systemd/system/commandsocket_57813.socket > > [Unit] > > Description=Port 57813 socket for host present check > > > > [Socket] > > ListenStream=57813 > > Accept=yes > > > > [Install] > > WantedBy=sockets.target > > > > --- > > > > journalctl: > > > > Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Failed to start service for > /bin/true ListenStream Port 57813 host present check. > > Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Unit > commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service > entered failed state. > > Apr 24 03:38:11 bscqs01.server.lan systemd[1]: > commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service > failed. > > Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Starting service for > /bin/true ListenStream Port 57813 host present check... > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Cannot add dependency job > for unit firewalld.service, ignoring: Unit is masked. > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: > commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service > failed to run 'start' task: Transport endpoint is not connected > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Failed to start service for > /bin/true ListenStream Port 57813 host present check (10.88.44.27:51161). > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Unit > commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service > entered failed state. > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: > commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service > failed. > > Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Starting service for > /bin/true ListenStream Port 57813 host present check (10.88.44.27:51161 > )... > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Cannot add dependency job > for unit firewalld.service, ignoring: Unit is masked. > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: > commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service > failed to run 'start' task: Transport endpoint is not connected > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Failed to start service for > /bin/true ListenStream Port 57813 host present check (10.88.44.27:51346). > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Unit > commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service > entered failed state. > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: > commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service > failed. > > Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Starting service for > /bin/true ListenStream Port 57813 host present check (10.88.44.27:51346 > )... > > Apr 24 03:38:31 bscqs01.server.lan systemd[1]: Cannot add dependency job > for unit firewalld.service, ignoring: Unit is masked. > > Apr 24 03:38:31 bscqs01.server.lan systemd[1]: > commandsocket_57813@65311-10.88.44.62:57813-10.88.44.27:51435.service > failed to run 'start' task: Transport endpoint is not connected > > > > -- > > > > systemctl start commandsocket_57813.socket > > > > systemctl status commandsocket_57813.socket > > ● commandsocket_57813.socket - Port 57813 socket for host present check > > Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; > enabled; vendor preset: disabled) > > Active: active (listening) since Fri 2017-05-05 08:48:48 CEST; 5s ago > > Listen: [::]:57813 (Stream) > > Accepted: 65359; Connected: 0 > > > > May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 > socket for host present check. > > May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket > for host present check. > > > > systemctl status commandsocket_57813.socket > > ● commandsocket_57813.socket - Port 57813 socket for host present check > > Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; > enabled; vendor preset: disabled) > > Active: failed (Result: resources) since Fri 2017-05-05 08:48:54 CEST; > 7s ago > > Listen: [::]:57813 (Stream) > > Accepted: 65359; Connected: 0 > > > > May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 > socket for host present check. > > May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket > for host present check. > > May 05 08:48:54 bscqs01.server.lan systemd[1]: commandsocket_57813.socket > failed to queue service startup job (Maybe the service file is missing or > not a template unit?): Argument list too long > > May 05 08:48:54 bscqs01.server.lan systemd[1]: Unit > commandsocket_57813.socket entered failed state. > > > > -- > > > > systemctl |grep commandsocket_57813: > > [...] > > ● commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:54933) > > ● commandsocket_57813@9992-10.88.44.62:57813-10.88.44.27:55028.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55028) > > ● commandsocket_57813@9993-10.88.44.62:57813-10.88.44.27:55119.service > loaded failed failed service for /bin/true ListenStream Port 57813 > host present check (10.88.44.27:55119) > > ● commandsocket_57813@9994-10.88.44.62:57813-10.88.44.27:55207.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55207) > > ● commandsocket_57813@9995-10.88.44.62:57813-10.88.44.27:55302.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55302) > > ● commandsocket_57813@9996-10.88.44.62:57813-10.88.44.27:55393.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55393) > > ● commandsocket_57813@9997-10.88.44.62:57813-10.88.44.27:55481.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55481) > > ● commandsocket_57813@9998-10.88.44.62:57813-10.88.44.27:55576.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55576) > > ● commandsocket_57813@9999-10.88.44.62:57813-10.88.44.27:55667.service > loaded failed failed service for /bin/true ListenStream Port 57813 host > present check (10.88.44.27:55667) > > > system-commandsocket_57813.slice > loaded active active system-commandsocket_57813.slice > > ● > commandsocket_57813.socket > loaded failed failed Port 57813 socket for host present check > > > > systemctl |grep commandsocket_57813 | wc -l: > > 65361 > > > > Port number limit? > > > > --- > > > > Any idea on how to fix this problem? > > > > Do you think that our current unit implementation is a reasonable solution > for providing healthchecks? Before systemd, we were using xinetd for that. > > Is this large number of additional entries > (commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service > etc.) normal? I have not seen them before the upgrade. > > > > Regards > > Robert > _______________________________________________ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel > -- Mantas Mikulėnas <graw...@gmail.com> Sent from my phone
_______________________________________________ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
