You could also try something like "watch fuser $(which ip)" or (if you can) write a program using inotify and IN_OPEN to see which procrees are opening the binary.
>>> Thomas CAS <t...@ikoula.com> schrieb am 21.12.2022 um 09:24 in Nachricht <pr3p193mb08297ba8173a31e40e59f3a4a5...@pr3p193mb0829.eurp193.prod.outlook.com>: > Ken, > > Antivirus (sophos‑av) is running but not in "real time access scanning", the > scheduled scan is however at 9pm every day. > 7 minutes later, we got these alerts. > The anti virus may indeed be the cause. > > I had the case on December 13 (with systemctl here): > > pacemaker.log‑20221217.gz:Dec 13 21:07:53 wd‑websqlng01 pacemaker‑controld > [5082] (process_lrm_event) notice: wd‑websqlng01‑NGINX_monitor_15000:454 [ > /etc/init.d/nginx: 33: /lib/lsb/init‑functions.d/40‑systemd: systemctl: Text > file busy\n/etc/init.d/nginx: 82: /lib/lsb/init‑functions.d/40‑systemd: > /bin/systemctl: Text file busy\n ] > pacemaker.log‑20221217.gz:Dec 13 21:07:53 wd‑websqlng01 pacemaker‑controld > [5082] (process_lrm_event) notice: wd‑websqlng01‑NGINX_monitor_15000:454 [ > /etc/init.d/nginx: 33: /lib/lsb/init‑functions.d/40‑systemd: systemctl: Text > file busy\n/etc/init.d/nginx: 82: /lib/lsb/init‑functions.d/40‑systemd: > /bin/systemctl: Text file busy\n ] > > After, this happens rarely, we had the case in August: > > pacemaker.log‑20220826.gz:Aug 25 21:06:31 wd‑websqlng01 pacemaker‑controld > [3718] (process_lrm_event) notice: > wd‑websqlng01‑NGINX‑VIP‑232_monitor_10000:2877 [ > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file > busy\nocf‑exit‑reason:IPaddr2 only supported Linux.\n ] > pacemaker.log‑20220826.gz:Aug 25 21:06:31 wd‑websqlng01 pacemaker‑controld > [3718] (process_lrm_event) notice: > wd‑websqlng01‑NGINX‑VIP‑231_monitor_10000:2880 [ > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: uname: Text file > busy\nocf‑exit‑reason:IPaddr2 only supported Linux.\n ] > > It's always around 9:00‑9:07 pm, > I'll move the virus scan to 10pm and see. > > Thanks, > Best regards, > > Thomas Cas | Technicien du support infogérance > PHONE : +33 3 51 25 23 26 WEB : www.ikoula.com/en > IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims ‑ FRANCE > Before printing this letter, think about the impact on the environment! > > ‑‑‑‑‑Message d'origine‑‑‑‑‑ > De : Reid Wahl <nw...@redhat.com> > Envoyé : mardi 20 décembre 2022 20:34 > À : Cluster Labs ‑ All topics related to open‑source clustering welcomed > <users@clusterlabs.org> > Cc : Ken Gaillot <kgail...@redhat.com>; Service Infogérance > <infogera...@ikoula.com> > Objet : Re: [ClusterLabs] Bug pacemaker with multiple IP > > [Vous ne recevez pas souvent de courriers de nw...@redhat.com. Découvrez > pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ] > > On Tue, Dec 20, 2022 at 6:25 AM Thomas CAS <t...@ikoula.com> wrote: >> >> Hello Ken, >> >> Thanks for your answer. >> There was no update running at the time of the bug, which is why I thought > that having too many IPs caused this type of error. >> The /usr/sbin/ip executable was not being modified either. >> >> We have many clusters, and only this one has so many IPs and this problem. > > How often does this happen, and is it reliably reproducible under any > circumstances? Any antivirus software running? It'd be nice to check > something like lsof or strace while it's happening, but that may not be > feasible if it's sporadic; running those at every monitor would generate lots > of logs. > > AFAICT, having multiple processes execute (or read) the `ip` binary > simultaneously *shouldn't* cause problems, as long as nothing opens it for > write. > >> >> Best regards, >> >> Thomas Cas | Technicien du support infogérance >> PHONE : +33 3 51 25 23 26 WEB : > https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ikoula.c > om%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C9aab91944bd6454a773808dae2c13ae4%7C > cb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638071616800939086%7CUnknown%7CTWF > pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 > C3000%7C%7C%7C&sdata=oYe7ws2%2BPx3sMblOFBgkuXuSHTdguzB%2Flk83O5W2MjE%3D&reserve > d=0 >> IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims ‑ FRANCE Before >> printing this letter, think about the impact on the environment! >> >> ‑‑‑‑‑Message d'origine‑‑‑‑‑ >> De : Ken Gaillot <kgail...@redhat.com> Envoyé : lundi 19 décembre 2022 >> 22:08 À : Cluster Labs ‑ All topics related to open‑source clustering >> welcomed <users@clusterlabs.org> Cc : Service Infogérance >> <infogera...@ikoula.com> Objet : Re: [ClusterLabs] Bug pacemaker with >> multiple IP >> >> [Vous ne recevez pas souvent de courriers de kgail...@redhat.com. >> Découvrez pourquoi ceci est important à >> https://aka.ms/LearnAboutSenderIdentification ] >> >> On Mon, 2022‑12‑19 at 09:48 +0000, Thomas CAS wrote: >> > Hello Clusterlabs, >> > >> > I would like to report a bug on Pacemaker with the "IPaddr2" >> > resource: >> > >> > OS: Debian 10 >> > Kernel: Linux wd‑websqlng01 4.19.0‑18‑amd64 #1 SMP Debian 4.19.208‑1 >> > (2021‑09‑29) x86_64 GNU/Linux >> > Pacemaker version: 2.0.1‑5+deb10u2 >> > >> > You will find the configuration of our cluster with 2 nodes attached. >> > >> > Bug : >> > >> > We have several IP configured in the cluster configuration (12) >> > Sometimes the cluster is unstable with the following errors in the >> > pacemaker logs: >> > >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 232_monitor_10000:28835:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> >> This doesn't sound like a bug in the agent; "Text file busy" suggests that > the system "ip" command is being modified while the command is running. Is a > software update happening when the problem occurs? >> >> I'm not sure whether there's some other situation that could cause that > error, but simply executing the command a bunch of times simultaneously > shouldn't cause it as far as I know. >> >> If simultaneous monitors is somehow causing the problem, you should be able > to work around it by using different intervals for different monitors. >> >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 239_monitor_10000:28877:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 239_monitor_10000:28877:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 234_monitor_10000:28830:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 231_monitor_10000:28900:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 231_monitor_10000:28900:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 235_monitor_10000:28905:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 235_monitor_10000:28905:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 237_monitor_10000:28890:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 237_monitor_10000:28890:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 238_monitor_10000:28876:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP‑ >> > 238_monitor_10000:28876:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP_monitor_10000:28880:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker‑execd [5079] >> > (operation_finished) notice: NGINX‑VIP_monitor_10000:28880:stderr [ >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1: >> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ] >> > >> > The reason is that there are a lot of IPs configured and if the >> > monitors take place at the same time it causes this type of error. >> > >> > Best regards, >> > >> > Thomas Cas | Technicien du support infogérance >> > PHONE : +33 3 51 25 23 26 WEB : > https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ikoula.c > om%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C9aab91944bd6454a773808dae2c13ae4%7C > cb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638071616800939086%7CUnknown%7CTWF > pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7 > C3000%7C%7C%7C&sdata=oYe7ws2%2BPx3sMblOFBgkuXuSHTdguzB%2Flk83O5W2MjE%3D&reserve > d=0 >> > IKOULA Data Center 34 rue Pont Assy ‑ 51100 Reims ‑ FRANCE Before >> > printing this letter, think about the impact on the environment! >> > >> > >> > >> > >> > _______________________________________________ >> > Manage your subscription: >> > https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli >> > st >> > s.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=05%7C01%7Ctcas%4 >> > 0i >> > koula.com%7C541f4960600340f90a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f8 >> > 0d >> > b4a66f1c7%7C0%7C0%7C638070808660951911%7CUnknown%7CTWFpbGZsb3d8eyJWI >> > jo >> > iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7 >> > C% >> > 7C%7C&sdata=U9osKXkKgjcqp6PN0%2F%2FB%2BzZyX0JMe6WMqRPVDTEGyWg%3D&res >> > er >> > ved=0 >> > >> > ClusterLabs home: >> > https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. >> > clusterlabs.org%2F&data=05%7C01%7Ctcas%40ikoula.com%7C541f4960600340 >> > f9 >> > 0a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638070 >> > 80 >> > 8660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM >> > zI >> > iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FfODTlNES3on >> > Dk >> > %2FfLgs6bWR2iikLdfqx7ePxzZfR%2BIU%3D&reserved=0 >> ‑‑ >> Ken Gaillot <kgail...@redhat.com> >> >> _______________________________________________ >> Manage your subscription: >> https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist >> s.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=05%7C01%7Ctcas%40i >> koula.com%7C9aab91944bd6454a773808dae2c13ae4%7Ccb7a4a4ea7f747cc931f80d >> b4a66f1c7%7C0%7C0%7C638071616800939086%7CUnknown%7CTWFpbGZsb3d8eyJWIjo >> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C% >> 7C%7C&sdata=3jtVFwvmy127OwWr9ZNbr6B%2FefuvNeZl9YsM31QxHJM%3D&reserved= >> 0 >> >> ClusterLabs home: >> https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. >> clusterlabs.org%2F&data=05%7C01%7Ctcas%40ikoula.com%7C9aab91944bd6454a >> 773808dae2c13ae4%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C63807161 >> 6800939086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI >> iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2E8uF0uNDw4djwcy >> %2FjVJ%2FDdJu5E77LQZfU9yrf0dVBI%3D&reserved=0 >> > > > ‑‑ > Regards, > > Reid Wahl (He/Him) > Senior Software Engineer, Red Hat > RHEL High Availability ‑ Pacemaker > > _______________________________________________ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
