I've never used Zabbix. Maybe Zabbix is the issue … (it has some similarities to Monit in a way) Try to disable it for a while on the problematic boxes …
> Am 16.09.2020 um 23:09 schrieb Michael Knill > <michael.kn...@ipcsolutions.com.au>: > > I do have Zabbix running on these boxes which checks memory and will notify > me if its getting too low and this has not happened. This is what alerted me > to the Zabbix client memory leak. > Also the lockups are very sporadic. Can happen after a couple of weeks or the > next day. > > Regards > Michael Knill > > On 16/9/20, 11:21 pm, "Michael Keuter" <li...@mksolutions.info> wrote: > > Hi all, > > it might be completely unrelated, but it reminds me at an issue that only > occured many years ago only on 32bit Geode boxes (Alix/net5501) after we > integrated Monit into AstLinux. > On these boxes all RAM was eaten up slowly and then these boxes locked up > after a few days. That's why we didn't installed Monit for these build types. > > To figure it out I then created a cronjob that logged uptime, RAM and used > Asterisk channels into a logfile every 10 minutes. > >> Am 16.09.2020 um 15:07 schrieb Lonnie Abelbeck <li...@lonnie.abelbeck.com>: >> >> Hi Daryl, >> >> Has the AstLinux version changed in the last couple weeks ? >> >> What version are you running ? >> >> Lonnie >> >> >> >>> On Sep 16, 2020, at 8:01 AM, Daryl Richards via Astlinux-users >>> <astlinux-users@lists.sourceforge.net> wrote: >>> >>> This is good timing (in a bad way..) Time for a "Me too!" >>> >>> Recently, over the last couple weeks my APU2 has started locking up exactly >>> the same way, just hard lock. I have a serial console cable hooked up and >>> there are no messages printed out before, there's just nothing. When the >>> problem first started I didn't have a console cable hooked up but I put it >>> in to see if any messages were appearing.. >>> >>> My system is on a UPS. Nothing else in the rack glitches. >>> >>> On 2020-09-15 8:36 p.m., Michael Knill wrote: >>>> Thanks Chris >>>> Yep I have plenty out there too with no issues. It also happened on a >>>> Qotom box so not hardware related. >>>> Regards >>>> Michael Knill >>>> *From: *AstLinux List <astlinux-users@lists.sourceforge.net> >>>> *Reply to: *AstLinux List <astlinux-users@lists.sourceforge.net> >>>> *Date: *Wednesday, 16 September 2020 at 8:53 am >>>> *To: *AstLinux List <astlinux-users@lists.sourceforge.net> >>>> *Cc: *The Cadillac Kid <eldorado...@yahoo.com> >>>> *Subject: *Re: [Astlinux-users] APU2 keeps locking up >>>> for whatever its worth.. from a hardware standpoint I have probably 200 >>>> APU2s in the field and dont have then just freeze like that.. granted >>>> they arent running astlinux, but just mentioning it from a power / >>>> hardware point of view. >>>> mine are all running Centos 6.X and asterisk 11 or 13 >>>> I have had a few chan_sip freezes.. (we wrote a watchdog to catch those >>>> and restart asterisk).. we get maybe 1 every 3 or 4 months .. (not each >>>> site but collectively) >>>> I have kenrel panicked an APU before by upping and downing the ethernet >>>> port too much (or so it seems).. it doesnt happen very often and is hard >>>> to repeat in the lab. but have done it esp on an install.. where one is >>>> prone to plug and unplug cables multiple times in succession for dressing >>>> in.. >>>> I have had a few power bricks go bad.. in all cases there was just no >>>> output.. no lights on the board at all. its not been that many 3 or 4.. >>>> considering we have sites in some pretty lightning-prone areas i dont feel >>>> too bad about a few power bricks. >>>> we run UPSs on all of our sites >>>> we run the US power supply that is sold on the PC-Engines store. >>>> On Tuesday, September 15, 2020, 6:04:44 PM EDT, Michael Knill >>>> <michael.kn...@ipcsolutions.com.au> wrote: >>>> Yep I would say environmental for 1, maybe 2 sites but for 3 sites and all >>>> only recently I cant see how it could be. >>>> No serial or long ethernet cables are connected. >>>> Yes completely different power adaptors when I changed from APU2 -> Qotom. >>>> Yes they are all DSL however ALL different modems and service types. >>>> I'm thinking I will upgrade all Runnix versions at these sites and if >>>> still happening upgrade to 1.3.10 and if still happening then I have no >>>> idea what to do. >>>> Regards >>>> Michael Knill >>>> On 16/9/20, 7:36 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com >>>> <mailto:li...@lonnie.abelbeck.com>> wrote: >>>> So a UPS may solve the issue as with Site 1 ? >>>> Do you have any serial cables connected ? >>>> Any long ethernet cables connected ? >>>> Is this with a variety of power adapters ? ie. the APU2 -> Qotom switch >>>> did the power adapter change as well ? >>>> Sure sounds environmental to me. >>>> I've seen DC-to-DC UPSs for about $40 USD, but never tried one. >>>> https://protectli.com/product/uninterruptible-power-supply/ >>>> Lonnie >>>>> On Sep 15, 2020, at 3:58 PM, Michael Knill >>>>> <michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>> >>>>> wrote: >>>>> >>>>> Ok I'm reviving this thread as I have now had my third site with this >>>>> issue. >>>>> >>>>> Symptom: >>>>> Astlinux completely locks up and requires a power reset. The log shows >>>>> NOTHING in all cases. >>>>> >>>>> Troubleshooting conducted: >>>>> Site 1 - The problem has not occurred since both a UPS AND Power Filter >>>>> have been added >>>>> Site 2 - A power filter has been added and the problem has reoccurred. I >>>>> have completely changed the system from an APU2 to Qotom and it did the >>>>> same thing again >>>>> Site 3 - A known working APU2 was installed and it locked up yesterday. >>>>> >>>>> I have checked mSATA cards and they were different across systems at >>>>> these sites. >>>>> All systems are running 1.3.7.1 but I am running this at many other sites >>>>> on the same hardware with no issues. >>>>> >>>>> Power quality testing is extremely expensive and surely I cant be having >>>>> a power issue at 3 sites! >>>>> The whole thing just doesn't make any sense and I don't know where to go >>>>> from here. >>>>> Any ideas? >>>>> >>>>> Regards >>>>> Michael Knill >>>>> >>>>> On 22/4/20, 9:24 pm, "Michael Knill" >>>>> <michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>> >>>>> wrote: >>>>> >>>>> Looks like this problem was bad power. I was told by the local IT Guy >>>>> that there are regular brownouts so I installed a UPS. No more problems >>>>> since doing so. >>>>> Seems like APU's don't like low voltage scenarios. >>>>> >>>>> Thanks for your help. >>>>> >>>>> Regards >>>>> Michael Knill >>>>> >>>>> On 13/4/20, 1:05 pm, "Michael Knill" >>>>> <michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>> >>>>> wrote: >>>>> >>>>> Yes could be but very unusual. I have never had this problem with any >>>>> other APU. >>>>> Maybe I will look for a good surge suppressor. >>>>> >>>>> Regards >>>>> Michael Knill >>>>> >>>>> On 13/4/20, 12:47 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com >>>>> <mailto:li...@lonnie.abelbeck.com>> wrote: >>>>> >>>>> Interesting, maybe bad power (spikes, noise, etc.) >>>>> >>>>> Test with a UPS attached or good surge suppresser. >>>>> >>>>> Lonnie >>>>> >>>>> >>>>> >>>>>> On Apr 12, 2020, at 9:17 PM, Michael Knill >>>>>> <michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>> >>>>>> wrote: >>>>>> >>>>>> Hi Lonnie >>>>>> >>>>>> I have replaced the hardware already and it did EXACTLY the same thing ☹ >>>>>> >>>>>> Regards >>>>>> Michael Knill >>>>>> >>>>>> On 13/4/20, 12:15 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com >>>>>> <mailto:li...@lonnie.abelbeck.com>> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> On Apr 12, 2020, at 7:03 PM, Michael Knill >>>>>>> <michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>> >>>>>>> wrote: >>>>>>> >>>>>>> May not have anything to do with Astlinux but I have a site which >>>>>>> completely locks up e.g. cannot even communicate using serial port. >>>>>>> On reboot its fine but there is nothing in the logs but the bootup >>>>>>> messages. >>>>>>> So far I have replaced the hardware with new storage as well and power >>>>>>> supply and it is still doing it. >>>>>>> >>>>>>> It is currently running Astlinux 1.3.7.1 which has been running fine on >>>>>>> another APU2 but its only been 8 days. >>>>>>> >>>>>>> Any ideas what I can do next? >>>>>>> >>>>>>> Regards >>>>>>> Michael Knill >>>>>> >>>>>> Sounds like an APU2 hardware issue. >>>>>> >>>>>> Maybe Pascal will give you a replacement. >>>>>> >>>>>> Lonnie >>>>>> Michael http://www.mksolutions.info _______________________________________________ Astlinux-users mailing list Astlinux-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/astlinux-users Donations to support AstLinux are graciously accepted via PayPal to pay...@krisk.org.
