Yay some progress on this problem. I had my 4th site lock up yesterday. It was a site I moved from one location to another. There were no changes to the Astlinux box at all other than PPPoE credentials but after a couple of hours it locked up. So realistically the only change is the internet provider which is a new one that I am trialling and is the same provider as two of the other sites that are failing.
As we are also using this provider in our home office, I set up another box
this morning and connected the serial port not expecting anything to happen but
it locked up and we captured it. Yay! It is attached.
I'm hoping it will help the resolution of this problem.
Regards
Michael Knill
On 17/9/20, 10:29 am, "Michael Knill" <michael.kn...@ipcsolutions.com.au>
wrote:
That's correct.
Regards
Michael Knill
On 17/9/20, 9:34 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote:
> I have had EXACTLY the same problem on a Qotom box.
But no UPS, correct ?
Lonnie
> On Sep 16, 2020, at 5:54 PM, Michael Knill
<michael.kn...@ipcsolutions.com.au> wrote:
>
> Hi sorry all but this is not an APU problem. I have had EXACTLY the
same problem on a Qotom box.
>
> At all three sites the common denominators are:
> -- Astlinux 1.3.7.1
> -- PPPoE
>
> My next tests will be (in this order):
> 1) Install another system onsite connected to the same environment
e.g. power to see if it happens to both systems
> 2) Put a firewall in front of the Astlinux box to do PPPoE
> 3) OS upgrade
>
> Regards
> Michael Knill
>
> On 17/9/20, 8:12 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com>
wrote:
>
> Yes, this is very strange.
>
> I have had an APU2 for years, never locked-up once ... and it is
currently running a UniFi controller in a Linux container.
>
> I also ran my APU2 without a UPS for a year or two, though it is
using one now. But we have good quality power where I live outside of
thunder-storms.
>
> Possibly the Coreboot BIOS version also needs to be tracked to
find some common denominator. I have never updated my APU2 coreboot BIOS.
>
> Also my APU2 sits inside my network, without a public interface.
>
> A common denominator list for APU2 lock-up reports:
>
> -- AstLinux version:
> -- Asterisk: 13se, 13 or 16
> -- RAM: 2GB or 4GB
> -- Coreboot BIOS version:
> -- Public interface yes/no, if 'yes' public SSH, HTTPS, etc.
access yes/no
> -- PPPoE: yes/no
> -- UPS Power: yes/no
>
>
> Lonnie
>
>
>
>
>> On Sep 16, 2020, at 4:09 PM, Michael Knill
<michael.kn...@ipcsolutions.com.au> wrote:
>>
>> I do have Zabbix running on these boxes which checks memory and will
notify me if its getting too low and this has not happened. This is what
alerted me to the Zabbix client memory leak.
>> Also the lockups are very sporadic. Can happen after a couple of
weeks or the next day.
>>
>> Regards
>> Michael Knill
>>
>> On 16/9/20, 11:21 pm, "Michael Keuter" <li...@mksolutions.info>
wrote:
>>
>> Hi all,
>>
>> it might be completely unrelated, but it reminds me at an issue
that only occured many years ago only on 32bit Geode boxes (Alix/net5501) after
we integrated Monit into AstLinux.
>> On these boxes all RAM was eaten up slowly and then these boxes
locked up after a few days. That's why we didn't installed Monit for these
build types.
>>
>> To figure it out I then created a cronjob that logged uptime, RAM
and used Asterisk channels into a logfile every 10 minutes.
>>
>>> Am 16.09.2020 um 15:07 schrieb Lonnie Abelbeck
<li...@lonnie.abelbeck.com>:
>>>
>>> Hi Daryl,
>>>
>>> Has the AstLinux version changed in the last couple weeks ?
>>>
>>> What version are you running ?
>>>
>>> Lonnie
>>>
>>>
>>>
>>>> On Sep 16, 2020, at 8:01 AM, Daryl Richards via Astlinux-users
<astlinux-users@lists.sourceforge.net> wrote:
>>>>
>>>> This is good timing (in a bad way..) Time for a "Me too!"
>>>>
>>>> Recently, over the last couple weeks my APU2 has started locking
up exactly the same way, just hard lock. I have a serial console cable hooked
up and there are no messages printed out before, there's just nothing. When the
problem first started I didn't have a console cable hooked up but I put it in
to see if any messages were appearing..
>>>>
>>>> My system is on a UPS. Nothing else in the rack glitches.
>>>>
>>>> On 2020-09-15 8:36 p.m., Michael Knill wrote:
>>>>> Thanks Chris
>>>>> Yep I have plenty out there too with no issues. It also happened
on a Qotom box so not hardware related.
>>>>> Regards
>>>>> Michael Knill
>>>>> *From: *AstLinux List <astlinux-users@lists.sourceforge.net>
>>>>> *Reply to: *AstLinux List <astlinux-users@lists.sourceforge.net>
>>>>> *Date: *Wednesday, 16 September 2020 at 8:53 am
>>>>> *To: *AstLinux List <astlinux-users@lists.sourceforge.net>
>>>>> *Cc: *The Cadillac Kid <eldorado...@yahoo.com>
>>>>> *Subject: *Re: [Astlinux-users] APU2 keeps locking up
>>>>> for whatever its worth.. from a hardware standpoint I have
probably 200 APU2s in the field and dont have then just freeze like that..
granted they arent running astlinux, but just mentioning it from a power /
hardware point of view.
>>>>> mine are all running Centos 6.X and asterisk 11 or 13
>>>>> I have had a few chan_sip freezes.. (we wrote a watchdog to catch
those and restart asterisk).. we get maybe 1 every 3 or 4 months .. (not each
site but collectively)
>>>>> I have kenrel panicked an APU before by upping and downing the
ethernet port too much (or so it seems).. it doesnt happen very often and is
hard to repeat in the lab. but have done it esp on an install.. where one is
prone to plug and unplug cables multiple times in succession for dressing in..
>>>>> I have had a few power bricks go bad.. in all cases there was
just no output.. no lights on the board at all. its not been that many 3 or
4.. considering we have sites in some pretty lightning-prone areas i dont feel
too bad about a few power bricks.
>>>>> we run UPSs on all of our sites
>>>>> we run the US power supply that is sold on the PC-Engines store.
>>>>> On Tuesday, September 15, 2020, 6:04:44 PM EDT, Michael Knill
<michael.kn...@ipcsolutions.com.au> wrote:
>>>>> Yep I would say environmental for 1, maybe 2 sites but for 3
sites and all only recently I cant see how it could be.
>>>>> No serial or long ethernet cables are connected.
>>>>> Yes completely different power adaptors when I changed from APU2
-> Qotom.
>>>>> Yes they are all DSL however ALL different modems and service
types.
>>>>> I'm thinking I will upgrade all Runnix versions at these sites
and if still happening upgrade to 1.3.10 and if still happening then I have no
idea what to do.
>>>>> Regards
>>>>> Michael Knill
>>>>> On 16/9/20, 7:36 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com
<mailto:li...@lonnie.abelbeck.com>> wrote:
>>>>> So a UPS may solve the issue as with Site 1 ?
>>>>> Do you have any serial cables connected ?
>>>>> Any long ethernet cables connected ?
>>>>> Is this with a variety of power adapters ? ie. the APU2 ->
Qotom switch did the power adapter change as well ?
>>>>> Sure sounds environmental to me.
>>>>> I've seen DC-to-DC UPSs for about $40 USD, but never tried one.
>>>>> https://protectli.com/product/uninterruptible-power-supply/
>>>>> Lonnie
>>>>>> On Sep 15, 2020, at 3:58 PM, Michael Knill
<michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>>
wrote:
>>>>>>
>>>>>> Ok I'm reviving this thread as I have now had my third site with
this issue.
>>>>>>
>>>>>> Symptom:
>>>>>> Astlinux completely locks up and requires a power reset. The log
shows NOTHING in all cases.
>>>>>>
>>>>>> Troubleshooting conducted:
>>>>>> Site 1 - The problem has not occurred since both a UPS AND Power
Filter have been added
>>>>>> Site 2 - A power filter has been added and the problem has
reoccurred. I have completely changed the system from an APU2 to Qotom and it
did the same thing again
>>>>>> Site 3 - A known working APU2 was installed and it locked up
yesterday.
>>>>>>
>>>>>> I have checked mSATA cards and they were different across
systems at these sites.
>>>>>> All systems are running 1.3.7.1 but I am running this at many
other sites on the same hardware with no issues.
>>>>>>
>>>>>> Power quality testing is extremely expensive and surely I cant
be having a power issue at 3 sites!
>>>>>> The whole thing just doesn't make any sense and I don't know
where to go from here.
>>>>>> Any ideas?
>>>>>>
>>>>>> Regards
>>>>>> Michael Knill
>>>>>>
>>>>>> On 22/4/20, 9:24 pm, "Michael Knill"
<michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>>
wrote:
>>>>>>
>>>>>> Looks like this problem was bad power. I was told by the local
IT Guy that there are regular brownouts so I installed a UPS. No more problems
since doing so.
>>>>>> Seems like APU's don't like low voltage scenarios.
>>>>>>
>>>>>> Thanks for your help.
>>>>>>
>>>>>> Regards
>>>>>> Michael Knill
>>>>>>
>>>>>> On 13/4/20, 1:05 pm, "Michael Knill"
<michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>>
wrote:
>>>>>>
>>>>>> Yes could be but very unusual. I have never had this problem
with any other APU.
>>>>>> Maybe I will look for a good surge suppressor.
>>>>>>
>>>>>> Regards
>>>>>> Michael Knill
>>>>>>
>>>>>> On 13/4/20, 12:47 pm, "Lonnie Abelbeck"
<li...@lonnie.abelbeck.com <mailto:li...@lonnie.abelbeck.com>> wrote:
>>>>>>
>>>>>> Interesting, maybe bad power (spikes, noise, etc.)
>>>>>>
>>>>>> Test with a UPS attached or good surge suppresser.
>>>>>>
>>>>>> Lonnie
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Apr 12, 2020, at 9:17 PM, Michael Knill
<michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>>
wrote:
>>>>>>>
>>>>>>> Hi Lonnie
>>>>>>>
>>>>>>> I have replaced the hardware already and it did EXACTLY the
same thing ☹
>>>>>>>
>>>>>>> Regards
>>>>>>> Michael Knill
>>>>>>>
>>>>>>> On 13/4/20, 12:15 pm, "Lonnie Abelbeck"
<li...@lonnie.abelbeck.com <mailto:li...@lonnie.abelbeck.com>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Apr 12, 2020, at 7:03 PM, Michael Knill
<michael.kn...@ipcsolutions.com.au<mailto:michael.kn...@ipcsolutions.com.au>>
wrote:
>>>>>>>>
>>>>>>>> May not have anything to do with Astlinux but I have a site
which completely locks up e.g. cannot even communicate using serial port.
>>>>>>>> On reboot its fine but there is nothing in the logs but the
bootup messages.
>>>>>>>> So far I have replaced the hardware with new storage as well
and power supply and it is still doing it.
>>>>>>>>
>>>>>>>> It is currently running Astlinux 1.3.7.1 which has been
running fine on another APU2 but its only been 8 days.
>>>>>>>>
>>>>>>>> Any ideas what I can do next?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Michael Knill
>>>>>>>
>>>>>>> Sounds like an APU2 hardware issue.
>>>>>>>
>>>>>>> Maybe Pascal will give you a replacement.
>>>>>>>
>>>>>>> Lonnie
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Astlinux-users mailing list
>>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>>
>>>>>>> Donations to support AstLinux are graciously accepted via
PayPal to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Astlinux-users mailing list
>>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>>
>>>>>>> Donations to support AstLinux are graciously accepted via
PayPal to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Astlinux-users mailing list
>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>
>>>>>> Donations to support AstLinux are graciously accepted
via PayPal to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Astlinux-users mailing list
>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>
>>>>>> Donations to support AstLinux are graciously accepted via
PayPal to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Astlinux-users mailing list
>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>
>>>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Astlinux-users mailing list
>>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>>>
>>>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Astlinux-users@lists.sourceforge.net
<mailto:Astlinux-users@lists.sourceforge.net>
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org. <mailto:pay...@krisk.org.>
>>>>> _______________________________________________
>>>>> Astlinux-users mailing list
>>>>> Astlinux-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org.
>>>>
>>>> --
>>>> Daryl Richards
>>>> Isle Technical Services Inc.
>>>>
>>>>
>>>> _______________________________________________
>>>> Astlinux-users mailing list
>>>> Astlinux-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>>
>>>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org.
>>>
>>>
>>>
>>> _______________________________________________
>>> Astlinux-users mailing list
>>> Astlinux-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>>
>>> Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
>>
>>
>> Michael
>>
>> http://www.mksolutions.info
>>
>>
>>
>>
>>
>> _______________________________________________
>> Astlinux-users mailing list
>> Astlinux-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>
>> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org.
>>
>>
>> _______________________________________________
>> Astlinux-users mailing list
>> Astlinux-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>>
>> Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
>
>
>
> _______________________________________________
> Astlinux-users mailing list
> Astlinux-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal
to pay...@krisk.org.
>
>
> _______________________________________________
> Astlinux-users mailing list
> Astlinux-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
_______________________________________________
Astlinux-users mailing list
Astlinux-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/astlinux-users
Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
_______________________________________________
Astlinux-users mailing list
Astlinux-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/astlinux-users
Donations to support AstLinux are graciously accepted via PayPal to
pay...@krisk.org.
APU Crash.log
Description: APU Crash.log
_______________________________________________ Astlinux-users mailing list Astlinux-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/astlinux-users Donations to support AstLinux are graciously accepted via PayPal to pay...@krisk.org.
