Thanks Michael for closing the loop on this issue. Additionally, AstLinux 1.4.0 should solve this issue without needing the workaround.
Lonnie > On Oct 11, 2020, at 2:34 PM, Michael Knill > <michael.kn...@ipcsolutions.com.au> wrote: > > PS just letting the group know that I haven’t had any lockups since adding > this workaround which is certainly a relief. > Thanks all for your help. > > Regards > Michael Knill > > On 21/9/20, 8:34 am, "Michael Knill" <michael.kn...@ipcsolutions.com.au> > wrote: > > Thanks Lonnie for all your help. > > Regards > Michael Knill > > On 21/9/20, 7:37 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote: > > Hi Michael, > > Considering this is a very obscure kernel bug that has been around for > a long time, and AstLinux 1.4.x will have it fixed ... I personally would not > get carried away implementing the workaround. > > BTW, the PHYETH_DISABLE_OFFLOAD workaround only applies to physical > ethernet NICs. > > You understand the situation, so do what you think is best. > > Lonnie > > > > >> On Sep 20, 2020, at 3:45 PM, Michael Knill >> <michael.kn...@ipcsolutions.com.au> wrote: >> >> Thanks Lonnie >> >> One last question; could I just blanket add this to all my systems or are >> there any that will certainly not be affected by this bug e.g. VM or virtual >> NIC? >> The plan is to have this command active by default and commented out if >> desired. >> >> Regards >> Michael Knill >> >> On 20/9/20, 9:58 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote: >> >>> Do you think I should remove PHYETH_DISABLE_OFFLOAD and try a BIOS upgrade >>> to see if it fixes the problem? >> >> No ... first make sure setting PHYETH_DISABLE_OFFLOAD solves the issue >> with your DLS provider and your APU2s. >> >> After that, AstLinux 1.4.0 will be the solution without setting >> PHYETH_DISABLE_OFFLOAD . >> >> Ideally, you could test a 1.4-pre-release on an APU2 with the new DLS >> provider to make certain we have identified the issue. >> https://www.astlinux-project.org/dev.html >> >> Lonnie >> >> >>> On Sep 20, 2020, at 5:24 AM, Michael Knill >>> <michael.kn...@ipcsolutions.com.au> wrote: >>> >>> Yes my BIOS is quite old: >>> 3999-IPCBuild-CM1 kd # dmesg | grep DMI >>> [ 0.000000] DMI: PC Engines APU, BIOS SageBios_PCEngines_APU-45 >>> 04/05/2014 >>> >>> Funny as it's a pretty new box. >>> >>> Do you think I should remove PHYETH_DISABLE_OFFLOAD and try a BIOS upgrade >>> to see if it fixes the problem? >>> I don't think I will for existing sites as a BIOS upgrade looks pretty hard >>> to do and I assume cannot be done remotely. >>> >>> Regards >>> Michael Knill >>> >>> On 20/9/20, 12:18 pm, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote: >>> >>> Hi Michael, >>> >>> Ahhh, very good ... looks like we are on to something. >>> >>> Add it to one/some of your APU2s and let us know how it goes. >>> >>> As far as my Qotom Q190G4N, I initially had to set PHYETH_DISABLE_OFFLOAD >>> to keep it from locking-up with sustained high network traffic but then >>> switched the RAM SO-DIMM with another brand and did not need >>> PHYETH_DISABLE_OFFLOAD anymore. My comments probably got you started adding >>> PHYETH_DISABLE_OFFLOAD. >>> >>> This is a very obscure kernel bug, as such it never got back-ported to >>> Linux 3.16.x . >>> >>> For the APU2, the BIOS could play a role in how it initializes the NICs >>> and whether this kernel bug is triggered. >>> >>> Lonnie >>> >>> >>> >>>> On Sep 19, 2020, at 7:49 PM, Michael Knill >>>> <michael.kn...@ipcsolutions.com.au> wrote: >>>> >>>> Awesome thanks Lonnie. >>>> Yes its all making sense now. I already have this directive in my template >>>> against the Qotom Q190G4U for some reason (should I have?) >>>> I have two Qotoms connected to the problem provider, one had this >>>> directive set already and has not failed and one did not (I forgot to >>>> change when I changed hardware) which fails. >>>> All my APU's don't have this set so all have problems with this provider. >>>> >>>> I'm thinking we have finally solved this issue. >>>> Thanks so much for your help >>>> >>>> Regards >>>> Michael Knill >>>> >>>> On 20/9/20, 9:29 am, "Lonnie Abelbeck" <li...@lonnie.abelbeck.com> wrote: >>>> >>>> I would try this first >>>> -- >>>> PHYETH_DISABLE_OFFLOAD="tso gso gro" >>>> -- >>>> and see if it fixes the problem. >>>> >>>> If by chance it does fix it, then it would not be needed in AstLinux 1.4.x. >>>> >>>> The PHYETH_DISABLE_OFFLOAD settings disable some of the "offload" features >>>> of the NICs in an effort to work around this (somewhat obscure) kernel bug. >>>> >>>> It all kind of makes sense that a particular provider is fragmenting >>>> packets in ways others do not, and hits this kernel bug. >>>> >>>> The PHYETH_DISABLE_OFFLOAD setting above is very "safe", only drawback is >>>> it slightly reduces network performance near the 1 Gbps level. >>>> >>>> BTW, if traffic shaping is enabled this PHYETH_DISABLE_OFFLOAD setting is >>>> already applied to external ethernet NIC(s). >>>> >>>> Lonnie >>>> >>>> >>>> >>>>> On Sep 19, 2020, at 6:09 PM, Michael Knill >>>>> <michael.kn...@ipcsolutions.com.au> wrote: >>>>> >>>>> Awesome thanks Lonnie. >>>>> >>>>> I will give it a try although I have no idea what it does! >>>>> I assume I can remove this when I go to Astlinux 1.4? >>>>> >>>>> Regards >>>>> Michael Knill >>>>> >>>>> Sent from my iPhone so please excuse my brevity. >>>>> >>>>>> On 19 Sep 2020, at 11:56 pm, Lonnie Abelbeck <li...@lonnie.abelbeck.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>> Hi Michael, >>>>>> >>>>>> Great info! >>>>>> >>>>>> Try this in your user.conf, and reboot. >>>>>> -- >>>>>> PHYETH_DISABLE_OFFLOAD="tso gso gro" >>>>>> -- >>>>>> >>>>>> If my hunch is correct, this kernel fix added in 4.1.17 may be related >>>>>> ... >>>>>> >>>>>> net: preserve IP control block during GSO segmentation >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/include/linux/skbuff.h?h=v4.1.17&id=abefd1b4087b9b5e83e7b4e7689f8b8e3cb2899c >>>>>> >>>>>> Lonnie >>>>>> >>>>>> >>>>>> >>>>>>> On Sep 18, 2020, at 10:56 PM, Michael Knill >>>>>>> <michael.kn...@ipcsolutions.com.au> wrote: >>>>>>> >>>>>>> Yay some progress on this problem. >>>>>>> >>>>>>> I had my 4th site lock up yesterday. It was a site I moved from one >>>>>>> location to another. There were no changes to the Astlinux box at all >>>>>>> other than PPPoE credentials but after a couple of hours it locked up. >>>>>>> So realistically the only change is the internet provider which is a >>>>>>> new one that I am trialling and is the same provider as two of the >>>>>>> other sites that are failing. >>>>>>> >>>>>>> As we are also using this provider in our home office, I set up another >>>>>>> box this morning and connected the serial port not expecting anything >>>>>>> to happen but it locked up and we captured it. Yay! It is attached. >>>>>>> >>>>>>> I'm hoping it will help the resolution of this problem. >>>>>>> >>>>>>> Regards >>>>>>> Michael Knill >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Astlinux-users mailing list >>>>>> Astlinux-users@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>>>>> >>>>>> Donations to support AstLinux are graciously accepted via PayPal to >>>>>> pay...@krisk.org. >>>>>> <APU Crash.log> >>>>> _______________________________________________ >>>>> Astlinux-users mailing list >>>>> Astlinux-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>>>> >>>>> Donations to support AstLinux are graciously accepted via PayPal to >>>>> pay...@krisk.org. >>>> >>>> >>>> >>>> _______________________________________________ >>>> Astlinux-users mailing list >>>> Astlinux-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>>> >>>> Donations to support AstLinux are graciously accepted via PayPal to >>>> pay...@krisk.org. >>>> >>>> >>>> _______________________________________________ >>>> Astlinux-users mailing list >>>> Astlinux-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>>> >>>> Donations to support AstLinux are graciously accepted via PayPal to >>>> pay...@krisk.org. >>> >>> >>> >>> _______________________________________________ >>> Astlinux-users mailing list >>> Astlinux-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>> >>> Donations to support AstLinux are graciously accepted via PayPal to >>> pay...@krisk.org. >>> >>> >>> _______________________________________________ >>> Astlinux-users mailing list >>> Astlinux-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/astlinux-users >>> >>> Donations to support AstLinux are graciously accepted via PayPal to >>> pay...@krisk.org. >> >> >> >> _______________________________________________ >> Astlinux-users mailing list >> Astlinux-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/astlinux-users >> >> Donations to support AstLinux are graciously accepted via PayPal to >> pay...@krisk.org. >> >> >> _______________________________________________ >> Astlinux-users mailing list >> Astlinux-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/astlinux-users >> >> Donations to support AstLinux are graciously accepted via PayPal to >> pay...@krisk.org. > > > > _______________________________________________ > Astlinux-users mailing list > Astlinux-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/astlinux-users > > Donations to support AstLinux are graciously accepted via PayPal to > pay...@krisk.org. > > > _______________________________________________ > Astlinux-users mailing list > Astlinux-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/astlinux-users > > Donations to support AstLinux are graciously accepted via PayPal to > pay...@krisk.org. > > > _______________________________________________ > Astlinux-users mailing list > Astlinux-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/astlinux-users > > Donations to support AstLinux are graciously accepted via PayPal to > pay...@krisk.org. _______________________________________________ Astlinux-users mailing list Astlinux-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/astlinux-users Donations to support AstLinux are graciously accepted via PayPal to pay...@krisk.org.
