Yes, I've been struggelig with something similiar this week. Ganesha dying with SIGABRT -- nothing else logged. After catching a few coredumps, it has been identified as a problem with some udp-communication during mounts from solaris clients. Disabling udp as transport on the shares serverside didn't help. It was suggested to use "mount -o tcp" or whatever the solaris version of this is -- but we haven't tested this. So far the downgrade to v2.3.2 has been our workaround.
PMR: 48669,080,678 -jf On Mon, Apr 23, 2018 at 12:38 AM, Ray Coetzee <[email protected]> wrote: > Good evening all > > I'm working with IBM on a PMR where ganesha is segfaulting or causing > kernel panics on one group of CES nodes. > > We have 12 identical CES nodes split into two groups of 6 nodes each & > have been running with RHEL 7.3 & GPFS 5.0.0-1 since 5.0.0-1 was released. > > Only one group started having issues Monday morning where ganesha would > segfault and the mounts would move over to the remaining nodes. > The remaining nodes then start to fall over like dominos within minutes or > hours to the point that all CES nodes are "failed" according to "mmces node > list" and the VIP's are unassigned. > > Recovering the nodes are extremely finicky and works for a few minutes or > hours before segfaulting again. > Most times a complete stop of Ganesha on all nodes & then only starting it > on two random nodes allow mounts to recover for a while. > > None of the following has helped: > A reboot of all nodes. > Refresh CCR config file with mmsdrrestore > Remove/add CES from nodes. > Reinstall GPFS & protocol rpms > Update to 5.0.0-2 > Fresh reinstall of a node > Network checks out with no dropped packets on either data or export > networks. > > The only temporary fix so far has been to downrev ganesha to 2.3.2 from > 2.5.3 on the affected nodes. > > While waiting for IBM development, has anyone seen something similar maybe? > > Kind regards > > Ray Coetzee > > > > On Sat, Apr 21, 2018 at 12:00 PM, <gpfsug-discuss-request@ > spectrumscale.org> wrote: > >> Send gpfsug-discuss mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of gpfsug-discuss digest..." >> >> >> Today's Topics: >> >> 1. Re: UK Meeting - tooling Spectrum Scale (Grunenberg, Renar) >> 2. Re: UK Meeting - tooling Spectrum Scale >> (Simon Thompson (IT Research Support)) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Fri, 20 Apr 2018 14:01:55 +0000 >> From: "Grunenberg, Renar" <[email protected]> >> To: "'[email protected]'" >> <[email protected]> >> Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset="utf-8" >> >> Hallo Simon, >> are there any reason why the link of the presentation from Yong ZY >> Zheng(Cognitive, ML, Hortonworks) is not linked. >> >> Renar Grunenberg >> Abteilung Informatik ? Betrieb >> >> HUK-COBURG >> Bahnhofsplatz >> 96444 Coburg >> Telefon: 09561 96-44110 >> Telefax: 09561 96-44104 >> E-Mail: [email protected] >> Internet: www.huk.de >> ________________________________ >> HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter >> Deutschlands a. G. in Coburg >> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021 >> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg >> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin. >> Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans >> Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas. >> ________________________________ >> Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte >> Informationen. >> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich >> erhalten haben, >> informieren Sie bitte sofort den Absender und vernichten Sie diese >> Nachricht. >> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht >> ist nicht gestattet. >> >> This information may contain confidential and/or privileged information. >> If you are not the intended recipient (or have received this information >> in error) please notify the >> sender immediately and destroy this information. >> Any unauthorized copying, disclosure or distribution of the material in >> this information is strictly forbidden. >> ________________________________ >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/2018 >> 0420/91e3d84d/attachment-0001.html> >> >> ------------------------------ >> >> Message: 2 >> Date: Fri, 20 Apr 2018 14:12:11 +0000 >> From: "Simon Thompson (IT Research Support)" <[email protected]> >> To: gpfsug main discussion list <[email protected]> >> Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset="utf-8" >> >> Sorry, it was a typo from my side. >> >> The talks that are missing we are chasing for copies of the slides that >> we can release. >> >> Simon >> >> From: <[email protected]> on behalf of " >> [email protected]" <[email protected]> >> Reply-To: "[email protected]" < >> [email protected]> >> Date: Friday, 20 April 2018 at 15:02 >> To: "[email protected]" <[email protected]> >> Subject: Re: [gpfsug-discuss] UK Meeting - tooling Spectrum Scale >> >> Hallo Simon, >> are there any reason why the link of the presentation from Yong ZY >> Zheng(Cognitive, ML, Hortonworks) is not linked. >> >> Renar Grunenberg >> Abteilung Informatik ? Betrieb >> >> HUK-COBURG >> Bahnhofsplatz >> 96444 Coburg >> Telefon: >> >> 09561 96-44110 >> >> Telefax: >> >> 09561 96-44104 >> >> E-Mail: >> >> [email protected] >> >> Internet: >> >> www.huk.de >> >> ________________________________ >> HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter >> Deutschlands a. G. in Coburg >> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021 >> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg >> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin. >> Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans >> Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas. >> ________________________________ >> Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte >> Informationen. >> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich >> erhalten haben, >> informieren Sie bitte sofort den Absender und vernichten Sie diese >> Nachricht. >> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht >> ist nicht gestattet. >> >> This information may contain confidential and/or privileged information. >> If you are not the intended recipient (or have received this information >> in error) please notify the >> sender immediately and destroy this information. >> Any unauthorized copying, disclosure or distribution of the material in >> this information is strictly forbidden. >> ________________________________ >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/2018 >> 0420/0b8e9ffa/attachment-0001.html> >> >> ------------------------------ >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> End of gpfsug-discuss Digest, Vol 75, Issue 34 >> ********************************************** >> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
