On 19/09/2017 23:02, Buterbaugh, Kevin L wrote: > Hi All, > > We have a small test cluster that is CCR enabled. It only had/has 3 NSD > servers (testnsd1, 2, and 3) and maybe 3-6 clients. testnsd3 died a while > back. I did nothing about it at the time because it was due to be > life-cycled as soon as I finished a couple of higher priority projects. > > Yesterday, testnsd1 also died, which took the whole cluster down. So now > resolving this has become higher priority… ;-) > > I took two other boxes and set them up as testnsd1 and 3, respectively. I’ve > done a “mmsdrrestore -p testnsd2 -R /usr/bin/scp” on both of them. I’ve also > done a "mmccr setup -F” and copied the ccr.disks and ccr.nodes files from > testnsd2 to them. And I’ve copied /var/mmfs/gen/mmsdrfs from testnsd2 to > testnsd1 and 3. In case it’s not obvious from the above, networking is fine > … ssh without a password between those 3 boxes is fine. > > However, when I try to startup GPFS … or run any GPFS command I get: > > /root > root@testnsd2# mmstartup -a > get file failed: Not enough CCR quorum nodes available (err 809) > gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158 > mmstartup: Command failed. Examine previous error messages to determine cause. > /root > root@testnsd2# > > I’ve got to run to a meeting right now, so I hope I’m not leaving out any > crucial details here … does anyone have an idea what I need to do? Thanks… > Hello, I have had the same issue multiple times.
The "trick" is to execute "/usr/lpp/mmfs/bin/mmcommon startCcrMonitor" on a majority of quorum nodes (once they have the correct configuration files) to be able to start the cluster. I noticed a call to the above command in the "gpfs.gplbin" spec file in the "%postun" section (when doing RPM upgrades, if I'm not mistaken). <Insert here rant about CCR design & testing>. Loïc. -- | Loïc Tortay <[email protected]> - IN2P3 Computing Centre | _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
