On Sat, Jul 9, 2016 at 10:02 PM, Atin Mukherjee <[email protected]> wrote: > We have hit a bug 1347250 in downstream (applicable upstream too) where it > was seen that glusterd didnt regenerate the volfiles when it was interimly > brought up with upgrade mode by yum. Log file captured that gsyncd --version > failed to execute and hence glusterd init couldnt proceed till the volfile > regeneration. Since the ret code is not handled here in spec file users > wouldnt come to know about this and going forward this is going to cause > major issues in healing and all and finally it exploits the possibility of > split brains at its best. > > Further analysis by Kotresh & Raghavendra Talur reveals that gsyncd failed > here because of the compatibility issue where gsyncd was still not upgraded > where as glusterfs-server was and this failure was mainly because of change > in the mem type enum. We have seen a similar issue for RDMA as well > (probably a year back). So to be very generic this can happen in any upgrade > path from one version to another where new mem type is introduced. We have > seen this from 3.7.8 to 3.7.12 and 3.8. People upgrading from 3.6 to 3.7/3.8 > will also experience this issue. > > Till we work on this fix, I suggest all the release managers to highlight > this in the release note of the latest releases with the following work > around after yum update: > > 1. grep -irns "geo-replication module not working as desired" > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l > > If the output is non-zero, then go to step 2 else follow the rest of the > steps as per the guide. > > 2.Check if glusterd instance is running or not by 'ps aux | grep glusterd', > if it is, then stop the glusterd service. > > 3. glusterd --xlator-option *.upgrade=on -N > > and then proceed ahead with rest of the steps as per the guide. > > Thoughts?
Proper .so versioning of libglusterfs should help with problems like this. I don't know how to do this though. But I do have some thoughts to share on using GlusterDs upgrade-mode. GlusterD depends on the cluster op-version when generating volfiles, to insert new features/xlators into the volfile graph. This was done to make sure that the homogeneity of the volfiles is preserved across the cluster. This behaviour makes running GlusterD in upgrade mode after a package upgrade, essentially a noop. The cluster op-version doesn't change automatically when packages are upgraded, so the regenerated volfiles in the post-upgrade section are basically the same as before. (If something is getting added into volfiles after this, it is incorrect, and is something I'm yet to check). The correct time to regenerate the volfiles is after all members of the cluster have been upgraded and the cluster op-version has been bumped. (Bumping op-version doesn't regenerate anything, it is just an indication that the cluster is now ready to use new features.) We don't have a direct way to get volfiles regenerated on all members with a single command yet. We can implement such a command with relative ease. For now, volfiles can regenerated by making use of the `volume set` command, by setting a `user.upgrade` option on a volume. Options in the `user.` namespace are passed on to hook scripts and not added into any volfiles, but setting such an option on a volume causes GlusterD to regenerate volfiles for the volume. My suggestion would be to stop using glusterd in upgrade mode during post-upgrade to regenerate volfiles, and document the above way to get volfiles regenerated across the cluster correctly. We could do away with upgrade mode itself, but it could be useful for other things (Though I can't think of any right now). What do the other maintainers feel about this? ~kaushal PS: If this discussion is distracting from the original conversation, I'll start a new thread. > > P.S : this email is limited to maintainers till we decide on the approach to > highlight this issues to the users > > > -- > Atin > Sent from iPhone > > _______________________________________________ > maintainers mailing list > [email protected] > http://www.gluster.org/mailman/listinfo/maintainers > _______________________________________________ maintainers mailing list [email protected] http://www.gluster.org/mailman/listinfo/maintainers
