Thanks for the review, Lisa. Have incorporated all your suggestions. Thanks & Regards, Sambit
Lisa.Shepherd at Sun.COM wrote: > Sambit, > > Some minor changes in the user-visible text: > > ccradm.cc > - Change "...changes after split-brain" to "...changes after split > brain" (no hyphen is needed when there is no noun being modified). > > updatable_copy_impl.cc > - In the SCMSGS user action, change "split-brain" to "split brain", but > add a hyphen to change "weak membership model" to "weak-membership > model" (here "weak-membership" is modifying the noun "model"). > > path_manager.cc > - In the PM_DBG messages immediately before the SCMSGS text, change > "done after split-brain" to "done after split brain" (no hyphen). > - In the SCMSGS user action, change "non-cluster mode" to "noncluster > mode" (multiple instances). > - In the error message itself, change "after split-brain" to "after > split brain". > > quorum_impl.cc > - In the SCMSGS explanation, change "...determine the required amount of > quorum votes required for survival..." to "...determine the number of > quorum votes that are required for survival...". > - Change "CMM: We could not..." to "CMM: Could not..." (multiple instances). > > Otherwise, user-visible text looks good. > > Thanks. > > Lisa Shepherd > Sun Cluster Technical Publications > "We're the M in RTFM" > > > > On 12/15/08 01:39, Sambit Nayak wrote: > >> Hi everyone, >> >> Please review the changes in CMM and CCR that support >> the new "weaker" form of membership being introduced >> with Project Colorado. This "weaker" form of membership >> allows multiple partitions to survive if a split brain >> situation develops. >> >> You can access the webrev at : >> http://cr.opensolaris.org/~samnayak/colorado-I-CMM-CCR/ >> >> >> ********* >> Here is a summary of the CMM and CCR changes. >> Please refer to the Requirements Document for Project Colorado >> to understand what these changes in CMM and CCR are trying to achieve. >> >> >> SUMMARY OF CMM CHANGES >> ---------------------- >> 1. cmm_impl::read_membership_info_from_ccr() reads membership properties >> from CCR and stores them into the CMM 'conf' structure >> (structure declaration in cmm_config.h). >> >> The properties read from infrastructure table in CCR are >> a. multiple_partitions : true/false which indicates whether >> cluster allows multiple partitions to survive >> b. ping_targets : comma separated list of IP addresses >> that the cluster nodes ping to check their own health >> during CMM reconfiguration. This ping is done only >> if cluster allows multiple partitions to survive >> (weaker form of membership) >> >> This 'read' is done when the cluster node boots up first, >> and also when infrastructure table (that stores these properties) >> is modified. >> >> >> 2. CMM has methods register_cmm_for_infr_callback() and >> unregister_cmm_for_infr_callback() to register/unregister with CCR >> for infrastructure table update callbacks. >> When infrastructure table changes, CCR delivers callbacks to CMM >> and CMM reads in its required membership information. >> >> The callback object registered by CMM is of type infr_cb_impl_for_cmm >> (introduced in this set of changes). >> >> >> 3. cmm_config.h declares the 'conf' structure that holds >> properties read in by CMM from CCR. >> We add a boolean value called multiple_partitions and >> a list of strings called ping_targets that stores the IP addresses. >> >> 4. During CMM reconfiguration, the CMM automaton does the ping check >> if cluster is configured to allow multiple partitions to survive. >> This functionality is implemented in automaton_impl::ping_health_check(). >> It essentially does a door upcall to userland qd_userd daemon >> in order to execute the ping. >> If the door upcall cannot be performed or the ping fails, >> then the node is panicked by the automaton considering that >> the ping check failed. >> >> 5. For the weaker form of membership that allows multiple partitions >> to survive, we alter the definition of required quorum votes. >> For the usual strong form of membership, the definition was : >> Q = (V/2) +1 where Q is the required quorum votes and V is the total votes. >> For the new weaker form of membership, the definition is : >> Q = 1, if V = 1 or 2; >> and we do not allow weaker form of membership if V > 2. >> If the cluster has more than 2 nodes, strong form of membership is used. >> >> 6. When a node tries to join another node, a path is first formed >> before CMM knows that a remote node is reachable and starts reconfiguration. >> When a node receives a request to form path, >> path_manager::update_node_incarnation() does checks based on >> incarnation numbers before allowing the path formation. >> An additional check is introduced here to see if the local node >> has any CCR changes done during split-brain that are still unresolved. >> If so, then the node will reject the request of path formation. >> Thus, a node that has unresolved post-split-brain CCR changes >> will not allow other nodes to join it. >> The CCR changes have to be resolved (nodes have to be marked winner/loser) >> before the nodes can join. >> >> >> SUMMARY OF CCR CHANGES >> ---------------------- >> >> 1. Flag To Indicate CCR Change During Split-Brain >> >> Existence of a file "/etc/cluster/.split_brain_ccr_change" >> on a cluster node serves as an indicator that CCR changes >> were done during split-brain on that node. >> >> The file is created (if it doesn't already exist) upon >> successful completion of a CCR transaction during split-brain. >> >> The file is defined as SPLIT_BRAIN_CHANGE_FILE: >> >> #define SPLIT_BRAIN_CHANGE_FILE >> "/etc/cluster/.split_brain_ccr_change" >> >> It is created with 600 permission bits, owned by root. >> >> The split-brain change file will be created at the end of >> updatable_copy_impl::commit_transaction(), if all the previous >> steps of the commit were successful. >> >> 2. New convenience functions in os class >> >> os::file_create() to create files. >> The creation mode wil be O_EXCL | O_CREAT. >> >> os::file_exists() will be used to query whether or not >> there is CCR change during split-brain on this node. >> >> 3. How To Indicate A Table Was Changed During Split-Brain >> >> No special indication present in the changed table itself. >> Let the generation number increase normally whenever the table >> is changed. The presence of the SPLIT_BRAIN_CHANGE_FILE will >> indicate that some change exists. >> >> 4. How To Select The Winning CCR Copy >> >> The administrator will run a CLI utility to mark the CCR >> on a cluster node as the truth copy. >> (The CLI is not part of the present set of changes, >> but the CCR interfaces that it will use are present.) >> >> The CCR interface will delete the SPLIT_BRAIN_CHANGE_FILE. >> The CCR tables themselves are not modified. >> >> 5. How To Select The Loser CCR Copy >> >> The administrator will run a CLI utility to mark the CCR >> on a cluster node as the losing copy. >> (The CLI is not part of the present set of changes, >> but the CCR interfaces that it will use are present.) >> >> The CCR interface will delete the SPLIT_BRAIN_CHANGE_FILE >> is deleted, and then every table on the losing node >> is marked as invalid (generation number -1). Marking every >> table with this generation number -1 will ensure that >> when this node joins the winner node to form cluster, >> it will update its CCR copy using the winner node's CCR. >> The loser node cannot form a cluster of its own, as well. >> >> Note that this command to mark the loser has to be run >> in non-cluster mode. The node must then be rebooted >> into cluster mode. >> >> 6. CCR Interfaces Provided >> >> a. Interface to query if this node has split-brain CCR change(s). >> Exists both in kernel and userland. >> Implementation : Check if SPLIT_BRAIN_CHANGE_FILE exists. >> >> bool ccrlib::split_brain_ccr_change_exists() >> >> b. Interface to mark this node's CCR as winning copy. >> Removes SPLIT_BRAIN_CHANGE_FILE. >> Exists in userland. >> >> int ccrlib::mark_ccr_as_winner() >> >> c. Interface to mark this node's CCR as valid until cluster join; >> in other words, mark this node's CCR as loser copy. >> Removes SPLIT_BRAIN_CHANGE_FILE file. >> Set the gennum of every CCR table to -1. >> Exists in userland. >> >> int ccrlib::mark_ccr_as_loser() >> >> >> 7. Internal utility - ccradm - is modified to provide the following >> features until the CLI is available. >> a. ccradm -q : check whether split-brain CCR change exist >> b. ccradm -m <"winner"|"loser"> : >> mark a node as the "winner" or "loser" >> >> >> *********** >> >> >> Apologies for the short notice; >> we are targetting a putback to Colorado staging gate on Dec 17 >> if there are no major objections, >> and hence would request quick reviews. :) >> >> >> >> Thanks & Regards, >> Zoram, Sambit >> >> >> >> _______________________________________________ >> ha-clusters-discuss mailing list >> ha-clusters-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/ha-clusters-discuss >> >> > _______________________________________________ > ha-clusters-discuss mailing list > ha-clusters-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/ha-clusters-discuss >
