[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352051#comment-17352051 ] ASF subversion and git services commented on KUDU-2181: --- Commit 04262dc8d20ac441890971d8487eda586fdbe43e in kudu's branch refs/heads/branch-1.15.x from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=04262dc ] [master] KUDU-2181 Fix duplicate master address and remove unsafe flag tag While testing add master on a physical cluster, observed that supplying only hostnames without port resulted in duplicates being supplied to bring up new master which in turn leads to failure in creating distributed Raft config on startup. For e.g. kudu master add hp1,hp2 h2 Reason being hp2 is compared as hp2:7051 whereas in the vector of strings "master_addresses", it contains hp1,hp2. This changes adds a new function ParseAddresses() in HostPort class that's a variant of existing ParseStrings() function and takes a vector of strings instead. This new function is used in the duplicate detection logic. This change also removes the unsafe tag from --master_addr_add_new_master as the feature is ready but keeps it hidden as it's only meant to be used by the add master orchestration tool. This one is tricky to test locally and write a test because we need to specify ports for starting masters/tservers. Verified the fix on physical cluster. Along with CM integration we can add a systest later. Change-Id: Icf29730e3a6b225adb24ff161cac2ad777b46b81 Reviewed-on: http://gerrit.cloudera.org:8080/17500 Tested-by: Bankim Bhavsar Reviewed-by: Alexey Serbin (cherry picked from commit 261d71ef71fcfdf96c26c9d604d6fc0147c2ee6a) Reviewed-on: http://gerrit.cloudera.org:8080/17516 Tested-by: Kudu Jenkins > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17352036#comment-17352036 ] ASF subversion and git services commented on KUDU-2181: --- Commit eda678de47856910c3b2aaef6e0e9e11473eb5ff in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=eda678d ] [master] KUDU-2181 Fix duplicate master address and remove unsafe flag tag While testing add master on a physical cluster, observed that supplying only hostnames without port resulted in duplicates being supplied to bring up new master which in turn leads to failure in creating distributed Raft config on startup. For e.g. kudu master add hp1,hp2 h2 Reason being hp2 is compared as hp2:7051 whereas in the vector of strings "master_addresses", it contains hp1,hp2. This changes adds a new function ParseAddresses() in HostPort class that's a variant of existing ParseStrings() function and takes a vector of strings instead. This new function is used in the duplicate detection logic. This change also removes the unsafe tag from --master_addr_add_new_master as the feature is ready but keeps it hidden as it's only meant to be used by the add master orchestration tool. This one is tricky to test locally and write a test because we need to specify ports for starting masters/tservers. Verified the fix on physical cluster. Along with CM integration we can add a systest later. Change-Id: Icf29730e3a6b225adb24ff161cac2ad777b46b81 Reviewed-on: http://gerrit.cloudera.org:8080/17500 Tested-by: Bankim Bhavsar Reviewed-by: Alexey Serbin > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344963#comment-17344963 ] ASF subversion and git services commented on KUDU-2181: --- Commit 863b3374000795256b0b73d248f60010dc51b78f in kudu's branch refs/heads/branch-1.15.x from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=863b337 ] [master] KUDU-2181 Unhide master change config flag & other improvements This change: - Unhides and turns the master Raft change config flag ON by default - Updates the description of flags that are not passed to the new master - Uses sync logging on bringing up the new master to help debug any issues in startup of the new master - Updates the wait timeout while bringing up a new master to account for the higher ntp wait timeout. This timeout is also used for checking whether the new master's system catalog has caught up from WAL. So this'll bump up timeout for that case as well. Thought about introducing separate wait timeout for that case but felt like exposing too many knobs and user shouldn't care whether time is being taken for master bringup or catchup from WAL. Change-Id: Id1fc1dc5985601158ae58d5d190a60c1e542ea1d Reviewed-on: http://gerrit.cloudera.org:8080/17401 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar (cherry picked from commit c5648cf189e0413420a553ad5f51aa8252a1fd6f) Reviewed-on: http://gerrit.cloudera.org:8080/17450 Reviewed-by: Andrew Wong > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344893#comment-17344893 ] ASF subversion and git services commented on KUDU-2181: --- Commit 207e4e693beb37c1f44c7621b244198bdbe895c1 in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=207e4e6 ] [master] KUDU-2181 Unhide master change config flag & other improvements This change: - Unhides and turns the master Raft change config flag ON by default - Updates the description of flags that are not passed to the new master - Uses sync logging on bringing up the new master to help debug any issues in startup of the new master - Updates the wait timeout while bringing up a new master to account for the higher ntp wait timeout. This timeout is also used for checking whether the new master's system catalog has caught up from WAL. So this'll bump up timeout for that case as well. Thought about introducing separate wait timeout for that case but felt like exposing too many knobs and user shouldn't care whether time is being taken for master bringup or catchup from WAL. Change-Id: Id1fc1dc5985601158ae58d5d190a60c1e542ea1d Reviewed-on: http://gerrit.cloudera.org:8080/17401 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344870#comment-17344870 ] ASF subversion and git services commented on KUDU-2181: --- Commit 336c65bfd84a5115dc5c6be6521751d53aa0c286 in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=336c65b ] [doc] KUDU-2181 Update multi-master addition/removal/recovery documentation This change updates the documentation for: 1) "Migrating to Multiple Masters" that uses the new `kudu master add` CLI command merged recently. 2) "Removing Kudu Masters from a Multi-Master Deployment" that uses the `kudu master remove` CLI tool. 3) "Recovering from a dead Kudu Master in a Multi-Master Deployment" that uses a combination of master remove and add CLI tools. This change doesn't include any version specific steps as this doc is meant for the latest Kudu version 1.15.0. The idea is to introduce an index page for documentation that points to version specific docs. This change also removes Cloudera Manager(CM) specific instructions as they could change with automation in CM. The rendered version of the doc can be viewed here: https://github.com/bbhavsar/kudu/blob/bankim/r5/docs/administration.adoc#migrate_to_multi_master Change-Id: I6a1d5bc6bbf4bc3e82e7046469d2682bf016d3a8 Reviewed-on: http://gerrit.cloudera.org:8080/17352 Reviewed-by: Andrew Wong Tested-by: Kudu Jenkins > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326047#comment-17326047 ] ASF subversion and git services commented on KUDU-2181: --- Commit d5d8fadadac750bd9a71c4bc928dc7de05168a9a in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=d5d8fad ] [master][tool] KUDU-2181 Tool to orchestrate adding a master This tool orchestrates the mechanism to add a master to an existing Kudu cluster using the changes made previously including the add master Raft change config. This tool must be run from the master being added. Outline of steps: - Bring up master from supplied flags - Invoke add master Raft change config RPC - Determine whether new master has caught up from WAL and ksck is healthy. - If yes, we are done else continue below - Delete system catalog on the new master - Copy system catalog from one of the existing masters - Verify ksck is healthy. - Shutdown the new master The tool now manages bringing up the new master process, copying system catalog and hence bunch of functionality from the dynamic_multi_master test is no longer needed and hence deleted. Using Subprocess we need to shutdown the new master and hence can't verify certain state which included contacting new master directly immediately after adding new master. Function VerifyClusterAfterMasterAddition() covers comprehensive verification steps so not missing out by removing those steps. This change also includes some refactoring in ExternalDaemon to fetch Kerberos related configuration to be used outside ExternalMiniCluster. Change-Id: I8f99cf2b3f1738c4127c7e7288beab61daf42e7b Reviewed-on: http://gerrit.cloudera.org:8080/17267 Reviewed-by: Andrew Wong Tested-by: Kudu Jenkins > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287342#comment-17287342 ] ASF subversion and git services commented on KUDU-2181: --- Commit 645f9dc72f62bc4cca8f59f9e043c37586e87f63 in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=645f9dc ] [tool] KUDU-2181 CLI to remove master This change adds CLI to remove master from the cluster which in turn invokes the RemoveMaster ChangeConfig RPC. This CLI will be part of the workflow to remove master from an existing cluster or recover dead master at the same HostPort. Change-Id: I5c97b03475b0ffc7b387d7dfc17acc4b13858fb7 Reviewed-on: http://gerrit.cloudera.org:8080/17010 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar Reviewed-by: Andrew Wong > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287341#comment-17287341 ] ASF subversion and git services commented on KUDU-2181: --- Commit df61b71f0093008f6f2cdc88b4380a575f42bf18 in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=df61b71 ] [tool] KUDU-2181 CLI to add master This change adds a CLI that invokes the AddMaster RPC to perform Raft ChangeConfig. This CLI will be part of the workflow to migrate to multiple masters in a Kudu cluster. Change-Id: I507f301d1aba17327eb35728eed0d765e86ef4cc Reviewed-on: http://gerrit.cloudera.org:8080/16530 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar Reviewed-by: Andrew Wong > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281420#comment-17281420 ] ASF subversion and git services commented on KUDU-2181: --- Commit 3c363d499ec5c0d675d78d871712e579d966548c in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=3c363d4 ] [master] KUDU-2181 Raft ChangeConfig request to remove a master This change adds RPC to request removal of an existing master from a cluster. It builds on top of earlier change 9fc6c77 to add master. Added tests for removal of non-leader master whether it's up or down. Change-Id: I76c03b8850faef60b65f85184c0a4db7cc6759ee Reviewed-on: http://gerrit.cloudera.org:8080/16936 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260502#comment-17260502 ] ASF subversion and git services commented on KUDU-2181: --- Commit 47075a180ca46366a56af2ff97f65b199b4a1f4e in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=47075a1 ] [master] KUDU-2181 Procedure for copying sys catalog on adding master This change outlines procedure to copy system catalog for the newly added master using existing CLI tools and the master ChangeConfig RPC. Only functional change is to hookup the master runtime flag --master_consensus_allow_status_msg_for_failed_peer to Raft consensus. New master could go into a FAILED_RECOVERABLE state if the leader master's system catalog WAL has been GC'ed. This change allows the new master to be promoted after copying the system catalog externally. Outline of the test procedure: 1) Runtime flag --master_consensus_allow_status_msg_for_failed_peer must be turned on for existing masters. 2) Start the new master with --master_address_add_new_master= and --master_addresses that contains itself and existing masters. 3) Invoke ChangeConfig to add the master. 4) Verify the new master is part of the Raft config even if it's a LEARNER/NON_VOTER or goes into FAILED_RECOVERABLE state. If not, above steps will have to be repeated. 5) If the new master is promoted to being a VOTER then following tablet copy steps can be skipped. 6) Shutdown the new master. 7) Delete the system catalog on the new master. 8) Copy the system catalog from the leader master to the new master. 9) Bring up the new master. 10) Verify the new master is promoted as VOTER. If the new master doesn't get promoted to a VOTER then double check whether the new master is part of the Raft config for masters by running "kudu master list". - If yes, repeat procedure from step 6. - Else repeat the entire procedure Change-Id: I142c1dec442ec72c38c5be9d62cdf270e441d6e3 Reviewed-on: http://gerrit.cloudera.org:8080/16830 Tested-by: Kudu Jenkins Reviewed-by: Bankim Bhavsar > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213233#comment-17213233 ] Bankim Bhavsar commented on KUDU-2181: -- Current state and remaining tasks: https://docs.google.com/document/d/1itZjdKrvQyiyK4DtmnkoeENb0R9gPrkNwHluistx5Ek/edit > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201707#comment-17201707 ] ASF subversion and git services commented on KUDU-2181: --- Commit 9fc6c772aff2bbfa008f52da8e7ba6e2c2d9012f in kudu's branch refs/heads/master from Bankim Bhavsar [ https://gitbox.apache.org/repos/asf?p=kudu.git;h=9fc6c77 ] [master] KUDU-2181 Raft ChangeConfig request to add a master This change: - Adds hidden feature flag "--master_support_change_config" off by default. - RPC changes to add a master that initiates Raft config change and responds asynchronously. - RPC changes to report back member type(VOTER/NON_VOTER) of masters - Removes the cached master_addresses in catalog manager as it's no longer static. - Updates and adds comments in MasterOptions such that it's used to fetch master addresses only during master init time as masters can be added/removed dynamically with this change. - Updates ListMasters() to look at local Raft config instead of MasterOptions as the masters can be added/removed dynamically. If the new master can be caught up from the WAL then the master gets promoted to VOTER else it remains as NON_VOTER without master tablet copying support. Change-Id: I0ac7e6e55220bcb01cad0fa386daaf656258088c Reviewed-on: http://gerrit.cloudera.org:8080/16321 Reviewed-by: Alexey Serbin Tested-by: Bankim Bhavsar > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193932#comment-17193932 ] Bankim Bhavsar commented on KUDU-2181: -- Change config implementation: https://gerrit.cloudera.org/c/16321/ > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2181) Multi-master config change support
[ https://issues.apache.org/jira/browse/KUDU-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162333#comment-17162333 ] Bankim Bhavsar commented on KUDU-2181: -- Design: https://docs.google.com/document/d/1PKA4juROjl5qRJAbi4X-eTUTvVoff6M49idvV4aEZLA/edit > Multi-master config change support > -- > > Key: KUDU-2181 > URL: https://issues.apache.org/jira/browse/KUDU-2181 > Project: Kudu > Issue Type: Improvement > Components: consensus, master >Reporter: Mike Percy >Assignee: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > It would be very useful to add support to the Kudu master for dynamic config > change. The current procedure for replacing a failed master is fairly arduous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
