This is an automated email from the ASF dual-hosted git repository. yjhjstz pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 63e48e661e0544de41b69d4a29fd50cc63c5b638 Author: Nihal Jain <[email protected]> AuthorDate: Wed Sep 13 18:25:54 2023 +0530 gpexpand: Fix error when database has tablespaces **Issue:** Currently `gpexpand` errors out whenever it is run using a user-created input file (not created using the `gpexpand` interview process) on a cluster that has `custom tablespaces` created with the following error - ``` $ cat gpexpand_inputfile_20230914_201220 jnihal3MD6M.vmware.com|jnihal3MD6M.vmware.com|7005|/tmp/demoDataDir3|5|3|p $ gpexpand -i gpexpand_inputfile_20230914_201220 20230914:20:13:04:066896 gpexpand:jnihal3MD6M:jnihal-[ERROR]:-gpexpand failed: [Errno 2] No such file or directory: 'gpexpand_inputfile_20230914_201220.ts' ``` **RCA:** This is happening due to the commit 9b70ba8698f656c40ee62e5519314f7db1e4655e. This commit introduced a change, where it requires `gpexpand` to have a separate tablespace input configuration file (`<input_file>.ts`) whenever there are `custom tablespaces` in the database. However, this file only gets created whenever the user uses the `gpexpand` interview process to create the input file. In cases where the user manually creates the input file, the tablespace file is missing which causes the above error. **Fix:** Add a check in the `read_tablespace_file()` function to assert if the file is present or not. In cases where the file is not present, create the file automatically and exit from the process to give users a chance to review them (if they want to change the `tablespace` location) and prompt them to re-run `gpexpand`. The call to the `read_tablespace_file()` is also moved before we start the expansion process. This is because we want to exit from the process before we start the expansion so that the user does not have to `rollback` when they re-run `gpexpand`. ``` $ gpexpand -i gpexpand_inputfile_20230914_201220 20230914:20:24:00:014186 gpexpand:jnihal3MD6M:jnihal-[WARNING]:-Could not locate tablespace input configuration file 'gpexpand_inputfile_20230914_201220.ts'. A new tablespace input configuration file is written to 'gpexpand_inputfile_20230914_201220.ts'. Please review the file and re-run with: gpexpand -i gpexpand_inputfile_20230914_201220 20230914:20:24:00:014186 gpexpand:jnihal3MD6M:jnihal-[INFO]:-Exiting... $ gpexpand -i gpexpand_inputfile_20230914_201220 --> re-run with the same input file ``` --- gpMgmt/bin/gpexpand | 16 +++++++++++++- gpMgmt/test/behave/mgmt_utils/gpexpand.feature | 27 +++++++++++++++++++++++ gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py | 11 +++++++++ 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/gpMgmt/bin/gpexpand b/gpMgmt/bin/gpexpand index 4ab60e1e7c..a7481b65c2 100755 --- a/gpMgmt/bin/gpexpand +++ b/gpMgmt/bin/gpexpand @@ -1240,6 +1240,20 @@ class gpexpand: tablespace_inputfile = self.options.filename + ".ts" + """ + Check if the tablespace input file exists or not + In cases where the user manually creates an input file, the file + will not be present. In such cases create the file and exit giving the + user a chance to review it and re-run gpexpand. + """ + if not os.path.exists(tablespace_inputfile): + self.generate_tablespace_inputfile(tablespace_inputfile) + self.logger.warning("Could not locate tablespace input configuration file '{0}'. A new tablespace input configuration file is written " \ + "to '{0}'. Please review the file and re-run with: gpexpand -i {1}".format(tablespace_inputfile, self.options.filename)) + + logger.info("Exiting...") + sys.exit(1) + new_tblspc_info = {} with open(tablespace_inputfile) as f: @@ -2481,10 +2495,10 @@ def main(options, args, parser): _gp_expand.validate_heap_checksums() newSegList = _gp_expand.read_input_files() _gp_expand.addNewSegments(newSegList) + newTableSpaceInfo = _gp_expand.read_tablespace_file() _gp_expand.sync_packages() _gp_expand.start_prepare() _gp_expand.lock_catalog() - newTableSpaceInfo = _gp_expand.read_tablespace_file() _gp_expand.add_segments(newTableSpaceInfo) _gp_expand.update_original_segments() _gp_expand.cleanup_new_segments() diff --git a/gpMgmt/test/behave/mgmt_utils/gpexpand.feature b/gpMgmt/test/behave/mgmt_utils/gpexpand.feature index 5cb2810f84..bac03a799e 100644 --- a/gpMgmt/test/behave/mgmt_utils/gpexpand.feature +++ b/gpMgmt/test/behave/mgmt_utils/gpexpand.feature @@ -207,6 +207,33 @@ Feature: expand the cluster by adding more segments When the user runs gpexpand to redistribute Then the tablespace is valid after gpexpand + @gpexpand_no_mirrors + Scenario: expand a cluster with tablespace when there is no tablespace configuration file + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' + And the user runs command "rm -rf /data/gpdata/gpexpand/*" + And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into + And a cluster is created with no mirrors on "cdw" and "sdw1" + And database "gptest" exists + And a tablespace is created with data + And another tablespace is created with data + And there are no gpexpand_inputfiles + And the cluster is setup for an expansion on hosts "cdw" + And the user runs gpexpand interview to add 1 new segment and 0 new host "ignore.host" + And the number of segments have been saved + And there are no gpexpand tablespace input configuration files + When the user runs gpexpand with the latest gpexpand_inputfile without ret code check + Then gpexpand should return a return code of 1 + And gpexpand should print "[WARNING]:-Could not locate tablespace input configuration file" escaped to stdout + And gpexpand should print "A new tablespace input configuration file is written to" escaped to stdout + And gpexpand should print "Please review the file and re-run with: gpexpand -i" escaped to stdout + And verify if a gpexpand tablespace input configuration file is created + When the user runs gpexpand with the latest gpexpand_inputfile with additional parameters "--silent" + And verify that the cluster has 1 new segments + And all the segments are running + When the user runs gpexpand to redistribute + Then the tablespace is valid after gpexpand + @gpexpand_verify_redistribution Scenario: Verify data is correctly redistributed after expansion Given the database is not running diff --git a/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py b/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py index 8ba11b7933..ca7d35f63a 100644 --- a/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py +++ b/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py @@ -2943,6 +2943,17 @@ def impl(context, num_of_segments, num_of_hosts, hostnames): def impl(context): list(map(os.remove, glob.glob("gpexpand_inputfile*"))) +@given('there are no gpexpand tablespace input configuration files') +def impl(context): + list(map(os.remove, glob.glob("{}/*.ts".format(context.working_directory)))) + if len(glob.glob('{}/*.ts'.format(context.working_directory))) != 0: + raise Exception("expected no gpexpand tablespace input configuration files") + +@then('verify if a gpexpand tablespace input configuration file is created') +def impl(context): + if len(glob.glob('{}/*.ts'.format(context.working_directory))) != 1: + raise Exception("expected gpexpand tablespace input configuration file to be created") + @when('the user runs gpexpand with the latest gpexpand_inputfile with additional parameters {additional_params}') def impl(context, additional_params=''): gpexpand = Gpexpand(context, working_directory=context.working_directory) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
