Terry Bowman wrote: > Hotplug removing a CXL device and hotplug reinserting the same device > currently requires manual interaction for managing the device > region. The CXL device region must be manually destroyed before > hotplug removal and manually created after hotplug reinsertion. > > Create a script to automatically destroy and recreate the region > during CXL device hotplug remove-reinsert. Save region characteristics to > a file before destroying the region and hotplug remove. Use the region > characteristics stored in the file to recreate the region after > hotplug reinsert.
It strikes me that functionality like this might be formalized under a command like "cxl export-region [--destroy]" where import support might be automatic if the exported configuration is saved somewhere udev can find it, otherwise "cxl import-region" could take hot added device back to its defined region. > > Signed-off-by: Terry Bowman <[email protected]> > --- > README.txt | 311 +++++++++++++++++++++++++++++++++++++++++ > cxl-hotplug.py | 366 +++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 677 insertions(+) > create mode 100644 README.txt > create mode 100755 cxl-hotplug.py > > diff --git a/README.txt b/README.txt > new file mode 100644 > index 0000000..97fa793 > --- /dev/null > +++ b/README.txt > @@ -0,0 +1,311 @@ > + ____________________ > + > + CXL-HOTPLUG.README > + ____________________ mmm, documentation. > + > + > +Table of Contents > +_________________ > + > +1. Purpose > +2. Requirements > +.. 1. Kernel - v6.4.0 > +.. 2. ndctl - v77 or later > +.. 3. QEMU - v8.0.3 + the following patches: > +.. 4. Python modules: > +.. 5. Additional tool details > +3. Usage > +4. Examples > +.. 1. Swap a device > +.. 2. Manually unplug and plugin a device > +.. 3. Manually unplug and plugin a device (w/ step by step details) > + > + > + > + > + > +1 Purpose > +========= > + > + Hotplug adding and removing CXL devices requires region management not > + automatically provided. For instance, if a CXL device is added then a > + region must be 'created' before the memory can be used. Likewise, a > + region must be 'destroyed' before a CXL device can be > + removed. Removing a CXL device before a region is 'destroyed' can > + result in CXL device data loss or corruption. > + > + This tool aims to provide region delete and create automation for an > + existing CXL device. The typical usage is to hotplug remove an > + existing CXL device and then hotplug readd the same device immediately > + or at some later time. > + > + An unplug function is provided that will 'destroy' the region making > + the device ready for hotplug removal. Note, 'destroying' a PMEM region > + incurrs no loss of data. 'destroying' a RAM region will lose the > + region data. This tool saves the region structure information so that > + the device's region can be created in the future. After the device is > + hot hotplug added in the future the region information is used to > + recreate the region. The region information is saved to a default file > + or can be directed to a specific file provided on the tool comandline. > + > + This tool provides a swap function that automatically executes the > + unplug and plugin functions. > + > + This tool is currently limited to non interleaved devices. > + > + > +2 Requirements > +============== > + > + Tested using the following: > + > + > +2.1 Kernel - v6.4.0 > +~~~~~~~~~~~~~~~~~~~ > + > + To include the following kernel config settings: > + ,---- > + | scripts/config --enable LIBNVDIMM > + | scripts/config --enable CONFIG_CXL_BUS > + | scripts/config --enable CONFIG_CXL_PCI > + | scripts/config --enable CONFIG_CXL_ACPI > + | scripts/config --enable CONFIG_CXL_MEM > + | scripts/config --enable CONFIG_CXL_PMEM > + | scripts/config --enable CONFIG_CXL_PORT > + | scripts/config --enable CONFIG_CXL_SUSPEND > + | scripts/config --enable CONFIG_CXL_REGION > + | scripts/config --enable CONFIG_CXL_REGION_INVALIDATION_TEST > + | scripts/config --enable CONFIG_PCIEAER_CXL > + | scripts/config --enable CONFIG_TRANSPARENT_HUGEPAGE > + | scripts/config --enable CONFIG_DEV_DAX > + | scripts/config --enable CONFIG_DEV_DAX_HMEM > + | scripts/config --enable CONFIG_DEV_DAX_KMEM > + | scripts/config --enable CONFIG_DEV_DAX_PMEM > + `---- > + The above configures for statically linked drivers. They could be > + dynamically linked as modules as well. > + > + > +2.2 ndctl - v77 or later > +~~~~~~~~~~~~~~~~~~~~~~~~ > + > + > +2.3 QEMU - v8.0.3 + the following patches: > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + [email protected] > + [email protected] > + > <https://lore.kernel.org/linux-cxl/[email protected]/\#r> > + > + > +2.4 Python modules: > +~~~~~~~~~~~~~~~~~~~ > + > + The following python modules are required: json subprocess os argparse > + logging > + > + > +2.5 Additional tool details > +~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + <https://confluence.amd.com/display/ALK/CXL+QEMU> Internal-only web page? > + > + > +3 Usage > +======= > + > + The utility requires one of the sub-commands: plugin,unplug,list,swap. > + > + Commandline usage information: > + ,---- > + | usage: cxl-hotplug.py [-h] {plugin,unplug,list,swap} ... > + | > + | positional arguments: > + | {plugin,unplug,list,swap} > + | > + | options: > + | -h, --help show this help message and exit > + `---- > + > + Commandline plugin usage information: > + ,---- > + | usage: cxl-hotplug.py plugin [-h] -m MEMDEV [-c CONFIG_FILE] [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to prepare for unplug > + | -c CONFIG_FILE CXL JSON configuration file to use > + | -d, --debug Enable debugging > + `---- > + > + Commandline unplug usage information: > + ,---- > + | usage: cxl-hotplug.py unplug [-h] -m MEMDEV [-c CONFIG_FILE] [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to prepare for unplug > + | -c CONFIG_FILE CXL JSON configuration file to save > + | -d, --debug Enable debugging > + `---- > + > + Commandline swap usage information: > + ,---- > + | usage: cxl-hotplug.py swap [-h] -m MEMDEV [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to swap > + | -d, --debug Enable debugging > + `---- > + > + > +4 Examples > +========== > + > +4.1 Swap a device > +~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | # ./cxl-hotplug.py swap -m mem0 > + | Device 'mem0' can now be safely removed. > + | Card is ready for removal. > + | Press any key to continue after card reinsertion. > + | Region created for 'mem0'. > + `---- > + > + > +4.2 Manually unplug and plugin a device > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | # ./cxl-hotplug.py unplug -m mem0 -c my-cxl.json > + | # ./cxl-hotplug.py plugin -m mem0 -c my-cxl.json > + `---- > + > + > +4.3 Manually unplug and plugin a device (w/ step by step details) > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl list > + | [ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 NA NA NA > + | > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | { > + | "region":"region0", > + | "resource":"0x890000000", > + | "size":"256.00 MiB (268.44 MB)", > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit", > + | "mappings":[ > + | { > + | "position":0, > + | "memdev":"mem0", > + | "decoder":"decoder2.0" > + | } > + | ] > + | } > + | cxl region: cmd_create_region: created 1 region > + | > + | # cxl list > + | [ > + | { > + | "memdevs":[ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | }, > + | { > + | "regions":[ > + | { > + | "region":"region0", > + | "resource":36775657472, > + | "size":268435456, > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit" > + | } > + | ] > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 region0 268435456 True > + | > + | # ./cxl-hotplug.py unplug -m mem0 > + | Device 'mem0' can now be safely removed. > + | > + | # cxl list > + | [ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 NA NA NA > + | > + | # ./cxl-hotplug.py plugin -m mem0 > + | Region created for 'mem0'. > + | > + | # cxl list > + | [ > + | { > + | "memdevs":[ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | }, > + | { > + | "regions":[ > + | { > + | "region":"region0", > + | "resource":36775657472, > + | "size":268435456, > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit" > + | } > + | ] > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 region0 268435456 True > + `---- Overall I like wrapping all the details of the focused sub-commands into something that is more goal oriented. I have no heartburn with carrying this in cxl/contrib/ while figuring out if this should also be hooked as a formal sub-command set. > diff --git a/cxl-hotplug.py b/cxl-hotplug.py > new file mode 100755 > index 0000000..971e4e7 > --- /dev/null > +++ b/cxl-hotplug.py > @@ -0,0 +1,366 @@ > +#!/usr/bin/python3 > +# SPDX-License-Identifier: LGPL-2.1 > +# > +# Copyright (C) 2023, Advanced Micro Devices (AMD). All rights reserved. > +# > +# @Author - Terry Bowman > +# > +# Utility to support CXL hotplug removal and re-insertion. The purpose > +# is to recreate a CXL region during insertion that existed before > +# removal. > +# > +# '--unplug' sub-command option will save the CXL region information to > +# a file before a manual hotplug removal. > +# > +# '--plugin' sub-command after hotplug insertion will recreate a device > +# region using the details from the configuration file. > +# > +# '--swap' sub-command will run the same functionality as '--unplug', > +# then prompt for when the card is plugged in, and then execute the > +# '--plugin' functionality. > +# > +# '--list' sub-command will list the CXL devices and associated regions. > +# > + > +import json > +import subprocess > +import os > +import argparse > +import logging > +import tempfile > + > +class cxl_json: > + > + cxl_memdev_json = {} > + decoders_root_json = {} > + regions_decoder_json = {} > + daxregion_json = {} > + daxregion_devices_json = {} > + > + def cxl_list_memdev(self, memdev): > + result = subprocess.run(["cxl", "list", "-m", memdev, "-RBMTXEPD"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl list command failed for: " + memdev) > + exit(1) > + > + result_json = json.loads(result.stdout) > + > + if not result_json: > + print("CXL list is empty for: " + memdev) > + exit(1) > + > + return(result_json[0]) > + > + # Cache embedded json dictionaries to make more easily accessible. > + # This helps in later processing . > + # > + # If the CXL json dictionary is missing any CXL components than this > + # implies a CXL device is not configured. In this case exit with an > + # error and message. > + def decode_json(self, fatal_error = True): > + for key in self.cxl_memdev_json: > + if key.startswith('decoders:root'): > + self.decoders_root_json = self.cxl_memdev_json[key][0] > + if self.decoders_root_json == None or \ > + len(self.decoders_root_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find decoder root CXL json.") > + exit(1) > + else: > + return > + > + regions_decoder_key = "regions:" + > self.decoders_root_json.get('decoder') > + if self.decoders_root_json.get(regions_decoder_key) == None: > + if fatal_error == True: > + print("Error: Failed to find region decoder in CXL json.") > + exit(1) > + else: > + return > + > + self.regions_decoder_json = > self.decoders_root_json.get(regions_decoder_key)[0] > + if len(self.regions_decoder_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find region decoder in CXL json.") > + exit(1) > + else: > + return > + > + # PMEM CXL JSON does not include DAX keys searched for below, return > early > + if 'pmem' == self.regions_decoder_json.get('type'): > + return; > + > + for key in self.regions_decoder_json: > + if key.startswith('daxregion'): > + self.daxregion_json = self.regions_decoder_json[key] > + if len(self.daxregion_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find daxregion in CXL json.") > + exit(1) > + else: > + return > + > + for key in self.daxregion_json: > + if key.startswith('devices'): > + self.daxregion_devices_json = self.daxregion_json[key][0] > + if len(self.daxregion_devices_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find daxregion devices in CXL json.") > + exit(1) > + else: > + return > + > + def get_block_size_bytes(self): > + result = subprocess.run(["cat", > "/sys/devices/system/memory/block_size_bytes"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + return int(result.stdout, 16) > + > + def is_region_movable(self): > + > + # PMEM doesn't have movable concept > + if 'pmem' == self.regions_decoder_json.get('type'): > + return True; > + > + block_size_bytes = self.get_block_size_bytes() > + resource = self.regions_decoder_json.get('resource') > + resource_size = self.regions_decoder_json.get('size') > + start_block = int(resource/block_size_bytes) > + stop_block = int((resource+resource_size)/block_size_bytes - 1) > + is_movable = True > + > + for i in range(0,(stop_block-start_block + 1)): > + block = start_block + i > + block_str = ("/sys/devices/system/memory/memory" + str(block) + > + "/valid_zones") > + result = subprocess.run(["cat", block_str], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: Block cat command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + if "Movable" not in result.stdout: > + is_movable = False > + break; > + > + return is_movable > + > +class cxl_unplug(cxl_json): > + > + def __init__(self, memdev, config_file): > + # Capture the current CXL (and DAX) configurations. Save > + # configurations to file for using later with '--plugin' > + # hot-plug adding the memory device. > + self.cxl_memdev_json = self.cxl_list_memdev(memdev) > + self.serialize_json(self.cxl_memdev_json, config_file) > + self.decode_json() > + > + if (self.regions_decoder_json.get('interleave_ways')>1): > + print("Interleaved devices are not supported.") > + exit(1) > + > + def serialize_json(self, json_str, filename): > + with open( filename , "w" ) as write: > + json.dump(json_str, write) > + > + def is_dax_memory_offline(self, dax_dev_json): > + result = subprocess.run(["daxctl", "list"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: daxctl offline command failed command. rc = " + > + str(result.returncode)) > + exit(1) > + > + dax_dev_online_mb = json.loads(result.stdout)[0]["online_memblocks"] > + logging.debug("online_memblocks = " + > + str(json.loads(result.stdout)[0]["online_memblocks"])) > + return dax_dev_online_mb == 0 > + > + def offline_dax_memory(self): > + # Note: self.daxregion_devices_json['movable'] JSON key is missing > + # when True (ndctl v77). As a result, use the function > region_movable() > + if self.is_region_movable() == False: > + print("Error: Entire region memory is not zone movable.") > + exit(1) > + > + dax_dev = self.daxregion_devices_json['chardev'] > + result=subprocess.run(["daxctl", "offline-memory", dax_dev], > + capture_output=True, text=True) > + if result.returncode != 0: > + if self.is_dax_memory_offline(dax_dev) == False: > + print("Error: daxctl offline command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def offline_memory(self): > + if 'ram' == self.regions_decoder_json.get('type'): > + self.offline_dax_memory() > + > + def cxl_destroy_region(self): > + result = subprocess.run(["cxl", "destroy-region", > + > str(self.regions_decoder_json.get('region')), > + "--force"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def unplug(self): > + unplug_dev.offline_memory() > + unplug_dev.cxl_destroy_region() > + print("Device \'" + args.memdev + "\' can now be safely removed.") > + > +class cxl_plugin(cxl_json): > + > + def __init__(self, config_file): > + f = open(config_file) > + self.cxl_memdev_json = json.load(f) > + self.decode_json() > + > + if (self.regions_decoder_json.get('interleave_ways')>1): > + print("Interleaved devices are not supported.") > + exit(1) > + > + def cxl_create_region(self, cxl_memdev_json, memdev): > + type = self.regions_decoder_json.get('type'); > + decoder = self.decoders_root_json.get('decoder') > + interleave_ways = self.regions_decoder_json.get('interleave_ways') > + interleave_granularity = > self.regions_decoder_json.get('interleave_granularity') > + size = self.regions_decoder_json.get('size') > + > + result = subprocess.run(["cxl", "create-region", > + "-m", memdev, > + "-t", type, > + "-d", decoder, > + "-w", str(interleave_ways), > + "-s", str(size), > + "--debug"], > + capture_output=True, text=True) > + if result.returncode!=0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def plugin(self, memdev): > + self.cxl_create_region(self.cxl_memdev_json, memdev) > + print("Region created for \'" + memdev + "\'.") > + > +class cxl_list(cxl_json): > + > + cxl_memdevs_json = {} > + > + def __init__(self): > + self.cxl_memdevs_json = self.cxl_list_memdevs() > + > + def display_memdevs(self): > + if (self.cxl_memdevs_json == None): > + print("Error: Failed to find memory devices") > + return > + > + if (len(self.cxl_memdevs_json) == 0): > + print("Error: Failed to find memory devices") > + return > + > + print("Device Region Name Region Size Movable") > + print("====== =========== =========== =======") > + > + for memdev in self.cxl_memdevs_json: > + self.cxl_memdev_json = self.cxl_list_memdev(memdev['memdev']) > + self.decode_json(fatal_error = False) > + if len(self.regions_decoder_json) != 0: > + region = self.regions_decoder_json.get('region'); > + size = self.regions_decoder_json.get('size') > + is_movable_str = "True" > + if self.is_region_movable() == 0: > + is_movable_str = "False" > + else: > + region = 'NA' > + size = 'NA' > + is_movable_str = 'NA' > + > + print("%6s %18s %18s %14s" % > + (memdev['memdev'], region, size, is_movable_str)) > + > + def cxl_list_memdevs(self): > + result = subprocess.run(["cxl", "list", "-M"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl list command failed.") > + exit(1) > + > + logging.debug("cxl_memdevs_json = " + result.stdout) > + self.cxl_memdevs_json = json.loads(result.stdout); > + > + return(self.cxl_memdevs_json) > + > +def init_args(): > + parser = argparse.ArgumentParser() > + subparsers = parser.add_subparsers(dest='subparser_name', required=True) > + > + parser_plugin = subparsers.add_parser('plugin') > + parser_plugin.add_argument('-m', dest='memdev', > + help='CXL memory device to prepare for > unplug', > + required=True) > + parser_plugin.add_argument('-c', dest='config_file', > + help='CXL JSON configuration file to > save/use', > + required=False, default="cxl.json") > + parser_plugin.add_argument('-d', '--debug', help='Enable debug logging', > + required=False, action='store_true') > + > + parser_unplug = subparsers.add_parser('unplug') > + parser_unplug.add_argument('-m', dest='memdev', > + help='CXL memory device to prepare for > unplug', > + required=True) > + parser_unplug.add_argument('-c', dest='config_file', > + help='CXL JSON configuration file to > save/use', > + required=False, default="cxl.json") > + parser_unplug.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + parser_list = subparsers.add_parser('list') > + parser_list.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + parser_swap = subparsers.add_parser('swap') > + parser_swap.add_argument('-m', dest='memdev', > + help='CXL memory device to swap', > + required=True) > + parser_swap.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + return parser.parse_args() > + > +args = init_args() > +if args.debug: > + logging.basicConfig(level=logging.DEBUG) > + > +logging.debug("args = " + str(args)) > + > +if args.subparser_name == 'list': > + list_dev = cxl_list(); > + list_dev.display_memdevs() > + > +if args.subparser_name == 'unplug': > + unplug_dev = cxl_unplug(args.memdev, args.config_file); > + unplug_dev.unplug() > + > +if args.subparser_name == 'plugin': > + plugin_dev = cxl_plugin(args.config_file); > + plugin_dev.plugin(args.memdev) > + > +if args.subparser_name == 'swap': > + f, filename = tempfile.mkstemp() > + unplug_dev = cxl_unplug(args.memdev, filename); > + unplug_dev.unplug() > + > + print("Card is ready for removal.") > + input("Press any key to continue after card reinsertion.") > + > + plugin_dev = cxl_plugin(filename); > + plugin_dev.plugin(args.memdev) > + os.close(f) > -- > 2.34.1 >
