Reviewed: https://review.openstack.org/530662 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=65736fd7d49284efd6c28be21aceb0ba1ea0ec4a Submitter: Zuul Branch: master
commit 65736fd7d49284efd6c28be21aceb0ba1ea0ec4a Author: Andreas Karis <aka...@redhat.com> Date: Mon Jan 1 17:31:56 2018 -0500 Add debug output for selected page size Adds debug output for selected page size when hw:mem_page_size is specified. This output is especially useful in the case of hw:mem_page_size=any. Change-Id: Ie43228dbfa5623f880e9be032d8ebd9d0be42870 Closes-Bug: #1740795 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1740795 Title: nova lacks debug output for selected page size when hw:mem_page_size is specified Status in OpenStack Compute (nova): Fix Released Bug description: nova lacks debug output for selected page size when hw:mem_page_size is specified Administrators currently are left completely in the dark as to which page size is selected by nova and why. This output is especially useful in the case of hw:mem_page_size=any. /usr/lib/python2.7/site-packages/nova/virt/hardware.py ~~~ 37 MEMPAGES_SMALL = -1 38 MEMPAGES_LARGE = -2 39 MEMPAGES_ANY = -3 (...) 933 def _get_flavor_image_meta(key, flavor, image_meta): 934 """Extract both flavor- and image-based variants of metadata.""" 935 flavor_key = ':'.join(['hw', key]) 936 image_key = '_'.join(['hw', key]) 937 938 flavor_policy = flavor.get('extra_specs', {}).get(flavor_key) 939 image_policy = image_meta.properties.get(image_key) 940 941 return flavor_policy, image_policy (...) 944 def _numa_get_pagesize_constraints(flavor, image_meta): 945 """Return the requested memory page size 946 947 :param flavor: a Flavor object to read extra specs from 948 :param image_meta: nova.objects.ImageMeta object instance 949 950 :raises: MemoryPageSizeInvalid if flavor extra spec or image 951 metadata provides an invalid hugepage value 952 :raises: MemoryPageSizeForbidden if flavor extra spec request 953 conflicts with image metadata request 954 :returns: a page size requested or MEMPAGES_* 955 """ 956 957 def check_and_return_pages_size(request): 958 if request == "any": 959 return MEMPAGES_ANY 960 elif request == "large": 961 return MEMPAGES_LARGE 962 elif request == "small": 963 return MEMPAGES_SMALL 964 else: 965 try: 966 request = int(request) 967 except ValueError: 968 try: 969 request = strutils.string_to_bytes( 970 request, return_int=True) / units.Ki 971 except ValueError: 972 request = 0 973 974 if request <= 0: 975 raise exception.MemoryPageSizeInvalid(pagesize=request) 976 977 return request 978 979 flavor_request, image_request = _get_flavor_image_meta( 980 'mem_page_size', flavor, image_meta) 981 982 if not flavor_request and image_request: 983 raise exception.MemoryPageSizeForbidden( 984 pagesize=image_request, 985 against="<empty>") 986 987 if not flavor_request: 988 # Nothing was specified for hugepages, 989 # let's the default process running. 990 return None 991 992 pagesize = check_and_return_pages_size(flavor_request) 993 if image_request and (pagesize in (MEMPAGES_ANY, MEMPAGES_LARGE)): 994 return check_and_return_pages_size(image_request) 995 elif image_request: 996 raise exception.MemoryPageSizeForbidden( 997 pagesize=image_request, 998 against=flavor_request) 999 1000 return pagesize ~~~ If the flavor is set to any, and the image properties are not set, then this will return: MEMPAGES_ANY In the same file, there is the following code: ~~~ 620 def _numa_cell_supports_pagesize_request(host_cell, inst_cell): 621 """Determine whether the cell can accept the request. 622 623 :param host_cell: host cell to fit the instance cell onto 624 :param inst_cell: instance cell we want to fit 625 626 :raises: exception.MemoryPageSizeNotSupported if custom page 627 size not supported in host cell 628 :returns: the page size able to be handled by host_cell 629 """ 630 avail_pagesize = [page.size_kb for page in host_cell.mempages] 631 avail_pagesize.sort(reverse=True) 632 633 def verify_pagesizes(host_cell, inst_cell, avail_pagesize): 634 inst_cell_mem = inst_cell.memory * units.Ki 635 for pagesize in avail_pagesize: 636 if host_cell.can_fit_hugepages(pagesize, inst_cell_mem): 637 return pagesize 638 639 if inst_cell.pagesize == MEMPAGES_SMALL: 640 return verify_pagesizes(host_cell, inst_cell, avail_pagesize[-1:]) 641 elif inst_cell.pagesize == MEMPAGES_LARGE: 642 return verify_pagesizes(host_cell, inst_cell, avail_pagesize[:-1]) 643 elif inst_cell.pagesize == MEMPAGES_ANY: 644 return verify_pagesizes(host_cell, inst_cell, avail_pagesize) 645 else: 646 return verify_pagesizes(host_cell, inst_cell, [inst_cell.pagesize]) ~~~ Let's quickly verify the implications of the pagesize.sort and the avail_pagesize[-1:] and [:-1]: ~~~ >>> arr = [0,1,2,3,4,5] >>> arr.sort(reverse=True) >>> arr [5, 4, 3, 2, 1, 0] >>> arr[-1:] [0] >>> arr[:-1] [5, 4, 3, 2, 1] ~~~ So MEMPAGES_SMALL will offer only the smallest page size. MEMPAGES_LARGE will offer any but the smallest page size. MEMPAGES_ANY will offer all page sizes. This in then fed into: ~~~ 633 def verify_pagesizes(host_cell, inst_cell, avail_pagesize): 634 inst_cell_mem = inst_cell.memory * units.Ki 635 for pagesize in avail_pagesize: 636 if host_cell.can_fit_hugepages(pagesize, inst_cell_mem): 637 return pagesize ~~~ The page sizes are ordered from largest to smallest. verify_pagesizes walks the pagesizes in descending order. If `host_cell.can_fit_hugepages(pagesize, inst_cell_mem)`, then we return the pagesize. In your case, this means that we walk over all large pages, and ignore them due to the fact that can_fit_hugepages returns `false` when comparing pagesize against inst_cell_mem. From can_fit_hugepages ~~~ 146 def can_fit_hugepages(self, pagesize, memory): 147 """Returns whether memory can fit into hugepages size 148 149 :param pagesize: a page size in KibB 150 :param memory: a memory size asked to fit in KiB 151 152 :returns: whether memory can fit in hugepages 153 :raises: MemoryPageSizeNotSupported if page size not supported 154 """ 155 for pages in self.mempages: 156 if pages.size_kb == pagesize: 157 return (memory <= pages.free_kb and 158 (memory % pages.size_kb) == 0) 159 raise exception.MemoryPageSizeNotSupported(pagesize=pagesize) ~~~ So: return `true` if requested instance memory <= free_kb for the page category and the requested memory can be divided by page size_kb without a remainder (modulo operation returns 0). Finally, here is the piece of code that calls all of the above: ~~~ 919 pagesize = None 920 if instance_cell.pagesize: 921 pagesize = _numa_cell_supports_pagesize_request( 922 host_cell, instance_cell) 923 if not pagesize: 924 LOG.debug('Host does not support requested memory pagesize. ' 925 'Requested: %d kB', instance_cell.pagesize) 926 return 927 928 instance_cell.id = host_cell.id 929 instance_cell.pagesize = pagesize 930 return instance_cell ~~~ Observe how instance_cell.pagesize is replaced from whatever is specified to a numeric value! Unfortunately, the code lacks a `LOG.debug` which would make troubleshooting easier. What would help us gain some insight is this little modification: /usr/lib/python2.7/site-packages/nova/virt/hardware.py ~~~ 919 pagesize = None 920 if instance_cell.pagesize: 921 pagesize = _numa_cell_supports_pagesize_request( 922 host_cell, instance_cell) 923 if not pagesize: 924 LOG.debug('Host does not support requested memory pagesize. ' 925 'Requested: %d kB', instance_cell.pagesize) 926 return 927 + 928 LOG.debug('Selected memory pagesize: %(selected_mem_pagesize)d kB. ' + 929 'Requested memory pagesize: %(requested_mem_pagesize)d ' + 930 '(small = -1, large = -2, any = -3)', + 931 {'selected_mem_pagesize' : pagesize, + 932 'requested_mem_pagesize': instance_cell.pagesize}) 933 934 instance_cell.id = host_cell.id 935 instance_cell.pagesize = pagesize 936 return instance_cell ~~~ The additional log output would then yield: ~~~ [root@overcloud-compute-0 ~]# tail -f /var/log/nova/nova-compute.log | grep Selec 2018-01-01 22:02:34.018 433399 DEBUG nova.virt.hardware [req-fc00ae35-7ec0-427f-b1e5-3d8f904292f7 ae9fb97fb2c24155be8850edebb1814e 75f18c3dbade48bfb6f6040091cd71ba - - -] Selected memory pagesize: 2048 kB. Requested memory pagesize: -2 (small = -1, large = -2, any = -3) _numa_fit_instance_cell /usr/lib/python2.7/site-packages/nova/virt/hardware.py:944 2018-01-01 22:03:12.160 433399 DEBUG nova.virt.hardware [req-1e0be24d-e444-495c-9817-79642615e9f7 ae9fb97fb2c24155be8850edebb1814e 75f18c3dbade48bfb6f6040091cd71ba - - -] Selected memory pagesize: 2048 kB. Requested memory pagesize: 2048 (small = -1, large = -2, any = -3) _numa_fit_instance_cell /usr/lib/python2.7/site-packages/nova/virt/hardware.py:944 2018-01-01 22:03:37.910 433399 DEBUG nova.virt.hardware [req-4c63a1c0-76c7-4b53-87e0-a94fa6daa344 ae9fb97fb2c24155be8850edebb1814e 75f18c3dbade48bfb6f6040091cd71ba - - -] Selected memory pagesize: 2048 kB. Requested memory pagesize: -3 (small = -1, large = -2, any = -3) _numa_fit_instance_cell /usr/lib/python2.7/site-packages/nova/virt/hardware.py:944 ~~~ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1740795/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp