Changes in v7:
* Split free_heap_pages() buddy merge changes into a separate patch (patch 1)
* Changed type for page_info.u.free.need_tlbflush to bool:1
* Added BUILD_BUG_ON
* Adjusted datatype of temp variable in check_and_stop_scrub()
* Formatting changes
(see per-patch changes)
When a domain is destroyed the hypervisor must scrub domain's pages before
giving them to another guest in order to prevent leaking the deceased
guest's data. Currently this is done during guest's destruction, possibly
causing very lengthy cleanup process.
This series adds support for scrubbing released pages from idle loop,
making guest destruction significantly faster. For example, destroying a
1TB guest can now be completed in 40+ seconds as opposed to about 9 minutes
using existing scrubbing algorithm.
Briefly, the new algorithm places dirty pages at the end of heap's page list
for each node/zone/order to avoid having to scan full list while searching
for dirty pages. One processor form each node checks whether the node has any
dirty pages and, if such pages are found, scrubs them. Scrubbing itself
happens without holding heap lock so other users may access heap in the
meantime. If while idle loop is scrubbing a particular chunk of pages this
chunk is requested by the heap allocator, scrubbing is immediately stopped.
On the allocation side, alloc_heap_pages() first tries to satisfy allocation
request using only clean pages. If this is not possible, the search is
repeated and dirty pages are scrubbed by the allocator.
This series is somewhat based on earlier work by Bob Liu.
* Only set PGC_need_scrub bit for the buddy head, thus making it unnecessary
to scan whole buddy
* Fix spin_lock_cb()
* Scrub CPU-less nodes
* ARM support. Note that I have not been able to test this, only built the
* Added scrub test patch (last one). Not sure whether it should be considered
for committing but I have been running with it.
* merge_chunks() returns new buddy head
* scrub_free_pages() returns softirq pending status in addition to (factored
status of unscrubbed memory
* spin_lock uses inlined spin_lock_cb()
* scrub debugging code checks whole page, not just the first word.
* Keep dirty bit per page
* Simplify merge_chunks() (now merge_and_free_buddy())
* When scrubbing memmory-only nodes try to find the closest node.
* Keep track of dirty pages in a buddy with page_info.u.free.first_dirty.
* Drop patch 1 (factoring out merge_and_free_buddy()) since there is only
one caller now
* Drop patch patch 5 (from V3) since we are not breaking partially-scrubbed
* Extract search loop in alloc_heap_pages() into get_free_buddy() (patch 2)
* Add MEMF_no_scrub flag
* Make page_info.u.free and union and use bitfields there.
* Bug fixes
* Changed first_dirty tracking from pointer-based to index-based (patch 1)
* Added/modified a few ASSERT()s
* Moved/modifed a couple of comments
* Adjusted width of INVALID_DIRTY_IDX
* Per-node heap locks. In addition to (presumably) improving performance in
general, once they are available we can parallelize scrubbing further by
allowing more than one core per node to do idle loop scrubbing.
* AVX-based scrubbing
* Use idle loop scrubbing during boot.
Boris Ostrovsky (9):
mm: Clean up free_heap_pages()
mm: Place unscrubbed pages at the end of pagelist
mm: Extract allocation loop from alloc_heap_pages()
mm: Scrub pages in alloc_heap_pages() if needed
mm: Scrub memory from idle loop
spinlock: Introduce spin_lock_cb()
mm: Keep heap accessible to others while scrubbing
mm: Print number of unscrubbed pages in 'H' debug handler
mm: Make sure pages are scrubbed
xen/Kconfig.debug | 7 +
xen/arch/arm/domain.c | 8 +-
xen/arch/x86/domain.c | 8 +-
xen/arch/x86/domain_page.c | 6 +-
xen/common/page_alloc.c | 607 ++++++++++++++++++++++++++++++++++++++-------
xen/common/spinlock.c | 9 +-
xen/include/asm-arm/mm.h | 30 ++-
xen/include/asm-x86/mm.h | 31 ++-
xen/include/xen/mm.h | 5 +-
xen/include/xen/spinlock.h | 8 +
10 files changed, 612 insertions(+), 107 deletions(-)
Xen-devel mailing list