Hi,
This RFC series explores integrating the vm-memory API into QEMU's
rust/memory bindings.
Thanks to Paolo and Manos's many suggestions and feedback, I have
resolved many issues over the past few months, but there are still
some open issues that I would like to discuss.
This series finally provides the following safe interfaces in Rust:
* AddressSpace::write in Rust <=> address_space_write in C
- **but only** supports MEMTXATTRS_UNSPECIFIED
* AddressSpace::read in Rust <=> address_space_read_full in C
- **but only** supports MEMTXATTRS_UNSPECIFIED.
* AddressSpace::store in Rust <=> address_space_st{size} in C
- **but only** supports MEMTXATTRS_UNSPECIFIED and native endian.
* AddressSpace::load in Rust <=> address_space_ld{size} in C
- **but only** supports MEMTXATTRS_UNSPECIFIED and native endian.
And this series involves changes mainly to these three parts:
* NEW QEMU memory APIs wrapper at C side.
* Extra changes for vm-memory (downstream for now).
* NEW QEMU memory bindings/APIs based on vm-memory at Rust side.
Although the number of line changes appears to be significant, more
than half of them are documentation and comments.
(Note: the latest vm-memory v0.16.2 crate didn't contain Paolo's
commit 5f59e29c3d30 ("guest_memory: let multiple regions slice one
global bitmap"), so I have to pull the vm-memory from github directly.)
Thanks for your feedback!
Background
==========
About vm-memory crate, it's design documentation said:
"The vm-memory crate focuses on defining consumer side interfaces to
access the physical memory of the VM. It does not define how the
underlying VM memory provider is implemented. Lightweight VMMs like
CrosVM and Firecracker can make assumptions about the structure of VM's
physical memory and implement a lightweight backend to access it. For
VMMs like Qemu, a high performance and full functionality backend may
be implemented with less assumptions."
At present, in addition to the memory model abstractions (including
GuestMemoryRegion, GuestMemory, and AddressSpace) it provides, it also
implements a simple memory management backend based on mmap for RAM
access.
However, for QEMU, the backend implementation based on vm-memory is more
complex, as QEMU not only needs to consider MMIO/IOMMU, but also complex
situations such as different endian/memory attributes.
This series tries to be simple as much as possible, and leaves different
endian/memory attributes support as the open issues.
But... wait, why vm-memory is necessary?
QEMU needs the safe Rust bindings for memory access. Whatever vm-memory
is used or not, there'll be the similar wrappers over AddressSpace/
FlatView/MemorySection, and there'll be the safe bindings for translation/
memory store, load, read and write.
Even if we don't use vm-memory, we will likely end up creating something
similar to vm-memory.
Furthermore, many components in vm-memory are also inspiring for
enhancements on the QEMU Rust side.
So, why not have a try?
Introduction
============
The core idea of this series is simple:
* Implement vm_memory::GuestMemoryRegion trait for MemoryRegionSection
to represent (non-overlapping) memory region.
vm_memory::GuestMemoryRegion trait itself doesn't provide any
interface to access memory. So vm_memory::Bytes trait is also
necessary for MemoryRegionSection to access memory region.
* Implement vm_memory::GuestMemory trait for FlatView to manage
Guest memory regions (that's MemoryRegionSection).
Similarly, vm_memory::Bytes trait is also needed for FlatView
to provide methods to write/read/store/load memory.
* Implement vm_memory::AddressSpace tarit for AddressSpace, to
provide a safe address space abstraction at Rust side.
For the above three parts, the most critical stuff is related to
MemoryRegionSection.
Currently QEMU's memory API is built around MemoryRegion, and
MemoryRegionSection is only for internal use at the most time.
But vm_memory::GuestMemoryRegion trait requires us to wrap unsafe C
bindings based on MemoryRegionSection. So it's necessary to expose
some C memory APIs based MemoryRegionSection. This is the following
section:
NEW QEMU memory APIs wrappers at C side
=======================================
Around MemoryRegionSection, this series provides these interfaces:
* Some straightforward wrappers over original C interfaces:
- section_access_allowed
- section_covers_region_addr,
- section_fuzz_dma_read
- section_get_host_addr
* Critical wrappers for intermediate memroy read/write (they're still
simple wrapper over C functions):
- section_rust_read_continue_step
- section_rust_write_continue_step
* And, special C helpers for memory load/store:
- section_rust_load
MemTxResult section_rust_load(MemoryRegionSection *section,
hwaddr mr_offset, uint8_t *buf,
MemTxAttrs attrs, hwaddr len);
- section_rust_store
MemTxResult section_rust_store(MemoryRegionSection *section,
hwaddr mr_offset, const uint8_t *buf,
MemTxAttrs attrs, hwaddr len);
These 2 load/store helpers are so different: comparing with the
detail implementations of address_space_ld{size}/
address_space_st{size}, these 2 functions aren't bound with specific
type (l, q, or w), and transfer the value via byte array.
This is because of the AtomicAccess bound in vm_memory::Bytes, which
makes it difficult to convert AtomicAccess type to u64! (For more
details, please refer the comments of Bytes::store/Bytes::load in
patch 22 "rust/memory: Implement vm_memory::GuestMemoryRegion for
MemoryRegionSection").
Of course, some other wrappers are also needed for FlatView and
AddressSpace:
* For FlatView:
- flatview_ref
- flatview_translate_section
- flatview_unref
* For AddressSpace:
- address_space_lookup_section
- address_space_memory
- address_space_to_flatview
They're all simple wrappers.
But, though QEMU's native C memory API could support complex conditions,
such as different endian formats or different memory attributes,
especially for memory write/read/store/load inferfaces. For now, Rust
side can only limit the support to only native endian and only
MEMTXATTRS_UNSPECIFIED.
This issue is related with why and how to adjust vm-memory's API for
QEMU:
Extra changes for vm-memory (downstream for now)
================================================
(All the patches for vm-memory can be found at patch 10 "subprojects/
vm-memory: Patch vm-memory for QEMU memory".)
As a minimum requirement, vm-memory still needs at least two changes:
* the 0001.diff file under subprojects/packagefiles/vm-memory-0.16-rs:
guest_memory: Add a marker tarit to implement Bytes<GuestAddress> for
GuestMemory
- This patch allows QEMU to customize its own Bytes trait
implementation for FlatView, which makes it possible to have the
memory write/read process in Rust similar to
flatview_write/flatview_read in C side.
So this patch is straightforward with low risk.
* the 0002.diff file under subprojects/packagefiles/vm-memory-0.16-rs:
guest_memory: Add is_write argument for GuestMemory::try_access()
- This patch is related with how to extend vm-memory to support more
complex cases.
Paolo suggested to implement Bytes<(GuestAddress, MemTxAttrs)> for
FlatView, but I found it's tricky since memory read/write will
finally depend on GuestMemory::try_access() to do iteration. Then
try_access() should konw more information.
1) One option is like the 0002.diff, just to add more arguments,
but this option is not very flexible. It is difficult to extend
for MemTxAttrs because vm-memory does not have MemTxAttrs at all.
(But perhaps vm-memory could support more memory attributes?)
2) Another option is to move try_access() under Bytes trait, then
try_access could accepts the tuple like (GuestAddress, MemTxAttrs)
or (GuestAddress, bool, MemTxAttrs) - boolean is is_write. But
my concern is I'm not sure whether it's proper to let Bytes have
try_access() method, especially there's no other need expect
GuestMemory.
Therefore, this is the issue that blocks us to provide more flexible
(or more complex) memory access interfaces at Rust side.
NEW QEMU memory bindings/APIs based on vm-memory at Rust side
=============================================================
At least, now we could have the safe bindings for the most basic (and
minimum) memory access in Rust side - no special memory attributs, and
only naive endian.
Speaking endian, this is another open issue. The current implementation
only supports native endian, i.e., endian that is consistent with the
target. Users can obtain the current endian and adjust it, for example,
```
use qemu_api::memory::{ADDRESS_SPACE_MEMORY, target_is_big_endian};
let addr = GuestAddress(0x123438000);
let val: u32 = 5;
let val_end = if target_is_big_endian() {
val.to_be()
} else {
val.to_le()
};
assert!(ADDRESS_SPACE_MEMORY.store(addr, val_end).is_ok());
```
It can work, But idealy, for Rust, type itself could tell enough
information.
But unfortunately, the Bytes::store/load accepts AtomicAccess, it doesn't
provide any information about endian (though vm-memory has other endian
type, e.g., Le32/be32).
For more endian support, I think there would be 2 options:
1) Implement Bytes<(GuestAddress, bool, MemTxAttrs, DeviceEndian)>, and
ask C side to handle endian issues.
2) Consider to add endian information in AtomicAccess.
Option 1 seems easier, but option 2 seems more reasonable? Because only
the Bytes::store/load care about endian, and no other methods requires
it. Considering endian for the entire Bytes seems a bit overkill.
Open issues
===========
Alright, the open issues are talked in the above sections. But let me
make a summary:
* About how to support more MemTxAttrs, please see the section "Extra
changes for vm-memory (downstream for now)".
* About how to support more endian formats, please see the section
"NEW QEMU memory bindings/APIs based on vm-memory at Rust side".
Thanks and Best Regards,
Zhao
---
Zhao Liu (26):
rust/hpet: Fix the error caused by vm-memory
rust/cargo: Add the support for vm-memory
subprojects: Add thiserror-impl crate
subprojects: Add thiserror crate
subprojects: Add winapi-i686-pc-windows-gnu crate
subprojects: Add winapi-x86_64-pc-windows-gnu crate
subprojects: Add winapi crate
subprojects: Add vm-memory crate
rust: Add vm-memory in meson
subprojects/vm-memory: Patch vm-memory for QEMU memory backend
rust/cargo: Specify the patched vm-memory crate
rcu: Make rcu_read_lock & rcu_read_unlock not inline
rust: Add RCU bindings
memory: Expose interfaces about Flatview reference count to Rust side
memory: Rename address_space_lookup_region and expose it to Rust side
memory: Make flatview_do_translate() return a pointer to
MemoryRegionSection
memory: Add a translation helper to return MemoryRegionSection
memory: Rename flatview_access_allowed() to
memory_region_access_allowed()
memory: Add MemoryRegionSection based misc helpers
memory: Add wrappers of intermediate steps for read/write
memory: Add store/load interfaces for Rust side
rust/memory: Implement vm_memory::GuestMemoryRegion for
MemoryRegionSection
rust/memory: Implement vm_memory::GuestMemory for FlatView
rust/memory: Provide AddressSpace bindings
rust/memory: Add binding to check target endian
rust/hpet: Use safe binding to access address space
include/qemu/rcu.h | 45 +-
include/system/memory.h | 313 +++++-
rust/Cargo.lock | 51 +
rust/Cargo.toml | 3 +
rust/hw/timer/hpet/src/device.rs | 29 +-
rust/meson.build | 2 +
rust/qemu-api/Cargo.toml | 11 +
rust/qemu-api/meson.build | 5 +-
rust/qemu-api/src/lib.rs | 1 +
rust/qemu-api/src/memory.rs | 971 +++++++++++++++++-
rust/qemu-api/src/rcu.rs | 26 +
rust/qemu-api/wrapper.h | 2 +
scripts/archive-source.sh | 4 +-
scripts/make-release | 4 +-
subprojects/.gitignore | 6 +
.../packagefiles/thiserror-1-rs/meson.build | 23 +
.../thiserror-impl-1-rs/meson.build | 41 +
.../packagefiles/vm-memory-0.16-rs/0001.diff | 81 ++
.../packagefiles/vm-memory-0.16-rs/0002.diff | 111 ++
.../vm-memory-0.16-rs/meson.build | 35 +
.../packagefiles/winapi-0.3-rs/meson.build | 46 +
.../meson.build | 20 +
.../meson.build | 20 +
subprojects/thiserror-1-rs.wrap | 10 +
subprojects/thiserror-impl-1-rs.wrap | 10 +
subprojects/vm-memory-0.16-rs.wrap | 14 +
subprojects/winapi-0.3-rs.wrap | 10 +
.../winapi-i686-pc-windows-gnu-0.4-rs.wrap | 10 +
.../winapi-x86_64-pc-windows-gnu-0.4-rs.wrap | 10 +
system/memory-internal.h | 1 -
system/memory.c | 7 +-
system/physmem.c | 200 +++-
util/rcu.c | 43 +
33 files changed, 2040 insertions(+), 125 deletions(-)
create mode 100644 rust/qemu-api/src/rcu.rs
create mode 100644 subprojects/packagefiles/thiserror-1-rs/meson.build
create mode 100644 subprojects/packagefiles/thiserror-impl-1-rs/meson.build
create mode 100644 subprojects/packagefiles/vm-memory-0.16-rs/0001.diff
create mode 100644 subprojects/packagefiles/vm-memory-0.16-rs/0002.diff
create mode 100644 subprojects/packagefiles/vm-memory-0.16-rs/meson.build
create mode 100644 subprojects/packagefiles/winapi-0.3-rs/meson.build
create mode 100644
subprojects/packagefiles/winapi-i686-pc-windows-gnu-0.4-rs/meson.build
create mode 100644
subprojects/packagefiles/winapi-x86_64-pc-windows-gnu-0.4-rs/meson.build
create mode 100644 subprojects/thiserror-1-rs.wrap
create mode 100644 subprojects/thiserror-impl-1-rs.wrap
create mode 100644 subprojects/vm-memory-0.16-rs.wrap
create mode 100644 subprojects/winapi-0.3-rs.wrap
create mode 100644 subprojects/winapi-i686-pc-windows-gnu-0.4-rs.wrap
create mode 100644 subprojects/winapi-x86_64-pc-windows-gnu-0.4-rs.wrap
--
2.34.1
