Hi Stefano, Thanks for review.
On Tue, Dec 12, 2017 at 12:10 AM, Stefano Stabellini <sstabell...@kernel.org > wrote: > On Wed, 9 Aug 2017, Mirela Simonovic wrote: > > This document contains our draft proposal for implementing "suspend to > RAM" > > support for ARM in Xen, as discussed during the last Xen ARM community > call. > > It covers the basic suspend to RAM mechanism based on ARM PSCI standard, > > that would allow individual guests and Xen itself to suspend. > > > > We would appreciate your feedback. > > > > Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com> > > I am really sorry, for the very late reply. This email fell through the > cracks. In general, if I forget about an email, please send a follow up > to help me remember :-) Do you have an more up-to-date proposal or is > this still the latest? > I've just sent new version, you're cc-ed. > > From an high level, it looks good, only one minor comment: don't call it > "hotplug" because usually in the Xen community hotplug means adding more > vcpus or removing vcpus at runtime (removing entirely, not suspending). > > PSCI-based suspend relies on hotplug mechanism for suspending/resuming non-boot CPUs. I believe we are aligned with the terminology, but if I missed something please let me know. I have updated the design spec, hope it's better now. Thanks, Mirela > But the approach looks good to me. > > > > --- > > docs/misc/arm/suspend-to-ram.txt | 210 ++++++++++++++++++++++++++++++ > +++++++++ > > 1 file changed, 210 insertions(+) > > create mode 100644 docs/misc/arm/suspend-to-ram.txt > > > > diff --git a/docs/misc/arm/suspend-to-ram.txt > b/docs/misc/arm/suspend-to-ram.txt > > new file mode 100644 > > index 0000000000..ec8080fc64 > > --- /dev/null > > +++ b/docs/misc/arm/suspend-to-ram.txt > > @@ -0,0 +1,210 @@ > > +% Suspend to RAM Support in Xen for ARM > > +% Revision 1.0 > > + > > +======== > > +Overview > > +======== > > + > > +Suspend to RAM (in the following text 'suspend') for ARM in Xen should > be > > +coordinated using ARM PSCI standard [1]. > > + > > +EL1/2 should suspend in the following order: > > +1) Unprivileged guests (DomUs) suspend > > +2) Privileged guest (Dom0) suspends > > +3) Xen suspends > > + > > +Since this proposal is focused on implementing PSCI-based suspend > mechanisms in > > +Xen, communication with or among the guests is not covered by this > document. > > +The order of suspending the guests is assumed to be guaranteed by the > software > > +running in EL1. > > + > > +----------------- > > +Suspending Guests > > +----------------- > > + > > +Suspend procedure for a guest consists of the following: > > +1) Suspending devices > > +2) Suspending non-boot CPUs > > +3) System suspend, performed by the boot CPU > > + > > +Each guest should suspend the devices it owns. Suspending of devices is > not > > +covered by this document. The document covers only mechanisms for > suspending > > +non-boot CPUs, as well as the system suspend. > > + > > +Guests should suspend their non-boot vCPUs using the hotplug mechanism. > > +Virtual CPUs should be put offline using the already implemented PSCI > vCPU_OFF > > +call (prefix 'v' is added to distinguish PSCI calls made by guests to > Xen, which > > +affect virtual machines; as opposed to PSCI calls made by Xen to the > EL3, which > > +can affect power state of the physical machine). > > + > > +After suspending its non-boot vCPUs a guest should finalize the suspend > by > > +making the vSYSTEM_SUSPEND PSCI call. The resume address is specified > by the > > +guest via the vSYSTEM_SUSPEND entry_point_address argument. The > vSYSTEM_SUSPEND > > +call is currently not implemented in Xen. > > + > > +It is expected that a guest leaves enabled all interrupts that should > wake it > > +up. Other interrupts should be disabled by the guest prior to calling > > +vSYSTEM_SUSPEND. > > + > > +After an unprivileged guest suspends, Xen will not suspend. Xen would > suspend > > +only after the Dom0 completes the system suspend. > > + > > +-------------- > > +Suspending Xen > > +-------------- > > + > > +Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND > call > > +from the last running guest (Dom0). At that moment all physical CPUs > are still > > +online (taking offline a vCPU or suspending a VM does not affect > physical CPUs). > > +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI > call > > +to EL3. The CPU_OFF PSCI function is currently not implemented in Xen. > > + > > +After putting offline the non-boot cores Xen must save the context and > finalize > > +suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to EL3. > > +The resume point of Xen is specified by the entry_point_address > argument of the > > +SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and context saving is > not > > +implemented in Xen for ARM today. > > + > > +------------ > > +Resuming Xen > > +------------ > > + > > +Xen must be resumed prior to any software running in EL1. Starting from > the > > +resume point, Xen should restore the context and resume Dom0. Dom0 > shall always > > +be resumed whenever Xen resumes. > > +The whole Xen resume flow for the ARM architecture has to be > implemented. > > + > > +--------------- > > +Resuming Guests > > +--------------- > > + > > +Resume of the privileged guest (Dom0) is always following the Xen > resume. > > + > > +An unprivileged guest shall resume once a device it owns triggers a > wake-up > > +interrupt, regardless of whether Xen was suspended when the wake-up > interrupt > > +was triggered. If Xen was suspended, it is assumed that Dom0 will be > running > > +before the DomU guest starts to resume. The synchronization mechanism to > > +enforce the assumed condition is TBD. > > + > > +If the ARM's GIC was powered down after the ARM subsystem suspended, it > is > > +assumed that Xen needs to restore the GIC interface for a VM prior to > handing > > +over control to the guest. However, the guest should restore its own > context > > +upon entering the resume point (out of scope of this document). > > + > > +======================= > > +Implementation Proposal > > +======================= > > + > > +-------- > > +Overview > > +-------- > > + > > +In order to enable the suspend/resume of VMs and Xen itself, the > following PSCI > > +calls have to be implemented and integrated in Xen: > > +1) vSYSTEM_SUSPEND > > +2) CPU_OFF > > +3) SYSTEM_SUSPEND > > + > > +In addition, the following have to be implemented: > > +* Save/restore of EL2 context > > +* Save/restore of GIC configuration for each VM > > + > > +Implementation details are provided in the sections below. Function > names and > > +paths used below are consistent within the document but may not always > match the > > +names used in future implementation. Existing functions and paths are > named as > > +in Xen source tree. > > + > > +Note: The proposal is still incomplete and shall be refined in future > revisions. > > +Specific issues that are not addressed are marked as "TBD". > > + > > +------------------------------------- > > +Suspend/Resume Implementation Details > > +------------------------------------- > > + > > +PSCI Implementation and Integration > > +----------------------------------- > > +vSYSTEM_SUSPEND > > +--------------- > > +vSYSTEM_SUSPEND shall be implemented in > > +* do_psci_system_suspend() in arch/arm/vpsci.c > > + > > +The implementation shall include the following steps: > > +* Block the current vCPU > > +* If the hardware domain made the call trigger Xen suspend, i.e. > > + call machine_suspend() which will be implemented in arch/arm/suspend.c > > + (similar as the machine_restart() is implemented in > arch/arm/shutdown.c) > > + > > +The function do_psci_system_suspend() shall be called from > > +* do_trap_psci() in arch/arm/traps.c > > + > > +CPU_OFF (physical CPUs) > > +----------------------- > > +The CPU_OFF function shall be implemented in > > +* call_psci_cpu_off() in arch/arm/psci.c > > + > > +The implementation shall consist just of making the SMC call to EL3. > > + > > +This function needs to be called when Xen generic code disables > non-boot CPUs, > > +which is done by > > +* disable_nonboot_cpus() in common/cpu.c > > +This function calls architecture specific > > +* __cpu_die() implemented in arch/arm/smpboot.c > > +The call_psci_cpu_off() shall be invoked when the respective CPU dies. > To make > > +that happen, the > > +* arch_cpu_die() would be implemented in arch/arm/arm64/smpboot.c > > +and called from __cpu_die(). > > +Finally the call_psci_cpu_off() shall be invoked from arch_cpu_die(). > > + > > +Such a control flow would be similar to the already existing flow for > enabling > > +non-boot CPUs, which looks like this: > > +enable_nonboot_cpus() -> cpu_up() -> __cpu_up() -> arch_cpu_up() -> > > +call_psci_cpu_on() > > + > > +SYSTEM_SUSPEND (physical) > > +------------------------- > > +The SYSTEM_SUSPEND function shall be implemented in > > +* call_psci_system_suspend() in arch/arm/psci.c > > + > > +The implementation shall consist just of making the SMC call to EL3. The > > +entry_point_address argument of the SMC call needs to be an ARM > architecture > > +resume address. The call_psci_system_suspend() function does not return. > > + > > +The function needs to be called from machine_suspend() to finalize the > suspend > > +procedure. > > + > > +------------------ > > +Additional Changes > > +------------------ > > + > > +Suspend Flow > > +------------ > > +The suspend procedure shall be implemented in > > +* machine_suspend() in arch/arm/suspend.c > > + > > +The implementation shall include the following steps: > > +* Set the system_state variable to SYS_STATE_suspend > > +* Freeze domains by calling domain_pause() for each domain (we assume > this needs > > + to be done) > > +* Disable non-boot CPUs by calling disable_nonboot_cpus() > > +* Save ARM specific context > > + > > +Resume Flow > > +------------ > > +The resume procedure shall be implemented in > > +* machine_resume() in arch/arm/suspend.c > > + > > +The machine_resume() implementation shall include the following steps: > > +* Restore ARM specific context > > +* Enable non-boot CPUs by calling enable_nonboot_cpus() > > +* Thaw domains by calling domain_unpause() for each domain (we assume > this needs > > + to be done) > > +* Set the system_state variable to SYS_STATE_resume > > +* TBD: how to resume Dom0, i.e. how to hand over control to Dom0? > > + > > +========== > > +References > > +========== > > + > > +[1] Power State Coordination Interface (ARM): > > +http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_ > Coordination_Interface_PDD_v1_1_DEN0022D.pdf > > + > > -- > > 2.13.0 > > > > > > _______________________________________________ > > Xen-devel mailing list > > xen-de...@lists.xen.org > > https://lists.xen.org/xen-devel > > >
_______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
