Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Stefano Stabellini]

On Tue, 13 Nov 2018, Julien Grall wrote:
> Hi Stefano,
> 
> On 13/11/2018 02:22, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> > > Hi Mirela,
> > > 
> > > Thank you for posting the series. Could you provide a branch with the
> > > patch
> > > applied?
> > > 
> > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > 
> > > > 
> > > > 
> > > > The series does not include:
> > > > a) UART driver-specific suspend/resume that gets called when console
> > > > suspends
> > > 
> > > I feel it will be difficult to test this series without UART driver
> > > support.
> > > Would it be possible to integrate them in that series?
> > 
> > Actually, you can test this series simply using upstream Linux and
> > upstream Xen + this series. You can use echo mem > /sys/power/state in
> > dom0 to go into suspend, and for example setup an RTC wakeup event to
> > wakeup after a certain amount of time. For instance echo +30 >
> > /sys/class/rtc/rtc0/wakealarm.
> 
> I am quite surprised this series is enough given there are a BUG() in the
> suspend path for the console.
> 
> So what is your xen command line?

No, I made a mistake in rebasing Mirela's branch: I tested a branch
which included the two unsent patches that removed exactly the BUG() you
are talking about. Without those two patches, Xen is unable to resume
correctly on the MPSoC due to the UART.

Mirela, it is better to include the two patches in your next version of
the series. If you think they are not up to quality for being merged
upstream, i.e. they are just hacks for testing, that's OK, tag them with
[PATCH DO-NOT-APPLY] to make it clear. It is also useful to post a git
branch in your 0 email when the patch series is large.

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

Hi Stefano,

On 13/11/2018 02:22, Stefano Stabellini wrote:

On Mon, 12 Nov 2018, Julien Grall wrote:

Hi Mirela,

Thank you for posting the series. Could you provide a branch with the patch
applied?

On 11/12/18 11:30 AM, Mirela Simonovic wrote:



The series does not include:
a) UART driver-specific suspend/resume that gets called when console
suspends


I feel it will be difficult to test this series without UART driver support.
Would it be possible to integrate them in that series?


Actually, you can test this series simply using upstream Linux and
upstream Xen + this series. You can use echo mem > /sys/power/state in
dom0 to go into suspend, and for example setup an RTC wakeup event to
wakeup after a certain amount of time. For instance echo +30 >
/sys/class/rtc/rtc0/wakealarm.


I am quite surprised this series is enough given there are a BUG() in the 
suspend path for the console.


So what is your xen command line?

Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Stefano Stabellini]

On Mon, 12 Nov 2018, Julien Grall wrote:
> Hi Mirela,
> 
> Thank you for posting the series. Could you provide a branch with the patch
> applied?
> 
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > 
> > 
> > The series does not include:
> > a) UART driver-specific suspend/resume that gets called when console
> > suspends
> 
> I feel it will be difficult to test this series without UART driver support.
> Would it be possible to integrate them in that series?

Actually, you can test this series simply using upstream Linux and
upstream Xen + this series. You can use echo mem > /sys/power/state in
dom0 to go into suspend, and for example setup an RTC wakeup event to
wakeup after a certain amount of time. For instance echo +30 >
/sys/class/rtc/rtc0/wakealarm.

For the Xilinx MPSoC, Linux 4.19 works fine with defconfig. Linux 4.20
(current master) gained EEMI support, but I tested it and it seems to
suspend successfully even if Xen doesn't know how to handle the EEMI
calls that Linux makes when suspending. 

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 1:08 PM Julien Grall  wrote:
>
> On 11/12/18 12:01 PM, Mirela Simonovic wrote:
> > Hi Julien,
>
> Hi Mirela,
>
> Please configure your e-mail client to avoid quoting using "space".
> Otherwise, this is going to make difficult to follow the discussions.
>

Sure, sorry.

> > On Mon, Nov 12, 2018 at 12:50 PM Julien Grall  > > wrote:
> >
> > Hi Mirela,
> >
> > Thank you for posting the series. Could you provide a branch with the
> > patch applied?
> >
> >
> > I have applied patches on top of upstream/staging.
>
> patches are applied every day to staging. So can you please provide the
> exact commit?
>

Oh yes, the staging already moved. It was this one: 388c55b
tools/misc: fix hard tabs in xen-hvmctx.c

> Also, it is common for big series to provide a branch with the patch
> series applied. This makes easier for people to try your series.
>
> >
> > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >  >
> > 
> > 
> >  > The series does not include:
> >  > a) UART driver-specific suspend/resume that gets called when
> > console suspends
> >
> > I feel it will be difficult to test this series without UART driver
> > support. Would it be possible to integrate them in that series?
> >
> >
> > Yes that's possible, but you'll need more than a UART driver's
> > suspend/resume to test this. E.g. for testing the series on Xilinx
> > Ultrascale+ MPSoC we need a set of patches that provides the basic
> > platform support needed to boot, which is not yet upstreamed. Luckily,
> > the suspend support is independent (ARM specific).
>
> I am a bit confused. I thought it was possible to boot Xen out-of-box on
> the Zynq MP. What is missing?
>
> > Please let me check with Xilinx people and Stefano how to deal with
> > this, and I'll come back with a real answer.
>
> If that's too difficult to provide in short term. Then I would
> appreciate if testing is done on another platform (e.g Foundation
> Model). So I can at least verify the series before we merge it.
>

No, it's not difficult at all, I just need Xilinx support on this.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

On 11/12/18 12:01 PM, Mirela Simonovic wrote:

Hi Julien,


Hi Mirela,

Please configure your e-mail client to avoid quoting using "space". 
Otherwise, this is going to make difficult to follow the discussions.


On Mon, Nov 12, 2018 at 12:50 PM Julien Grall > wrote:


Hi Mirela,

Thank you for posting the series. Could you provide a branch with the
patch applied?


I have applied patches on top of upstream/staging.


patches are applied every day to staging. So can you please provide the 
exact commit?


Also, it is common for big series to provide a branch with the patch 
series applied. This makes easier for people to try your series.




On 11/12/18 11:30 AM, Mirela Simonovic wrote:
 >


 > The series does not include:
 > a) UART driver-specific suspend/resume that gets called when
console suspends

I feel it will be difficult to test this series without UART driver
support. Would it be possible to integrate them in that series?


Yes that's possible, but you'll need more than a UART driver's 
suspend/resume to test this. E.g. for testing the series on Xilinx 
Ultrascale+ MPSoC we need a set of patches that provides the basic 
platform support needed to boot, which is not yet upstreamed. Luckily, 
the suspend support is independent (ARM specific).


I am a bit confused. I thought it was possible to boot Xen out-of-box on 
the Zynq MP. What is missing?


Please let me check with Xilinx people and Stefano how to deal with 
this, and I'll come back with a real answer.


If that's too difficult to provide in short term. Then I would 
appreciate if testing is done on another platform (e.g Foundation 
Model). So I can at least verify the series before we merge it.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 12:50 PM Julien Grall  wrote:

> Hi Mirela,
>
> Thank you for posting the series. Could you provide a branch with the
> patch applied?
>
>
I have applied patches on top of upstream/staging.


> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >
> 
> > The series does not include:
> > a) UART driver-specific suspend/resume that gets called when console
> suspends
>
> I feel it will be difficult to test this series without UART driver
> support. Would it be possible to integrate them in that series?
>
>
Yes that's possible, but you'll need more than a UART driver's
suspend/resume to test this. E.g. for testing the series on Xilinx
Ultrascale+ MPSoC we need a set of patches that provides the basic platform
support needed to boot, which is not yet upstreamed. Luckily, the suspend
support is independent (ARM specific).
Please let me check with Xilinx people and Stefano how to deal with this,
and I'll come back with a real answer.

Thanks,
Mirela


> Cheers,
>
> --
> Julien Grall
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Julien Grall]

Hi Mirela,

Thank you for posting the series. Could you provide a branch with the 
patch applied?


On 11/12/18 11:30 AM, Mirela Simonovic wrote:


The series does not include:
a) UART driver-specific suspend/resume that gets called when console suspends


I feel it will be difficult to test this series without UART driver 
support. Would it be possible to integrate them in that series?


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi,

One thing I screwed - I forgot to remove changes log from an internal
review, so please ignore it. This is officially the first version.

Thanks,
Mirela

On Mon, Nov 12, 2018 at 12:31 PM Mirela Simonovic <
mirela.simono...@aggios.com> wrote:

> This series contains support for suspend to RAM (in the following text just
> 'suspend') for Xen on arm64. The implementation is aligned with the design
> specification that has been proposed on xen-devel list:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
> At a high-level the series contains:
> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
> 2) Support for resuming a guest on an interrupt targeted to that guest
> 3) Support for suspending Xen after dom0 finalizes the suspend
> 4) Support for resuming Xen on an interrupt that is targeted to a guest
>
>
> 
> In more details:
>
> *** About suspending/resuming guests
>
> The patches included in this series allow PSCI compliant guests that have
> support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
> suspend and resume on top of Xen without any EL1 code changes.
>
> During their suspend procedure guests will hot-unplug their secondary CPUs,
> triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
> suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
> Guests will save/restore their own EL1 context on suspend/resume.
>
> A guest is expected to leave enabled interrupts that are considered to be
> its
> wake-up sources. Those interrupts will be able to wake up the guest. This
> holds
> regardless of the state of the underlying software layers, i.e. whether
> Xen gets
> suspended or not doesn't affect the ability of the guest to wake up.
>
> First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from
> which
> the guest assumes to start on resume. On resume, guests assume to be
> running in
> an environment whose state matches the CPU state after reset, e.g. with
> reset
> register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
> VCPU context and save the resume entry point into program counter before
> the
> guest's VCPU gets scheduled in on resume. This is done when the guest
> finalizes
> its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure
> that
> the resume-ready VCPU's context does not get overwritten later upon the
> context
> switch when the VCPU is scheduled out.
> Xen also needs to take care that the guest's view of GIC and timer gets
> saved.
> Also, while a guest is suspended its watchdogs are paused, in order to
> avoid
> watchdog triggered shutdown of a guest that has been asleep for a period
> of time
> that is longer than the watchdog period.
>
> After this point, from Xen's point of view a suspended guest has one VCPU
> blocked, waiting for an interrupt. When such an interrupt comes, Xen will
> unblock the VCPU of the suspended domain, which results in the guest
> resuming.
>
> *** About suspending/resuming Xen
>
> Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
> considered to be the decision maker for EL1 and EL2.
> On suspend, Xen will first freeze all domains. Then, Xen disables physical
> secondary CPUs, which leads to physical CPU_OFF to be called by each
> secondary
> CPU. After that Xen finalizes the suspend from the boot CPU.
>
> This consists of suspending the timer, i.e. suppressing its interrupts (we
> don't
> want to be woken up by a timer, there is no VCPU ready to be scheduled).
> Then
> the state of GIC is saved, console is suspended, and CPU context is saved.
> The
> saved context tells where Xen needs to continue execution on resume.
> Since Xen will resume with MMU disabled, the first thing to do in resume
> is to
> resume memory management in order to be able to access the context that
> needs to
> be restored (we know virtual address of the context data). Finally Xen
> calls
> SYSTEM_SUSPEND PSCI to the EL3.
>
> When resuming, all the steps done in suspend need to be reverted. This is
> completed by unblocking dom0's VCPU, because we always want the dom0 to
> resume,
> regardless of the target domain whose interrupt woke up Xen.
>
> *** Handling of unprivileged guests during Xen suspend/resume
>
> Any domU that is not suspended when dom0 suspends will be frozen, domUs
> that are
> already suspended remain suspended. On resume the suspended domUs still
> remain
> suspended (unless their wake interrupt caused Xen to wake) while the
> others will
> be thawed.
>
> For more details please refer to patches or the design specification:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
>
> 
> The series does not include:
> a) UART driver-specific suspend/resume that gets called when console
> suspends
> b) 

[Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

This series contains support for suspend to RAM (in the following text just
'suspend') for Xen on arm64. The implementation is aligned with the design
specification that has been proposed on xen-devel list:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

At a high-level the series contains:
1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
2) Support for resuming a guest on an interrupt targeted to that guest
3) Support for suspending Xen after dom0 finalizes the suspend
4) Support for resuming Xen on an interrupt that is targeted to a guest


In more details:

*** About suspending/resuming guests

The patches included in this series allow PSCI compliant guests that have
support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
suspend and resume on top of Xen without any EL1 code changes.

During their suspend procedure guests will hot-unplug their secondary CPUs,
triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
Guests will save/restore their own EL1 context on suspend/resume.

A guest is expected to leave enabled interrupts that are considered to be its
wake-up sources. Those interrupts will be able to wake up the guest. This holds
regardless of the state of the underlying software layers, i.e. whether Xen gets
suspended or not doesn't affect the ability of the guest to wake up.

First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from which
the guest assumes to start on resume. On resume, guests assume to be running in
an environment whose state matches the CPU state after reset, e.g. with reset
register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
VCPU context and save the resume entry point into program counter before the
guest's VCPU gets scheduled in on resume. This is done when the guest finalizes
its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure that
the resume-ready VCPU's context does not get overwritten later upon the context
switch when the VCPU is scheduled out.
Xen also needs to take care that the guest's view of GIC and timer gets saved.
Also, while a guest is suspended its watchdogs are paused, in order to avoid
watchdog triggered shutdown of a guest that has been asleep for a period of time
that is longer than the watchdog period.

After this point, from Xen's point of view a suspended guest has one VCPU
blocked, waiting for an interrupt. When such an interrupt comes, Xen will
unblock the VCPU of the suspended domain, which results in the guest resuming.

*** About suspending/resuming Xen

Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
considered to be the decision maker for EL1 and EL2.
On suspend, Xen will first freeze all domains. Then, Xen disables physical
secondary CPUs, which leads to physical CPU_OFF to be called by each secondary
CPU. After that Xen finalizes the suspend from the boot CPU.

This consists of suspending the timer, i.e. suppressing its interrupts (we don't
want to be woken up by a timer, there is no VCPU ready to be scheduled). Then
the state of GIC is saved, console is suspended, and CPU context is saved. The
saved context tells where Xen needs to continue execution on resume.
Since Xen will resume with MMU disabled, the first thing to do in resume is to
resume memory management in order to be able to access the context that needs to
be restored (we know virtual address of the context data). Finally Xen calls
SYSTEM_SUSPEND PSCI to the EL3.

When resuming, all the steps done in suspend need to be reverted. This is
completed by unblocking dom0's VCPU, because we always want the dom0 to resume,
regardless of the target domain whose interrupt woke up Xen.

*** Handling of unprivileged guests during Xen suspend/resume

Any domU that is not suspended when dom0 suspends will be frozen, domUs that are
already suspended remain suspended. On resume the suspended domUs still remain
suspended (unless their wake interrupt caused Xen to wake) while the others will
be thawed.

For more details please refer to patches or the design specification:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html


The series does not include:
a) UART driver-specific suspend/resume that gets called when console suspends
b) SMMU suspend/resume
c) Suspend coordination support that would allow dom0 to request domUs to
suspend
These will be submitted in the following series.

Mirela Simonovic (16):
  xen/arm: Implement PSCI system suspend call (virtual interface)
  xen/arm: Save GIC and virtual timer context when the domain suspends
  xen/arm: While a domain is suspended put its watchdogs on pause
  xen/arm: Trigger Xen suspend when Dom0 completes suspend
  xen/x86: Move