Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Tue, Nov 27, 2018 at 7:36 PM Julien Grall  wrote:
>
>
>
> On 11/17/18 4:01 PM, Mirela Simonovic wrote:
> > Hi,
>
> Hi Mirela,
>
> >
> > On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> >  wrote:
> >>
> >> On Sat, 17 Nov 2018, Dario Faggioli wrote:
> >>> On Fri, 2018-11-16 at 21:58 +, Julien Grall wrote:
> >>>> On 16/11/2018 21:41, Mirela Simonovic wrote:
> >>>>> On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> >>>>>  wrote:
> >>>>>>> It should be possible to figure out which domain needs to
> >>>>>>> awaken from
> >>>>>>> there.
> >>>>>>
> >>>>>> Actually, evtchn_send eventually will trigger a proper interrupt
> >>>>>> injection into the domain
> >>>>>> (xen/arch/arm/vgic.c:arch_evtchn_inject),
> >>>>>> which will necessarely wake it up. So it is possible that it will
> >>>>>> already work without any need for additional changes?
> >>>>>>
> >>>>>
> >>>>> Absolutely, that sounds great :) Then we could just drop this
> >>>>> patch.
> >>>>
> >>>> I don't think you can drop this patch... As you tie the host suspend
> >>>> to
> >>>> the hardware domain suspend, it may makes sense to resume at the same
> >>>> time.
> >>>>
> >>> FWIW, I think that too.
> >>>
> >>> In fact, let's assume a *fully* disaggregated setup, where dom0 only
> >>> has the toolstack, while it has no hardware, no PV backend, etc... If
> >>> we don't resume it explicitly together with Xen, who is going to resume
> >>> it? :-O
> >>
> >> Yes, that's right. However, it should work for driver domains: there is
> >> no need to wake up driver domains explicitly because they will be
> >> woken up by the frontends?
> >>
> >
> > I think we all agree, except some of us weren't so clear about it :)
> > For now, dom0 issues suspend and should resume as well when Xen
> > suspends. This is done in the series, resume is covered by this patch,
> > and commit message should be clarified.
> >
> > If a domU has a backend, we should verify that it can be woken-up by
> > an event triggered by a frontend driver in another domain.
> >
> > One day, this patch could be dropped/reverted if one come up with a
> > different logic for triggering Xen suspend. This should be of the
> > table for now, but a good option to remember for future.
>
> Such change cannot be easily dropped because some hardware domain OS may
> rely on that behavior.
>
> I am also interested to see how this is going to fit with the Dom0less
> use case. The end goal is to have no Dom0/Hardware domain. So how do you
> expect suspend/resume to work in that case? Note that I am not asking to
> implement it :).
>

From the implementation perspective and future changes which would be
required in this series it's not going to be the problem to support
dom0less approach - just one line of code (the if statement that
checks whether the hardware domain suspended) has to be replaced with
some other check. That other check would be a new condition to suspend
Xen that needs to be implemented. What that check would do depends on
the system architecture and target use cases that are specific to
dom0less architecture. I'm not so familiar with dom0less work, so
can't really say what would be the best approach to suspend condition
rules.
Do you have some whitepaper or anything similar that describes an
example of a target system architecture and/or use cases for dom0less
setup?
In dom0less world, who creates other domains? Is some domain still
more privileged than the others? E.g. is there a domain which knows
about other domains in the system so that it could do the coordination
in EL1?

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi,

On Sat, Nov 17, 2018 at 5:02 PM Mirela Simonovic
 wrote:
>
> On Sat, Nov 17, 2018 at 5:01 PM Mirela Simonovic
>  wrote:
> >
> > Hi,
> >
> > On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
> >  wrote:
> > >
> > > On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > > > On Fri, 2018-11-16 at 21:58 +, Julien Grall wrote:
> > > > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > > > >  wrote:
> > > > > > > > It should be possible to figure out which domain needs to
> > > > > > > > awaken from
> > > > > > > > there.
> > > > > > >
> > > > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > > > injection into the domain
> > > > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > > > which will necessarely wake it up. So it is possible that it will
> > > > > > > already work without any need for additional changes?
> > > > > > >
> > > > > >
> > > > > > Absolutely, that sounds great :) Then we could just drop this
> > > > > > patch.
> > > > >
> > > > > I don't think you can drop this patch... As you tie the host suspend
> > > > > to
> > > > > the hardware domain suspend, it may makes sense to resume at the same
> > > > > time.
> > > > >
> > > > FWIW, I think that too.
> > > >
> > > > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > > > has the toolstack, while it has no hardware, no PV backend, etc... If
> > > > we don't resume it explicitly together with Xen, who is going to resume
> > > > it? :-O

Forgot to answer this - when someone needs a toolstack, it will try to
type. That will raise a UART interrupt (UART used by Xen console). It
could be implemented that a UART interrupt by default wakes up dom0 if
it's asleep.
But I would keep anything like that out of this series.

> > >
> > > Yes, that's right. However, it should work for driver domains: there is
> > > no need to wake up driver domains explicitly because they will be
> > > woken up by the frontends?
> > >
> >
> > I think we all agree, except some of us weren't so clear about it :)
> > For now, dom0 issues suspend and should resume as well when Xen
> > suspends. This is done in the series, resume is covered by this patch,
> resumes
>
> > and commit message should be clarified.
> >
> > If a domU has a backend, we should verify that it can be woken-up by
> > an event triggered by a frontend driver in another domain.
> >
> > One day, this patch could be dropped/reverted if one come up with a
> > different logic for triggering Xen suspend. This should be of the
> > table for now, but a good option to remember for future.
> >
> > >
> > > > <> (Raistlin Majere)
> > > > -
> > > > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > > > Software Engineer @ SUSE https://www.suse.com/
> > > >

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

On Sat, Nov 17, 2018 at 5:01 PM Mirela Simonovic
 wrote:
>
> Hi,
>
> On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
>  wrote:
> >
> > On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > > On Fri, 2018-11-16 at 21:58 +, Julien Grall wrote:
> > > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > > >  wrote:
> > > > > > > It should be possible to figure out which domain needs to
> > > > > > > awaken from
> > > > > > > there.
> > > > > >
> > > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > > injection into the domain
> > > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > > which will necessarely wake it up. So it is possible that it will
> > > > > > already work without any need for additional changes?
> > > > > >
> > > > >
> > > > > Absolutely, that sounds great :) Then we could just drop this
> > > > > patch.
> > > >
> > > > I don't think you can drop this patch... As you tie the host suspend
> > > > to
> > > > the hardware domain suspend, it may makes sense to resume at the same
> > > > time.
> > > >
> > > FWIW, I think that too.
> > >
> > > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > > has the toolstack, while it has no hardware, no PV backend, etc... If
> > > we don't resume it explicitly together with Xen, who is going to resume
> > > it? :-O
> >
> > Yes, that's right. However, it should work for driver domains: there is
> > no need to wake up driver domains explicitly because they will be
> > woken up by the frontends?
> >
>
> I think we all agree, except some of us weren't so clear about it :)
> For now, dom0 issues suspend and should resume as well when Xen
> suspends. This is done in the series, resume is covered by this patch,
resumes

> and commit message should be clarified.
>
> If a domU has a backend, we should verify that it can be woken-up by
> an event triggered by a frontend driver in another domain.
>
> One day, this patch could be dropped/reverted if one come up with a
> different logic for triggering Xen suspend. This should be of the
> table for now, but a good option to remember for future.
>
> >
> > > <> (Raistlin Majere)
> > > -
> > > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > > Software Engineer @ SUSE https://www.suse.com/
> > >

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi,

On Sat, Nov 17, 2018 at 12:06 AM Stefano Stabellini
 wrote:
>
> On Sat, 17 Nov 2018, Dario Faggioli wrote:
> > On Fri, 2018-11-16 at 21:58 +, Julien Grall wrote:
> > > On 16/11/2018 21:41, Mirela Simonovic wrote:
> > > > On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
> > > >  wrote:
> > > > > > It should be possible to figure out which domain needs to
> > > > > > awaken from
> > > > > > there.
> > > > >
> > > > > Actually, evtchn_send eventually will trigger a proper interrupt
> > > > > injection into the domain
> > > > > (xen/arch/arm/vgic.c:arch_evtchn_inject),
> > > > > which will necessarely wake it up. So it is possible that it will
> > > > > already work without any need for additional changes?
> > > > >
> > > >
> > > > Absolutely, that sounds great :) Then we could just drop this
> > > > patch.
> > >
> > > I don't think you can drop this patch... As you tie the host suspend
> > > to
> > > the hardware domain suspend, it may makes sense to resume at the same
> > > time.
> > >
> > FWIW, I think that too.
> >
> > In fact, let's assume a *fully* disaggregated setup, where dom0 only
> > has the toolstack, while it has no hardware, no PV backend, etc... If
> > we don't resume it explicitly together with Xen, who is going to resume
> > it? :-O
>
> Yes, that's right. However, it should work for driver domains: there is
> no need to wake up driver domains explicitly because they will be
> woken up by the frontends?
>

I think we all agree, except some of us weren't so clear about it :)
For now, dom0 issues suspend and should resume as well when Xen
suspends. This is done in the series, resume is covered by this patch,
and commit message should be clarified.

If a domU has a backend, we should verify that it can be woken-up by
an event triggered by a frontend driver in another domain.

One day, this patch could be dropped/reverted if one come up with a
different logic for triggering Xen suspend. This should be of the
table for now, but a good option to remember for future.

>
> > <> (Raistlin Majere)
> > -
> > Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> > Software Engineer @ SUSE https://www.suse.com/
> >

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Stefano,

On Fri, Nov 16, 2018 at 8:09 PM Stefano Stabellini
 wrote:
>
> On Fri, 16 Nov 2018, Stefano Stabellini wrote:
> > On Fri, 16 Nov 2018, Julien Grall wrote:
> > > On 16/11/2018 12:34, Mirela Simonovic wrote:
> > > > Hi Julien,
> > >
> > > Hi Mirela,
> > >
> > > >
> > > > On Fri, Nov 16, 2018 at 12:44 PM Julien Grall  
> > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 16/11/2018 11:29, Mirela Simonovic wrote:
> > > > > > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > On Thu, Nov 15, 2018 at 9:31 PM Julien Grall 
> > > > > > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > > > > > > > The resume of Dom0 should always follow Xen's resume. This is
> > > > > > > > > done by unblocking the first vCPU of Dom0.
> > > > > > > >
> > > > > > > > Please explain why you always need to resume Dom0 afterwards.
> > > > > > > >
> > > > > > >
> > > > > > > We don't need to, but that is what is promised in the design spec.
> > > > >
> > > > > You surely had some rationale when writing the promise in the design
> > > > > document,
> > > > > right?
> > > > >
> > > > > So what is the reason behind it? I don't want to resume a domain if 
> > > > > that's
> > > > > not
> > > > > necessary.
> > > > >
> > > > > > >
> > > > > >
> > > > > > To be more specific - a domU that doesn't depend on dom0 can resume
> > > > > > and work happily without dom0 being resumed, i.e. just Xen and domU
> > > > > > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > > > > > resumes" is not a must (when there are no PV drivers involved).
> > > > >
> > > > > PV backends don't necessarily reside in the hardware domain. So how is
> > > > > this
> > > > > going to work for the other case?
> > > > >
> > > >
> > > > I honestly believe that this is not necessary, and is sub-optimal. It
> > > > relies on an assumption that dom0 contains all the PV drivers, which
> > > > is not always correct.
> > >
> > > Well, there are other reasons to resume the hardware domain. The hardware
> > > domain owns most the devices and may be part of the suspend/resume path.
> > >
> > > As you tie the host suspend to the hardware domain suspend, it may makes 
> > > sense
> > > to resume at the same time. It is the kind of rationale I would expect in 
> > > the
> > > commit message.
> > >
> > > > I would prefer if someone can tell us that any frontend will trigger
> > > > an event to the backend, and the event would go through Xen. That way,
> > > > this event would cause a domain containing the backend driver to
> > > > resume. I think this is the best possible solution, but it relies on
> > > > an assumption that the event will go through Xen, and I'm not
> > > > knowledgeable enough to claim that this is indeed the case.
> > >
> > > I think it is should work, the best way to find out if building a test 
> > > case
> > > for it.
> >
> > Yes, PV protocols use hypercalls to send notifications to the other end.
> > Specifically, the function at the Linux side is
> > include/xen/events.h:notify_remote_via_evtchn. The Xen implementation
> > for the hypercall is xen/common/event_channel.c:evtchn_send, where 'rd'
> > is the destination domain.
> >
> > It should be possible to figure out which domain needs to awaken from
> > there.
>
> Actually, evtchn_send eventually will trigger a proper interrupt
> injection into the domain (xen/arch/arm/vgic.c:arch_evtchn_inject),
> which will necessarely wake it up. So it is possible that it will
> already work without any need for additional changes?
>

Absolutely, that sounds great :) Then we could just drop this patch.
Of course, we need to test to confirm

>
> > It would fantastic to have that in this patch series, and might not be
> > too difficult to do. However, I also think that always waking up the
> > hardware domain could be a decent way to start and could be OK for this
> > patch series which is the very first to introduce suspend/resume
> > functionalities on Xen on ARM. But the limitation should be well
> > explained in the commit message, as Julien wrote, and possibly even as
> > an in-code comment with a TODO.
>

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Fri, Nov 16, 2018 at 12:44 PM Julien Grall  wrote:
>
>
>
> On 16/11/2018 11:29, Mirela Simonovic wrote:
> > On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
> >  wrote:
> >>
> >> Hi Julien,
> >>
> >> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall  wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>>> The resume of Dom0 should always follow Xen's resume. This is
> >>>> done by unblocking the first vCPU of Dom0.
> >>>
> >>> Please explain why you always need to resume Dom0 afterwards.
> >>>
> >>
> >> We don't need to, but that is what is promised in the design spec.
>
> You surely had some rationale when writing the promise in the design document,
> right?
>
> So what is the reason behind it? I don't want to resume a domain if that's not
> necessary.
>
> >>
> >
> > To be more specific - a domU that doesn't depend on dom0 can resume
> > and work happily without dom0 being resumed, i.e. just Xen and domU
> > resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
> > resumes" is not a must (when there are no PV drivers involved).
>
> PV backends don't necessarily reside in the hardware domain. So how is this
> going to work for the other case?
>

I honestly believe that this is not necessary, and is sub-optimal. It
relies on an assumption that dom0 contains all the PV drivers, which
is not always correct.

I would prefer if someone can tell us that any frontend will trigger
an event to the backend, and the event would go through Xen. That way,
this event would cause a domain containing the backend driver to
resume. I think this is the best possible solution, but it relies on
an assumption that the event will go through Xen, and I'm not
knowledgeable enough to claim that this is indeed the case.
If this assumption is correct, I strongly suggest to remove this patch
and let the wake-up for all domains happen automatically after the Xen
resumes.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

On Fri, Nov 16, 2018 at 11:33 AM Mirela Simonovic
 wrote:
>
> Hi Julien,
>
> On Thu, Nov 15, 2018 at 9:31 PM Julien Grall  wrote:
> >
> > Hi,
> >
> > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > > The resume of Dom0 should always follow Xen's resume. This is
> > > done by unblocking the first vCPU of Dom0.
> >
> > Please explain why you always need to resume Dom0 afterwards.
> >
>
> We don't need to, but that is what is promised in the design spec.
>

To be more specific - a domU that doesn't depend on dom0 can resume
and work happily without dom0 being resumed, i.e. just Xen and domU
resume. So patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
resumes" is not a must (when there are no PV drivers involved).

> > Cheers,
> >
> > --
> > Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 9:31 PM Julien Grall  wrote:
>
> Hi,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > The resume of Dom0 should always follow Xen's resume. This is
> > done by unblocking the first vCPU of Dom0.
>
> Please explain why you always need to resume Dom0 afterwards.
>

We don't need to, but that is what is promised in the design spec.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 7:23 PM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > When Dom0 finalizes its suspend procedure the suspend of Xen is
> > triggered by calling system_suspend(). Dom0 finalizes the suspend from
> > its boot core (VCPU#0), which could be mapped to any physical CPU,
> > i.e. the system_suspend() function could be executed by any physical
> > CPU. Since Xen suspend procedure has to be run by the boot CPU
> > (non-boot CPUs will be disabled at some point in suspend procedure),
> > system_suspend() execution has to continue on CPU#0.
> >
> > When the system_suspend() returns 0, it means that the system was
> > suspended and it is coming out of the resume procedure. Regardless
> > of the system_suspend() return value, after this function returns
> > Xen is fully functional, and its state, including all devices and data
> > structures, matches the state prior to calling system_suspend().
> > The status is returned by system_suspend() for debugging/logging
> > purposes and function prototype compatibility.
> >
> > Signed-off-by: Mirela Simonovic 
> > Signed-off-by: Saeed Nowshadi 
> > ---
> >   xen/arch/arm/suspend.c | 34 ++
> >   1 file changed, 34 insertions(+)
> >
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index f2338e41db..21b45f8248 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, 
> > register_t cid)
> >   _arch_set_info_guest(v, );
> >   }
> >
> > +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> > +static long system_suspend(void *data)
> > +{
> > +BUG_ON(system_state != SYS_STATE_active);
> > +
> > +return -ENOSYS;
> > +}
> > +
> >   int32_t domain_suspend(register_t epoint, register_t cid)
> >   {
> >   struct vcpu *v;
> >   struct domain *d = current->domain;
> >   bool is_thumb = epoint & 1;
> > +int status;
> >
> >   dprintk(XENLOG_DEBUG,
> >   "Dom%d suspend: epoint=0x%"PRIregister", 
> > cid=0x%"PRIregister"\n",
> > @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t 
> > cid)
> >*/
> >   vcpu_block_unless_event_pending(current);
> >
> > +/* If this was dom0 the whole system should suspend: trigger Xen 
> > suspend */
> > +if ( is_hardware_domain(d) )
> > +{
> > +/*
> > + * system_suspend should be called when Dom0 finalizes the suspend
> > + * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 
> > could
> > + * be mapped to any PCPU (this function could be executed by any 
> > PCPU).
> > + * The suspend procedure has to be finalized by the PCPU#0 
> > (non-boot
> > + * PCPUs will be disabled during the suspend).
> > + */
> > +status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> > +/*
> > + * If an error happened, there is nothing that needs to be done 
> > here
> > + * because the system_suspend always returns in fully functional 
> > state
> > + * no matter what the outcome of suspend procedure is. If the 
> > system
> > + * suspended successfully the function will return 0 after the 
> > resume.
> > + * Otherwise, if an error is returned it means Xen did not 
> > suspended,
> > + * but it is still in the same state as if the system_suspend was 
> > never
> > + * called. We dump a debug message in case of an error for 
> > debugging/
> > + * logging purpose.
> > + */
> > +if ( status )
> > +dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
> > +}
>
> After discussing we Stefano today, I have one more question regarding
> Dom0 suspend.
>
>  From my understanding, the host may resume because of an event
> targeting a guest. This means that Dom0 would still be blocked. As Dom0
> would contain PV backend, how do you expect this to work?
>
> Is there any potential dependency between frontend and backend? Or would
> Dom0 be resume when the PV frontend probe the backend?
>

We have assumed that Dom0 has to resume whenever Xen resume. So if the
wake-up interrupt was targeted to DomU, the Dom0 would resume
regarless. Vice versa does not hold.
This is done in patch "[PATCH 17/18] xen/arm: Resume Dom0 after Xen
resumes" - the Dom0 is simply unblocked at the end of Xen's resume.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 7:48 PM Julien Grall  wrote:
>
> Hi,
>
> On 14/11/2018 17:35, Mirela Simonovic wrote:
> > On Wed, Nov 14, 2018 at 6:10 PM Julien Grall  wrote:
> >> On 14/11/2018 15:40, Mirela Simonovic wrote:
> >>> On Wed, Nov 14, 2018 at 4:07 PM Julien Grall  wrote:
> >>>> On 12/11/2018 11:30, Mirela Simonovic wrote:
> >>>>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>>>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>>>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>>>> i.e. the system_suspend() function could be executed by any physical
> >>>>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>>>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>>>> system_suspend() execution has to continue on CPU#0.
> >>>>>
> >>>>> When the system_suspend() returns 0, it means that the system was
> >>>>> suspended and it is coming out of the resume procedure. Regardless
> >>>>> of the system_suspend() return value, after this function returns
> >>>>> Xen is fully functional, and its state, including all devices and data
> >>>>> structures, matches the state prior to calling system_suspend().
> >>>>> The status is returned by system_suspend() for debugging/logging
> >>>>> purposes and function prototype compatibility.
> >>>>>
> >>>>> Signed-off-by: Mirela Simonovic 
> >>>>> Signed-off-by: Saeed Nowshadi 
> >>>>> ---
> >>>>> xen/arch/arm/suspend.c | 34 ++
> >>>>> 1 file changed, 34 insertions(+)
> >>>>>
> >>>>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> >>>>> index f2338e41db..21b45f8248 100644
> >>>>> --- a/xen/arch/arm/suspend.c
> >>>>> +++ b/xen/arch/arm/suspend.c
> >>>>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, 
> >>>>> register_t cid)
> >>>>> _arch_set_info_guest(v, );
> >>>>> }
> >>>>>
> >>>>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) 
> >>>>> */
> >>>>> +static long system_suspend(void *data)
> >>>>> +{
> >>>>> +BUG_ON(system_state != SYS_STATE_active);
> >>>>> +
> >>>>> +return -ENOSYS;
> >>>>> +}
> >>>>> +
> >>>>> int32_t domain_suspend(register_t epoint, register_t cid)
> >>>>> {
> >>>>> struct vcpu *v;
> >>>>> struct domain *d = current->domain;
> >>>>> bool is_thumb = epoint & 1;
> >>>>> +int status;
> >>>>>
> >>>>> dprintk(XENLOG_DEBUG,
> >>>>> "Dom%d suspend: epoint=0x%"PRIregister", 
> >>>>> cid=0x%"PRIregister"\n",
> >>>>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, 
> >>>>> register_t cid)
> >>>>>  */
> >>>>> vcpu_block_unless_event_pending(current);
> >>>>>
> >>>>> +/* If this was dom0 the whole system should suspend: trigger Xen 
> >>>>> suspend */
> >>>>> +if ( is_hardware_domain(d) )
> >>>>> +{
> >>>>> +/*
> >>>>> + * system_suspend should be called when Dom0 finalizes the 
> >>>>> suspend
> >>>>> + * procedure from its boot core (VCPU#0). However, Dom0's 
> >>>>> VCPU#0 could
> >>>>> + * be mapped to any PCPU (this function could be executed by 
> >>>>> any PCPU).
> >>>>> + * The suspend procedure has to be finalized by the PCPU#0 
> >>>>> (non-boot
> >>>>> + * PCPUs will be disabled during the suspend).
> >>>>> + */
> >>>>> +status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> >>>>
> >>>> Based on my comment in patch #2, I don't think this will do the correct 
> &g

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 12:38 PM Julien Grall  wrote:
>
> Hi,
>
> On 11/15/18 11:10 AM, Mirela Simonovic wrote:
> > Hi Julien,
> >
> > On Thu, Nov 15, 2018 at 11:59 AM Julien Grall  wrote:
> >>
> >> Hi Mirela,
> >>
> >> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> >>> On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> >>>  wrote:
> >>>>
> >>>> On 15/11/2018 10:13, Julien Grall wrote:
> >>>>> (+ Andre)
> >>>>>
> >>>>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >>>>>> On 14/11/2018 12:49, Julien Grall wrote:
> >>>>>>> Hi Mirela,
> >>>>>>>
> >>>>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>>>>>> Hi Andrew,
> >>>>>>>>>>
> >>>>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>>>>>if ( is_idle_vcpu(p) )
> >>>>>>>>>>>>>>return;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +/* VCPU's context should not be saved if its domain is
> >>>>>>>>>>>>>> suspended */
> >>>>>>>>>>>>>> +if ( p->domain->is_shut_down &&
> >>>>>>>>>>>>>> +(p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>>>>>> +return;
> >>>>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>>>>>> state
> >>>>>>>>>>>>> to be saved on suspend.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>>>>>> believe
> >>>>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>>>>>> Let's come back on this.
> >>>>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>>>>>> Xen
> >>>>>>>>>>> tree and you'll find several pieces of documentation, including 
> >>>>>>>>>>> the
> >>>>>>>>>>> description of what this shutdown code means>>>>>>>>>>>
> >>>>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>>>>>> with
> >>>>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>>>>>> Xen's
> >>>>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>>>>>> the
> >>>>>>>>>>> toolstack.
> >>>>>>>>>> Would domain pause/unpause be a better solution?
> >>>>>>>>> Actually yes - that sounds like a very neat solution.
> >>>>>>>>
> >>>>>>>> I believe domain pause will not work here - a domain cannot pause
> >>>>>>>> itself, i.e. current domain cannot be paused. This functionality
> >>>>>>&

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Thu, Nov 15, 2018 at 11:59 AM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 11/15/18 10:33 AM, Mirela Simonovic wrote:
> > On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
> >  wrote:
> >>
> >> On 15/11/2018 10:13, Julien Grall wrote:
> >>> (+ Andre)
> >>>
> >>> On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >>>> On 14/11/2018 12:49, Julien Grall wrote:
> >>>>> Hi Mirela,
> >>>>>
> >>>>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>>>> Hi Andrew,
> >>>>>>>>
> >>>>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>>>   if ( is_idle_vcpu(p) )
> >>>>>>>>>>>>   return;
> >>>>>>>>>>>>
> >>>>>>>>>>>> +/* VCPU's context should not be saved if its domain is
> >>>>>>>>>>>> suspended */
> >>>>>>>>>>>> +if ( p->domain->is_shut_down &&
> >>>>>>>>>>>> +(p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>>>> +return;
> >>>>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>>>> state
> >>>>>>>>>>> to be saved on suspend.
> >>>>>>>>>>>
> >>>>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>>>> believe
> >>>>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>>>> Let's come back on this.
> >>>>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>>>> Xen
> >>>>>>>>> tree and you'll find several pieces of documentation, including the
> >>>>>>>>> description of what this shutdown code means.
> >>>>>>>>>
> >>>>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>>>> with
> >>>>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>>>> Xen's
> >>>>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>>>> the
> >>>>>>>>> toolstack.
> >>>>>>>> Would domain pause/unpause be a better solution?
> >>>>>>> Actually yes - that sounds like a very neat solution.
> >>>>>>
> >>>>>> I believe domain pause will not work here - a domain cannot pause
> >>>>>> itself, i.e. current domain cannot be paused. This functionality
> >>>>>> seems to assume that a domain is pausing another domain. Or I missed
> >>>>>> the point.
> >>>>>
> >>>>> Yes domain pause/unpause will not work. However, you can introduce a
> >>>>> boolean to tell you whether the domain was suspend.
> >>>>>
> >>>>> I actually quite like how suspend work for x86 HVM. This is based on
> >>>>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> >>>>
> >>>> That only exists because the ACPI controller is/was implemented in
> >>>> QEMU.  I wouldn't recommend copying it.
> >>>
> >&

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On Thu, Nov 15, 2018 at 11:26 AM Andrew Cooper
 wrote:
>
> On 15/11/2018 10:13, Julien Grall wrote:
> > (+ Andre)
> >
> > On 11/15/18 12:47 AM, Andrew Cooper wrote:
> >> On 14/11/2018 12:49, Julien Grall wrote:
> >>> Hi Mirela,
> >>>
> >>> On 14/11/2018 12:08, Mirela Simonovic wrote:
> >>>>
> >>>>
> >>>> On 11/13/2018 09:32 AM, Andrew Cooper wrote:
> >>>>> On 12/11/2018 19:56, Julien Grall wrote:
> >>>>>> Hi Andrew,
> >>>>>>
> >>>>>> On 11/12/18 4:41 PM, Andrew Cooper wrote:
> >>>>>>> On 12/11/18 16:35, Mirela Simonovic wrote:
> >>>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>>>>>> index e594b48d81..7f8105465c 100644
> >>>>>>>>>> --- a/xen/arch/arm/domain.c
> >>>>>>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>>>>>> @@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>>>>>>>>  if ( is_idle_vcpu(p) )
> >>>>>>>>>>  return;
> >>>>>>>>>>
> >>>>>>>>>> +/* VCPU's context should not be saved if its domain is
> >>>>>>>>>> suspended */
> >>>>>>>>>> +if ( p->domain->is_shut_down &&
> >>>>>>>>>> +(p->domain->shutdown_code == SHUTDOWN_suspend) )
> >>>>>>>>>> +return;
> >>>>>>>>> SHUTDOWN_suspend is used in Xen for other purpose (see
> >>>>>>>>> SCHEDOP_shutdown). The other user of that code relies on all the
> >>>>>>>>> state
> >>>>>>>>> to be saved on suspend.
> >>>>>>>>>
> >>>>>>>> We just need a flag to mark a domain as suspended, and I do
> >>>>>>>> believe
> >>>>>>>> SHUTDOWN_suspend is not used anywhere else.
> >>>>>>>> Let's come back on this.
> >>>>>>> SHUTDOWN_suspend is used for migration.  Grep for it through the
> >>>>>>> Xen
> >>>>>>> tree and you'll find several pieces of documentation, including the
> >>>>>>> description of what this shutdown code means.
> >>>>>>>
> >>>>>>> What you are introducing here is not a shutdown - it is a suspend
> >>>>>>> with
> >>>>>>> the intent to resume executing later.  As such, it shouldn't use
> >>>>>>> Xen's
> >>>>>>> shutdown infrastructure, which exists mainly to communicate with
> >>>>>>> the
> >>>>>>> toolstack.
> >>>>>> Would domain pause/unpause be a better solution?
> >>>>> Actually yes - that sounds like a very neat solution.
> >>>>
> >>>> I believe domain pause will not work here - a domain cannot pause
> >>>> itself, i.e. current domain cannot be paused. This functionality
> >>>> seems to assume that a domain is pausing another domain. Or I missed
> >>>> the point.
> >>>
> >>> Yes domain pause/unpause will not work. However, you can introduce a
> >>> boolean to tell you whether the domain was suspend.
> >>>
> >>> I actually quite like how suspend work for x86 HVM. This is based on
> >>> pause/unpause. Have a look at hvm_s3_{suspend/resume}.
> >>
> >> That only exists because the ACPI controller is/was implemented in
> >> QEMU.  I wouldn't recommend copying it.
> >
> > May I ask why? I don't think the properties are very different from
> > Arm here.
>
> If you observe, that can only be actioned by a hypercall from qemu.  It
> can't be actioned by an ACPI controller emulated by Xen.
>
> >
> >>
> >> Having spent some more time thinking about this problem, what properties
> >> does the PSCI call have?
> >>
> >> I gather from other parts of this thread that there may be a partial
> >> reset of state?  Beyond that, what else?
> >>
> >> I ask, because conceptually, domU suspend to RAM is literally just
> >> "de-schedule until we want to wake up", and

Re: [Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 6:10 PM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 14/11/2018 15:40, Mirela Simonovic wrote:
> > On Wed, Nov 14, 2018 at 4:07 PM Julien Grall  wrote:
> >> On 12/11/2018 11:30, Mirela Simonovic wrote:
> >>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>> i.e. the system_suspend() function could be executed by any physical
> >>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>> system_suspend() execution has to continue on CPU#0.
> >>>
> >>> When the system_suspend() returns 0, it means that the system was
> >>> suspended and it is coming out of the resume procedure. Regardless
> >>> of the system_suspend() return value, after this function returns
> >>> Xen is fully functional, and its state, including all devices and data
> >>> structures, matches the state prior to calling system_suspend().
> >>> The status is returned by system_suspend() for debugging/logging
> >>> purposes and function prototype compatibility.
> >>>
> >>> Signed-off-by: Mirela Simonovic 
> >>> Signed-off-by: Saeed Nowshadi 
> >>> ---
> >>>xen/arch/arm/suspend.c | 34 ++
> >>>1 file changed, 34 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> >>> index f2338e41db..21b45f8248 100644
> >>> --- a/xen/arch/arm/suspend.c
> >>> +++ b/xen/arch/arm/suspend.c
> >>> @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, 
> >>> register_t cid)
> >>>_arch_set_info_guest(v, );
> >>>}
> >>>
> >>> +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> >>> +static long system_suspend(void *data)
> >>> +{
> >>> +BUG_ON(system_state != SYS_STATE_active);
> >>> +
> >>> +return -ENOSYS;
> >>> +}
> >>> +
> >>>int32_t domain_suspend(register_t epoint, register_t cid)
> >>>{
> >>>struct vcpu *v;
> >>>struct domain *d = current->domain;
> >>>bool is_thumb = epoint & 1;
> >>> +int status;
> >>>
> >>>dprintk(XENLOG_DEBUG,
> >>>"Dom%d suspend: epoint=0x%"PRIregister", 
> >>> cid=0x%"PRIregister"\n",
> >>> @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t 
> >>> cid)
> >>> */
> >>>vcpu_block_unless_event_pending(current);
> >>>
> >>> +/* If this was dom0 the whole system should suspend: trigger Xen 
> >>> suspend */
> >>> +if ( is_hardware_domain(d) )
> >>> +{
> >>> +/*
> >>> + * system_suspend should be called when Dom0 finalizes the 
> >>> suspend
> >>> + * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 
> >>> could
> >>> + * be mapped to any PCPU (this function could be executed by any 
> >>> PCPU).
> >>> + * The suspend procedure has to be finalized by the PCPU#0 
> >>> (non-boot
> >>> + * PCPUs will be disabled during the suspend).
> >>> + */
> >>> +status = continue_hypercall_on_cpu(0, system_suspend, NULL);
> >>
> >> Based on my comment in patch #2, I don't think this will do the correct 
> >> thing on
> >> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
> >> do_vpsci_0_2_call.
> >>
> >
> > Could you please explain? I can't follow
>
> General purpose (e.g xN, pc) registers live at the bottom of the vCPU stack. 
> The
> function vcpu_suspend will reset all of them to 0 but x0 (Context ID) and pc
> (entry point).
>
> You rely on those registers to be untouched in the return path. However, this 
> is
> not the case. If you look at do_vpsci_0_2_call, x0 will be set to whatever is
> the return value of domain_suspend (e.g PSCI_SUSPEND). So x0 will not contain
> anymore the Context ID as expected by the guest.

This is expected, the system

Re: [Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 4:07 PM Julien Grall  wrote:
>
> Hi,
>
> On 12/11/2018 11:30, Mirela Simonovic wrote:
> > When Dom0 finalizes its suspend procedure the suspend of Xen is
> > triggered by calling system_suspend(). Dom0 finalizes the suspend from
> > its boot core (VCPU#0), which could be mapped to any physical CPU,
> > i.e. the system_suspend() function could be executed by any physical
> > CPU. Since Xen suspend procedure has to be run by the boot CPU
> > (non-boot CPUs will be disabled at some point in suspend procedure),
> > system_suspend() execution has to continue on CPU#0.
> >
> > When the system_suspend() returns 0, it means that the system was
> > suspended and it is coming out of the resume procedure. Regardless
> > of the system_suspend() return value, after this function returns
> > Xen is fully functional, and its state, including all devices and data
> > structures, matches the state prior to calling system_suspend().
> > The status is returned by system_suspend() for debugging/logging
> > purposes and function prototype compatibility.
> >
> > Signed-off-by: Mirela Simonovic 
> > Signed-off-by: Saeed Nowshadi 
> > ---
> >   xen/arch/arm/suspend.c | 34 ++
> >   1 file changed, 34 insertions(+)
> >
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index f2338e41db..21b45f8248 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, 
> > register_t cid)
> >   _arch_set_info_guest(v, );
> >   }
> >
> > +/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
> > +static long system_suspend(void *data)
> > +{
> > +BUG_ON(system_state != SYS_STATE_active);
> > +
> > +return -ENOSYS;
> > +}
> > +
> >   int32_t domain_suspend(register_t epoint, register_t cid)
> >   {
> >   struct vcpu *v;
> >   struct domain *d = current->domain;
> >   bool is_thumb = epoint & 1;
> > +int status;
> >
> >   dprintk(XENLOG_DEBUG,
> >   "Dom%d suspend: epoint=0x%"PRIregister", 
> > cid=0x%"PRIregister"\n",
> > @@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t 
> > cid)
> >*/
> >   vcpu_block_unless_event_pending(current);
> >
> > +/* If this was dom0 the whole system should suspend: trigger Xen 
> > suspend */
> > +if ( is_hardware_domain(d) )
> > +{
> > +/*
> > + * system_suspend should be called when Dom0 finalizes the suspend
> > + * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 
> > could
> > + * be mapped to any PCPU (this function could be executed by any 
> > PCPU).
> > + * The suspend procedure has to be finalized by the PCPU#0 
> > (non-boot
> > + * PCPUs will be disabled during the suspend).
> > + */
> > +status = continue_hypercall_on_cpu(0, system_suspend, NULL);
>
> Based on my comment in patch #2, I don't think this will do the correct thing 
> on
> Dom0. x0 will not contain cid but PSCI_SUCCESS has it is overriden in
> do_vpsci_0_2_call.
>

Could you please explain? I can't follow

> As upper layer may modify the vCPU context, I think it would be best to run
> system_suspend in a tasklet bound to CPU#0.
>

I'm not following this too, please explain

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 4:36 PM Mirela Simonovic
 wrote:
>
> Hi Julien,
>
> On Wed, Nov 14, 2018 at 3:49 PM Julien Grall  wrote:
> >
> >
> >
> > On 14/11/2018 13:05, Julien Grall wrote:
> > >
> > >
> > > On 14/11/2018 12:35, Mirela Simonovic wrote:
> > >> Hi Julien,
> > >
> > > Hi,
> > >
> > >>
> > >> On 11/14/2018 11:45 AM, Julien Grall wrote:
> > >>> Hi,
> > >>>
> > >>> On 13/11/2018 20:39, Stefano Stabellini wrote:
> > >>>> On Mon, 12 Nov 2018, Julien Grall wrote:
> > >>>>>>> However, what is the issue with saving all the registers here?
> > >>>>>>>
> > >>>>>>
> > >>>>>> We need to save arguments that are provided by a guest with system
> > >>>>>> suspend PSCI call. These arguments are the entry point that needs to
> > >>>>>> be saved in program counter and context ID that needs to be saved in
> > >>>>>> x0/r0. We don't have these arguments here. Context switch happens
> > >>>>>> after processing the system suspend PSCI call, so it's too late.
> > >>>>>
> > >>>>> It does not feel right to modify ctxt_switch{from,to} for 
> > >>>>> suspend/resume. If
> > >>>>> you want to reset the vCPU state before blocking the vCPU, then you 
> > >>>>> should
> > >>>>> instead
> > >>>>
> > >>
> > >> I think it's not clear what problem are we discussing here, at least 
> > >> it's not
> > >> to me. So I'll make an assumption, and please correct me if I'm wrong.
> > >> In the patches we submitted, the VCPU context is not reset in
> > >> ctxt_switch{from,to}. My understanding is that you suggested/asked to 
> > >> reset
> > >> the VCPU context when switch happens, and I explained why is that not 
> > >> possible
> > >> - at least not without additional code changes, that may not be so 
> > >> small. I
> > >> agree with Andrew's comment in this perspective - reset of VCPU should 
> > >> not
> > >> (and right now cannot) be done when the context is switched.
> > >
> > > I didn't ask to reset the vCPU context in the switch. Instead we should 
> > > make
> > > sure the vCPU context is synced to memory before modifying it.
> > >
> > > It seems that solution works on x86 using domain_pause (see
> > > hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. 
> > > Note
> > > that it may require more work.
> > >
> > >>
> > >>>> You missed the end of the suggestion here
> > >>>
> > >>> Whoops. I meant that instead you should save the context of the vCPU in
> > >>> advance or reset the vCPU using the system registers directly.
> > >>>
> > >>> But my preference is to reset the vCPU when you receive the wake-up 
> > >>> interrupt.
> > >>>
> > >>
> > >> Without you presenting more details how would that work I cannot really
> > >> provide any comment, nor say that your preference could work or be better
> > >> compared to what is in this series. Honestly, I don't understand what 
> > >> exactly
> > >> you're proposing, because more things needs to be think-through beyond 
> > >> the
> > >> place to put a code.
> > >> We submitted a code that works, which is very elegant and nice in my 
> > >> opinion
> > >> (fair to say we may not share opinions here), and does not require lots 
> > >> of
> > >> code changes. So there's the reference.
> > >> Could you please clarify why do you think the proposed solution is not 
> > >> good?
> > >
> > > The context switch is about saving/restore the context from the hardware. 
> > > We can
> > > decide to optimize it in the suspend case (though it might be premature), 
> > > but it
> > > is clearly the wrong place to decide to resume a domain.
> >
> > Actually, I just found a good example of why I think it is wrong and 
> > broken. You
> > rely on a context switch to always happen after suspending and on resuming 
> > the
> > guest. There are path where context/switch will not happen. An example is 
> > if you
> > have interrupt pending, you may return to the guest directly if the 
> > scheduler
> > does not have anything else to schedule.
> >
>
> Can we check whether the vcpu blocked, right? Let me be more specific - after 
> calling vcpu_block_unless_event_pending we can check whether the vcpu is 
> indeed blocked?
>
> > The problem is the variable is_shut_down and shutdown_code are only be 
> > reset on
> > restoring the vCPU. This means the next context switch will skip the saving 
> > part
> > and result to vCPU corruption.
> >
> > In general, you cannot rely on the context/switch code as it may or may not
> > happen (that's up to the scheduler).
> >
>

I wanted to write also that this is a very good example, thank you for
putting it together

>
>
> > Cheers,
> >
> > --
> > Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

On Wed, Nov 14, 2018 at 3:49 PM Julien Grall  wrote:
>
>
>
> On 14/11/2018 13:05, Julien Grall wrote:
> >
> >
> > On 14/11/2018 12:35, Mirela Simonovic wrote:
> >> Hi Julien,
> >
> > Hi,
> >
> >>
> >> On 11/14/2018 11:45 AM, Julien Grall wrote:
> >>> Hi,
> >>>
> >>> On 13/11/2018 20:39, Stefano Stabellini wrote:
> >>>> On Mon, 12 Nov 2018, Julien Grall wrote:
> >>>>>>> However, what is the issue with saving all the registers here?
> >>>>>>>
> >>>>>>
> >>>>>> We need to save arguments that are provided by a guest with system
> >>>>>> suspend PSCI call. These arguments are the entry point that needs to
> >>>>>> be saved in program counter and context ID that needs to be saved in
> >>>>>> x0/r0. We don't have these arguments here. Context switch happens
> >>>>>> after processing the system suspend PSCI call, so it's too late.
> >>>>>
> >>>>> It does not feel right to modify ctxt_switch{from,to} for 
> >>>>> suspend/resume. If
> >>>>> you want to reset the vCPU state before blocking the vCPU, then you 
> >>>>> should
> >>>>> instead
> >>>>
> >>
> >> I think it's not clear what problem are we discussing here, at least it's 
> >> not
> >> to me. So I'll make an assumption, and please correct me if I'm wrong.
> >> In the patches we submitted, the VCPU context is not reset in
> >> ctxt_switch{from,to}. My understanding is that you suggested/asked to reset
> >> the VCPU context when switch happens, and I explained why is that not 
> >> possible
> >> - at least not without additional code changes, that may not be so small. I
> >> agree with Andrew's comment in this perspective - reset of VCPU should not
> >> (and right now cannot) be done when the context is switched.
> >
> > I didn't ask to reset the vCPU context in the switch. Instead we should make
> > sure the vCPU context is synced to memory before modifying it.
> >
> > It seems that solution works on x86 using domain_pause (see
> > hvm_s3_{resume,suspend}). So I am not sure why it cannot be use on Arm. Note
> > that it may require more work.
> >
> >>
> >>>> You missed the end of the suggestion here
> >>>
> >>> Whoops. I meant that instead you should save the context of the vCPU in
> >>> advance or reset the vCPU using the system registers directly.
> >>>
> >>> But my preference is to reset the vCPU when you receive the wake-up 
> >>> interrupt.
> >>>
> >>
> >> Without you presenting more details how would that work I cannot really
> >> provide any comment, nor say that your preference could work or be better
> >> compared to what is in this series. Honestly, I don't understand what 
> >> exactly
> >> you're proposing, because more things needs to be think-through beyond the
> >> place to put a code.
> >> We submitted a code that works, which is very elegant and nice in my 
> >> opinion
> >> (fair to say we may not share opinions here), and does not require lots of
> >> code changes. So there's the reference.
> >> Could you please clarify why do you think the proposed solution is not 
> >> good?
> >
> > The context switch is about saving/restore the context from the hardware. 
> > We can
> > decide to optimize it in the suspend case (though it might be premature), 
> > but it
> > is clearly the wrong place to decide to resume a domain.
>
> Actually, I just found a good example of why I think it is wrong and broken. 
> You
> rely on a context switch to always happen after suspending and on resuming the
> guest. There are path where context/switch will not happen. An example is if 
> you
> have interrupt pending, you may return to the guest directly if the scheduler
> does not have anything else to schedule.
>

Can we check whether the vcpu blocked, right? Let me be more specific
- after calling vcpu_block_unless_event_pending we can check whether
the vcpu is indeed blocked?

> The problem is the variable is_shut_down and shutdown_code are only be reset 
> on
> restoring the vCPU. This means the next context switch will skip the saving 
> part
> and result to vCPU corruption.
>
> In general, you cannot rely on the context/switch code as it may or may not
> happen (that's up to the scheduler).
>



> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Mirela Simonovic]

On 11/14/2018 11:52 AM, Julien Grall wrote:

Hi Mirela,

On 12/11/2018 11:30, Mirela Simonovic wrote:

Non-boot CPUs have to be disabled on suspend and enabled on resume
(hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
CPU_OFF to be called by each non-boot CPU. Depending on the underlying
platform capabilities, this may lead to the physical powering down of
CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
each non-boot CPU).

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
  xen/arch/arm/suspend.c | 15 ++-
  1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 575afd5eb8..dae1b1f7d6 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -1,4 +1,5 @@
  #include 
+#include 
  #include 
  #include 
  #include 
@@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, 
register_t cid)
  /* Xen suspend. Note: data is not used (suspend is the suspend to 
RAM) */

  static long system_suspend(void *data)
  {
+int status;
+
  BUG_ON(system_state != SYS_STATE_active);
system_state = SYS_STATE_suspend;
  freeze_domains();
  +status = disable_nonboot_cpus();
+if ( status )
+{
+system_state = SYS_STATE_resume;
+goto resume_nonboot_cpus;
+}
+
  system_state = SYS_STATE_resume;
  +resume_nonboot_cpus:
+enable_nonboot_cpus();
  thaw_domains();
  system_state = SYS_STATE_active;
+dsb(sy);


Why do you need a dsb(sy) here?



Updated value of system_state variable needs to be visible to other CPUs 
before we move on



  -return -ENOSYS;


Why do you return -ENOSYS until this patch? Should not have it be 0?



To distinguish that Xen suspend wasn't supported until we at least do 
something useful in suspend procedure. This is not so important, we can 
return 0 but needs to be fixed in previous patches.



+return status;
  }
int32_t domain_suspend(register_t epoint, register_t cid)



Cheers,




___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

Thanks for the review


On 11/14/2018 01:41 PM, Julien Grall wrote:

Hi,

I am not entirely sure where to ask it, so I will do it here. From my 
understanding of the PSCI spec, all the interrupts should have been 
migrated away from turned off CPU. Where do you ensure that in the 
suspend path?




disable_nonboot_cpus will lead to CPU_OFF PSCI to be called. That is 
orthogonal to suspend support in this series




On 12/11/2018 11:30, Mirela Simonovic wrote:

System suspend may lead to a state where GIC would be powered down.
Therefore, Xen should save/restore the context of GIC on suspend/resume.
Note that the context consists of states of registers which are
controlled by the hypervisor. Other GIC registers which are accessible
by guests are saved/restored on context switch.
Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
the GIC.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
  xen/arch/arm/gic-v2.c | 147 
++

  xen/arch/arm/gic.c|  27 +
  xen/include/asm-arm/gic.h |   8 +++
  3 files changed, 182 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index e7eb01f30a..bb52b64ecb 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
  /* Maximum cpu interface per GIC */
  #define NR_GIC_CPU_IF 8
  +/* GICv2 registers to be saved/restored on system suspend/resume */
+struct gicv2_context {
+/* GICC context */
+uint32_t gicc_ctlr;
+uint32_t gicc_pmr;
+uint32_t gicc_bpr;
+/* GICD context */
+uint32_t gicd_ctlr;
+uint32_t *gicd_isenabler;
+uint32_t *gicd_isactiver;
+uint32_t *gicd_ipriorityr;
+uint32_t *gicd_itargetsr;
+uint32_t *gicd_icfgr;
+};


It took me a long time to understand that you will only save the 
context for the CPU0 and Distributor.


Yes, other physical CPUs are hot-unplugged at this point.



I would prefer if we keep separate per-CPU context and common context. 
This would keep the logic very similar to the rest of the GIC drivers.



+
+static struct gicv2_context gicv2_context;
+
+static void gicv2_alloc_context(struct gicv2_context *gc);


Please don't do forward declaration. The code should be added in the 
correct place.




Agreed


+
  static inline void writeb_gicd(uint8_t val, unsigned int offset)
  {
  writeb_relaxed(val, gicv2.map_dbase + offset);
@@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
spin_unlock();
  +/* Allocate memory to be used for saving GIC context during 
the suspend */

+gicv2_alloc_context(_context);
+
  return 0;
  }
  @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
  BUG();
  }
  +static void gicv2_alloc_context(struct gicv2_context *gc)
+{


Is it necessary to allocate them at boot? Can we make them static or 
allocate them when we suspend?




We need to allocate dynamically because the size of allocated data 
depends on the number of irq lines, which is not known at the compile time.
Alternative is to allocate on suspend, but I believe it is better to do 
this when the system boots.



+uint32_t n = gicv2_info.nr_lines;
+
+gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+if ( !gc->gicd_isenabler )
+return;
+
+gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+if ( !gc->gicd_isactiver )
+goto free_gicd_isenabler;
+
+gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+if ( !gc->gicd_itargetsr )
+goto free_gicd_isactiver;
+
+gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+if ( !gc->gicd_ipriorityr )
+goto free_gicd_itargetsr;
+
+gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
+if ( gc->gicd_icfgr )
+return;
+
+xfree(gc->gicd_ipriorityr);
+
+free_gicd_itargetsr:
+xfree(gc->gicd_itargetsr);
+
+free_gicd_isactiver:
+xfree(gc->gicd_isactiver);
+
+free_gicd_isenabler:
+xfree(gc->gicd_isenabler);
+gc->gicd_isenabler = NULL;
+}
+
+static int gicv2_suspend(void)
+{
+int i;
+
+/* Save GICC configuration */
+gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
+gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
+gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);


AFAICT, those values should always be the same. So I would just 
restore them with the hardcoded value.


This could likely be factored in a separate function that could be 
used in gicv2_cpu_init as well.



+
+/* If gicv2_alloc_context() hasn't allocated memory, return */
+if ( !gicv2_context.gicd_isenabler )
+return -ENOMEM;
+
+/* Save GICD configuration */
+gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);


Same here.


+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+gicv2_context.gicd_isenabler[i] = readl_gicd(

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On 11/13/2018 04:09 PM, Julien Grall wrote:



On 13/11/2018 10:23, Julien Grall wrote:

Hi,

On 12/11/2018 11:30, Mirela Simonovic wrote:

+/*
+ * This function sets the context of current VCPU to the state 
which is expected

+ * by the guest on resume. The expected VCPU state is:
+ * 1) pc to contain resume entry point (1st argument of PSCI 
SYSTEM_SUSPEND)
+ * 2) r0/x0 to contain context ID (2nd argument of PSCI 
SYSTEM_SUSPEND)
+ * 3) All other general purpose and system registers should have 
reset values

+ *
+ * Note: this function has to return void because it has to always 
succeed. In
+ * other words, this function is called from virtual PSCI 
SYSTEM_SUSPEND
+ * implementation, which can return only a limited number of 
possible errors,
+ * none of which could represent the fact that an error occurred 
when preparing

+ * the domain for suspend.
+ * Consequently, dynamic memory allocation cannot be done within 
this function,

+ * because if malloc fails the error has nowhere to propagate.
+ */
+static void vcpu_suspend(register_t epoint, register_t cid)
+{
+/* Static allocation because dynamic would need a non-void 
return */

+static struct vcpu_guest_context ctxt;
+struct vcpu *v = current;
+
+/* Make sure that VCPU guest regs are zeroied */
+memset(, 0, sizeof(ctxt));
+
+/* Set non-zero values to the registers prior to copying */
+ctxt.user_regs.pc64 = (u64)epoint;
+
+if ( is_32bit_domain(current->domain) )
+{
+ctxt.user_regs.r0_usr = cid;
+ctxt.user_regs.cpsr = PSR_GUEST32_INIT;


This is going to disable the MMU and Cache as requested by the PSCI 
spec. As the guest is not required to clean the cache when turning 
off the CPU/suspending, the data may not have reached the main memory.


So do you need to perform cache maintenance to avoid stale information?


Answering to myself, I have discussed about the cache with others Arm 
folks today. SYSTEM_SUSPEND may exit early due to a pending event (BTW 
you don't seem to handle it) and could jump to the specified entry 
point address.




I believe it is handled - with the vcpu_block_unless_event_pending in 
domain_suspend


In that case, the only guarantee is the data would not be lost, yet 
can still be in the cache. This means that any data used before the 
MMU & cache are enabled in the entry point should have been cleaned to 
PoC and if you modify data make sure the cache is invalidated.


Linux is doing that properly. Looking at Xen, we don't really properly 
handle the cache in the boot path. So you may randomly crash.


Cheers,




___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,


On 11/14/2018 11:45 AM, Julien Grall wrote:

Hi,

On 13/11/2018 20:39, Stefano Stabellini wrote:

On Mon, 12 Nov 2018, Julien Grall wrote:

However, what is the issue with saving all the registers here?



We need to save arguments that are provided by a guest with system
suspend PSCI call. These arguments are the entry point that needs to
be saved in program counter and context ID that needs to be saved in
x0/r0. We don't have these arguments here. Context switch happens
after processing the system suspend PSCI call, so it's too late.


It does not feel right to modify ctxt_switch{from,to} for 
suspend/resume. If
you want to reset the vCPU state before blocking the vCPU, then you 
should

instead




I think it's not clear what problem are we discussing here, at least 
it's not to me. So I'll make an assumption, and please correct me if I'm 
wrong.
In the patches we submitted, the VCPU context is not reset in 
ctxt_switch{from,to}. My understanding is that you suggested/asked to 
reset the VCPU context when switch happens, and I explained why is that 
not possible - at least not without additional code changes, that may 
not be so small. I agree with Andrew's comment in this perspective - 
reset of VCPU should not (and right now cannot) be done when the context 
is switched.



You missed the end of the suggestion here


Whoops. I meant that instead you should save the context of the vCPU 
in advance or reset the vCPU using the system registers directly.


But my preference is to reset the vCPU when you receive the wake-up 
interrupt.




Without you presenting more details how would that work I cannot really 
provide any comment, nor say that your preference could work or be 
better compared to what is in this series. Honestly, I don't understand 
what exactly you're proposing, because more things needs to be 
think-through beyond the place to put a code.
We submitted a code that works, which is very elegant and nice in my 
opinion (fair to say we may not share opinions here), and does not 
require lots of code changes. So there's the reference.

Could you please clarify why do you think the proposed solution is not good?
And why do you think that what you're proposing is better? Lets be more 
clear here - how exactly you propose to implement that?


I haven't understood so far why do you think that the proposed approach 
is not good. Maybe the whole discussion drifted a bit for no reason.


Thanks,
Mirela


Cheers,




___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

On 11/13/2018 09:32 AM, Andrew Cooper wrote:

On 12/11/2018 19:56, Julien Grall wrote:

Hi Andrew,

On 11/12/18 4:41 PM, Andrew Cooper wrote:

On 12/11/18 16:35, Mirela Simonovic wrote:

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e594b48d81..7f8105465c 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
if ( is_idle_vcpu(p) )
return;

+/* VCPU's context should not be saved if its domain is
suspended */
+if ( p->domain->is_shut_down &&
+(p->domain->shutdown_code == SHUTDOWN_suspend) )
+return;

SHUTDOWN_suspend is used in Xen for other purpose (see
SCHEDOP_shutdown). The other user of that code relies on all the state
to be saved on suspend.


We just need a flag to mark a domain as suspended, and I do believe
SHUTDOWN_suspend is not used anywhere else.
Let's come back on this.

SHUTDOWN_suspend is used for migration.  Grep for it through the Xen
tree and you'll find several pieces of documentation, including the
description of what this shutdown code means.

What you are introducing here is not a shutdown - it is a suspend with
the intent to resume executing later.  As such, it shouldn't use Xen's
shutdown infrastructure, which exists mainly to communicate with the
toolstack.

Would domain pause/unpause be a better solution?

Actually yes - that sounds like a very neat solution.


I believe domain pause will not work here - a domain cannot pause 
itself, i.e. current domain cannot be paused. This functionality seems 
to assume that a domain is pausing another domain. Or I missed the point.



~Andrew



___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 1:14 AM Stefano Stabellini
 wrote:
>
> On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> > PSCI system suspend function shall be invoked to finalize Xen suspend
> > procedure. Resume entry point, which needs to be passed via 1st argument
> > of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
> > is just a placeholder that will be implemented in assembly. Context ID,
> > which is 2nd argument of system suspend PSCI call, is unused, as in Linux.
> >
> > Signed-off-by: Mirela Simonovic 
> > Signed-off-by: Saeed Nowshadi 
> >
> > ---
> > Changes in v2:
> >
> > -The commit message was stale - referring to the do_suspend function
> > that has been renamed long time ago. Fixed commit message
> > ---
> >  xen/arch/arm/arm64/entry.S|  3 +++
> >  xen/arch/arm/psci.c   | 16 
> >  xen/arch/arm/suspend.c|  4 
> >  xen/include/asm-arm/psci.h|  1 +
> >  xen/include/asm-arm/suspend.h |  1 +
> >  5 files changed, 25 insertions(+)
> >
> > diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> > index 97b05f53ea..dbc4717903 100644
> > --- a/xen/arch/arm/arm64/entry.S
> > +++ b/xen/arch/arm/arm64/entry.S
> > @@ -533,6 +533,9 @@ ENTRY(__context_switch)
> >  mov sp, x9
> >  ret
> >
> > +ENTRY(hyp_resume)
> > +b .
> > +
> >  /*
> >   * Local variables:
> >   * mode: ASM
> > diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
> > index a93121f43b..b100bd8ad2 100644
> > --- a/xen/arch/arm/psci.c
> > +++ b/xen/arch/arm/psci.c
> > @@ -24,6 +24,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >
> >  /*
> >   * While a 64-bit OS can make calls with SMC32 calling conventions, for
> > @@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
> >  }
> >  }
> >
> > +int call_psci_system_suspend(void)
> > +{
> > +#ifdef CONFIG_ARM_64
> > +struct arm_smccc_res res;
> > +
> > +/* 2nd argument (context ID) is not used */
> > +arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), );
> > +
> > +return PSCI_RET(res);
> > +#else
> > +/* not supported */
> > +return 1;
> > +#endif
> > +}
>
> Given that suspend is unimplemented on arm32, the #ifdef is OK. But
> in that case return PSCI_NOT_SUPPORTED for arm32.
>
> Otherwise you should be able to remove this #ifdef by introducing
> something similar to PSCI_0_2_FN_NATIVE, but for PSCI_1_0 calls.
>
>
> >  void call_psci_system_off(void)
> >  {
> >  if ( psci_ver > PSCI_VERSION(0, 1) )
> > diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
> > index d1b48c339a..37926374bc 100644
> > --- a/xen/arch/arm/suspend.c
> > +++ b/xen/arch/arm/suspend.c
> > @@ -141,6 +141,10 @@ static long system_suspend(void *data)
> >  goto resume_irqs;
> >  }
> >
> > +status = call_psci_system_suspend();
> > +if ( status )
> > +dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", 
> > status);
> > +
> >  system_state = SYS_STATE_resume;
> >
> >  gic_resume();
> > diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
> > index 26462d0c47..9f6116a224 100644
> > --- a/xen/include/asm-arm/psci.h
> > +++ b/xen/include/asm-arm/psci.h
> > @@ -20,6 +20,7 @@ extern uint32_t psci_ver;
> >
> >  int psci_init(void);
> >  int call_psci_cpu_on(int cpu);
> > +int call_psci_system_suspend(void);
> >  void call_psci_cpu_off(void);
> >  void call_psci_system_off(void);
> >  void call_psci_system_reset(void);
> > diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
> > index de787d296a..7604e2e2e2 100644
> > --- a/xen/include/asm-arm/suspend.h
> > +++ b/xen/include/asm-arm/suspend.h
> > @@ -2,6 +2,7 @@
> >  #define __ASM_ARM_SUSPEND_H__
> >
> >  int32_t domain_suspend(register_t epoint, register_t cid);
> > +void hyp_resume(void);
>
> I think it would be better to separate physical suspend from virtual
> suspend, like Julien suggested. As you separate the C implementation,
> please also introduce separate header files (for instance vsuspend.h and
> suspend.h).
>

AFAIU Julien came back with Andrew's feedback suggesting that this
should rather not be done. Please see discussion "[PATCH 02/18]
xen/arm: Implement PSCI system suspend call (virtual interface)"

>
> >  #endif
> >
> > --
> > 2.13.0
> >

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

On Wed, Nov 14, 2018 at 10:13 AM Julien Grall  wrote:
>
> Hi Stefano,
>
> On 11/13/18 11:41 PM, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Mirela Simonovic wrote:
> >> System suspend may lead to a state where GIC would be powered down.
> >> Therefore, Xen should save/restore the context of GIC on suspend/resume.
> >> Note that the context consists of states of registers which are
> >> controlled by the hypervisor. Other GIC registers which are accessible
> >> by guests are saved/restored on context switch.
> >> Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
> >> the GIC.
> >>
> >> Signed-off-by: Mirela Simonovic 
> >> Signed-off-by: Saeed Nowshadi 
> >> ---
> >>   xen/arch/arm/gic-v2.c | 147 
> >> ++
> >>   xen/arch/arm/gic.c|  27 +
> >>   xen/include/asm-arm/gic.h |   8 +++
> >>   3 files changed, 182 insertions(+)
> >>
> >> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> >> index e7eb01f30a..bb52b64ecb 100644
> >> --- a/xen/arch/arm/gic-v2.c
> >> +++ b/xen/arch/arm/gic-v2.c
> >> @@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
> >>   /* Maximum cpu interface per GIC */
> >>   #define NR_GIC_CPU_IF 8
> >>
> >> +/* GICv2 registers to be saved/restored on system suspend/resume */
> >> +struct gicv2_context {
> >> +/* GICC context */
> >> +uint32_t gicc_ctlr;
> >> +uint32_t gicc_pmr;
> >> +uint32_t gicc_bpr;
> >> +/* GICD context */
> >> +uint32_t gicd_ctlr;
> >> +uint32_t *gicd_isenabler;
> >> +uint32_t *gicd_isactiver;
> >> +uint32_t *gicd_ipriorityr;
> >> +uint32_t *gicd_itargetsr;
> >> +uint32_t *gicd_icfgr;
> >> +};
> >> +
> >> +static struct gicv2_context gicv2_context;
> >> +
> >> +static void gicv2_alloc_context(struct gicv2_context *gc);
> >> +
> >>   static inline void writeb_gicd(uint8_t val, unsigned int offset)
> >>   {
> >>   writeb_relaxed(val, gicv2.map_dbase + offset);
> >> @@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
> >>
> >>   spin_unlock();
> >>
> >> +/* Allocate memory to be used for saving GIC context during the 
> >> suspend */
> >> +gicv2_alloc_context(_context);
> >
> > Please check for the return of gicv2_alloc_context and return error
> > accordingly.
>
> Suspend/resume is not a critical feature in common case. So I would
> prefer if we disable it when we can't alloc memory.
>
> I would also be tempt to #ifdef all related suspend/resume code to allow
> the integrator disabling the feature when they don't want it.
>
> >
> >
> >>   return 0;
> >>   }
> >>
> >> @@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
> >>   BUG();
> >>   }
> >>
> >> +static void gicv2_alloc_context(struct gicv2_context *gc)
> >> +{
> >> +uint32_t n = gicv2_info.nr_lines;
> >> +
> >> +gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> >> +if ( !gc->gicd_isenabler )
> >> +return;
> >
> > I would return error and return error also below for all the other
> > similar cases.
> >
> >
> >> +gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
> >> +if ( !gc->gicd_isactiver )
> >> +goto free_gicd_isenabler;
> >> +
> >> +gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> >> +if ( !gc->gicd_itargetsr )
> >> +goto free_gicd_isactiver;
> >> +
> >> +gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
> >> +if ( !gc->gicd_ipriorityr )
> >> +goto free_gicd_itargetsr;
> >> +
> >> +gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
> >> +if ( gc->gicd_icfgr )
> >> +return;
> >> +
> >> +xfree(gc->gicd_ipriorityr);
> >> +
> >> +free_gicd_itargetsr:
> >
> > You can have just one label that frees everything, as you can rely on
> > xfree working fine (doing nothing) for NULL pointers.
> >
> >
> >> +xfree(gc->gicd_itargetsr);
> >> +
> >> +free_gicd_isactiver:
> >> +

Re: [Xen-devel] [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Mirela Simonovic]

On Mon, Nov 12, 2018 at 2:15 PM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > From: Saeed Nowshadi 
> >
> > The arch_set_info_guest() has code to initialize the context of a VCPU.
> > When a VCPU is resumed it needs to go through the same context
> > initialization excluding all the validations that this routine does.
> > We move the actual VCPU context setting into a function so that it can be
> > shared with the resume path.
>
> I would rather not introduce a function that skip validation. They are
> here to catch error when setting up vCPU registers. If those errors are
> not caught, then the hypervisor will likely crash on return to the guest.
>

Ok, then we should just use arch_set_info_guest.

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

Hi,

On Tue, Nov 13, 2018 at 10:43 AM Julien Grall  wrote:
>
> Hi Stefano,
>
> On 12/11/2018 23:46, Stefano Stabellini wrote:
> > On Mon, 12 Nov 2018, Julien Grall wrote:
> >> Hi,
> >>
> >> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>> When Dom0 finalizes its suspend procedure the suspend of Xen is
> >>> triggered by calling system_suspend(). Dom0 finalizes the suspend from
> >>> its boot core (VCPU#0), which could be mapped to any physical CPU,
> >>> i.e. the system_suspend() function could be executed by any physical
> >>> CPU. Since Xen suspend procedure has to be run by the boot CPU
> >>> (non-boot CPUs will be disabled at some point in suspend procedure),
> >>> system_suspend() execution has to continue on CPU#0.
> >>
> >> Nothing in the domain_suspend code checks that domain_suspend is called by
> >> vCPU0. I also can't find any restriction in the PSCI spec to run on pCPU0. 
> >> Can
> >> you provide more details why this required?
> >
> > The spec says that "to use this API, a calling OS must power down all
> > but one core through calls to CPU_OFF". It is natural to think that the
> > remaining core would be (physical or virtual) cpu0, but actually it is
> > not clearly stated by the spec. For dom0, we could simply check that
> > only 1 vcpu is left ON.
>
> It is what we already do in the code. The comment and commit message does not
> match the code.
>
> > For Xen and the physical system_suspend call,
> > it makes sense to make the call on pcpu0.
>
> This needs some rationale. What's wrong with issue the suspend from the 
> current
> pCPU?
>

The guest doesn't have to finalize its suspend procedure from VCPU0.
That is not required by the spec, although it is commonly the case in
implementations. We don't make any assumptions on CPU ID until Dom0
suspends. In the context of Dom0 and comments in this patch, linux
will finalize the suspend procedure from its boot core/CPU0, so the
comment is correct.

When it comes to suspending Xen, we use generic, existing functions to
disable secondary cpus (disable_nonboot_cpus) later in the suspend
procedure. This function makes the assumption that non-boot CPUs are
the ones whose index is not zero, and the pCPU0 will not be disabled
in this process. So we have to switch the execution to physical CPU0
before we move on with suspend. The same is done for x86 suspend.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 6:00 PM Julien Grall  wrote:
>
>
>
> On 11/12/18 4:52 PM, Mirela Simonovic wrote:
> > Hi Julien,
>
> Hi,
>
> > Thanks for the feedback.
> >
> > On Mon, Nov 12, 2018 at 4:36 PM Julien Grall  wrote:
> >>
> >> Hi Mirela,
> >>
> >> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >>> GIC and virtual timer context must be saved when the domain suspends.
> >>
> >> Please provide the rationale for this. Is it required by the spec?
> >>
> >
> > This is required for GIC because a guest leaves enabled interrupts in
> > the GIC that could wake it up, and on resume it should be able to
> > detect which interrupt woke it up. Without saving/restoring the state
> > of GIC this would not be possible.
>
> I am confused. In patch #10, you save the GIC host because the GIC can
> be powered-down. Linux is also saving the GIC state. So how the
> interrupt can come up if the GIC is powered down?
>

After Xen (or Linux in the config without Xen) hands over the control
to EL3 on suspend (calls system suspend PSCI to EL3), it leaves
enabled interrupts that are its wake-up sources. Saving a GIC state
only means that its current configuration will be remembered somewhere
in data structures, but the configuration is not changed on suspend.
Everything that happens with interrupt configuration from this point
on is platform specific. Now there are 2 options: 1) EL3 will never
allow CPU0 to be powered down and the wake-up interrupt will indeed
propagate via GIC; or 2) CPU0 will be powered down and the GIC may be
powered down as well, so an external help is needed to deal with
wake-up and resume (e.g. in order to react to a wake-up interrupt
while the GIC is down, and power up CPU0). This external help is
provided by some low-power processor outside of the Cortex-A cluster.

So the platform firmware is responsible for properly configuring a
wake-up path if GIC goes down. This is commonly handled by EL3
communicating with low-power processor. When the GIC power comes up,
the interrupt triggered by a peripheral is still active and the
software on Cortex-A cluster should be able to observe it once the GIC
state is restored, i.e. interrupts get enabled at GIC.

> > For timer, my understanding is that vtimer accounts for how long the
> > guest was executing and this tracking should not be affected by the
> > suspend/resume sequence. Without saving/restoring the state of vtimer,
> > the time tracking would be affected by the suspend/resume.
>
> Currently, vtimer does not take such things into account. It only setup
> a timer in Xen to wake up the guest later on.
>
> > Hope this is ok, I'll add it to the commit message.
> >
> >>> This is done by moving the respective code in ctxt_switch_from()
> >>> before the return that happens if the domain suspended.
> >>>
> >>> Signed-off-by: Mirela Simonovic 
> >>> Signed-off-by: Saeed Nowshadi 
> >>> ---
> >>>xen/arch/arm/domain.c | 14 +++---
> >>>1 file changed, 7 insertions(+), 7 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>> index 7f8105465c..bebe3238e8 100644
> >>> --- a/xen/arch/arm/domain.c
> >>> +++ b/xen/arch/arm/domain.c
> >>> @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
> >>>if ( is_idle_vcpu(p) )
> >>>return;
> >>>
> >>> +/* VGIC */
> >>> +gic_save_state(p);
> >>> +
> >>> +/* Arch timer */
> >>> +p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> >>> +virt_timer_save(p);
> >>> +
> >>>/* VCPU's context should not be saved if its domain is suspended */
> >>
> >> The GIC and timer are part of the vCPU context. So this comment looks a
> >> bit stale.
> >
> > It's not stale, but incorrect in that perspective. The comment should
> > be updated to "VCPU architecture specific context should ..."
>
> But the timer is part of the architecture...
>
> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

Hi Julien,

Thanks for the feedback.

On Mon, Nov 12, 2018 at 4:36 PM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > GIC and virtual timer context must be saved when the domain suspends.
>
> Please provide the rationale for this. Is it required by the spec?
>

This is required for GIC because a guest leaves enabled interrupts in
the GIC that could wake it up, and on resume it should be able to
detect which interrupt woke it up. Without saving/restoring the state
of GIC this would not be possible.
For timer, my understanding is that vtimer accounts for how long the
guest was executing and this tracking should not be affected by the
suspend/resume sequence. Without saving/restoring the state of vtimer,
the time tracking would be affected by the suspend/resume.
Hope this is ok, I'll add it to the commit message.

> > This is done by moving the respective code in ctxt_switch_from()
> > before the return that happens if the domain suspended.
> >
> > Signed-off-by: Mirela Simonovic 
> > Signed-off-by: Saeed Nowshadi 
> > ---
> >   xen/arch/arm/domain.c | 14 +++---
> >   1 file changed, 7 insertions(+), 7 deletions(-)
> >
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index 7f8105465c..bebe3238e8 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
> >   if ( is_idle_vcpu(p) )
> >   return;
> >
> > +/* VGIC */
> > +gic_save_state(p);
> > +
> > +/* Arch timer */
> > +p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> > +virt_timer_save(p);
> > +
> >   /* VCPU's context should not be saved if its domain is suspended */
>
> The GIC and timer are part of the vCPU context. So this comment looks a
> bit stale.

It's not stale, but incorrect in that perspective. The comment should
be updated to "VCPU architecture specific context should ..."

>
> >   if ( p->domain->is_shut_down &&
> >   (p->domain->shutdown_code == SHUTDOWN_suspend) )
> > @@ -115,10 +122,6 @@ static void ctxt_switch_from(struct vcpu *p)
> >   p->arch.tpidrro_el0 = READ_SYSREG(TPIDRRO_EL0);
> >   p->arch.tpidr_el1 = READ_SYSREG(TPIDR_EL1);
> >
> > -/* Arch timer */
> > -p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
> > -virt_timer_save(p);
> > -
> >   if ( is_32bit_domain(p->domain) && cpu_has_thumbee )
> >   {
> >   p->arch.teecr = READ_SYSREG32(TEECR32_EL1);
> > @@ -170,9 +173,6 @@ static void ctxt_switch_from(struct vcpu *p)
> >   /* VFP */
> >   vfp_save_state(p);
> >
> > -/* VGIC */
> > -gic_save_state(p);
> > -
> >   isb();
> >   }
> >
> >
>
> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

Hi Julien,

Thanks for your feedback, I'll need to answer in iterations.

On Mon, Nov 12, 2018 at 4:27 PM Julien Grall  wrote:
>
> Hi Mirela,
>
> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> > The implementation consists of:
> > -Adding PSCI system suspend call as new PSCI function
> > -Trapping PSCI system_suspend HVC
> > -Implementing PSCI system suspend call (virtual interface that allows
> >   guests to suspend themselves)
> >
> > The PSCI system suspend should be called by a guest from its boot
> > VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
> > CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
> > are left enabled by the guest are assumed to be its wake-up interrupts.
> > Therefore, a wake-up interrupt triggers the resume of the guest. Guest
> > should resume regardless of the state of Xen (suspended or not).
> >
> > When a guest calls PSCI system suspend the respective domain will be
> > suspended if the following conditions are met:
> > 1) Given resume entry point is not invalid
> > 2) Other (if any) VCPUs of the calling guest are hot-unplugged
> >
> > If the conditions above are met the calling domain is labeled as
> > suspended and the calling VCPU is blocked. If nothing else wouldn't
> > be done the suspended domain would resume from the place where it
> > called PSCI system suspend. This is expected if processing of the PSCI
> > system suspend call fails. However, in the case of success the calling
> > guest should resume (continue execution after the wake-up) from the entry
> > point which is given as the first argument of the PSCI system suspend
> > call. In addition to the entry point, the guest expects to start within
> > the environment whose state matches the state after reset. This means
> > that the guest should find reset register values, MMU disabled, etc.
> > Thereby, the context of VCPU should be 'reset' (as if the system is
> > comming out of reset), the program counter should contain entry point,
> > which is 1st argument, and r0/x0 should contain context ID which is 2nd
> > argument of PSCI system suspend call. The context of VCPU is set
> > accordingly when the PSCI system suspend is processed, so that nothing
> > needs to be done on resume/wake-up path. However, in order to ensure that
> > this context doesn't get overwritten by the scheduler when scheduling out
> > this VCPU (would normally happen after the calling CPU is blocked), we need
> > to check whether to return early from ctxt_switch_from().
> >
> > There are variables in domain structure to keep track of domain shutdown.
> > One of existing shutdown reason is 'suspend' which this patch is using to
> > track the suspend state of a domain. Those variables are used to determine
> > whether to early return from ctxt_switch_from() or not.
> >
> > A suspended domain will resume after the Xen receives an interrupt which is
> > targeted to the domain, unblocks the domain's VCPU, and schedules it in.
> > When the VCPU is scheduled in, the VCPU context is already reset, and
> > contains the right resume entry point in program counter that will be
> > restored in ctxt_switch_to(). The only thing that needs to be done at this
> > point is to clear the variables that marked the domain state as suspended.
> >
> > Signed-off-by: Mirela Simonovic 
> > Signed-off-by: Saeed Nowshadi 
> >
> > ---
> > Changes in v2:
> >
> > -Fix print to compile for arm32 and to align with Xen coding style
> > ---
> >   xen/arch/arm/Makefile|   1 +
> >   xen/arch/arm/domain.c|  13 +++
> >   xen/arch/arm/suspend.c   | 166 
> > +++
> >   xen/arch/arm/vpsci.c |  19 +
> >   xen/include/asm-arm/perfc_defn.h |   1 +
> >   xen/include/asm-arm/psci.h   |   2 +
> >   xen/include/asm-arm/suspend.h|  16 
> >   xen/include/xen/sched.h  |   1 +
> >   8 files changed, 219 insertions(+)
> >   create mode 100644 xen/arch/arm/suspend.c
> >   create mode 100644 xen/include/asm-arm/suspend.h
> >
> > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> > index 23c5d9adbc..744b1a4dc8 100644
> > --- a/xen/arch/arm/Makefile
> > +++ b/xen/arch/arm/Makefile
> > @@ -43,6 +43,7 @@ obj-y += setup.o
> >   obj-y += shutdown.o
> >   obj-y += smp.o
> >   obj-y += smpboot.o
> > +obj-y += suspend.o
> >   obj-y += sysctl.o
> >   obj-y += time.o
> >   obj-y += traps.o
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/

Re: [Xen-devel] [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

Hi Jan,

On Mon, Nov 12, 2018 at 4:23 PM Jan Beulich  wrote:
>
> >>> On 12.11.18 at 16:17,  wrote:
> > Hi Jan,
> >
> > On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich  wrote:
> >>
> >> >>> On 12.11.18 at 12:30,  wrote:
> >> > --- a/xen/include/xen/timer.h
> >> > +++ b/xen/include/xen/timer.h
> >> > @@ -18,6 +18,9 @@ struct timer {
> >> >  /* System time expiry value (nanoseconds since boot). */
> >> >  s_time_t expires;
> >> >
> >> > +/* Suspend timestamp value (nanoseconds since boot). */
> >> > +s_time_t suspended;
> >>
> >> I can't see how this becomes a universally useful field, so I
> >> don't think it belongs here. For your purpose deriving a new
> >> structure containing struct timer and this one field should
> >> work.
> >
> > Each 'expires' value needs to be coupled with 'suspended' value to be
> > able to deal with watchdogs when suspending a domain. This is specific
> > to a watchdog.
> > If we add a derived structure to contain this field we still need to
> > associate such a structure with each watchdog of a domain, which is
> > also domain generic info and requires a lot more (in my opinion
> > unnecessary) changes. Please let me know if I misinterpreted your
> > proposal.
>
> You add the field to all timer instances, yet you need it only for
> very few. Why not have a struct watchdog_timer, and use it
> for struct domain's watchdog_timer[]? I'm afraid I don't see the
> difficulties you appear to see.
>

Thanks, now it's clear. We need to change the type for watchdog_timer
to be the derived structure of timer that additionally contains
'suspended' variable. That makes sense, we'll do so.

Thanks,
Mirela

> Jan
>
>

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

Hi Jan,

On Mon, Nov 12, 2018 at 12:47 PM Jan Beulich  wrote:
>
> >>> On 12.11.18 at 12:30,  wrote:
> > --- a/xen/include/xen/timer.h
> > +++ b/xen/include/xen/timer.h
> > @@ -18,6 +18,9 @@ struct timer {
> >  /* System time expiry value (nanoseconds since boot). */
> >  s_time_t expires;
> >
> > +/* Suspend timestamp value (nanoseconds since boot). */
> > +s_time_t suspended;
>
> I can't see how this becomes a universally useful field, so I
> don't think it belongs here. For your purpose deriving a new
> structure containing struct timer and this one field should
> work.

Each 'expires' value needs to be coupled with 'suspended' value to be
able to deal with watchdogs when suspending a domain. This is specific
to a watchdog.
If we add a derived structure to contain this field we still need to
associate such a structure with each watchdog of a domain, which is
also domain generic info and requires a lot more (in my opinion
unnecessary) changes. Please let me know if I misinterpreted your
proposal.

Thanks,
Mirela

>
> Jan
>
>

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 1:08 PM Julien Grall  wrote:
>
> On 11/12/18 12:01 PM, Mirela Simonovic wrote:
> > Hi Julien,
>
> Hi Mirela,
>
> Please configure your e-mail client to avoid quoting using "space".
> Otherwise, this is going to make difficult to follow the discussions.
>

Sure, sorry.

> > On Mon, Nov 12, 2018 at 12:50 PM Julien Grall  > <mailto:julien.gr...@arm.com>> wrote:
> >
> > Hi Mirela,
> >
> > Thank you for posting the series. Could you provide a branch with the
> > patch applied?
> >
> >
> > I have applied patches on top of upstream/staging.
>
> patches are applied every day to staging. So can you please provide the
> exact commit?
>

Oh yes, the staging already moved. It was this one: 388c55b
tools/misc: fix hard tabs in xen-hvmctx.c

> Also, it is common for big series to provide a branch with the patch
> series applied. This makes easier for people to try your series.
>
> >
> > On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >  >
> > 
> > 
> >  > The series does not include:
> >  > a) UART driver-specific suspend/resume that gets called when
> > console suspends
> >
> > I feel it will be difficult to test this series without UART driver
> > support. Would it be possible to integrate them in that series?
> >
> >
> > Yes that's possible, but you'll need more than a UART driver's
> > suspend/resume to test this. E.g. for testing the series on Xilinx
> > Ultrascale+ MPSoC we need a set of patches that provides the basic
> > platform support needed to boot, which is not yet upstreamed. Luckily,
> > the suspend support is independent (ARM specific).
>
> I am a bit confused. I thought it was possible to boot Xen out-of-box on
> the Zynq MP. What is missing?
>
> > Please let me check with Xilinx people and Stefano how to deal with
> > this, and I'll come back with a real answer.
>
> If that's too difficult to provide in short term. Then I would
> appreciate if testing is done on another platform (e.g Foundation
> Model). So I can at least verify the series before we merge it.
>

No, it's not difficult at all, I just need Xilinx support on this.

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi Julien,

On Mon, Nov 12, 2018 at 12:50 PM Julien Grall  wrote:

> Hi Mirela,
>
> Thank you for posting the series. Could you provide a branch with the
> patch applied?
>
>
I have applied patches on top of upstream/staging.


> On 11/12/18 11:30 AM, Mirela Simonovic wrote:
> >
> 
> > The series does not include:
> > a) UART driver-specific suspend/resume that gets called when console
> suspends
>
> I feel it will be difficult to test this series without UART driver
> support. Would it be possible to integrate them in that series?
>
>
Yes that's possible, but you'll need more than a UART driver's
suspend/resume to test this. E.g. for testing the series on Xilinx
Ultrascale+ MPSoC we need a set of patches that provides the basic platform
support needed to boot, which is not yet upstreamed. Luckily, the suspend
support is independent (ARM specific).
Please let me check with Xilinx people and Stefano how to deal with this,
and I'll come back with a real answer.

Thanks,
Mirela


> Cheers,
>
> --
> Julien Grall
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

What?

On Mon, Nov 12, 2018 at 12:42 PM Jan Beulich  wrote:

> >>> On 12.11.18 at 12:30,  wrote:
> > --- a/xen/include/xen/sched.h
> > +++ b/xen/include/xen/sched.h
> > @@ -24,6 +24,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> >  #include 
>
> Why?
>
> Jan.
>
>
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

Hi,

One thing I screwed - I forgot to remove changes log from an internal
review, so please ignore it. This is officially the first version.

Thanks,
Mirela

On Mon, Nov 12, 2018 at 12:31 PM Mirela Simonovic <
mirela.simono...@aggios.com> wrote:

> This series contains support for suspend to RAM (in the following text just
> 'suspend') for Xen on arm64. The implementation is aligned with the design
> specification that has been proposed on xen-devel list:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>
> At a high-level the series contains:
> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
> 2) Support for resuming a guest on an interrupt targeted to that guest
> 3) Support for suspending Xen after dom0 finalizes the suspend
> 4) Support for resuming Xen on an interrupt that is targeted to a guest
>
>
> 
> In more details:
>
> *** About suspending/resuming guests
>
> The patches included in this series allow PSCI compliant guests that have
> support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
> suspend and resume on top of Xen without any EL1 code changes.
>
> During their suspend procedure guests will hot-unplug their secondary CPUs,
> triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
> suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
> Guests will save/restore their own EL1 context on suspend/resume.
>
> A guest is expected to leave enabled interrupts that are considered to be
> its
> wake-up sources. Those interrupts will be able to wake up the guest. This
> holds
> regardless of the state of the underlying software layers, i.e. whether
> Xen gets
> suspended or not doesn't affect the ability of the guest to wake up.
>
> First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from
> which
> the guest assumes to start on resume. On resume, guests assume to be
> running in
> an environment whose state matches the CPU state after reset, e.g. with
> reset
> register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
> VCPU context and save the resume entry point into program counter before
> the
> guest's VCPU gets scheduled in on resume. This is done when the guest
> finalizes
> its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure
> that
> the resume-ready VCPU's context does not get overwritten later upon the
> context
> switch when the VCPU is scheduled out.
> Xen also needs to take care that the guest's view of GIC and timer gets
> saved.
> Also, while a guest is suspended its watchdogs are paused, in order to
> avoid
> watchdog triggered shutdown of a guest that has been asleep for a period
> of time
> that is longer than the watchdog period.
>
> After this point, from Xen's point of view a suspended guest has one VCPU
> blocked, waiting for an interrupt. When such an interrupt comes, Xen will
> unblock the VCPU of the suspended domain, which results in the guest
> resuming.
>
> *** About suspending/resuming Xen
>
> Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
> considered to be the decision maker for EL1 and EL2.
> On suspend, Xen will first freeze all domains. Then, Xen disables physical
> secondary CPUs, which leads to physical CPU_OFF to be called by each
> secondary
> CPU. After that Xen finalizes the suspend from the boot CPU.
>
> This consists of suspending the timer, i.e. suppressing its interrupts (we
> don't
> want to be woken up by a timer, there is no VCPU ready to be scheduled).
> Then
> the state of GIC is saved, console is suspended, and CPU context is saved.
> The
> saved context tells where Xen needs to continue execution on resume.
> Since Xen will resume with MMU disabled, the first thing to do in resume
> is to
> resume memory management in order to be able to access the context that
> needs to
> be restored (we know virtual address of the context data). Finally Xen
> calls
> SYSTEM_SUSPEND PSCI to the EL3.
>
> When resuming, all the steps done in suspend need to be reverted. This is
> completed by unblocking dom0's VCPU, because we always want the dom0 to
> resume,
> regardless of the target domain whose interrupt woke up Xen.
>
> *** Handling of unprivileged guests during Xen suspend/resume
>
> Any domU that is not suspended when dom0 suspends will be frozen, domUs
> that are
> already suspended remain suspended. On resume the suspended domUs still
> remain
> suspended (unless their wake interrupt caused Xen to wake) while the
> others will
> be thawed.
>
> For more details please refer to

[Xen-devel] [PATCH 10/18] xen/arm: Implement GIC suspend/resume functions (gicv2 only)

[Mirela Simonovic]

System suspend may lead to a state where GIC would be powered down.
Therefore, Xen should save/restore the context of GIC on suspend/resume.
Note that the context consists of states of registers which are
controlled by the hypervisor. Other GIC registers which are accessible
by guests are saved/restored on context switch.
Tested on Xilinx Ultrascale+ MPSoC with (and without) powering down
the GIC.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/gic-v2.c | 147 ++
 xen/arch/arm/gic.c|  27 +
 xen/include/asm-arm/gic.h |   8 +++
 3 files changed, 182 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index e7eb01f30a..bb52b64ecb 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -123,6 +123,25 @@ static DEFINE_PER_CPU(u8, gic_cpu_id);
 /* Maximum cpu interface per GIC */
 #define NR_GIC_CPU_IF 8
 
+/* GICv2 registers to be saved/restored on system suspend/resume */
+struct gicv2_context {
+/* GICC context */
+uint32_t gicc_ctlr;
+uint32_t gicc_pmr;
+uint32_t gicc_bpr;
+/* GICD context */
+uint32_t gicd_ctlr;
+uint32_t *gicd_isenabler;
+uint32_t *gicd_isactiver;
+uint32_t *gicd_ipriorityr;
+uint32_t *gicd_itargetsr;
+uint32_t *gicd_icfgr;
+};
+
+static struct gicv2_context gicv2_context;
+
+static void gicv2_alloc_context(struct gicv2_context *gc);
+
 static inline void writeb_gicd(uint8_t val, unsigned int offset)
 {
 writeb_relaxed(val, gicv2.map_dbase + offset);
@@ -1310,6 +1329,9 @@ static int __init gicv2_init(void)
 
 spin_unlock();
 
+/* Allocate memory to be used for saving GIC context during the suspend */
+gicv2_alloc_context(_context);
+
 return 0;
 }
 
@@ -1319,6 +1341,129 @@ static void gicv2_do_LPI(unsigned int lpi)
 BUG();
 }
 
+static void gicv2_alloc_context(struct gicv2_context *gc)
+{
+uint32_t n = gicv2_info.nr_lines;
+
+gc->gicd_isenabler = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+if ( !gc->gicd_isenabler )
+return;
+
+gc->gicd_isactiver = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 32));
+if ( !gc->gicd_isactiver )
+goto free_gicd_isenabler;
+
+gc->gicd_itargetsr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+if ( !gc->gicd_itargetsr )
+goto free_gicd_isactiver;
+
+gc->gicd_ipriorityr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 4));
+if ( !gc->gicd_ipriorityr )
+goto free_gicd_itargetsr;
+
+gc->gicd_icfgr = xzalloc_array(uint32_t, DIV_ROUND_UP(n, 16));
+if ( gc->gicd_icfgr )
+return;
+
+xfree(gc->gicd_ipriorityr);
+
+free_gicd_itargetsr:
+xfree(gc->gicd_itargetsr);
+
+free_gicd_isactiver:
+xfree(gc->gicd_isactiver);
+
+free_gicd_isenabler:
+xfree(gc->gicd_isenabler);
+gc->gicd_isenabler = NULL;
+}
+
+static int gicv2_suspend(void)
+{
+int i;
+
+/* Save GICC configuration */
+gicv2_context.gicc_ctlr = readl_gicc(GICC_CTLR);
+gicv2_context.gicc_pmr = readl_gicc(GICC_PMR);
+gicv2_context.gicc_bpr = readl_gicc(GICC_BPR);
+
+/* If gicv2_alloc_context() hasn't allocated memory, return */
+if ( !gicv2_context.gicd_isenabler )
+return -ENOMEM;
+
+/* Save GICD configuration */
+gicv2_context.gicd_ctlr = readl_gicd(GICD_CTLR);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+gicv2_context.gicd_isenabler[i] = readl_gicd(GICD_ISENABLER + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+gicv2_context.gicd_isactiver[i] = readl_gicd(GICD_ISACTIVER + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+gicv2_context.gicd_ipriorityr[i] = readl_gicd(GICD_IPRIORITYR + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 4); i++ )
+gicv2_context.gicd_itargetsr[i] = readl_gicd(GICD_ITARGETSR + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 16); i++ )
+gicv2_context.gicd_icfgr[i] = readl_gicd(GICD_ICFGR + i * 4);
+
+return 0;
+}
+
+static void gicv2_resume(void)
+{
+int i;
+
+ASSERT(gicv2_context.gicd_isenabler);
+ASSERT(gicv2_context.gicd_isactiver);
+ASSERT(gicv2_context.gicd_ipriorityr);
+ASSERT(gicv2_context.gicd_itargetsr);
+ASSERT(gicv2_context.gicd_icfgr);
+
+/* Disable CPU interface and distributor */
+writel_gicc(0, GICC_CTLR);
+writel_gicd(0, GICD_CTLR);
+isb();
+
+/* Restore GICD configuration */
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+writel_gicd(0x, GICD_ICENABLER + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+writel_gicd(gicv2_context.gicd_isenabler[i], GICD_ISENABLER + i * 4);
+
+for ( i = 0; i < DIV_ROUND_UP(gicv2_info.nr_lines, 32); i++ )
+writel_gicd(0x, GICD_ICACTIVER + i *

[Xen-devel] [PATCH 18/18] xen/arm: Suspend/resume console on Xen suspend/resume

[Mirela Simonovic]

This is done using generic console_suspend/resume functions that cause
uart driver specific suspend/resume handlers to be called for each
initialized port (if the port has suspend/resume driver handlers
implemented).

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index a05aea9c25..6d7d69539b 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -1,5 +1,6 @@
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -149,6 +150,15 @@ static long system_suspend(void *data)
 goto resume_irqs;
 }
 
+dprintk(XENLOG_DEBUG, "Suspend\n");
+status = console_suspend();
+if ( status )
+{
+dprintk(XENLOG_ERR, "Failed to suspend the console, err=%d\n", status);
+system_state = SYS_STATE_resume;
+goto resume_console;
+}
+
 if ( hyp_suspend(_context) )
 {
 status = call_psci_system_suspend();
@@ -175,6 +185,10 @@ static long system_suspend(void *data)
  */
 mmu_init_secondary_cpu();
 
+resume_console:
+console_resume();
+dprintk(XENLOG_DEBUG, "Resume\n");
+
 gic_resume();
 
 resume_irqs:
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 13/18] xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)

[Mirela Simonovic]

PSCI system suspend function shall be invoked to finalize Xen suspend
procedure. Resume entry point, which needs to be passed via 1st argument
of PSCI system suspend call to the EL3, is hyp_resume. For now, hyp_resume
is just a placeholder that will be implemented in assembly. Context ID,
which is 2nd argument of system suspend PSCI call, is unused, as in Linux.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 

---
Changes in v2:

-The commit message was stale - referring to the do_suspend function
that has been renamed long time ago. Fixed commit message
---
 xen/arch/arm/arm64/entry.S|  3 +++
 xen/arch/arm/psci.c   | 16 
 xen/arch/arm/suspend.c|  4 
 xen/include/asm-arm/psci.h|  1 +
 xen/include/asm-arm/suspend.h |  1 +
 5 files changed, 25 insertions(+)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 97b05f53ea..dbc4717903 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -533,6 +533,9 @@ ENTRY(__context_switch)
 mov sp, x9
 ret
 
+ENTRY(hyp_resume)
+b .
+
 /*
  * Local variables:
  * mode: ASM
diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
index a93121f43b..b100bd8ad2 100644
--- a/xen/arch/arm/psci.c
+++ b/xen/arch/arm/psci.c
@@ -24,6 +24,7 @@
 #include 
 #include 
 #include 
+#include 
 
 /*
  * While a 64-bit OS can make calls with SMC32 calling conventions, for
@@ -67,6 +68,21 @@ void call_psci_cpu_off(void)
 }
 }
 
+int call_psci_system_suspend(void)
+{
+#ifdef CONFIG_ARM_64
+struct arm_smccc_res res;
+
+/* 2nd argument (context ID) is not used */
+arm_smccc_smc(PSCI_1_0_FN64_SYSTEM_SUSPEND, __pa(hyp_resume), );
+
+return PSCI_RET(res);
+#else
+/* not supported */
+return 1;
+#endif
+}
+
 void call_psci_system_off(void)
 {
 if ( psci_ver > PSCI_VERSION(0, 1) )
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index d1b48c339a..37926374bc 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -141,6 +141,10 @@ static long system_suspend(void *data)
 goto resume_irqs;
 }
 
+status = call_psci_system_suspend();
+if ( status )
+dprintk(XENLOG_ERR, "PSCI system suspend failed, err=%d\n", status);
+
 system_state = SYS_STATE_resume;
 
 gic_resume();
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 26462d0c47..9f6116a224 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -20,6 +20,7 @@ extern uint32_t psci_ver;
 
 int psci_init(void);
 int call_psci_cpu_on(int cpu);
+int call_psci_system_suspend(void);
 void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
diff --git a/xen/include/asm-arm/suspend.h b/xen/include/asm-arm/suspend.h
index de787d296a..7604e2e2e2 100644
--- a/xen/include/asm-arm/suspend.h
+++ b/xen/include/asm-arm/suspend.h
@@ -2,6 +2,7 @@
 #define __ASM_ARM_SUSPEND_H__
 
 int32_t domain_suspend(register_t epoint, register_t cid);
+void hyp_resume(void);
 
 #endif
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 12/18] xen/arm: Suspend/resume timer interrupt generation

[Mirela Simonovic]

Timer interrupts have to be disabled while the system is in suspend.
Otherwise, a timer interrupt would fire and wake-up the system.
Suspending the timer interrupts consists of disabling physical EL1
and EL2 timers. The resume consists only of raising timer softirq,
which will trigger the generic timer code to reprogram the EL2 timer
as needed. Enabling of EL1 physical timer will be triggered by an
entity which uses it.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c |  4 
 xen/arch/arm/time.c| 22 ++
 xen/include/asm-arm/time.h |  3 +++
 3 files changed, 29 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index b7940fe03f..d1b48c339a 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -131,6 +131,8 @@ static long system_suspend(void *data)
 goto resume_nonboot_cpus;
 }
 
+time_suspend();
+
 local_irq_save(flags);
 status = gic_suspend();
 if ( status )
@@ -146,6 +148,8 @@ static long system_suspend(void *data)
 resume_irqs:
 local_irq_restore(flags);
 
+time_resume();
+
 resume_nonboot_cpus:
 rcu_barrier();
 enable_nonboot_cpus();
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index bbccee742e..aaefcb6441 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -357,6 +357,28 @@ void domain_set_time_offset(struct domain *d, int64_t 
time_offset_seconds)
 /* XXX update guest visible wallclock time */
 }
 
+void time_suspend(void)
+{
+/* Disable physical EL1 timer */
+WRITE_SYSREG32(0, CNTP_CTL_EL0);
+
+/* Disable hypervisor's timer */
+WRITE_SYSREG32(0, CNTHP_CTL_EL2);
+isb();
+}
+
+void time_resume(void)
+{
+/*
+ * Raising timer softirq will trigger generic timer code to reprogram_timer
+ * with the correct timeout value (which is not known here). There is no
+ * need to do anything else in order to recover the time keeping from power
+ * down, because the system counter is not affected by the power down (it
+ * resides out of the ARM's cluster in an always-on part of the SoC).
+ */
+raise_softirq(TIMER_SOFTIRQ);
+}
+
 static int cpu_time_callback(struct notifier_block *nfb,
  unsigned long action,
  void *hcpu)
diff --git a/xen/include/asm-arm/time.h b/xen/include/asm-arm/time.h
index 19a4515e72..9275b39a4a 100644
--- a/xen/include/asm-arm/time.h
+++ b/xen/include/asm-arm/time.h
@@ -42,6 +42,9 @@ extern uint64_t ns_to_ticks(s_time_t ns);
 
 void preinit_xen_time(void);
 
+void time_suspend(void);
+void time_resume(void);
+
 #endif /* __ARM_TIME_H__ */
 /*
  * Local variables:
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 16/18] xen/arm: Save/restore context on suspend/resume

[Mirela Simonovic]

The context of CPU general purpose and system control registers
has to be saved on suspend and restored on resume. This is
implemented in hyp_suspend and before the return from hyp_resume
function. The hyp_suspend is invoked just before the PSCI system
suspend call is issued to the ATF. The hyp_suspend has to return a
non-zero value so that the calling 'if' statement evaluates to true,
causing the system suspend to be invoked. Upon the resume, context
saved on suspend will be restored, including the link register.
Therefore, after restoring the context the control flow will
return to the address pointed by the saved link register, which
is the place from which the hyp_suspend was called. To ensure
that the calling 'if' statement doesn't again evaluate to true
and initiate system suspend, hyp_resume has to return a zero value
after restoring the context.
Note that the order of saving register context into cpu_context
structure has to match the order of restoring.
Since the suspend/resume is supported only for arm64, we define
a null cpu_context structure so arm32 could compile.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/arm64/entry.S| 89 ++-
 xen/arch/arm/suspend.c| 28 --
 xen/include/asm-arm/suspend.h | 22 +++
 3 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 5efa30e8ee..861a401b18 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -534,6 +534,52 @@ ENTRY(__context_switch)
 mov sp, x9
 ret
 
+/*
+ * void hyp_suspend(struct cpu_context *ptr)
+ *
+ * x0 - pointer to the storage where callee's context will be saved
+ *
+ * CPU context saved here will be restored on resume in hyp_resume function.
+ * hyp_suspend shall return a non-zero value. Upon restoring context
+ * hyp_resume shall return value zero instead. From C code that invokes
+ * hyp_suspend, the return value is interpreted to determine whether the 
context
+ * is saved (hyp_suspend) or restored (hyp_resume).
+ */
+ENTRY(hyp_suspend)
+/* Store callee-saved registers */
+stp x19, x20, [x0], #16
+stp x21, x22, [x0], #16
+stp x23, x24, [x0], #16
+stp x25, x26, [x0], #16
+stp x27, x28, [x0], #16
+stp x29, lr, [x0], #16
+
+/* Store stack-pointer */
+mov x2, sp
+str x2, [x0], #8
+
+/* Store system control registers */
+mrs x2, VBAR_EL2
+str x2, [x0], #8
+mrs x2, VTCR_EL2
+str x2, [x0], #8
+mrs x2, VTTBR_EL2
+str x2, [x0], #8
+mrs x2, TPIDR_EL2
+str x2, [x0], #8
+mrs x2, MDCR_EL2
+str x2, [x0], #8
+mrs x2, HSTR_EL2
+str x2, [x0], #8
+mrs x2, CPTR_EL2
+str x2, [x0], #8
+mrs x2, HCR_EL2
+str x2, [x0], #8
+
+/* hyp_suspend must return a non-zero value */
+mov x0, #1
+ret
+
 ENTRY(hyp_resume)
 msr   DAIFSet, 0xf   /* Disable all interrupts */
 
@@ -622,7 +668,48 @@ mmu_resumed:
 tlbi  alle2
 dsb   sy /* Ensure completion of TLB flush */
 isb
-b .
+
+/* Now we can access the cpu_context, so restore the context here */
+ldr x0, =cpu_context
+
+/* Restore callee-saved registers */
+ldp x19, x20, [x0], #16
+ldp x21, x22, [x0], #16
+ldp x23, x24, [x0], #16
+ldp x25, x26, [x0], #16
+ldp x27, x28, [x0], #16
+ldp x29, lr, [x0], #16
+
+/* Restore stack pointer */
+ldr x2, [x0], #8
+mov sp, x2
+
+/* Restore system control registers */
+ldr x2, [x0], #8
+msr VBAR_EL2, x2
+ldr x2, [x0], #8
+msr VTCR_EL2, x2
+ldr x2, [x0], #8
+msr VTTBR_EL2, x2
+ldr x2, [x0], #8
+msr TPIDR_EL2, x2
+ldr x2, [x0], #8
+msr MDCR_EL2, x2
+ldr x2, [x0], #8
+msr HSTR_EL2, x2
+ldr x2, [x0], #8
+msr CPTR_EL2, x2
+ldr x2, [x0], #8
+msr HCR_EL2, x2
+isb
+
+/* Since context is restored return from this function will appear as
+ * return from hyp_suspend. To distinguish a return from hyp_suspend
+ * which is called upon finalizing the suspend, as opposed to return
+ * from this function which executes on resume, we need to return zero
+ * value here. */
+mov x0, #0
+ret
 
 /*
  * Local variables:
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 365c32ec3c..423d9c0e90 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -3,6 +3,9 @@
 #include

[Xen-devel] [PATCH 14/18] xen/arm: Convert setting MMU page tables code into a routine

[Mirela Simonovic]

From: Saeed Nowshadi 

The code that sets up MMU page tables during the boot is also
needed when the system resumes. Convert that code in head.S
into a routine so the resume code can use it as well. This patch
does not include any functional change.

Signed-off-by: Saeed Nowshadi 
Signed-off-by: Mirela Simonovic 
---
 xen/arch/arm/arm64/head.S | 265 --
 1 file changed, 138 insertions(+), 127 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index ef87b5c254..f95390dcfe 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -379,134 +379,10 @@ skip_bss:
  * than SP_EL0. */
 msr spsel, #1
 
-/* Rebuild the boot pagetable's first-level entries. The structure
- * is described in mm.c.
- *
- * After the CPU enables paging it will add the fixmap mapping
- * to these page tables, however this may clash with the 1:1
- * mapping. So each CPU must rebuild the page tables here with
- * the 1:1 in place. */
+/* If setting up page_tables are not successful, fail to boot */
+blsetup_page_tables
+cbz   x25, fail
 
-/* If Xen is loaded at exactly XEN_VIRT_START then we don't
- * need an additional 1:1 mapping, the virtual mapping will
- * suffice.
- */
-cmp   x19, #XEN_VIRT_START
-cset  x25, eq/* x25 := identity map in place, or not */
-
-/* Write Xen's PT's paddr into TTBR0_EL2 */
-load_paddr x4, boot_pgtable
-msr   TTBR0_EL2, x4
-
-/* Setup boot_pgtable: */
-load_paddr x1, boot_first
-
-/* ... map boot_first in boot_pgtable[0] */
-mov   x3, #PT_PT /* x2 := table map of boot_first */
-orr   x2, x1, x3 /*   + rights for linear PT */
-str   x2, [x4, #0]   /* Map it in slot 0 */
-
-/* ... map of paddr(start) in boot_pgtable+boot_first_id */
-lsr   x1, x19, #ZEROETH_SHIFT/* Offset of base paddr in boot_pgtable */
-cbz   x1, 1f /* It's in slot 0, map in boot_first
-  * or boot_second later on */
-
-/* Level zero does not support superpage mappings, so we have
- * to use an extra first level page in which we create a 1GB mapping.
- */
-load_paddr x2, boot_first_id
-
-mov   x3, #PT_PT /* x2 := table map of boot_first_id */
-orr   x2, x2, x3 /*   + rights for linear PT */
-lsl   x1, x1, #3 /* x1 := Slot offset */
-str   x2, [x4, x1]
-
-load_paddr x4, boot_first_id
-
-lsr   x1, x19, #FIRST_SHIFT  /* x1 := Offset of base paddr in 
boot_first_id */
-lsl   x2, x1, #FIRST_SHIFT   /* x2 := Base address for 1GB mapping */
-mov   x3, #PT_MEM/* x2 := Section map */
-orr   x2, x2, x3
-and   x1, x1, #LPAE_ENTRY_MASK /* x1 := Slot offset */
-lsl   x1, x1, #3
-str   x2, [x4, x1]   /* Mapping of paddr(start) */
-mov   x25, #1/* x25 := identity map now in place */
-
-1:  /* Setup boot_first: */
-load_paddr x4, boot_first   /* Next level into boot_first */
-
-/* ... map boot_second in boot_first[0] */
-load_paddr x1, boot_second
-mov   x3, #PT_PT /* x2 := table map of boot_second */
-orr   x2, x1, x3 /*   + rights for linear PT */
-str   x2, [x4, #0]   /* Map it in slot 0 */
-
-/* ... map of paddr(start) in boot_first */
-cbnz  x25, 1f/* x25 is set if already created */
-lsr   x2, x19, #FIRST_SHIFT  /* x2 := Offset of base paddr in 
boot_first */
-and   x1, x2, #LPAE_ENTRY_MASK /* x1 := Slot to use */
-cbz   x1, 1f /* It's in slot 0, map in boot_second */
-
-lsl   x2, x2, #FIRST_SHIFT   /* Base address for 1GB mapping */
-mov   x3, #PT_MEM/* x2 := Section map */
-orr   x2, x2, x3
-lsl   x1, x1, #3 /* x1 := Slot offset */
-str   x2, [x4, x1]   /* Create mapping of paddr(start)*/
-mov   x25, #1/* x25 := identity map now in place */
-
-1:  /* Setup boot_second: */
-load_paddr x4, boot_second
-
-/* ... map boot_third in boot_second[1] */
-load_paddr x1, boot_third
-mov   x3, #PT_PT /* x2 := table map of boot_third */
-orr   x2, x1, x3 /*   + rights for linear PT */
-str   x2, [x4, #8]   /* Map it in slot 1 */
-
-/* ... map of paddr(start) in boot_second */
-cbnz  x25, 1f/* x25 is set if already created */
-lsr   x2, x19, #SECOND_SHIFT /* x2 := Offset of base paddr in 
boot_second */
-and   x1, x2, #LPAE_ENTRY_MASK /* x1

[Xen-devel] [PATCH 05/18] xen/arm: Trigger Xen suspend when Dom0 completes suspend

[Mirela Simonovic]

When Dom0 finalizes its suspend procedure the suspend of Xen is
triggered by calling system_suspend(). Dom0 finalizes the suspend from
its boot core (VCPU#0), which could be mapped to any physical CPU,
i.e. the system_suspend() function could be executed by any physical
CPU. Since Xen suspend procedure has to be run by the boot CPU
(non-boot CPUs will be disabled at some point in suspend procedure),
system_suspend() execution has to continue on CPU#0.

When the system_suspend() returns 0, it means that the system was
suspended and it is coming out of the resume procedure. Regardless
of the system_suspend() return value, after this function returns
Xen is fully functional, and its state, including all devices and data
structures, matches the state prior to calling system_suspend().
The status is returned by system_suspend() for debugging/logging
purposes and function prototype compatibility.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index f2338e41db..21b45f8248 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -112,11 +112,20 @@ static void vcpu_suspend(register_t epoint, register_t 
cid)
 _arch_set_info_guest(v, );
 }
 
+/* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
+static long system_suspend(void *data)
+{
+BUG_ON(system_state != SYS_STATE_active);
+
+return -ENOSYS;
+}
+
 int32_t domain_suspend(register_t epoint, register_t cid)
 {
 struct vcpu *v;
 struct domain *d = current->domain;
 bool is_thumb = epoint & 1;
+int status;
 
 dprintk(XENLOG_DEBUG,
 "Dom%d suspend: epoint=0x%"PRIregister", cid=0x%"PRIregister"\n",
@@ -156,6 +165,31 @@ int32_t domain_suspend(register_t epoint, register_t cid)
  */
 vcpu_block_unless_event_pending(current);
 
+/* If this was dom0 the whole system should suspend: trigger Xen suspend */
+if ( is_hardware_domain(d) )
+{
+/*
+ * system_suspend should be called when Dom0 finalizes the suspend
+ * procedure from its boot core (VCPU#0). However, Dom0's VCPU#0 could
+ * be mapped to any PCPU (this function could be executed by any PCPU).
+ * The suspend procedure has to be finalized by the PCPU#0 (non-boot
+ * PCPUs will be disabled during the suspend).
+ */
+status = continue_hypercall_on_cpu(0, system_suspend, NULL);
+/*
+ * If an error happened, there is nothing that needs to be done here
+ * because the system_suspend always returns in fully functional state
+ * no matter what the outcome of suspend procedure is. If the system
+ * suspended successfully the function will return 0 after the resume.
+ * Otherwise, if an error is returned it means Xen did not suspended,
+ * but it is still in the same state as if the system_suspend was never
+ * called. We dump a debug message in case of an error for debugging/
+ * logging purpose.
+ */
+if ( status )
+dprintk(XENLOG_ERR, "Failed to suspend, errno=%d\n", status);
+}
+
 return PSCI_SUCCESS;
 }
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 15/18] xen/arm: Resume memory management on Xen resume

[Mirela Simonovic]

The MMU needs to be enabled in the resume flow before the context
can be restored (we need to be able to access the context data by
virtual address in order to restore it). The configuration of system
registers prior to branching to the routine that sets up the page
tables is copied from xen/arch/arm/arm64/head.S.
After the MMU is enabled, the content of TTBR0_EL2 is changed to
point to init_ttbr (page tables used at runtime).

At boot the init_ttbr variable is updated when a secondary CPU is
hotplugged. In the scenario where there is only one physical CPU in
the system, the init_ttbr would not be initialized for the use in
resume flow. To get the variable initialized in all scenarios in this
patch we add that the boot CPU updates init_ttbr after it sets the
page tables for runtime.

After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2
has to be set in order to configure that a mapping cannot be both
writable and executable (this was configured prior to suspend).
This is done using an existing function (mmu_init_secondary_cpu).

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 

---
Changes in v2:

- Patch from v1:
"[XEN PATCH 17/21] xen/arm: Set SCTLR_WXN in SCTLR_EL2 on resume"
is squashed with this patch, because it is indeed related to resuming
the memory management
- Since the original patch was named 'Enable the MMU', and this is
not only enabling anymore, but the full resume of functionality, the
commit title and message is fixed
---
 xen/arch/arm/arm64/entry.S | 88 ++
 xen/arch/arm/mm.c  |  1 +
 xen/arch/arm/suspend.c |  6 
 3 files changed, 95 insertions(+)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index dbc4717903..5efa30e8ee 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -1,6 +1,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -534,6 +535,93 @@ ENTRY(__context_switch)
 ret
 
 ENTRY(hyp_resume)
+msr   DAIFSet, 0xf   /* Disable all interrupts */
+
+tlbi  alle2
+dsb   sy /* Ensure completion of TLB flush */
+isb
+
+ldr   x0, =start
+adr   x19, start /* x19 := paddr (start) */
+sub   x20, x19, x0   /* x20 := phys-offset */
+
+/*  call PROCINFO_cpu_init here */
+
+/* Set up memory attribute type tables */
+ldr   x0, =MAIRVAL
+msr   mair_el2, x0
+
+/* Set up TCR_EL2:
+ * PS -- Based on ID_AA64MMFR0_EL1.PARange
+ * Top byte is used
+ * PT walks use Inner-Shareable accesses,
+ * PT walks are write-back, write-allocate in both cache levels,
+ * 48-bit virtual address space goes through this table. */
+ldr   x0, 
=(TCR_RES1|TCR_SH0_IS|TCR_ORGN0_WBWA|TCR_IRGN0_WBWA|TCR_T0SZ(64-48))
+/* ID_AA64MMFR0_EL1[3:0] (PARange) corresponds to TCR_EL2[18:16] (PS) 
*/
+mrs   x1, ID_AA64MMFR0_EL1
+bfi   x0, x1, #16, #3
+
+msr   tcr_el2, x0
+
+/* Set up the SCTLR_EL2:
+ * Exceptions in LE ARM,
+ * Low-latency IRQs disabled,
+ * Write-implies-XN disabled (for now),
+ * D-cache disabled (for now),
+ * I-cache enabled,
+ * Alignment checking disabled,
+ * MMU translation disabled (for now). */
+ldr   x0, =(HSCTLR_BASE)
+msr   SCTLR_EL2, x0
+
+/* Ensure that any exceptions encountered at EL2
+ * are handled using the EL2 stack pointer, rather
+ * than SP_EL0. */
+msr spsel, #1
+
+/* Rebuild the boot pagetable's first-level entries. The structure
+ * is described in mm.c.
+ *
+ * After the CPU enables paging it will add the fixmap mapping
+ * to these page tables, however this may clash with the 1:1
+ * mapping. So each CPU must rebuild the page tables here with
+ * the 1:1 in place. */
+
+/* If Xen is loaded at exactly XEN_VIRT_START then we don't
+ * need an additional 1:1 mapping, the virtual mapping will
+ * suffice.
+ */
+cmp   x19, #XEN_VIRT_START
+cset  x25, eq/* x25 := identity map in place, or not */
+
+/* Write Xen's PT's paddr into TTBR0_EL2 */
+ldr   x4, =boot_pgtable /* xen_pgtable*/
+add   x4, x4, x20   /* x4 := paddr (boot_pagetable) */
+msr   TTBR0_EL2, x4
+
+/* Set up page tables */
+blsetup_page_tables
+
+ldr   x1, =mmu_resumed  /* Explicit vaddr, not RIP-relative */
+mrs   x0, SCTLR_EL2
+orr   x0, x0, #SCTLR_M  /* Enable MMU */
+orr   x0, x0, #SCTLR_C  /* Enable D-cache */
+dsb   sy/* Flush PTE writes and finish reads */
+msr   SCTLR_EL2, x0 /* now paging is enabled */
+isb 

[Xen-devel] [PATCH 17/18] xen/arm: Resume Dom0 after Xen resumes

[Mirela Simonovic]

The resume of Dom0 should always follow Xen's resume. This is
done by unblocking the first vCPU of Dom0.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 423d9c0e90..a05aea9c25 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -189,6 +189,9 @@ resume_nonboot_cpus:
 system_state = SYS_STATE_active;
 dsb(sy);
 
+/* Wake-up hardware domain (should always resume after Xen) */
+vcpu_unblock(hardware_domain->vcpu[0]);
+
 return status;
 }
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 11/18] xen/arm: Suspend/resume GIC on system suspend/resume

[Mirela Simonovic]

GIC state is saved on system suspend by calling gic_suspend
(this function does not change current state of the GIC but only
saves the values of configuration registers).
The state of GIC has to be restored by calling gic_resume, but only
if the gic_suspend has succeeded. If gic_suspend fails, we'll just
restore interrupts configuration and abort suspend.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 8e8e531d61..b7940fe03f 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -117,6 +117,7 @@ static void vcpu_suspend(register_t epoint, register_t cid)
 static long system_suspend(void *data)
 {
 int status;
+unsigned long flags;
 
 BUG_ON(system_state != SYS_STATE_active);
 
@@ -130,8 +131,21 @@ static long system_suspend(void *data)
 goto resume_nonboot_cpus;
 }
 
+local_irq_save(flags);
+status = gic_suspend();
+if ( status )
+{
+system_state = SYS_STATE_resume;
+goto resume_irqs;
+}
+
 system_state = SYS_STATE_resume;
 
+gic_resume();
+
+resume_irqs:
+local_irq_restore(flags);
+
 resume_nonboot_cpus:
 rcu_barrier();
 enable_nonboot_cpus();
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 08/18] xen/arm: Disable/enable non-boot physical CPUs on suspend/resume

[Mirela Simonovic]

Non-boot CPUs have to be disabled on suspend and enabled on resume
(hotplug-based mechanism). Disabling non-boot CPUs will lead to PSCI
CPU_OFF to be called by each non-boot CPU. Depending on the underlying
platform capabilities, this may lead to the physical powering down of
CPUs. Tested on Xilinx Zynq Ultrascale+ MPSoC (including power down of
each non-boot CPU).

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 575afd5eb8..dae1b1f7d6 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -1,4 +1,5 @@
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -115,17 +116,29 @@ static void vcpu_suspend(register_t epoint, register_t 
cid)
 /* Xen suspend. Note: data is not used (suspend is the suspend to RAM) */
 static long system_suspend(void *data)
 {
+int status;
+
 BUG_ON(system_state != SYS_STATE_active);
 
 system_state = SYS_STATE_suspend;
 freeze_domains();
 
+status = disable_nonboot_cpus();
+if ( status )
+{
+system_state = SYS_STATE_resume;
+goto resume_nonboot_cpus;
+}
+
 system_state = SYS_STATE_resume;
 
+resume_nonboot_cpus:
+enable_nonboot_cpus();
 thaw_domains();
 system_state = SYS_STATE_active;
+dsb(sy);
 
-return -ENOSYS;
+return status;
 }
 
 int32_t domain_suspend(register_t epoint, register_t cid)
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 01/18] xen/arm: Move code that initializes VCPU context into a separate function

[Mirela Simonovic]

From: Saeed Nowshadi 

The arch_set_info_guest() has code to initialize the context of a VCPU.
When a VCPU is resumed it needs to go through the same context
initialization excluding all the validations that this routine does.
We move the actual VCPU context setting into a function so that it can be
shared with the resume path.

Signed-off-by: Saeed Nowshadi 
Signed-off-by: Mirela Simonovic 
---
 xen/arch/arm/domain.c| 34 +-
 xen/include/xen/domain.h |  1 +
 2 files changed, 22 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 80432872d6..e594b48d81 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -781,6 +781,26 @@ static int is_guest_pv64_psr(uint32_t psr)
 #endif
 
 /*
+ * The actual VCPU initialization after all validations are passed.
+ */
+void _arch_set_info_guest(struct vcpu *v, struct vcpu_guest_context *ctxt)
+{
+vcpu_regs_user_to_hyp(v, >user_regs);
+
+v->arch.sctlr = ctxt->sctlr;
+v->arch.ttbr0 = ctxt->ttbr0;
+v->arch.ttbr1 = ctxt->ttbr1;
+v->arch.ttbcr = ctxt->ttbcr;
+
+v->is_initialised = 1;
+
+if ( ctxt->flags & VGCF_online )
+clear_bit(_VPF_down, >pause_flags);
+else
+set_bit(_VPF_down, >pause_flags);
+}
+
+/*
  * Initialise VCPU state. The context can be supplied by either the
  * toolstack (XEN_DOMCTL_setvcpucontext) or the guest
  * (VCPUOP_initialise) and therefore must be properly validated.
@@ -818,19 +838,7 @@ int arch_set_info_guest(
 }
 #endif
 
-vcpu_regs_user_to_hyp(v, regs);
-
-v->arch.sctlr = ctxt->sctlr;
-v->arch.ttbr0 = ctxt->ttbr0;
-v->arch.ttbr1 = ctxt->ttbr1;
-v->arch.ttbcr = ctxt->ttbcr;
-
-v->is_initialised = 1;
-
-if ( ctxt->flags & VGCF_online )
-clear_bit(_VPF_down, >pause_flags);
-else
-set_bit(_VPF_down, >pause_flags);
+_arch_set_info_guest(v, ctxt);
 
 return 0;
 }
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 33e41486cb..904624e070 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -73,6 +73,7 @@ int arch_domain_soft_reset(struct domain *d);
 void arch_p2m_set_access_required(struct domain *d, bool access_required);
 
 int arch_set_info_guest(struct vcpu *, vcpu_guest_context_u);
+void _arch_set_info_guest(struct vcpu *, struct vcpu_guest_context *);
 void arch_get_info_guest(struct vcpu *, vcpu_guest_context_u);
 
 int arch_initialise_vcpu(struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 07/18] xen/arm: Freeze domains on suspend and thaw them on resume

[Mirela Simonovic]

Freeze and thaw of domains is reused as implemented for x86. In
addition, system_state variable is updated to represent the actual
state of the system.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 21b45f8248..575afd5eb8 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -117,6 +117,14 @@ static long system_suspend(void *data)
 {
 BUG_ON(system_state != SYS_STATE_active);
 
+system_state = SYS_STATE_suspend;
+freeze_domains();
+
+system_state = SYS_STATE_resume;
+
+thaw_domains();
+system_state = SYS_STATE_active;
+
 return -ENOSYS;
 }
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 09/18] xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume

[Mirela Simonovic]

The rcu_barrier() has to be added to ensure that the per cpu area is
freed before a non-boot CPU tries to initialize it (_free_percpu_area()
has to be called before the init_percpu_area()). This scenario occurs
when non-boot CPUs are hot-unplugged on suspend and hotplugged on resume.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/suspend.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index dae1b1f7d6..8e8e531d61 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -133,6 +133,7 @@ static long system_suspend(void *data)
 system_state = SYS_STATE_resume;
 
 resume_nonboot_cpus:
+rcu_barrier();
 enable_nonboot_cpus();
 thaw_domains();
 system_state = SYS_STATE_active;
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 00/18] xen/arm64: Suspend to RAM support for Xen

[Mirela Simonovic]

This series contains support for suspend to RAM (in the following text just
'suspend') for Xen on arm64. The implementation is aligned with the design
specification that has been proposed on xen-devel list:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

At a high-level the series contains:
1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
2) Support for resuming a guest on an interrupt targeted to that guest
3) Support for suspending Xen after dom0 finalizes the suspend
4) Support for resuming Xen on an interrupt that is targeted to a guest


In more details:

*** About suspending/resuming guests

The patches included in this series allow PSCI compliant guests that have
support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
suspend and resume on top of Xen without any EL1 code changes.

During their suspend procedure guests will hot-unplug their secondary CPUs,
triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
Guests will save/restore their own EL1 context on suspend/resume.

A guest is expected to leave enabled interrupts that are considered to be its
wake-up sources. Those interrupts will be able to wake up the guest. This holds
regardless of the state of the underlying software layers, i.e. whether Xen gets
suspended or not doesn't affect the ability of the guest to wake up.

First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from which
the guest assumes to start on resume. On resume, guests assume to be running in
an environment whose state matches the CPU state after reset, e.g. with reset
register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
VCPU context and save the resume entry point into program counter before the
guest's VCPU gets scheduled in on resume. This is done when the guest finalizes
its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure that
the resume-ready VCPU's context does not get overwritten later upon the context
switch when the VCPU is scheduled out.
Xen also needs to take care that the guest's view of GIC and timer gets saved.
Also, while a guest is suspended its watchdogs are paused, in order to avoid
watchdog triggered shutdown of a guest that has been asleep for a period of time
that is longer than the watchdog period.

After this point, from Xen's point of view a suspended guest has one VCPU
blocked, waiting for an interrupt. When such an interrupt comes, Xen will
unblock the VCPU of the suspended domain, which results in the guest resuming.

*** About suspending/resuming Xen

Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
considered to be the decision maker for EL1 and EL2.
On suspend, Xen will first freeze all domains. Then, Xen disables physical
secondary CPUs, which leads to physical CPU_OFF to be called by each secondary
CPU. After that Xen finalizes the suspend from the boot CPU.

This consists of suspending the timer, i.e. suppressing its interrupts (we don't
want to be woken up by a timer, there is no VCPU ready to be scheduled). Then
the state of GIC is saved, console is suspended, and CPU context is saved. The
saved context tells where Xen needs to continue execution on resume.
Since Xen will resume with MMU disabled, the first thing to do in resume is to
resume memory management in order to be able to access the context that needs to
be restored (we know virtual address of the context data). Finally Xen calls
SYSTEM_SUSPEND PSCI to the EL3.

When resuming, all the steps done in suspend need to be reverted. This is
completed by unblocking dom0's VCPU, because we always want the dom0 to resume,
regardless of the target domain whose interrupt woke up Xen.

*** Handling of unprivileged guests during Xen suspend/resume

Any domU that is not suspended when dom0 suspends will be frozen, domUs that are
already suspended remain suspended. On resume the suspended domUs still remain
suspended (unless their wake interrupt caused Xen to wake) while the others will
be thawed.

For more details please refer to patches or the design specification:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html


The series does not include:
a) UART driver-specific suspend/resume that gets called when console suspends
b) SMMU suspend/resume
c) Suspend coordination support that would allow dom0 to request domUs to
suspend
These will be submitted in the following series.

Mirela Simonovic (16):
  xen/arm: Implement PSCI system suspend call (virtual interface)
  xen/arm: Save GIC and virtual timer context when the domain suspends
  xen/arm: While a domain is suspended put its watchdogs on pause
  xen/arm: Trigger Xen suspend when Dom0 completes suspend
  xen/x86: M

[Xen-devel] [PATCH 02/18] xen/arm: Implement PSCI system suspend call (virtual interface)

[Mirela Simonovic]

The implementation consists of:
-Adding PSCI system suspend call as new PSCI function
-Trapping PSCI system_suspend HVC
-Implementing PSCI system suspend call (virtual interface that allows
 guests to suspend themselves)

The PSCI system suspend should be called by a guest from its boot
VCPU. Non-boot VCPUs of the guest should be hot-unplugged using PSCI
CPU_OFF call prior to issuing PSCI system suspend. Interrupts that
are left enabled by the guest are assumed to be its wake-up interrupts.
Therefore, a wake-up interrupt triggers the resume of the guest. Guest
should resume regardless of the state of Xen (suspended or not).

When a guest calls PSCI system suspend the respective domain will be
suspended if the following conditions are met:
1) Given resume entry point is not invalid
2) Other (if any) VCPUs of the calling guest are hot-unplugged

If the conditions above are met the calling domain is labeled as
suspended and the calling VCPU is blocked. If nothing else wouldn't
be done the suspended domain would resume from the place where it
called PSCI system suspend. This is expected if processing of the PSCI
system suspend call fails. However, in the case of success the calling
guest should resume (continue execution after the wake-up) from the entry
point which is given as the first argument of the PSCI system suspend
call. In addition to the entry point, the guest expects to start within
the environment whose state matches the state after reset. This means
that the guest should find reset register values, MMU disabled, etc.
Thereby, the context of VCPU should be 'reset' (as if the system is
comming out of reset), the program counter should contain entry point,
which is 1st argument, and r0/x0 should contain context ID which is 2nd
argument of PSCI system suspend call. The context of VCPU is set
accordingly when the PSCI system suspend is processed, so that nothing
needs to be done on resume/wake-up path. However, in order to ensure that
this context doesn't get overwritten by the scheduler when scheduling out
this VCPU (would normally happen after the calling CPU is blocked), we need
to check whether to return early from ctxt_switch_from().

There are variables in domain structure to keep track of domain shutdown.
One of existing shutdown reason is 'suspend' which this patch is using to
track the suspend state of a domain. Those variables are used to determine
whether to early return from ctxt_switch_from() or not.

A suspended domain will resume after the Xen receives an interrupt which is
targeted to the domain, unblocks the domain's VCPU, and schedules it in.
When the VCPU is scheduled in, the VCPU context is already reset, and
contains the right resume entry point in program counter that will be
restored in ctxt_switch_to(). The only thing that needs to be done at this
point is to clear the variables that marked the domain state as suspended.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 

---
Changes in v2:

-Fix print to compile for arm32 and to align with Xen coding style
---
 xen/arch/arm/Makefile|   1 +
 xen/arch/arm/domain.c|  13 +++
 xen/arch/arm/suspend.c   | 166 +++
 xen/arch/arm/vpsci.c |  19 +
 xen/include/asm-arm/perfc_defn.h |   1 +
 xen/include/asm-arm/psci.h   |   2 +
 xen/include/asm-arm/suspend.h|  16 
 xen/include/xen/sched.h  |   1 +
 8 files changed, 219 insertions(+)
 create mode 100644 xen/arch/arm/suspend.c
 create mode 100644 xen/include/asm-arm/suspend.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 23c5d9adbc..744b1a4dc8 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -43,6 +43,7 @@ obj-y += setup.o
 obj-y += shutdown.o
 obj-y += smp.o
 obj-y += smpboot.o
+obj-y += suspend.o
 obj-y += sysctl.o
 obj-y += time.o
 obj-y += traps.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e594b48d81..7f8105465c 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -97,6 +97,11 @@ static void ctxt_switch_from(struct vcpu *p)
 if ( is_idle_vcpu(p) )
 return;
 
+/* VCPU's context should not be saved if its domain is suspended */
+if ( p->domain->is_shut_down &&
+(p->domain->shutdown_code == SHUTDOWN_suspend) )
+return;
+
 p2m_save_state(p);
 
 /* CP 15 */
@@ -181,6 +186,14 @@ static void ctxt_switch_to(struct vcpu *n)
 if ( is_idle_vcpu(n) )
 return;
 
+/* If the domain was suspended, it is resuming now */
+if ( n->domain->is_shut_down &&
+(n->domain->shutdown_code == SHUTDOWN_suspend) )
+{
+n->domain->is_shut_down = 0;
+n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
+}
+
 p2m_restore_state(n);
 
 vpidr = READ_SYSREG32(MIDR_EL1);
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
new file mode 100644
index 00..9eea9214e1
--- /

[Xen-devel] [PATCH 06/18] xen/x86: Move freeze/thaw_domains into common files

[Mirela Simonovic]

These functions will be reused by suspend/resume support for ARM.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/x86/acpi/power.c | 28 
 xen/common/domain.c   | 29 +
 xen/include/xen/sched.h   |  3 +++
 3 files changed, 32 insertions(+), 28 deletions(-)

diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
index 93e967fe8f..794750e45b 100644
--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -109,34 +109,6 @@ static void device_power_up(enum dev_power_saved saved)
 }
 }
 
-static void freeze_domains(void)
-{
-struct domain *d;
-
-rcu_read_lock(_read_lock);
-/*
- * Note that we iterate in order of domain-id. Hence we will pause dom0
- * first which is required for correctness (as only dom0 can add domains to
- * the domain list). Otherwise we could miss concurrently-created domains.
- */
-for_each_domain ( d )
-domain_pause(d);
-rcu_read_unlock(_read_lock);
-}
-
-static void thaw_domains(void)
-{
-struct domain *d;
-
-rcu_read_lock(_read_lock);
-for_each_domain ( d )
-{
-restore_vcpu_affinity(d);
-domain_unpause(d);
-}
-rcu_read_unlock(_read_lock);
-}
-
 static void acpi_sleep_prepare(u32 state)
 {
 void *wakeup_vector_va;
diff --git a/xen/common/domain.c b/xen/common/domain.c
index d6650f0656..fdd00dc661 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -1666,6 +1666,35 @@ int continue_hypercall_on_cpu(
 return 0;
 }
 
+
+void freeze_domains(void)
+{
+struct domain *d;
+
+rcu_read_lock(_read_lock);
+/*
+ * Note that we iterate in order of domain-id. Hence we will pause dom0
+ * first which is required for correctness (as only dom0 can add domains to
+ * the domain list). Otherwise we could miss concurrently-created domains.
+ */
+for_each_domain ( d )
+domain_pause(d);
+rcu_read_unlock(_read_lock);
+}
+
+void thaw_domains(void)
+{
+struct domain *d;
+
+rcu_read_lock(_read_lock);
+for_each_domain ( d )
+{
+restore_vcpu_affinity(d);
+domain_unpause(d);
+}
+rcu_read_unlock(_read_lock);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 366acaf69a..c7a6d9504a 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -821,6 +821,9 @@ static inline int 
domain_pause_by_systemcontroller_nosync(struct domain *d)
 void domain_pause_except_self(struct domain *d);
 void domain_unpause_except_self(struct domain *d);
 
+void freeze_domains(void);
+void thaw_domains(void);
+
 void cpu_init(void);
 
 struct scheduler;
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 04/18] xen/arm: While a domain is suspended put its watchdogs on pause

[Mirela Simonovic]

While a domain is suspended its watchdogs must be paused. Otherwise,
if the domain stays in the suspend state for a longer period of time
compared to the watchdog period, the domain would be shutdown on resume.
Proper solution to this problem is to stop (suspend) the watchdog timers
after the domain suspends and to restart (resume) the watchdog timers
before the domain resumes. The suspend/resume of watchdog timers is done
in Xen and is invisible to the guests.
Just before the domain starts resuming the watchdog timers are programmed
with a new expire value. The new expire value is equal to the expire
value prior to suspend plus the period of time for which the watchdog
was on pause (the domain was suspended). In order to save the suspend
timestamp and afterwards calculate for how long the domain was suspended,
a 'suspended' variable is added into the generic timer structure.
Programming of the timers is triggered when a VCPU of the suspended
domain is scheduled in on resume.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 

---
Changes in v2:

-Fixed typo in commit message
---
 xen/arch/arm/domain.c   |  1 +
 xen/arch/arm/suspend.c  |  3 +++
 xen/common/schedule.c   | 38 ++
 xen/include/xen/sched.h |  7 +++
 xen/include/xen/timer.h |  3 +++
 5 files changed, 52 insertions(+)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index bebe3238e8..68f038458f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -192,6 +192,7 @@ static void ctxt_switch_to(struct vcpu *n)
 {
 n->domain->is_shut_down = 0;
 n->domain->shutdown_code = SHUTDOWN_CODE_INVALID;
+watchdog_domain_resume(n->domain);
 }
 
 p2m_restore_state(n);
diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c
index 9eea9214e1..f2338e41db 100644
--- a/xen/arch/arm/suspend.c
+++ b/xen/arch/arm/suspend.c
@@ -146,6 +146,9 @@ int32_t domain_suspend(register_t epoint, register_t cid)
 d->is_shut_down = 1;
 d->shutdown_code = SHUTDOWN_suspend;
 
+/* Disable watchdogs of this domain */
+watchdog_domain_suspend(d);
+
 /*
  * The calling domain is suspended by blocking its last running VCPU. If an
  * event is pending the domain will resume right away (VCPU will not block,
diff --git a/xen/common/schedule.c b/xen/common/schedule.c
index a957c5e57c..9b2882a168 100644
--- a/xen/common/schedule.c
+++ b/xen/common/schedule.c
@@ -1172,6 +1172,44 @@ void watchdog_domain_destroy(struct domain *d)
 kill_timer(>watchdog_timer[i]);
 }
 
+void watchdog_domain_suspend(struct domain *d)
+{
+unsigned int i;
+
+spin_lock(>watchdog_lock);
+
+for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+{
+if ( test_bit(i, >watchdog_inuse_map) )
+{
+struct timer *timer = >watchdog_timer[i];
+timer->suspended = NOW();
+stop_timer(timer);
+}
+}
+
+spin_unlock(>watchdog_lock);
+}
+
+void watchdog_domain_resume(struct domain *d)
+{
+unsigned int i;
+
+spin_lock(>watchdog_lock);
+
+for ( i = 0; i < NR_DOMAIN_WATCHDOG_TIMERS; i++ )
+{
+if ( test_bit(i, >watchdog_inuse_map) )
+{
+struct timer *timer = >watchdog_timer[i];
+s_time_t sleep_interval = NOW() - timer->suspended;
+set_timer(timer, timer->expires + sleep_interval);
+}
+}
+
+spin_unlock(>watchdog_lock);
+}
+
 int vcpu_pin_override(struct vcpu *v, int cpu)
 {
 spinlock_t *lock;
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 1f4e86524f..366acaf69a 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -862,6 +862,13 @@ void watchdog_domain_init(struct domain *d);
 void watchdog_domain_destroy(struct domain *d);
 
 /*
+ * Suspend/resume watchdogs of domain (while the domain is suspended its
+ * watchdogs should be on pause)
+ */
+void watchdog_domain_suspend(struct domain *d);
+void watchdog_domain_resume(struct domain *d);
+
+/*
  * Use this check when the following are both true:
  *  - Using this feature or interface requires full access to the hardware
  *(that is, this would not be suitable for a driver domain)
diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
index 4513260b0d..4a88af7de0 100644
--- a/xen/include/xen/timer.h
+++ b/xen/include/xen/timer.h
@@ -18,6 +18,9 @@ struct timer {
 /* System time expiry value (nanoseconds since boot). */
 s_time_t expires;
 
+/* Suspend timestamp value (nanoseconds since boot). */
+s_time_t suspended;
+
 /* Position in active-timer data structure. */
 union {
 /* Timer-heap offset (TIMER_STATUS_in_heap). */
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 03/18] xen/arm: Save GIC and virtual timer context when the domain suspends

[Mirela Simonovic]

GIC and virtual timer context must be saved when the domain suspends.
This is done by moving the respective code in ctxt_switch_from()
before the return that happens if the domain suspended.

Signed-off-by: Mirela Simonovic 
Signed-off-by: Saeed Nowshadi 
---
 xen/arch/arm/domain.c | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 7f8105465c..bebe3238e8 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -97,6 +97,13 @@ static void ctxt_switch_from(struct vcpu *p)
 if ( is_idle_vcpu(p) )
 return;
 
+/* VGIC */
+gic_save_state(p);
+
+/* Arch timer */
+p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
+virt_timer_save(p);
+
 /* VCPU's context should not be saved if its domain is suspended */
 if ( p->domain->is_shut_down &&
 (p->domain->shutdown_code == SHUTDOWN_suspend) )
@@ -115,10 +122,6 @@ static void ctxt_switch_from(struct vcpu *p)
 p->arch.tpidrro_el0 = READ_SYSREG(TPIDRRO_EL0);
 p->arch.tpidr_el1 = READ_SYSREG(TPIDR_EL1);
 
-/* Arch timer */
-p->arch.cntkctl = READ_SYSREG32(CNTKCTL_EL1);
-virt_timer_save(p);
-
 if ( is_32bit_domain(p->domain) && cpu_has_thumbee )
 {
 p->arch.teecr = READ_SYSREG32(TEECR32_EL1);
@@ -170,9 +173,6 @@ static void ctxt_switch_from(struct vcpu *p)
 /* VFP */
 vfp_save_state(p);
 
-/* VGIC */
-gic_save_state(p);
-
 isb();
 }
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

Hi Julien,

On Tue, Jun 5, 2018 at 8:00 PM, Julien Grall  wrote:

>
>
> On 05/06/18 18:24, Julien Grall wrote:
>
>> Hi Mirela,
>>
>> On 01/06/18 14:17, Mirela Simonovic wrote:
>>
>>> Linux/dom0 accesses OSLSR register when saving CPU context during the
>>> suspend procedure. Xen traps access to this register, but has no handling
>>> for it. Consequently, Xen injects undef exception to linux, causing it to
>>> crash. This patch adds handling of the trapped access to OSLSR as read
>>> only as a fixed value.
>>>
>>
>> Can you please mention that you introduced handle_ro_read_val() and
>> rework handle_ro_raz()?
>>
>>
>>> Signed-off-by: Mirela Simonovic 
>>> Reviewed-by: Stefano Stabellini 
>>> Acked-by: Julien Grall 
>>>
>>
>> The reviewed-by/acked-by tags should only be kept when there are minor
>> changes in the code (and the reviewer is happy with them). This is very
>> important for the former as the tag means the reviewer read your code and
>> confirm this is correct.
>>
>> As you change quite a bit the patch, those 2 tags should have been
>> removed.
>>
>> Stefano, are you still happy with the Reviewed-by?
>>
>
> Stefano mentioned on IRC he was happy with the reviewed-by kept.
>
> I will go ahead with merging the series. For this patch, I will add the
> following in the commit message:
>
> "For convenience, introduce a new helper handle_ro_read_val() based on
> handle_ro_raz() that will return a specified value on read and re-implement
> handle_ro_raz() based on the new helper".
>
>
Great, thanks!


> Cheers,
>
> --
> Julien Grall
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] ARM: Issues while Enabling hibernation in domU(linux) on jacinto-j6

[Mirela Simonovic]

Hi,

On Wed, Jun 6, 2018 at 8:00 PM, Julien Grall  wrote:
> (+ Stefano, Mirela, Juergen and Boris)
>
> On 06/06/18 09:17, moin anjnawala wrote:
>>
>> Hi Julien,
>
>
> Hi,
>
>> As you specified earlier I am now able to boot using xen4.10. I am
>> using linux4.4 as dom0 as well as domU.In domU, I have enabled
>> hibenation related configs and I am trying suspend to disk.With xen
>> 4.10, hibernation is done sucessfuly but resume crashes.
>
>
> It would have been nice to switch to a more recent Linux as well because I
> don't think anyone worked on suspend/resume 2 years ago. There are effort to
> support suspend/resume in Xen (and I guess guest) but I suspect some work is
> still missing in newer kernel. Although that would be a better start than a
> 2 years old kernel for the guest.
>
>
>> I have used given config file for domU.
>> name = "domU"
>> kernel = "/ZI-uh1"
>> memory = 200
>> vcpus = 1
>> cpus = 1
>> disk = [ 'phy:/rfs1.img,xvda,rw']
>>
>> extra = 'no_console_suspend loglevel=8 console=hvc0 xenconsole=tty
>> earlyprintk=xx
>> en rw dhcp=off resume=/dev/xvda resume_offset=331776 root=/dev/xvda
>> rootwait'
>>
>> The hibernate(domU console) and resume logs are shown as below:
>>
>>
>> root@debian-jessie:~# swapon /swapfile.img
>>
>> swapon: /swapfile.img: insecure permissions 0644, 0600 suggested.
>> [   50.064545] Adding 102396k swap on /swapfile.img.  Priority:-1
>> extents:1 across:102396k SS
>> root@debian-jessie:~#
>> root@debian-jessie:~#
>> root@debian-jessie:~# echo disk > /sys/power/state
>> [   56.649793] PM: Syncing filesystems ... done.
>> [   56.686770] Freezing user space processes ... (elapsed 0.002 seconds)
>> done.
>> [   56.689454] PM: Basic memory bitmaps created
>> [   56.689497] PM: Preallocating image memory... done (allocated 11077
>> pages)
>> [   57.119960] PM: Allocated 44308 kbytes in 0.43 seconds (103.04 MB/s)
>> [   57.120010] Freezing remaining freezable tasks ... (elapsed 0.009
>> seconds) done.
>> [   58.922629] PM: freeze of devices complete after 1791.756 msecs
>> [   58.925167] PM: late freeze of devices complete after 2.485 msecs
>> [   58.927231] PM: noirq freeze of devices complete after 2.010 msecs
>> [   58.927275] Disabling non-boot CPUs ...
>> [   58.927307] PM: Creating hibernation image:
>> [   58.927307] PM: Need to copy 10790 pages
>> [   58.927307] PM: Normal pages needed: 10790 + 1024, available pages:
>> 40398
>> [   58.927307] PM: Hibernation image created (10790 pages copied)
>> [   58.928629] PM: noirq thaw of devices complete after 1.310 msecs
>> [   58.930334] PM: early thaw of devices complete after 1.589 msecs
>> [   60.634498] PM: thaw of devices complete after 1704.114 msecs
>> [   60.635878] PM: writing image.
>> [   60.680038] PM: Using 1 thread(s) for compression.
>> [   60.680038] PM: Compressing and saving image data (10801 pages)...
>> [   60.680170] PM: Image saving progress:   0%
>> [   61.288932] PM: Image saving progress:  10%
>> [   61.809937] PM: Image saving progress:  20%
>> [   61.994594] PM: Image saving progress:  30%
>> [   62.360616] PM: Image saving progress:  40%
>> [   62.583613] random: nonblocking pool is initialized
>> [   62.592295] PM: Image saving progress:  50%
>> [   62.665052] PM: Image saving progress:  60%
>> [   62.700022] PM: Image saving progress:  70%
>> [   62.759685] PM: Image saving progress:  80%
>> [   62.821796] PM: Image saving progress:  90%
>> [   62.854694] PM: Image saving progress: 100%
>> [   62.855175] PM: Image saving done.
>> [   62.855210] PM: Wrote 43204 kbytes in 2.17 seconds (19.90 MB/s)
>> [   62.863308] PM: S|
>> [   62.921029] reboot: Power down
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~#
>> root@Dom0:~# xl create -c /vnc_l.cfg
>> Parsing config from /vnc_l.cfg
>> (XEN) d2v0: vGICD: unhandled word write 0x to ICACTIVER0
>> [0.00] Booting Linux on physical CPU 0x0
>> [0.00] Initializing cgroup subsys cpuset
>> [0.00] Initializing cgroup subsys cpu
>> [0.00] Initializing cgroup subsys cpuacct
>> [0.00] Linux version 4.4.91-2-g89baae6-dirty
>
>
> I don't seem to be able to find that commit in Linux and it also says dirty.
> What are your modification on top of Linux 4.4?
>
>
>> (omkar.bolla@CPU-308U) (gcc version 4.8.4 (Ubuntu/Linaro
>> 4.8.4-2ubuntu1~14.04.1) ) #107 SMP PR
>> EEMPT Tue Jun 5 16:51:21 IST 2018
>> [0.00] CPU: ARMv7 Processor [412fc0f2] revision 2 (ARMv7),
>> cr=50c5387d
>> [0.00] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction
>> cache
>> [0.00] Machine model: XENVM-4.10
>> [0.00] cma: Reserved 24 MiB at 0x4b00
>> [0.00] Memory policy: Data cache writealloc
>> [0.00] On node 0 totalpages: 51200
>> [0.00] free_area_init_node: node 0, pgdat c07eb740,
>> node_mem_map cae2b000
>> [0.00]   Normal zone: 450 pages used for memmap
>> [0.00]   Normal zone: 0 

[Xen-devel] [PATCH v5 06/10] xen/common: Restore IRQ affinity when hotplugging a pCPU

[Mirela Simonovic]

Non-boot pCPUs are being hot-unplugged during the system suspend to
RAM and hotplugged during the resume. When non-boot pCPUs are
hot-unplugged the interrupts that were targeted to them are migrated
to the boot pCPU.
On suspend, each guest could have its own wake-up devices/interrupts
(passthrough) that could trigger the system resume. These interrupts
could be targeted to a non-boot pCPU, e.g. if the guest's vCPU is
pinned to a non-boot pCPU. Due to the hot-unplug of non-boot pCPUs
during the suspend such interrupts will be migrated from non-boot pCPUs
to the boot pCPU (this is fine). However, when non-boot pCPUs are
hotplugged on resume, these interrupts are not migrated back to non-boot
pCPUs, i.e. IRQ affinity is not restored on resume (this is wrong).
This patch adds the restoration of IRQ affinity when a pCPU is hotplugged.

Signed-off-by: Mirela Simonovic 
Reviewed-by: Dario Faggioli 

---
CC: George Dunlap 
CC: Dario Faggioli 
---
Changes in v2:
-Instead of checking whether the affinity was broken check whether
 vcpu's processor has changed in order to trigger restoring of the
 IRQ affinity
-Fix commit message

Changes in v4:
-Added reviewed by Dario
---
 xen/common/schedule.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/xen/common/schedule.c b/xen/common/schedule.c
index 049f93f7aa..ccf936db83 100644
--- a/xen/common/schedule.c
+++ b/xen/common/schedule.c
@@ -737,6 +737,7 @@ void restore_vcpu_affinity(struct domain *d)
 for_each_vcpu ( d, v )
 {
 spinlock_t *lock;
+unsigned int old_cpu = v->processor;
 
 ASSERT(!vcpu_runnable(v));
 
@@ -769,6 +770,9 @@ void restore_vcpu_affinity(struct domain *d)
 lock = vcpu_schedule_lock_irq(v);
 v->processor = SCHED_OP(vcpu_scheduler(v), pick_cpu, v);
 spin_unlock_irq(lock);
+
+if ( old_cpu != v->processor )
+sched_move_irqs(v);
 }
 
 domain_update_node_affinity(d);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 05/10] xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume

[Mirela Simonovic]

In existing code the virtual paging for non-boot CPUs is setup only on boot.
The setup is triggered from start_xen() after all CPUs are brought online.
In other words, the initialization of VTCR_EL2 register is done out of the
cpu_up/start_secondary() control flow. However, the cpu_up flow is also used
to hotplug non-boot CPUs on resume from suspend to RAM state, in which case
the virtual paging will not be configured.

With this patch the setting of paging is triggered from start_secondary()
function using cpu starting notifier (notify_cpu_starting() call). The
notifier is registered in p2m.c using init call. This has to be done with
init call rather than presmp_init because the registered callback depends
on vtcr configuration value which is setup after the presmp init calls
are executed (do_presmp_initcalls() called from start_xen()). Init calls
are executed after initial virtual paging is set up for all CPUs on boot.
This ensures that no callback can fire until the vtcr value is calculated
by Xen and virtual paging is set up initially for all CPUs. Also, this way
the virtual paging setup in boot scenario remains unchanged.

It is assumed here that after the system completed the boot, CPUs that
execute start_secondary() were booted as well when the Xen itself was
booted. According to this assumption non-boot CPUs will always be compliant
with the VTCR_EL2 value that was selected by Xen on boot.
Currently, there is no mechanism to trigger hotplugging of a CPU. This
will be added with the suspend to RAM support for ARM, where the hotplug
of non-boot CPUs will be triggered via enable_nonboot_cpus() call.

Signed-off-by: Mirela Simonovic 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v2:
-Fix commit message
-Save configured VTCR_EL2 value into static variable that will be used
 by non-boot CPUs on hotplug
-Add setup_virt_paging_secondary() and invoke it from start_secondary()
 if that CPU has to setup virtual paging (if the system state is not boot)

Changes in v3:
-Fix commit message
-Remove setup_virt_paging_secondary() and use notifier to setup virtual
 paging for non-boot CPU on hotplug.
-In setup_virt_paging() use vtcr static variable instead of local val
-In setup_virt_paging_one() use vtcr static variable instead of provided
 argument

Changes in v4:
-Add includes alphabetically
-Add newline before return in cpu_virt_paging_init()
-Fix indentation in cpu_virt_paging_callback() definition
-Use local val in setup_virt_paging() for calculation, assign it to vtcr
 after the calculation is done
-Remove priority initialization in the notifier structure (priority
 doesn't matter here)

Changes in v5:
-Define vtcr as uint32_t instead uint64_t
---
 xen/arch/arm/p2m.c | 53 -
 1 file changed, 48 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index d43c3aa896..14791388ad 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -8,6 +8,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -1451,10 +1453,12 @@ err:
 return page;
 }
 
-static void __init setup_virt_paging_one(void *data)
+/* VTCR value to be configured by all CPUs. Set only once by the boot CPU */
+static uint32_t __read_mostly vtcr;
+
+static void setup_virt_paging_one(void *data)
 {
-unsigned long val = (unsigned long)data;
-WRITE_SYSREG32(val, VTCR_EL2);
+WRITE_SYSREG32(vtcr, VTCR_EL2);
 isb();
 }
 
@@ -1538,10 +1542,49 @@ void __init setup_virt_paging(void)
 
 /* It is not allowed to concatenate a level zero root */
 BUG_ON( P2M_ROOT_LEVEL == 0 && P2M_ROOT_ORDER > 0 );
-setup_virt_paging_one((void *)val);
-smp_call_function(setup_virt_paging_one, (void *)val, 1);
+vtcr = val;
+setup_virt_paging_one(NULL);
+smp_call_function(setup_virt_paging_one, NULL, 1);
+}
+
+static int cpu_virt_paging_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+switch ( action )
+{
+case CPU_STARTING:
+ASSERT(system_state != SYS_STATE_boot);
+setup_virt_paging_one(NULL);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
 }
 
+static struct notifier_block cpu_virt_paging_nfb = {
+.notifier_call = cpu_virt_paging_callback,
+};
+
+static int __init cpu_virt_paging_init(void)
+{
+register_cpu_notifier(_virt_paging_nfb);
+
+return 0;
+}
+/*
+ * Initialization of the notifier has to be done at init rather than 
presmp_init
+ * phase because: the registered notifier is used to setup virtual paging for
+ * non-boot CPUs after the initial virtual paging for all CPUs is already 
setup,
+ * i.e. when a non-boot CPU is hotplugged after the system has booted. In other
+ * words, the notifier should be registered after the virtual paging is
+ * initially setup (setup_virt_paging() is called fro

[Xen-devel] [PATCH v5 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

Linux/dom0 accesses OSLSR register when saving CPU context during the
suspend procedure. Xen traps access to this register, but has no handling
for it. Consequently, Xen injects undef exception to linux, causing it to
crash. This patch adds handling of the trapped access to OSLSR as read
only as a fixed value.

Signed-off-by: Mirela Simonovic 
Reviewed-by: Stefano Stabellini 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v2:
- Commit message fix (arm64 related change instead of arm)
- Add Stefano's reviewed-by

Changes in v3:
- Added Julien's acked-by

Changes in v5:
-Insted of zero the reading of OSLSR_EL1 should return set bit 3
-Implement new helper handle_ro_read_val() to support read only as a value.
 handle_ro_read_val() reuses the implementation of handle_ro_raz() and
 extends it with additional argument for passing the value to be returned
-Use handle_ro_read_val() for handle_ro_raz() implementation to avoid code
 duplication
-Fix commit message to reflect changes made in this version
---
 xen/arch/arm/arm64/vsysreg.c |  4 +++-
 xen/arch/arm/traps.c | 26 ++
 xen/include/asm-arm/traps.h  |  4 
 3 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/xen/arch/arm/arm64/vsysreg.c b/xen/arch/arm/arm64/vsysreg.c
index c57ac12503..6e60824572 100644
--- a/xen/arch/arm/arm64/vsysreg.c
+++ b/xen/arch/arm/arm64/vsysreg.c
@@ -57,13 +57,15 @@ void do_sysreg(struct cpu_user_regs *regs,
  * ARMv8 (DDI 0487A.d): D1-1509 Table D1-58
  *
  * Unhandled:
- *OSLSR_EL1
  *DBGPRCR_EL1
  */
 case HSR_SYSREG_OSLAR_EL1:
 return handle_wo_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
 case HSR_SYSREG_OSDLR_EL1:
 return handle_raz_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
+case HSR_SYSREG_OSLSR_EL1:
+return handle_ro_read_val(regs, regidx, hsr.sysreg.read, hsr, 1,
+  1 << 3);
 
 /*
  * MDCR_EL2.TDA
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 5c18e918b0..d71adfa745 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -1739,12 +1739,13 @@ void handle_wo_wi(struct cpu_user_regs *regs,
 advance_pc(regs, hsr);
 }
 
-/* Read only as read as zero */
-void handle_ro_raz(struct cpu_user_regs *regs,
-   int regidx,
-   bool read,
-   const union hsr hsr,
-   int min_el)
+/* Read only as value provided with 'val' argument of this function */
+void handle_ro_read_val(struct cpu_user_regs *regs,
+int regidx,
+bool read,
+const union hsr hsr,
+int min_el,
+register_t val)
 {
 ASSERT((min_el == 0) || (min_el == 1));
 
@@ -1753,13 +1754,22 @@ void handle_ro_raz(struct cpu_user_regs *regs,
 
 if ( !read )
 return inject_undef_exception(regs, hsr);
-/* else: raz */
 
-set_user_reg(regs, regidx, 0);
+set_user_reg(regs, regidx, val);
 
 advance_pc(regs, hsr);
 }
 
+/* Read only as read as zero */
+inline void handle_ro_raz(struct cpu_user_regs *regs,
+  int regidx,
+  bool read,
+  const union hsr hsr,
+  int min_el)
+{
+handle_ro_read_val(regs, regidx, read, hsr, min_el, 0);
+}
+
 void dump_guest_s1_walk(struct domain *d, vaddr_t addr)
 {
 register_t ttbcr = READ_SYSREG(TCR_EL1);
diff --git a/xen/include/asm-arm/traps.h b/xen/include/asm-arm/traps.h
index a0e5e92ebb..70b52d1d16 100644
--- a/xen/include/asm-arm/traps.h
+++ b/xen/include/asm-arm/traps.h
@@ -27,6 +27,10 @@ void handle_wo_wi(struct cpu_user_regs *regs, int regidx, 
bool read,
 void handle_ro_raz(struct cpu_user_regs *regs, int regidx, bool read,
const union hsr hsr, int min_el);
 
+/* Read only as value provided with 'val' argument */
+void handle_ro_read_val(struct cpu_user_regs *regs, int regidx, bool read,
+const union hsr hsr, int min_el, register_t val);
+
 /* Co-processor registers emulation (see arch/arm/vcpreg.c). */
 void do_cp15_32(struct cpu_user_regs *regs, const union hsr hsr);
 void do_cp15_64(struct cpu_user_regs *regs, const union hsr hsr);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 04/10] xen/arm: Remove __initdata and __init to enable CPU hotplug

[Mirela Simonovic]

CPU up flow is currently used during the initial boot to start secondary
CPUs. However, the same flow should be used for CPU hotplug, e.g. when
hotplugging secondary CPUs within the resume procedure (resume from the
suspend to RAM). Therefore, prefixes __initdata and __init had to be removed
from few data structures and functions that are used within the cpu up flow.

Signed-off-by: Mirela Simonovic 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v3:
- Added acked-by Julien
---
 xen/arch/arm/arm64/smpboot.c   | 2 +-
 xen/arch/arm/irq.c | 2 +-
 xen/arch/arm/processor.c   | 2 +-
 xen/arch/arm/smpboot.c | 4 ++--
 xen/include/asm-arm/procinfo.h | 4 ++--
 5 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/arm64/smpboot.c b/xen/arch/arm/arm64/smpboot.c
index 4fd0ac68b7..694fbf67e6 100644
--- a/xen/arch/arm/arm64/smpboot.c
+++ b/xen/arch/arm/arm64/smpboot.c
@@ -104,7 +104,7 @@ int __init arch_cpu_init(int cpu, struct dt_device_node *dn)
 return smp_psci_init(cpu);
 }
 
-int __init arch_cpu_up(int cpu)
+int arch_cpu_up(int cpu)
 {
 if ( !smp_enable_ops[cpu].prepare_cpu )
 return -ENODEV;
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index aa4e832cae..098281f8ab 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -65,7 +65,7 @@ irq_desc_t *__irq_to_desc(int irq)
 return _desc[irq-NR_LOCAL_IRQS];
 }
 
-int __init arch_init_one_irq_desc(struct irq_desc *desc)
+int arch_init_one_irq_desc(struct irq_desc *desc)
 {
 desc->arch.type = IRQ_TYPE_INVALID;
 return 0;
diff --git a/xen/arch/arm/processor.c b/xen/arch/arm/processor.c
index ce4385064a..acad8b31d6 100644
--- a/xen/arch/arm/processor.c
+++ b/xen/arch/arm/processor.c
@@ -20,7 +20,7 @@
 
 static DEFINE_PER_CPU(struct processor *, processor);
 
-void __init processor_setup(void)
+void processor_setup(void)
 {
 const struct proc_info_list *procinfo;
 
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index 8b1e274bf3..ad1f6b751b 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -52,8 +52,8 @@ nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 static unsigned char __initdata cpu0_boot_stack[STACK_SIZE]
__attribute__((__aligned__(STACK_SIZE)));
 
-/* Initial boot cpu data */
-struct init_info __initdata init_data =
+/* Boot cpu data */
+struct init_info init_data =
 {
 .stack = cpu0_boot_stack,
 };
diff --git a/xen/include/asm-arm/procinfo.h b/xen/include/asm-arm/procinfo.h
index 26306b35f8..02be56e348 100644
--- a/xen/include/asm-arm/procinfo.h
+++ b/xen/include/asm-arm/procinfo.h
@@ -35,9 +35,9 @@ struct proc_info_list {
 struct processor*processor;
 };
 
-const __init struct proc_info_list *lookup_processor_type(void);
+const struct proc_info_list *lookup_processor_type(void);
 
-void __init processor_setup(void);
+void processor_setup(void);
 void processor_vcpu_initialise(struct vcpu *v);
 
 #endif
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 07/10] xen/arm: Release maintenance interrupt when CPU is hot-unplugged

[Mirela Simonovic]

When a CPU is hot-unplugged the maintenance interrupt has to be
released in order to free the memory that was allocated when the CPU
was hotplugged and interrupt requested. The interrupt was requested
using request_irq() which is called from start_secondary->
init_maintenance_interrupt. With this patch the interrupt will be
released when the CPU_DYING event is received by the callback which
is added in gic.c.

Signed-off-by: Mirela Simonovic 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v3:
-Add notifier in order to trigger releasing of the  maintenance
 interrupt when the CPU is dying.

Changes in v4:
-Add includes alphabetically
-Added newline before the return in cpu_gic_notifier_init()
-Fix indentation in cpu_gic_callback() definition
-Added acked-by Julien
---
 xen/arch/arm/gic.c | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 653a815127..5474030386 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -27,6 +27,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -462,6 +464,35 @@ int gic_iomem_deny_access(const struct domain *d)
 return gic_hw_ops->iomem_deny_access(d);
 }
 
+static int cpu_gic_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+switch ( action )
+{
+case CPU_DYING:
+/* This is reverting the work done in init_maintenance_interrupt */
+release_irq(gic_hw_ops->info->maintenance_irq, NULL);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_gic_nfb = {
+.notifier_call = cpu_gic_callback,
+};
+
+static int __init cpu_gic_notifier_init(void)
+{
+register_cpu_notifier(_gic_nfb);
+
+return 0;
+}
+__initcall(cpu_gic_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 08/10] xen/arm: Disable timers and release their interrupts on CPU hot-unplug

[Mirela Simonovic]

When a CPU is hot-unplugged we need to disable timers and release
their interrupts in order to free the memory that was allocated when
interrupts were requested (using request_irq()). The request_irq()
is called for each timer interrupt when the CPU gets hotplugged
(start_secondary->init_timer_interrupt->request_irq).
With this patch timers will be disabled and interrupts will be
released when the newly added callback receives CPU_DYING event.

Signed-off-by: Mirela Simonovic 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v3:
-Trigger releasing of timer interrupts using notifiers

Changes in v4:
-Fix commit message to include disabling of timers
-Disable timers prior to releasing interrupts
-Add new line before the return in cpu_time_notifier_init()
-Add includes alphabetically
-Fix indentation in cpu_time_callback() definition

Changes in v5:
-Added acked-by Julien
---
 xen/arch/arm/time.c | 45 +
 1 file changed, 45 insertions(+)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index c11fcfeadd..1635c8822d 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -29,6 +29,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -312,6 +314,21 @@ void init_timer_interrupt(void)
 check_timer_irq_cfg(timer_irq[TIMER_PHYS_NONSECURE_PPI], "NS-physical");
 }
 
+/*
+ * Revert actions done in init_timer_interrupt that are required to properly
+ * disable this CPU.
+ */
+static void deinit_timer_interrupt(void)
+{
+WRITE_SYSREG32(0, CNTP_CTL_EL0);/* Disable physical timer */
+WRITE_SYSREG32(0, CNTHP_CTL_EL2);   /* Disable hypervisor's timer */
+isb();
+
+release_irq(timer_irq[TIMER_HYP_PPI], NULL);
+release_irq(timer_irq[TIMER_VIRT_PPI], NULL);
+release_irq(timer_irq[TIMER_PHYS_NONSECURE_PPI], NULL);
+}
+
 /* Wait a set number of microseconds */
 void udelay(unsigned long usecs)
 {
@@ -340,6 +357,34 @@ void domain_set_time_offset(struct domain *d, int64_t 
time_offset_seconds)
 /* XXX update guest visible wallclock time */
 }
 
+static int cpu_time_callback(struct notifier_block *nfb,
+ unsigned long action,
+ void *hcpu)
+{
+switch ( action )
+{
+case CPU_DYING:
+deinit_timer_interrupt();
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_time_nfb = {
+.notifier_call = cpu_time_callback,
+};
+
+static int __init cpu_time_notifier_init(void)
+{
+register_cpu_notifier(_time_nfb);
+
+return 0;
+}
+__initcall(cpu_time_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 03/10] xen/arm: Implement CPU_OFF PSCI call (physical interface)

[Mirela Simonovic]

During the system suspend to RAM non-boot CPUs will be hotplugged.
This will be triggered via disable_nonboot_cpus() call. When
hotplugged the CPU will end up in an infinite wfi loop in stop_cpu().
This patch adds PSCI CPU_OFF call to the EL3 with the aim to get powered
down the calling CPU during the suspend. The CPU_OFF call will be made
only if the PSCI version is higher than v0.1 (Note that the CPU_OFF
function is mandatory since PSCI v0.2).
If PSCI CPU_OFF call to the EL3 succeeds it will not return. Otherwise,
when the PSCI CPU_OFF call returns we'll raise panic, because the
calling CPU couldn't be enabled afterwards (stays in WFI loop forever).
Note that if the PSCI version is higher than v0.1 the CPU_OFF will be
called regardless of the system state. This is done because scenarios
other than suspend may benefit from powering off the CPU.

Signed-off-by: Mirela Simonovic 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v2:
-Issue PSCI CPU_OFF only if the system is suspending
-If PSCI CPU_OFF call fails (unlikely to ever happen) raise panic
-Fixed commit message

Changes in v3:
-Check for PSCI version prior to calling CPU_OFF
-Don't check for system state - invoke CPU_OFF in all system states
-Don't check if returned error is not zero because it's always not
 zero if CPU_OFF SMC returns
-Fixed commit message

Changes in v4:
-Use smp_processor_id() instead of get_processor_id()
-Fixed indentation
-Added acked-by Julien
---
 xen/arch/arm/psci.c| 13 +
 xen/arch/arm/smpboot.c |  2 ++
 xen/include/asm-arm/psci.h |  1 +
 3 files changed, 16 insertions(+)

diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
index 94b616df9b..3cf5ecf0f3 100644
--- a/xen/arch/arm/psci.c
+++ b/xen/arch/arm/psci.c
@@ -46,6 +46,19 @@ int call_psci_cpu_on(int cpu)
 return call_smc(psci_cpu_on_nr, cpu_logical_map(cpu), 
__pa(init_secondary), 0);
 }
 
+void call_psci_cpu_off(void)
+{
+if ( psci_ver > PSCI_VERSION(0, 1) )
+{
+int errno;
+
+/* If successfull the PSCI cpu_off call doesn't return */
+errno = call_smc(PSCI_0_2_FN32_CPU_OFF, 0, 0, 0);
+panic("PSCI cpu off failed for CPU%d err=%d\n", smp_processor_id(),
+  errno);
+}
+}
+
 void call_psci_system_off(void)
 {
 if ( psci_ver > PSCI_VERSION(0, 1) )
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index b2116f0d2d..8b1e274bf3 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -395,6 +395,8 @@ void stop_cpu(void)
 /* Make sure the write happens before we sleep forever */
 dsb(sy);
 isb();
+call_psci_cpu_off();
+
 while ( 1 )
 wfi();
 }
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 9ac820e94a..832f77afff 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -20,6 +20,7 @@ extern uint32_t psci_ver;
 
 int psci_init(void);
 int call_psci_cpu_on(int cpu);
+void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes

[Mirela Simonovic]

This patch set contains fixes that are required as precondition for suspend to
RAM support, including the CPU hotplug which is required to suspend non-boot
CPUs.
The first two patches in this series:
1) xen/arm64: Added handling of the trapped access to OSLSR register
2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
patch set does not include the implementation of virtual PSCI system suspend
call that would allow guests to finalize their suspend procedures. This will
be submitted in the following series.

Remaining of the patches are related to enabling CPU hotplug for non-boot
CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for suspend
to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot CPUs
will be triggered with disable_nonboot_cpus(), while the hotplug is triggered
with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs could
be powered down/up on suspend/resume, respectively, if the underlying firmware
allows so. Calls to enable/disable_nonboot_cpus() functions currently do not
exist in Xen ARM code. This will be added with the suspend to RAM support for
ARM.

When non-boot pCPUs are hot-unplugged their interrupts are migrated to the boot
pCPU. This series also includes a fix that would restore the interrupts affinity
once non-boot pCPUs are hotplugged. Here only SPIs used by guests are covered.
Migration of Xen internal SPIs is not covered. According to my understanding
Xen internal SPIs are routed to the boot CPU which initializes the respective
devices. Therefore, there is no need to migrate Xen internal SPIs.

The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
physical power down/up of non-boot CPUs). The testing requires additional
patches for issuing system suspend. These patches and instructions for testing
will be submitted later, when we get closer to the final version of the series.

---
Changes in v2:
-Rename cover-letter title and emphasize that 2 patches from this series are not
specific to CPU hotplug (my initial fault, splitting it now could be confusing)
-Fix cover-letter explanations
-Address all the issues and comments as discussed on mailing list for v1
-Add 3 patches to ensure that suspend/resume does not cause any memory leaks.
All the memory allocated when a CPU was hotplugged is now freed when the CPU is
hot-unplugged.
-Remove from the v1 series the patch which incorrectly dealt with an issue:
[PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
One solution to the issue addressed by the patch above is to add rcu_barrier()
prior to calling enable_nonboot_cpus() during the suspend. This is how it is
done in x86 suspend implementation. Until the discussion here
https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
doesn't conclude differently, I need to assume that adding rcu_barrier() prior
to calling enable_nonboot_cpus() as it is done for x86 is the right way to go.
Therefore, the fix to the issue will be part of the suspend to RAM series.

Changes in v3:
-Add acked-by where needed
-Fix CPU_OFF PSCI implementation (physical interface)
-Use notifiers to implement freeing memory and releasing interrupts on CPU
hotplug
-Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
hotplug
-Add enabling errata workarounds on CPU hotplug, also based on a notifier
-Remove patch:
[PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU only on 
boot

Changes in v4:
-Add acked-by/reviewed-by where needed
-Cleanup: use smp_processor_id() instead of get_processor_id(), fixed
 indentation, add includes alphabetically, add newline before return, etc.
-Disable timers prior to releasing timer interrupts
-Initialize cpu_smpboot notifier at presmp_init rather than init phase
-In the last patch of the series errata notifier now returns an error

Changes in v5:
-Introduce handle_ro_read_val() to handle traps as read-only as fixed value
-Fix handling accesses to OSLSR_EL1
-Fix variable type in 5th patch

---
CC: Stefano Stabellini 
CC: Julien Grall 
CC: George Dunlap 
CC: Dario Faggioli 
---

Mirela Simonovic (10):
  xen/arm64: Added handling of the trapped access to OSLSR register
  xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
  xen/arm: Implement CPU_OFF PSCI call (physical interface)
  xen/arm: Remove __initdata and __init to enable CPU hotplug
  xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume
  xen/common: Restore IRQ affinity when hotplugging a pCPU
  xen/arm: Release maintenance interrupt when CPU is hot-unplugged
  xen/arm: Disable timers and release their interrupts on CPU hot-unplug
  xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug
  xen/arm: Enable errata for secondary CPU on hotplug after the boot

 xen/arch/arm/arm64/smpboot.c |  2 +-
 xen/arch/arm/arm64/vsysreg.c |  4 ++-
 xen/arch/arm

[Xen-devel] [PATCH v5 09/10] xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug

[Mirela Simonovic]

The memory allocated in setup_cpu_sibling_map() when a CPU is hotplugged
has to be freed when the CPU is hot-unplugged. This is done in
remove_cpu_sibling_map() and called when the CPU dies. The call to
remove_cpu_sibling_map() is made from a notifier callback when
CPU_DEAD event is received.

Signed-off-by: Mirela Simonovic 
Acked-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v3:
-Use notifier to trigger remove_cpu_sibling_map() when the CPU dies.

Changes in v4:
-Initialize cpu_smpboot notifier at presmp_init rather than init phase
 to cover the case where a secondary CPU dies beforehand the initcall
-Added newline before the return in cpu_smpboot_notifier_init()
-Fix indentation in cpu_smpboot_callback() definition

Changes in v5:
-Added acked-by Julien
---
 xen/arch/arm/smpboot.c | 36 
 1 file changed, 36 insertions(+)

diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ad1f6b751b..cf3a4ce659 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -89,6 +89,12 @@ static void setup_cpu_sibling_map(int cpu)
 cpumask_set_cpu(cpu, per_cpu(cpu_core_mask, cpu));
 }
 
+static void remove_cpu_sibling_map(int cpu)
+{
+free_cpumask_var(per_cpu(cpu_sibling_mask, cpu));
+free_cpumask_var(per_cpu(cpu_core_mask, cpu));
+}
+
 void __init
 smp_clear_cpu_maps (void)
 {
@@ -499,6 +505,36 @@ void __cpu_die(unsigned int cpu)
 smp_mb();
 }
 
+static int cpu_smpboot_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+unsigned int cpu = (unsigned long)hcpu;
+
+switch ( action )
+{
+case CPU_DEAD:
+remove_cpu_sibling_map(cpu);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_smpboot_nfb = {
+.notifier_call = cpu_smpboot_callback,
+};
+
+static int __init cpu_smpboot_notifier_init(void)
+{
+register_cpu_notifier(_smpboot_nfb);
+
+return 0;
+}
+presmp_initcall(cpu_smpboot_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v5 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

On boot, enabling errata workarounds will be triggered by the boot CPU
from start_xen(). On CPU hotplug (non-boot scenario) this would not be
done. This patch adds the code required to enable errata workarounds for
a CPU being hotplugged after the system boots. This is triggered using
a notifier. If the CPU fails to enable workarounds the notifier will
return an error and Xen will hit the BUG_ON() in notify_cpu_starting().
To avoid the BUG_ON() in an error case either enabling notifiers should
be fixed to return void (not propagate error to notify_cpu_starting())
and the errata notifier will always return success for CPU_STARTING
event, or the notify_cpu_starting() and other common code should be
fixed to expect an error at CPU_STARTING phase.

Signed-off-by: Mirela Simonovic 
Reviewed-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v4:
-Add includes alphabetically
-Added newline before the return in cpu_errata_notifier_init()
-Enabling capabilities returns an error if enabling a capability fails
 (enable_nonboot_cpu_caps() returns int instead of void). When enabling
 any of the capability fails the error is remembered into a variable and
 the remaining capabilities are enabled. If enabling multiple capabilities
 fails the error returned by enable_nonboot_cpu_caps() represents the
 error code of the last failure.
-Callback enable_nonboot_cpu_caps() can return an error when CPU_STARTING
 fires. This is not right because of the assumption that starting a CPU
 cannot fail at this phase. Consequently, if an error happens it will
 cause Xen to hit the BUG_ON() in notify_cpu_starting(). In future,
 either this notifier/enabling capabilities should be fixed to always
 return success/void, or notify_cpu_starting() and other common code
 should be fixed to expect an error at CPU_STARTING phase.
-Fix commit message to reflect changes in v4

Changes in v5:
-Added reviewed-by Julien
---
 xen/arch/arm/cpuerrata.c | 49 
 xen/arch/arm/cpufeature.c| 29 
 xen/include/asm-arm/cpufeature.h |  1 +
 3 files changed, 79 insertions(+)

diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
index 1baa20654b..b829d226ef 100644
--- a/xen/arch/arm/cpuerrata.c
+++ b/xen/arch/arm/cpuerrata.c
@@ -1,3 +1,4 @@
+#include 
 #include 
 #include 
 #include 
@@ -5,6 +6,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -349,6 +351,53 @@ void __init enable_errata_workarounds(void)
 enable_cpu_capabilities(arm_errata);
 }
 
+static int cpu_errata_callback(struct notifier_block *nfb,
+   unsigned long action,
+   void *hcpu)
+{
+int rc = 0;
+
+switch ( action )
+{
+case CPU_STARTING:
+/*
+ * At CPU_STARTING phase no notifier shall return an error, because the
+ * system is designed with the assumption that starting a CPU cannot
+ * fail at this point. If an error happens here it will cause Xen to 
hit
+ * the BUG_ON() in notify_cpu_starting(). In future, either this
+ * notifier/enabling capabilities should be fixed to always return
+ * success/void or notify_cpu_starting() and other common code should 
be
+ * fixed to expect an error at CPU_STARTING phase.
+ */
+ASSERT(system_state != SYS_STATE_boot);
+rc = enable_nonboot_cpu_caps(arm_errata);
+break;
+default:
+break;
+}
+
+return !rc ? NOTIFY_DONE : notifier_from_errno(rc);
+}
+
+static struct notifier_block cpu_errata_nfb = {
+.notifier_call = cpu_errata_callback,
+};
+
+static int __init cpu_errata_notifier_init(void)
+{
+register_cpu_notifier(_errata_nfb);
+
+return 0;
+}
+/*
+ * Initialization has to be done at init rather than presmp_init phase because
+ * the callback should execute only after the secondary CPUs are initially
+ * booted (in hotplug scenarios when the system state is not boot). On boot,
+ * the enabling of errata workarounds will be triggered by the boot CPU from
+ * start_xen().
+ */
+__initcall(cpu_errata_notifier_init);
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
index 525b45e22f..3aaff4c0e6 100644
--- a/xen/arch/arm/cpufeature.c
+++ b/xen/arch/arm/cpufeature.c
@@ -69,6 +69,35 @@ void __init enable_cpu_capabilities(const struct 
arm_cpu_capabilities *caps)
 }
 
 /*
+ * Run through the enabled capabilities and enable() them on the calling CPU.
+ * If enabling of any capability fails the error is returned. After enabling a
+ * capability fails the error will be remembered into 'rc' and the remaining
+ * capabilities will be enabled. If enabling multiple capabilities fail the
+ * error returned by this function represents the error code of the last
+ * failure.
+ */
+int enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *caps)
+{
+int rc = 0

[Xen-devel] [PATCH v5 02/10] xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)

[Mirela Simonovic]

Guests attempt to write into these registers on resume (for example Linux).
Without this patch a data abort exception will be raised to the guest.
This patch handles the write access by ignoring it, but only if the value
to be written is zero. This should be fine because reading these registers
is already handled as 'read as zero'.

Signed-off-by: Mirela Simonovic 
Reviewed-by: Julien Grall 

---
CC: Stefano Stabellini 
CC: Julien Grall 
---
Changes in v2:
- Write should be ignored only if the value to be written is zero
 (in v1 the write was ignored regardless of the value)

Changes in v3:
- Print warning only if the value to be written is not zero

Changes in v4:
- Added reviewed-by Julien
---
 xen/arch/arm/vgic-v2.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 646d1f3d12..f6c11f1e41 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -485,6 +485,8 @@ static int vgic_v2_distr_mmio_write(struct vcpu *v, 
mmio_info_t *info,
 
 case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
 if ( dabt.size != DABT_WORD ) goto bad_width;
+if ( r == 0 )
+goto write_ignore_32;
 printk(XENLOG_G_ERR
"%pv: vGICD: unhandled word write %#"PRIregister" to 
ISACTIVER%d\n",
v, r, gicd_reg - GICD_ISACTIVER);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v4 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

On Wed, May 30, 2018 at 11:48 AM, Mirela Simonovic <
mirela.simono...@aggios.com> wrote:

> Hi Julien,
>
> Thanks for the feedback.
>
> On Tue, May 29, 2018 at 3:19 PM, Julien Grall 
> wrote:
>
>> Hi,
>>
>>
>> On 15/05/18 12:44, Mirela Simonovic wrote:
>>
>>> Linux/dom0 accesses OSLSR register when saving CPU context during the
>>> suspend procedure. Xen traps access to this register, but has no handling
>>> for it. Consequently, Xen injects undef exception to linux, causing it to
>>> crash. This patch adds handling of the trapped access to OSLSR as ro/raz.
>>>
>>> Signed-off-by: Mirela Simonovic 
>>> Reviewed-by: Stefano Stabellini 
>>> Acked-by: Julien Grall 
>>>
>>> ---
>>> CC: Stefano Stabellini 
>>> CC: Julien Grall 
>>> ---
>>> Changes in v2:
>>> - Commit message fix (arm64 related change instead of arm)
>>> - Add Stefano's reviewed-by
>>>
>>> Changes in v3:
>>> - Added Julien's acked-by
>>> ---
>>>   xen/arch/arm/arm64/vsysreg.c | 3 ++-
>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/xen/arch/arm/arm64/vsysreg.c b/xen/arch/arm/arm64/vsysreg.c
>>> index c57ac12503..8f80e1735e 100644
>>> --- a/xen/arch/arm/arm64/vsysreg.c
>>> +++ b/xen/arch/arm/arm64/vsysreg.c
>>> @@ -57,13 +57,14 @@ void do_sysreg(struct cpu_user_regs *regs,
>>>* ARMv8 (DDI 0487A.d): D1-1509 Table D1-58
>>>*
>>>* Unhandled:
>>> - *OSLSR_EL1
>>>*DBGPRCR_EL1
>>>*/
>>>   case HSR_SYSREG_OSLAR_EL1:
>>>   return handle_wo_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
>>>   case HSR_SYSREG_OSDLR_EL1:
>>>   return handle_raz_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
>>> +case HSR_SYSREG_OSLSR_EL1:
>>> +return handle_ro_raz(regs, regidx, hsr.sysreg.read, hsr, 1);
>>>
>>
>> Looking at this patch again, OSLM (bit[3] and bit[1]) should be 10 as all
>> other values are reserved. So you want to set bit 3 when the guest read it.
>>
>>
> Could you please let me know is my proposal/understanding correct? Instead
> of handle_ro_raz, I assume we need this:
>
> if ( psr_mode_is_user(regs) )
> return inject_undef_exception(regs, hsr);
> if ( hsr.sysreg.read )
> set_user_reg(regs, regidx, 0x8);
> break;
>
>

Please note - I'm not sure how should write to this register be treated. In
the original patch (handle_ro_raz implementation) undef exception will be
raised. I realized in what I proposed that wouldn't be the case, it's a
write-ignore.


> Thanks,
> Mirela
>
> Cheers,
>>
>> --
>> Julien Grall
>>
>
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v4 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

Hi Julien,

Thanks for the feedback.

On Tue, May 29, 2018 at 3:19 PM, Julien Grall  wrote:

> Hi,
>
>
> On 15/05/18 12:44, Mirela Simonovic wrote:
>
>> Linux/dom0 accesses OSLSR register when saving CPU context during the
>> suspend procedure. Xen traps access to this register, but has no handling
>> for it. Consequently, Xen injects undef exception to linux, causing it to
>> crash. This patch adds handling of the trapped access to OSLSR as ro/raz.
>>
>> Signed-off-by: Mirela Simonovic 
>> Reviewed-by: Stefano Stabellini 
>> Acked-by: Julien Grall 
>>
>> ---
>> CC: Stefano Stabellini 
>> CC: Julien Grall 
>> ---
>> Changes in v2:
>> - Commit message fix (arm64 related change instead of arm)
>> - Add Stefano's reviewed-by
>>
>> Changes in v3:
>> - Added Julien's acked-by
>> ---
>>   xen/arch/arm/arm64/vsysreg.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/arm64/vsysreg.c b/xen/arch/arm/arm64/vsysreg.c
>> index c57ac12503..8f80e1735e 100644
>> --- a/xen/arch/arm/arm64/vsysreg.c
>> +++ b/xen/arch/arm/arm64/vsysreg.c
>> @@ -57,13 +57,14 @@ void do_sysreg(struct cpu_user_regs *regs,
>>* ARMv8 (DDI 0487A.d): D1-1509 Table D1-58
>>*
>>* Unhandled:
>> - *OSLSR_EL1
>>*DBGPRCR_EL1
>>*/
>>   case HSR_SYSREG_OSLAR_EL1:
>>   return handle_wo_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
>>   case HSR_SYSREG_OSDLR_EL1:
>>   return handle_raz_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
>> +case HSR_SYSREG_OSLSR_EL1:
>> +return handle_ro_raz(regs, regidx, hsr.sysreg.read, hsr, 1);
>>
>
> Looking at this patch again, OSLM (bit[3] and bit[1]) should be 10 as all
> other values are reserved. So you want to set bit 3 when the guest read it.
>
>
Could you please let me know is my proposal/understanding correct? Instead
of handle_ro_raz, I assume we need this:

if ( psr_mode_is_user(regs) )
return inject_undef_exception(regs, hsr);
if ( hsr.sysreg.read )
set_user_reg(regs, regidx, 0x8);
break;

Thanks,
Mirela

Cheers,
>
> --
> Julien Grall
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v4 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes

[Mirela Simonovic]

Hi,

Could you please provide feedback for the remaining patches?

Thanks,
Mirela

On Tue, May 15, 2018 at 1:44 PM, Mirela Simonovic <
mirela.simono...@aggios.com> wrote:

> This patch set contains fixes that are required as precondition for
> suspend to
> RAM support, including the CPU hotplug which is required to suspend
> non-boot
> CPUs.
> The first two patches in this series:
> 1) xen/arm64: Added handling of the trapped access to OSLSR register
> 2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
> are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
> patch set does not include the implementation of virtual PSCI system
> suspend
> call that would allow guests to finalize their suspend procedures. This
> will
> be submitted in the following series.
>
> Remaining of the patches are related to enabling CPU hotplug for non-boot
> CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for
> suspend
> to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot
> CPUs
> will be triggered with disable_nonboot_cpus(), while the hotplug is
> triggered
> with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs
> could
> be powered down/up on suspend/resume, respectively, if the underlying
> firmware
> allows so. Calls to enable/disable_nonboot_cpus() functions currently do
> not
> exist in Xen ARM code. This will be added with the suspend to RAM support
> for
> ARM.
>
> When non-boot pCPUs are hot-unplugged their interrupts are migrated to the
> boot
> pCPU. This series also includes a fix that would restore the interrupts
> affinity
> once non-boot pCPUs are hotplugged. Here only SPIs used by guests are
> covered.
> Migration of Xen internal SPIs is not covered. According to my
> understanding
> Xen internal SPIs are routed to the boot CPU which initializes the
> respective
> devices. Therefore, there is no need to migrate Xen internal SPIs.
>
> The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
> physical power down/up of non-boot CPUs). The testing requires additional
> patches for issuing system suspend. These patches and instructions for
> testing
> will be submitted later, when we get closer to the final version of the
> series.
>
> ---
> Changes in v2:
> -Rename cover-letter title and emphasize that 2 patches from this series
> are not
> specific to CPU hotplug (my initial fault, splitting it now could be
> confusing)
> -Fix cover-letter explanations
> -Address all the issues and comments as discussed on mailing list for v1
> -Add 3 patches to ensure that suspend/resume does not cause any memory
> leaks.
> All the memory allocated when a CPU was hotplugged is now freed when the
> CPU is
> hot-unplugged.
> -Remove from the v1 series the patch which incorrectly dealt with an issue:
> [PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
> One solution to the issue addressed by the patch above is to add
> rcu_barrier()
> prior to calling enable_nonboot_cpus() during the suspend. This is how it
> is
> done in x86 suspend implementation. Until the discussion here
> https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
> doesn't conclude differently, I need to assume that adding rcu_barrier()
> prior
> to calling enable_nonboot_cpus() as it is done for x86 is the right way to
> go.
> Therefore, the fix to the issue will be part of the suspend to RAM series.
>
> Changes in v3:
> -Add acked-by where needed
> -Fix CPU_OFF PSCI implementation (physical interface)
> -Use notifiers to implement freeing memory and releasing interrupts on CPU
> hotplug
> -Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
> hotplug
> -Add enabling errata workarounds on CPU hotplug, also based on a notifier
> -Remove patch:
> [PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU
> only on boot
>
> Changes in v4:
> -Add acked-by/reviewed-by where needed
> -Cleanup: use smp_processor_id() instead of get_processor_id(), fixed
>  indentation, add includes alphabetically, add newline before return, etc.
> -Disable timers prior to releasing timer interrupts
> -Initialize cpu_smpboot notifier at presmp_init rather than init phase
> -In the last patch of the series errata notifier now returns an error
>
> ---
> CC: Stefano Stabellini <sstabell...@kernel.org>
> CC: Julien Grall <julien.gr...@arm.com>
> CC: George Dunlap <george.dun...@eu.citrix.com>
> CC: Dario Faggioli <dfaggi...@suse.com>
> ---
>
> Mirela Simonovic (10):
>   xen/arm64: Added handling of the trapped access to OSLSR register
>   xen/arm: Ignore write 

[Xen-devel] [PATCH v4 05/10] xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume

[Mirela Simonovic]

In existing code the virtual paging for non-boot CPUs is setup only on boot.
The setup is triggered from start_xen() after all CPUs are brought online.
In other words, the initialization of VTCR_EL2 register is done out of the
cpu_up/start_secondary() control flow. However, the cpu_up flow is also used
to hotplug non-boot CPUs on resume from suspend to RAM state, in which case
the virtual paging will not be configured.

With this patch the setting of paging is triggered from start_secondary()
function using cpu starting notifier (notify_cpu_starting() call). The
notifier is registered in p2m.c using init call. This has to be done with
init call rather than presmp_init because the registered callback depends
on vtcr configuration value which is setup after the presmp init calls
are executed (do_presmp_initcalls() called from start_xen()). Init calls
are executed after initial virtual paging is set up for all CPUs on boot.
This ensures that no callback can fire until the vtcr value is calculated
by Xen and virtual paging is set up initially for all CPUs. Also, this way
the virtual paging setup in boot scenario remains unchanged.

It is assumed here that after the system completed the boot, CPUs that
execute start_secondary() were booted as well when the Xen itself was
booted. According to this assumption non-boot CPUs will always be compliant
with the VTCR_EL2 value that was selected by Xen on boot.
Currently, there is no mechanism to trigger hotplugging of a CPU. This
will be added with the suspend to RAM support for ARM, where the hotplug
of non-boot CPUs will be triggered via enable_nonboot_cpus() call.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
-Fix commit message
-Save configured VTCR_EL2 value into static variable that will be used
 by non-boot CPUs on hotplug
-Add setup_virt_paging_secondary() and invoke it from start_secondary()
 if that CPU has to setup virtual paging (if the system state is not boot)

Changes in v3:
-Fix commit message
-Remove setup_virt_paging_secondary() and use notifier to setup virtual
 paging for non-boot CPU on hotplug.
-In setup_virt_paging() use vtcr static variable instead of local val
-In setup_virt_paging_one() use vtcr static variable instead of provided
 argument

Changes in v4:
-Add includes alphabetically
-Add newline before return in cpu_virt_paging_init()
-Fix indentation in cpu_virt_paging_callback() definition
-Use local val in setup_virt_paging() for calculation, assign it to vtcr
 after the calculation is done
-Remove priority initialization in the notifier structure (priority
 doesn't matter here)

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
---
 xen/arch/arm/p2m.c | 53 -
 1 file changed, 48 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index d43c3aa896..924226f63c 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -8,6 +8,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -1451,10 +1453,12 @@ err:
 return page;
 }
 
-static void __init setup_virt_paging_one(void *data)
+/* VTCR value to be configured by all CPUs. Set only once by the boot CPU */
+static uint64_t __read_mostly vtcr;
+
+static void setup_virt_paging_one(void *data)
 {
-unsigned long val = (unsigned long)data;
-WRITE_SYSREG32(val, VTCR_EL2);
+WRITE_SYSREG32(vtcr, VTCR_EL2);
 isb();
 }
 
@@ -1538,10 +1542,49 @@ void __init setup_virt_paging(void)
 
 /* It is not allowed to concatenate a level zero root */
 BUG_ON( P2M_ROOT_LEVEL == 0 && P2M_ROOT_ORDER > 0 );
-setup_virt_paging_one((void *)val);
-smp_call_function(setup_virt_paging_one, (void *)val, 1);
+vtcr = val;
+setup_virt_paging_one(NULL);
+smp_call_function(setup_virt_paging_one, NULL, 1);
+}
+
+static int cpu_virt_paging_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+switch ( action )
+{
+case CPU_STARTING:
+ASSERT(system_state != SYS_STATE_boot);
+setup_virt_paging_one(NULL);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
 }
 
+static struct notifier_block cpu_virt_paging_nfb = {
+.notifier_call = cpu_virt_paging_callback,
+};
+
+static int __init cpu_virt_paging_init(void)
+{
+register_cpu_notifier(_virt_paging_nfb);
+
+return 0;
+}
+/*
+ * Initialization of the notifier has to be done at init rather than 
presmp_init
+ * phase because: the registered notifier is used to setup virtual paging for
+ * non-boot CPUs after the initial virtual paging for all CPUs is already 
setup,
+ * i.e. when a non-boot CPU is hotplugged after the system has booted. In other
+ * words, the notifier should be 

[Xen-devel] [PATCH v4 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

On boot, enabling errata workarounds will be triggered by the boot CPU
from start_xen(). On CPU hotplug (non-boot scenario) this would not be
done. This patch adds the code required to enable errata workarounds for
a CPU being hotplugged after the system boots. This is triggered using
a notifier. If the CPU fails to enable workarounds the notifier will
return an error and Xen will hit the BUG_ON() in notify_cpu_starting().
To avoid the BUG_ON() in an error case either enabling notifiers should
be fixed to return void (not propagate error to notify_cpu_starting())
and the errata notifier will always return success for CPU_STARTING
event, or the notify_cpu_starting() and other common code should be
fixed to expect an error at CPU_STARTING phase.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v4:
-Add includes alphabetically
-Added newline before the return in cpu_errata_notifier_init()
-Enabling capabilities returns an error if enabling a capability fails
 (enable_nonboot_cpu_caps() returns int instead of void). When enabling
 any of the capability fails the error is remembered into a variable and
 the remaining capabilities are enabled. If enabling multiple capabilities
 fails the error returned by enable_nonboot_cpu_caps() represents the
 error code of the last failure.
-Callback enable_nonboot_cpu_caps() can return an error when CPU_STARTING
 fires. This is not right because of the assumption that starting a CPU
 cannot fail at this phase. Consequently, if an error happens it will
 cause Xen to hit the BUG_ON() in notify_cpu_starting(). In future,
 either this notifier/enabling capabilities should be fixed to always
 return success/void, or notify_cpu_starting() and other common code
 should be fixed to expect an error at CPU_STARTING phase.
-Fix commit message to reflect changes in v4
---
 xen/arch/arm/cpuerrata.c | 49 
 xen/arch/arm/cpufeature.c| 29 
 xen/include/asm-arm/cpufeature.h |  1 +
 3 files changed, 79 insertions(+)

diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
index 1baa20654b..b829d226ef 100644
--- a/xen/arch/arm/cpuerrata.c
+++ b/xen/arch/arm/cpuerrata.c
@@ -1,3 +1,4 @@
+#include 
 #include 
 #include 
 #include 
@@ -5,6 +6,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -349,6 +351,53 @@ void __init enable_errata_workarounds(void)
 enable_cpu_capabilities(arm_errata);
 }
 
+static int cpu_errata_callback(struct notifier_block *nfb,
+   unsigned long action,
+   void *hcpu)
+{
+int rc = 0;
+
+switch ( action )
+{
+case CPU_STARTING:
+/*
+ * At CPU_STARTING phase no notifier shall return an error, because the
+ * system is designed with the assumption that starting a CPU cannot
+ * fail at this point. If an error happens here it will cause Xen to 
hit
+ * the BUG_ON() in notify_cpu_starting(). In future, either this
+ * notifier/enabling capabilities should be fixed to always return
+ * success/void or notify_cpu_starting() and other common code should 
be
+ * fixed to expect an error at CPU_STARTING phase.
+ */
+ASSERT(system_state != SYS_STATE_boot);
+rc = enable_nonboot_cpu_caps(arm_errata);
+break;
+default:
+break;
+}
+
+return !rc ? NOTIFY_DONE : notifier_from_errno(rc);
+}
+
+static struct notifier_block cpu_errata_nfb = {
+.notifier_call = cpu_errata_callback,
+};
+
+static int __init cpu_errata_notifier_init(void)
+{
+register_cpu_notifier(_errata_nfb);
+
+return 0;
+}
+/*
+ * Initialization has to be done at init rather than presmp_init phase because
+ * the callback should execute only after the secondary CPUs are initially
+ * booted (in hotplug scenarios when the system state is not boot). On boot,
+ * the enabling of errata workarounds will be triggered by the boot CPU from
+ * start_xen().
+ */
+__initcall(cpu_errata_notifier_init);
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
index 525b45e22f..3aaff4c0e6 100644
--- a/xen/arch/arm/cpufeature.c
+++ b/xen/arch/arm/cpufeature.c
@@ -69,6 +69,35 @@ void __init enable_cpu_capabilities(const struct 
arm_cpu_capabilities *caps)
 }
 
 /*
+ * Run through the enabled capabilities and enable() them on the calling CPU.
+ * If enabling of any capability fails the error is returned. After enabling a
+ * capability fails the error will be remembered into 'rc' and the remaining
+ * capabilities will be enabled. If enabling multiple capabilities fail the
+ * error returned by this function represents the error code of the last
+ * failure.
+ */
+int enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *ca

[Xen-devel] [PATCH v4 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

Linux/dom0 accesses OSLSR register when saving CPU context during the
suspend procedure. Xen traps access to this register, but has no handling
for it. Consequently, Xen injects undef exception to linux, causing it to
crash. This patch adds handling of the trapped access to OSLSR as ro/raz.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
Acked-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
- Commit message fix (arm64 related change instead of arm)
- Add Stefano's reviewed-by

Changes in v3:
- Added Julien's acked-by
---
 xen/arch/arm/arm64/vsysreg.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/arm64/vsysreg.c b/xen/arch/arm/arm64/vsysreg.c
index c57ac12503..8f80e1735e 100644
--- a/xen/arch/arm/arm64/vsysreg.c
+++ b/xen/arch/arm/arm64/vsysreg.c
@@ -57,13 +57,14 @@ void do_sysreg(struct cpu_user_regs *regs,
  * ARMv8 (DDI 0487A.d): D1-1509 Table D1-58
  *
  * Unhandled:
- *OSLSR_EL1
  *DBGPRCR_EL1
  */
 case HSR_SYSREG_OSLAR_EL1:
 return handle_wo_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
 case HSR_SYSREG_OSDLR_EL1:
 return handle_raz_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
+case HSR_SYSREG_OSLSR_EL1:
+return handle_ro_raz(regs, regidx, hsr.sysreg.read, hsr, 1);
 
 /*
  * MDCR_EL2.TDA
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 04/10] xen/arm: Remove __initdata and __init to enable CPU hotplug

[Mirela Simonovic]

CPU up flow is currently used during the initial boot to start secondary
CPUs. However, the same flow should be used for CPU hotplug, e.g. when
hotplugging secondary CPUs within the resume procedure (resume from the
suspend to RAM). Therefore, prefixes __initdata and __init had to be removed
from few data structures and functions that are used within the cpu up flow.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Acked-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v3:
- Added acked-by Julien
---
 xen/arch/arm/arm64/smpboot.c   | 2 +-
 xen/arch/arm/irq.c | 2 +-
 xen/arch/arm/processor.c   | 2 +-
 xen/arch/arm/smpboot.c | 4 ++--
 xen/include/asm-arm/procinfo.h | 4 ++--
 5 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/arm64/smpboot.c b/xen/arch/arm/arm64/smpboot.c
index 4fd0ac68b7..694fbf67e6 100644
--- a/xen/arch/arm/arm64/smpboot.c
+++ b/xen/arch/arm/arm64/smpboot.c
@@ -104,7 +104,7 @@ int __init arch_cpu_init(int cpu, struct dt_device_node *dn)
 return smp_psci_init(cpu);
 }
 
-int __init arch_cpu_up(int cpu)
+int arch_cpu_up(int cpu)
 {
 if ( !smp_enable_ops[cpu].prepare_cpu )
 return -ENODEV;
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index aa4e832cae..098281f8ab 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -65,7 +65,7 @@ irq_desc_t *__irq_to_desc(int irq)
 return _desc[irq-NR_LOCAL_IRQS];
 }
 
-int __init arch_init_one_irq_desc(struct irq_desc *desc)
+int arch_init_one_irq_desc(struct irq_desc *desc)
 {
 desc->arch.type = IRQ_TYPE_INVALID;
 return 0;
diff --git a/xen/arch/arm/processor.c b/xen/arch/arm/processor.c
index ce4385064a..acad8b31d6 100644
--- a/xen/arch/arm/processor.c
+++ b/xen/arch/arm/processor.c
@@ -20,7 +20,7 @@
 
 static DEFINE_PER_CPU(struct processor *, processor);
 
-void __init processor_setup(void)
+void processor_setup(void)
 {
 const struct proc_info_list *procinfo;
 
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index 8b1e274bf3..ad1f6b751b 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -52,8 +52,8 @@ nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 static unsigned char __initdata cpu0_boot_stack[STACK_SIZE]
__attribute__((__aligned__(STACK_SIZE)));
 
-/* Initial boot cpu data */
-struct init_info __initdata init_data =
+/* Boot cpu data */
+struct init_info init_data =
 {
 .stack = cpu0_boot_stack,
 };
diff --git a/xen/include/asm-arm/procinfo.h b/xen/include/asm-arm/procinfo.h
index 26306b35f8..02be56e348 100644
--- a/xen/include/asm-arm/procinfo.h
+++ b/xen/include/asm-arm/procinfo.h
@@ -35,9 +35,9 @@ struct proc_info_list {
 struct processor*processor;
 };
 
-const __init struct proc_info_list *lookup_processor_type(void);
+const struct proc_info_list *lookup_processor_type(void);
 
-void __init processor_setup(void);
+void processor_setup(void);
 void processor_vcpu_initialise(struct vcpu *v);
 
 #endif
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Stefano,

On Mon, May 14, 2018 at 6:59 PM, Stefano Stabellini
 wrote:
> On Mon, 14 May 2018, Julien Grall wrote:
>> On 11/05/18 22:47, Stefano Stabellini wrote:
>> > On Fri, 11 May 2018, Dario Faggioli wrote:
>> > > On Fri, 2018-05-11 at 14:08 +0100, Julien Grall wrote:
>> > > > The whole idea here is we have only one place taking the decision and
>> > > > we
>> > > > don't spread BUG_ON()/panic/stop_cpu everywhere. The benefit is
>> > > > having
>> > > > only one place to fix over multiple one because very likely the
>> > > > decision
>> > > > is the same everywhere.
>> > > >
>> > > > I agree that today it will end up to crashing the system because of
>> > > > the
>> > > > BUG_ON. But that's a separate topic.
>> > > >
>> > > Yes!!! :-D
>> > >
>> > > I.e., as I've said countless times, I would think that a series which
>> > > introduces a CPU_STARTING notifier that fails, should also deal with
>> > > adjusting the CPU process accordingly.
>> > >
>> > > *BUT* if you ARM people are ok with arch/arm/ code that does that,
>> > > perhaps with a comment saying something like:
>> > >
>> > > "This will cause us to hit the BUG_ON() in notify_cpu_starting(). To
>> > > fix that, we need to properly change the CPU bringup code, which will
>> > > happen in a leter series."
>> > >
>> > > that would also work, I guess. :-)
>> >
>> > Yes, I think that returning error with an in-code comment on top is the
>> > best solution.
>>
>> It is the second best solution ;). If we consider the notifier can return an
>> error, then best solution is to fix notify_cpu_starting().
>>
>> I would be ok with the second best solution if we have someone to fix it for
>> Xen 4.12. Per my understanding, Mirela is not going to do it. So what's the
>> plan here?
>
> I can look at fixing notify_cpu_starting(). I am also OK with you
> reworking the vmap code as you suggested below. Regardless, I think
> Mirela should go ahead with the comment now. Then, either you or me are
> going to come in and remove the comment one way or another (either
> fixing notify_cpu_starting or imposing all the callbacks to never return
> an error).
>

Thanks, I submitted v4.

Regards,
Mirela

>
>> Another solution is to impose all the enable callbacks to never return an
>> error (AFAICT Linux is just ignoring the return of the callback)).
>>
>> Today, we happen to return an error only in the case vmap is failing (used to
>> remapped vector table read-write). It might be possible to avoid the 
>> potential
>> re-mapping failure by reworking the code.
>>
>> I could explore that solution if we prefer going towards imposing all the
>> enable callbacks to never return an error.

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 08/10] xen/arm: Disable timers and release their interrupts on CPU hot-unplug

[Mirela Simonovic]

When a CPU is hot-unplugged we need to disable timers and release
their interrupts in order to free the memory that was allocated when
interrupts were requested (using request_irq()). The request_irq()
is called for each timer interrupt when the CPU gets hotplugged
(start_secondary->init_timer_interrupt->request_irq).
With this patch timers will be disabled and interrupts will be
released when the newly added callback receives CPU_DYING event.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v3:
-Trigger releasing of timer interrupts using notifiers

Changes in v4:
-Fix commit message to include disabling of timers
-Disable timers prior to releasing interrupts
-Add new line before the return in cpu_time_notifier_init()
-Add includes alphabetically
-Fix indentation in cpu_time_callback() definition
---
 xen/arch/arm/time.c | 45 +
 1 file changed, 45 insertions(+)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index c11fcfeadd..1635c8822d 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -29,6 +29,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -312,6 +314,21 @@ void init_timer_interrupt(void)
 check_timer_irq_cfg(timer_irq[TIMER_PHYS_NONSECURE_PPI], "NS-physical");
 }
 
+/*
+ * Revert actions done in init_timer_interrupt that are required to properly
+ * disable this CPU.
+ */
+static void deinit_timer_interrupt(void)
+{
+WRITE_SYSREG32(0, CNTP_CTL_EL0);/* Disable physical timer */
+WRITE_SYSREG32(0, CNTHP_CTL_EL2);   /* Disable hypervisor's timer */
+isb();
+
+release_irq(timer_irq[TIMER_HYP_PPI], NULL);
+release_irq(timer_irq[TIMER_VIRT_PPI], NULL);
+release_irq(timer_irq[TIMER_PHYS_NONSECURE_PPI], NULL);
+}
+
 /* Wait a set number of microseconds */
 void udelay(unsigned long usecs)
 {
@@ -340,6 +357,34 @@ void domain_set_time_offset(struct domain *d, int64_t 
time_offset_seconds)
 /* XXX update guest visible wallclock time */
 }
 
+static int cpu_time_callback(struct notifier_block *nfb,
+ unsigned long action,
+ void *hcpu)
+{
+switch ( action )
+{
+case CPU_DYING:
+deinit_timer_interrupt();
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_time_nfb = {
+.notifier_call = cpu_time_callback,
+};
+
+static int __init cpu_time_notifier_init(void)
+{
+register_cpu_notifier(_time_nfb);
+
+return 0;
+}
+__initcall(cpu_time_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 09/10] xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug

[Mirela Simonovic]

The memory allocated in setup_cpu_sibling_map() when a CPU is hotplugged
has to be freed when the CPU is hot-unplugged. This is done in
remove_cpu_sibling_map() and called when the CPU dies. The call to
remove_cpu_sibling_map() is made from a notifier callback when
CPU_DEAD event is received.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v3:
-Use notifier to trigger remove_cpu_sibling_map() when the CPU dies.

Changes in v4:
-Initialize cpu_smpboot notifier at presmp_init rather than init phase
 to cover the case where a secondary CPU dies beforehand the initcall
-Added newline before the return in cpu_smpboot_notifier_init()
-Fix indentation in cpu_smpboot_callback() definition
---
 xen/arch/arm/smpboot.c | 36 
 1 file changed, 36 insertions(+)

diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ad1f6b751b..cf3a4ce659 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -89,6 +89,12 @@ static void setup_cpu_sibling_map(int cpu)
 cpumask_set_cpu(cpu, per_cpu(cpu_core_mask, cpu));
 }
 
+static void remove_cpu_sibling_map(int cpu)
+{
+free_cpumask_var(per_cpu(cpu_sibling_mask, cpu));
+free_cpumask_var(per_cpu(cpu_core_mask, cpu));
+}
+
 void __init
 smp_clear_cpu_maps (void)
 {
@@ -499,6 +505,36 @@ void __cpu_die(unsigned int cpu)
 smp_mb();
 }
 
+static int cpu_smpboot_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+unsigned int cpu = (unsigned long)hcpu;
+
+switch ( action )
+{
+case CPU_DEAD:
+remove_cpu_sibling_map(cpu);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_smpboot_nfb = {
+.notifier_call = cpu_smpboot_callback,
+};
+
+static int __init cpu_smpboot_notifier_init(void)
+{
+register_cpu_notifier(_smpboot_nfb);
+
+return 0;
+}
+presmp_initcall(cpu_smpboot_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 03/10] xen/arm: Implement CPU_OFF PSCI call (physical interface)

[Mirela Simonovic]

During the system suspend to RAM non-boot CPUs will be hotplugged.
This will be triggered via disable_nonboot_cpus() call. When
hotplugged the CPU will end up in an infinite wfi loop in stop_cpu().
This patch adds PSCI CPU_OFF call to the EL3 with the aim to get powered
down the calling CPU during the suspend. The CPU_OFF call will be made
only if the PSCI version is higher than v0.1 (Note that the CPU_OFF
function is mandatory since PSCI v0.2).
If PSCI CPU_OFF call to the EL3 succeeds it will not return. Otherwise,
when the PSCI CPU_OFF call returns we'll raise panic, because the
calling CPU couldn't be enabled afterwards (stays in WFI loop forever).
Note that if the PSCI version is higher than v0.1 the CPU_OFF will be
called regardless of the system state. This is done because scenarios
other than suspend may benefit from powering off the CPU.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Acked-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
-Issue PSCI CPU_OFF only if the system is suspending
-If PSCI CPU_OFF call fails (unlikely to ever happen) raise panic
-Fixed commit message

Changes in v3:
-Check for PSCI version prior to calling CPU_OFF
-Don't check for system state - invoke CPU_OFF in all system states
-Don't check if returned error is not zero because it's always not
 zero if CPU_OFF SMC returns
-Fixed commit message

Changes in v4:
-Use smp_processor_id() instead of get_processor_id()
-Fixed indentation
-Added acked-by Julien
---
 xen/arch/arm/psci.c| 13 +
 xen/arch/arm/smpboot.c |  2 ++
 xen/include/asm-arm/psci.h |  1 +
 3 files changed, 16 insertions(+)

diff --git a/xen/arch/arm/psci.c b/xen/arch/arm/psci.c
index 94b616df9b..3cf5ecf0f3 100644
--- a/xen/arch/arm/psci.c
+++ b/xen/arch/arm/psci.c
@@ -46,6 +46,19 @@ int call_psci_cpu_on(int cpu)
 return call_smc(psci_cpu_on_nr, cpu_logical_map(cpu), 
__pa(init_secondary), 0);
 }
 
+void call_psci_cpu_off(void)
+{
+if ( psci_ver > PSCI_VERSION(0, 1) )
+{
+int errno;
+
+/* If successfull the PSCI cpu_off call doesn't return */
+errno = call_smc(PSCI_0_2_FN32_CPU_OFF, 0, 0, 0);
+panic("PSCI cpu off failed for CPU%d err=%d\n", smp_processor_id(),
+  errno);
+}
+}
+
 void call_psci_system_off(void)
 {
 if ( psci_ver > PSCI_VERSION(0, 1) )
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index b2116f0d2d..8b1e274bf3 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -395,6 +395,8 @@ void stop_cpu(void)
 /* Make sure the write happens before we sleep forever */
 dsb(sy);
 isb();
+call_psci_cpu_off();
+
 while ( 1 )
 wfi();
 }
diff --git a/xen/include/asm-arm/psci.h b/xen/include/asm-arm/psci.h
index 9ac820e94a..832f77afff 100644
--- a/xen/include/asm-arm/psci.h
+++ b/xen/include/asm-arm/psci.h
@@ -20,6 +20,7 @@ extern uint32_t psci_ver;
 
 int psci_init(void);
 int call_psci_cpu_on(int cpu);
+void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 06/10] xen/common: Restore IRQ affinity when hotplugging a pCPU

[Mirela Simonovic]

Non-boot pCPUs are being hot-unplugged during the system suspend to
RAM and hotplugged during the resume. When non-boot pCPUs are
hot-unplugged the interrupts that were targeted to them are migrated
to the boot pCPU.
On suspend, each guest could have its own wake-up devices/interrupts
(passthrough) that could trigger the system resume. These interrupts
could be targeted to a non-boot pCPU, e.g. if the guest's vCPU is
pinned to a non-boot pCPU. Due to the hot-unplug of non-boot pCPUs
during the suspend such interrupts will be migrated from non-boot pCPUs
to the boot pCPU (this is fine). However, when non-boot pCPUs are
hotplugged on resume, these interrupts are not migrated back to non-boot
pCPUs, i.e. IRQ affinity is not restored on resume (this is wrong).
This patch adds the restoration of IRQ affinity when a pCPU is hotplugged.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Reviewed-by: Dario Faggioli <dfaggi...@suse.com>

---
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Dario Faggioli <dfaggi...@suse.com>
---
Changes in v2:
-Instead of checking whether the affinity was broken check whether
 vcpu's processor has changed in order to trigger restoring of the
 IRQ affinity
-Fix commit message

Changes in v4:
-Added reviewed by Dario
---
 xen/common/schedule.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/xen/common/schedule.c b/xen/common/schedule.c
index 049f93f7aa..ccf936db83 100644
--- a/xen/common/schedule.c
+++ b/xen/common/schedule.c
@@ -737,6 +737,7 @@ void restore_vcpu_affinity(struct domain *d)
 for_each_vcpu ( d, v )
 {
 spinlock_t *lock;
+unsigned int old_cpu = v->processor;
 
 ASSERT(!vcpu_runnable(v));
 
@@ -769,6 +770,9 @@ void restore_vcpu_affinity(struct domain *d)
 lock = vcpu_schedule_lock_irq(v);
 v->processor = SCHED_OP(vcpu_scheduler(v), pick_cpu, v);
 spin_unlock_irq(lock);
+
+if ( old_cpu != v->processor )
+sched_move_irqs(v);
 }
 
 domain_update_node_affinity(d);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 02/10] xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)

[Mirela Simonovic]

Guests attempt to write into these registers on resume (for example Linux).
Without this patch a data abort exception will be raised to the guest.
This patch handles the write access by ignoring it, but only if the value
to be written is zero. This should be fine because reading these registers
is already handled as 'read as zero'.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Reviewed-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
- Write should be ignored only if the value to be written is zero
 (in v1 the write was ignored regardless of the value)

Changes in v3:
- Print warning only if the value to be written is not zero

Changes in v4:
- Added reviewed-by Julien
---
 xen/arch/arm/vgic-v2.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 646d1f3d12..f6c11f1e41 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -485,6 +485,8 @@ static int vgic_v2_distr_mmio_write(struct vcpu *v, 
mmio_info_t *info,
 
 case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
 if ( dabt.size != DABT_WORD ) goto bad_width;
+if ( r == 0 )
+goto write_ignore_32;
 printk(XENLOG_G_ERR
"%pv: vGICD: unhandled word write %#"PRIregister" to 
ISACTIVER%d\n",
v, r, gicd_reg - GICD_ISACTIVER);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v4 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes

[Mirela Simonovic]

This patch set contains fixes that are required as precondition for suspend to
RAM support, including the CPU hotplug which is required to suspend non-boot
CPUs.
The first two patches in this series:
1) xen/arm64: Added handling of the trapped access to OSLSR register
2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
patch set does not include the implementation of virtual PSCI system suspend
call that would allow guests to finalize their suspend procedures. This will
be submitted in the following series.

Remaining of the patches are related to enabling CPU hotplug for non-boot
CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for suspend
to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot CPUs
will be triggered with disable_nonboot_cpus(), while the hotplug is triggered
with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs could
be powered down/up on suspend/resume, respectively, if the underlying firmware
allows so. Calls to enable/disable_nonboot_cpus() functions currently do not
exist in Xen ARM code. This will be added with the suspend to RAM support for
ARM.

When non-boot pCPUs are hot-unplugged their interrupts are migrated to the boot
pCPU. This series also includes a fix that would restore the interrupts affinity
once non-boot pCPUs are hotplugged. Here only SPIs used by guests are covered.
Migration of Xen internal SPIs is not covered. According to my understanding
Xen internal SPIs are routed to the boot CPU which initializes the respective
devices. Therefore, there is no need to migrate Xen internal SPIs.

The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
physical power down/up of non-boot CPUs). The testing requires additional
patches for issuing system suspend. These patches and instructions for testing
will be submitted later, when we get closer to the final version of the series.

---
Changes in v2:
-Rename cover-letter title and emphasize that 2 patches from this series are not
specific to CPU hotplug (my initial fault, splitting it now could be confusing)
-Fix cover-letter explanations
-Address all the issues and comments as discussed on mailing list for v1
-Add 3 patches to ensure that suspend/resume does not cause any memory leaks.
All the memory allocated when a CPU was hotplugged is now freed when the CPU is
hot-unplugged.
-Remove from the v1 series the patch which incorrectly dealt with an issue:
[PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
One solution to the issue addressed by the patch above is to add rcu_barrier()
prior to calling enable_nonboot_cpus() during the suspend. This is how it is
done in x86 suspend implementation. Until the discussion here
https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
doesn't conclude differently, I need to assume that adding rcu_barrier() prior
to calling enable_nonboot_cpus() as it is done for x86 is the right way to go.
Therefore, the fix to the issue will be part of the suspend to RAM series.

Changes in v3:
-Add acked-by where needed
-Fix CPU_OFF PSCI implementation (physical interface)
-Use notifiers to implement freeing memory and releasing interrupts on CPU
hotplug
-Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
hotplug
-Add enabling errata workarounds on CPU hotplug, also based on a notifier
-Remove patch:
[PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU only on 
boot

Changes in v4:
-Add acked-by/reviewed-by where needed
-Cleanup: use smp_processor_id() instead of get_processor_id(), fixed
 indentation, add includes alphabetically, add newline before return, etc.
-Disable timers prior to releasing timer interrupts
-Initialize cpu_smpboot notifier at presmp_init rather than init phase
-In the last patch of the series errata notifier now returns an error

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Dario Faggioli <dfaggi...@suse.com>
---

Mirela Simonovic (10):
  xen/arm64: Added handling of the trapped access to OSLSR register
  xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
  xen/arm: Implement CPU_OFF PSCI call (physical interface)
  xen/arm: Remove __initdata and __init to enable CPU hotplug
  xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume
  xen/common: Restore IRQ affinity when hotplugging a pCPU
  xen/arm: Release maintenance interrupt when CPU is hot-unplugged
  xen/arm: Disable timers and release their interrupts on CPU hot-unplug
  xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug
  xen/arm: Enable errata for secondary CPU on hotplug after the boot

 xen/arch/arm/arm64/smpboot.c |  2 +-
 xen/arch/arm/arm64/vsysreg.c |  3 ++-
 xen/arch/arm/cpuerrata.c | 49 ++

[Xen-devel] [PATCH v4 07/10] xen/arm: Release maintenance interrupt when CPU is hot-unplugged

[Mirela Simonovic]

When a CPU is hot-unplugged the maintenance interrupt has to be
released in order to free the memory that was allocated when the CPU
was hotplugged and interrupt requested. The interrupt was requested
using request_irq() which is called from start_secondary->
init_maintenance_interrupt. With this patch the interrupt will be
released when the CPU_DYING event is received by the callback which
is added in gic.c.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Acked-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v3:
-Add notifier in order to trigger releasing of the  maintenance
 interrupt when the CPU is dying.

Changes in v4:
-Add includes alphabetically
-Added newline before the return in cpu_gic_notifier_init()
-Fix indentation in cpu_gic_callback() definition
-Added acked-by Julien
---
 xen/arch/arm/gic.c | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 653a815127..5474030386 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -27,6 +27,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -462,6 +464,35 @@ int gic_iomem_deny_access(const struct domain *d)
 return gic_hw_ops->iomem_deny_access(d);
 }
 
+static int cpu_gic_callback(struct notifier_block *nfb,
+unsigned long action,
+void *hcpu)
+{
+switch ( action )
+{
+case CPU_DYING:
+/* This is reverting the work done in init_maintenance_interrupt */
+release_irq(gic_hw_ops->info->maintenance_irq, NULL);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_gic_nfb = {
+.notifier_call = cpu_gic_callback,
+};
+
+static int __init cpu_gic_notifier_init(void)
+{
+register_cpu_notifier(_gic_nfb);
+
+return 0;
+}
+__initcall(cpu_gic_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi,

On Fri, May 11, 2018 at 2:07 PM, Mirela Simonovic
<mirela.simono...@aggios.com> wrote:
> Hi Julien,
>
> On Fri, May 11, 2018 at 12:54 PM, Julien Grall <julien.gr...@arm.com> wrote:
>>
>>
>> On 11/05/18 11:41, Mirela Simonovic wrote:
>>>
>>> Hi Dario,
>>>
>>> On Thu, May 10, 2018 at 6:24 PM, Dario Faggioli <dfaggi...@suse.com>
>>> wrote:
>>>>
>>>> On Thu, 2018-05-10 at 17:49 +0200, Mirela Simonovic wrote:
>>>>>
>>>>> Regardless of the fact that the notifier returns an error or not, I
>>>>> believe it would be good and safe to set priority and document that
>>>>> priority zero would cause racing issue in the scenario I debugged
>>>>> today. I'm pretty sure that this discussion would be forgotten soon
>>>>> and it really should be documented in code/comment.
>>>>>
>>>> I may very well be missing or misunderstanding something, but I
>>>> continue to think that the problem here is that CPU_STARTING can't,
>>>> right now, fail, while you need it to be able to.
>>>>
>>>> If that is the case, giving different priorities to the notifier, is
>>>> not a solution.
>>>>
>>>
>>> Let me try to clarify. The assumption is that the starting CPU can
>>> fail. Additional requirement set by Julien is that panic or BUG_ON is
>>> not acceptable.
>>
>>
>> Please don't twist my word. I never said it was not acceptable to have the
>> BUG_ON in notify_cpu_starting().
>
> I didn't say that either. You referenced notify_cpu_starting() here, not me.
> BTW, if you get the impression that I'm twisting words then we have a
> misunderstanding and your approach is not the best way toward
> clarifying it. Please have that in mind next time, because I do not
> have such an intent and I never will.
>
> I referred to panic/BUG_ON in this scenario regardless of the place
> from which it would be invoked. My understanding from your previous
> answers is that you think the system should not panic/BUG_ON in this
> scenario, because this kind of error the system should be able to
> survive.
>
>>
>> I am going to repeat the content of my answer to your last e-mail:
>>
>> I was aware about it since the beginning. The whole point of the
>> conversation was we should avoid to take the decision at the lower level and
>> let the upper layer decide what to do.
>>
>> If the system is failing today then that's fine and still fit what I said in
>> my first e-mail of that thread. For reminder:
>>
>> "We should really avoid to use panic(...) if this is something the system
>> can survive. In that specific case, it would only affect the current CPU. So
>> it would be better to return an error and let the caller decide what to do."
>>
>> To summarize:
>> 1) Notifiers should only report an error and let the upper caller
>> (here notify_cpu_starting()) deciding what to do.
>> 2) I am OK with the BUG_ON in notify_cpu_starting() for now.
>
> I agree with BUG_ON in notify_cpu_starting().
>
>>
>
> Let me just clarify consequence of your proposal according to my
> understanding. If instead of stopping the CPU when enabling a
> capability fails the notifier returns an error, the error will
> propagate to notify_cpu_starting() and BUG_ON will crash the system.
> The proposal with stop_cpu() in the enable_nonboot_cpu_caps() instead
> of panic that is submitted in this patch would stop only the erroneous
> CPU. The rest of the system will continue to work and I though that is
> what's the goal since we don't want to panic/BUG_ON.
> From that perspective I believe stop_cpu() in
> enable_nonboot_cpu_caps() is better compared to returning an error
> from the notifier.
>
> You said above "I would be ok having stop_cpu called here" and I did
> so (stop_cpu() in enable_nonboot_cpu_caps() instead of panic that
> submitted in this patch).
>
> If you believe my understanding is not correct, if I missed something
> or you have another proposal please let me know.
>

Also, if you just want to convert panic from this patch into print I
don't believe it's a good approach, but I can do that.

> Thanks,
> Mirela
>
>> Cheers,
>>
>> --
>> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Julien,

On Fri, May 11, 2018 at 12:54 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 11/05/18 11:41, Mirela Simonovic wrote:
>>
>> Hi Dario,
>>
>> On Thu, May 10, 2018 at 6:24 PM, Dario Faggioli <dfaggi...@suse.com>
>> wrote:
>>>
>>> On Thu, 2018-05-10 at 17:49 +0200, Mirela Simonovic wrote:
>>>>
>>>> Regardless of the fact that the notifier returns an error or not, I
>>>> believe it would be good and safe to set priority and document that
>>>> priority zero would cause racing issue in the scenario I debugged
>>>> today. I'm pretty sure that this discussion would be forgotten soon
>>>> and it really should be documented in code/comment.
>>>>
>>> I may very well be missing or misunderstanding something, but I
>>> continue to think that the problem here is that CPU_STARTING can't,
>>> right now, fail, while you need it to be able to.
>>>
>>> If that is the case, giving different priorities to the notifier, is
>>> not a solution.
>>>
>>
>> Let me try to clarify. The assumption is that the starting CPU can
>> fail. Additional requirement set by Julien is that panic or BUG_ON is
>> not acceptable.
>
>
> Please don't twist my word. I never said it was not acceptable to have the
> BUG_ON in notify_cpu_starting().

I didn't say that either. You referenced notify_cpu_starting() here, not me.
BTW, if you get the impression that I'm twisting words then we have a
misunderstanding and your approach is not the best way toward
clarifying it. Please have that in mind next time, because I do not
have such an intent and I never will.

I referred to panic/BUG_ON in this scenario regardless of the place
from which it would be invoked. My understanding from your previous
answers is that you think the system should not panic/BUG_ON in this
scenario, because this kind of error the system should be able to
survive.

>
> I am going to repeat the content of my answer to your last e-mail:
>
> I was aware about it since the beginning. The whole point of the
> conversation was we should avoid to take the decision at the lower level and
> let the upper layer decide what to do.
>
> If the system is failing today then that's fine and still fit what I said in
> my first e-mail of that thread. For reminder:
>
> "We should really avoid to use panic(...) if this is something the system
> can survive. In that specific case, it would only affect the current CPU. So
> it would be better to return an error and let the caller decide what to do."
>
> To summarize:
> 1) Notifiers should only report an error and let the upper caller
> (here notify_cpu_starting()) deciding what to do.
> 2) I am OK with the BUG_ON in notify_cpu_starting() for now.

I agree with BUG_ON in notify_cpu_starting().

>

Let me just clarify consequence of your proposal according to my
understanding. If instead of stopping the CPU when enabling a
capability fails the notifier returns an error, the error will
propagate to notify_cpu_starting() and BUG_ON will crash the system.
The proposal with stop_cpu() in the enable_nonboot_cpu_caps() instead
of panic that is submitted in this patch would stop only the erroneous
CPU. The rest of the system will continue to work and I though that is
what's the goal since we don't want to panic/BUG_ON.
From that perspective I believe stop_cpu() in
enable_nonboot_cpu_caps() is better compared to returning an error
from the notifier.

You said above "I would be ok having stop_cpu called here" and I did
so (stop_cpu() in enable_nonboot_cpu_caps() instead of panic that
submitted in this patch).

If you believe my understanding is not correct, if I missed something
or you have another proposal please let me know.

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Dario,

On Thu, May 10, 2018 at 6:24 PM, Dario Faggioli <dfaggi...@suse.com> wrote:
> On Thu, 2018-05-10 at 17:49 +0200, Mirela Simonovic wrote:
>> Regardless of the fact that the notifier returns an error or not, I
>> believe it would be good and safe to set priority and document that
>> priority zero would cause racing issue in the scenario I debugged
>> today. I'm pretty sure that this discussion would be forgotten soon
>> and it really should be documented in code/comment.
>>
> I may very well be missing or misunderstanding something, but I
> continue to think that the problem here is that CPU_STARTING can't,
> right now, fail, while you need it to be able to.
>
> If that is the case, giving different priorities to the notifier, is
> not a solution.
>

Let me try to clarify. The assumption is that the starting CPU can
fail. Additional requirement set by Julien is that panic or BUG_ON is
not acceptable. There are 2 ways to deal with this scenario:

1) Ignore and report the error, and let the erroneous CPU become online.
This cannot be done without changing the logic in either scheduler or
notify_cpu_starting(), or I least I don't see how would that be done.
In previous email when I said "escalating this to who knows where" I
did not refer to error escalation but the escalation of the scope of
this series.

2) Stop the erroneous CPU.
Taking this approach requires setting the priority for the notifier.
The key point is that notify_cpu_starting() and scheduler do not have
to be changed. If errata notifier has higher priority than scheduler's
notifier in the case of an error the CPU will not return into
notify_cpu_starting() and it will never execute BUG_ON because it will
be stopped. The rest of the system will continue to function without
that CPU.

> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Julien,

On Thu, May 10, 2018 at 5:13 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 10/05/18 16:00, Mirela Simonovic wrote:
>>
>> Hi Dario,
>>
>> On Thu, May 10, 2018 at 4:25 PM, Dario Faggioli <dfaggi...@suse.com>
>> wrote:
>>>
>>> On Thu, 2018-05-10 at 15:24 +0200, Mirela Simonovic wrote:
>>>>
>>>> On Thu, May 10, 2018 at 1:57 PM, Mirela Simonovic
>>>>
>>>>> Please take a look at function cpu_schedule_callback in schedule.c.
>>>>> Within switch, case CPU_DEAD doesn't have a break, causing the
>>>>> bellow
>>>>> CPU_UP_CANCELED to execute as well when the CPU goes down. This
>>>>> looks
>>>>> wrong to me.
>>>>> Dario, could you please confirm that this is a bug? Otherwise could
>>>>> you please confirm the reasoning beyond?
>>>>>
>>>>
>>>> Dario sorry, this looked wrong in my scenario but actually it is
>>>> correct. I understand the purpose of the missing break now.
>>>>
>>> No problem.
>>>
>>>> For the curious ones (if any) here is detailed description: errata
>>>> notifier added in this patch had the same priority as scheduler
>>>> notifier. I though priority doesn't matter, but I was wrong. In this
>>>> particular scenario where a CPU fails to enable capabilities
>>>> (triggered by errata notifier added in this patch), scheduler
>>>> callback
>>>> executed before the errata callback for CPU_STARTING event.
>>>>
>>> So, you're adding your errata callback to the CPU_STARTING notifier,
>>> right? (Sorry for having to ask, I don't have the patch handy right
>>> now.)
>>>
>>>> In other
>>>> words, scheduler already called init_pdata before the errata callback
>>>> fired (and stopped the CPU).
>>>> Later on when errata callback fired, enabling of capabilities has
>>>> failed, so the erroneous non-boot CPU stopped itself and never
>>>> declared to be online.
>>>> Then CPU#0 fired new notification with CPU_UP_CANCELED event in order
>>>> to clean up for the job done on CPU_STARTING. However, this broke the
>>>> assumption (which is good) made in cpu_schedule_callback. The
>>>> assumption is that the sequence of steps should be
>>>> alloc_pdata-->init_pdata-->deinit_pdata-->free_pdata. In this
>>>> particular case deinit_pdata was not done because this would be done
>>>> only upon CPU_DEAD event which makes no sense in this scenario.
>>>> In order to avoid running into the invalid scenario described above,
>>>> the errata callback should fire before the scheduler callback. If
>>>> enabling capabilities fails, the scheduler callback for CPU_STARTING
>>>> will never execute afterwards, so the following CPU_UP_CANCELED event
>>>> triggered by the CPU#0 will do free_pdata, which is ok because
>>>> init_pdata was not executed and alloc_pdata-->free_pdata flow is also
>>>> valid. Congratulations to the reader who reached this point :)
>>>>
>>> Ok, but the flow is, AFAICR, CPU_UP_PREPARE->CPU_STARTING->CPU_ONLINE.
>>>
>>> If you add your callback to CPU_UP_PREPARE, instead than to
>>> CPU_STARTING, SCHED_OP(init_pdata) wouldn't be called, without having
>>> to fiddle with priorities.
>
> This function will enable capabilities on a given CPU, most of them are
> touching system registers. So it is necessary to add the callback to
> CPU_STARTING.
>
>>>
>>
>> Difference between CPU_UP_PREPARE and CPU_STARTING (in addition to the
>> sequential ordering) is about which CPU executes the callback.
>> In CPU_UP_PREPARE case the CPU which called cpu_up for another CPU
>> will execute the callback. If I have 2 CPUs: CPU#0 executes callback
>> when trying to hotplug CPU#1. I need CPU#1 to execute this callback.
>> In CPU_STARTING case the CPU#1 will execute the callback, that is the
>> reason why it has to be CPU_STARTING event.
>>
>> I tried CPU_UP_PREPARE in my tuned scenario and I needed few moment to
>> realize why the system died (CPU#0 stopped himself :)
>>
>>> Is there any reason why you can't do it that way? It would look more
>>> natural to me, and it's definitely going to be easier debug and
>>> maintain (e.g., look at how many callbacks CPU_UP_PREPARE has, as
>>> compared to CPU_STARTING ;-P).
>>>
>>
>> Julien is go

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Dario,

On Thu, May 10, 2018 at 4:25 PM, Dario Faggioli <dfaggi...@suse.com> wrote:
> On Thu, 2018-05-10 at 15:24 +0200, Mirela Simonovic wrote:
>> On Thu, May 10, 2018 at 1:57 PM, Mirela Simonovic
>>
>> > Please take a look at function cpu_schedule_callback in schedule.c.
>> > Within switch, case CPU_DEAD doesn't have a break, causing the
>> > bellow
>> > CPU_UP_CANCELED to execute as well when the CPU goes down. This
>> > looks
>> > wrong to me.
>> > Dario, could you please confirm that this is a bug? Otherwise could
>> > you please confirm the reasoning beyond?
>> >
>>
>> Dario sorry, this looked wrong in my scenario but actually it is
>> correct. I understand the purpose of the missing break now.
>>
> No problem.
>
>> For the curious ones (if any) here is detailed description: errata
>> notifier added in this patch had the same priority as scheduler
>> notifier. I though priority doesn't matter, but I was wrong. In this
>> particular scenario where a CPU fails to enable capabilities
>> (triggered by errata notifier added in this patch), scheduler
>> callback
>> executed before the errata callback for CPU_STARTING event.
>>
> So, you're adding your errata callback to the CPU_STARTING notifier,
> right? (Sorry for having to ask, I don't have the patch handy right
> now.)
>
>> In other
>> words, scheduler already called init_pdata before the errata callback
>> fired (and stopped the CPU).
>> Later on when errata callback fired, enabling of capabilities has
>> failed, so the erroneous non-boot CPU stopped itself and never
>> declared to be online.
>> Then CPU#0 fired new notification with CPU_UP_CANCELED event in order
>> to clean up for the job done on CPU_STARTING. However, this broke the
>> assumption (which is good) made in cpu_schedule_callback. The
>> assumption is that the sequence of steps should be
>> alloc_pdata-->init_pdata-->deinit_pdata-->free_pdata. In this
>> particular case deinit_pdata was not done because this would be done
>> only upon CPU_DEAD event which makes no sense in this scenario.
>> In order to avoid running into the invalid scenario described above,
>> the errata callback should fire before the scheduler callback. If
>> enabling capabilities fails, the scheduler callback for CPU_STARTING
>> will never execute afterwards, so the following CPU_UP_CANCELED event
>> triggered by the CPU#0 will do free_pdata, which is ok because
>> init_pdata was not executed and alloc_pdata-->free_pdata flow is also
>> valid. Congratulations to the reader who reached this point :)
>>
> Ok, but the flow is, AFAICR, CPU_UP_PREPARE->CPU_STARTING->CPU_ONLINE.
>
> If you add your callback to CPU_UP_PREPARE, instead than to
> CPU_STARTING, SCHED_OP(init_pdata) wouldn't be called, without having
> to fiddle with priorities.
>

Difference between CPU_UP_PREPARE and CPU_STARTING (in addition to the
sequential ordering) is about which CPU executes the callback.
In CPU_UP_PREPARE case the CPU which called cpu_up for another CPU
will execute the callback. If I have 2 CPUs: CPU#0 executes callback
when trying to hotplug CPU#1. I need CPU#1 to execute this callback.
In CPU_STARTING case the CPU#1 will execute the callback, that is the
reason why it has to be CPU_STARTING event.

I tried CPU_UP_PREPARE in my tuned scenario and I needed few moment to
realize why the system died (CPU#0 stopped himself :)

> Is there any reason why you can't do it that way? It would look more
> natural to me, and it's definitely going to be easier debug and
> maintain (e.g., look at how many callbacks CPU_UP_PREPARE has, as
> compared to CPU_STARTING ;-P).
>

Julien is going to maintain this :)))

> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Software Engineer @ SUSE https://www.suse.com/

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

On Thu, May 10, 2018 at 1:57 PM, Mirela Simonovic
<mirela.simono...@aggios.com> wrote:
> Hi,
>
> +Dario
>
> On Wed, May 9, 2018 at 6:32 PM, Julien Grall <julien.gr...@arm.com> wrote:
>>
>>
>> On 09/05/18 16:48, Mirela Simonovic wrote:
>>>
>>> Hi Julien,
>>
>>
>> Hi Mirela,
>>
>>
>>> On Mon, Apr 30, 2018 at 6:09 PM, Julien Grall <julien.gr...@arm.com>
>>> wrote:
>>>>
>>>> Hi Mirela,
>>>>
>>>>
>>>> On 27/04/18 18:12, Mirela Simonovic wrote:
>>>>>
>>>>>
>>>>> On boot, enabling errata workarounds will be triggered by the boot CPU
>>>>> from start_xen(). On CPU hotplug (non-boot scenario) this would not be
>>>>> done. This patch adds the code required to enable errata workarounds
>>>>> for a CPU being hotplugged after the system boots. This is triggered
>>>>> using a notifier. If the CPU fails to enable the errata Xen will panic.
>>>>> This is done because it is assumed that the CPU which is hotplugged
>>>>> after the system/Xen boots, was initially hotplugged during the
>>>>> system/Xen boot. Therefore, enabling errata workarounds should never
>>>>> fail.
>>>>>
>>>>> Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
>>>>>
>>>>> ---
>>>>> CC: Stefano Stabellini <sstabell...@kernel.org>
>>>>> CC: Julien Grall <julien.gr...@arm.com>
>>>>> ---
>>>>>xen/arch/arm/cpuerrata.c | 35
>>>>> +++
>>>>>xen/arch/arm/cpufeature.c| 23 +++
>>>>>xen/include/asm-arm/cpufeature.h |  1 +
>>>>>3 files changed, 59 insertions(+)
>>>>>
>>>>> diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
>>>>> index 1baa20654b..4040f781ec 100644
>>>>> --- a/xen/arch/arm/cpuerrata.c
>>>>> +++ b/xen/arch/arm/cpuerrata.c
>>>>> @@ -5,6 +5,8 @@
>>>>>#include 
>>>>>#include 
>>>>>#include 
>>>>> +#include 
>>>>> +#include 
>>>>>#include 
>>>>>#include 
>>>>>#include 
>>>>> @@ -349,6 +351,39 @@ void __init enable_errata_workarounds(void)
>>>>>enable_cpu_capabilities(arm_errata);
>>>>>}
>>>>>+static int cpu_errata_callback(
>>>>> +struct notifier_block *nfb, unsigned long action, void *hcpu)
>>>>> +{
>>>>> +switch ( action )
>>>>> +{
>>>>> +case CPU_STARTING:
>>>>> +enable_nonboot_cpu_caps(arm_errata);
>>>>> +break;
>>>>> +default:
>>>>> +break;
>>>>> +}
>>>>> +
>>>>> +return NOTIFY_DONE;
>>>>> +}
>>>>> +
>>>>> +static struct notifier_block cpu_errata_nfb = {
>>>>> +.notifier_call = cpu_errata_callback,
>>>>> +};
>>>>> +
>>>>> +static int __init cpu_errata_notifier_init(void)
>>>>> +{
>>>>> +register_cpu_notifier(_errata_nfb);
>>>>> +return 0;
>>>>> +}
>>>>> +/*
>>>>> + * Initialization has to be done at init rather than presmp_init phase
>>>>> because
>>>>> + * the callback should execute only after the secondary CPUs are
>>>>> initially
>>>>> + * booted (in hotplug scenarios when the system state is not boot). On
>>>>> boot,
>>>>> + * the enabling of errata workarounds will be triggered by the boot CPU
>>>>> from
>>>>> + * start_xen().
>>>>> + */
>>>>> +__initcall(cpu_errata_notifier_init);
>>>>> +
>>>>>/*
>>>>> * Local variables:
>>>>> * mode: C
>>>>> diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
>>>>> index 525b45e22f..dd30f0d29c 100644
>>>>> --- a/xen/arch/arm/cpufeature.c
>>>>> +++ b/xen/arch/arm/cpufeature.c
>>>>> @@ -68,6 +68,29 @@ void __init enable_cpu_capabilities(const struct
>>>>> arm_cpu_capabilities *caps)
>>>>>}
&g

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi,

+Dario

On Wed, May 9, 2018 at 6:32 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 09/05/18 16:48, Mirela Simonovic wrote:
>>
>> Hi Julien,
>
>
> Hi Mirela,
>
>
>> On Mon, Apr 30, 2018 at 6:09 PM, Julien Grall <julien.gr...@arm.com>
>> wrote:
>>>
>>> Hi Mirela,
>>>
>>>
>>> On 27/04/18 18:12, Mirela Simonovic wrote:
>>>>
>>>>
>>>> On boot, enabling errata workarounds will be triggered by the boot CPU
>>>> from start_xen(). On CPU hotplug (non-boot scenario) this would not be
>>>> done. This patch adds the code required to enable errata workarounds
>>>> for a CPU being hotplugged after the system boots. This is triggered
>>>> using a notifier. If the CPU fails to enable the errata Xen will panic.
>>>> This is done because it is assumed that the CPU which is hotplugged
>>>> after the system/Xen boots, was initially hotplugged during the
>>>> system/Xen boot. Therefore, enabling errata workarounds should never
>>>> fail.
>>>>
>>>> Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
>>>>
>>>> ---
>>>> CC: Stefano Stabellini <sstabell...@kernel.org>
>>>> CC: Julien Grall <julien.gr...@arm.com>
>>>> ---
>>>>xen/arch/arm/cpuerrata.c | 35
>>>> +++
>>>>xen/arch/arm/cpufeature.c| 23 +++
>>>>xen/include/asm-arm/cpufeature.h |  1 +
>>>>3 files changed, 59 insertions(+)
>>>>
>>>> diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
>>>> index 1baa20654b..4040f781ec 100644
>>>> --- a/xen/arch/arm/cpuerrata.c
>>>> +++ b/xen/arch/arm/cpuerrata.c
>>>> @@ -5,6 +5,8 @@
>>>>#include 
>>>>#include 
>>>>#include 
>>>> +#include 
>>>> +#include 
>>>>#include 
>>>>#include 
>>>>#include 
>>>> @@ -349,6 +351,39 @@ void __init enable_errata_workarounds(void)
>>>>enable_cpu_capabilities(arm_errata);
>>>>}
>>>>+static int cpu_errata_callback(
>>>> +struct notifier_block *nfb, unsigned long action, void *hcpu)
>>>> +{
>>>> +switch ( action )
>>>> +{
>>>> +case CPU_STARTING:
>>>> +enable_nonboot_cpu_caps(arm_errata);
>>>> +break;
>>>> +default:
>>>> +break;
>>>> +}
>>>> +
>>>> +return NOTIFY_DONE;
>>>> +}
>>>> +
>>>> +static struct notifier_block cpu_errata_nfb = {
>>>> +.notifier_call = cpu_errata_callback,
>>>> +};
>>>> +
>>>> +static int __init cpu_errata_notifier_init(void)
>>>> +{
>>>> +register_cpu_notifier(_errata_nfb);
>>>> +return 0;
>>>> +}
>>>> +/*
>>>> + * Initialization has to be done at init rather than presmp_init phase
>>>> because
>>>> + * the callback should execute only after the secondary CPUs are
>>>> initially
>>>> + * booted (in hotplug scenarios when the system state is not boot). On
>>>> boot,
>>>> + * the enabling of errata workarounds will be triggered by the boot CPU
>>>> from
>>>> + * start_xen().
>>>> + */
>>>> +__initcall(cpu_errata_notifier_init);
>>>> +
>>>>/*
>>>> * Local variables:
>>>> * mode: C
>>>> diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
>>>> index 525b45e22f..dd30f0d29c 100644
>>>> --- a/xen/arch/arm/cpufeature.c
>>>> +++ b/xen/arch/arm/cpufeature.c
>>>> @@ -68,6 +68,29 @@ void __init enable_cpu_capabilities(const struct
>>>> arm_cpu_capabilities *caps)
>>>>}
>>>>}
>>>>+/* Run through the enabled capabilities and enable() them on the
>>>> calling CPU */
>>>> +void enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *caps)
>>>> +{
>>>> +ASSERT(system_state != SYS_STATE_boot);
>>>> +
>>>> +for ( ; caps->matches; caps++ )
>>>> +{
>>>> +if ( !cpus_have_cap(caps->capability) )
>>>> +continue;
&g

Re: [Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

Hi Julien,

On Mon, Apr 30, 2018 at 6:09 PM, Julien Grall <julien.gr...@arm.com> wrote:
> Hi Mirela,
>
>
> On 27/04/18 18:12, Mirela Simonovic wrote:
>>
>> On boot, enabling errata workarounds will be triggered by the boot CPU
>> from start_xen(). On CPU hotplug (non-boot scenario) this would not be
>> done. This patch adds the code required to enable errata workarounds
>> for a CPU being hotplugged after the system boots. This is triggered
>> using a notifier. If the CPU fails to enable the errata Xen will panic.
>> This is done because it is assumed that the CPU which is hotplugged
>> after the system/Xen boots, was initially hotplugged during the
>> system/Xen boot. Therefore, enabling errata workarounds should never
>> fail.
>>
>> Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
>>
>> ---
>> CC: Stefano Stabellini <sstabell...@kernel.org>
>> CC: Julien Grall <julien.gr...@arm.com>
>> ---
>>   xen/arch/arm/cpuerrata.c | 35
>> +++
>>   xen/arch/arm/cpufeature.c| 23 +++
>>   xen/include/asm-arm/cpufeature.h |  1 +
>>   3 files changed, 59 insertions(+)
>>
>> diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
>> index 1baa20654b..4040f781ec 100644
>> --- a/xen/arch/arm/cpuerrata.c
>> +++ b/xen/arch/arm/cpuerrata.c
>> @@ -5,6 +5,8 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>> +#include 
>>   #include 
>>   #include 
>>   #include 
>> @@ -349,6 +351,39 @@ void __init enable_errata_workarounds(void)
>>   enable_cpu_capabilities(arm_errata);
>>   }
>>   +static int cpu_errata_callback(
>> +struct notifier_block *nfb, unsigned long action, void *hcpu)
>> +{
>> +switch ( action )
>> +{
>> +case CPU_STARTING:
>> +enable_nonboot_cpu_caps(arm_errata);
>> +break;
>> +default:
>> +break;
>> +}
>> +
>> +return NOTIFY_DONE;
>> +}
>> +
>> +static struct notifier_block cpu_errata_nfb = {
>> +.notifier_call = cpu_errata_callback,
>> +};
>> +
>> +static int __init cpu_errata_notifier_init(void)
>> +{
>> +register_cpu_notifier(_errata_nfb);
>> +return 0;
>> +}
>> +/*
>> + * Initialization has to be done at init rather than presmp_init phase
>> because
>> + * the callback should execute only after the secondary CPUs are
>> initially
>> + * booted (in hotplug scenarios when the system state is not boot). On
>> boot,
>> + * the enabling of errata workarounds will be triggered by the boot CPU
>> from
>> + * start_xen().
>> + */
>> +__initcall(cpu_errata_notifier_init);
>> +
>>   /*
>>* Local variables:
>>* mode: C
>> diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
>> index 525b45e22f..dd30f0d29c 100644
>> --- a/xen/arch/arm/cpufeature.c
>> +++ b/xen/arch/arm/cpufeature.c
>> @@ -68,6 +68,29 @@ void __init enable_cpu_capabilities(const struct
>> arm_cpu_capabilities *caps)
>>   }
>>   }
>>   +/* Run through the enabled capabilities and enable() them on the
>> calling CPU */
>> +void enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *caps)
>> +{
>> +ASSERT(system_state != SYS_STATE_boot);
>> +
>> +for ( ; caps->matches; caps++ )
>> +{
>> +if ( !cpus_have_cap(caps->capability) )
>> +continue;
>> +
>> +if ( caps->enable )
>> +{
>> +/*
>> + * Since the CPU has enabled errata workarounds on boot, it
>> should
>
>
> This function is not really about errata, it is about capabilities. Errata
> is just a sub-category of them.
>

I've fixed the comment, thanks.

>> + * never fail to enable them here.
>> + */
>> +if ( caps->enable((void *)caps) )
>> +panic("CPU%u failed to enable capability %u\n",
>> +  smp_processor_id(), caps->capability);
>
>
> We should really avoid to use panic(...) if this is something the system can
> survive. In that specific case, it would only affect the current CPU. So it
> would be better to return an error and let the caller decide what to do.
>

I need to emphasize two points:
1) I don't see how is this different compared to PSCI CPU OFF where we
do panic. Essentially, in both cases the system will not be able to
use tha

Re: [Xen-devel] [PATCH v2 07/10] xen/arm: Release maintenance interrupt when CPU is hot-unplugged

[Mirela Simonovic]

Hi Julien,

On Wed, May 9, 2018 at 1:01 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 09/05/18 11:10, Mirela Simonovic wrote:
>>
>> On Fri, Apr 27, 2018 at 5:12 PM, Julien Grall <julien.gr...@arm.com>
>> wrote:
>>>
>>> On 27/04/18 15:38, Mirela Simonovic wrote:
>>>>
>>>> On Fri, Apr 27, 2018 at 4:15 PM, Tim Deegan <t...@xen.org> wrote:
>>>>>
>>>>> At 10:28 +0100 on 27 Apr (1524824906), Julien Grall wrote:
>>>>>>
>>>>>> On 26/04/18 15:23, Tim Deegan wrote:
>>>>>>>
>>>>>>> At 11:08 +0100 on 26 Apr (1524740921), Julien Grall wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On 20/04/18 13:25, Mirela Simonovic wrote:
>>>
>>> While I guess this code makes no harm, it does not do what is expected
>>> (i.e
>>> draining the interrupt). I can't see any reason to keep wrong code, we
>>> should really aim to have code that match the architecture. And better to
>>> fix it when we discover the problem rather than waiting until we
>>> rediscovered it later.
>>>
>>> So at least a patch to update the code/comment should be done.
>>>
>>
>> I don't feel comfortable removing these 3 lines because I have no way
>> to test and guarantee that the change will not introduce any issues.
>
>
> The work you are doing (suspend/resume, hotplug) is not easy to test and
> very subtle to get it right. Testing can only uncover obvious bug on your
> platform. IMHO, we can only rely on the specifications (ARM ARM, PSCI...)
> and extensive review of the series (and the code around).
>
>> However, if despite all you really want me to remove these lines
>> within this series I don't have a problem doing that in a separate
>> patch. Please just confirm the plan.
>
>
> I am quite confident that this code should not be there or at least not in
> its current form.
>

Do you want me to include the removal of that code in this series or not?

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2 07/10] xen/arm: Release maintenance interrupt when CPU is hot-unplugged

[Mirela Simonovic]

Hi Julien,

On Fri, Apr 27, 2018 at 5:12 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 27/04/18 15:38, Mirela Simonovic wrote:
>>
>> Hi,
>>
>> On Fri, Apr 27, 2018 at 4:15 PM, Tim Deegan <t...@xen.org> wrote:
>>>
>>> Hi,
>>>
>>> At 10:28 +0100 on 27 Apr (1524824906), Julien Grall wrote:
>>>>
>>>> On 26/04/18 15:23, Tim Deegan wrote:
>>>>>
>>>>> At 11:08 +0100 on 26 Apr (1524740921), Julien Grall wrote:
>>>>>>>>>>
>>>>>>>>>> On 20/04/18 13:25, Mirela Simonovic wrote:
>>>>>>>>
>>>>>>>> This looks a bit weird. AFAIU, if you disable the CPU interface,
>>>>>>>> then you
>>>>>>>> should never receive interrupt after. So why would you re-enable
>>>>>>>> them?
>>>>>>>>
>>>>>>>> I realize the code in __cpu_disbale do that, but this looks quite
>>>>>>>> wrong to
>>>>>>>> me. There are no way to receive queued timer interrupt afterwards.
>>>>>>>>
>>>>>>>
>>>>>>> That is what I took as a reference, but I asked myself the same.
>>>>>>> There is (extremely small, but it exists) time window between
>>>>>>> disabling irq locally and disabling CPU interface. An interrupt
>>>>>>> received in that time window would propagate to the CPU but I'm not
>>>>>>> sure would happen after the GIC CPU interface is disabled and
>>>>>>> interrupts are locally enabled. That is the only explanation I can
>>>>>>> come up with, although I believe the answer is nothing. Since you're
>>>>>>> at ARM you could check this internally.
>>>>>>
>>>>>>
>>>>>> Speaking with Andre (in CC), the GIC CPU interface may have forwarded
>>>>>> an
>>>>>> interrupt to the processor before it gets disabled. So when the
>>>>>> interrupt will be re-enabled, the processor will jump to the interrupt
>>>>>> exception entry.
>>>>>>
>>>>>> However, looking at the spec (4-78 in ARM IHI 0048B.b), Xen will read
>>>>>> a
>>>>>> spurious interrupt ID from GICC_IAR. So I am not sure what the point
>>>>>> of
>>>>>> that code. It looks like it has been taken from x86, but some bits are
>>>>>> missing.
>>>>>>
>>>>>> AFAIU, x86 will only suspend the timer afterwards (see time_suspend).
>>>>>> I
>>>>>> am not fully sure why this code is there on Arm. Whether we expect a
>>>>>> timer interrupt to come up. Stefano, Tim, do you have any insight on
>>>>>> that code?
>>>>>
>>>>>
>>>>> Sorry, no.  I pretty clearly copied this logic from x86, which copied
>>>>> it directly from Linux at some point in the past.  I don't know why
>>>>> x86 does it this way, and I haven't dived into linux to find out. :)
>>>>> But draining the outstanding IRQs seems like a polite thing to do if
>>>>> you're ever going to re-enable this CPU (at least without resetting
>>>>> it first).
>>>>
>>>>
>>>> I am not entirely sure what you mean by draining, do you mean they will
>>>> serviced by Xen? If so, what kind of interrupts do you expect to be
>>>> serviced (e.g PPI, SPIs) ?
>>>
>>>
>>> All I mean is, when you disable the GICC (or APIC, or whatever), you
>>> know that it won't send any more interrupts to the CPU.  But you would
>>> like to also be certain that any interrupts it already sent to the CPU
>>> get processed now.  Otherwise, if you bring the CPU up again later
>>> that interrupt could still be there.  Better to get it out of the way
>>> now, right?
>>>
>>> AIUI that's what x86 is doing by re-enabling interrupts and waiting a
>>> bit, which seems a bit crude but OK.  ARM could maybe do the same
>>> thing by disabling GICC, dsb(), then disable interrupts.  But I don't
>>> understand the interface between GICD, GICC and CPU well enough to
>>> reason about it properly.
>>>
>>> It's also possible that there's some subtlety of the timer interrupt
>>> handling that I don't know about -- I _thin

Re: [Xen-devel] [PATCH v3 05/10] xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume

[Mirela Simonovic]

Hi Julien,

On Tue, May 8, 2018 at 4:14 PM, Julien Grall <julien.gr...@arm.com> wrote:
>
>
> On 07/05/18 15:55, Mirela Simonovic wrote:
>>
>> Hi Julien,
>
>
> Hi Mirela,
>
>> On Mon, Apr 30, 2018 at 4:47 PM, Julien Grall <julien.gr...@arm.com>
>> wrote:
>>>
>>> On 27/04/18 18:12, Mirela Simonovic wrote:
>>>>
>>>>printk("P2M: %d levels with order-%d root, VTCR 0x%lx\n",
>>>> -   4 - P2M_ROOT_LEVEL, P2M_ROOT_ORDER, val);
>>>> +   4 - P2M_ROOT_LEVEL, P2M_ROOT_ORDER, vtcr);
>>>>  p2m_vmid_allocator_init();
>>>>  /* It is not allowed to concatenate a level zero root */
>>>>BUG_ON( P2M_ROOT_LEVEL == 0 && P2M_ROOT_ORDER > 0 );
>>>> -setup_virt_paging_one((void *)val);
>>>> -smp_call_function(setup_virt_paging_one, (void *)val, 1);
>>>> +setup_virt_paging_one(NULL);
>>>> +smp_call_function(setup_virt_paging_one, NULL, 1);
>>>> +}
>>>> +
>>>> +static int cpu_virt_paging_callback(
>>>> +struct notifier_block *nfb, unsigned long action, void *hcpu)
>>>
>>>
>>>
>>> The indentation looks wrong.
>>>
>>
>> Editor indented this for me and it looks the same as in other places
>> where a notifier is defined. I did
>> grep -r "struct notifier_block \*nfb,"
>> to check. It looks weird but seems correct
>
>
> Indeed, I am not sure why it is done like that for notifiers. I can't see
> any reason to split like that given the first parameter can fit on the first
> line without hitting the 80 columns.
>
> So I would much prefer if we follow Xen coding style:
>
> static int cpu_virt_paging_callback(struct notifier_block *nfb,
> unsigned long action,
> void *hcpu);
>

Please just one more clarification: why did you split line after 2nd
argument? 3rd argument could fit in 80 chars.
The only coding style I found is in CODING_STYLE file, which doesn't
specify that much details - I'm just trying to understand where to
find more info in order to avoid coding style-related iterations in
future. Is there any other source specifying coding style for Xen?

Thanks,
Mirela

> Cheers,
>
> --
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 08/10] xen/arm: Release timer interrupts when CPU is hot-unplugged

[Mirela Simonovic]

Hi Julien,

On Mon, Apr 30, 2018 at 5:58 PM, Julien Grall <julien.gr...@arm.com> wrote:
> Hi,
>
>
> On 27/04/18 18:12, Mirela Simonovic wrote:
>>
>> When a CPU is hot-unplugged timer interrupts have to be released
>> in order to free the memory that was allocated when the interrupts
>> were requested (using request_irq()). The request_irq is called
>> for each timer interrupt when the CPU gets hotplugged
>> (start_secondary->init_timer_interrupt->request_irq).
>> With this patch the timer interrupts will be released when the
>> newly added callback receives CPU_DYING event.
>>
>> Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
>>
>> ---
>> CC: Stefano Stabellini <sstabell...@kernel.org>
>> CC: Julien Grall <julien.gr...@arm.com>
>> ---
>> Changes in v3:
>> -Trigger releasing of timer interrupts using notifiers
>> ---
>>   xen/arch/arm/time.c | 39 +++
>>   1 file changed, 39 insertions(+)
>>
>> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
>> index c11fcfeadd..c7317e4639 100644
>> --- a/xen/arch/arm/time.c
>> +++ b/xen/arch/arm/time.c
>> @@ -29,6 +29,8 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>> +#include 
>>   #include 
>>   #include 
>>   #include 
>> @@ -312,6 +314,17 @@ void init_timer_interrupt(void)
>>   check_timer_irq_cfg(timer_irq[TIMER_PHYS_NONSECURE_PPI],
>> "NS-physical");
>>   }
>>   +/*
>> + * Revert actions done in init_timer_interrupt that are required to
>> properly
>> + * disable this CPU.
>> + */
>> +static void deinit_timer_interrupt(void)
>> +{
>
>
> Any reason to not disable the timer here? But I think we need to finish the
> discussion on the previous series regarding the purpose of the mdelay before
> going further with that patch. See patch v2 7/10. I would also appreciate
> answer to my question there.
>

I just missed it. Will add disabling timers.

>
>> +release_irq(timer_irq[TIMER_HYP_PPI], NULL);
>> +release_irq(timer_irq[TIMER_VIRT_PPI], NULL);
>> +release_irq(timer_irq[TIMER_PHYS_NONSECURE_PPI], NULL);
>> +}
>> +
>>   /* Wait a set number of microseconds */
>>   void udelay(unsigned long usecs)
>>   {
>> @@ -340,6 +353,32 @@ void domain_set_time_offset(struct domain *d, int64_t
>> time_offset_seconds)
>>   /* XXX update guest visible wallclock time */
>>   }
>>   +static int cpu_time_callback(
>> +struct notifier_block *nfb, unsigned long action, void *hcpu)
>> +{
>> +switch ( action )
>> +{
>> +case CPU_DYING:
>> +deinit_timer_interrupt();
>> +break;
>> +default:
>> +break;
>> +}
>> +
>> +return NOTIFY_DONE;
>> +}
>> +
>> +static struct notifier_block cpu_time_nfb = {
>> +.notifier_call = cpu_time_callback,
>> +};
>> +
>> +static int __init cpu_time_notifier_init(void)
>> +{
>> +register_cpu_notifier(_time_nfb);
>> +return 0;
>> +}
>> +__initcall(cpu_time_notifier_init);
>> +
>>   /*
>>* Local variables:
>>* mode: C
>>
>
> Cheers,
>
> --
> Julien Grall


Thanks,
Mirela

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 05/10] xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume

[Mirela Simonovic]

Hi Julien,


On Mon, Apr 30, 2018 at 4:47 PM, Julien Grall <julien.gr...@arm.com> wrote:
> Hi Mirela,
>
>
> On 27/04/18 18:12, Mirela Simonovic wrote:
>>
>> In existing code the virtual paging for non-boot CPUs is setup only on
>> boot.
>> The setup is triggered from start_xen() after all CPUs are brought online.
>> In other words, the initialization of VTCR_EL2 register is done out of the
>> cpu_up/start_secondary() control flow. However, the cpu_up flow is also
>> used
>> to hotplug non-boot CPUs on resume from suspend to RAM state, in which
>> case
>> the virtual paging will not be configured.
>>
>> With this patch the setting of paging is triggered from start_secondary()
>> function using cpu starting notifier (notify_cpu_starting() call). The
>> notifier is registered in p2m.c using init call. This has to be done with
>> init call rather than presmp_init because the registered callback depends
>> on vtcr configuration value which is setup after the presmp init calls
>> are executed (do_presmp_initcalls() called from start_xen()). Init calls
>> are executed after initial virtual paging is set up for all CPUs on boot.
>> This ensures that no callback can fire until the vtcr value is calculated
>> by Xen and virtual paging is set up initially for all CPUs. Also, this way
>> the virtual paging setup in boot scenario remains unchanged.
>>
>> It is assumed here that after the system completed the boot, CPUs that
>> execute start_secondary() were booted as well when the Xen itself was
>> booted. According to this assumption non-boot CPUs will always be
>> compliant
>> with the VTCR_EL2 value that was selected by Xen on boot.
>> Currently, there is no mechanism to trigger hotplugging of a CPU. This
>> will be added with the suspend to RAM support for ARM, where the hotplug
>> of non-boot CPUs will be triggered via enable_nonboot_cpus() call.
>>
>> Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
>>
>> ---
>> CC: Stefano Stabellini <sstabell...@kernel.org>
>> CC: Julien Grall <julien.gr...@arm.com>
>> ---
>> Changes in v2:
>> -Fix commit message
>> -Save configured VTCR_EL2 value into static variable that will be used
>>   by non-boot CPUs on hotplug
>> -Add setup_virt_paging_secondary() and invoke it from start_secondary()
>>   if that CPU has to setup virtual paging (if the system state is not
>> boot)
>>
>> Changes in v3:
>> -Fix commit message
>> -Remove setup_virt_paging_secondary() and use notifier to setup virtual
>>   paging for non-boot CPU on hotplug.
>> -In setup_virt_paging() use vtcr static variable instead of local val
>> -In setup_virt_paging_one() use vtcr static variable instead of provided
>>   argument
>> ---
>>   xen/arch/arm/p2m.c | 82
>> +-
>>   1 file changed, 62 insertions(+), 20 deletions(-)
>>
>> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
>> index d43c3aa896..98a1fe6de9 100644
>> --- a/xen/arch/arm/p2m.c
>> +++ b/xen/arch/arm/p2m.c
>> @@ -8,6 +8,8 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>> +#include 
>
>
> Please add them alphabetically.
>
>
>>   #include 
>>   #include 
>>   #include 
>> @@ -1451,24 +1453,17 @@ err:
>>   return page;
>>   }
>>   -static void __init setup_virt_paging_one(void *data)
>> +/* VTCR value to be configured by all CPUs. Set only once by the boot CPU
>> */
>> +static uint64_t __read_mostly vtcr;
>> +
>> +static void setup_virt_paging_one(void *data)
>>   {
>> -unsigned long val = (unsigned long)data;
>> -WRITE_SYSREG32(val, VTCR_EL2);
>> +WRITE_SYSREG32(vtcr, VTCR_EL2);
>>   isb();
>>   }
>> void __init setup_virt_paging(void)
>>   {
>> -/* Setup Stage 2 address translation */
>> -unsigned long val =
>> VTCR_RES1|VTCR_SH0_IS|VTCR_ORGN0_WBWA|VTCR_IRGN0_WBWA;
>> -
>> -#ifdef CONFIG_ARM_32
>> -printk("P2M: 40-bit IPA\n");
>> -p2m_ipa_bits = 40;
>> -val |= VTCR_T0SZ(0x18); /* 40 bit IPA */
>> -val |= VTCR_SL0(0x1); /* P2M starts at first level */
>> -#else /* CONFIG_ARM_64 */
>>   const struct {
>>   unsigned int pabits; /* Physical Address Size */
>>   unsigned int t0sz;   /* Desired T0SZ, minimum in comment */
>> @@ -1491,6 +1486,16 @@ void __init setup_virt_paging(void)
>>   unsigned int pa_range = 0x10; /* Larger than any

[Xen-devel] [PATCH v3 01/10] xen/arm64: Added handling of the trapped access to OSLSR register

[Mirela Simonovic]

Linux/dom0 accesses OSLSR register when saving CPU context during the
suspend procedure. Xen traps access to this register, but has no handling
for it. Consequently, Xen injects undef exception to linux, causing it to
crash. This patch adds handling of the trapped access to OSLSR as ro/raz.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>
Reviewed-by: Stefano Stabellini <sstabell...@kernel.org>
Acked-by: Julien Grall <julien.gr...@arm.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
- Commit message fix (arm64 related change instead of arm)
- Add Stefano's reviewed-by

Changes in v3:
- Added Julien's acked-by
---
 xen/arch/arm/arm64/vsysreg.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/arm64/vsysreg.c b/xen/arch/arm/arm64/vsysreg.c
index c57ac12503..8f80e1735e 100644
--- a/xen/arch/arm/arm64/vsysreg.c
+++ b/xen/arch/arm/arm64/vsysreg.c
@@ -57,13 +57,14 @@ void do_sysreg(struct cpu_user_regs *regs,
  * ARMv8 (DDI 0487A.d): D1-1509 Table D1-58
  *
  * Unhandled:
- *OSLSR_EL1
  *DBGPRCR_EL1
  */
 case HSR_SYSREG_OSLAR_EL1:
 return handle_wo_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
 case HSR_SYSREG_OSDLR_EL1:
 return handle_raz_wi(regs, regidx, hsr.sysreg.read, hsr, 1);
+case HSR_SYSREG_OSLSR_EL1:
+return handle_ro_raz(regs, regidx, hsr.sysreg.read, hsr, 1);
 
 /*
  * MDCR_EL2.TDA
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v3 02/10] xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)

[Mirela Simonovic]

Guests attempt to write into these registers on resume (for example Linux).
Without this patch a data abort exception will be raised to the guest.
This patch handles the write access by ignoring it, but only if the value
to be written is zero. This should be fine because reading these registers
is already handled as 'read as zero'.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v2:
- Write should be ignored only if the value to be written is zero
 (in v1 the write was ignored regardless of the value)

Changes in v3:
- Print warning only if the value to be written is not zero
---
 xen/arch/arm/vgic-v2.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 646d1f3d12..f6c11f1e41 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -485,6 +485,8 @@ static int vgic_v2_distr_mmio_write(struct vcpu *v, 
mmio_info_t *info,
 
 case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
 if ( dabt.size != DABT_WORD ) goto bad_width;
+if ( r == 0 )
+goto write_ignore_32;
 printk(XENLOG_G_ERR
"%pv: vGICD: unhandled word write %#"PRIregister" to 
ISACTIVER%d\n",
v, r, gicd_reg - GICD_ISACTIVER);
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v3 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes

[Mirela Simonovic]

This patch set contains fixes that are required as precondition for suspend to
RAM support, including the CPU hotplug which is required to suspend non-boot
CPUs.
The first two patches in this series:
1) xen/arm64: Added handling of the trapped access to OSLSR register
2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
patch set does not include the implementation of virtual PSCI system suspend
call that would allow guests to finalize their suspend procedures. This will
be submitted in the following series.

Remaining of the patches are related to enabling CPU hotplug for non-boot
CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for suspend
to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot CPUs
will be triggered with disable_nonboot_cpus(), while the hotplug is triggered
with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs could
be powered down/up on suspend/resume, respectively, if the underlying firmware
allows so. Calls to enable/disable_nonboot_cpus() functions currently do not
exist in Xen ARM code. This will be added with the suspend to RAM support for
ARM.

When non-boot pCPUs are hot-unplugged their interrupts are migrated to the boot
pCPU. This series also includes a fix that would restore the interrupts affinity
once non-boot pCPUs are hotplugged. Here only SPIs used by guests are covered.
Migration of Xen internal SPIs is not covered. According to my understanding
Xen internal SPIs are routed to the boot CPU which initializes the respective
devices.

The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
physical power down/up of non-boot CPUs). The testing requires additional
patches for issuing system suspend. These patches and instructions for testing
will be submitted later, when we get closer to the final version of the series.

---
Changes in v2:
-Rename cover-letter title and emphasize that 2 patches from this series are not
specific to CPU hotplug (my initial fault, splitting it now could be confusing)
-Fix cover-letter explanations
-Address all the issues and comments as discussed on mailing list for v1
-Add 3 patches to ensure that suspend/resume does not cause any memory leaks.
All the memory allocated when a CPU was hotplugged is now freed when the CPU is
hot-unplugged.
-Remove from the v1 series the patch which incorrectly dealt with an issue:
[PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
One solution to the issue addressed by the patch above is to add rcu_barrier()
prior to calling enable_nonboot_cpus() during the suspend. This is how it is
done in x86 suspend implementation. Until the discussion here
https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
doesn't conclude differently, I need to assume that adding rcu_barrier() prior
to calling enable_nonboot_cpus() as it is done for x86 is the right way to go.
Therefore, the fix to the issue will be part of the suspend to RAM series.

Changes in v3:
-Add acked-by where needed
-Fix CPU_OFF PSCI implementation (physical interface)
-Use notifiers to implement freeing memory and releasing interrupts on CPU
hotplug
-Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
hotplug
-Add enabling errata workarounds on CPU hotplug, also based on a notifier
-Remove patch:
[PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU only on 
boot

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Dario Faggioli <dfaggi...@suse.com>
---

Mirela Simonovic (10):
  xen/arm64: Added handling of the trapped access to OSLSR register
  xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
  xen/arm: Implement CPU_OFF PSCI call (physical interface)
  xen/arm: Remove __initdata and __init to enable CPU hotplug
  xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume
  xen/common: Restore IRQ affinity when hotplugging a pCPU
  xen/arm: Release maintenance interrupt when CPU is hot-unplugged
  xen/arm: Release timer interrupts when CPU is hot-unplugged
  xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug
  xen/arm: Enable errata for secondary CPU on hotplug after the boot

 xen/arch/arm/arm64/smpboot.c |  2 +-
 xen/arch/arm/arm64/vsysreg.c |  3 +-
 xen/arch/arm/cpuerrata.c | 35 +
 xen/arch/arm/cpufeature.c| 23 +++
 xen/arch/arm/gic.c   | 29 ++
 xen/arch/arm/irq.c   |  2 +-
 xen/arch/arm/p2m.c   | 82 ++--
 xen/arch/arm/processor.c |  2 +-
 xen/arch/arm/psci.c  | 13 +++
 xen/arch/arm/smpboot.c   | 40 +++-
 xen/arch/arm/time.c  | 39 +++
 xen/arch/arm/vgic-v2.c 

[Xen-devel] [PATCH v3 09/10] xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug

[Mirela Simonovic]

The memory allocated in setup_cpu_sibling_map() when a CPU is hotplugged
has to be freed when the CPU is hot-unplugged. This is done in
remove_cpu_sibling_map() and called when the CPU dies. The call to
remove_cpu_sibling_map() is made from a notifier callback when
CPU_DEAD event is received.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
Changes in v3:
-Use notifier to trigger remove_cpu_sibling_map() when the CPU dies.
---
 xen/arch/arm/smpboot.c | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ad1f6b751b..b833e3a754 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -89,6 +89,12 @@ static void setup_cpu_sibling_map(int cpu)
 cpumask_set_cpu(cpu, per_cpu(cpu_core_mask, cpu));
 }
 
+static void remove_cpu_sibling_map(int cpu)
+{
+free_cpumask_var(per_cpu(cpu_sibling_mask, cpu));
+free_cpumask_var(per_cpu(cpu_core_mask, cpu));
+}
+
 void __init
 smp_clear_cpu_maps (void)
 {
@@ -499,6 +505,34 @@ void __cpu_die(unsigned int cpu)
 smp_mb();
 }
 
+static int cpu_smpboot_callback(
+struct notifier_block *nfb, unsigned long action, void *hcpu)
+{
+unsigned int cpu = (unsigned long)hcpu;
+
+switch ( action )
+{
+case CPU_DEAD:
+remove_cpu_sibling_map(cpu);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_smpboot_nfb = {
+.notifier_call = cpu_smpboot_callback,
+};
+
+static int __init cpu_smpboot_notifier_init(void)
+{
+register_cpu_notifier(_smpboot_nfb);
+return 0;
+}
+__initcall(cpu_smpboot_notifier_init);
+
 /*
  * Local variables:
  * mode: C
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v3 10/10] xen/arm: Enable errata for secondary CPU on hotplug after the boot

[Mirela Simonovic]

On boot, enabling errata workarounds will be triggered by the boot CPU
from start_xen(). On CPU hotplug (non-boot scenario) this would not be
done. This patch adds the code required to enable errata workarounds
for a CPU being hotplugged after the system boots. This is triggered
using a notifier. If the CPU fails to enable the errata Xen will panic.
This is done because it is assumed that the CPU which is hotplugged
after the system/Xen boots, was initially hotplugged during the
system/Xen boot. Therefore, enabling errata workarounds should never
fail.

Signed-off-by: Mirela Simonovic <mirela.simono...@aggios.com>

---
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
---
 xen/arch/arm/cpuerrata.c | 35 +++
 xen/arch/arm/cpufeature.c| 23 +++
 xen/include/asm-arm/cpufeature.h |  1 +
 3 files changed, 59 insertions(+)

diff --git a/xen/arch/arm/cpuerrata.c b/xen/arch/arm/cpuerrata.c
index 1baa20654b..4040f781ec 100644
--- a/xen/arch/arm/cpuerrata.c
+++ b/xen/arch/arm/cpuerrata.c
@@ -5,6 +5,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -349,6 +351,39 @@ void __init enable_errata_workarounds(void)
 enable_cpu_capabilities(arm_errata);
 }
 
+static int cpu_errata_callback(
+struct notifier_block *nfb, unsigned long action, void *hcpu)
+{
+switch ( action )
+{
+case CPU_STARTING:
+enable_nonboot_cpu_caps(arm_errata);
+break;
+default:
+break;
+}
+
+return NOTIFY_DONE;
+}
+
+static struct notifier_block cpu_errata_nfb = {
+.notifier_call = cpu_errata_callback,
+};
+
+static int __init cpu_errata_notifier_init(void)
+{
+register_cpu_notifier(_errata_nfb);
+return 0;
+}
+/*
+ * Initialization has to be done at init rather than presmp_init phase because
+ * the callback should execute only after the secondary CPUs are initially
+ * booted (in hotplug scenarios when the system state is not boot). On boot,
+ * the enabling of errata workarounds will be triggered by the boot CPU from
+ * start_xen().
+ */
+__initcall(cpu_errata_notifier_init);
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c
index 525b45e22f..dd30f0d29c 100644
--- a/xen/arch/arm/cpufeature.c
+++ b/xen/arch/arm/cpufeature.c
@@ -68,6 +68,29 @@ void __init enable_cpu_capabilities(const struct 
arm_cpu_capabilities *caps)
 }
 }
 
+/* Run through the enabled capabilities and enable() them on the calling CPU */
+void enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *caps)
+{
+ASSERT(system_state != SYS_STATE_boot);
+
+for ( ; caps->matches; caps++ )
+{
+if ( !cpus_have_cap(caps->capability) )
+continue;
+
+if ( caps->enable )
+{
+/*
+ * Since the CPU has enabled errata workarounds on boot, it should
+ * never fail to enable them here.
+ */
+if ( caps->enable((void *)caps) )
+panic("CPU%u failed to enable capability %u\n",
+  smp_processor_id(), caps->capability);
+}
+}
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
index e557a095af..b14e226401 100644
--- a/xen/include/asm-arm/cpufeature.h
+++ b/xen/include/asm-arm/cpufeature.h
@@ -88,6 +88,7 @@ void update_cpu_capabilities(const struct 
arm_cpu_capabilities *caps,
  const char *info);
 
 void enable_cpu_capabilities(const struct arm_cpu_capabilities *caps);
+void enable_nonboot_cpu_caps(const struct arm_cpu_capabilities *caps);
 
 #endif /* __ASSEMBLY__ */
 
-- 
2.13.0


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel