Hi Dario,

On 21/09/2016 16:45, Dario Faggioli wrote:
On Wed, 2016-09-21 at 14:06 +0100, Julien Grall wrote:
(CC a couple of ARM folks)

Yay, thanks for this! :-)

I had few discussions and  more thought about big.LITTLE support in
The main goal of big.LITTLE is power efficiency by moving task
and been able to idle one cluster. All the solutions suggested
(including mine) so far, can be replicated by hand (except the VPIDR)
they are mostly an automatic way.

I'm sorry, how is this (going to be) handled in Linux? Is it that any
arbitrary task executing any arbitrary binary code can be run on both
big and LITTLE pcpus, depending on the scheduler's and energy
management's decisions?

This does not seem to match with what has been said at some point in
this thread... And if it's like that, how's that possible, if the
pcpus' ISAs are (even only slightly) different?

Right, at some point I mentioned that the set of errata and features will be different between processor.

However, it is possible to sanitize the feature registers to expose a common set to the guest. This is what is done in Linux at boot time, only the features common to all the CPUs will be enabled.

This allows a task to migrate between big and LITTLE CPUs seamlessly.

This will also remove the real
benefits of big.LITTLE because Xen will not be able to migrate vCPU
across cluster for power efficiency.

If we care about power efficiency, we would have to handle
big.LITTLE in Xen (i.e a guess would only see a kind of CPU).

Well, I'm a big fan of an approach that leaves the guests' scheduler
dumb about things like these (i.e., load balancing, energy efficiency,
etc), and hence puts Xen in charge. In fact, on a Xen system, it is
only Xen that has all the info necessary to make wise decisions (e.g.,
the load of the _whole_ host, the effect of any decisions on the
_whole_ host, etc).

But this case may be a LITTLE.bit ( :-PP ) different.

Anyway, I guess I'll way your reply to my question above before
commenting more.

This arise
quite few problem, nothing insurmountable, similar to migration
two platforms with different micro-architecture (e.g processors):
errata, features supported... The guest would have to know the union
all the errata (this is done so far via the MIDR, so we would a PV
to do it), and only the intersection of features would be exposed to
guest. This also means the scheduler would have to be modified to
power efficiency (not strictly necessary at the beginning).

I agree that a such solution would require some work to implement,
although Xen will have a better control of the energy consumption of

So the question here, is what do we want to achieve with big.LITTLE?

Just thinking out loud here. So, instead of "just", as George

 vcpuclass=["0-1:A35","2-5:A53", "6-7:A72"]

we can allow something like the following (note that I'm tossing out
random numbers next to the 'A's):

 vcpuclass = ["0-1:A35", "2-5:A53,A17", "6-7:A72,A24,A31", "12-13:A8"]

with the following meaning:
 - vcpus 0, 1 can only run on pcpus of class A35
 - vcpus 2,3,4,5 can run on pcpus of class A53 _and_ on pcpus of class
 - vcpus 6,7 can run on pcpus of class A72, A24, A31
 - vcpus 8,9,10,11 --since they're not mentioned, can run on pcpus of
   any class
 - vcpus 12,13 can only run on pcpus of class A8

This will set the "boundaries", for each vcpu. Then, within these
boundaries, once in the (Xen's) scheduler, we can implement whatever
complex/magic/silly logic we want, e.g.:
 - only use a pcpu of class A53 for vcpus that have an average load
   above 50%
 - only use a pcpu of class A31 if there are no idle pcpus of class A24
 - only use a pcpu of class A17 for a vcpu if the total system load
   divided by the vcpu ID give 42 as result
 - whatever

This allows us to achieve both the following goals:
 - allow Xen to take smart decisions, considering the load and the
   efficiency of the host as a whole
 - allow the guest to take smart decisions, like running lightweight
   tasks on low power vcpus (which then Xen will run on low
   power pcpus, at least on a properly configured system)

Of course this **requires** that, for instance, vcpu 6 must be able to
run on A72, A24 and A31 just fine, i.e., it must be possible for it to
block on I/O when executing on an A72 pcpu, and, later, after wakeup,
restart executing on an A24 pcpu.

With a bit of work in Xen, it would be possible to do move the vCPU between big and LITTLE cpus. As mentioned above, we could sanitize the features to only enable a common set. You can view the big.LITTLE problem as a local live migration between two kind of CPUs.

In your suggestion you don't mention what would happen if the guest configuration does not contain the affinity. Does it mean the vCPU will be scheduled anywhere? A pCPU/class will be chosen randomly?

To be honest, I quite like this idea. It could be used as soft/hard affinity for the moment. But can be extended in the future if/when the scheduler gain knowledge of power efficiency and vCPU can migrate between big and LITTLE.


Julien Grall

Xen-devel mailing list

Reply via email to