Re: [Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-09-28 Thread George Dunlap 
On 17/08/16 18:19, Dario Faggioli wrote:
> This is done by means of the "usual" two steps loop:
>  - soft affinity balance step;
>  - hard affinity balance step.
> 
> The entire logic implemented in runq_tickle() is
> applied, during the first step, considering only the
> CPUs in the vcpu's soft affinity. In the second step,
> we fall back to use all the CPUs from its hard
> affinity (as it is doing now, without this patch).
> 
> Signed-off-by: Dario Faggioli 
> Signed-off-by: Justin T. Weaver 
> ---
> Cc: George Dunlap 
> Cc: Anshul Makkar 
> ---
>  xen/common/sched_credit2.c |  243 
> 
>  1 file changed, 157 insertions(+), 86 deletions(-)
> 
> diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
> index 0d83bd7..3aef1b4 100644
> --- a/xen/common/sched_credit2.c
> +++ b/xen/common/sched_credit2.c
> @@ -902,6 +902,42 @@ __runq_remove(struct csched2_vcpu *svc)
>  list_del_init(>runq_elem);
>  }
>  
> +/*
> + * During the soft-affinity step, only actually preempt someone if
> + * he does not have soft-affinity with cpu (while we have).
> + *
> + * BEWARE that this uses cpumask_scratch, trowing away what's in there!
> + */
> +static inline bool_t soft_aff_check_preempt(unsigned int bs, unsigned int 
> cpu)
> +{
> +struct csched2_vcpu * cur = CSCHED2_VCPU(curr_on_cpu(cpu));
> +
> +/*
> + * If we're doing hard-affinity, always check whether to preempt cur.
> + * If we're doing soft-affinity, but cur doesn't have one, check as well.
> + */
> +if ( bs == BALANCE_HARD_AFFINITY ||
> + !has_soft_affinity(cur->vcpu, cur->vcpu->cpu_hard_affinity) )
> +return 1;
> +
> +/*
> + * We're doing soft-affinity, and we know that the current vcpu on cpu
> + * has a soft affinity. We now want to know whether cpu itself is in
> + * such affinity. In fact, since we now that new (in runq_tickle()) is:

This is a bit confusing.  I think you mean, "We know that the vcpu we
want to place has soft affinity with the target cpu; now we want to know
whether the vcpu running on the target cpu has soft affinity with that
cpu or not."

> + *  - if cpu is not in cur's soft-affinity, we should indeed check to
> + *see whether new should preempt cur. If that will be the case, that
> + *would be an improvement wrt respecting soft affinity;
> + *  - if cpu is in cur's soft-affinity, we leave it alone and (in
> + *runq_tickle()) move on to another cpu. In fact, we don't want to
> + *be too harsh with someone which is running within its 
> soft-affinity.
> + *This is safe because later, if we don't fine anyone else during the
> + *soft-affinity step, we will check cpu for preemption anyway, when
> + *doing hard-affinity.

But by doing this, isn't it actually more likely that we'll end up
somewhere outside our soft affinity, even though there are cpus inside
our soft affinity where we have higher credit?

It would be nice if we could pre-empt somebody outside their soft
affinity before pre-empting somebody inside their soft affinity.

It occurs to me -- in the normal case, the number of cpus involved here
should be lower than 8, and often lower than that.  Rather than loop
around twice, with potentially two "inner" loops, would it make sense
just to sweep through the hard affinity once, calculating a "score" that
factored in the different things we want to factor in, and then choosing
the highest score (if any)?

Something like the attached (compile-tested only)?

 -George


From 93346e02da5def9c1bca502e0e47aa8be9b3f2a6 Mon Sep 17 00:00:00 2001
From: Dario Faggioli 
Date: Thu, 15 Sep 2016 12:35:05 +0100
Subject: [PATCH] xen: credit2: soft-affinity awareness in runq_tickle()

Rather than the usual two-step loop, after first checking for idlers,
we scan through each cpu in the runqueue and find a "score" for the
utility of tickling each cpu.

FIXME - needs filling out. :-)

Signed-off-by: George Dunlap 
---
 xen/common/sched_credit2.c | 126 ++---
 1 file changed, 84 insertions(+), 42 deletions(-)

diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 0d83bd7..36acf82 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -904,6 +904,64 @@ __runq_remove(struct csched2_vcpu *svc)
 
 void burn_credits(struct csched2_runqueue_data *rqd, struct csched2_vcpu *, s_time_t);
 
+/* 
+ * Score to preempt the target cpu.  Return a negative number if the
+ * credit isn't high enough; if it is, favor preemptions in this
+ * order:
+ * - cpu is in new's soft affinity, not in cur's soft affinity
+ * - cpu is in new's soft affinity and cur's soft affinity
+ * - cpu is not in new's soft affinity
+ * - Within the same class, the highest difference of credit

Re: [Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-09-07 Thread Dario Faggioli 
On Wed, 2016-09-07 at 14:24 +0100, anshul makkar wrote:
> On 05/09/16 15:55, Dario Faggioli wrote:
> > On Thu, 2016-09-01 at 11:52 +0100, anshul makkar wrote:
> > So, yes, we know already that it's running in a cpu at least from
> > its
> > hard affinity, what is it exactly that you are not understanding?
> If I put it simply ,  can  (X being a vcpu)
> x {soft affinity pcpus} Intersect x { hard affinity pcpu} -> be Null
> or 
> disjoint set ?
> and
> 
The user can setup things such that:

 soft-affinity{X} intersection hard-affinity{X} = O

but this, here inside the scheduler, is considered like X does not have
any soft-affinity at all, i.e., only X's hard-affinity is considered,
and all the balancing steps and operations and consideration related to
soft-affinity are ignored/skipped.

That's because it's absolutely pointless to try figure out where to
execute X, among the set of the pCPUs it prefers to run on, if it
_can't_ actually run on any pCPU from that same set.

So the answer to your question is: "it seems to be possible for the
intersection to be void, but in practise, it is not." :-)

> x{runnable pcpu} intersect (x{hard affinity pcpu} union x{soft
> affinity 
> pcpu} ) -> be null or disjoint ??
> 
I still don't get this. In particular, I'm not sure what
'x{runnable pcpu}' is. Also the union of a vcpu's soft and hard
affinity is never done (it's, as explained above, the intersection that
counts).

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-09-07 Thread anshul makkar 

On 05/09/16 15:55, Dario Faggioli wrote:

On Thu, 2016-09-01 at 11:52 +0100, anshul makkar wrote:

On 17/08/16 18:19, Dario Faggioli wrote:


+/*
+ * We're doing soft-affinity, and we know that the current
vcpu on cpu
+ * has a soft affinity. We now want to know whether cpu itself
is in

Please can you explain the above statment. If the vcpu has soft
affinity
and its currently executing, doesn;t it always means that its running
on
one of the pcpu which is there in its soft affinity or hard affinity?


A vcpu will always run on a pcpu from its own hard-affinity (that's the
definition of hard-affinity).

On the other hand, a vcpu will, most of the time, run on a cpu from its
own soft affinity, but can run on a cpu that is in its hard-affinity,
but *IS NOT* in its soft-affinity.

That's the definition of soft-affinity: the scheduler will try to run
it there, but it that can't happen, it will run it will run it outside
of it (but still within its hard-affinity, of course).

So, yes, we know already that it's running in a cpu at least from its
hard affinity, what is it exactly that you are not understanding?


If I put it simply ,  can  (X being a vcpu)
x {soft affinity pcpus} Intersect x { hard affinity pcpu} -> be Null or 
disjoint set ?

and

x{runnable pcpu} intersect (x{hard affinity pcpu} union x{soft affinity 
pcpu} ) -> be null or disjoint ??





+ * such affinity. In fact, since we now that new (in
runq_tickle()) is

Typo:   * such affinity. In fact, since now we know that new (in
runq_tickle()) is


Thanks. :-)



+ */



Regards,
Dario


Anshul Makkar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-09-05 Thread Dario Faggioli 
On Thu, 2016-09-01 at 11:52 +0100, anshul makkar wrote:
> On 17/08/16 18:19, Dario Faggioli wrote:
> > 
> > +/*
> > + * We're doing soft-affinity, and we know that the current
> > vcpu on cpu
> > + * has a soft affinity. We now want to know whether cpu itself
> > is in
> Please can you explain the above statment. If the vcpu has soft
> affinity 
> and its currently executing, doesn;t it always means that its running
> on 
> one of the pcpu which is there in its soft affinity or hard affinity?
>
A vcpu will always run on a pcpu from its own hard-affinity (that's the
definition of hard-affinity).

On the other hand, a vcpu will, most of the time, run on a cpu from its
own soft affinity, but can run on a cpu that is in its hard-affinity,
but *IS NOT* in its soft-affinity.

That's the definition of soft-affinity: the scheduler will try to run
it there, but it that can't happen, it will run it will run it outside
of it (but still within its hard-affinity, of course).

So, yes, we know already that it's running in a cpu at least from its
hard affinity, what is it exactly that you are not understanding?

> > + * such affinity. In fact, since we now that new (in
> > runq_tickle()) is
> Typo:   * such affinity. In fact, since now we know that new (in 
> runq_tickle()) is
>
Thanks. :-)

> > + *  - if cpu is not in cur's soft-affinity, we should indeed
> > check to
> > + *see whether new should preempt cur. If that will be the
> > case, that
> > + *would be an improvement wrt respecting soft affinity;
> > + *  - if cpu is in cur's soft-affinity, we leave it alone and
> > (in
> > + *runq_tickle()) move on to another cpu. In fact, we don't
> > want to
> > + *be too harsh with someone which is running within its
> > soft-affinity.
> > + *This is safe because later, if we don't fine anyone else
> > during the
> > + *soft-affinity step, we will check cpu for preemption
> > anyway, when
> > + *doing hard-affinity.
> > + */
>
Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-09-01 Thread anshul makkar 

On 17/08/16 18:19, Dario Faggioli wrote:

This is done by means of the "usual" two steps loop:
  - soft affinity balance step;
  - hard affinity balance step.

The entire logic implemented in runq_tickle() is
applied, during the first step, considering only the
CPUs in the vcpu's soft affinity. In the second step,
we fall back to use all the CPUs from its hard
affinity (as it is doing now, without this patch).

Signed-off-by: Dario Faggioli 
Signed-off-by: Justin T. Weaver 
---
Cc: George Dunlap 
Cc: Anshul Makkar 
---
  xen/common/sched_credit2.c |  243 
  1 file changed, 157 insertions(+), 86 deletions(-)

diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 0d83bd7..3aef1b4 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -902,6 +902,42 @@ __runq_remove(struct csched2_vcpu *svc)
  list_del_init(>runq_elem);
  }

+/*
+ * During the soft-affinity step, only actually preempt someone if
+ * he does not have soft-affinity with cpu (while we have).
+ *
+ * BEWARE that this uses cpumask_scratch, trowing away what's in there!

Typo:* BEWARE that this uses cpumask_scratch, throwing away what's in there!


+ */
+static inline bool_t soft_aff_check_preempt(unsigned int bs, unsigned int cpu)
+{
+struct csched2_vcpu * cur = CSCHED2_VCPU(curr_on_cpu(cpu));
+
+/*
+ * If we're doing hard-affinity, always check whether to preempt cur.
+ * If we're doing soft-affinity, but cur doesn't have one, check as well.
+ */
+if ( bs == BALANCE_HARD_AFFINITY ||
+ !has_soft_affinity(cur->vcpu, cur->vcpu->cpu_hard_affinity) )
+return 1;
+
+/*
+ * We're doing soft-affinity, and we know that the current vcpu on cpu
+ * has a soft affinity. We now want to know whether cpu itself is in
Please can you explain the above statment. If the vcpu has soft affinity 
and its currently executing, doesn;t it always means that its running on 
one of the pcpu which is there in its soft affinity or hard affinity?

+ * such affinity. In fact, since we now that new (in runq_tickle()) is
Typo:   * such affinity. In fact, since now we know that new (in 
runq_tickle()) is

+ *  - if cpu is not in cur's soft-affinity, we should indeed check to
+ *see whether new should preempt cur. If that will be the case, that
+ *would be an improvement wrt respecting soft affinity;
+ *  - if cpu is in cur's soft-affinity, we leave it alone and (in
+ *runq_tickle()) move on to another cpu. In fact, we don't want to
+ *be too harsh with someone which is running within its soft-affinity.
+ *This is safe because later, if we don't fine anyone else during the
+ *soft-affinity step, we will check cpu for preemption anyway, when
+ *doing hard-affinity.
+ */
+affinity_balance_cpumask(cur->vcpu, BALANCE_SOFT_AFFINITY, 
cpumask_scratch);
+return !cpumask_test_cpu(cpu, cpumask_scratch);
+}
+
  void burn_credits(struct csched2_runqueue_data *rqd, struct csched2_vcpu *, 
s_time_t);

  /*
@@ -925,7 +961,7 @@ runq_tickle(const struct scheduler *ops, struct 
csched2_vcpu *new, s_time_t now)
  {
  int i, ipid = -1;
  s_time_t lowest = (1<<30);
-unsigned int cpu = new->vcpu->processor;
+unsigned int bs, cpu = new->vcpu->processor;
  struct csched2_runqueue_data *rqd = RQD(ops, cpu);
  cpumask_t mask;
  struct csched2_vcpu * cur;
@@ -947,109 +983,144 @@ runq_tickle(const struct scheduler *ops, struct 
csched2_vcpu *new, s_time_t now)
  (unsigned char *));
  }

-/*
- * First of all, consider idle cpus, checking if we can just
- * re-use the pcpu where we were running before.
- *
- * If there are cores where all the siblings are idle, consider
- * them first, honoring whatever the spreading-vs-consolidation
- * SMT policy wants us to do.
- */
-if ( unlikely(sched_smt_power_savings) )
-cpumask_andnot(, >idle, >smt_idle);
-else
-cpumask_copy(, >smt_idle);
-cpumask_and(, , new->vcpu->cpu_hard_affinity);
-i = cpumask_test_or_cycle(cpu, );
-if ( i < nr_cpu_ids )
+for_each_affinity_balance_step( bs )
  {
-SCHED_STAT_CRANK(tickled_idle_cpu);
-ipid = i;
-goto tickle;
-}
+/*
+ * First things first: if we are at the first (soft affinity) step,
+ * but new doesn't have a soft affinity, skip this step.
+ */
+if ( bs == BALANCE_SOFT_AFFINITY &&
+ !has_soft_affinity(new->vcpu, new->vcpu->cpu_hard_affinity) )
+continue;

-/*
- * If there are no fully idle cores, check all idlers, after
- * having filtered out pcpus that have been tickled but haven't
- * gone through the scheduler yet.
- */
-cpumask_andnot(, >idle, >tickled);
-

[Xen-devel] [PATCH 17/24] xen: credit2: soft-affinity awareness in runq_tickle()

2016-08-17 Thread Dario Faggioli 
This is done by means of the "usual" two steps loop:
 - soft affinity balance step;
 - hard affinity balance step.

The entire logic implemented in runq_tickle() is
applied, during the first step, considering only the
CPUs in the vcpu's soft affinity. In the second step,
we fall back to use all the CPUs from its hard
affinity (as it is doing now, without this patch).

Signed-off-by: Dario Faggioli 
Signed-off-by: Justin T. Weaver 
---
Cc: George Dunlap 
Cc: Anshul Makkar 
---
 xen/common/sched_credit2.c |  243 
 1 file changed, 157 insertions(+), 86 deletions(-)

diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 0d83bd7..3aef1b4 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -902,6 +902,42 @@ __runq_remove(struct csched2_vcpu *svc)
 list_del_init(>runq_elem);
 }
 
+/*
+ * During the soft-affinity step, only actually preempt someone if
+ * he does not have soft-affinity with cpu (while we have).
+ *
+ * BEWARE that this uses cpumask_scratch, trowing away what's in there!
+ */
+static inline bool_t soft_aff_check_preempt(unsigned int bs, unsigned int cpu)
+{
+struct csched2_vcpu * cur = CSCHED2_VCPU(curr_on_cpu(cpu));
+
+/*
+ * If we're doing hard-affinity, always check whether to preempt cur.
+ * If we're doing soft-affinity, but cur doesn't have one, check as well.
+ */
+if ( bs == BALANCE_HARD_AFFINITY ||
+ !has_soft_affinity(cur->vcpu, cur->vcpu->cpu_hard_affinity) )
+return 1;
+
+/*
+ * We're doing soft-affinity, and we know that the current vcpu on cpu
+ * has a soft affinity. We now want to know whether cpu itself is in
+ * such affinity. In fact, since we now that new (in runq_tickle()) is:
+ *  - if cpu is not in cur's soft-affinity, we should indeed check to
+ *see whether new should preempt cur. If that will be the case, that
+ *would be an improvement wrt respecting soft affinity;
+ *  - if cpu is in cur's soft-affinity, we leave it alone and (in
+ *runq_tickle()) move on to another cpu. In fact, we don't want to
+ *be too harsh with someone which is running within its soft-affinity.
+ *This is safe because later, if we don't fine anyone else during the
+ *soft-affinity step, we will check cpu for preemption anyway, when
+ *doing hard-affinity.
+ */
+affinity_balance_cpumask(cur->vcpu, BALANCE_SOFT_AFFINITY, 
cpumask_scratch);
+return !cpumask_test_cpu(cpu, cpumask_scratch);
+}
+
 void burn_credits(struct csched2_runqueue_data *rqd, struct csched2_vcpu *, 
s_time_t);
 
 /*
@@ -925,7 +961,7 @@ runq_tickle(const struct scheduler *ops, struct 
csched2_vcpu *new, s_time_t now)
 {
 int i, ipid = -1;
 s_time_t lowest = (1<<30);
-unsigned int cpu = new->vcpu->processor;
+unsigned int bs, cpu = new->vcpu->processor;
 struct csched2_runqueue_data *rqd = RQD(ops, cpu);
 cpumask_t mask;
 struct csched2_vcpu * cur;
@@ -947,109 +983,144 @@ runq_tickle(const struct scheduler *ops, struct 
csched2_vcpu *new, s_time_t now)
 (unsigned char *));
 }
 
-/*
- * First of all, consider idle cpus, checking if we can just
- * re-use the pcpu where we were running before.
- *
- * If there are cores where all the siblings are idle, consider
- * them first, honoring whatever the spreading-vs-consolidation
- * SMT policy wants us to do.
- */
-if ( unlikely(sched_smt_power_savings) )
-cpumask_andnot(, >idle, >smt_idle);
-else
-cpumask_copy(, >smt_idle);
-cpumask_and(, , new->vcpu->cpu_hard_affinity);
-i = cpumask_test_or_cycle(cpu, );
-if ( i < nr_cpu_ids )
+for_each_affinity_balance_step( bs )
 {
-SCHED_STAT_CRANK(tickled_idle_cpu);
-ipid = i;
-goto tickle;
-}
+/*
+ * First things first: if we are at the first (soft affinity) step,
+ * but new doesn't have a soft affinity, skip this step.
+ */
+if ( bs == BALANCE_SOFT_AFFINITY &&
+ !has_soft_affinity(new->vcpu, new->vcpu->cpu_hard_affinity) )
+continue;
 
-/*
- * If there are no fully idle cores, check all idlers, after
- * having filtered out pcpus that have been tickled but haven't
- * gone through the scheduler yet.
- */
-cpumask_andnot(, >idle, >tickled);
-cpumask_and(, , new->vcpu->cpu_hard_affinity);
-i = cpumask_test_or_cycle(cpu, );
-if ( i < nr_cpu_ids )
-{
-SCHED_STAT_CRANK(tickled_idle_cpu);
-ipid = i;
-goto tickle;
-}
+affinity_balance_cpumask(new->vcpu, bs, cpumask_scratch);
 
-/*
- * Otherwise, look for the non-idle (and non-tickled) processors with
- * the lowest credit, among the ones new is allowed to run on. Again,
- * the