Ingo Molnar wrote on Tuesday, April 05, 2005 11:46 PM
> ok, the delay of 16 secs is alot better. Could you send me the full
> detection log, how stable is the curve?
Full log attached.
begin 666 boot.log
M0F]O="!PF5D($E40R!W:71H($-052 P("AL87-T(&1I9F8@,R!C>6-L97,L(>&5R
M@I#86QI8G)A=
Ingo Molnar wrote on Tuesday, April 05, 2005 11:46 PM
ok, the delay of 16 secs is alot better. Could you send me the full
detection log, how stable is the curve?
Full log attached.
begin 666 boot.log
M0F]O=!PF]C97-S;W(@:60@,'@P+S!X8S0Q. I#4%4@,3H@WEN8VAR;VYI
MF5D($E40R!W:71H($-052
* Chen, Kenneth W <[EMAIL PROTECTED]> wrote:
> > tested on x86, the calibration results look ok there.
>
> Calibration result on ia64 (1.5 GHz, 9 MB), somewhat smaller in this
> version compare to earlier estimate of 10.4ms. The optimal setting
> found by a db workload is around 16 ms.
with
* Chen, Kenneth W [EMAIL PROTECTED] wrote:
tested on x86, the calibration results look ok there.
Calibration result on ia64 (1.5 GHz, 9 MB), somewhat smaller in this
version compare to earlier estimate of 10.4ms. The optimal setting
found by a db workload is around 16 ms.
with idle
Ingo Molnar wrote on Monday, April 04, 2005 8:05 PM
>
> latest patch attached. Changes:
>
> - stabilized calibration even more, by using cache flushing
>instructions to generate a predictable working set. The cache
>flushing itself is not timed, it is used to create quiescent
>cache
Ingo Molnar wrote on Sunday, April 03, 2005 11:24 PM
> great! How long does the benchmark take (hours?), and is there any way
> to speed up the benchmarking (without hurting accuracy), so that
> multiple migration-cost settings could be tried? Would it be possible to
> try a few other values via
Ingo Molnar wrote on Sunday, April 03, 2005 11:24 PM
great! How long does the benchmark take (hours?), and is there any way
to speed up the benchmarking (without hurting accuracy), so that
multiple migration-cost settings could be tried? Would it be possible to
try a few other values via the
Ingo Molnar wrote on Monday, April 04, 2005 8:05 PM
latest patch attached. Changes:
- stabilized calibration even more, by using cache flushing
instructions to generate a predictable working set. The cache
flushing itself is not timed, it is used to create quiescent
cache state.
latest patch attached. Changes:
- stabilized calibration even more, by using cache flushing
instructions to generate a predictable working set. The cache
flushing itself is not timed, it is used to create quiescent
cache state.
I only guessed the ia64 version - e.g. i didnt know
* Chen, Kenneth W <[EMAIL PROTECTED]> wrote:
> Perhaps, I'm not getting the latest patch? It skipped measuring
> because migration cost array is non-zero (initialized to -1LL):
yeah ... some mixup here. I've attached the latest.
> Also, the cost calculation in measure_one() looks fishy to me
* Chen, Kenneth W <[EMAIL PROTECTED]> wrote:
> The cache size information on ia64 is already available at the finger
> tip. Here is a patch that I whipped up to set max_cache_size for ia64.
Ingo Molnar wrote on Monday, April 04, 2005 4:38 AM
> thanks - i've added this to my tree.
>
> i've
Ingo wrote:
> i've attached the latest snapshot.
I ran your latest snapshot on 64 CPU (well, 62 - one node wasn't
working) system. I made one change - chop the matrix lines at 8 terms.
It's a hack - don't know if it's a good idea. But the long lines were
hard to read (and would only get worse
* Chen, Kenneth W <[EMAIL PROTECTED]> wrote:
> Ingo Molnar wrote on Saturday, April 02, 2005 11:04 PM
> > the default on ia64 (32MB) was way too large and caused the search to
> > start from 64MB. That can take a _long_ time.
> >
> > i've attached a new patch with your changes included, and a
Ingo wrote:
> the problem i mentioned earlier is that there is no other use
Eh ... whatever. The present seems straight forward enough, with a
simple sched domain tree and your auto-tune migration cost calculation
bolted directly on top of that.
I'd better leave the futures to those more
Ingo wrote:
> agreed - i've changed it to domain_distance() in my tree.
Good - cool - thanks.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[EMAIL PROTECTED]> 1.650.933.1373,
1.925.600.0401
-
To
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Would be a good idea to rename 'cpu_distance()' to something more
> specific, like 'cpu_dist_ndx()', and reserve the generic name
> 'cpu_distance()' for later use to return a scaled integer distance,
> rather like 'node_distance()' does now. [...]
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Nick wrote:
> > In a sense, the information *is* already there - in node_distance.
> > What I think should be done is probably to use node_distance when
> > calculating costs, ...
>
> Hmmm ... perhaps I'm confused, but this sure sounds like the
Nick wrote:
> In a sense, the information *is* already there - in node_distance.
> What I think should be done is probably to use node_distance when
> calculating costs, ...
Hmmm ... perhaps I'm confused, but this sure sounds like the alternative
implementation of cpu_distance using node_distance
* Ingo Molnar <[EMAIL PROTECTED]> wrote:
> > a numa scheduler domain at the top level and cache_hot_time will be
> > set to 0 in that case on smp box. Though this will be a mutt point
> > with recent patch from Suresh Siddha for removing the extra bogus
> > scheduler domains.
> >
* Chen, Kenneth W <[EMAIL PROTECTED]> wrote:
> Ingo Molnar wrote on Sunday, April 03, 2005 7:30 AM
> > how close are these numbers to the real worst-case migration costs on
> > that box?
>
> I booted your latest patch on a 4-way SMP box (1.5 GHz, 9MB ia64). This
> is what it produces. I think
* Chen, Kenneth W [EMAIL PROTECTED] wrote:
Ingo Molnar wrote on Sunday, April 03, 2005 7:30 AM
how close are these numbers to the real worst-case migration costs on
that box?
I booted your latest patch on a 4-way SMP box (1.5 GHz, 9MB ia64). This
is what it produces. I think the
Nick wrote:
In a sense, the information *is* already there - in node_distance.
What I think should be done is probably to use node_distance when
calculating costs, ...
Hmmm ... perhaps I'm confused, but this sure sounds like the alternative
implementation of cpu_distance using node_distance
* Ingo Molnar [EMAIL PROTECTED] wrote:
a numa scheduler domain at the top level and cache_hot_time will be
set to 0 in that case on smp box. Though this will be a mutt point
with recent patch from Suresh Siddha for removing the extra bogus
scheduler domains.
* Paul Jackson [EMAIL PROTECTED] wrote:
Nick wrote:
In a sense, the information *is* already there - in node_distance.
What I think should be done is probably to use node_distance when
calculating costs, ...
Hmmm ... perhaps I'm confused, but this sure sounds like the alternative
* Paul Jackson [EMAIL PROTECTED] wrote:
Would be a good idea to rename 'cpu_distance()' to something more
specific, like 'cpu_dist_ndx()', and reserve the generic name
'cpu_distance()' for later use to return a scaled integer distance,
rather like 'node_distance()' does now. [...]
agreed
Ingo wrote:
the problem i mentioned earlier is that there is no other use
Eh ... whatever. The present seems straight forward enough, with a
simple sched domain tree and your auto-tune migration cost calculation
bolted directly on top of that.
I'd better leave the futures to those more
* Chen, Kenneth W [EMAIL PROTECTED] wrote:
Ingo Molnar wrote on Saturday, April 02, 2005 11:04 PM
the default on ia64 (32MB) was way too large and caused the search to
start from 64MB. That can take a _long_ time.
i've attached a new patch with your changes included, and a couple of
Ingo wrote:
i've attached the latest snapshot.
I ran your latest snapshot on 64 CPU (well, 62 - one node wasn't
working) system. I made one change - chop the matrix lines at 8 terms.
It's a hack - don't know if it's a good idea. But the long lines were
hard to read (and would only get worse
* Chen, Kenneth W [EMAIL PROTECTED] wrote:
The cache size information on ia64 is already available at the finger
tip. Here is a patch that I whipped up to set max_cache_size for ia64.
Ingo Molnar wrote on Monday, April 04, 2005 4:38 AM
thanks - i've added this to my tree.
i've attached the
* Chen, Kenneth W [EMAIL PROTECTED] wrote:
Perhaps, I'm not getting the latest patch? It skipped measuring
because migration cost array is non-zero (initialized to -1LL):
yeah ... some mixup here. I've attached the latest.
Also, the cost calculation in measure_one() looks fishy to me in
latest patch attached. Changes:
- stabilized calibration even more, by using cache flushing
instructions to generate a predictable working set. The cache
flushing itself is not timed, it is used to create quiescent
cache state.
I only guessed the ia64 version - e.g. i didnt know
On Sun, 2005-04-03 at 20:55 -0700, Paul Jackson wrote:
> But if we knew the CPU hierarchy in more detail, and if we had some
> other use for that detail (we don't that I know), then I take it from
> your comment that we should be reluctant to push those details into the
> sched domains. Put them
Ingo wrote:
> There's no other place to push them
One could make a place, if the need arose.
> but trying and benchmarking it is necessary to tell for sure.
Hard to argue with that ... ;).
--
I won't rest till it's the best ...
Programmer, Linux
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Ingo, if I understood correctly, suggested pushing any necessary
> detail of the CPU hierarchy into the scheduler domains, so that his
> latest work tuning migration costs could pick it up from there.
>
> It makes good sense for the migration cost
Andy wrote:
> Not that I really know what I'm talking about here, but this sounds
> highly parallelizable.
I doubt it. If we are testing the cost of a migration between CPUs
alpha and beta, and at the same time testing betweeen CPUs gamma and
delta, then often there will be some hardware that
Paul Jackson wrote:
Ok - that flies, or at least walks. It took 53 seconds to
compute this cost matrix.
Not that I really know what I'm talking about here, but this sounds
highly parallelizable. It seems like you could do N/2 measurements at a
time, so this should be O(N) to compute the matrix
Paul wrote:
> I should push in the direction of improving the
> SN2 sched domain hierarchy.
Nick wrote:
> I'd just be a bit careful about this.
Good point - thanks.
I will - be careful. I have no delusions that I know what would be an
"improvement" to the scheduler - if anything.
Ingo, if I
Paul Jackson wrote:
Ingo wrote:
if you create a sched-domains hierarchy (based on the SLIT tables, or in
whatever other way) that matches the CPU hierarchy then you'll
automatically get the proper distances detected.
Yes - agreed. I should push in the direction of improving the
SN2 sched
Ingo Molnar wrote on Sunday, April 03, 2005 7:30 AM
> how close are these numbers to the real worst-case migration costs on
> that box?
I booted your latest patch on a 4-way SMP box (1.5 GHz, 9MB ia64). This
is what it produces. I think the estimate is excellent.
[00]: -10.4(0) 10.4(0)
Ingo Molnar wrote on Saturday, April 02, 2005 11:04 PM
> the default on ia64 (32MB) was way too large and caused the search to
> start from 64MB. That can take a _long_ time.
>
> i've attached a new patch with your changes included, and a couple of
> new things added:
>
> - removed the 32MB
Ingo wrote:
> how close are these numbers to the real worst-case migration costs on
> that box? What are the cache sizes and what is their hierarchies?
> ...
> is there any workload that shows the same scheduling related performance
> regressions, other than Ken's $1m+ benchmark kit?
I'll have
Ingo wrote:
> if you create a sched-domains hierarchy (based on the SLIT tables, or in
> whatever other way) that matches the CPU hierarchy then you'll
> automatically get the proper distances detected.
Yes - agreed. I should push in the direction of improving the
SN2 sched domain hierarchy.
Ingo wrote:
> if_ there is a significant hierarchy between CPUs it
> should be represented via a matching sched-domains hierarchy,
Agreed.
I'll see how the sched domains hierarchy looks on a bigger SN2 systems.
If the CPU hierarchy is not reflected in the sched-domain hierarchy any
better
* Paul Jackson <[EMAIL PROTECTED]> wrote:
>
> 3) I was noticing that my test system was only showing a couple of
> distinct values for cpu_distance, even though it has 4 distinct
> distances for values of node_distance. So I coded up a variant of
> cpu_distance that converts
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Three more observations.
>
> 1) The slowest measure_one() calls are, not surprisingly, those for the
> largest sizes. At least on my test system of the moment, the plot
> of cost versus size has one major maximum (a one hump camel, not two).
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Ok - that flies, or at least walks. It took 53 seconds to compute
> this cost matrix.
53 seconds is too much - i'm working on reducing it.
> Here's what it prints, on a small 8 CPU ia64 SN2 Altix, with
> the migration_debug prints formatted
Three more observations.
1) The slowest measure_one() calls are, not surprisingly, those for the
largest sizes. At least on my test system of the moment, the plot
of cost versus size has one major maximum (a one hump camel, not two).
Seems like if we computed from smallest size
Ok - that flies, or at least walks. It took 53 seconds to
compute this cost matrix.
Here's what it prints, on a small 8 CPU ia64 SN2 Altix, with
the migration_debug prints formatted separately from the primary
table, for ease of reading:
Total of 8 processors activated (15548.60 BogoMIPS).
Earlier, Paul wrote:
> Note the first 3 chars of the panic message "4.5". This looks like it
> might be the [00]-[01] entry of Ingo's table, flushed out when the
> newlines of the panic came through.
For the record, the above speculation is probably wrong.
More likely, the first six characters
> the default on ia64 (32MB) was way too large
Agreed. It took about 9 minutes to search the first pair of cpus
(cpu 0 to cpu 1) from a size of 67107840 down to a size of 62906,
based on some prints I added since my last message.
> it seems the screen blanking timer hit
Ah - yes. That makes
the default on ia64 (32MB) was way too large
Agreed. It took about 9 minutes to search the first pair of cpus
(cpu 0 to cpu 1) from a size of 67107840 down to a size of 62906,
based on some prints I added since my last message.
it seems the screen blanking timer hit
Ah - yes. That makes
Earlier, Paul wrote:
Note the first 3 chars of the panic message 4.5. This looks like it
might be the [00]-[01] entry of Ingo's table, flushed out when the
newlines of the panic came through.
For the record, the above speculation is probably wrong.
More likely, the first six characters
Ok - that flies, or at least walks. It took 53 seconds to
compute this cost matrix.
Here's what it prints, on a small 8 CPU ia64 SN2 Altix, with
the migration_debug prints formatted separately from the primary
table, for ease of reading:
Total of 8 processors activated (15548.60 BogoMIPS).
Three more observations.
1) The slowest measure_one() calls are, not surprisingly, those for the
largest sizes. At least on my test system of the moment, the plot
of cost versus size has one major maximum (a one hump camel, not two).
Seems like if we computed from smallest size
* Paul Jackson [EMAIL PROTECTED] wrote:
Ok - that flies, or at least walks. It took 53 seconds to compute
this cost matrix.
53 seconds is too much - i'm working on reducing it.
Here's what it prints, on a small 8 CPU ia64 SN2 Altix, with
the migration_debug prints formatted separately
* Paul Jackson [EMAIL PROTECTED] wrote:
Three more observations.
1) The slowest measure_one() calls are, not surprisingly, those for the
largest sizes. At least on my test system of the moment, the plot
of cost versus size has one major maximum (a one hump camel, not two).
* Paul Jackson [EMAIL PROTECTED] wrote:
3) I was noticing that my test system was only showing a couple of
distinct values for cpu_distance, even though it has 4 distinct
distances for values of node_distance. So I coded up a variant of
cpu_distance that converts the
Ingo wrote:
if_ there is a significant hierarchy between CPUs it
should be represented via a matching sched-domains hierarchy,
Agreed.
I'll see how the sched domains hierarchy looks on a bigger SN2 systems.
If the CPU hierarchy is not reflected in the sched-domain hierarchy any
better there,
Ingo wrote:
if you create a sched-domains hierarchy (based on the SLIT tables, or in
whatever other way) that matches the CPU hierarchy then you'll
automatically get the proper distances detected.
Yes - agreed. I should push in the direction of improving the
SN2 sched domain hierarchy.
Ingo wrote:
how close are these numbers to the real worst-case migration costs on
that box? What are the cache sizes and what is their hierarchies?
...
is there any workload that shows the same scheduling related performance
regressions, other than Ken's $1m+ benchmark kit?
I'll have to
Ingo Molnar wrote on Saturday, April 02, 2005 11:04 PM
the default on ia64 (32MB) was way too large and caused the search to
start from 64MB. That can take a _long_ time.
i've attached a new patch with your changes included, and a couple of
new things added:
- removed the 32MB
Paul wrote:
I should push in the direction of improving the
SN2 sched domain hierarchy.
Nick wrote:
I'd just be a bit careful about this.
Good point - thanks.
I will - be careful. I have no delusions that I know what would be an
improvement to the scheduler - if anything.
Ingo, if I
Paul Jackson wrote:
Ok - that flies, or at least walks. It took 53 seconds to
compute this cost matrix.
Not that I really know what I'm talking about here, but this sounds
highly parallelizable. It seems like you could do N/2 measurements at a
time, so this should be O(N) to compute the matrix
Andy wrote:
Not that I really know what I'm talking about here, but this sounds
highly parallelizable.
I doubt it. If we are testing the cost of a migration between CPUs
alpha and beta, and at the same time testing betweeen CPUs gamma and
delta, then often there will be some hardware that is
* Paul Jackson [EMAIL PROTECTED] wrote:
Ingo, if I understood correctly, suggested pushing any necessary
detail of the CPU hierarchy into the scheduler domains, so that his
latest work tuning migration costs could pick it up from there.
It makes good sense for the migration cost
Ingo wrote:
There's no other place to push them
One could make a place, if the need arose.
but trying and benchmarking it is necessary to tell for sure.
Hard to argue with that ... ;).
--
I won't rest till it's the best ...
Programmer, Linux Scalability
On Sun, 2005-04-03 at 20:55 -0700, Paul Jackson wrote:
But if we knew the CPU hierarchy in more detail, and if we had some
other use for that detail (we don't that I know), then I take it from
your comment that we should be reluctant to push those details into the
sched domains. Put them
* Paul Jackson <[EMAIL PROTECTED]> wrote:
> Just so as no else wastes time repeating the little bit I've done so
> far, and so I don't waste time figuring out what is already known,
> here's what I have so far, trying out Ingo's "sched: auto-tune
> migration costs" on ia64 SN2:
>
> To get it
Just so as no else wastes time repeating the little bit I've done so
far, and so I don't waste time figuring out what is already known,
here's what I have so far, trying out Ingo's "sched: auto-tune migration
costs" on ia64 SN2:
To get it to compile against 2.6.12-rc1-mm4, I did thus:
1.
Ingo wrote:
> in theory the code should work fine on ia64 as well,
Nice. I'll try it on our SN2 Altix IA64 as well.
Though I am being delayed a day or two in this
by irrelevant problems.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Ingo wrote:
in theory the code should work fine on ia64 as well,
Nice. I'll try it on our SN2 Altix IA64 as well.
Though I am being delayed a day or two in this
by irrelevant problems.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Just so as no else wastes time repeating the little bit I've done so
far, and so I don't waste time figuring out what is already known,
here's what I have so far, trying out Ingo's sched: auto-tune migration
costs on ia64 SN2:
To get it to compile against 2.6.12-rc1-mm4, I did thus:
1.
* Paul Jackson [EMAIL PROTECTED] wrote:
Just so as no else wastes time repeating the little bit I've done so
far, and so I don't waste time figuring out what is already known,
here's what I have so far, trying out Ingo's sched: auto-tune
migration costs on ia64 SN2:
To get it to
73 matches
Mail list logo
