Hi Guy,
The latency statistics will be different with SRAT enabled or disabled.
When SRAT is enabled, all latency information will be read from SLIT table,
which is generated by BIOS/firmware. When SRAT is disabled (or more
precise, SLIT is disabled), lgrp will try to probe the latency among nodes by
memory copy operation or PCI configuration space reading on AMD platform.
So the key idea here is that ther are not comparable when SLIT enabled or
disabled. Entry in "Lgroup latencies" table only has relative meaning, they
could only be compared with other entry in the same table instead of entry
from another table.
To me, the mdb output from you sounds like that my patch works
correctly.
Thanks!
Guy <> wrote:
> Hello gerry, hello kit
>
> I tested your fix proposal on my opteron system (before kit answered).
>
> With this fix, the system boots,
> but the lgrp framework doesn't seem to be initialized properly.
> The latency stats are weird.
>
>> lgrp_plat_node_domain::print
> [
> {
> exists = 0x1
> prox_domain = 0x2
> }
> {
> exists = 0x1
> prox_domain = 0x1
> }
> {
> exists = 0
> prox_domain = 0
> }
> {
> exists = 0
> prox_domain = 0
> }
> {
> exists = 0
> prox_domain = 0
> }
> {
> exists = 0
> prox_domain = 0
> }
> {
> exists = 0
> prox_domain = 0
> }
> {
> exists = 0
> prox_domain = 0
> }
> ]
>> lgrp_plat_cpu_node::print
> [
> {
> exists = 0x1
> node = 0x1
> apicid = 0
> prox_domain = 0x1
> }
> {
> exists = 0x1
> node = 0
> apicid = 0x2
> prox_domain = 0x2
> }
> {
> exists = 0x1
> node = 0x1
> apicid = 0x1
> prox_domain = 0x1
> }
> {
> exists = 0x1
> node = 0
> apicid = 0x3
> prox_domain = 0x2
> }
> {
> exists = 0
> node = 0
> apicid = 0
> prox_domain = 0
> }
> {
> exists = 0
> node = 0
> apicid = 0
> prox_domain = 0
> }
> {
> exists = 0
> node = 0
> apicid = 0
> prox_domain = 0
> }
> {
> exists = 0
> node = 0
> apicid = 0
> prox_domain = 0
> }
> {
> exists = 0
> node = 0
> apicid = 0
>
>
>> lgrp_plat_node_memory::print
> [
> {
> start = 0x80000
> end = 0xfffff
> exists = 0x1
> prox_domain = 0x2
> }
> {
> start = 0
> end = 0x7ffff
> exists = 0x1
> prox_domain = 0x1
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> {
> start = 0
> end = 0
> exists = 0
> prox_domain = 0
> }
> ]
>> lgrp_plat_lat_stats::print
> {
> latencies = [
> [ 0x2, 0x3, 0, 0, 0, 0, 0, 0 ]
> [ 0x3, 0x2, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> [ 0, 0, 0, 0, 0, 0, 0, 0 ]
> ]
> latency_max = 0x3
> latency_min = 0x2
> }
>
>
> bash-3.2# lgrpinfo -Ta
> $0
>> -- 1
>> CPUs: 0 2
>> Memory: installed 2.0G, allocated 196M, free 1.8G Load: 4.58e-05
>> Latency: 2
> `-- 2
> CPUs: 1 3
> Memory: installed 1.8G, allocated 519M, free 1.3G
> Load: 0.603
> Latency: 2
>
> Lgroup latencies:
>
> ---------
> | 0 1 2
> ---------
> 0 | 3 3 3
> 1 | 3 2 3
> 2 | 3 3 2
> ---------
> bash-3.2# lgrpinfo
> lgroup 0 (root):
> Children: 1 2
> CPUs: 0-3
> Memory: installed 3.8G, allocated 715M, free 3.1G
> Lgroup resources: 1 2 (CPU); 1 2 (memory)
> Latency: 3
> lgroup 1 (leaf):
> Children: none, Parent: 0
> CPUs: 0 2
> Memory: installed 2.0G, allocated 198M, free 1.8G
> Lgroup resources: 1 (CPU); 1 (memory)
> Load: 0.503
> Latency: 2
> lgroup 2 (leaf):
> Children: none, Parent: 0
> CPUs: 1 3
> Memory: installed 1.8G, allocated 517M, free 1.3G
> Lgroup resources: 2 (CPU); 2 (memory)
> Load: 0.13
> Latency: 2
>
>
>
>
>
>
>
>
> The result is very different from what we get when srat is disabled
> (with my patch proposal) :
>
> bash-3.2# lgrpinfo
> lgroup 0 (root):
> Children: 1 2
> CPUs: 0-3
> Memory: installed 3.8G, allocated 1.1G, free 2.7G
> Lgroup resources: 1 2 (CPU); 1 2 (memory)
> Latency: 139
> lgroup 1 (leaf):
> Children: none, Parent: 0
> CPUs: 0 2
> Memory: installed 2.0G, allocated 411M, free 1.6G
> Lgroup resources: 1 (CPU); 1 (memory)
> Load: 0.497
> Latency: 105
> lgroup 2 (leaf):
> Children: none, Parent: 0
> CPUs: 1 3
> Memory: installed 1.8G, allocated 722M, free 1.1G
> Lgroup resources: 2 (CPU); 2 (memory)
> Load: 0.0722
> Latency: 105
>
> bash-3.2# lgrpinfo -Ta
> 0
>> -- 1
>> CPUs: 0 2
>> Memory: installed 2.0G, allocated 409M, free 1.6G Load: 0.0392
>> Latency: 105
> `-- 2
> CPUs: 1 3
> Memory: installed 1.8G, allocated 724M, free 1.1G
> Load: 0.54
> Latency: 105
>
> Lgroup latencies:
>
> ---------------
> | 0 1 2
> ---------------
> 0 | 139 139 139
> 1 | 139 105 139
> 2 | 139 139 105
> ---------------
>
>
> Thank you
> Guy
Liu Jiang (Gerry)
OpenSolaris, OTC, SSG, Intel
_______________________________________________
opensolaris-code mailing list
opensolaris-code@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/opensolaris-code
