Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
David Gibson wrote:
> On Tue, Apr 17, 2018 at 09:28:42AM +0530, Bharata B Rao wrote:
>> On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote:
>>> Bharata B Rao wrote:
On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote:
> Now recent kernels (i.e. since linux-stable commit a346137e9142
> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible
> nodes")
> support this property to mark initially memory-less NUMA nodes as
> "possible"
> to allow further memory hot-add to them.
>
> Advertise this property for pSeries machines to let guest kernels detect
> maximum supported node configuration and benefit from kernel side change
> when hot-add memory to specific, possibly empty before, NUMA node.
>
> Signed-off-by: Serhii Popovych
> ---
> hw/ppc/spapr.c | 10 ++
> 1 file changed, 10 insertions(+)
>
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index a81570e..c05bbad 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> void *fdt)
> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE),
> cpu_to_be32(max_cpus / smp_threads),
> };
> +uint32_t maxdomains[] = {
> +cpu_to_be32(4),
> +cpu_to_be32(0),
> +cpu_to_be32(0),
> +cpu_to_be32(0),
> +cpu_to_be32(nb_numa_nodes - 1),
> +};
>
> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas"));
>
> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> void *fdt)
> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points",
> refpoints, sizeof(refpoints)));
>
> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains",
> + maxdomains, sizeof(maxdomains)));
> +
> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max",
>RTAS_ERROR_LOG_MAX));
> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate",
This commit causes hash guest with latest guest kernel to hang at early
boot.
>>>
>>> I use v4.16 tag from stable and can't reproduce on P8 machine reported
>>> issue.
>>>
>>> Could you please share more details about your setup, kernel commit id
>>> you spot problem?
>>
>> I am on 4.16.0-rc7 (commit id: 0b412605ef5f)
>>
>> BTW this happens only for non-NUMA guest.
>
> Ah, that might explain it. With no NUMA nodes specified, I think this
> code will put a -1 into the max-associativity-domains property, which
> is probably causing the mess. If we don't have NUMA (nb_numa_nodes ==
> 0) we probably want to either omit the property entirely, or clamp
> that 5th cell to 0.
>
Ok, proposed fix already posted to qemu-devel. Sorry forgot to CC.
Mail subject contains (w/o quotas):
"spapr: Correct max associativity domains value for non-NUMA configs"
Tested with v4.16 tag:
o Before : non-NUMA configs: able to reproduce, stall during boot
o After : non-NUMA configs: not reproducible, boot is ok
--
Thanks,
Serhii
signature.asc
Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Tue, Apr 17, 2018 at 09:28:42AM +0530, Bharata B Rao wrote:
> On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote:
> > Bharata B Rao wrote:
> > > On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote:
> > >> Now recent kernels (i.e. since linux-stable commit a346137e9142
> > >> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible
> > >> nodes")
> > >> support this property to mark initially memory-less NUMA nodes as
> > >> "possible"
> > >> to allow further memory hot-add to them.
> > >>
> > >> Advertise this property for pSeries machines to let guest kernels detect
> > >> maximum supported node configuration and benefit from kernel side change
> > >> when hot-add memory to specific, possibly empty before, NUMA node.
> > >>
> > >> Signed-off-by: Serhii Popovych
> > >> ---
> > >> hw/ppc/spapr.c | 10 ++
> > >> 1 file changed, 10 insertions(+)
> > >>
> > >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > >> index a81570e..c05bbad 100644
> > >> --- a/hw/ppc/spapr.c
> > >> +++ b/hw/ppc/spapr.c
> > >> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> > >> void *fdt)
> > >> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE),
> > >> cpu_to_be32(max_cpus / smp_threads),
> > >> };
> > >> +uint32_t maxdomains[] = {
> > >> +cpu_to_be32(4),
> > >> +cpu_to_be32(0),
> > >> +cpu_to_be32(0),
> > >> +cpu_to_be32(0),
> > >> +cpu_to_be32(nb_numa_nodes - 1),
> > >> +};
> > >>
> > >> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas"));
> > >>
> > >> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> > >> void *fdt)
> > >> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points",
> > >> refpoints, sizeof(refpoints)));
> > >>
> > >> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains",
> > >> + maxdomains, sizeof(maxdomains)));
> > >> +
> > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max",
> > >>RTAS_ERROR_LOG_MAX));
> > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate",
> > >
> > > This commit causes hash guest with latest guest kernel to hang at early
> > > boot.
> >
> > I use v4.16 tag from stable and can't reproduce on P8 machine reported
> > issue.
> >
> > Could you please share more details about your setup, kernel commit id
> > you spot problem?
>
> I am on 4.16.0-rc7 (commit id: 0b412605ef5f)
>
> BTW this happens only for non-NUMA guest.
Ah, that might explain it. With no NUMA nodes specified, I think this
code will put a -1 into the max-associativity-domains property, which
is probably causing the mess. If we don't have NUMA (nb_numa_nodes ==
0) we probably want to either omit the property entirely, or clamp
that 5th cell to 0.
--
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
signature.asc
Description: PGP signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote:
> Bharata B Rao wrote:
> > On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote:
> >> Now recent kernels (i.e. since linux-stable commit a346137e9142
> >> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible
> >> nodes")
> >> support this property to mark initially memory-less NUMA nodes as
> >> "possible"
> >> to allow further memory hot-add to them.
> >>
> >> Advertise this property for pSeries machines to let guest kernels detect
> >> maximum supported node configuration and benefit from kernel side change
> >> when hot-add memory to specific, possibly empty before, NUMA node.
> >>
> >> Signed-off-by: Serhii Popovych
> >> ---
> >> hw/ppc/spapr.c | 10 ++
> >> 1 file changed, 10 insertions(+)
> >>
> >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> >> index a81570e..c05bbad 100644
> >> --- a/hw/ppc/spapr.c
> >> +++ b/hw/ppc/spapr.c
> >> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> >> void *fdt)
> >> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE),
> >> cpu_to_be32(max_cpus / smp_threads),
> >> };
> >> +uint32_t maxdomains[] = {
> >> +cpu_to_be32(4),
> >> +cpu_to_be32(0),
> >> +cpu_to_be32(0),
> >> +cpu_to_be32(0),
> >> +cpu_to_be32(nb_numa_nodes - 1),
> >> +};
> >>
> >> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas"));
> >>
> >> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
> >> void *fdt)
> >> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points",
> >> refpoints, sizeof(refpoints)));
> >>
> >> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains",
> >> + maxdomains, sizeof(maxdomains)));
> >> +
> >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max",
> >>RTAS_ERROR_LOG_MAX));
> >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate",
> >> --
> >> 1.8.3.1
> >
> > This commit causes hash guest with latest guest kernel to hang at early
> > boot.
>
> I use v4.16 tag from stable and can't reproduce on P8 machine reported
> issue.
>
> Could you please share more details about your setup, kernel commit id
> you spot problem?
I am on 4.16.0-rc7 (commit id: 0b412605ef5f)
BTW this happens only for non-NUMA guest.
Regards,
Bharata.
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
Bharata B Rao wrote:
> On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote:
>> Now recent kernels (i.e. since linux-stable commit a346137e9142
>> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible
>> nodes")
>> support this property to mark initially memory-less NUMA nodes as "possible"
>> to allow further memory hot-add to them.
>>
>> Advertise this property for pSeries machines to let guest kernels detect
>> maximum supported node configuration and benefit from kernel side change
>> when hot-add memory to specific, possibly empty before, NUMA node.
>>
>> Signed-off-by: Serhii Popovych
>> ---
>> hw/ppc/spapr.c | 10 ++
>> 1 file changed, 10 insertions(+)
>>
>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
>> index a81570e..c05bbad 100644
>> --- a/hw/ppc/spapr.c
>> +++ b/hw/ppc/spapr.c
>> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr,
>> void *fdt)
>> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE),
>> cpu_to_be32(max_cpus / smp_threads),
>> };
>> +uint32_t maxdomains[] = {
>> +cpu_to_be32(4),
>> +cpu_to_be32(0),
>> +cpu_to_be32(0),
>> +cpu_to_be32(0),
>> +cpu_to_be32(nb_numa_nodes - 1),
>> +};
>>
>> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas"));
>>
>> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void
>> *fdt)
>> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points",
>> refpoints, sizeof(refpoints)));
>>
>> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains",
>> + maxdomains, sizeof(maxdomains)));
>> +
>> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max",
>>RTAS_ERROR_LOG_MAX));
>> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate",
>> --
>> 1.8.3.1
>
> This commit causes hash guest with latest guest kernel to hang at early boot.
I use v4.16 tag from stable and can't reproduce on P8 machine reported
issue.
Could you please share more details about your setup, kernel commit id
you spot problem?
>
> Quiescing Open Firmware ...
> Booting Linux via __start() @ 0x0200 ...
> [0.00] hash-mmu: Page sizes from device-tree:
> [0.00] hash-mmu: base_shift=12: shift=12, sllp=0x,
> avpnm=0x, tlbiel=1, penc=0
> [0.00] hash-mmu: base_shift=16: shift=16, sllp=0x0110,
> avpnm=0x, tlbiel=1, penc=1
> [0.00] Using 1TB segments
> [0.00] hash-mmu: Initializing hash mmu with SLB
> [0.00] Linux version 4.16.0-rc7+ (root@localhost.localdomain) (gcc
> version 7.1.1 20170622 (Red Hat 7.1.1-3) (GCC)) #60 SMP Wed Apr 11 10:36:22
> IST 2018
> [0.00] Found initrd at 0xc3c0:0xc4f9a34c
> [0.00] Using pSeries machine description
> [0.00] bootconsole [udbg0] enabled
> [0.00] Partition configured for 32 cpus.
> [0.00] CPU maps initialized for 1 thread per core
> [0.00] -
> [0.00] ppc64_pft_size= 0x1a
> [0.00] phys_mem_size = 0x2
> [0.00] dcache_bsize = 0x80
> [0.00] icache_bsize = 0x80
> [0.00] cpu_features = 0x077c7a6c18500249
> [0.00] possible= 0x18500649
> [0.00] always = 0x18100040
> [0.00] cpu_user_features = 0xdc0065c2 0xae00
> [0.00] mmu_features = 0x78006001
> [0.00] firmware_features = 0x0001415a445f
> [0.00] htab_hash_mask= 0x7
> [0.00] -
>
> No progess after this.
>
>
--
Thanks,
Serhii
signature.asc
Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote:
> Now recent kernels (i.e. since linux-stable commit a346137e9142
> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible nodes")
> support this property to mark initially memory-less NUMA nodes as "possible"
> to allow further memory hot-add to them.
>
> Advertise this property for pSeries machines to let guest kernels detect
> maximum supported node configuration and benefit from kernel side change
> when hot-add memory to specific, possibly empty before, NUMA node.
>
> Signed-off-by: Serhii Popovych
> ---
> hw/ppc/spapr.c | 10 ++
> 1 file changed, 10 insertions(+)
>
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index a81570e..c05bbad 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void
> *fdt)
> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE),
> cpu_to_be32(max_cpus / smp_threads),
> };
> +uint32_t maxdomains[] = {
> +cpu_to_be32(4),
> +cpu_to_be32(0),
> +cpu_to_be32(0),
> +cpu_to_be32(0),
> +cpu_to_be32(nb_numa_nodes - 1),
> +};
>
> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas"));
>
> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void
> *fdt)
> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points",
> refpoints, sizeof(refpoints)));
>
> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains",
> + maxdomains, sizeof(maxdomains)));
> +
> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max",
>RTAS_ERROR_LOG_MAX));
> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate",
> --
> 1.8.3.1
This commit causes hash guest with latest guest kernel to hang at early boot.
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0200 ...
[0.00] hash-mmu: Page sizes from device-tree:
[0.00] hash-mmu: base_shift=12: shift=12, sllp=0x,
avpnm=0x, tlbiel=1, penc=0
[0.00] hash-mmu: base_shift=16: shift=16, sllp=0x0110,
avpnm=0x, tlbiel=1, penc=1
[0.00] Using 1TB segments
[0.00] hash-mmu: Initializing hash mmu with SLB
[0.00] Linux version 4.16.0-rc7+ (root@localhost.localdomain) (gcc
version 7.1.1 20170622 (Red Hat 7.1.1-3) (GCC)) #60 SMP Wed Apr 11 10:36:22 IST
2018
[0.00] Found initrd at 0xc3c0:0xc4f9a34c
[0.00] Using pSeries machine description
[0.00] bootconsole [udbg0] enabled
[0.00] Partition configured for 32 cpus.
[0.00] CPU maps initialized for 1 thread per core
[0.00] -
[0.00] ppc64_pft_size= 0x1a
[0.00] phys_mem_size = 0x2
[0.00] dcache_bsize = 0x80
[0.00] icache_bsize = 0x80
[0.00] cpu_features = 0x077c7a6c18500249
[0.00] possible= 0x18500649
[0.00] always = 0x18100040
[0.00] cpu_user_features = 0xdc0065c2 0xae00
[0.00] mmu_features = 0x78006001
[0.00] firmware_features = 0x0001415a445f
[0.00] htab_hash_mask= 0x7
[0.00] -
No progess after this.
