Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
David Gibson wrote: > On Tue, Apr 17, 2018 at 09:28:42AM +0530, Bharata B Rao wrote: >> On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote: >>> Bharata B Rao wrote: On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote: > Now recent kernels (i.e. since linux-stable commit a346137e9142 > ("powerpc/numa: Use ibm,max-associativity-domains to discover possible > nodes") > support this property to mark initially memory-less NUMA nodes as > "possible" > to allow further memory hot-add to them. > > Advertise this property for pSeries machines to let guest kernels detect > maximum supported node configuration and benefit from kernel side change > when hot-add memory to specific, possibly empty before, NUMA node. > > Signed-off-by: Serhii Popovych > --- > hw/ppc/spapr.c | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index a81570e..c05bbad 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > void *fdt) > 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), > cpu_to_be32(max_cpus / smp_threads), > }; > +uint32_t maxdomains[] = { > +cpu_to_be32(4), > +cpu_to_be32(0), > +cpu_to_be32(0), > +cpu_to_be32(0), > +cpu_to_be32(nb_numa_nodes - 1), > +}; > > _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); > > @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > void *fdt) > _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", > refpoints, sizeof(refpoints))); > > +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", > + maxdomains, sizeof(maxdomains))); > + > _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", >RTAS_ERROR_LOG_MAX)); > _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", This commit causes hash guest with latest guest kernel to hang at early boot. >>> >>> I use v4.16 tag from stable and can't reproduce on P8 machine reported >>> issue. >>> >>> Could you please share more details about your setup, kernel commit id >>> you spot problem? >> >> I am on 4.16.0-rc7 (commit id: 0b412605ef5f) >> >> BTW this happens only for non-NUMA guest. > > Ah, that might explain it. With no NUMA nodes specified, I think this > code will put a -1 into the max-associativity-domains property, which > is probably causing the mess. If we don't have NUMA (nb_numa_nodes == > 0) we probably want to either omit the property entirely, or clamp > that 5th cell to 0. > Ok, proposed fix already posted to qemu-devel. Sorry forgot to CC. Mail subject contains (w/o quotas): "spapr: Correct max associativity domains value for non-NUMA configs" Tested with v4.16 tag: o Before : non-NUMA configs: able to reproduce, stall during boot o After : non-NUMA configs: not reproducible, boot is ok -- Thanks, Serhii signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Tue, Apr 17, 2018 at 09:28:42AM +0530, Bharata B Rao wrote: > On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote: > > Bharata B Rao wrote: > > > On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote: > > >> Now recent kernels (i.e. since linux-stable commit a346137e9142 > > >> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible > > >> nodes") > > >> support this property to mark initially memory-less NUMA nodes as > > >> "possible" > > >> to allow further memory hot-add to them. > > >> > > >> Advertise this property for pSeries machines to let guest kernels detect > > >> maximum supported node configuration and benefit from kernel side change > > >> when hot-add memory to specific, possibly empty before, NUMA node. > > >> > > >> Signed-off-by: Serhii Popovych > > >> --- > > >> hw/ppc/spapr.c | 10 ++ > > >> 1 file changed, 10 insertions(+) > > >> > > >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > > >> index a81570e..c05bbad 100644 > > >> --- a/hw/ppc/spapr.c > > >> +++ b/hw/ppc/spapr.c > > >> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > > >> void *fdt) > > >> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), > > >> cpu_to_be32(max_cpus / smp_threads), > > >> }; > > >> +uint32_t maxdomains[] = { > > >> +cpu_to_be32(4), > > >> +cpu_to_be32(0), > > >> +cpu_to_be32(0), > > >> +cpu_to_be32(0), > > >> +cpu_to_be32(nb_numa_nodes - 1), > > >> +}; > > >> > > >> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); > > >> > > >> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > > >> void *fdt) > > >> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", > > >> refpoints, sizeof(refpoints))); > > >> > > >> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", > > >> + maxdomains, sizeof(maxdomains))); > > >> + > > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", > > >>RTAS_ERROR_LOG_MAX)); > > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", > > > > > > This commit causes hash guest with latest guest kernel to hang at early > > > boot. > > > > I use v4.16 tag from stable and can't reproduce on P8 machine reported > > issue. > > > > Could you please share more details about your setup, kernel commit id > > you spot problem? > > I am on 4.16.0-rc7 (commit id: 0b412605ef5f) > > BTW this happens only for non-NUMA guest. Ah, that might explain it. With no NUMA nodes specified, I think this code will put a -1 into the max-associativity-domains property, which is probably causing the mess. If we don't have NUMA (nb_numa_nodes == 0) we probably want to either omit the property entirely, or clamp that 5th cell to 0. -- David Gibson| I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson signature.asc Description: PGP signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Mon, Apr 16, 2018 at 07:47:29PM +0300, Serhii Popovych wrote: > Bharata B Rao wrote: > > On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote: > >> Now recent kernels (i.e. since linux-stable commit a346137e9142 > >> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible > >> nodes") > >> support this property to mark initially memory-less NUMA nodes as > >> "possible" > >> to allow further memory hot-add to them. > >> > >> Advertise this property for pSeries machines to let guest kernels detect > >> maximum supported node configuration and benefit from kernel side change > >> when hot-add memory to specific, possibly empty before, NUMA node. > >> > >> Signed-off-by: Serhii Popovych > >> --- > >> hw/ppc/spapr.c | 10 ++ > >> 1 file changed, 10 insertions(+) > >> > >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > >> index a81570e..c05bbad 100644 > >> --- a/hw/ppc/spapr.c > >> +++ b/hw/ppc/spapr.c > >> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > >> void *fdt) > >> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), > >> cpu_to_be32(max_cpus / smp_threads), > >> }; > >> +uint32_t maxdomains[] = { > >> +cpu_to_be32(4), > >> +cpu_to_be32(0), > >> +cpu_to_be32(0), > >> +cpu_to_be32(0), > >> +cpu_to_be32(nb_numa_nodes - 1), > >> +}; > >> > >> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); > >> > >> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, > >> void *fdt) > >> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", > >> refpoints, sizeof(refpoints))); > >> > >> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", > >> + maxdomains, sizeof(maxdomains))); > >> + > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", > >>RTAS_ERROR_LOG_MAX)); > >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", > >> -- > >> 1.8.3.1 > > > > This commit causes hash guest with latest guest kernel to hang at early > > boot. > > I use v4.16 tag from stable and can't reproduce on P8 machine reported > issue. > > Could you please share more details about your setup, kernel commit id > you spot problem? I am on 4.16.0-rc7 (commit id: 0b412605ef5f) BTW this happens only for non-NUMA guest. Regards, Bharata.
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
Bharata B Rao wrote: > On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote: >> Now recent kernels (i.e. since linux-stable commit a346137e9142 >> ("powerpc/numa: Use ibm,max-associativity-domains to discover possible >> nodes") >> support this property to mark initially memory-less NUMA nodes as "possible" >> to allow further memory hot-add to them. >> >> Advertise this property for pSeries machines to let guest kernels detect >> maximum supported node configuration and benefit from kernel side change >> when hot-add memory to specific, possibly empty before, NUMA node. >> >> Signed-off-by: Serhii Popovych >> --- >> hw/ppc/spapr.c | 10 ++ >> 1 file changed, 10 insertions(+) >> >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c >> index a81570e..c05bbad 100644 >> --- a/hw/ppc/spapr.c >> +++ b/hw/ppc/spapr.c >> @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, >> void *fdt) >> 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), >> cpu_to_be32(max_cpus / smp_threads), >> }; >> +uint32_t maxdomains[] = { >> +cpu_to_be32(4), >> +cpu_to_be32(0), >> +cpu_to_be32(0), >> +cpu_to_be32(0), >> +cpu_to_be32(nb_numa_nodes - 1), >> +}; >> >> _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); >> >> @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void >> *fdt) >> _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", >> refpoints, sizeof(refpoints))); >> >> +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", >> + maxdomains, sizeof(maxdomains))); >> + >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", >>RTAS_ERROR_LOG_MAX)); >> _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", >> -- >> 1.8.3.1 > > This commit causes hash guest with latest guest kernel to hang at early boot. I use v4.16 tag from stable and can't reproduce on P8 machine reported issue. Could you please share more details about your setup, kernel commit id you spot problem? > > Quiescing Open Firmware ... > Booting Linux via __start() @ 0x0200 ... > [0.00] hash-mmu: Page sizes from device-tree: > [0.00] hash-mmu: base_shift=12: shift=12, sllp=0x, > avpnm=0x, tlbiel=1, penc=0 > [0.00] hash-mmu: base_shift=16: shift=16, sllp=0x0110, > avpnm=0x, tlbiel=1, penc=1 > [0.00] Using 1TB segments > [0.00] hash-mmu: Initializing hash mmu with SLB > [0.00] Linux version 4.16.0-rc7+ (root@localhost.localdomain) (gcc > version 7.1.1 20170622 (Red Hat 7.1.1-3) (GCC)) #60 SMP Wed Apr 11 10:36:22 > IST 2018 > [0.00] Found initrd at 0xc3c0:0xc4f9a34c > [0.00] Using pSeries machine description > [0.00] bootconsole [udbg0] enabled > [0.00] Partition configured for 32 cpus. > [0.00] CPU maps initialized for 1 thread per core > [0.00] - > [0.00] ppc64_pft_size= 0x1a > [0.00] phys_mem_size = 0x2 > [0.00] dcache_bsize = 0x80 > [0.00] icache_bsize = 0x80 > [0.00] cpu_features = 0x077c7a6c18500249 > [0.00] possible= 0x18500649 > [0.00] always = 0x18100040 > [0.00] cpu_user_features = 0xdc0065c2 0xae00 > [0.00] mmu_features = 0x78006001 > [0.00] firmware_features = 0x0001415a445f > [0.00] htab_hash_mask= 0x7 > [0.00] - > > No progess after this. > > -- Thanks, Serhii signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
On Wed, Apr 11, 2018 at 02:41:59PM -0400, Serhii Popovych wrote: > Now recent kernels (i.e. since linux-stable commit a346137e9142 > ("powerpc/numa: Use ibm,max-associativity-domains to discover possible nodes") > support this property to mark initially memory-less NUMA nodes as "possible" > to allow further memory hot-add to them. > > Advertise this property for pSeries machines to let guest kernels detect > maximum supported node configuration and benefit from kernel side change > when hot-add memory to specific, possibly empty before, NUMA node. > > Signed-off-by: Serhii Popovych > --- > hw/ppc/spapr.c | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c > index a81570e..c05bbad 100644 > --- a/hw/ppc/spapr.c > +++ b/hw/ppc/spapr.c > @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void > *fdt) > 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), > cpu_to_be32(max_cpus / smp_threads), > }; > +uint32_t maxdomains[] = { > +cpu_to_be32(4), > +cpu_to_be32(0), > +cpu_to_be32(0), > +cpu_to_be32(0), > +cpu_to_be32(nb_numa_nodes - 1), > +}; > > _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); > > @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void > *fdt) > _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", > refpoints, sizeof(refpoints))); > > +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", > + maxdomains, sizeof(maxdomains))); > + > _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", >RTAS_ERROR_LOG_MAX)); > _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", > -- > 1.8.3.1 This commit causes hash guest with latest guest kernel to hang at early boot. Quiescing Open Firmware ... Booting Linux via __start() @ 0x0200 ... [0.00] hash-mmu: Page sizes from device-tree: [0.00] hash-mmu: base_shift=12: shift=12, sllp=0x, avpnm=0x, tlbiel=1, penc=0 [0.00] hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x, tlbiel=1, penc=1 [0.00] Using 1TB segments [0.00] hash-mmu: Initializing hash mmu with SLB [0.00] Linux version 4.16.0-rc7+ (root@localhost.localdomain) (gcc version 7.1.1 20170622 (Red Hat 7.1.1-3) (GCC)) #60 SMP Wed Apr 11 10:36:22 IST 2018 [0.00] Found initrd at 0xc3c0:0xc4f9a34c [0.00] Using pSeries machine description [0.00] bootconsole [udbg0] enabled [0.00] Partition configured for 32 cpus. [0.00] CPU maps initialized for 1 thread per core [0.00] - [0.00] ppc64_pft_size= 0x1a [0.00] phys_mem_size = 0x2 [0.00] dcache_bsize = 0x80 [0.00] icache_bsize = 0x80 [0.00] cpu_features = 0x077c7a6c18500249 [0.00] possible= 0x18500649 [0.00] always = 0x18100040 [0.00] cpu_user_features = 0xdc0065c2 0xae00 [0.00] mmu_features = 0x78006001 [0.00] firmware_features = 0x0001415a445f [0.00] htab_hash_mask= 0x7 [0.00] - No progess after this.
[Qemu-devel] [PATCH for 2.13 v3 1/2] spapr: Add ibm, max-associativity-domains property
Now recent kernels (i.e. since linux-stable commit a346137e9142 ("powerpc/numa: Use ibm,max-associativity-domains to discover possible nodes") support this property to mark initially memory-less NUMA nodes as "possible" to allow further memory hot-add to them. Advertise this property for pSeries machines to let guest kernels detect maximum supported node configuration and benefit from kernel side change when hot-add memory to specific, possibly empty before, NUMA node. Signed-off-by: Serhii Popovych --- hw/ppc/spapr.c | 10 ++ 1 file changed, 10 insertions(+) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index a81570e..c05bbad 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -910,6 +910,13 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void *fdt) 0, cpu_to_be32(SPAPR_MEMORY_BLOCK_SIZE), cpu_to_be32(max_cpus / smp_threads), }; +uint32_t maxdomains[] = { +cpu_to_be32(4), +cpu_to_be32(0), +cpu_to_be32(0), +cpu_to_be32(0), +cpu_to_be32(nb_numa_nodes - 1), +}; _FDT(rtas = fdt_add_subnode(fdt, 0, "rtas")); @@ -946,6 +953,9 @@ static void spapr_dt_rtas(sPAPRMachineState *spapr, void *fdt) _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", refpoints, sizeof(refpoints))); +_FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", + maxdomains, sizeof(maxdomains))); + _FDT(fdt_setprop_cell(fdt, rtas, "rtas-error-log-max", RTAS_ERROR_LOG_MAX)); _FDT(fdt_setprop_cell(fdt, rtas, "rtas-event-scan-rate", -- 1.8.3.1