Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
David Gibson writes:
> On Thu, Jun 24, 2021 at 01:50:34PM +0530, Aneesh Kumar K.V wrote:
>> David Gibson writes:
>>
>> > On Thu, Jun 17, 2021 at 10:21:05PM +0530, Aneesh Kumar K.V wrote:
>> >> PAPR interface currently supports two different ways of communicating
>> >> resource
>> >> grouping details to the OS. These are referred to as Form 0 and Form 1
>> >> associativity grouping. Form 0 is the older format and is now considered
>> >> deprecated. This patch adds another resource grouping named FORM2.
>> >>
>> >> Signed-off-by: Daniel Henrique Barboza
>> >> Signed-off-by: Aneesh Kumar K.V
>> >> ---
>> >> Documentation/powerpc/associativity.rst | 135
>> >> arch/powerpc/include/asm/firmware.h | 3 +-
>> >> arch/powerpc/include/asm/prom.h | 1 +
>> >> arch/powerpc/kernel/prom_init.c | 3 +-
>> >> arch/powerpc/mm/numa.c| 149 +-
>> >> arch/powerpc/platforms/pseries/firmware.c | 1 +
>> >> 6 files changed, 286 insertions(+), 6 deletions(-)
>> >> create mode 100644 Documentation/powerpc/associativity.rst
>> >>
>> >> diff --git a/Documentation/powerpc/associativity.rst
>> >> b/Documentation/powerpc/associativity.rst
>> >> new file mode 100644
>> >> index ..93be604ac54d
>> >> --- /dev/null
>> >> +++ b/Documentation/powerpc/associativity.rst
>> >> @@ -0,0 +1,135 @@
>> >> +
>> >> +NUMA resource associativity
>> >> +=
>> >> +
>> >> +Associativity represents the groupings of the various platform resources
>> >> into
>> >> +domains of substantially similar mean performance relative to resources
>> >> outside
>> >> +of that domain. Resources subsets of a given domain that exhibit better
>> >> +performance relative to each other than relative to other resources
>> >> subsets
>> >> +are represented as being members of a sub-grouping domain. This
>> >> performance
>> >> +characteristic is presented in terms of NUMA node distance within the
>> >> Linux kernel.
>> >> +From the platform view, these groups are also referred to as domains.
>> >> +
>> >> +PAPR interface currently supports different ways of communicating these
>> >> resource
>> >> +grouping details to the OS. These are referred to as Form 0, Form 1 and
>> >> Form2
>> >> +associativity grouping. Form 0 is the older format and is now considered
>> >> deprecated.
>> >> +
>> >> +Hypervisor indicates the type/form of associativity used via
>> >> "ibm,arcitecture-vec-5 property".
>> >> +Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage
>> >> of Form 0 or Form 1.
>> >> +A value of 1 indicates the usage of Form 1 associativity. For Form 2
>> >> associativity
>> >> +bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
>> >> +
>> >> +Form 0
>> >> +-
>> >> +Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
>> >> +
>> >> +Form 1
>> >> +-
>> >> +With Form 1 a combination of ibm,associativity-reference-points and
>> >> ibm,associativity
>> >> +device tree properties are used to determine the NUMA distance between
>> >> resource groups/domains.
>> >> +
>> >> +The “ibm,associativity” property contains one or more lists of numbers
>> >> (domainID)
>> >> +representing the resource’s platform grouping domains.
>> >> +
>> >> +The “ibm,associativity-reference-points” property contains one or more
>> >> list of numbers
>> >> +(domainID index) that represents the 1 based ordinal in the
>> >> associativity lists.
>> >> +The list of domainID index represnets increasing hierachy of
>> >> resource grouping.
>> >
>> > Typo "represnets". Also s/hierachy/hierarchy/
>> >
>> >> +
>> >> +ex:
>> >> +{ primary domainID index, secondary domainID index, tertiary domainID
>> >> index.. }
>> >
>> >> +Linux kernel uses the domainID at the primary domainID index as the NUMA
>> >> node id.
>> >> +Linux kernel computes NUMA distance between two domains by recursively
>> >> comparing
>> >> +if they belong to the same higher-level domains. For mismatch at every
>> >> higher
>> >> +level of the resource group, the kernel doubles the NUMA distance
>> >> between the
>> >> +comparing domains.
>> >
>> > The Form1 description is still kinda confusing, but I don't really
>> > care. Form1 *is* confusing, it's Form2 that I hope will be clearer.
>> >
>> >> +
>> >> +Form 2
>> >> +---
>> >> +Form 2 associativity format adds separate device tree properties
>> >> representing NUMA node distance
>> >> +thereby making the node distance computation flexible. Form 2 also
>> >> allows flexible primary
>> >> +domain numbering. With numa distance computation now detached from the
>> >> index value of
>> >> +"ibm,associativity" property, Form 2 allows a large number of primary
>> >> domain ids at the
>> >> +same domainID index representing resource groups of different
>> >> performance/latency characteristics.
>> >
>> > So, see you've removed the special
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
On Thu, Jun 24, 2021 at 01:50:34PM +0530, Aneesh Kumar K.V wrote:
> David Gibson writes:
>
> > On Thu, Jun 17, 2021 at 10:21:05PM +0530, Aneesh Kumar K.V wrote:
> >> PAPR interface currently supports two different ways of communicating
> >> resource
> >> grouping details to the OS. These are referred to as Form 0 and Form 1
> >> associativity grouping. Form 0 is the older format and is now considered
> >> deprecated. This patch adds another resource grouping named FORM2.
> >>
> >> Signed-off-by: Daniel Henrique Barboza
> >> Signed-off-by: Aneesh Kumar K.V
> >> ---
> >> Documentation/powerpc/associativity.rst | 135
> >> arch/powerpc/include/asm/firmware.h | 3 +-
> >> arch/powerpc/include/asm/prom.h | 1 +
> >> arch/powerpc/kernel/prom_init.c | 3 +-
> >> arch/powerpc/mm/numa.c| 149 +-
> >> arch/powerpc/platforms/pseries/firmware.c | 1 +
> >> 6 files changed, 286 insertions(+), 6 deletions(-)
> >> create mode 100644 Documentation/powerpc/associativity.rst
> >>
> >> diff --git a/Documentation/powerpc/associativity.rst
> >> b/Documentation/powerpc/associativity.rst
> >> new file mode 100644
> >> index ..93be604ac54d
> >> --- /dev/null
> >> +++ b/Documentation/powerpc/associativity.rst
> >> @@ -0,0 +1,135 @@
> >> +
> >> +NUMA resource associativity
> >> +=
> >> +
> >> +Associativity represents the groupings of the various platform resources
> >> into
> >> +domains of substantially similar mean performance relative to resources
> >> outside
> >> +of that domain. Resources subsets of a given domain that exhibit better
> >> +performance relative to each other than relative to other resources
> >> subsets
> >> +are represented as being members of a sub-grouping domain. This
> >> performance
> >> +characteristic is presented in terms of NUMA node distance within the
> >> Linux kernel.
> >> +From the platform view, these groups are also referred to as domains.
> >> +
> >> +PAPR interface currently supports different ways of communicating these
> >> resource
> >> +grouping details to the OS. These are referred to as Form 0, Form 1 and
> >> Form2
> >> +associativity grouping. Form 0 is the older format and is now considered
> >> deprecated.
> >> +
> >> +Hypervisor indicates the type/form of associativity used via
> >> "ibm,arcitecture-vec-5 property".
> >> +Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage
> >> of Form 0 or Form 1.
> >> +A value of 1 indicates the usage of Form 1 associativity. For Form 2
> >> associativity
> >> +bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
> >> +
> >> +Form 0
> >> +-
> >> +Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
> >> +
> >> +Form 1
> >> +-
> >> +With Form 1 a combination of ibm,associativity-reference-points and
> >> ibm,associativity
> >> +device tree properties are used to determine the NUMA distance between
> >> resource groups/domains.
> >> +
> >> +The “ibm,associativity” property contains one or more lists of numbers
> >> (domainID)
> >> +representing the resource’s platform grouping domains.
> >> +
> >> +The “ibm,associativity-reference-points” property contains one or more
> >> list of numbers
> >> +(domainID index) that represents the 1 based ordinal in the associativity
> >> lists.
> >> +The list of domainID index represnets increasing hierachy of
> >> resource grouping.
> >
> > Typo "represnets". Also s/hierachy/hierarchy/
> >
> >> +
> >> +ex:
> >> +{ primary domainID index, secondary domainID index, tertiary domainID
> >> index.. }
> >
> >> +Linux kernel uses the domainID at the primary domainID index as the NUMA
> >> node id.
> >> +Linux kernel computes NUMA distance between two domains by recursively
> >> comparing
> >> +if they belong to the same higher-level domains. For mismatch at every
> >> higher
> >> +level of the resource group, the kernel doubles the NUMA distance between
> >> the
> >> +comparing domains.
> >
> > The Form1 description is still kinda confusing, but I don't really
> > care. Form1 *is* confusing, it's Form2 that I hope will be clearer.
> >
> >> +
> >> +Form 2
> >> +---
> >> +Form 2 associativity format adds separate device tree properties
> >> representing NUMA node distance
> >> +thereby making the node distance computation flexible. Form 2 also allows
> >> flexible primary
> >> +domain numbering. With numa distance computation now detached from the
> >> index value of
> >> +"ibm,associativity" property, Form 2 allows a large number of primary
> >> domain ids at the
> >> +same domainID index representing resource groups of different
> >> performance/latency characteristics.
> >
> > So, see you've removed the special handling of secondary IDs for pmem
> > - big improvement, thanks. IIUC, in this revised version, for Form2
> > there's really no reason for
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
On 6/24/21 4:03 PM, Laurent Dufour wrote:
Hi Aneesh,
A little bit of wordsmithing below...
Le 17/06/2021 à 18:51, Aneesh Kumar K.V a écrit :
PAPR interface currently supports two different ways of communicating
resource
grouping details to the OS. These are referred to as Form 0 and Form 1
associativity grouping. Form 0 is the older format and is now considered
deprecated. This patch adds another resource grouping named FORM2.
Signed-off-by: Daniel Henrique Barboza
Signed-off-by: Aneesh Kumar K.V
---
Documentation/powerpc/associativity.rst | 135
arch/powerpc/include/asm/firmware.h | 3 +-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 3 +-
arch/powerpc/mm/numa.c | 149 +-
arch/powerpc/platforms/pseries/firmware.c | 1 +
6 files changed, 286 insertions(+), 6 deletions(-)
create mode 100644 Documentation/powerpc/associativity.rst
diff --git a/Documentation/powerpc/associativity.rst
b/Documentation/powerpc/associativity.rst
new file mode 100644
index ..93be604ac54d
--- /dev/null
+++ b/Documentation/powerpc/associativity.rst
@@ -0,0 +1,135 @@
+
+NUMA resource associativity
+=
+
+Associativity represents the groupings of the various platform
resources into
+domains of substantially similar mean performance relative to
resources outside
+of that domain. Resources subsets of a given domain that exhibit better
+performance relative to each other than relative to other resources
subsets
+are represented as being members of a sub-grouping domain. This
performance
+characteristic is presented in terms of NUMA node distance within the
Linux kernel.
+From the platform view, these groups are also referred to as domains.
+
+PAPR interface currently supports different ways of communicating
these resource
+grouping details to the OS. These are referred to as Form 0, Form 1
and Form2
+associativity grouping. Form 0 is the older format and is now
considered deprecated.
+
+Hypervisor indicates the type/form of associativity used via
"ibm,arcitecture-vec-5 property".
architecture ^
fixed
+Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates
usage of Form 0 or Form 1.
+A value of 1 indicates the usage of Form 1 associativity. For Form 2
associativity
+bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
+
+Form 0
+-
+Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
+
+Form 1
+-
+With Form 1 a combination of ibm,associativity-reference-points and
ibm,associativity
+device tree properties are used to determine the NUMA distance
between resource groups/domains.
+
+The “ibm,associativity” property contains one or more lists of
numbers (domainID)
+representing the resource’s platform grouping domains.
+
+The “ibm,associativity-reference-points” property contains one or
more list of numbers
+(domainID index) that represents the 1 based ordinal in the
associativity lists.
+The list of domainID index represnets increasing hierachy of resource
grouping.
represents ^
fixed
+
+ex:
+{ primary domainID index, secondary domainID index, tertiary domainID
index.. }
+
+Linux kernel uses the domainID at the primary domainID index as the
NUMA node id.
+Linux kernel computes NUMA distance between two domains by
recursively comparing
+if they belong to the same higher-level domains. For mismatch at
every higher
+level of the resource group, the kernel doubles the NUMA distance
between the
+comparing domains.
+
+Form 2
+---
+Form 2 associativity format adds separate device tree properties
representing NUMA node distance
+thereby making the node distance computation flexible. Form 2 also
allows flexible primary
+domain numbering. With numa distance computation now detached from
the index value of
+"ibm,associativity" property, Form 2 allows a large number of primary
domain ids at the
+same domainID index representing resource groups of different
performance/latency characteristics.
+
+Hypervisor indicates the usage of FORM2 associativity using bit 2 of
byte 5 in the
+"ibm,architecture-vec-5" property.
+
+"ibm,numa-lookup-index-table" property contains one or more list
numbers representing
+the domainIDs present in the system. The offset of the domainID in
this property is considered
+the domainID index.
+
+prop-encoded-array: The number N of the domainIDs encoded as with
encode-int, followed by
+N domainID encoded as with encode-int
+
+For ex:
+ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index
for domainID 8 is 1.
+
+"ibm,numa-distance-table" property contains one or more list of
numbers representing the NUMA
+distance between resource groups/domains present in the system.
+
+prop-encoded-array: The number N of the
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
Hi Aneesh,
A little bit of wordsmithing below...
Le 17/06/2021 à 18:51, Aneesh Kumar K.V a écrit :
PAPR interface currently supports two different ways of communicating resource
grouping details to the OS. These are referred to as Form 0 and Form 1
associativity grouping. Form 0 is the older format and is now considered
deprecated. This patch adds another resource grouping named FORM2.
Signed-off-by: Daniel Henrique Barboza
Signed-off-by: Aneesh Kumar K.V
---
Documentation/powerpc/associativity.rst | 135
arch/powerpc/include/asm/firmware.h | 3 +-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 3 +-
arch/powerpc/mm/numa.c| 149 +-
arch/powerpc/platforms/pseries/firmware.c | 1 +
6 files changed, 286 insertions(+), 6 deletions(-)
create mode 100644 Documentation/powerpc/associativity.rst
diff --git a/Documentation/powerpc/associativity.rst
b/Documentation/powerpc/associativity.rst
new file mode 100644
index ..93be604ac54d
--- /dev/null
+++ b/Documentation/powerpc/associativity.rst
@@ -0,0 +1,135 @@
+
+NUMA resource associativity
+=
+
+Associativity represents the groupings of the various platform resources into
+domains of substantially similar mean performance relative to resources outside
+of that domain. Resources subsets of a given domain that exhibit better
+performance relative to each other than relative to other resources subsets
+are represented as being members of a sub-grouping domain. This performance
+characteristic is presented in terms of NUMA node distance within the Linux
kernel.
+From the platform view, these groups are also referred to as domains.
+
+PAPR interface currently supports different ways of communicating these
resource
+grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
+associativity grouping. Form 0 is the older format and is now considered
deprecated.
+
+Hypervisor indicates the type/form of associativity used via "ibm,arcitecture-vec-5
property".
architecture ^
+Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
Form 0 or Form 1.
+A value of 1 indicates the usage of Form 1 associativity. For Form 2
associativity
+bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
+
+Form 0
+-
+Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
+
+Form 1
+-
+With Form 1 a combination of ibm,associativity-reference-points and
ibm,associativity
+device tree properties are used to determine the NUMA distance between
resource groups/domains.
+
+The “ibm,associativity” property contains one or more lists of numbers
(domainID)
+representing the resource’s platform grouping domains.
+
+The “ibm,associativity-reference-points” property contains one or more list of
numbers
+(domainID index) that represents the 1 based ordinal in the associativity
lists.
+The list of domainID index represnets increasing hierachy of resource grouping.
represents ^
+
+ex:
+{ primary domainID index, secondary domainID index, tertiary domainID index.. }
+
+Linux kernel uses the domainID at the primary domainID index as the NUMA node
id.
+Linux kernel computes NUMA distance between two domains by recursively
comparing
+if they belong to the same higher-level domains. For mismatch at every higher
+level of the resource group, the kernel doubles the NUMA distance between the
+comparing domains.
+
+Form 2
+---
+Form 2 associativity format adds separate device tree properties representing
NUMA node distance
+thereby making the node distance computation flexible. Form 2 also allows
flexible primary
+domain numbering. With numa distance computation now detached from the index
value of
+"ibm,associativity" property, Form 2 allows a large number of primary domain
ids at the
+same domainID index representing resource groups of different
performance/latency characteristics.
+
+Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in
the
+"ibm,architecture-vec-5" property.
+
+"ibm,numa-lookup-index-table" property contains one or more list numbers
representing
+the domainIDs present in the system. The offset of the domainID in this
property is considered
+the domainID index.
+
+prop-encoded-array: The number N of the domainIDs encoded as with encode-int,
followed by
+N domainID encoded as with encode-int
+
+For ex:
+ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index for
domainID 8 is 1.
+
+"ibm,numa-distance-table" property contains one or more list of numbers
representing the NUMA
+distance between resource groups/domains present in the system.
+
+prop-encoded-array: The number N of the distance values encoded as with
encode-int, followed by
+N distance values encoded as with
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
David Gibson writes:
> On Thu, Jun 17, 2021 at 10:21:05PM +0530, Aneesh Kumar K.V wrote:
>> PAPR interface currently supports two different ways of communicating
>> resource
>> grouping details to the OS. These are referred to as Form 0 and Form 1
>> associativity grouping. Form 0 is the older format and is now considered
>> deprecated. This patch adds another resource grouping named FORM2.
>>
>> Signed-off-by: Daniel Henrique Barboza
>> Signed-off-by: Aneesh Kumar K.V
>> ---
>> Documentation/powerpc/associativity.rst | 135
>> arch/powerpc/include/asm/firmware.h | 3 +-
>> arch/powerpc/include/asm/prom.h | 1 +
>> arch/powerpc/kernel/prom_init.c | 3 +-
>> arch/powerpc/mm/numa.c| 149 +-
>> arch/powerpc/platforms/pseries/firmware.c | 1 +
>> 6 files changed, 286 insertions(+), 6 deletions(-)
>> create mode 100644 Documentation/powerpc/associativity.rst
>>
>> diff --git a/Documentation/powerpc/associativity.rst
>> b/Documentation/powerpc/associativity.rst
>> new file mode 100644
>> index ..93be604ac54d
>> --- /dev/null
>> +++ b/Documentation/powerpc/associativity.rst
>> @@ -0,0 +1,135 @@
>> +
>> +NUMA resource associativity
>> +=
>> +
>> +Associativity represents the groupings of the various platform resources
>> into
>> +domains of substantially similar mean performance relative to resources
>> outside
>> +of that domain. Resources subsets of a given domain that exhibit better
>> +performance relative to each other than relative to other resources subsets
>> +are represented as being members of a sub-grouping domain. This performance
>> +characteristic is presented in terms of NUMA node distance within the Linux
>> kernel.
>> +From the platform view, these groups are also referred to as domains.
>> +
>> +PAPR interface currently supports different ways of communicating these
>> resource
>> +grouping details to the OS. These are referred to as Form 0, Form 1 and
>> Form2
>> +associativity grouping. Form 0 is the older format and is now considered
>> deprecated.
>> +
>> +Hypervisor indicates the type/form of associativity used via
>> "ibm,arcitecture-vec-5 property".
>> +Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
>> Form 0 or Form 1.
>> +A value of 1 indicates the usage of Form 1 associativity. For Form 2
>> associativity
>> +bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
>> +
>> +Form 0
>> +-
>> +Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
>> +
>> +Form 1
>> +-
>> +With Form 1 a combination of ibm,associativity-reference-points and
>> ibm,associativity
>> +device tree properties are used to determine the NUMA distance between
>> resource groups/domains.
>> +
>> +The “ibm,associativity” property contains one or more lists of numbers
>> (domainID)
>> +representing the resource’s platform grouping domains.
>> +
>> +The “ibm,associativity-reference-points” property contains one or more list
>> of numbers
>> +(domainID index) that represents the 1 based ordinal in the associativity
>> lists.
>> +The list of domainID index represnets increasing hierachy of
>> resource grouping.
>
> Typo "represnets". Also s/hierachy/hierarchy/
>
>> +
>> +ex:
>> +{ primary domainID index, secondary domainID index, tertiary domainID
>> index.. }
>
>> +Linux kernel uses the domainID at the primary domainID index as the NUMA
>> node id.
>> +Linux kernel computes NUMA distance between two domains by recursively
>> comparing
>> +if they belong to the same higher-level domains. For mismatch at every
>> higher
>> +level of the resource group, the kernel doubles the NUMA distance between
>> the
>> +comparing domains.
>
> The Form1 description is still kinda confusing, but I don't really
> care. Form1 *is* confusing, it's Form2 that I hope will be clearer.
>
>> +
>> +Form 2
>> +---
>> +Form 2 associativity format adds separate device tree properties
>> representing NUMA node distance
>> +thereby making the node distance computation flexible. Form 2 also allows
>> flexible primary
>> +domain numbering. With numa distance computation now detached from the
>> index value of
>> +"ibm,associativity" property, Form 2 allows a large number of primary
>> domain ids at the
>> +same domainID index representing resource groups of different
>> performance/latency characteristics.
>
> So, see you've removed the special handling of secondary IDs for pmem
> - big improvement, thanks. IIUC, in this revised version, for Form2
> there's really no reason for ibm,associativity-reference-points to
> have >1 entry. Is that right?
>
> In Form2 everything revolves around the primary domain ID - so much
> that I suggest we come up with a short name for it. How about just
> "node id" since that's how Linux uses it.
We don't really refer primary domain ID in rest of FORM2
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
On Thu, Jun 17, 2021 at 10:21:05PM +0530, Aneesh Kumar K.V wrote:
> PAPR interface currently supports two different ways of communicating resource
> grouping details to the OS. These are referred to as Form 0 and Form 1
> associativity grouping. Form 0 is the older format and is now considered
> deprecated. This patch adds another resource grouping named FORM2.
>
> Signed-off-by: Daniel Henrique Barboza
> Signed-off-by: Aneesh Kumar K.V
> ---
> Documentation/powerpc/associativity.rst | 135
> arch/powerpc/include/asm/firmware.h | 3 +-
> arch/powerpc/include/asm/prom.h | 1 +
> arch/powerpc/kernel/prom_init.c | 3 +-
> arch/powerpc/mm/numa.c| 149 +-
> arch/powerpc/platforms/pseries/firmware.c | 1 +
> 6 files changed, 286 insertions(+), 6 deletions(-)
> create mode 100644 Documentation/powerpc/associativity.rst
>
> diff --git a/Documentation/powerpc/associativity.rst
> b/Documentation/powerpc/associativity.rst
> new file mode 100644
> index ..93be604ac54d
> --- /dev/null
> +++ b/Documentation/powerpc/associativity.rst
> @@ -0,0 +1,135 @@
> +
> +NUMA resource associativity
> +=
> +
> +Associativity represents the groupings of the various platform resources into
> +domains of substantially similar mean performance relative to resources
> outside
> +of that domain. Resources subsets of a given domain that exhibit better
> +performance relative to each other than relative to other resources subsets
> +are represented as being members of a sub-grouping domain. This performance
> +characteristic is presented in terms of NUMA node distance within the Linux
> kernel.
> +From the platform view, these groups are also referred to as domains.
> +
> +PAPR interface currently supports different ways of communicating these
> resource
> +grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
> +associativity grouping. Form 0 is the older format and is now considered
> deprecated.
> +
> +Hypervisor indicates the type/form of associativity used via
> "ibm,arcitecture-vec-5 property".
> +Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
> Form 0 or Form 1.
> +A value of 1 indicates the usage of Form 1 associativity. For Form 2
> associativity
> +bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
> +
> +Form 0
> +-
> +Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
> +
> +Form 1
> +-
> +With Form 1 a combination of ibm,associativity-reference-points and
> ibm,associativity
> +device tree properties are used to determine the NUMA distance between
> resource groups/domains.
> +
> +The “ibm,associativity” property contains one or more lists of numbers
> (domainID)
> +representing the resource’s platform grouping domains.
> +
> +The “ibm,associativity-reference-points” property contains one or more list
> of numbers
> +(domainID index) that represents the 1 based ordinal in the associativity
> lists.
> +The list of domainID index represnets increasing hierachy of
> resource grouping.
Typo "represnets". Also s/hierachy/hierarchy/
> +
> +ex:
> +{ primary domainID index, secondary domainID index, tertiary domainID
> index.. }
> +Linux kernel uses the domainID at the primary domainID index as the NUMA
> node id.
> +Linux kernel computes NUMA distance between two domains by recursively
> comparing
> +if they belong to the same higher-level domains. For mismatch at every higher
> +level of the resource group, the kernel doubles the NUMA distance between the
> +comparing domains.
The Form1 description is still kinda confusing, but I don't really
care. Form1 *is* confusing, it's Form2 that I hope will be clearer.
> +
> +Form 2
> +---
> +Form 2 associativity format adds separate device tree properties
> representing NUMA node distance
> +thereby making the node distance computation flexible. Form 2 also allows
> flexible primary
> +domain numbering. With numa distance computation now detached from the index
> value of
> +"ibm,associativity" property, Form 2 allows a large number of primary domain
> ids at the
> +same domainID index representing resource groups of different
> performance/latency characteristics.
So, see you've removed the special handling of secondary IDs for pmem
- big improvement, thanks. IIUC, in this revised version, for Form2
there's really no reason for ibm,associativity-reference-points to
have >1 entry. Is that right?
In Form2 everything revolves around the primary domain ID - so much
that I suggest we come up with a short name for it. How about just
"node id" since that's how Linux uses it.
> +Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5
> in the
> +"ibm,architecture-vec-5" property.
> +
> +"ibm,numa-lookup-index-table" property contains one or more list numbers
> representing
>
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
On 6/22/21 9:07 AM, Aneesh Kumar K.V wrote:
Daniel Henrique Barboza writes:
On 6/17/21 1:51 PM, Aneesh Kumar K.V wrote:
PAPR interface currently supports two different ways of communicating resource
grouping details to the OS. These are referred to as Form 0 and Form 1
associativity grouping. Form 0 is the older format and is now considered
deprecated. This patch adds another resource grouping named FORM2.
Signed-off-by: Daniel Henrique Barboza
Signed-off-by: Aneesh Kumar K.V
---
Documentation/powerpc/associativity.rst | 135
arch/powerpc/include/asm/firmware.h | 3 +-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 3 +-
arch/powerpc/mm/numa.c| 149 +-
arch/powerpc/platforms/pseries/firmware.c | 1 +
6 files changed, 286 insertions(+), 6 deletions(-)
create mode 100644 Documentation/powerpc/associativity.rst
diff --git a/Documentation/powerpc/associativity.rst
b/Documentation/powerpc/associativity.rst
new file mode 100644
index ..93be604ac54d
--- /dev/null
+++ b/Documentation/powerpc/associativity.rst
@@ -0,0 +1,135 @@
+
+NUMA resource associativity
+=
+
+Associativity represents the groupings of the various platform resources into
+domains of substantially similar mean performance relative to resources outside
+of that domain. Resources subsets of a given domain that exhibit better
+performance relative to each other than relative to other resources subsets
+are represented as being members of a sub-grouping domain. This performance
+characteristic is presented in terms of NUMA node distance within the Linux
kernel.
+From the platform view, these groups are also referred to as domains.
+
+PAPR interface currently supports different ways of communicating these
resource
+grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
+associativity grouping. Form 0 is the older format and is now considered
deprecated.
+
+Hypervisor indicates the type/form of associativity used via "ibm,arcitecture-vec-5
property".
+Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
Form 0 or Form 1.
+A value of 1 indicates the usage of Form 1 associativity. For Form 2
associativity
+bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
+
+Form 0
+-
+Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
+
+Form 1
+-
+With Form 1 a combination of ibm,associativity-reference-points and
ibm,associativity
+device tree properties are used to determine the NUMA distance between
resource groups/domains.
+
+The “ibm,associativity” property contains one or more lists of numbers
(domainID)
+representing the resource’s platform grouping domains.
+
+The “ibm,associativity-reference-points” property contains one or more list of
numbers
+(domainID index) that represents the 1 based ordinal in the associativity
lists.
+The list of domainID index represnets increasing hierachy of resource grouping.
+
+ex:
+{ primary domainID index, secondary domainID index, tertiary domainID index.. }
+
+Linux kernel uses the domainID at the primary domainID index as the NUMA node
id.
+Linux kernel computes NUMA distance between two domains by recursively
comparing
+if they belong to the same higher-level domains. For mismatch at every higher
+level of the resource group, the kernel doubles the NUMA distance between the
+comparing domains.
+
+Form 2
+---
+Form 2 associativity format adds separate device tree properties representing
NUMA node distance
+thereby making the node distance computation flexible. Form 2 also allows
flexible primary
+domain numbering. With numa distance computation now detached from the index
value of
+"ibm,associativity" property, Form 2 allows a large number of primary domain
ids at the
+same domainID index representing resource groups of different
performance/latency characteristics.
+
+Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in
the
+"ibm,architecture-vec-5" property.
+
+"ibm,numa-lookup-index-table" property contains one or more list numbers
representing
+the domainIDs present in the system. The offset of the domainID in this
property is considered
+the domainID index.
+
+prop-encoded-array: The number N of the domainIDs encoded as with encode-int,
followed by
+N domainID encoded as with encode-int
+
+For ex:
+ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index for
domainID 8 is 1.
+
+"ibm,numa-distance-table" property contains one or more list of numbers
representing the NUMA
+distance between resource groups/domains present in the system.
+
+prop-encoded-array: The number N of the distance values encoded as with
encode-int, followed by
+N distance values encoded as with encode-bytes. The max distance value we
could encode is 255.
+
+For ex:
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
Daniel Henrique Barboza writes:
> On 6/17/21 1:51 PM, Aneesh Kumar K.V wrote:
>> PAPR interface currently supports two different ways of communicating
>> resource
>> grouping details to the OS. These are referred to as Form 0 and Form 1
>> associativity grouping. Form 0 is the older format and is now considered
>> deprecated. This patch adds another resource grouping named FORM2.
>>
>> Signed-off-by: Daniel Henrique Barboza
>> Signed-off-by: Aneesh Kumar K.V
>> ---
>> Documentation/powerpc/associativity.rst | 135
>> arch/powerpc/include/asm/firmware.h | 3 +-
>> arch/powerpc/include/asm/prom.h | 1 +
>> arch/powerpc/kernel/prom_init.c | 3 +-
>> arch/powerpc/mm/numa.c| 149 +-
>> arch/powerpc/platforms/pseries/firmware.c | 1 +
>> 6 files changed, 286 insertions(+), 6 deletions(-)
>> create mode 100644 Documentation/powerpc/associativity.rst
>>
>> diff --git a/Documentation/powerpc/associativity.rst
>> b/Documentation/powerpc/associativity.rst
>> new file mode 100644
>> index ..93be604ac54d
>> --- /dev/null
>> +++ b/Documentation/powerpc/associativity.rst
>> @@ -0,0 +1,135 @@
>> +
>> +NUMA resource associativity
>> +=
>> +
>> +Associativity represents the groupings of the various platform resources
>> into
>> +domains of substantially similar mean performance relative to resources
>> outside
>> +of that domain. Resources subsets of a given domain that exhibit better
>> +performance relative to each other than relative to other resources subsets
>> +are represented as being members of a sub-grouping domain. This performance
>> +characteristic is presented in terms of NUMA node distance within the Linux
>> kernel.
>> +From the platform view, these groups are also referred to as domains.
>> +
>> +PAPR interface currently supports different ways of communicating these
>> resource
>> +grouping details to the OS. These are referred to as Form 0, Form 1 and
>> Form2
>> +associativity grouping. Form 0 is the older format and is now considered
>> deprecated.
>> +
>> +Hypervisor indicates the type/form of associativity used via
>> "ibm,arcitecture-vec-5 property".
>> +Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
>> Form 0 or Form 1.
>> +A value of 1 indicates the usage of Form 1 associativity. For Form 2
>> associativity
>> +bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
>> +
>> +Form 0
>> +-
>> +Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
>> +
>> +Form 1
>> +-
>> +With Form 1 a combination of ibm,associativity-reference-points and
>> ibm,associativity
>> +device tree properties are used to determine the NUMA distance between
>> resource groups/domains.
>> +
>> +The “ibm,associativity” property contains one or more lists of numbers
>> (domainID)
>> +representing the resource’s platform grouping domains.
>> +
>> +The “ibm,associativity-reference-points” property contains one or more list
>> of numbers
>> +(domainID index) that represents the 1 based ordinal in the associativity
>> lists.
>> +The list of domainID index represnets increasing hierachy of resource
>> grouping.
>> +
>> +ex:
>> +{ primary domainID index, secondary domainID index, tertiary domainID
>> index.. }
>> +
>> +Linux kernel uses the domainID at the primary domainID index as the NUMA
>> node id.
>> +Linux kernel computes NUMA distance between two domains by recursively
>> comparing
>> +if they belong to the same higher-level domains. For mismatch at every
>> higher
>> +level of the resource group, the kernel doubles the NUMA distance between
>> the
>> +comparing domains.
>> +
>> +Form 2
>> +---
>> +Form 2 associativity format adds separate device tree properties
>> representing NUMA node distance
>> +thereby making the node distance computation flexible. Form 2 also allows
>> flexible primary
>> +domain numbering. With numa distance computation now detached from the
>> index value of
>> +"ibm,associativity" property, Form 2 allows a large number of primary
>> domain ids at the
>> +same domainID index representing resource groups of different
>> performance/latency characteristics.
>> +
>> +Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5
>> in the
>> +"ibm,architecture-vec-5" property.
>> +
>> +"ibm,numa-lookup-index-table" property contains one or more list numbers
>> representing
>> +the domainIDs present in the system. The offset of the domainID in this
>> property is considered
>> +the domainID index.
>> +
>> +prop-encoded-array: The number N of the domainIDs encoded as with
>> encode-int, followed by
>> +N domainID encoded as with encode-int
>> +
>> +For ex:
>> +ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index for
>> domainID 8 is 1.
>> +
>> +"ibm,numa-distance-table" property contains one or more
Re: [PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
On 6/17/21 1:51 PM, Aneesh Kumar K.V wrote:
PAPR interface currently supports two different ways of communicating resource
grouping details to the OS. These are referred to as Form 0 and Form 1
associativity grouping. Form 0 is the older format and is now considered
deprecated. This patch adds another resource grouping named FORM2.
Signed-off-by: Daniel Henrique Barboza
Signed-off-by: Aneesh Kumar K.V
---
Documentation/powerpc/associativity.rst | 135
arch/powerpc/include/asm/firmware.h | 3 +-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 3 +-
arch/powerpc/mm/numa.c| 149 +-
arch/powerpc/platforms/pseries/firmware.c | 1 +
6 files changed, 286 insertions(+), 6 deletions(-)
create mode 100644 Documentation/powerpc/associativity.rst
diff --git a/Documentation/powerpc/associativity.rst
b/Documentation/powerpc/associativity.rst
new file mode 100644
index ..93be604ac54d
--- /dev/null
+++ b/Documentation/powerpc/associativity.rst
@@ -0,0 +1,135 @@
+
+NUMA resource associativity
+=
+
+Associativity represents the groupings of the various platform resources into
+domains of substantially similar mean performance relative to resources outside
+of that domain. Resources subsets of a given domain that exhibit better
+performance relative to each other than relative to other resources subsets
+are represented as being members of a sub-grouping domain. This performance
+characteristic is presented in terms of NUMA node distance within the Linux
kernel.
+From the platform view, these groups are also referred to as domains.
+
+PAPR interface currently supports different ways of communicating these
resource
+grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
+associativity grouping. Form 0 is the older format and is now considered
deprecated.
+
+Hypervisor indicates the type/form of associativity used via "ibm,arcitecture-vec-5
property".
+Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
Form 0 or Form 1.
+A value of 1 indicates the usage of Form 1 associativity. For Form 2
associativity
+bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
+
+Form 0
+-
+Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
+
+Form 1
+-
+With Form 1 a combination of ibm,associativity-reference-points and
ibm,associativity
+device tree properties are used to determine the NUMA distance between
resource groups/domains.
+
+The “ibm,associativity” property contains one or more lists of numbers
(domainID)
+representing the resource’s platform grouping domains.
+
+The “ibm,associativity-reference-points” property contains one or more list of
numbers
+(domainID index) that represents the 1 based ordinal in the associativity
lists.
+The list of domainID index represnets increasing hierachy of resource grouping.
+
+ex:
+{ primary domainID index, secondary domainID index, tertiary domainID index.. }
+
+Linux kernel uses the domainID at the primary domainID index as the NUMA node
id.
+Linux kernel computes NUMA distance between two domains by recursively
comparing
+if they belong to the same higher-level domains. For mismatch at every higher
+level of the resource group, the kernel doubles the NUMA distance between the
+comparing domains.
+
+Form 2
+---
+Form 2 associativity format adds separate device tree properties representing
NUMA node distance
+thereby making the node distance computation flexible. Form 2 also allows
flexible primary
+domain numbering. With numa distance computation now detached from the index
value of
+"ibm,associativity" property, Form 2 allows a large number of primary domain
ids at the
+same domainID index representing resource groups of different
performance/latency characteristics.
+
+Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in
the
+"ibm,architecture-vec-5" property.
+
+"ibm,numa-lookup-index-table" property contains one or more list numbers
representing
+the domainIDs present in the system. The offset of the domainID in this
property is considered
+the domainID index.
+
+prop-encoded-array: The number N of the domainIDs encoded as with encode-int,
followed by
+N domainID encoded as with encode-int
+
+For ex:
+ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index for
domainID 8 is 1.
+
+"ibm,numa-distance-table" property contains one or more list of numbers
representing the NUMA
+distance between resource groups/domains present in the system.
+
+prop-encoded-array: The number N of the distance values encoded as with
encode-int, followed by
+N distance values encoded as with encode-bytes. The max distance value we
could encode is 255.
+
+For ex:
+ibm,numa-lookup-index-table = {3, 0, 8, 40}
+ibm,numa-distance-table = {9, 10, 20, 80, 20,
[PATCH v4 7/7] powerpc/pseries: Add support for FORM2 associativity
PAPR interface currently supports two different ways of communicating resource
grouping details to the OS. These are referred to as Form 0 and Form 1
associativity grouping. Form 0 is the older format and is now considered
deprecated. This patch adds another resource grouping named FORM2.
Signed-off-by: Daniel Henrique Barboza
Signed-off-by: Aneesh Kumar K.V
---
Documentation/powerpc/associativity.rst | 135
arch/powerpc/include/asm/firmware.h | 3 +-
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom_init.c | 3 +-
arch/powerpc/mm/numa.c| 149 +-
arch/powerpc/platforms/pseries/firmware.c | 1 +
6 files changed, 286 insertions(+), 6 deletions(-)
create mode 100644 Documentation/powerpc/associativity.rst
diff --git a/Documentation/powerpc/associativity.rst
b/Documentation/powerpc/associativity.rst
new file mode 100644
index ..93be604ac54d
--- /dev/null
+++ b/Documentation/powerpc/associativity.rst
@@ -0,0 +1,135 @@
+
+NUMA resource associativity
+=
+
+Associativity represents the groupings of the various platform resources into
+domains of substantially similar mean performance relative to resources outside
+of that domain. Resources subsets of a given domain that exhibit better
+performance relative to each other than relative to other resources subsets
+are represented as being members of a sub-grouping domain. This performance
+characteristic is presented in terms of NUMA node distance within the Linux
kernel.
+From the platform view, these groups are also referred to as domains.
+
+PAPR interface currently supports different ways of communicating these
resource
+grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
+associativity grouping. Form 0 is the older format and is now considered
deprecated.
+
+Hypervisor indicates the type/form of associativity used via
"ibm,arcitecture-vec-5 property".
+Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of
Form 0 or Form 1.
+A value of 1 indicates the usage of Form 1 associativity. For Form 2
associativity
+bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
+
+Form 0
+-
+Form 0 associativity supports only two NUMA distance (LOCAL and REMOTE).
+
+Form 1
+-
+With Form 1 a combination of ibm,associativity-reference-points and
ibm,associativity
+device tree properties are used to determine the NUMA distance between
resource groups/domains.
+
+The “ibm,associativity” property contains one or more lists of numbers
(domainID)
+representing the resource’s platform grouping domains.
+
+The “ibm,associativity-reference-points” property contains one or more list of
numbers
+(domainID index) that represents the 1 based ordinal in the associativity
lists.
+The list of domainID index represnets increasing hierachy of resource
grouping.
+
+ex:
+{ primary domainID index, secondary domainID index, tertiary domainID index.. }
+
+Linux kernel uses the domainID at the primary domainID index as the NUMA node
id.
+Linux kernel computes NUMA distance between two domains by recursively
comparing
+if they belong to the same higher-level domains. For mismatch at every higher
+level of the resource group, the kernel doubles the NUMA distance between the
+comparing domains.
+
+Form 2
+---
+Form 2 associativity format adds separate device tree properties representing
NUMA node distance
+thereby making the node distance computation flexible. Form 2 also allows
flexible primary
+domain numbering. With numa distance computation now detached from the index
value of
+"ibm,associativity" property, Form 2 allows a large number of primary domain
ids at the
+same domainID index representing resource groups of different
performance/latency characteristics.
+
+Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in
the
+"ibm,architecture-vec-5" property.
+
+"ibm,numa-lookup-index-table" property contains one or more list numbers
representing
+the domainIDs present in the system. The offset of the domainID in this
property is considered
+the domainID index.
+
+prop-encoded-array: The number N of the domainIDs encoded as with encode-int,
followed by
+N domainID encoded as with encode-int
+
+For ex:
+ibm,numa-lookup-index-table = {4, 0, 8, 250, 252}, domainID index for
domainID 8 is 1.
+
+"ibm,numa-distance-table" property contains one or more list of numbers
representing the NUMA
+distance between resource groups/domains present in the system.
+
+prop-encoded-array: The number N of the distance values encoded as with
encode-int, followed by
+N distance values encoded as with encode-bytes. The max distance value we
could encode is 255.
+
+For ex:
+ibm,numa-lookup-index-table = {3, 0, 8, 40}
+ibm,numa-distance-table = {9, 10, 20, 80, 20, 10, 160, 80, 160, 10}
+
+ | 08 40
