Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Thomas,

On 2/19/2018 2:19 PM, Thomas Gleixner wrote:
>> It is required that the user space application self sets affinity to
>> cores associated with the cache. This is also highlighted in the example
>> application code (later in this patch) within the comments as well as
>> the example usage of sched_setaffinity(). The enforcement done in the
>> kernel code is done as a check that the user space application did so,
>> no the actual affinity management.
> 
> Right, but your documentation claims it's enforced. There is no enforcement
> aside of the initial sanity check.

I see the confusion. I will fix the documentation to clarify that it is
a sanity check.

Thank you

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Thomas,

On 2/19/2018 2:19 PM, Thomas Gleixner wrote:
>> It is required that the user space application self sets affinity to
>> cores associated with the cache. This is also highlighted in the example
>> application code (later in this patch) within the comments as well as
>> the example usage of sched_setaffinity(). The enforcement done in the
>> kernel code is done as a check that the user space application did so,
>> no the actual affinity management.
> 
> Right, but your documentation claims it's enforced. There is no enforcement
> aside of the initial sanity check.

I see the confusion. I will fix the documentation to clarify that it is
a sanity check.

Thank you

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Randy,

On 2/19/2018 1:27 PM, Randy Dunlap wrote:
> On 02/13/18 07:46, Reinette Chatre wrote:
>> Add description of Cache Pseudo-Locking feature, its interface,
>> as well as an example of its usage.
>>
>> Signed-off-by: Reinette Chatre 
>> ---
>>  Documentation/x86/intel_rdt_ui.txt | 229 
>> -
>>  1 file changed, 228 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/x86/intel_rdt_ui.txt 
>> b/Documentation/x86/intel_rdt_ui.txt
>> index 756fd76b78a6..bb3d6fe0a3e4 100644
>> --- a/Documentation/x86/intel_rdt_ui.txt
>> +++ b/Documentation/x86/intel_rdt_ui.txt
> 
>> @@ -329,6 +332,149 @@ L3CODE:0=f;1=f;2=f;3=f
>>  L3DATA:0=f;1=f;2=3c0;3=f
>>  L3CODE:0=f;1=f;2=f;3=f
>>  
>> +Cache Pseudo-Locking
>> +
>> +CAT enables a user to specify the amount of cache space into which an
> 
>  space that an

Will fix.

> 
>> +application can fill. Cache pseudo-locking builds on the fact that a
>> +CPU can still read and write data pre-allocated outside its current
>> +allocated area on a cache hit. With cache pseudo-locking, data can be
>> +preloaded into a reserved portion of cache that no application can
>> +fill, and from that point on will only serve cache hits. The cache
>> +pseudo-locked memory is made accessible to user space where an
>> +application can map it into its virtual address space and thus have
>> +a region of memory with reduced average read latency.
>> +
>> +Cache pseudo-locking increases the probability that data will remain
>> +in the cache via carefully configuring the CAT feature and controlling
>> +application behavior. There is no guarantee that data is placed in
>> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
>> +“locked” data from cache. Power management C-states may shrink or
>> +power off cache. It is thus recommended to limit the processor maximum
>> +C-state, for example, by setting the processor.max_cstate kernel parameter.
>> +
>> +It is required that an application using a pseudo-locked region runs
>> +with affinity to the cores (or a subset of the cores) associated
>> +with the cache on which the pseudo-locked region resides. This is
>> +enforced by the implementation.
>> +
>> +Pseudo-locking is accomplished in two stages:
>> +1) During the first stage the system administrator allocates a portion
>> +   of cache that should be dedicated to pseudo-locking. At this time an
>> +   equivalent portion of memory is allocated, loaded into allocated
>> +   cache portion, and exposed as a character device.
>> +2) During the second stage a user-space application maps (mmap()) the
>> +   pseudo-locked memory into its address space.
>> +
>> +Cache Pseudo-Locking Interface
>> +--
>> +Platforms supporting cache pseudo-locking will expose a new
>> +"/sys/fs/restrl/pseudo_lock" directory after successful mount of the
>> +resctrl filesystem. Initially this directory will contain a single file,
>> +"avail" that contains the schemata, one line per resource, of cache region
>> +available for pseudo-locking.
> 
> uh, sysfs is supposed to be one value per file.

This builds on how the schemata file currently works.

>> +A pseudo-locked region is created by creating a new directory within
>> +/sys/fs/resctrl/pseudo_lock. On success two new files will appear in
>> +the directory:
>> +
>> +"schemata":
>> +Shows the schemata representing the pseudo-locked cache region.
>> +User writes schemata of requested locked area to file.
> 
>   use complete sentences, please. E.g.:
> 
>   The user writes the schemata of the requested locked area to the file.
> 
>> +Only one id of single resource accepted - can only lock from
> 
>   of a single resource is accepted -
> 

Will fix both.

>> +single cache instance. Writing of schemata to this file will
>> +return success on successful pseudo-locked region setup.
>> +"size":
>> +After successful pseudo-locked region setup this read-only file
>> +will contain the size in bytes of pseudo-locked region.
> 
> 

Thank you very much for taking a look!

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Randy,

On 2/19/2018 1:27 PM, Randy Dunlap wrote:
> On 02/13/18 07:46, Reinette Chatre wrote:
>> Add description of Cache Pseudo-Locking feature, its interface,
>> as well as an example of its usage.
>>
>> Signed-off-by: Reinette Chatre 
>> ---
>>  Documentation/x86/intel_rdt_ui.txt | 229 
>> -
>>  1 file changed, 228 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/x86/intel_rdt_ui.txt 
>> b/Documentation/x86/intel_rdt_ui.txt
>> index 756fd76b78a6..bb3d6fe0a3e4 100644
>> --- a/Documentation/x86/intel_rdt_ui.txt
>> +++ b/Documentation/x86/intel_rdt_ui.txt
> 
>> @@ -329,6 +332,149 @@ L3CODE:0=f;1=f;2=f;3=f
>>  L3DATA:0=f;1=f;2=3c0;3=f
>>  L3CODE:0=f;1=f;2=f;3=f
>>  
>> +Cache Pseudo-Locking
>> +
>> +CAT enables a user to specify the amount of cache space into which an
> 
>  space that an

Will fix.

> 
>> +application can fill. Cache pseudo-locking builds on the fact that a
>> +CPU can still read and write data pre-allocated outside its current
>> +allocated area on a cache hit. With cache pseudo-locking, data can be
>> +preloaded into a reserved portion of cache that no application can
>> +fill, and from that point on will only serve cache hits. The cache
>> +pseudo-locked memory is made accessible to user space where an
>> +application can map it into its virtual address space and thus have
>> +a region of memory with reduced average read latency.
>> +
>> +Cache pseudo-locking increases the probability that data will remain
>> +in the cache via carefully configuring the CAT feature and controlling
>> +application behavior. There is no guarantee that data is placed in
>> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
>> +“locked” data from cache. Power management C-states may shrink or
>> +power off cache. It is thus recommended to limit the processor maximum
>> +C-state, for example, by setting the processor.max_cstate kernel parameter.
>> +
>> +It is required that an application using a pseudo-locked region runs
>> +with affinity to the cores (or a subset of the cores) associated
>> +with the cache on which the pseudo-locked region resides. This is
>> +enforced by the implementation.
>> +
>> +Pseudo-locking is accomplished in two stages:
>> +1) During the first stage the system administrator allocates a portion
>> +   of cache that should be dedicated to pseudo-locking. At this time an
>> +   equivalent portion of memory is allocated, loaded into allocated
>> +   cache portion, and exposed as a character device.
>> +2) During the second stage a user-space application maps (mmap()) the
>> +   pseudo-locked memory into its address space.
>> +
>> +Cache Pseudo-Locking Interface
>> +--
>> +Platforms supporting cache pseudo-locking will expose a new
>> +"/sys/fs/restrl/pseudo_lock" directory after successful mount of the
>> +resctrl filesystem. Initially this directory will contain a single file,
>> +"avail" that contains the schemata, one line per resource, of cache region
>> +available for pseudo-locking.
> 
> uh, sysfs is supposed to be one value per file.

This builds on how the schemata file currently works.

>> +A pseudo-locked region is created by creating a new directory within
>> +/sys/fs/resctrl/pseudo_lock. On success two new files will appear in
>> +the directory:
>> +
>> +"schemata":
>> +Shows the schemata representing the pseudo-locked cache region.
>> +User writes schemata of requested locked area to file.
> 
>   use complete sentences, please. E.g.:
> 
>   The user writes the schemata of the requested locked area to the file.
> 
>> +Only one id of single resource accepted - can only lock from
> 
>   of a single resource is accepted -
> 

Will fix both.

>> +single cache instance. Writing of schemata to this file will
>> +return success on successful pseudo-locked region setup.
>> +"size":
>> +After successful pseudo-locked region setup this read-only file
>> +will contain the size in bytes of pseudo-locked region.
> 
> 

Thank you very much for taking a look!

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Thomas Gleixner]

On Mon, 19 Feb 2018, Reinette Chatre wrote:
> Hi Thomas,
> 
> On 2/19/2018 12:35 PM, Thomas Gleixner wrote:
> > On Tue, 13 Feb 2018, Reinette Chatre wrote:
> >> +Cache Pseudo-Locking
> >> +
> >> +CAT enables a user to specify the amount of cache space into which an
> >> +application can fill. Cache pseudo-locking builds on the fact that a
> >> +CPU can still read and write data pre-allocated outside its current
> >> +allocated area on a cache hit. With cache pseudo-locking, data can be
> >> +preloaded into a reserved portion of cache that no application can
> >> +fill, and from that point on will only serve cache hits.
> > 
> > This lacks explanation how that preloading works.
> 
> Following this text you quote there is a brief explanation starting with
> "Pseudo-locking is accomplished in two stages:" - I'll add more details
> to that area.
> 
> > 
> >> The cache
> >> +pseudo-locked memory is made accessible to user space where an
> >> +application can map it into its virtual address space and thus have
> >> +a region of memory with reduced average read latency.
> >> +
> >> +Cache pseudo-locking increases the probability that data will remain
> >> +in the cache via carefully configuring the CAT feature and controlling
> >> +application behavior. There is no guarantee that data is placed in
> >> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> >> +“locked” data from cache. Power management C-states may shrink or
> >> +power off cache. It is thus recommended to limit the processor maximum
> >> +C-state, for example, by setting the processor.max_cstate kernel 
> >> parameter.
> >> +
> >> +It is required that an application using a pseudo-locked region runs
> >> +with affinity to the cores (or a subset of the cores) associated
> >> +with the cache on which the pseudo-locked region resides. This is
> >> +enforced by the implementation.
> > 
> > Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
> > to the right CPUs. But that's not a guarantee that the task stays there.
> 
> It is required that the user space application self sets affinity to
> cores associated with the cache. This is also highlighted in the example
> application code (later in this patch) within the comments as well as
> the example usage of sched_setaffinity(). The enforcement done in the
> kernel code is done as a check that the user space application did so,
> no the actual affinity management.

Right, but your documentation claims it's enforced. There is no enforcement
aside of the initial sanity check.

Thanks,

tglx

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Thomas Gleixner]

On Mon, 19 Feb 2018, Reinette Chatre wrote:
> Hi Thomas,
> 
> On 2/19/2018 12:35 PM, Thomas Gleixner wrote:
> > On Tue, 13 Feb 2018, Reinette Chatre wrote:
> >> +Cache Pseudo-Locking
> >> +
> >> +CAT enables a user to specify the amount of cache space into which an
> >> +application can fill. Cache pseudo-locking builds on the fact that a
> >> +CPU can still read and write data pre-allocated outside its current
> >> +allocated area on a cache hit. With cache pseudo-locking, data can be
> >> +preloaded into a reserved portion of cache that no application can
> >> +fill, and from that point on will only serve cache hits.
> > 
> > This lacks explanation how that preloading works.
> 
> Following this text you quote there is a brief explanation starting with
> "Pseudo-locking is accomplished in two stages:" - I'll add more details
> to that area.
> 
> > 
> >> The cache
> >> +pseudo-locked memory is made accessible to user space where an
> >> +application can map it into its virtual address space and thus have
> >> +a region of memory with reduced average read latency.
> >> +
> >> +Cache pseudo-locking increases the probability that data will remain
> >> +in the cache via carefully configuring the CAT feature and controlling
> >> +application behavior. There is no guarantee that data is placed in
> >> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> >> +“locked” data from cache. Power management C-states may shrink or
> >> +power off cache. It is thus recommended to limit the processor maximum
> >> +C-state, for example, by setting the processor.max_cstate kernel 
> >> parameter.
> >> +
> >> +It is required that an application using a pseudo-locked region runs
> >> +with affinity to the cores (or a subset of the cores) associated
> >> +with the cache on which the pseudo-locked region resides. This is
> >> +enforced by the implementation.
> > 
> > Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
> > to the right CPUs. But that's not a guarantee that the task stays there.
> 
> It is required that the user space application self sets affinity to
> cores associated with the cache. This is also highlighted in the example
> application code (later in this patch) within the comments as well as
> the example usage of sched_setaffinity(). The enforcement done in the
> kernel code is done as a check that the user space application did so,
> no the actual affinity management.

Right, but your documentation claims it's enforced. There is no enforcement
aside of the initial sanity check.

Thanks,

tglx

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Thomas,

On 2/19/2018 12:35 PM, Thomas Gleixner wrote:
> On Tue, 13 Feb 2018, Reinette Chatre wrote:
>> +Cache Pseudo-Locking
>> +
>> +CAT enables a user to specify the amount of cache space into which an
>> +application can fill. Cache pseudo-locking builds on the fact that a
>> +CPU can still read and write data pre-allocated outside its current
>> +allocated area on a cache hit. With cache pseudo-locking, data can be
>> +preloaded into a reserved portion of cache that no application can
>> +fill, and from that point on will only serve cache hits.
> 
> This lacks explanation how that preloading works.

Following this text you quote there is a brief explanation starting with
"Pseudo-locking is accomplished in two stages:" - I'll add more details
to that area.

> 
>> The cache
>> +pseudo-locked memory is made accessible to user space where an
>> +application can map it into its virtual address space and thus have
>> +a region of memory with reduced average read latency.
>> +
>> +Cache pseudo-locking increases the probability that data will remain
>> +in the cache via carefully configuring the CAT feature and controlling
>> +application behavior. There is no guarantee that data is placed in
>> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
>> +“locked” data from cache. Power management C-states may shrink or
>> +power off cache. It is thus recommended to limit the processor maximum
>> +C-state, for example, by setting the processor.max_cstate kernel parameter.
>> +
>> +It is required that an application using a pseudo-locked region runs
>> +with affinity to the cores (or a subset of the cores) associated
>> +with the cache on which the pseudo-locked region resides. This is
>> +enforced by the implementation.
> 
> Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
> to the right CPUs. But that's not a guarantee that the task stays there.

It is required that the user space application self sets affinity to
cores associated with the cache. This is also highlighted in the example
application code (later in this patch) within the comments as well as
the example usage of sched_setaffinity(). The enforcement done in the
kernel code is done as a check that the user space application did so,
no the actual affinity management.

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Reinette Chatre]

Hi Thomas,

On 2/19/2018 12:35 PM, Thomas Gleixner wrote:
> On Tue, 13 Feb 2018, Reinette Chatre wrote:
>> +Cache Pseudo-Locking
>> +
>> +CAT enables a user to specify the amount of cache space into which an
>> +application can fill. Cache pseudo-locking builds on the fact that a
>> +CPU can still read and write data pre-allocated outside its current
>> +allocated area on a cache hit. With cache pseudo-locking, data can be
>> +preloaded into a reserved portion of cache that no application can
>> +fill, and from that point on will only serve cache hits.
> 
> This lacks explanation how that preloading works.

Following this text you quote there is a brief explanation starting with
"Pseudo-locking is accomplished in two stages:" - I'll add more details
to that area.

> 
>> The cache
>> +pseudo-locked memory is made accessible to user space where an
>> +application can map it into its virtual address space and thus have
>> +a region of memory with reduced average read latency.
>> +
>> +Cache pseudo-locking increases the probability that data will remain
>> +in the cache via carefully configuring the CAT feature and controlling
>> +application behavior. There is no guarantee that data is placed in
>> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
>> +“locked” data from cache. Power management C-states may shrink or
>> +power off cache. It is thus recommended to limit the processor maximum
>> +C-state, for example, by setting the processor.max_cstate kernel parameter.
>> +
>> +It is required that an application using a pseudo-locked region runs
>> +with affinity to the cores (or a subset of the cores) associated
>> +with the cache on which the pseudo-locked region resides. This is
>> +enforced by the implementation.
> 
> Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
> to the right CPUs. But that's not a guarantee that the task stays there.

It is required that the user space application self sets affinity to
cores associated with the cache. This is also highlighted in the example
application code (later in this patch) within the comments as well as
the example usage of sched_setaffinity(). The enforcement done in the
kernel code is done as a check that the user space application did so,
no the actual affinity management.

Reinette

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Randy Dunlap]

On 02/13/18 07:46, Reinette Chatre wrote:
> Add description of Cache Pseudo-Locking feature, its interface,
> as well as an example of its usage.
> 
> Signed-off-by: Reinette Chatre 
> ---
>  Documentation/x86/intel_rdt_ui.txt | 229 
> -
>  1 file changed, 228 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/x86/intel_rdt_ui.txt 
> b/Documentation/x86/intel_rdt_ui.txt
> index 756fd76b78a6..bb3d6fe0a3e4 100644
> --- a/Documentation/x86/intel_rdt_ui.txt
> +++ b/Documentation/x86/intel_rdt_ui.txt

> @@ -329,6 +332,149 @@ L3CODE:0=f;1=f;2=f;3=f
>  L3DATA:0=f;1=f;2=3c0;3=f
>  L3CODE:0=f;1=f;2=f;3=f
>  
> +Cache Pseudo-Locking
> +
> +CAT enables a user to specify the amount of cache space into which an

 space that an

> +application can fill. Cache pseudo-locking builds on the fact that a
> +CPU can still read and write data pre-allocated outside its current
> +allocated area on a cache hit. With cache pseudo-locking, data can be
> +preloaded into a reserved portion of cache that no application can
> +fill, and from that point on will only serve cache hits. The cache
> +pseudo-locked memory is made accessible to user space where an
> +application can map it into its virtual address space and thus have
> +a region of memory with reduced average read latency.
> +
> +Cache pseudo-locking increases the probability that data will remain
> +in the cache via carefully configuring the CAT feature and controlling
> +application behavior. There is no guarantee that data is placed in
> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> +“locked” data from cache. Power management C-states may shrink or
> +power off cache. It is thus recommended to limit the processor maximum
> +C-state, for example, by setting the processor.max_cstate kernel parameter.
> +
> +It is required that an application using a pseudo-locked region runs
> +with affinity to the cores (or a subset of the cores) associated
> +with the cache on which the pseudo-locked region resides. This is
> +enforced by the implementation.
> +
> +Pseudo-locking is accomplished in two stages:
> +1) During the first stage the system administrator allocates a portion
> +   of cache that should be dedicated to pseudo-locking. At this time an
> +   equivalent portion of memory is allocated, loaded into allocated
> +   cache portion, and exposed as a character device.
> +2) During the second stage a user-space application maps (mmap()) the
> +   pseudo-locked memory into its address space.
> +
> +Cache Pseudo-Locking Interface
> +--
> +Platforms supporting cache pseudo-locking will expose a new
> +"/sys/fs/restrl/pseudo_lock" directory after successful mount of the
> +resctrl filesystem. Initially this directory will contain a single file,
> +"avail" that contains the schemata, one line per resource, of cache region
> +available for pseudo-locking.

uh, sysfs is supposed to be one value per file.

> +A pseudo-locked region is created by creating a new directory within
> +/sys/fs/resctrl/pseudo_lock. On success two new files will appear in
> +the directory:
> +
> +"schemata":
> + Shows the schemata representing the pseudo-locked cache region.
> + User writes schemata of requested locked area to file.

use complete sentences, please. E.g.:

The user writes the schemata of the requested locked area to the file.

> + Only one id of single resource accepted - can only lock from

of a single resource is accepted -

> + single cache instance. Writing of schemata to this file will
> + return success on successful pseudo-locked region setup.
> +"size":
> + After successful pseudo-locked region setup this read-only file
> + will contain the size in bytes of pseudo-locked region.


-- 
~Randy

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Randy Dunlap]

On 02/13/18 07:46, Reinette Chatre wrote:
> Add description of Cache Pseudo-Locking feature, its interface,
> as well as an example of its usage.
> 
> Signed-off-by: Reinette Chatre 
> ---
>  Documentation/x86/intel_rdt_ui.txt | 229 
> -
>  1 file changed, 228 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/x86/intel_rdt_ui.txt 
> b/Documentation/x86/intel_rdt_ui.txt
> index 756fd76b78a6..bb3d6fe0a3e4 100644
> --- a/Documentation/x86/intel_rdt_ui.txt
> +++ b/Documentation/x86/intel_rdt_ui.txt

> @@ -329,6 +332,149 @@ L3CODE:0=f;1=f;2=f;3=f
>  L3DATA:0=f;1=f;2=3c0;3=f
>  L3CODE:0=f;1=f;2=f;3=f
>  
> +Cache Pseudo-Locking
> +
> +CAT enables a user to specify the amount of cache space into which an

 space that an

> +application can fill. Cache pseudo-locking builds on the fact that a
> +CPU can still read and write data pre-allocated outside its current
> +allocated area on a cache hit. With cache pseudo-locking, data can be
> +preloaded into a reserved portion of cache that no application can
> +fill, and from that point on will only serve cache hits. The cache
> +pseudo-locked memory is made accessible to user space where an
> +application can map it into its virtual address space and thus have
> +a region of memory with reduced average read latency.
> +
> +Cache pseudo-locking increases the probability that data will remain
> +in the cache via carefully configuring the CAT feature and controlling
> +application behavior. There is no guarantee that data is placed in
> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> +“locked” data from cache. Power management C-states may shrink or
> +power off cache. It is thus recommended to limit the processor maximum
> +C-state, for example, by setting the processor.max_cstate kernel parameter.
> +
> +It is required that an application using a pseudo-locked region runs
> +with affinity to the cores (or a subset of the cores) associated
> +with the cache on which the pseudo-locked region resides. This is
> +enforced by the implementation.
> +
> +Pseudo-locking is accomplished in two stages:
> +1) During the first stage the system administrator allocates a portion
> +   of cache that should be dedicated to pseudo-locking. At this time an
> +   equivalent portion of memory is allocated, loaded into allocated
> +   cache portion, and exposed as a character device.
> +2) During the second stage a user-space application maps (mmap()) the
> +   pseudo-locked memory into its address space.
> +
> +Cache Pseudo-Locking Interface
> +--
> +Platforms supporting cache pseudo-locking will expose a new
> +"/sys/fs/restrl/pseudo_lock" directory after successful mount of the
> +resctrl filesystem. Initially this directory will contain a single file,
> +"avail" that contains the schemata, one line per resource, of cache region
> +available for pseudo-locking.

uh, sysfs is supposed to be one value per file.

> +A pseudo-locked region is created by creating a new directory within
> +/sys/fs/resctrl/pseudo_lock. On success two new files will appear in
> +the directory:
> +
> +"schemata":
> + Shows the schemata representing the pseudo-locked cache region.
> + User writes schemata of requested locked area to file.

use complete sentences, please. E.g.:

The user writes the schemata of the requested locked area to the file.

> + Only one id of single resource accepted - can only lock from

of a single resource is accepted -

> + single cache instance. Writing of schemata to this file will
> + return success on successful pseudo-locked region setup.
> +"size":
> + After successful pseudo-locked region setup this read-only file
> + will contain the size in bytes of pseudo-locked region.


-- 
~Randy

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Thomas Gleixner]

On Tue, 13 Feb 2018, Reinette Chatre wrote:
> +Cache Pseudo-Locking
> +
> +CAT enables a user to specify the amount of cache space into which an
> +application can fill. Cache pseudo-locking builds on the fact that a
> +CPU can still read and write data pre-allocated outside its current
> +allocated area on a cache hit. With cache pseudo-locking, data can be
> +preloaded into a reserved portion of cache that no application can
> +fill, and from that point on will only serve cache hits.

This lacks explanation how that preloading works.

> The cache
> +pseudo-locked memory is made accessible to user space where an
> +application can map it into its virtual address space and thus have
> +a region of memory with reduced average read latency.
> +
> +Cache pseudo-locking increases the probability that data will remain
> +in the cache via carefully configuring the CAT feature and controlling
> +application behavior. There is no guarantee that data is placed in
> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> +“locked” data from cache. Power management C-states may shrink or
> +power off cache. It is thus recommended to limit the processor maximum
> +C-state, for example, by setting the processor.max_cstate kernel parameter.
> +
> +It is required that an application using a pseudo-locked region runs
> +with affinity to the cores (or a subset of the cores) associated
> +with the cache on which the pseudo-locked region resides. This is
> +enforced by the implementation.

Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
to the right CPUs. But that's not a guarantee that the task stays there.

Thanks,

tglx

Re: [RFC PATCH V2 01/22] x86/intel_rdt: Documentation for Cache Pseudo-Locking

[Thomas Gleixner]

On Tue, 13 Feb 2018, Reinette Chatre wrote:
> +Cache Pseudo-Locking
> +
> +CAT enables a user to specify the amount of cache space into which an
> +application can fill. Cache pseudo-locking builds on the fact that a
> +CPU can still read and write data pre-allocated outside its current
> +allocated area on a cache hit. With cache pseudo-locking, data can be
> +preloaded into a reserved portion of cache that no application can
> +fill, and from that point on will only serve cache hits.

This lacks explanation how that preloading works.

> The cache
> +pseudo-locked memory is made accessible to user space where an
> +application can map it into its virtual address space and thus have
> +a region of memory with reduced average read latency.
> +
> +Cache pseudo-locking increases the probability that data will remain
> +in the cache via carefully configuring the CAT feature and controlling
> +application behavior. There is no guarantee that data is placed in
> +cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
> +“locked” data from cache. Power management C-states may shrink or
> +power off cache. It is thus recommended to limit the processor maximum
> +C-state, for example, by setting the processor.max_cstate kernel parameter.
> +
> +It is required that an application using a pseudo-locked region runs
> +with affinity to the cores (or a subset of the cores) associated
> +with the cache on which the pseudo-locked region resides. This is
> +enforced by the implementation.

Well, you only enforce in pseudo_lock_dev_mmap() that the caller is affine
to the right CPUs. But that's not a guarantee that the task stays there.

Thanks,

tglx