Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 08:25:28PM -0200, Marcelo Tosatti wrote: > On Wed, Nov 30, 2016 at 02:05:31PM -0800, Fenghua Yu wrote: > > On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > > > > > There is a locking problem between different applications > > > reading/writing to resctrlfs directory at the same time (read the patch > > > below for details). > > > > > > Suggest a standard locking scheme for applications to use. > > > > > > Signed-off-by: Marcelo Tosatti> > > > > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 > > > 13:40:33.080233101 -0200 > > > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > > > -0200 > > > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > > > kernel and the tasks running there get 50% of the cache. > > > > > > # echo C0 > p0/cpus > > > + > > > +4) Locking between applications > > > + > > > +The allocation of an exclusive reservation > > > +of L3 cache involves: > > > + > > > +1. read list of cbmmasks for each directory > > > +2. find a contiguous set of bits in the global CBM bitmask > > > + that is clear in any of the directory cbmmasks > > > +3. create a new directory > > > +4. set the bits found in step 2 to the new directory "schemata" > > > + file > > > > This is one example of why locking is needed. There are other scenarios > > that need the locking as well. For example, two applications scan each > > directory to find an empty/less loaded "tasks". Both of them find that > > directory p1 has empty "tasks" and write their own thread ids into the > > "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or > > workloads. A locking can solve this race scenario too. > > > > As a user interface document, maybe we need a generic explanation why > > locking plus the example. > > Well, agreed there are other races, but in this particular example > taking the file lock does not solve the "tasks" race: the contents of > the tasks file can change in face of fork. The "tasks" example is only for resolving the race when allocating two tasks to an empty rdtgroup. Once a task is allocated to a "tasks", the task's forked children will automatically stay with the task unless they are moved. Without the locking, task A and task B are allocated to the same directory because both of them thought "tasks" in the directory is empty. Then all forked children of both A and B will populate the directory and cause crowded cache. Sure the contents of the "tasks" can change in face of fork. But the race of the allocating two groups of tasks can cause wrong decision to allocate them at the beginning. > > So i've added your suggestion but can't use this example, if you have > another one you'd like to see added, please let me know... Replying with > V2. Thanks. -Fenghua
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 08:25:28PM -0200, Marcelo Tosatti wrote: > On Wed, Nov 30, 2016 at 02:05:31PM -0800, Fenghua Yu wrote: > > On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > > > > > There is a locking problem between different applications > > > reading/writing to resctrlfs directory at the same time (read the patch > > > below for details). > > > > > > Suggest a standard locking scheme for applications to use. > > > > > > Signed-off-by: Marcelo Tosatti > > > > > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 > > > 13:40:33.080233101 -0200 > > > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > > > -0200 > > > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > > > kernel and the tasks running there get 50% of the cache. > > > > > > # echo C0 > p0/cpus > > > + > > > +4) Locking between applications > > > + > > > +The allocation of an exclusive reservation > > > +of L3 cache involves: > > > + > > > +1. read list of cbmmasks for each directory > > > +2. find a contiguous set of bits in the global CBM bitmask > > > + that is clear in any of the directory cbmmasks > > > +3. create a new directory > > > +4. set the bits found in step 2 to the new directory "schemata" > > > + file > > > > This is one example of why locking is needed. There are other scenarios > > that need the locking as well. For example, two applications scan each > > directory to find an empty/less loaded "tasks". Both of them find that > > directory p1 has empty "tasks" and write their own thread ids into the > > "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or > > workloads. A locking can solve this race scenario too. > > > > As a user interface document, maybe we need a generic explanation why > > locking plus the example. > > Well, agreed there are other races, but in this particular example > taking the file lock does not solve the "tasks" race: the contents of > the tasks file can change in face of fork. The "tasks" example is only for resolving the race when allocating two tasks to an empty rdtgroup. Once a task is allocated to a "tasks", the task's forked children will automatically stay with the task unless they are moved. Without the locking, task A and task B are allocated to the same directory because both of them thought "tasks" in the directory is empty. Then all forked children of both A and B will populate the directory and cause crowded cache. Sure the contents of the "tasks" can change in face of fork. But the race of the allocating two groups of tasks can cause wrong decision to allocate them at the beginning. > > So i've added your suggestion but can't use this example, if you have > another one you'd like to see added, please let me know... Replying with > V2. Thanks. -Fenghua
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 02:05:31PM -0800, Fenghua Yu wrote: > On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > > > There is a locking problem between different applications > > reading/writing to resctrlfs directory at the same time (read the patch > > below for details). > > > > Suggest a standard locking scheme for applications to use. > > > > Signed-off-by: Marcelo Tosatti> > > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > > -0200 > > +++ Documentation/x86/intel_rdt_ui.txt 2016-11-30 13:45:01.253703259 > > -0200 > > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > > kernel and the tasks running there get 50% of the cache. > > > > # echo C0 > p0/cpus > > + > > +4) Locking between applications > > + > > +The allocation of an exclusive reservation > > +of L3 cache involves: > > + > > +1. read list of cbmmasks for each directory > > +2. find a contiguous set of bits in the global CBM bitmask > > + that is clear in any of the directory cbmmasks > > +3. create a new directory > > +4. set the bits found in step 2 to the new directory "schemata" > > + file > > This is one example of why locking is needed. There are other scenarios > that need the locking as well. For example, two applications scan each > directory to find an empty/less loaded "tasks". Both of them find that > directory p1 has empty "tasks" and write their own thread ids into the > "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or > workloads. A locking can solve this race scenario too. > > As a user interface document, maybe we need a generic explanation why > locking plus the example. Well, agreed there are other races, but in this particular example taking the file lock does not solve the "tasks" race: the contents of the tasks file can change in face of fork. So i've added your suggestion but can't use this example, if you have another one you'd like to see added, please let me know... Replying with V2.
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 02:05:31PM -0800, Fenghua Yu wrote: > On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > > > There is a locking problem between different applications > > reading/writing to resctrlfs directory at the same time (read the patch > > below for details). > > > > Suggest a standard locking scheme for applications to use. > > > > Signed-off-by: Marcelo Tosatti > > > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > > -0200 > > +++ Documentation/x86/intel_rdt_ui.txt 2016-11-30 13:45:01.253703259 > > -0200 > > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > > kernel and the tasks running there get 50% of the cache. > > > > # echo C0 > p0/cpus > > + > > +4) Locking between applications > > + > > +The allocation of an exclusive reservation > > +of L3 cache involves: > > + > > +1. read list of cbmmasks for each directory > > +2. find a contiguous set of bits in the global CBM bitmask > > + that is clear in any of the directory cbmmasks > > +3. create a new directory > > +4. set the bits found in step 2 to the new directory "schemata" > > + file > > This is one example of why locking is needed. There are other scenarios > that need the locking as well. For example, two applications scan each > directory to find an empty/less loaded "tasks". Both of them find that > directory p1 has empty "tasks" and write their own thread ids into the > "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or > workloads. A locking can solve this race scenario too. > > As a user interface document, maybe we need a generic explanation why > locking plus the example. Well, agreed there are other races, but in this particular example taking the file lock does not solve the "tasks" race: the contents of the tasks file can change in face of fork. So i've added your suggestion but can't use this example, if you have another one you'd like to see added, please let me know... Replying with V2.
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > There is a locking problem between different applications > reading/writing to resctrlfs directory at the same time (read the patch > below for details). > > Suggest a standard locking scheme for applications to use. > > Signed-off-by: Marcelo Tosatti> > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > -0200 > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > -0200 > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > kernel and the tasks running there get 50% of the cache. > > # echo C0 > p0/cpus > + > +4) Locking between applications > + > +The allocation of an exclusive reservation > +of L3 cache involves: > + > +1. read list of cbmmasks for each directory > +2. find a contiguous set of bits in the global CBM bitmask > + that is clear in any of the directory cbmmasks > +3. create a new directory > +4. set the bits found in step 2 to the new directory "schemata" > + file This is one example of why locking is needed. There are other scenarios that need the locking as well. For example, two applications scan each directory to find an empty/less loaded "tasks". Both of them find that directory p1 has empty "tasks" and write their own thread ids into the "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or workloads. A locking can solve this race scenario too. As a user interface document, maybe we need a generic explanation why locking plus the example. > + > +If two applications attempt to allocate space race with each other > +(if two processes execute the steps above in a interlocked fashion), > +they can end up using the same bits of CBMMASK, which renders the > +reservations non-exclusive but shared. > + > +To coordinate creation of reservations on resctrl and avoid the problem > +above, the following locking procedure is recommended: > + > +A) open /var/lock/resctrl/fs.lock with O_CREAT|O_EXCL. > +B) if success, write pid of program accessing the directory > + structure to this file. > +C) read/write the directory structure. > +D) remove file. > +
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, Nov 30, 2016 at 01:48:10PM -0200, Marcelo Tosatti wrote: > > There is a locking problem between different applications > reading/writing to resctrlfs directory at the same time (read the patch > below for details). > > Suggest a standard locking scheme for applications to use. > > Signed-off-by: Marcelo Tosatti > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > -0200 > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > -0200 > @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new > kernel and the tasks running there get 50% of the cache. > > # echo C0 > p0/cpus > + > +4) Locking between applications > + > +The allocation of an exclusive reservation > +of L3 cache involves: > + > +1. read list of cbmmasks for each directory > +2. find a contiguous set of bits in the global CBM bitmask > + that is clear in any of the directory cbmmasks > +3. create a new directory > +4. set the bits found in step 2 to the new directory "schemata" > + file This is one example of why locking is needed. There are other scenarios that need the locking as well. For example, two applications scan each directory to find an empty/less loaded "tasks". Both of them find that directory p1 has empty "tasks" and write their own thread ids into the "tasks" in p1. Turns out the "tasks" in p1 will have crowded threads or workloads. A locking can solve this race scenario too. As a user interface document, maybe we need a generic explanation why locking plus the example. > + > +If two applications attempt to allocate space race with each other > +(if two processes execute the steps above in a interlocked fashion), > +they can end up using the same bits of CBMMASK, which renders the > +reservations non-exclusive but shared. > + > +To coordinate creation of reservations on resctrl and avoid the problem > +above, the following locking procedure is recommended: > + > +A) open /var/lock/resctrl/fs.lock with O_CREAT|O_EXCL. > +B) if success, write pid of program accessing the directory > + structure to this file. > +C) read/write the directory structure. > +D) remove file. > +
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, 30 Nov 2016, Marcelo Tosatti wrote: > > There is a locking problem between different applications > reading/writing to resctrlfs directory at the same time (read the patch > below for details). > > Suggest a standard locking scheme for applications to use. > > Signed-off-by: Marcelo Tosatti> > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > -0200 > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > -0200 I can't remember that we changed the -p1 patch format to -p0 :(
Re: [PATCH] intelrdt: resctrl: recommend locking for resctrlfs
On Wed, 30 Nov 2016, Marcelo Tosatti wrote: > > There is a locking problem between different applications > reading/writing to resctrlfs directory at the same time (read the patch > below for details). > > Suggest a standard locking scheme for applications to use. > > Signed-off-by: Marcelo Tosatti > > --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 > -0200 > +++ Documentation/x86/intel_rdt_ui.txt2016-11-30 13:45:01.253703259 > -0200 I can't remember that we changed the -p1 patch format to -p0 :(
[PATCH] intelrdt: resctrl: recommend locking for resctrlfs
There is a locking problem between different applications reading/writing to resctrlfs directory at the same time (read the patch below for details). Suggest a standard locking scheme for applications to use. Signed-off-by: Marcelo Tosatti--- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 -0200 +++ Documentation/x86/intel_rdt_ui.txt 2016-11-30 13:45:01.253703259 -0200 @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new kernel and the tasks running there get 50% of the cache. # echo C0 > p0/cpus + +4) Locking between applications + +The allocation of an exclusive reservation +of L3 cache involves: + +1. read list of cbmmasks for each directory +2. find a contiguous set of bits in the global CBM bitmask + that is clear in any of the directory cbmmasks +3. create a new directory +4. set the bits found in step 2 to the new directory "schemata" + file + +If two applications attempt to allocate space race with each other +(if two processes execute the steps above in a interlocked fashion), +they can end up using the same bits of CBMMASK, which renders the +reservations non-exclusive but shared. + +To coordinate creation of reservations on resctrl and avoid the problem +above, the following locking procedure is recommended: + +A) open /var/lock/resctrl/fs.lock with O_CREAT|O_EXCL. +B) if success, write pid of program accessing the directory + structure to this file. +C) read/write the directory structure. +D) remove file. +
[PATCH] intelrdt: resctrl: recommend locking for resctrlfs
There is a locking problem between different applications reading/writing to resctrlfs directory at the same time (read the patch below for details). Suggest a standard locking scheme for applications to use. Signed-off-by: Marcelo Tosatti --- Documentation/x86/intel_rdt_ui.txt.orig 2016-11-30 13:40:33.080233101 -0200 +++ Documentation/x86/intel_rdt_ui.txt 2016-11-30 13:45:01.253703259 -0200 @@ -212,3 +212,30 @@ Finally we move core 4-7 over to the new kernel and the tasks running there get 50% of the cache. # echo C0 > p0/cpus + +4) Locking between applications + +The allocation of an exclusive reservation +of L3 cache involves: + +1. read list of cbmmasks for each directory +2. find a contiguous set of bits in the global CBM bitmask + that is clear in any of the directory cbmmasks +3. create a new directory +4. set the bits found in step 2 to the new directory "schemata" + file + +If two applications attempt to allocate space race with each other +(if two processes execute the steps above in a interlocked fashion), +they can end up using the same bits of CBMMASK, which renders the +reservations non-exclusive but shared. + +To coordinate creation of reservations on resctrl and avoid the problem +above, the following locking procedure is recommended: + +A) open /var/lock/resctrl/fs.lock with O_CREAT|O_EXCL. +B) if success, write pid of program accessing the directory + structure to this file. +C) read/write the directory structure. +D) remove file. +