Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-15 Thread John Ferlan


On 10/15/18 11:25 AM, Wang, Huaqiang wrote:
> 
> On 10/13/2018 6:29 AM, John Ferlan wrote:
>>
>> On 10/12/18 3:10 AM, Wang, Huaqiang wrote:
 -Original Message-

[...]

 IOW: What is cache_occupancy measuring? Each cache? The entire
 thing? If
 there's no cache elements, then what?
>>> cache_occupancy is measuring based on cache bank. For Intel 2 socket
>>> xeon CPU,
>>> it is considered as two cache banks, one cache bank per socket. The
>>> typical
>>> output for each monitor of this case is:
>>>
>>>   cpu.cache.0.name=vcpus_1
>>>   cpu.cache.0.vcpus=1
>>>   cpu.cache.0.bank.count=2  <--- 2 cache banks
>>>   cpu.cache.0.bank.0.id=0   <--- bank.id.0 cache_occypancy
>>>   cpu.cache.0.bank.0.bytes=9371648    _|
>>>   cpu.cache.0.bank.1.id=1   <--- bank.id.1 cache_occypancy
>>>   cpu.cache.0.bank.1.bytes=1081344    _|
>>>
>>> If you want to know the total cache occupancy for VM vcpu threads of
>>> this
>>> monitor, you need to add them up.
>>>
>> So if you have:
>>
>>     
>>
>> what do you get in output for cache_occupancy? 0 + 1?
> 
> Yes. Output is sum of two vcpus.
> 
> for cache bank 0
>     vcpus_0-1.bank.0.bytes =   vcpus_0.bank.0.bytes + vcpus_1.bank.0.bytes
> for cache bank 1
>     vcpus_0-1.bank.1.bytes =   vcpus_0.bank.1.bytes + vcpus_1.bank.1.bytes
> 
>>
 I honestly think this just needs to be simplified as much as possible.
> 
> "I honestly think this just needs to be simplified as much as possible."
> 
> I reconsidered your comment ( in above line), do you mean the XML
> configuration for 'monitor' need to be simplified also?
> 

This is/was a comment regarding default stuff which you are removing.

> What I think is, even after the removal of 'default monitor' and
> 'default allocation' concepts, the XML
> configuration for monitors (with type 'all', 'm-to-n', 'one to one')
> still need such kind of arrangement.
> 
> Take an example, a VM has 4 vcpus, vcpu 0 and 1 run cache sensitive
> workload, and wants to hold
> private L3 caches, and there is no specific requirement for left vcpus
> but still need a monitoring on
> the cache usage.
> 
> Then we could create an cache allocation for vcpu 0 and 1 as well as a
> monitor on getting the
> actual cache that these two vcpus used. For vcpu 2 and 3, create a
> monitor for it.
> 
> The XML configurations are: (no change in general rules comparing to my
> previous examples)
> 
>     
>   
>   
>   
>     
>     
>   
>     
>  
> Any suggestion from you is welcome.
> 


I'm not sure what the question is and I'm not sure it matters at this
point. If you only create an allocation for any  or
 entry, then that's all that'll be reported which is what I
was trying to point out. Its not that something else may or may not
exist, it's what gets reported and can be queried via the XML.

> 
 When you monitor specific vcpus within a cachetune, then you get what?
>>> In this case, the monitor you created only monitors the specific vcpus
>>> you added for monitor.
>>>
>>> Following two configurations satisfy your scenario, and the only monitor
>>> will detect the cache usage of thread of vcpu 2.
>>>
>>>   
>>>     
>>>     
>>>     
>>>   
>>>
>>>   
>>>     
>>>   
>>>
>> Perhaps my question was mistyped or misinterpreted. In the above top
>> example, if we have , then do the values in
>>  have any impact on the calculation as opposed to if they weren't
>> there?
> 
> I perhaps still not understand you well ...
> There will have significant influence for the output of monitor if
>  entry
> exist and if vcpu2-4 demands much more caches that allocation can offer;
> If the
> cache that the allocation offers is much bigger than vcpu2-4 actually
> used, the
> influence will be tiny.
> 
> But in another case, that, if there is no 'cache' entries, just showing
> in the second
> example, it still influenced by the cache that the 'allocation' offers.
> Its difference
> with the first example is: the top example is using the cache resources
> allocated
> by the allocation of itself, while the second example uses the
> allocation of resources
> defined in /sys/fs/resctrl/schemata, and this cache is shared by
> multiple system tasks.
> 

The question was related to how  is defined and trying to
further describe my feeling that default was necessary.

>>
>>>
 If the cachetune has no specific cache entries, you get what?
>>> If no cache entry in cachetune, it will also get vcpu threads' cache
>>> utilization information based on cache bank.
>>> No cache entry specified for the cachetune, means it will use the cache
>>> allocating policy of default cache allocation, which is file
>>> /sys/fs/resctrl/schemata.
>>>
>>> If valid cache entries are provided in cachetune, then an allocation
>>> will
>>> be created for the threads of vcpu listed in  'vcpus'
>>> attribute. Supposing the allocation is the directory /sys/fs/resctrl/p0,
>>> then the 

Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-15 Thread Wang, Huaqiang


On 10/13/2018 6:29 AM, John Ferlan wrote:


On 10/12/18 3:10 AM, Wang, Huaqiang wrote:

-Original Message-
From: John Ferlan [mailto:jfer...@redhat.com]
Sent: Thursday, October 11, 2018 4:58 AM
To: Wang, Huaqiang ; libvir-list@redhat.com
Cc: Feng, Shaohe ; Niu, Bing ;
Ding, Jian-feng ; Zang, Rui 
Subject: Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration



On 10/9/18 6:30 AM, Wang Huaqiang wrote:

Introducing  element under  to represent a cache
monitor.

Supports two kind of monitors, which are, monitor under default
allocation or monitor under particular allocation.

I still don't see what the big difference is with singling out default.
Does it really matter - what advantage is there. Having a monitor for 'n' vcpus 
is
a choice.



Monitor supervises the cache or memory bandwidth usage for

At this point memory bandwidth is not in play...

Yes. Will remove the memory BW related words.


interested vcpu thread set, if the vcpu thread set is belong to some
resctrl allocation, then the monitor will be created under this
allocation, that is, creating a resctrl monitoring group directory
under the directory of '@alloc->path/mon_group'. Otherwise, the
monitor will be created under default allocation.

This is becoming increasing difficult to describe/decipher - makes me wonder
who would really use it.

Monitor in default allocation could be commonly used when we want to
observe the resource usage for VM vcpu threads that are not specified
with any dedicated resource limitation through CAT or MBA technology.

And another common case is it will be used if CAT/MBA is not supported
but CMT/MBM is enabled in libvirt.

The reason I have introduced the monitor in previous patch and further
introducing the monitor in default allocation in this patch is the monitor
n two different allocations has different behavior.

Let's focus on CAT and CMT technology in my below lines, since MBA and
MBM are very similar cases to them.

As we know, before the CAT technology was introduced, any process in Linux
is sharing the L3 cache along with all other active processes. After
CAT is enabled in libvirt, it has the capability to apply cache isolation
and assign dedicated amount of cache to some key VM vcpu threads.

If we want to observe the real L3 cache usage information, then we
need the help of monitor.

== Monitor usage case 1: monitor in default allocation ==

If you want to get the cache utilization data before applying any cache
isolation to the VM vcpu threads, you need to create a monitor in the
default allocation, because you haven't create any cache allocation.

== Monitor usage case 2: monitor in non-default allocation ==

If you have created a cache allocation for specific VM vcpu treads
and not sure how many cache-lines these VM vcpu threads are really used,
you need to create a monitor under the this allocation to get real cache
usage information.

If you find your VM vcpu threads only used a small part of cache that
have allocated, you might think about to reduce its allocation.

== Usage for default monitor and non-default monitor ==

Since we have introduced the 'default monitor' and 'non-default monitor'
concepts in previous patches, and now, you can monitor the cache usage
for all VM vcpu threads that added to this allocation, and also you has
the capability to monitor a subset of vcpu list of this allocation.

Without 'default monitor', it is impossible to get the cache usage for
all vcpu threads in the allocation and at the same time get the cache
usage for some highly interested vcpu threads of allocation.

"Default monitor" is in name only. It's just "a" monitor for all the
threads in cachetune (or memorytune eventually); whereas, a "non default
monitor" is "a" monitor for specific vcpus within the range in cachetune.


Agree.
As stated in reply of patch10, 'default monitor' and will be removed, 
and related

code will be cleaned.



Thus your description is that a monitor can be a monitor all vcpus or a
monitor for some subset of of all vcpus. A  describes which
vcpus of a cachetune (or memorytune) can be monitored - there are 3
options - "all", "m-to-n", or "one-to-one".


Thanks for suggestion. Then no things will be mentioned as 'default', 
code and document

will be cleaned accordingly.



What they relate to in the filesystem is the magic of the code of paths
to the data. What data structures are called or how this is described in
docs just doesn't seem to need to make a delineation.


Agree.




== Monitor usage case 3: allocating dedicated cache and monitoring its
usage of one VM, and getting its influence over another VM==

Think about the scenario that there are two VMs, it is a known information
that one VM is very cache sensitive and don't want to share cache with
other workloads, and for another VM, we have no knowledge about cache
requirement, but it is required to monitor the cach

Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-12 Thread John Ferlan



On 10/12/18 3:10 AM, Wang, Huaqiang wrote:
> 
>> -Original Message-
>> From: John Ferlan [mailto:jfer...@redhat.com]
>> Sent: Thursday, October 11, 2018 4:58 AM
>> To: Wang, Huaqiang ; libvir-list@redhat.com
>> Cc: Feng, Shaohe ; Niu, Bing ;
>> Ding, Jian-feng ; Zang, Rui 
>> Subject: Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor 
>> configuration
>>
>>
>>
>> On 10/9/18 6:30 AM, Wang Huaqiang wrote:
>>> Introducing  element under  to represent a cache
>>> monitor.
>>>
>>> Supports two kind of monitors, which are, monitor under default
>>> allocation or monitor under particular allocation.
>>
>> I still don't see what the big difference is with singling out default.
>> Does it really matter - what advantage is there. Having a monitor for 'n' 
>> vcpus is
>> a choice.
>>
>>
>>>
>>> Monitor supervises the cache or memory bandwidth usage for
>>
>> At this point memory bandwidth is not in play...
> 
> Yes. Will remove the memory BW related words.
> 
>>
>>> interested vcpu thread set, if the vcpu thread set is belong to some
>>> resctrl allocation, then the monitor will be created under this
>>> allocation, that is, creating a resctrl monitoring group directory
>>> under the directory of '@alloc->path/mon_group'. Otherwise, the
>>> monitor will be created under default allocation.
>>
>> This is becoming increasing difficult to describe/decipher - makes me wonder
>> who would really use it.
> 
> Monitor in default allocation could be commonly used when we want to
> observe the resource usage for VM vcpu threads that are not specified
> with any dedicated resource limitation through CAT or MBA technology.
> 
> And another common case is it will be used if CAT/MBA is not supported
> but CMT/MBM is enabled in libvirt.
> 
> The reason I have introduced the monitor in previous patch and further
> introducing the monitor in default allocation in this patch is the monitor
> n two different allocations has different behavior.
> 
> Let's focus on CAT and CMT technology in my below lines, since MBA and
> MBM are very similar cases to them.
> 
> As we know, before the CAT technology was introduced, any process in Linux
> is sharing the L3 cache along with all other active processes. After
> CAT is enabled in libvirt, it has the capability to apply cache isolation
> and assign dedicated amount of cache to some key VM vcpu threads.
> 
> If we want to observe the real L3 cache usage information, then we
> need the help of monitor.
> 
> == Monitor usage case 1: monitor in default allocation ==
> 
> If you want to get the cache utilization data before applying any cache
> isolation to the VM vcpu threads, you need to create a monitor in the
> default allocation, because you haven't create any cache allocation.
> 
> == Monitor usage case 2: monitor in non-default allocation ==
> 
> If you have created a cache allocation for specific VM vcpu treads
> and not sure how many cache-lines these VM vcpu threads are really used,
> you need to create a monitor under the this allocation to get real cache
> usage information.
> 
> If you find your VM vcpu threads only used a small part of cache that
> have allocated, you might think about to reduce its allocation.
> 
> == Usage for default monitor and non-default monitor ==
> 
> Since we have introduced the 'default monitor' and 'non-default monitor'
> concepts in previous patches, and now, you can monitor the cache usage
> for all VM vcpu threads that added to this allocation, and also you has
> the capability to monitor a subset of vcpu list of this allocation.
> 
> Without 'default monitor', it is impossible to get the cache usage for
> all vcpu threads in the allocation and at the same time get the cache
> usage for some highly interested vcpu threads of allocation.

"Default monitor" is in name only. It's just "a" monitor for all the
threads in cachetune (or memorytune eventually); whereas, a "non default
monitor" is "a" monitor for specific vcpus within the range in cachetune.

Thus your description is that a monitor can be a monitor all vcpus or a
monitor for some subset of of all vcpus. A  describes which
vcpus of a cachetune (or memorytune) can be monitored - there are 3
options - "all", "m-to-n", or "one-to-one".

What they relate to in the filesystem is the magic of the code of paths
to the data. What data structures are called or how this is described in
docs just doesn't seem to need to make a delineation.

> 
> == Monitor usage case 3: allocating dedicated cach

Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-12 Thread Wang, Huaqiang


> -Original Message-
> From: John Ferlan [mailto:jfer...@redhat.com]
> Sent: Thursday, October 11, 2018 4:58 AM
> To: Wang, Huaqiang ; libvir-list@redhat.com
> Cc: Feng, Shaohe ; Niu, Bing ;
> Ding, Jian-feng ; Zang, Rui 
> Subject: Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration
> 
> 
> 
> On 10/9/18 6:30 AM, Wang Huaqiang wrote:
> > Introducing  element under  to represent a cache
> > monitor.
> >
> > Supports two kind of monitors, which are, monitor under default
> > allocation or monitor under particular allocation.
> 
> I still don't see what the big difference is with singling out default.
> Does it really matter - what advantage is there. Having a monitor for 'n' 
> vcpus is
> a choice.
> 
> 
> >
> > Monitor supervises the cache or memory bandwidth usage for
> 
> At this point memory bandwidth is not in play...

Yes. Will remove the memory BW related words.

> 
> > interested vcpu thread set, if the vcpu thread set is belong to some
> > resctrl allocation, then the monitor will be created under this
> > allocation, that is, creating a resctrl monitoring group directory
> > under the directory of '@alloc->path/mon_group'. Otherwise, the
> > monitor will be created under default allocation.
> 
> This is becoming increasing difficult to describe/decipher - makes me wonder
> who would really use it.

Monitor in default allocation could be commonly used when we want to
observe the resource usage for VM vcpu threads that are not specified
with any dedicated resource limitation through CAT or MBA technology.

And another common case is it will be used if CAT/MBA is not supported
but CMT/MBM is enabled in libvirt.

The reason I have introduced the monitor in previous patch and further
introducing the monitor in default allocation in this patch is the monitor
n two different allocations has different behavior.

Let's focus on CAT and CMT technology in my below lines, since MBA and
MBM are very similar cases to them.

As we know, before the CAT technology was introduced, any process in Linux
is sharing the L3 cache along with all other active processes. After
CAT is enabled in libvirt, it has the capability to apply cache isolation
and assign dedicated amount of cache to some key VM vcpu threads.

If we want to observe the real L3 cache usage information, then we
need the help of monitor.

== Monitor usage case 1: monitor in default allocation ==

If you want to get the cache utilization data before applying any cache
isolation to the VM vcpu threads, you need to create a monitor in the
default allocation, because you haven't create any cache allocation.

== Monitor usage case 2: monitor in non-default allocation ==

If you have created a cache allocation for specific VM vcpu treads
and not sure how many cache-lines these VM vcpu threads are really used,
you need to create a monitor under the this allocation to get real cache
usage information.

If you find your VM vcpu threads only used a small part of cache that
have allocated, you might think about to reduce its allocation.

== Usage for default monitor and non-default monitor ==

Since we have introduced the 'default monitor' and 'non-default monitor'
concepts in previous patches, and now, you can monitor the cache usage
for all VM vcpu threads that added to this allocation, and also you has
the capability to monitor a subset of vcpu list of this allocation.

Without 'default monitor', it is impossible to get the cache usage for
all vcpu threads in the allocation and at the same time get the cache
usage for some highly interested vcpu threads of allocation.

== Monitor usage case 3: allocating dedicated cache and monitoring its
usage of one VM, and getting its influence over another VM==

Think about the scenario that there are two VMs, it is a known information
that one VM is very cache sensitive and don't want to share cache with
other workloads, and for another VM, we have no knowledge about cache
requirement, but it is required to monitor the cache usage for the two
VMs.

With the concepts introduced until now. We need to create an allocation
and for this VM, then create a default-monitor in this allocation for
monitoring. For another VM, it is required to create a non-default
monitor under default allocation.


With introduced concepts, the allocation, the default allocation, the monitor
and the default monitor, it is possible to fulfill requirement all scenarios.

The creation of monitor does not have too much flexibility, as I stated
in my reply of v5patch0's review comments, the monitors need to follow
below rules:

1. In each  entry more than one monitors could be specified.
2. In each  entry up to one allocation could be specified.
3. The allocation is using the vcpu list specified in  attribute
'vcpus'.
4. A monitor has the same vcpu list as allocation is allowed, an

Re: [libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-10 Thread John Ferlan



On 10/9/18 6:30 AM, Wang Huaqiang wrote:
> Introducing  element under  to represent
> a cache monitor.
> 
> Supports two kind of monitors, which are, monitor under default
> allocation or monitor under particular allocation.

I still don't see what the big difference is with singling out default.
Does it really matter - what advantage is there. Having a monitor for
'n' vcpus is a choice.


> 
> Monitor supervises the cache or memory bandwidth usage for

At this point memory bandwidth is not in play...

> interested vcpu thread set, if the vcpu thread set is belong to some
> resctrl allocation, then the monitor will be created under this
> allocation, that is, creating a resctrl monitoring group directory under
> the directory of '@alloc->path/mon_group'. Otherwise, the monitor
> will be created under default allocation.

This is becoming increasing difficult to describe/decipher - makes me
wonder who would really use it.

> 
> For default allocation monitor, it will have such kind of XML layout:
> 
> 
>   
> 
> 
> For other type monitor, the XML layout will be something like:
> 
> 
>   
>   
>   
> 
> 

Since all we get is the "cache_occupancy" that would seem to me to be
most important when there is a  bank, right?  So what would the
first (or your default style) collect/display. Or does cache not really
matter.

IOW: What is cache_occupancy measuring? Each cache? The entire thing? If
there's no cache elements, then what?

I honestly think this just needs to be simplified as much as possible.
When you monitor specific vcpus within a cachetune, then you get what?
If the cachetune has no specific cache entries, you get what?  If you
monitor multiple vcpus within a cachetune then you get what? (?An
aggregation of all?).  This whole default and specific description
doesn't make sense.

> Signed-off-by: Wang Huaqiang 
> ---
>  docs/formatdomain.html.in  |  26 +++
>  docs/schemas/domaincommon.rng  |  10 +
>  src/conf/domain_conf.c | 217 
> -
>  src/conf/domain_conf.h |  11 ++
>  tests/genericxml2xmlindata/cachetune-cdp.xml   |   3 +
>  .../cachetune-colliding-monitor.xml|  30 +++
>  tests/genericxml2xmlindata/cachetune-small.xml |   7 +
>  tests/genericxml2xmltest.c |   2 +
>  8 files changed, 301 insertions(+), 5 deletions(-)
>  create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml
> 
> diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
> index b1651e3..2fd665c 100644
> --- a/docs/formatdomain.html.in
> +++ b/docs/formatdomain.html.in
> @@ -759,6 +759,12 @@
>  cachetune vcpus='0-3'
>cache id='0' level='3' type='both' size='3' unit='MiB'/
>cache id='1' level='3' type='both' size='3' unit='MiB'/
> +  monitor level='3' vcpus='1'/
> +  monitor level='3' vcpus='0-3'/
> +/cachetune
> +cachetune vcpus='4-5'
> +  monitor level='3' vcpus='4'/
> +  monitor level='3' vcpus='5'/
>  /cachetune
>  memorytune vcpus='0-3'
>node id='0' bandwidth='60'/
> @@ -978,6 +984,26 @@
>
>  
>
> +  monitor
> +  
> +The optional element monitor creates the cache
> +monitor(s) for current cache allocation and has the following
> +required attributes:
> +
> +  level
> +  
> +Host cache level the monitor belongs to.
> +  
> +  vcpus
> +  
> +vCPU list the monitor applies to. A monitor's vCPU list
> +can only be the member(s) of the vCPU list of associating
> +allocation. The default monitor has the same vCPU list as the
> +associating allocation. For non-default monitors, there
> +are no vCPU overlap permitted.
> +  
> +
> +  
>  
>
>  
> diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng
> index 5c533d6..7ce49d3 100644
> --- a/docs/schemas/domaincommon.rng
> +++ b/docs/schemas/domaincommon.rng
> @@ -981,6 +981,16 @@
>  
>
>  
> +
> +  
> +
> +  
> +
> +
> +  
> +
> +  
> +
>
>  
>  
> diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
> index 9a514a6..4f4604f 100644
> --- a/src/conf/domain_conf.c
> +++ b/src/conf/domain_conf.c
> @@ -2955,13 +2955,30 @@ virDomainLoaderDefFree(virDomainLoaderDefPtr loader)
>  
>  
>  static void
> +virDomainResctrlMonDefFree(virDomainResctrlMonDefPtr domresmon)
> +{
> +if (!domresmon)
> +return;
> +
> +virBitmapFree(domresmon->vcpus);
> +

[libvirt] [PATCHv5 13/19] conf: Add resctrl monitor configuration

2018-10-09 Thread Wang Huaqiang
Introducing  element under  to represent
a cache monitor.

Supports two kind of monitors, which are, monitor under default
allocation or monitor under particular allocation.

Monitor supervises the cache or memory bandwidth usage for
interested vcpu thread set, if the vcpu thread set is belong to some
resctrl allocation, then the monitor will be created under this
allocation, that is, creating a resctrl monitoring group directory under
the directory of '@alloc->path/mon_group'. Otherwise, the monitor
will be created under default allocation.

For default allocation monitor, it will have such kind of XML layout:


  


For other type monitor, the XML layout will be something like:


  
  
  


Signed-off-by: Wang Huaqiang 
---
 docs/formatdomain.html.in  |  26 +++
 docs/schemas/domaincommon.rng  |  10 +
 src/conf/domain_conf.c | 217 -
 src/conf/domain_conf.h |  11 ++
 tests/genericxml2xmlindata/cachetune-cdp.xml   |   3 +
 .../cachetune-colliding-monitor.xml|  30 +++
 tests/genericxml2xmlindata/cachetune-small.xml |   7 +
 tests/genericxml2xmltest.c |   2 +
 8 files changed, 301 insertions(+), 5 deletions(-)
 create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitor.xml

diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
index b1651e3..2fd665c 100644
--- a/docs/formatdomain.html.in
+++ b/docs/formatdomain.html.in
@@ -759,6 +759,12 @@
 cachetune vcpus='0-3'
   cache id='0' level='3' type='both' size='3' unit='MiB'/
   cache id='1' level='3' type='both' size='3' unit='MiB'/
+  monitor level='3' vcpus='1'/
+  monitor level='3' vcpus='0-3'/
+/cachetune
+cachetune vcpus='4-5'
+  monitor level='3' vcpus='4'/
+  monitor level='3' vcpus='5'/
 /cachetune
 memorytune vcpus='0-3'
   node id='0' bandwidth='60'/
@@ -978,6 +984,26 @@
   
 
   
+  monitor
+  
+The optional element monitor creates the cache
+monitor(s) for current cache allocation and has the following
+required attributes:
+
+  level
+  
+Host cache level the monitor belongs to.
+  
+  vcpus
+  
+vCPU list the monitor applies to. A monitor's vCPU list
+can only be the member(s) of the vCPU list of associating
+allocation. The default monitor has the same vCPU list as the
+associating allocation. For non-default monitors, there
+are no vCPU overlap permitted.
+  
+
+  
 
   
 
diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng
index 5c533d6..7ce49d3 100644
--- a/docs/schemas/domaincommon.rng
+++ b/docs/schemas/domaincommon.rng
@@ -981,6 +981,16 @@
 
   
 
+
+  
+
+  
+
+
+  
+
+  
+
   
 
 
diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c
index 9a514a6..4f4604f 100644
--- a/src/conf/domain_conf.c
+++ b/src/conf/domain_conf.c
@@ -2955,13 +2955,30 @@ virDomainLoaderDefFree(virDomainLoaderDefPtr loader)
 
 
 static void
+virDomainResctrlMonDefFree(virDomainResctrlMonDefPtr domresmon)
+{
+if (!domresmon)
+return;
+
+virBitmapFree(domresmon->vcpus);
+virObjectUnref(domresmon->instance);
+}
+
+
+static void
 virDomainResctrlDefFree(virDomainResctrlDefPtr resctrl)
 {
+size_t i = 0;
+
 if (!resctrl)
 return;
 
+for (i = 0; i < resctrl->nmonitors; i++)
+virDomainResctrlMonDefFree(resctrl->monitors[i]);
+
 virObjectUnref(resctrl->alloc);
 virBitmapFree(resctrl->vcpus);
+VIR_FREE(resctrl->monitors);
 VIR_FREE(resctrl);
 }
 
@@ -18919,6 +18936,154 @@ virDomainCachetuneDefParseCache(xmlXPathContextPtr 
ctxt,
 return ret;
 }
 
+/* Checking if the monitor's vcpus is conflicted with existing allocation
+ * and monitors.
+ *
+ * Returns 1 if @vcpus equals to @resctrl->vcpus, means it is a default
+ * monitor. Returns - 1 if a conflict found. Returns 0 if no conflict and
+ * @vcpus is not equal to @resctrl->vcpus.
+ * */
+static int
+virDomainResctrlMonValidateVcpu(virDomainResctrlDefPtr resctrl,
+virBitmapPtr vcpus)
+{
+size_t i = 0;
+int vcpu = -1;
+
+if (virBitmapIsAllClear(vcpus)) {
+virReportError(VIR_ERR_INVALID_ARG, "%s",
+   _("vcpus is empty"));
+return -1;
+}
+
+while ((vcpu = virBitmapNextSetBit(vcpus, vcpu)) >= 0) {
+if (!virBitmapIsBitSet(resctrl->vcpus, vcpu)) {
+