Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-09 Thread Meng Xu 
On Wed, Aug 9, 2017 at 3:32 AM, Dario Faggioli
 wrote:
> On Tue, 2017-08-08 at 15:55 -0700, Meng Xu wrote:
>> On Tue, Aug 8, 2017 at 3:24 PM, Dario Faggioli
>>  wrote:
>> >
>> > Therefore, I think I would set extratime as on by default in both
>> > Xen
>> > an xl. What do you think?
>> >
>>
>> Right now, the domain is created with its VCPUs' extratime flag on.
>> So
>> by default, extratime is set on in Xen.
>>
>> I'm not sure what do you suggest setting the extratime flag on by
>> default in xl?
>> Did you mean if users do not input -e option, the extratime flag will
>> be set as on?
>>
> No, as I said, I'm ok with the requirement of -e 0/1 always having to
> be present, when changing the vCPU(s) parameters with xl.
>
> I'm talking about what happens at domain creation time.
>
> If the default in Xen is already 'extratime on', I think we're mostly
> fine.

Yes. It is.

>
> As for xl/libxl, I think it would probably be good to take care of
> extratime, e.g., in sched_rtds_domain_set() (in a similar way to how we
> deal with period and budget, i.e., taking advantage of
> LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT).
>

OK.

Thanks,

Meng

---
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-09 Thread Dario Faggioli 
On Tue, 2017-08-08 at 15:55 -0700, Meng Xu wrote:
> On Tue, Aug 8, 2017 at 3:24 PM, Dario Faggioli
>  wrote:
> > 
> > Therefore, I think I would set extratime as on by default in both
> > Xen
> > an xl. What do you think?
> > 
> 
> Right now, the domain is created with its VCPUs' extratime flag on.
> So
> by default, extratime is set on in Xen.
> 
> I'm not sure what do you suggest setting the extratime flag on by
> default in xl?
> Did you mean if users do not input -e option, the extratime flag will
> be set as on?
>
No, as I said, I'm ok with the requirement of -e 0/1 always having to
be present, when changing the vCPU(s) parameters with xl.

I'm talking about what happens at domain creation time.

If the default in Xen is already 'extratime on', I think we're mostly
fine.

As for xl/libxl, I think it would probably be good to take care of
extratime, e.g., in sched_rtds_domain_set() (in a similar way to how we
deal with period and budget, i.e., taking advantage of
LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT).

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-08 Thread Meng Xu 
On Tue, Aug 8, 2017 at 3:24 PM, Dario Faggioli
 wrote:
> On Tue, 2017-08-08 at 12:16 -0700, Meng Xu wrote:
>> On Tue, Aug 8, 2017 at 9:09 AM, Dario Faggioli
>>  wrote:
>> > On Sun, 2017-08-06 at 22:43 -0400, Meng Xu wrote:
>> > >
>> > > As to (1), if users want to set some VCPUs with extratime flag
>> > > set
>> > > and
>> > > some with extratime flag clear, there are two types of input:
>> > > (a) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -e 0 -v 2 -p 1
>> > > -b
>> > > 4000 -e 1 -v 5 -p 1 -b 4000 -e 0
>> > > (b) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -v 2 -p 1 -b
>> > > 4000 -e
>> > > 1 -v 5 -p 1 -b 4000
>> > > I felt that the style (a) is more intuitive and the input
>> > > commands
>> > > have very static pattern, i.e., each vcpu must have -v -p -b -e
>> > > options set.
>> > >
>> >
>> > Exactly, I do think that (b) is indeed a better user interface.
>> >
>> With the approach (b), what I have in my mind was: if users do not
>> use
>> -e option for a vcpu index, the vcpu will have its extratime flag
>> cleared.
>> If not-setting -e option for a VCPU means using the current extratime
>> value for the VCPU, then how should users clear the extratime flag
>> for
>> a VCPU?
>>
> Yeah, I know... Well, it's an hard interface to get right.
>
> So, I think, considering how things currently work for budget and
> period, I guess I'm fine with the -e switch taking a 0/1 value.
>
> I've checked how it was in SEDF, and it was like that in there too
> (see, e.g. commit 1583cdd1fdded49698503a699c5868643051e391).
>
>> If you look at the -p and -b option for the xl sched-rtds, we will
>> find that users will have to first read both parameters of a VCPU
>> even
>> if they only want to change the value for one parameter, either -p or
>> -b. We don't allow users to specify -p or -b without an input value.
>>
> Yes. Which I now remember as something I've never really liked. But
> again, it's an interface which is a bit hard to get right. And it's
> certainly not this patch series' job to change it.
>
> So, let's stick with it. Thanks for bearing with me. :-)

No problem at all. :-)
I also checked the SEDF scheduler's commands while I was working on
this patch version.
I felt that keeping the same format for the -p, -b and -e options is a
better idea.

>
>
> I now want to bring something new on the table, though: what should the
> default be?
>
> I mean, what do we expect most people to want, e.g., at domain creation
> time, if they don't include an 'extratime=1' in their config file
> (actually, I don't think it's even possible to do that! :-O) ?
>
> I believe that --kind of unlikely wrt what happens in the real-time
> research and papers-- most users would expect a work conserving
> scheduler, unless they specify otherwise.
>
> As in, they hopefully will enjoy being able to reserve some CPU
> bandwidth in a very precise and deterministic way, for their vCPUs. But
> I don't think they see as a good thing the fact that those vCPUs stops
> running at some point, even if the system is idle.
>
> Also, I think we really should set dom0 to be in extratime mode.
>
> Therefore, I think I would set extratime as on by default in both Xen
> an xl. What do you think?
>

Right now, the domain is created with its VCPUs' extratime flag on. So
by default, extratime is set on in Xen.

I'm not sure what do you suggest setting the extratime flag on by default in xl?
Did you mean if users do not input -e option, the extratime flag will
be set as on?
If so, users may get confused IMHO. Some users may think not setting
-e option indicating clear the extratime flag, while some who
carefully read the instruction of the commands know the xl set the
extratime flag by default if -e option is not provided.

Thanks,

Meng

---
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-08 Thread Dario Faggioli 
On Tue, 2017-08-08 at 12:16 -0700, Meng Xu wrote:
> On Tue, Aug 8, 2017 at 9:09 AM, Dario Faggioli
>  wrote:
> > On Sun, 2017-08-06 at 22:43 -0400, Meng Xu wrote:
> > > 
> > > As to (1), if users want to set some VCPUs with extratime flag
> > > set
> > > and
> > > some with extratime flag clear, there are two types of input:
> > > (a) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -e 0 -v 2 -p 1
> > > -b
> > > 4000 -e 1 -v 5 -p 1 -b 4000 -e 0
> > > (b) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -v 2 -p 1 -b
> > > 4000 -e
> > > 1 -v 5 -p 1 -b 4000
> > > I felt that the style (a) is more intuitive and the input
> > > commands
> > > have very static pattern, i.e., each vcpu must have -v -p -b -e
> > > options set.
> > > 
> > 
> > Exactly, I do think that (b) is indeed a better user interface.
> > 
> With the approach (b), what I have in my mind was: if users do not
> use
> -e option for a vcpu index, the vcpu will have its extratime flag
> cleared.
> If not-setting -e option for a VCPU means using the current extratime
> value for the VCPU, then how should users clear the extratime flag
> for
> a VCPU? 
>
Yeah, I know... Well, it's an hard interface to get right.

So, I think, considering how things currently work for budget and
period, I guess I'm fine with the -e switch taking a 0/1 value.

I've checked how it was in SEDF, and it was like that in there too
(see, e.g. commit 1583cdd1fdded49698503a699c5868643051e391).

> If you look at the -p and -b option for the xl sched-rtds, we will
> find that users will have to first read both parameters of a VCPU
> even
> if they only want to change the value for one parameter, either -p or
> -b. We don't allow users to specify -p or -b without an input value.
> 
Yes. Which I now remember as something I've never really liked. But
again, it's an interface which is a bit hard to get right. And it's
certainly not this patch series' job to change it.

So, let's stick with it. Thanks for bearing with me. :-)


I now want to bring something new on the table, though: what should the
default be?

I mean, what do we expect most people to want, e.g., at domain creation
time, if they don't include an 'extratime=1' in their config file
(actually, I don't think it's even possible to do that! :-O) ?

I believe that --kind of unlikely wrt what happens in the real-time
research and papers-- most users would expect a work conserving
scheduler, unless they specify otherwise.

As in, they hopefully will enjoy being able to reserve some CPU
bandwidth in a very precise and deterministic way, for their vCPUs. But
I don't think they see as a good thing the fact that those vCPUs stops
running at some point, even if the system is idle.

Also, I think we really should set dom0 to be in extratime mode.

Therefore, I think I would set extratime as on by default in both Xen
an xl. What do you think?

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-08 Thread Meng Xu 
On Tue, Aug 8, 2017 at 9:09 AM, Dario Faggioli
 wrote:
> On Sun, 2017-08-06 at 22:43 -0400, Meng Xu wrote:
>> On Sun, Aug 6, 2017 at 12:22 PM, Meng Xu 
>> wrote:
>> >
>> > diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
>> > index 2c71a9f..88933a4 100644
>> > --- a/tools/xl/xl_cmdtable.c
>> > +++ b/tools/xl/xl_cmdtable.c
>> > @@ -272,12 +272,13 @@ struct cmd_spec cmd_table[] = {
>> >  { "sched-rtds",
>> >_sched_rtds, 0, 1,
>> >"Get/set rtds scheduler parameters",
>> > -  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-
>> > b[=BUDGET]]]",
>> > +  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]]
>> > [-e[=EXTRATIME]]]",
>> >"-d DOMAIN, --domain=DOMAIN Domain to modify\n"
>> >"-v VCPUID/all, --vcpuid=VCPUID/allVCPU to modify or
>> > output;\n"
>> >"   Using '-v all' to modify/output all vcpus\n"
>> >"-p PERIOD, --period=PERIOD Period (us)\n"
>> >"-b BUDGET, --budget=BUDGET Budget (us)\n"
>> > +  "-e EXTRATIME, --extratime=EXTRATIME EXTRATIME (1=yes,
>> > 0=no)\n"
>>
>> Hi Dario,
>>
>> I kept the EXTRATIME value for -e option because: (1) it may be more
>> intuitive for users; (2) it needs much less code change than the
>> input
>> style that does not need EXTRATIME value.
>>
> I'm of the opposite view.
>
> But again, it's tools' people views' that count. :-D
>
>> As to (1), if users want to set some VCPUs with extratime flag set
>> and
>> some with extratime flag clear, there are two types of input:
>> (a) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -e 0 -v 2 -p 1 -b
>> 4000 -e 1 -v 5 -p 1 -b 4000 -e 0
>> (b) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -v 2 -p 1 -b 4000 -e
>> 1 -v 5 -p 1 -b 4000
>> I felt that the style (a) is more intuitive and the input commands
>> have very static pattern, i.e., each vcpu must have -v -p -b -e
>> options set.
>>
> Exactly, I do think that (b) is indeed a better user interface.
>
> For instance, what if I want to change period and budget of vcpu 1 of
> domain 3, _without_ changing whether or not it can use the extra time.

With the approach (b), what I have in my mind was: if users do not use
-e option for a vcpu index, the vcpu will have its extratime flag
cleared.
If not-setting -e option for a VCPU means using the current extratime
value for the VCPU, then how should users clear the extratime flag for
a VCPU? Are you indicating the -e option has three meanings:
If -e option is set without value, keep the extratime value unchanged;
If -e option is set with value 0, clear the extratime value;
If -e option is set with value 1, set the extratime value.


If you look at the -p and -b option for the xl sched-rtds, we will
find that users will have to first read both parameters of a VCPU even
if they only want to change the value for one parameter, either -p or
-b. We don't allow users to specify -p or -b without an input value.

By looking at how -p and -b options are handled, I leaned to the
approach (a): users must input a value for the -e option,  similar to
how  the -p and -b options are handled.

>
> With (a), I don't think I can do that. Or at least, I'd have to either
> remember or check what extratime is right now, and pass that same value
> explicitly to `xl sched-rtds -d 3 -v 1 ...'.
>
> That does not look super user-friendly to me.
>
>> As to (2), if we go with -e without EXTRATIME, we will have to keep
>> track of the vcpu that has no -e option. I thought about this option,
>> we can pre-set the extratime value to false when -v option is
>> assigned:
>> case 'v':
>> ...
>> extratimes[v_index]  = 0;
>>
>> and set the extratimes[v_index] = 0 when -e is set.
>>
> Yes, something like that. Or, even better, use its current value.
>
> That would require calling libxl_vcpu_sched_params_get() (or, at times,
> libxl_vcpu_sched_params_get_all()), which I assumed you were doing
> already, while you apparently don't. Mmm...
>
>> This approach is not very neat in the code: we have to reallocate
>> memory for extratimes array when its size is not enough; we also have
>> to deal with the special case when -e is set before -v, such as the
>> command "xl sched-rtds -p 1 -b 4000 -e -v 0"
>>
> Err... sorry, there's code for reallocations in this patch already,
> isn't this the case?
>
> Also, it may be me, but I don't understand how this is different from
> how -b and -p are dealt with.
>
> Regards,
> Dario
> --
> <> (Raistlin Majere)
> -
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

Thanks,

Meng

-- 
---
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-08 Thread Dario Faggioli 
On Sun, 2017-08-06 at 22:43 -0400, Meng Xu wrote:
> On Sun, Aug 6, 2017 at 12:22 PM, Meng Xu 
> wrote:
> > 
> > diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
> > index 2c71a9f..88933a4 100644
> > --- a/tools/xl/xl_cmdtable.c
> > +++ b/tools/xl/xl_cmdtable.c
> > @@ -272,12 +272,13 @@ struct cmd_spec cmd_table[] = {
> >  { "sched-rtds",
> >    _sched_rtds, 0, 1,
> >    "Get/set rtds scheduler parameters",
> > -  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-
> > b[=BUDGET]]]",
> > +  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]]
> > [-e[=EXTRATIME]]]",
> >    "-d DOMAIN, --domain=DOMAIN Domain to modify\n"
> >    "-v VCPUID/all, --vcpuid=VCPUID/allVCPU to modify or
> > output;\n"
> >    "   Using '-v all' to modify/output all vcpus\n"
> >    "-p PERIOD, --period=PERIOD Period (us)\n"
> >    "-b BUDGET, --budget=BUDGET Budget (us)\n"
> > +  "-e EXTRATIME, --extratime=EXTRATIME EXTRATIME (1=yes,
> > 0=no)\n"
> 
> Hi Dario,
> 
> I kept the EXTRATIME value for -e option because: (1) it may be more
> intuitive for users; (2) it needs much less code change than the
> input
> style that does not need EXTRATIME value.
> 
I'm of the opposite view.

But again, it's tools' people views' that count. :-D

> As to (1), if users want to set some VCPUs with extratime flag set
> and
> some with extratime flag clear, there are two types of input:
> (a) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -e 0 -v 2 -p 1 -b
> 4000 -e 1 -v 5 -p 1 -b 4000 -e 0
> (b) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -v 2 -p 1 -b 4000 -e
> 1 -v 5 -p 1 -b 4000
> I felt that the style (a) is more intuitive and the input commands
> have very static pattern, i.e., each vcpu must have -v -p -b -e
> options set.
>
Exactly, I do think that (b) is indeed a better user interface.

For instance, what if I want to change period and budget of vcpu 1 of
domain 3, _without_ changing whether or not it can use the extra time. 

With (a), I don't think I can do that. Or at least, I'd have to either
remember or check what extratime is right now, and pass that same value
explicitly to `xl sched-rtds -d 3 -v 1 ...'.

That does not look super user-friendly to me.

> As to (2), if we go with -e without EXTRATIME, we will have to keep
> track of the vcpu that has no -e option. I thought about this option,
> we can pre-set the extratime value to false when -v option is
> assigned:
> case 'v':
> ...
> extratimes[v_index]  = 0;
> 
> and set the extratimes[v_index] = 0 when -e is set.
> 
Yes, something like that. Or, even better, use its current value.

That would require calling libxl_vcpu_sched_params_get() (or, at times,
libxl_vcpu_sched_params_get_all()), which I assumed you were doing
already, while you apparently don't. Mmm...

> This approach is not very neat in the code: we have to reallocate
> memory for extratimes array when its size is not enough; we also have
> to deal with the special case when -e is set before -v, such as the
> command "xl sched-rtds -p 1 -b 4000 -e -v 0"
> 
Err... sorry, there's code for reallocations in this patch already,
isn't this the case?

Also, it may be me, but I don't understand how this is different from
how -b and -p are dealt with.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-06 Thread Meng Xu 
On Sun, Aug 6, 2017 at 12:22 PM, Meng Xu  wrote:
> Change main_sched_rtds and related output functions to support
> per-VCPU extratime flag.
>
> Signed-off-by: Meng Xu 
>
> ---
> Changes from RFC v1
> Changes work_conserving flag to extratime flag
> ---
>  tools/xl/xl_cmdtable.c |  3 ++-
>  tools/xl/xl_sched.c| 56 
> ++
>  2 files changed, 40 insertions(+), 19 deletions(-)
>
> diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
> index 2c71a9f..88933a4 100644
> --- a/tools/xl/xl_cmdtable.c
> +++ b/tools/xl/xl_cmdtable.c
> @@ -272,12 +272,13 @@ struct cmd_spec cmd_table[] = {
>  { "sched-rtds",
>_sched_rtds, 0, 1,
>"Get/set rtds scheduler parameters",
> -  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]]]",
> +  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]] 
> [-e[=EXTRATIME]]]",
>"-d DOMAIN, --domain=DOMAIN Domain to modify\n"
>"-v VCPUID/all, --vcpuid=VCPUID/allVCPU to modify or output;\n"
>"   Using '-v all' to modify/output all vcpus\n"
>"-p PERIOD, --period=PERIOD Period (us)\n"
>"-b BUDGET, --budget=BUDGET Budget (us)\n"
> +  "-e EXTRATIME, --extratime=EXTRATIME EXTRATIME (1=yes, 0=no)\n"

Hi Dario,

I kept the EXTRATIME value for -e option because: (1) it may be more
intuitive for users; (2) it needs much less code change than the input
style that does not need EXTRATIME value.

As to (1), if users want to set some VCPUs with extratime flag set and
some with extratime flag clear, there are two types of input:
(a) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -e 0 -v 2 -p 1 -b
4000 -e 1 -v 5 -p 1 -b 4000 -e 0
(b) xl sched-rtds -d 1 -v 1 -p 1 -b 4000 -v 2 -p 1 -b 4000 -e
1 -v 5 -p 1 -b 4000
I felt that the style (a) is more intuitive and the input commands
have very static pattern, i.e., each vcpu must have -v -p -b -e
options set.

As to (2), if we go with -e without EXTRATIME, we will have to keep
track of the vcpu that has no -e option. I thought about this option,
we can pre-set the extratime value to false when -v option is
assigned:
case 'v':
...
extratimes[v_index]  = 0;

and set the extratimes[v_index] = 0 when -e is set.

This approach is not very neat in the code: we have to reallocate
memory for extratimes array when its size is not enough; we also have
to deal with the special case when -e is set before -v, such as the
command "xl sched-rtds -p 1 -b 4000 -e -v 0"

Best,

Meng

---
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH v1 3/3] xl: enable per-VCPU extratime flag for RTDS

2017-08-06 Thread Meng Xu 
Change main_sched_rtds and related output functions to support
per-VCPU extratime flag.

Signed-off-by: Meng Xu 

---
Changes from RFC v1
Changes work_conserving flag to extratime flag
---
 tools/xl/xl_cmdtable.c |  3 ++-
 tools/xl/xl_sched.c| 56 ++
 2 files changed, 40 insertions(+), 19 deletions(-)

diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 2c71a9f..88933a4 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -272,12 +272,13 @@ struct cmd_spec cmd_table[] = {
 { "sched-rtds",
   _sched_rtds, 0, 1,
   "Get/set rtds scheduler parameters",
-  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]]]",
+  "[-d  [-v[=VCPUID/all]] [-p[=PERIOD]] [-b[=BUDGET]] 
[-e[=EXTRATIME]]]",
   "-d DOMAIN, --domain=DOMAIN Domain to modify\n"
   "-v VCPUID/all, --vcpuid=VCPUID/allVCPU to modify or output;\n"
   "   Using '-v all' to modify/output all vcpus\n"
   "-p PERIOD, --period=PERIOD Period (us)\n"
   "-b BUDGET, --budget=BUDGET Budget (us)\n"
+  "-e EXTRATIME, --extratime=EXTRATIME EXTRATIME (1=yes, 0=no)\n"
 },
 { "domid",
   _domid, 0, 0,
diff --git a/tools/xl/xl_sched.c b/tools/xl/xl_sched.c
index 85722fe..5138012 100644
--- a/tools/xl/xl_sched.c
+++ b/tools/xl/xl_sched.c
@@ -251,7 +251,7 @@ static int sched_rtds_domain_output(
 libxl_domain_sched_params scinfo;
 
 if (domid < 0) {
-printf("%-33s %4s %9s %9s\n", "Name", "ID", "Period", "Budget");
+printf("%-33s %4s %9s %9s %10s\n", "Name", "ID", "Period", "Budget", 
"Extra time");
 return 0;
 }
 
@@ -262,11 +262,12 @@ static int sched_rtds_domain_output(
 }
 
 domname = libxl_domid_to_name(ctx, domid);
-printf("%-33s %4d %9d %9d\n",
+printf("%-33s %4d %9d %9d %10s\n",
 domname,
 domid,
 scinfo.period,
-scinfo.budget);
+scinfo.budget,
+scinfo.extratime ? "yes" : "no");
 free(domname);
 libxl_domain_sched_params_dispose();
 return 0;
@@ -279,8 +280,8 @@ static int sched_rtds_vcpu_output(int domid, 
libxl_vcpu_sched_params *scinfo)
 int i;
 
 if (domid < 0) {
-printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
-   "VCPU", "Period", "Budget");
+printf("%-33s %4s %4s %9s %9s %10s\n", "Name", "ID",
+   "VCPU", "Period", "Budget", "Extra time");
 return 0;
 }
 
@@ -290,12 +291,13 @@ static int sched_rtds_vcpu_output(int domid, 
libxl_vcpu_sched_params *scinfo)
 
 domname = libxl_domid_to_name(ctx, domid);
 for ( i = 0; i < scinfo->num_vcpus; i++ ) {
-printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32" %10s\n",
domname,
domid,
scinfo->vcpus[i].vcpuid,
scinfo->vcpus[i].period,
-   scinfo->vcpus[i].budget);
+   scinfo->vcpus[i].budget,
+   scinfo->vcpus[i].extratime ? "yes" : "no");
 }
 free(domname);
 return 0;
@@ -309,8 +311,8 @@ static int sched_rtds_vcpu_output_all(int domid,
 int i;
 
 if (domid < 0) {
-printf("%-33s %4s %4s %9s %9s\n", "Name", "ID",
-   "VCPU", "Period", "Budget");
+printf("%-33s %4s %4s %9s %9s %10s\n", "Name", "ID",
+   "VCPU", "Period", "Budget", "Extra time");
 return 0;
 }
 
@@ -321,12 +323,13 @@ static int sched_rtds_vcpu_output_all(int domid,
 
 domname = libxl_domid_to_name(ctx, domid);
 for ( i = 0; i < scinfo->num_vcpus; i++ ) {
-printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32"\n",
+printf("%-33s %4d %4d %9"PRIu32" %9"PRIu32" %10s\n",
domname,
domid,
scinfo->vcpus[i].vcpuid,
scinfo->vcpus[i].period,
-   scinfo->vcpus[i].budget);
+   scinfo->vcpus[i].budget,
+   scinfo->vcpus[i].extratime ? "yes" : "no");
 }
 free(domname);
 return 0;
@@ -702,14 +705,18 @@ int main_sched_rtds(int argc, char **argv)
 int *vcpus = (int *)xmalloc(sizeof(int)); /* IDs of VCPUs that change */
 int *periods = (int *)xmalloc(sizeof(int)); /* period is in microsecond */
 int *budgets = (int *)xmalloc(sizeof(int)); /* budget is in microsecond */
+bool *extratimes = (bool *)xmalloc(sizeof(bool)); /* extratime is bool */
 int v_size = 1; /* size of vcpus array */
 int p_size = 1; /* size of periods array */
 int b_size = 1; /* size of budgets array */
+int e_size = 1; /* size of extratimes array */
 int v_index = 0; /* index in vcpus array */
 int p_index =0; /* index in periods array */
 int b_index =0; /* index for in budgets array */
+int e_index = 0; /* index in extratimes array */
 bool opt_p = false;
 bool opt_b = false;
+bool opt_e = false;
 bool opt_v = false;
 bool opt_all =