Re: [openstack-dev] Custom Nova scheduler based on CPU queue length (Clint Byrum)

2015-10-21 Thread Rahul Nair 
Hi Clint,

Yes this is a specific use case, I am trying to analyze if there would be
any performance improvement if we use processor queue length as a weight in
the filter scheduler.

Regards,
Rahul U Nair

On Wed, Oct 21, 2015 at 2:54 AM, 
wrote:

> Send OpenStack-dev mailing list submissions to
> openstack-dev@lists.openstack.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> or, via email, send a message with subject or body 'help' to
> openstack-dev-requ...@lists.openstack.org
>
> You can reach the person managing the list at
> openstack-dev-ow...@lists.openstack.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of OpenStack-dev digest..."
>
>
> Today's Topics:
>
>1. Re: [Neutron] Do not merge until further notice (Armando M.)
>2. Re: [Neutron] HenryG addition to the Neutron  Drivers team
>   (Kevin Benton)
>3. Re: [Fuel][QA][Plugins] Move functional tests from fuel-qa to
>   the plugins (Anastasia Urlapova)
>4. Re: [Fuel] Modularization activity POC (Mike Scherbakov)
>5. Re: [Fuel][Plugins] Plugin deployment questions (Mike Scherbakov)
>6. Re: [Fuel][QA][Plugins] Move functional tests from fuel-qa to
>   the plugins (Mike Scherbakov)
>7. Re: [all][heat] Which repo to use in docs --
>   git.openstack.org or github.com? (Flavio Percoco)
>8. Re: [mistral] How to call 3rd-party tools(such as Ansible) in
>   Mistral (Renat Akhmerov)
>9. Re: [mistral] How to call 3rd-party tools(such as Ansible) in
>   Mistral (Renat Akhmerov)
>   10. Re: [Fuel][QA][Plugins] Move functional tests from fuel-qa to
>   the plugins (Sebastian Kalinowski)
>   11. Re: [Heat] core team nomination (Qiming Teng)
>   12. Re: Custom Nova scheduler based on CPU queue length (Clint Byrum)
>   13. [openstack-dev]
>   openstack-barbican-authenticate-keystone-barbican-command
>   (OpenStack Mailing List Archive)
>   14. Re: [Neutron] HenryG addition to the Neutron  Drivers team
>   (Anna Kamyshnikova)
>   15. Re: [Heat] core team nomination (Sergey Kraynev)
>   16. [Magnum] Mitaka Design Summit Sessions in Tokyo (Adrian Otto)
>   17. Re: [Heat] core team nomination (Manickam, Kanagaraj)
>   18. [Glance] Spec review sprint in Tokyo (Flavio Percoco)
>   19. [security] The first version of the Logo for Openstack
>   Security Project (Michael Xin)
>
>
> --
>
> Message: 1
> Date: Tue, 20 Oct 2015 22:48:30 -0700
> From: "Armando M." 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Subject: Re: [openstack-dev] [Neutron] Do not merge until further
> notice
> Message-ID:
> <
> cak+rqebgqr__qchj2ca3y6drdnd2s1xrptzovxrrygmb4tt...@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On 20 October 2015 at 19:46, Takashi Yamamoto 
> wrote:
>
> > i missed the "further notice"?
> >
>
> No, you didn't. RC3 released, Liberty released, the world moved on and I
> didn't think of sending an email. Sorry.
>
>
> >
> > On Wed, Oct 14, 2015 at 4:07 AM, Armando M.  wrote:
> > > Hi folks,
> > >
> > > We are in the last hours of Liberty, let's pause for a second and
> > consider
> > > merging patches only if absolutely necessary. The gate is getting
> clogged
> > > and we need to give priority to potential RC3 fixes or gate stability
> > fixes.
> > >
> > > Thanks,
> > > Armando
> > >
> > >
> >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> -- next part --
> An HTML attachment was scrubbed...
> URL: <
> http://lists.openstack.org/pipermail/openstack-dev/attachments/20151020/11840378/attachment-0001.html
> >
>
> --
>
> Message: 2
> Date: Tue, 20 Oct 2015 22:49:32 -0700
> From: Kevin Benton 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Subject: Re: [openstack-dev] [Neutron] HenryG addition to the Neutron
> Drivers team
> Message-ID:
> <
> cao_f6jnjg9y1yxzb8baon7+krnm_icgitbpjb6kmre7mp62...@mail.gmail.com>
> Content-Type: text/plain;

Re: [openstack-dev] Custom Nova scheduler based on CPU queue length

2015-10-21 Thread Clint Byrum 
Excerpts from Rahul Nair's message of 2015-10-20 18:53:09 -0700:
> ​Hi All,
> 
> I am fairly new to the OpenStack community and is trying to create a custom
> scheduler for Nova. A scheduler based on CPU queue length. Kindly apologize
> if there are any inaccuracies in my statements.
> 
> As I understand a weigher for the filter scheduler that weighs filtered
> host machines based on the host weight option, 'cpu.percent' can be
> configured to prioritize the hosts based on CPU percentage, but there are
> only limited options when it comes to filtering of machines in the first
> place, that too using the CPU queue length.
> 
> Also, I understand that IBM has its platform resource scheduler that can be
> used to build custom plugins to get user defined metrics, is there a
> similar way in pure Openstack which can be used to get the CPU queue length.
> 
> As of now, I am thinking of writing a custom script to be kept in all
> compute nodes to retrieve the CPU queue length and send it to the Nova
> controller, is this the way to go, or is there a defined approach that I
> can follow to implement a scheduler that uses CPU queue length to filter
> physical machines.
> 
> Any pointers on this would be really helpful.

Rahul, I'm realy curious about what it is you're trying to accomplish
here.

The CPU queue length is an extremely volatile statistic. Is this maybe
for a single-purpose cloud where you do have some idea what might be
running?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Custom Nova scheduler based on CPU queue length

2015-10-21 Thread Murray, Paul (HP Cloud) 
Hi Rahul,

> As I understand a weigher for the filter scheduler that weighs filtered host 
> machines based on the host weight option, 'cpu.percent' can be
> configured to prioritize the hosts based on CPU percentage, but there are 
> only limited options when it comes to filtering of machines in the
> first place, that too using the CPU queue length.
>
> Also, I understand that IBM has its platform resource scheduler that can be 
> used to build custom plugins to get user defined metrics, is there
> a similar way in pure Openstack which can be used to get the CPU queue length.
>
> As of now, I am thinking of writing a custom script to be kept in all compute 
> nodes to retrieve the CPU queue length and send it to the Nova
> controller, is this the way to go, or is there a defined approach that I can 
> follow to implement a scheduler that uses CPU queue length to filter
> physical machines.

Nova has metric plugins (monitors) in the resource tracker at the compute 
manager that will report metric data like this to the scheduler.
Any weigher plugins can use that data.

To see how cpu.percentage is set and used see the following plugins:
nova/compute/monitors/cpu_monitor.py
nova/scheduler/weights/metrics.py

You can create new monitor and weigher plugins using these as a model or 
propose an addition to cpu_monitor.py if you think it is generally useful.

Paul
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Custom Nova scheduler based on CPU queue length

2015-10-20 Thread Rahul Nair 
​Hi All,

I am fairly new to the OpenStack community and is trying to create a custom
scheduler for Nova. A scheduler based on CPU queue length. Kindly apologize
if there are any inaccuracies in my statements.

As I understand a weigher for the filter scheduler that weighs filtered
host machines based on the host weight option, 'cpu.percent' can be
configured to prioritize the hosts based on CPU percentage, but there are
only limited options when it comes to filtering of machines in the first
place, that too using the CPU queue length.

Also, I understand that IBM has its platform resource scheduler that can be
used to build custom plugins to get user defined metrics, is there a
similar way in pure Openstack which can be used to get the CPU queue length.

As of now, I am thinking of writing a custom script to be kept in all
compute nodes to retrieve the CPU queue length and send it to the Nova
controller, is this the way to go, or is there a defined approach that I
can follow to implement a scheduler that uses CPU queue length to filter
physical machines.

Any pointers on this would be really helpful.

Regards,
Rahul U Nair
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev