Re: [openstack-dev] [new][cognitive] Announcing Cognitive - a project to deliver Machine Learning as a Service for OpenStack

[Debojyoti Dutta]

Sorry Steven, Sergey, saw the email late. If there is an informal meetup
tonight, please let me know :)

Else see you on IRC

debo

On Fri, May 22, 2015 at 9:54 AM, Steven Dake (stdake) 
wrote:

>  Debo,
>
>  The Sahara cats are having a contributors meetup right now in room 218.
> @ ODS.  I have prior commitments so won’t be able to attend, but perhaps
> some of the cognitive folks could make the meetup there.
>
>  Sergey,
>
>  If the Sahara contributors plan on walking the city tonight, feel free
> to send me info off-list and I’ll try to make it.
>
>  Regards
> -steve
>
>
>   From: Sergey Lukjanov 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Friday, May 22, 2015 at 9:21 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [new][cognitive] Announcing Cognitive - a
> project to deliver Machine Learning as a Service for OpenStack
>
>   So, in case if Cognitive will use Sahara to create big data clusters -
> then it sounds ok.
>
> On Fri, May 22, 2015 at 8:54 AM, Debojyoti Dutta  wrote:
>
>> One more thing and a typo in my mail  -  we would be happy to point folks
>> to an very early version (in Django) we prototyped and opened up recently.
>>
>>  thx
>> debo
>>
>>
>>
>> On Fri, May 22, 2015 at 8:48 AM, Debojyoti Dutta 
>> wrote:
>>
>>> Hi Sergey
>>>
>>>  Thanks a lot for your interest. The bare bones API proposal is up on
>>> the wiki page http://wiki.openstack.org/Cognitive
>>>
>>>  Sahara is about deploying and managing big data workloads like hadoop,
>>> spark etc. Cognitive is about a simple API to do predictive analytics,
>>> learning, data science workflow etc etc. Thus the goals are different. AWS
>>> has Elastic MapReduce (in the same space as Sahara) and also AWS Machine
>>> Learning (http://aws.amazon.com/machine-learning/) for which there is
>>> no parallel. Should we point them to our internal 1st version of Cognitive
>>> on our github which we opened.
>>>
>>>  If it requires to use big data toolchains to do our job, we will
>>> definitely leverage Sahara for that, and not replicate the good work done
>>> in Sahara. Our primary goal is to build (within the community) a simple
>>> machine learning API (and a service) that abstracts the pain of data
>>> science for the app developer.
>>>
>>>
>>>  thx
>>>
>>> debo
>>>
>>>
>>>  PS: FWIW  I am at the summit till tonight so we could catch up here.
>>>
>>>
>>>
>>> On Tue, May 19, 2015 at 2:16 PM, Sergey Lukjanov >> > wrote:
>>>
>>>> Hi,
>>>>
>>>>  as there is no any details on the project yet done, if this project
>>>> will deploy ML frameworks it'll be direct duplication of Sahara's
>>>> functionality (we already support HDP and CDH deployments and they are
>>>> provided tons of tools for ML). So, I think that it could be built on top
>>>> of Sahara or even as part of Sahara probably. I'd like to propose you to
>>>> take a deeper look on Sahara and avoid duplicating it.
>>>>
>>>>  Thanks.
>>>>
>>>> On Thu, May 14, 2015 at 8:47 PM, Debojyoti Dutta 
>>>> wrote:
>>>>
>>>>> Hi Salvatore
>>>>>
>>>>>  Thanks a lot for your comments.
>>>>>
>>>>>  Timing: Yes it is time to do this! The nature of applications
>>>>> running on clouds is indeed changing.
>>>>>
>>>>>  Initial group: We asked around for folks interested and we got a lot
>>>>> more people than we expected. The idea is to get something out there in a
>>>>> stack forge project and build something good. This group already has 
>>>>> people
>>>>> who have built things like this already in the past. Hence confident about
>>>>> the success.
>>>>>
>>>>>  Participation: We want this to be inclusive from scratch independent
>>>>> of who is a PTL or a contributor or merely a curious individual to give us
>>>>> ideas :) The community will get it right. Maybe I should have clarified
>>>>> that these are the members interested in seeing this happen.
>>>>>
>>>>>  Wiki page: The wiki page will be ready in 1-

Re: [openstack-dev] [new][cognitive] Announcing Cognitive - a project to deliver Machine Learning as a Service for OpenStack

[Debojyoti Dutta]

One more thing and a typo in my mail  -  we would be happy to point folks
to an very early version (in Django) we prototyped and opened up recently.

thx
debo



On Fri, May 22, 2015 at 8:48 AM, Debojyoti Dutta  wrote:

> Hi Sergey
>
> Thanks a lot for your interest. The bare bones API proposal is up on the
> wiki page http://wiki.openstack.org/Cognitive
>
> Sahara is about deploying and managing big data workloads like hadoop,
> spark etc. Cognitive is about a simple API to do predictive analytics,
> learning, data science workflow etc etc. Thus the goals are different. AWS
> has Elastic MapReduce (in the same space as Sahara) and also AWS Machine
> Learning (http://aws.amazon.com/machine-learning/) for which there is no
> parallel. Should we point them to our internal 1st version of Cognitive on
> our github which we opened.
>
> If it requires to use big data toolchains to do our job, we will
> definitely leverage Sahara for that, and not replicate the good work done
> in Sahara. Our primary goal is to build (within the community) a simple
> machine learning API (and a service) that abstracts the pain of data
> science for the app developer.
>
>
> thx
>
> debo
>
>
> PS: FWIW  I am at the summit till tonight so we could catch up here.
>
>
>
> On Tue, May 19, 2015 at 2:16 PM, Sergey Lukjanov 
> wrote:
>
>> Hi,
>>
>> as there is no any details on the project yet done, if this project will
>> deploy ML frameworks it'll be direct duplication of Sahara's functionality
>> (we already support HDP and CDH deployments and they are provided tons of
>> tools for ML). So, I think that it could be built on top of Sahara or even
>> as part of Sahara probably. I'd like to propose you to take a deeper look
>> on Sahara and avoid duplicating it.
>>
>> Thanks.
>>
>> On Thu, May 14, 2015 at 8:47 PM, Debojyoti Dutta 
>> wrote:
>>
>>> Hi Salvatore
>>>
>>> Thanks a lot for your comments.
>>>
>>> Timing: Yes it is time to do this! The nature of applications running on
>>> clouds is indeed changing.
>>>
>>> Initial group: We asked around for folks interested and we got a lot
>>> more people than we expected. The idea is to get something out there in a
>>> stack forge project and build something good. This group already has people
>>> who have built things like this already in the past. Hence confident about
>>> the success.
>>>
>>> Participation: We want this to be inclusive from scratch independent of
>>> who is a PTL or a contributor or merely a curious individual to give us
>>> ideas :) The community will get it right. Maybe I should have clarified
>>> that these are the members interested in seeing this happen.
>>>
>>> Wiki page: The wiki page will be ready in 1-2 days. Also we would like
>>> to have a discussion during the summit to see what we should build in the
>>> community. Would be delighted to get your thoughts.
>>>
>>> Services: Some of the services this could provide:
>>> * create experiments: define data sources, train models, then perform
>>> classification, clustering, data cleaning etc.
>>> * have experiment templates that can be reused
>>> * have an editor (maybe a horizon plugin) to drag and drop the workflow
>>> and generate an API that when called from an app would provide results
>>> * ML primitives that could be targeted initially: 1) classification  2)
>>> clustering 3) Anomaly detection
>>>
>>> thx
>>> debo
>>>
>>> On Thu, May 14, 2015 at 5:02 PM, Salvatore Orlando 
>>> wrote:
>>>
>>>>
>>>> On 15 May 2015 at 00:19, Debojyoti Dutta  wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> It is a great pleasure to announce the development of a new project
>>>>> called Cognitive.  Cognitive provides Machine Learning [1] as a Service
>>>>> that enables operators to offer next generation data science based 
>>>>> services
>>>>> on top of their OpenStack Clouds.
>>>>>
>>>>
>>>> I was indeed wondering when "Machine Learning as a Service" would come
>>>> up...
>>>>
>>>>
>>>>> This project will begin as a StackForge project baed upon an empty
>>>>> cookiecutter [2] repo.  The repos to work in are:
>>>>> Server: https://github.com/stackforge/cognitive
>>>>> Client: https://github.com/stackforge/python-cognitiveclient
>>>&g

Re: [openstack-dev] [new][cognitive] Announcing Cognitive - a project to deliver Machine Learning as a Service for OpenStack

[Debojyoti Dutta]

Hi Sergey

Thanks a lot for your interest. The bare bones API proposal is up on the
wiki page http://wiki.openstack.org/Cognitive

Sahara is about deploying and managing big data workloads like hadoop,
spark etc. Cognitive is about a simple API to do predictive analytics,
learning, data science workflow etc etc. Thus the goals are different. AWS
has Elastic MapReduce (in the same space as Sahara) and also AWS Machine
Learning (http://aws.amazon.com/machine-learning/) for which there is no
parallel. Should we point them to our internal 1st version of Cognitive on
our github which we opened.

If it requires to use big data toolchains to do our job, we will definitely
leverage Sahara for that, and not replicate the good work done in Sahara.
Our primary goal is to build (within the community) a simple machine
learning API (and a service) that abstracts the pain of data science for
the app developer.


thx

debo


PS: FWIW  I am at the summit till tonight so we could catch up here.



On Tue, May 19, 2015 at 2:16 PM, Sergey Lukjanov 
wrote:

> Hi,
>
> as there is no any details on the project yet done, if this project will
> deploy ML frameworks it'll be direct duplication of Sahara's functionality
> (we already support HDP and CDH deployments and they are provided tons of
> tools for ML). So, I think that it could be built on top of Sahara or even
> as part of Sahara probably. I'd like to propose you to take a deeper look
> on Sahara and avoid duplicating it.
>
> Thanks.
>
> On Thu, May 14, 2015 at 8:47 PM, Debojyoti Dutta  wrote:
>
>> Hi Salvatore
>>
>> Thanks a lot for your comments.
>>
>> Timing: Yes it is time to do this! The nature of applications running on
>> clouds is indeed changing.
>>
>> Initial group: We asked around for folks interested and we got a lot more
>> people than we expected. The idea is to get something out there in a stack
>> forge project and build something good. This group already has people who
>> have built things like this already in the past. Hence confident about the
>> success.
>>
>> Participation: We want this to be inclusive from scratch independent of
>> who is a PTL or a contributor or merely a curious individual to give us
>> ideas :) The community will get it right. Maybe I should have clarified
>> that these are the members interested in seeing this happen.
>>
>> Wiki page: The wiki page will be ready in 1-2 days. Also we would like to
>> have a discussion during the summit to see what we should build in the
>> community. Would be delighted to get your thoughts.
>>
>> Services: Some of the services this could provide:
>> * create experiments: define data sources, train models, then perform
>> classification, clustering, data cleaning etc.
>> * have experiment templates that can be reused
>> * have an editor (maybe a horizon plugin) to drag and drop the workflow
>> and generate an API that when called from an app would provide results
>> * ML primitives that could be targeted initially: 1) classification  2)
>> clustering 3) Anomaly detection
>>
>> thx
>> debo
>>
>> On Thu, May 14, 2015 at 5:02 PM, Salvatore Orlando 
>> wrote:
>>
>>>
>>> On 15 May 2015 at 00:19, Debojyoti Dutta  wrote:
>>>
>>>> Hi!
>>>>
>>>> It is a great pleasure to announce the development of a new project
>>>> called Cognitive.  Cognitive provides Machine Learning [1] as a Service
>>>> that enables operators to offer next generation data science based services
>>>> on top of their OpenStack Clouds.
>>>>
>>>
>>> I was indeed wondering when "Machine Learning as a Service" would come
>>> up...
>>>
>>>
>>>> This project will begin as a StackForge project baed upon an empty
>>>> cookiecutter [2] repo.  The repos to work in are:
>>>> Server: https://github.com/stackforge/cognitive
>>>> Client: https://github.com/stackforge/python-cognitiveclient
>>>>
>>>> Please join us via iRC on #openstack-cognitive on freenode.
>>>>
>>>> We will be holding a doodle poll to select times for our first meeting
>>>> the week after summit.  This doodle poll will close May 24th and meeting
>>>> times will be announced on the mailing list at that time.  At our first IRC
>>>> meeting, we will draft additional core team members. We would like to
>>>> invite interested individuals to join this exciting new development effort!
>>>>
>>>
>>> From my little experience, "drafting" core members before even actually
>

[openstack-dev] [surge] Introducing Surge - rapid deploy/scale stream processing systems on OpenStack

[Debojyoti Dutta]

Hi,

It gives me a great pleasure to introduce Surge - a system to rapidly
deploy and scale a stream processing system on OpenStack. It leverages
Vagrant and Ansible, and supports both OpenStack as well as the local mode
(with VirtualBox).

https://github.com/CiscoSystems/surge

Hope to see a lot of pull requests and comments.

thx
-Debo~
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ceph] Introducing CephEWS

[Debojyoti Dutta]

Hi!

Its a pleasure to introduce CephEWS, an awesome dashboard for Ceph with a
twist - it has ceph health checks and warnings built in along with OSD and
CRUSH map viz etc. https://github.com/CiscoSystems/cephEWS

Hope to see a lot of pull requests and feedback!

thx
-Debo~
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [new][cognitive] Announcing Cognitive - a project to deliver Machine Learning as a Service for OpenStack

[Debojyoti Dutta]

Hi Salvatore

Thanks a lot for your comments.

Timing: Yes it is time to do this! The nature of applications running on
clouds is indeed changing.

Initial group: We asked around for folks interested and we got a lot more
people than we expected. The idea is to get something out there in a stack
forge project and build something good. This group already has people who
have built things like this already in the past. Hence confident about the
success.

Participation: We want this to be inclusive from scratch independent of who
is a PTL or a contributor or merely a curious individual to give us ideas
:) The community will get it right. Maybe I should have clarified that
these are the members interested in seeing this happen.

Wiki page: The wiki page will be ready in 1-2 days. Also we would like to
have a discussion during the summit to see what we should build in the
community. Would be delighted to get your thoughts.

Services: Some of the services this could provide:
* create experiments: define data sources, train models, then perform
classification, clustering, data cleaning etc.
* have experiment templates that can be reused
* have an editor (maybe a horizon plugin) to drag and drop the workflow and
generate an API that when called from an app would provide results
* ML primitives that could be targeted initially: 1) classification  2)
clustering 3) Anomaly detection

thx
debo

On Thu, May 14, 2015 at 5:02 PM, Salvatore Orlando 
wrote:

>
> On 15 May 2015 at 00:19, Debojyoti Dutta  wrote:
>
>> Hi!
>>
>> It is a great pleasure to announce the development of a new project
>> called Cognitive.  Cognitive provides Machine Learning [1] as a Service
>> that enables operators to offer next generation data science based services
>> on top of their OpenStack Clouds.
>>
>
> I was indeed wondering when "Machine Learning as a Service" would come
> up...
>
>
>> This project will begin as a StackForge project baed upon an empty
>> cookiecutter [2] repo.  The repos to work in are:
>> Server: https://github.com/stackforge/cognitive
>> Client: https://github.com/stackforge/python-cognitiveclient
>>
>> Please join us via iRC on #openstack-cognitive on freenode.
>>
>> We will be holding a doodle poll to select times for our first meeting
>> the week after summit.  This doodle poll will close May 24th and meeting
>> times will be announced on the mailing list at that time.  At our first IRC
>> meeting, we will draft additional core team members. We would like to
>> invite interested individuals to join this exciting new development effort!
>>
>
> From my little experience, "drafting" core members before even actually
> having a code base has drawbacks. Also, it seems the initial starting team
> is already large enough for ensuring support for 1 or 2 release cycle.
>
>
>>
>>
>
>> Please commit your schedule in the doodle poll here:
>> http://doodle.com/drrka5tgbwpbfbxy
>>
>> Initial core team: Steven Dake, Aparupa Das Gupa, Debo~ Dutta, Johnu
>> George,  Kyle Mestery, Sarvesh Ranjan, Ralf Rantzau, Komei Shimamura, Marc
>> Solanas, Manoj Sharma, Yathi Udupi, Kai Zhang.
>>
>
> Hey! What's the Neutron PTL doing there? Sorry we need his reviews we
> can't loan it to you!
>
>
>>
>> A little bit about Cognitive:
>> Data driven applications on cloud infrastructure increasingly rely on
>> Machine Learning. Most data driven applications today use Machine Learning
>> (ML). This often requires application developers and data scientists to
>> write their own machine learning stack or deploy other packages to do any
>> kind of data science based applications. Data scientists also need to have
>> an easy way to rapidly experiment with data without having to write basic
>> infrastructure for data manipulations. Cognitive is a Machine Learning
>> service on top of OpenStack and provides machine learning based services to
>> tenants (API, workbench, compute service).
>>
>
> I wonder what kind of services you would offer; also you could have shared
> something about the architecture of this service. Is it providing a full
> machine learning stack, or just facilitating the use of existing one?
>
> But I see that there's a link to a wiki page below. This might have all
> the answers.
>
>
>>
>>
>> For information about blueprints check out:
>> https://blueprints.launchpad.net/cognitive
>> https://blueprints.launchpad.net/python-cognitiveclient
>>
>> For more details, check out our Wiki:
>> https://wiki.openstack.org/wiki/Cognitive
>>
>
> ... and unfortunately the wiki is empty ;)
>
>
>>
>> Please join the

[openstack-dev] [new][cognitive] Announcing Cognitive - a project to deliver Machine Learning as a Service for OpenStack

[Debojyoti Dutta]

Hi!

It is a great pleasure to announce the development of a new project called
Cognitive.  Cognitive provides Machine Learning [1] as a Service that
enables operators to offer next generation data science based services on
top of their OpenStack Clouds. This project will begin as a StackForge
project baed upon an empty cookiecutter [2] repo.  The repos to work in are:
Server: https://github.com/stackforge/cognitive
Client: https://github.com/stackforge/python-cognitiveclient

Please join us via iRC on #openstack-cognitive on freenode.

We will be holding a doodle poll to select times for our first meeting the
week after summit.  This doodle poll will close May 24th and meeting times
will be announced on the mailing list at that time.  At our first IRC
meeting, we will draft additional core team members. We would like to
invite interested individuals to join this exciting new development effort!

Please commit your schedule in the doodle poll here:
http://doodle.com/drrka5tgbwpbfbxy

Initial core team: Steven Dake, Aparupa Das Gupa, Debo~ Dutta, Johnu
George,  Kyle Mestery, Sarvesh Ranjan, Ralf Rantzau, Komei Shimamura, Marc
Solanas, Manoj Sharma, Yathi Udupi, Kai Zhang.

A little bit about Cognitive:
Data driven applications on cloud infrastructure increasingly rely on
Machine Learning. Most data driven applications today use Machine Learning
(ML). This often requires application developers and data scientists to
write their own machine learning stack or deploy other packages to do any
kind of data science based applications. Data scientists also need to have
an easy way to rapidly experiment with data without having to write basic
infrastructure for data manipulations. Cognitive is a Machine Learning
service on top of OpenStack and provides machine learning based services to
tenants (API, workbench, compute service).

For information about blueprints check out:
https://blueprints.launchpad.net/cognitive
https://blueprints.launchpad.net/python-cognitiveclient

For more details, check out our Wiki:
https://wiki.openstack.org/wiki/Cognitive

Please join the awesome Cognitive team in designing a world class Machine
Learning as a Service solution.

We look forward to seeing you on IRC on #openstack-cognitive.

Regards,
Debo~ Dutta (on behalf of the initial team)

[1] http://en.wikipedia.org/wiki/Machine_learning
[2] https://github.com/openstack-dev/cookiecutter
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

[Debojyoti Dutta]

Hi Ruby

Good point. If you assume congress to be present, then scheduling and most
actions are a result of a policy decision and might not impact the
scheduler. The results of the LP/CVP would allow you to spawn resources at
end points.

However if you assume that there is congress + other entities, then it
might be better/cleaner to use a separate solver scheduling layer to decide
if the policies and constraints from congress are not in conflict with
other entities that might be asking for resources.

I guess its about how the community wants to layer the components. We
wanted to first build the simple constraint solver layer and then hope that
policy layers would talk to the advanced scheduler and this scheduler
driver would fit into nova scheduler or Gantt.

debo

On Fri, Feb 13, 2015 at 7:51 AM,  wrote:

>  Hello Debo/Tim
>
>
>
> My understanding is that with Congress things like filters (e.g.
> anti-affinity or other aggregates) will be replaced to be written as
> policies with Datalog.
>
> Goals (a Policy), Constraints (policies in Congress) will also get
> translated to (for example) linear programs in some modelling language
> (e.g. PuLP).
>
>
>
> So this is likely to be a major change to the scheduler?
>
>
>
> Ruby
>
>
>
> *De :* Debojyoti Dutta [mailto:ddu...@gmail.com]
> *Envoyé :* vendredi 13 février 2015 14:06
> *À :* OpenStack Development Mailing List (not for usage questions)
> *Objet :* Re: [openstack-dev] [Congress][Delegation] Google doc for
> working notes
>
>
>
> Tim
>
>
>
> Wanted to clarify a bit. As I have mentioned before: Solver scheduler is
> work done before this work (Datalog->constraints) but we had kept it
> very generic to be integrated with something like congress. In fact Ramki
> (who was one of the members of the original thread when you reached out to
> us) joined us to in talk in Atlanta where we described some of the same use
> cases using PULP  congress was still ramping up then. We were not aware
> of the Datalog->constraints work that you guys were doing, else we would
> have joined hands before.
>
>
>
> The question is this: going forward, how do build this cool stuff together
> in the community? I am hoping the scheduler folks will be very excited too!
>
>
>
> debo
>
>
>
> On Thu, Feb 12, 2015 at 11:27 AM, Yathiraj Udupi (yudupi) <
> yud...@cisco.com> wrote:
>
> Hi Tim,
>
>
>
> Thanks for your response.  Excited too to extend the collaboration and
> ensure there is no need to duplicate effort in the open source community.
>
>  My responses inline.
>
>
>
>  1)  Choice of LP solver.
>
>
>
> I see solver-scheduler uses Pulp, which was on the Congress short list as
> well.  So we’re highly aligned on the choice of underlying solver.
>
>
>
> YATHI - This makes me wonder why can’t we easily adapt the
> solver-scheduler to your needs, rather than duplicating the effort!
>
>
>
>
>
> 2) User control over VM-placement.
>
>
>
>
>
> To choose the criteria for VM-placement, the solver-scheduler user picks
> from a list of predefined options, e.g. ActiveHostConstraint,
> MaxRamAllocationPerHostConstraint.
>
>
>
> We’re investigating a slightly different approach, where the user defines
> the criteria for VM-placement by writing any policy they like in Datalog.
> Under the hood we then convert that Datalog to an LP problem.  From the
> developer’s perspective, with the Congress approach we don’t attempt to
> anticipate the different policies the user might want and write code for
> each policy; instead, we as developers write a translator from Datalog to
> LP.  From the user’s perspective, the difference is that if the option they
> want isn’t on the solver-scheduler's list, they’re out of luck or need to
> write the code themselves.  But with the Congress approach, they can write
> any VM-placement policy they like.
>
>
>
> What I’d like to see is the best of both worlds.  Users write Datalog
> policies describing whatever VM-placement policy they want.  If the policy
> they’ve written is on the solver-scheduler’s list of options, we use the
> hard-coded implementation, but if the policy isn’t on that list we
> translate directly to LP.  This approach gives us the ability to write
> custom code to handle common cases while at the same time letting users
> write whatever policy they like.
>
>
>
>
>
> YATHI -  The idea of providing some default constraint classes in Solver
> Scheduler was to enable easy pluggability for various placement policy
> scenarios.  We can easily add a custom constraint class in solver
> scheduler, that enables adding additional constraints at runti

[openstack-dev] GSoC2015: Its time for potential mentors and participants!

[Debojyoti Dutta]

Hello Everyone

It is time for us to apply for slots for the annual Google Summer of Code
event https://developers.google.com/open-source/soc/?csw=1

Last year, we got a bunch of slots and had awesome projects
https://wiki.openstack.org/wiki/GSoC2014
We are hoping this year we will get even more slots, uber cool projects etc.

If you are interested - either as a mentor or as a participant, please feel
free to add your name, project ideas to the wiki page for this yr
https://wiki.openstack.org/wiki/GSoC2015

-- 
OpenStack GSoC team
(Debo~, Dims, Victoria)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

[Debojyoti Dutta]

r
> node (e.g. ceilometer’s data), and the algorithms for translating Datalog
> to LP look to be quite similar to the algorithms we’re using in our
> domain-agnostic policy engine.
>
>
>  YATHI – The entire scheduling community in Nova is planning on an
> external scheduler (Gantt), and we are pitching solver scheduler also as a
> stand-alone placement engine a scheduler as a service.  Nova integration is
> just to ensure it fits within the Nova workflow.   I am not quite familiar
> with the DSE architecture yet,  but the simple idea we have is, Congress
> policies, as part of the enforcement workflow, should set the VM placement
> constraints, and feed any additional data to be used for
> scheduling/placement decisions, which will be consumed dynamically by the
> Solver Scheduler, and after the delegation, the Solver scheduler module
> will calculate the placement decisions, and complete the VM initial
> placement or call the VM migration APIs and enable the required migrations.
>
>
>
>  Thanks,
> Yathi.
>
>
>   On 2/12/15, 10:02 AM, "Tim Hinrichs"  wrote:
>
>   Hi Debo and Yathiraj,
>
>  I took a third look at the solver-scheduler docs and code with your
> comments in mind.  A few things jumped out.
>
>
>
>
>
>  2) User control over VM-placement.
>
>
>  Tim
>
>
>  On Feb 11, 2015, at 4:50 PM, Debojyoti Dutta  wrote:
>
>  Hi Tim: moving our thread to the mailer. Excited to collaborate!
>
>
>
>   From: Debo~ Dutta 
> Date: Wednesday, February 11, 2015 at 4:48 PM
> To: Tim Hinrichs 
> Cc: "Yathiraj Udupi (yudupi)" , Gokul B Kandiraju <
> go...@us.ibm.com>, Prabhakar Kudva , "
> ruby.krishnasw...@orange.com" , "
> dilik...@in.ibm.com" , Norival Figueira <
> nfigu...@brocade.com>, Ramki Krishnan , "Xinyuan Huang
> (xinyuahu)" , "Rishabh Jain -X (rishabja - AAP3 INC
> at Cisco)" 
> Subject: Re: Nova solver scheduler and Congress
>
>   Hi Tim
>
>  To address your particular questions:
>
>1. translate some policy language into constraints for the LP/CVP and
>we had left that to congress hoping to integrate when the policy efforts in
>openstack were ready (our initial effort was pre congress)
>2. For migrations: we are currently doing that – its about incremental
>constraints into the same solver. Hence its a small deal ….
>
> Joining forces is a terrific idea. Would love to join the IRC call and see
> how we can build cool stuff in the community together. I hope we don’t have
> to replicate the vm placement engine while the work that was done in the
> community does something very similar (and be adapted)
>
>  debo
>
>   From: Tim Hinrichs 
> Date: Wednesday, February 11, 2015 at 4:43 PM
> To: Debo~ Dutta 
> Cc: "Yathiraj Udupi (yudupi)" , Gokul B Kandiraju <
> go...@us.ibm.com>, Prabhakar Kudva , "
> ruby.krishnasw...@orange.com" , "
> dilik...@in.ibm.com" , Norival Figueira <
> nfigu...@brocade.com>, Ramki Krishnan , "Xinyuan Huang
> (xinyuahu)" , "Rishabh Jain -X (rishabja - AAP3 INC
> at Cisco)" 
> Subject: Re: Nova solver scheduler and Congress
>
>  Hi Debo,
>
>  The 2 efforts share great similarities, which was why we investigated
> the state of solver-scheduler.  From what I understand, (i)
> solver-scheduler doesn’t currently have a policy language and (ii) it
> doesn’t do migrations.  (I realize these are both in the works.)  We needed
> both and wanted to make progress before those were complete.
>
>  In the long run, it may make perfect sense to replace our vm-placement
> engine with yours.  So joining forces sounds like a good idea.  At the very
> *least* we ought to keep up to date with each other’s progress.
>
>  I’m starting to wonder if we ought to schedule a (bi-) weekly IRC for
> this topic.
>
>  Tim
>
>  On Feb 11, 2015, at 4:35 PM, Debo Dutta (dedutta) 
> wrote:
>
>  Hi Tim
>
>  This looks awesome. Trying to figure out how this approach is different
> from the solver scheduler effort we did? We would be happy to fold our
> solver scheduler effort into this (that way you also get code up and
> running)
>
>  Will also respond on the thread.
>
>  thx
> debo
>
>   From: Tim Hinrichs 
> Date: Wednesday, February 11, 2015 at 4:11 PM
> To: "Yathiraj Udupi (yudupi)" 
> Cc: Gokul B Kandiraju , Prabhakar Kudva <
> ku...@us.ibm.com>, "ruby.krishnasw...@orange.com" <
> ruby.krishnasw...@orange.com>, "dilik...@in.ibm.com" ,
> Norival Figueira , Ramki Krishnan ,
> "Xinyuan Huang (xinyuahu)" , "Rishabh Jain -X
> (risha

Re: [openstack-dev] [Congress][Delegation] Google doc for working notes

[Debojyoti Dutta]

Hi Tim: moving our thread to the mailer. Excited to collaborate!



From: Debo~ Dutta 
Date: Wednesday, February 11, 2015 at 4:48 PM
To: Tim Hinrichs 
Cc: "Yathiraj Udupi (yudupi)" , Gokul B Kandiraju <
go...@us.ibm.com>, Prabhakar Kudva , "
ruby.krishnasw...@orange.com" , "
dilik...@in.ibm.com" , Norival Figueira <
nfigu...@brocade.com>, Ramki Krishnan , "Xinyuan Huang
(xinyuahu)" , "Rishabh Jain -X (rishabja - AAP3 INC at
Cisco)" 
Subject: Re: Nova solver scheduler and Congress

Hi Tim

To address your particular questions:

   1. translate some policy language into constraints for the LP/CVP and we
   had left that to congress hoping to integrate when the policy efforts in
   openstack were ready (our initial effort was pre congress)
   2. For migrations: we are currently doing that – its about incremental
   constraints into the same solver. Hence its a small deal ….

Joining forces is a terrific idea. Would love to join the IRC call and see
how we can build cool stuff in the community together. I hope we don’t have
to replicate the vm placement engine while the work that was done in the
community does something very similar (and be adapted)

debo

From: Tim Hinrichs 
Date: Wednesday, February 11, 2015 at 4:43 PM
To: Debo~ Dutta 
Cc: "Yathiraj Udupi (yudupi)" , Gokul B Kandiraju <
go...@us.ibm.com>, Prabhakar Kudva , "
ruby.krishnasw...@orange.com" , "
dilik...@in.ibm.com" , Norival Figueira <
nfigu...@brocade.com>, Ramki Krishnan , "Xinyuan Huang
(xinyuahu)" , "Rishabh Jain -X (rishabja - AAP3 INC at
Cisco)" 
Subject: Re: Nova solver scheduler and Congress

Hi Debo,

The 2 efforts share great similarities, which was why we investigated the
state of solver-scheduler.  From what I understand, (i) solver-scheduler
doesn’t currently have a policy language and (ii) it doesn’t do migrations.
 (I realize these are both in the works.)  We needed both and wanted to
make progress before those were complete.

In the long run, it may make perfect sense to replace our vm-placement
engine with yours.  So joining forces sounds like a good idea.  At the very
*least* we ought to keep up to date with each other’s progress.

I’m starting to wonder if we ought to schedule a (bi-) weekly IRC for this
topic.

Tim

On Feb 11, 2015, at 4:35 PM, Debo Dutta (dedutta)  wrote:

Hi Tim

This looks awesome. Trying to figure out how this approach is different
from the solver scheduler effort we did? We would be happy to fold our
solver scheduler effort into this (that way you also get code up and
running)

Will also respond on the thread.

thx
debo

From: Tim Hinrichs 
Date: Wednesday, February 11, 2015 at 4:11 PM
To: "Yathiraj Udupi (yudupi)" 
Cc: Gokul B Kandiraju , Prabhakar Kudva ,
"ruby.krishnasw...@orange.com" , "
dilik...@in.ibm.com" , Norival Figueira <
nfigu...@brocade.com>, Ramki Krishnan , "Xinyuan Huang
(xinyuahu)" , "Rishabh Jain -X (rishabja - AAP3 INC at
Cisco)" , Debo~ Dutta 
Subject: Re: Nova solver scheduler and Congress

Hi Yathiraj,

The group is getting big enough that we’ve decided to move the entire
discussion to the openstack-dev mailing list.  I sent a note today with the
google doc we’re working on.  We’re trying to include
[Congress][Delegation] in the subject line of relevant posts.  Here’s the
gdoc.

https://docs.google.com/document/d/1ksDilJYXV-5AXWON8PLMedDKr9NpS8VbT0jIy_MIEtI/edit?usp=sharing


Tim

On Feb 10, 2015, at 11:10 AM, Yathiraj Udupi (yudupi) 
wrote:

Hi Tim,

Thanks for your response.  I think Congress will have to appreciate
different APIs interacting with multiple components in the OpenStack
ecosystem.  So I will be happy to help figure out the integration plan in
general from the Congress side.

I will  be very interested and glad to participate in the discussions of
designing these interfaces in Congress.   Please share any preliminary
designs you may have.   I had participated in one of the Congress mid-cycle
meet ups, and I am interested in the upcoming work on these kind of
enforcement aspects (reactive, proactive) of Congress.  In terms of Nova
scheduling via Solver scheduler, it will also help us be ready with the
integration points.

Let’s be in sync.

Thanks,
Yathi.


On 2/10/15, 11:03 AM, "Tim Hinrichs"  wrote:

Hi Yathiraj,

Thanks for the help!

The reason I asked is that we’re trying to figure out the basic interface
for how two policy engines (in general) ought to interact.  We were hoping
Congress and solver-scheduler had very similar APIs, which would make that
interface relatively simple.  But it sounds like the two systems have
pretty different APIs.  So for now we’ll keep working on that interface,
and once we have something wor

Re: [openstack-dev] [Nova] [Gantt][Scheduler-split] Why we need a Smart Placement Engine as a Service! (was: Scheduler split status (updated))

[Debojyoti Dutta]

https://etherpad.openstack.org/p/SchedulerUseCases

[08:43:35]  #action all update the use case etherpad
athttps://etherpad.openstack.org/p/SchedulerUseCases

Please update your use cases here ..

debo

On Mon, Jul 14, 2014 at 7:25 PM, Yathiraj Udupi (yudupi)
 wrote:
> Hi all,
>
> Adding to the interesting discussion thread regarding the scheduler split
> and its importance, I would like to pitch in a couple of thoughts in favor
> of Gantt.  It was in the Icehouse summit in HKG in one of the scheduler
> design sessions, I along with a few others (cc’d) pitched a session on Smart
> Resource Placement
> (https://etherpad.openstack.org/p/NovaIcehouse-Smart-Resource-Placement),
> where we pitched for a  Smart Placement Decision Engine  as a Service ,
> addressing cross-service scheduling as one of the use cases.  We pitched the
> idea as to how a stand-alone service can act as a  smart resource placement
> engine, (see figure:
> https://docs.google.com/drawings/d/1BgK1q7gl5nkKWy3zLkP1t_SNmjl6nh66S0jHdP0-zbY/edit?pli=1)
> that can use state data from all the services, and make a unified placement
> decision.   We even have proposed a separate blueprint
> (https://blueprints.launchpad.net/nova/+spec/solver-scheduler with working
> code now here: https://github.com/CiscoSystems/nova-solver-scheduler) called
> Smart Scheduler (Solver Scheduler), which has the goals of being able to do
> smart resource placement taking into account complex constraints
> incorporating compute(nova), storage(cinder), and network constraints.   The
> existing Filter Scheduler or the projects like Smart (Solver) Scheduler (for
> covering the complex constraints scenarios) could easily fulfill the
> decision making aspects of the placement engine.
>
> I believe the Gantt project is the right direction in terms of separating
> out the placement decision concern, and creating a separate scheduler as a
> service, so that it can freely talk to any of the other services, or use a
> unified global state repository and make the unified decision.  Projects
> like Smart(Solver) Scheduler can easily fit into the Gantt Project as
> pluggable drivers to add the additional smarts required.
>
> To make our Smart Scheduler as a service, we currently have prototyped this
> Scheduler as a service providing a RESTful interface to the smart scheduler,
> that is detached from Nova (loosely connected):
> For example a RESTful request like this (where I am requests for 2 Vms, with
> a requirement of 1 GB disk, and another request for 1 Vm of flavor
> ‘m1.tiny’, but also has a special requirement that it should be close to the
> volume with uuid: “ef6348300bc511e4bc4cc03fd564d1bc" (Compute-Volume
> affinity constraint)) :
>
>
> curl -i -H "Content-Type: application/json" -X POST -d
> '{"instance_requests": [{"num_instances": 2, "request_properties":
> {"instance_type": {"root_gb": 1}}}, {"num_instances": 1,
> "request_properties": {"flavor": "m1.tiny”, “volume_affinity":
> "ef6348300bc511e4bc4cc03fd564d1bc"}}]}'
> http:///smart-scheduler-as-a-service/v1.0/placement
>
>
> provides a placement decision something like this:
>
> {
>
>   "result": [
>
> [
>
>   {
>
> "host": {
>
>   "host": "Host1",
>
>   "nodename": "Node1"
>
> },
>
> "instance_uuid": "VM_ID_0_0"
>
>   },
>
>   {
>
> "host": {
>
>   "host": "Host2",
>
>   "nodename": "Node2"
>
> },
>
> "instance_uuid": "VM_ID_0_1"
>
>   }
>
> ],
>
> [
>
>   {
>
> "host": {
>
>   "host": "Host1",
>
>   "nodename": "Node1"
>
> },
>
> "instance_uuid": "VM_ID_1_0"
>
>   }
>
> ]
>
>   ]
>
> }
>
>
> This placement result can be used by Nova to proceed and complete the
> scheduling.
>
>
> This is where I see the potential for Gantt, which will be a stand alone
> placement decision engine, and can easily accommodate different pluggable
> engines (such as Smart Scheduler
> (https://blueprints.launchpad.net/nova/+spec/solver-scheduler))  to do smart
> placement decisions.
>
>
> Pointers:
> Smart Resource Placement overview:
> https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit?pli=1
> Figure:
> https://docs.google.com/drawings/d/1BgK1q7gl5nkKWy3zLkP1t_SNmjl6nh66S0jHdP0-zbY/edit?pli=1
> Nova Design Session Etherpad:
> https://etherpad.openstack.org/p/NovaIcehouse-Smart-Resource-Placement
> https://etherpad.openstack.org/p/IceHouse-Nova-Scheduler-Sessions
> Smart Scheduler Blueprint:
> https://blueprints.launchpad.net/nova/+spec/solver-scheduler
> Working code: https://github.com/CiscoSystems/nova-solver-scheduler
>
>
> Thanks,
>
> Yathi.
>
>
>
>
>
>
> On 7/14/14, 1:40 PM, "Murray, Paul (HP Cloud)"  wrote:
>
> Hi All,
>
>
>
> I’m sorry I am so late to this lively discussion – it looks a good one! Jay
> has been driving the debate a bit so most of this is in response to his
> comments. But please, anyone should chip in.
>
>
>
> On e

Re: [openstack-dev] [Nova] [Gantt] Scheduler split status (updated)

[Debojyoti Dutta]

https://etherpad.openstack.org/p/SchedulerUseCases

[08:43:35]  #action all update the use case etherpad
athttps://etherpad.openstack.org/p/SchedulerUseCases

Please update your use cases here ..

thx
debo

On Tue, Jul 15, 2014 at 2:50 AM, Sylvain Bauza  wrote:
> Le 14/07/2014 20:10, Jay Pipes a écrit :
>> On 07/14/2014 10:16 AM, Sylvain Bauza wrote:
>>> Le 12/07/2014 06:07, Jay Pipes a écrit :
 On 07/11/2014 07:14 AM, John Garbutt wrote:
> On 10 July 2014 16:59, Sylvain Bauza  wrote:
>> Le 10/07/2014 15:47, Russell Bryant a écrit :
>>> On 07/10/2014 05:06 AM, Sylvain Bauza wrote:
 Hi all,

 === tl;dr: Now that we agree on waiting for the split
 prereqs to be done, we debate on if ResourceTracker should
 be part of the scheduler code and consequently Scheduler
 should expose ResourceTracker APIs so that Nova wouldn't
 own compute nodes resources. I'm proposing to first come
 with RT as Nova resource in Juno and move ResourceTracker
 in Scheduler for K, so we at least merge some patches by
 Juno. ===

 Some debates occured recently about the scheduler split, so
 I think it's important to loop back with you all to see
 where we are and what are the discussions. Again, feel free
 to express your opinions, they are welcome.
>>> Where did this resource tracker discussion come up?  Do you
>>> have any references that I can read to catch up on it?  I
>>> would like to see more detail on the proposal for what should
>>> stay in Nova vs. be moved.  What is the interface between
>>> Nova and the scheduler here?
>>
>> Oh, missed the most important question you asked. So, about
>> the interface in between scheduler and Nova, the original
>> agreed proposal is in the spec
>> https://review.openstack.org/82133 (approved) where the
>> Scheduler exposes : - select_destinations() : for querying the
>> scheduler to provide candidates - update_resource_stats() : for
>> updating the scheduler internal state (ie. HostState)
>>
>> Here, update_resource_stats() is called by the
>> ResourceTracker, see the implementations (in review)
>> https://review.openstack.org/82778 and
>> https://review.openstack.org/104556.
>>
>> The alternative that has just been raised this week is to
>> provide a new interface where ComputeNode claims for resources
>> and frees these resources, so that all the resources are fully
>> owned by the Scheduler. An initial PoC has been raised here
>> https://review.openstack.org/103598 but I tried to see what
>> would be a ResourceTracker proxified by a Scheduler client here
>> : https://review.openstack.org/105747. As the spec hasn't been
>> written, the names of the interfaces are not properly defined
>> but I made a proposal as : - select_destinations() : same as
>> above - usage_claim() : claim a resource amount -
>> usage_update() : update a resource amount - usage_drop(): frees
>> the resource amount
>>
>> Again, this is a dummy proposal, a spec has to written if we
>> consider moving the RT.
>
> While I am not against moving the resource tracker, I feel we
> could move this to Gantt after the core scheduling has been
> moved.

 Big -1 from me on this, John.

 Frankly, I see no urgency whatsoever -- and actually very little
 benefit -- to moving the scheduler out of Nova. The Gantt project I
 think is getting ahead of itself by focusing on a split instead of
 focusing on cleaning up the interfaces between nova-conductor,
 nova-scheduler, and nova-compute.

>>>
>>> -1 on saying there is no urgency. Don't you see the NFV group saying
>>> each meeting what is the status of the scheduler split ?
>>
>> Frankly, I don't think a lot of the NFV use cases are well-defined.
>>
>> Even more frankly, I don't see any benefit to a split-out scheduler to
>> a single NFV use case.
>
> I don't know if you can, but if you're interested in giving feedback to
> the NFV team, we do run weekly meeting on #openstack-meeting-alt every
> Wednesday 2pm UTC.
>
> You can find a list of all the associated blueprints here
> https://wiki.openstack.org/wiki/Teams/NFV#Active_Blueprints whose list
> is processed hourly by a backend script so it generates a Gerrit
> dashboard accessible here : http://nfv.russellbryant.net
>
> By saying that, you can find
> https://blueprints.launchpad.net/nova/+spec/solver-scheduler as a
> possible use-case for NFV.
> As Paul and Yathi said, there is a need for a global resource placement
> engine able to cope with both network and compute resources if we need
> to provide NFV use-cases, that appears to me quite clearly and that's
> why I joined the NFV team.
>
>>
>>> Don't you see each Summit the lots of talks (and people attending
>>> them) talking about how OpenStack should look 

Re: [openstack-dev] [gantt] scheduler group meeting agenda

[Debojyoti Dutta]

https://etherpad.openstack.org/p/SchedulerUseCases

[08:43:35]  #action all update the use case etherpad
athttps://etherpad.openstack.org/p/SchedulerUseCases

Please update your use cases here ..

2014-07-14 20:57 GMT-07:00 Dugger, Donald D :
> 1) Forklift (Tasks & status)
> 2) Opens
>
> --
> Don Dugger
> "Censeo Toto nos in Kansa esse decisse." - D. Gale
> Ph: 303/443-3786
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Schduler] Volunteers wanted for a modest proposal for an external scheduler in our lifetime

[Debojyoti Dutta]

I agree with RussellB on this … if the forklift's goal is to just separate
the scheduler, there should be no new features etc till the forklift is
done and it should work as is with very minor config changes.

A scheduler has several features like place resources correctly, for
example. Ideally, this should be a simple service that can allocate any
resources to any available bucket - balls in bins, VMs in host,
blocks/blobs on disk/SSD etc. Maybe the scheduler should operate on meta
level resource maps for each "type" and delegate the precise decisions to
the allocator for that "type".

debo


On Tue, Dec 3, 2013 at 9:58 AM, Russell Bryant  wrote:

> On 12/03/2013 07:22 AM, Boris Pavlovic wrote:
> > Hi all,
> >
> >
> > Finally found a bit time to write my thoughts.
> >
> > There are few blockers that make really complex to build scheduler as a
> > services or even to move main part of scheduler code to separated lib.
> > We already have one unsuccessfully effort
> > https://blueprints.launchpad.net/oslo/+spec/oslo-scheduler .
> >
> > Major problems that we faced were next:
> > 1) Hard connection with project db api layer (e.g. nova.db.api,
> > cinder.db.api)
> > 2) Hard connection between db.models and host_states
> > 3) Hardcoded host states objects structure
> > 4) There is no namespace support in host states (so we are not able to
> > keep all filters for all projects in the same place)
> > 5) Different API methods, that can't be effectively generalized.
> >
> >
> > Main goals of no-db-scheduler effort are:
> > 1) Make scheduling much faster, storing data locally on each scheduler
> > and just syncing states of them
> > 2) Remove connections between project.db.api and scheduler.db
> > 3) Make host_states just JSON like objects
> > 4) Add namespace support in host_states
> >
> > When this part will be finished, we will have actually only 1 problem
> > what to do with DB API methods, and business logic of each project. What
> > I see is that there are 2 different ways:
>
> If the new project is just a forklift of the existing code that still
> imports nova's db API and accesses Nova's DB, I don't think the initial
> forklift necessarily has to be blocked on completing no-db-scheduler.
> That can happen after just as easily (depending on which effort is ready
> first).
>
> > 1) Make scheduler as a big lib, then implement RPC methods + bit of
> > business logic in each project
> > 2) Move all RPC calls from nova,cinder,ironic,... and business logic in
> > 1 scheduler as a service
>
> Right now I think #2 is the approach we should take.  This is mainly
> because there is common information that is needed for the scheduling
> logic for resources in multiple projects.
>
> --
> Russell Bryant
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
-Debo~
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [scheduler] External scheduler design doc + review

[Debojyoti Dutta]

Hi

I think we should do a review of the design doc review and have rough
consensus (that it should work) followed by running code. ….

As of now all the design stuff is supposedly in (as per the scheduler
meeting today)
https://etherpad.openstack.org/p/icehouse-external-scheduler

Hope the core devs who decide to shepherd will review this and say yay or
nay.

thx
debo
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler meeting and Icehouse Summit

[Debojyoti Dutta]

Hi Phil

Good summary ...

I was wondering if we are shooting for too much to discuss by clubbing
 i) Group Scheduling / Smart Placement / Heat and Scheduling  (1),
(2), (3), & (7)
- How do you schedule something more complex that a single VM ?

I think specifying something more complex than a single VM is a great
theme. But dont know if we can do justice in 1 session. I think maybe
a simple nova scheduling API with groups/bundles of resources  itself
would be a lot for 1 session. In fact in order to specify what you
want in your resources bundle, you would need to think about policies.
So maybe just the simple Nova API and policies might be useful.

Also we might have a session correlating the different models of how
more than 1 VM can be requested - you could start from nova and then
generalize to cross services or you could start from heat workload
models and drill down. There are passionate people on both sides and
maybe that debate needs a session.

I think the smart resource placement is very interesting and might
need at least 1/2 a slot since one can show how it can be done today
in nova and how it can handle cross services scenarios.

See you tomorrow on IRC

debo


On Mon, Oct 14, 2013 at 10:56 AM, Alex Glikson  wrote:
> IMO, the three themes make sense, but I would suggest waiting until the
> submission deadline and discuss at the following IRC meeting on the 22nd.
> Maybe there will be more relevant proposals to consider.
>
> Regards,
> Alex
>
> P.S. I plan to submit a proposal regarding scheduling policies, and maybe
> one more related to theme #1 below
>
>
>
> From:"Day, Phil" 
> To:OpenStack Development Mailing List
> ,
> Date:14/10/2013 06:50 PM
> Subject:Re: [openstack-dev] Scheduler meeting and Icehouse Summit
> 
>
>
>
> Hi Folks,
>
> In the weekly scheduler meeting we've been trying to pull together a
> consolidated list of Summit sessions so that we can find logical groupings
> and make a more structured set of sessions for the limited time available at
> the summit.
>
> https://etherpad.openstack.org/p/IceHouse-Nova-Scheduler-Sessions
>
> With the deadline for sessions being this Thursday 17th, tomorrows IRC
> meeting is the last chance to decide which sessions we want to combine /
> prioritize.Russell has indicated that a starting assumption of three
> scheduler sessions is reasonable, with any extras depending on what else is
> submitted.
>
> I've matched the list on the Either pad to submitted sessions below, and
> added links to any other proposed sessions that look like they are related.
>
>
> 1) Instance Group Model and API
>Session Proposal:
> http://summit.openstack.org/cfp/details/190
>
> 2) Smart Resource Placement:
>   Session Proposal:
> http://summit.openstack.org/cfp/details/33
>Possibly related sessions:  Resource
> optimization service for nova  (http://summit.openstack.org/cfp/details/201)
>
> 3) Heat and Scheduling and Software, Oh My!:
> Session Proposal:
> http://summit.openstack.org/cfp/details/113
>
> 4) Generic Scheduler Metrics and Celiometer:
> Session Proposal:
> http://summit.openstack.org/cfp/details/218
> Possibly related sessions:  Making Ceilometer and Nova play
> nice  http://summit.openstack.org/cfp/details/73
>
> 5) Image Properties and Host Capabilities
> Session Proposal:  NONE
>
> 6) Scheduler Performance:
> Session Proposal:  NONE
> Possibly related Sessions: Rethinking Scheduler Design
> http://summit.openstack.org/cfp/details/34
>
> 7) Scheduling Across Services:
> Session Proposal: NONE
>
> 8) Private Clouds:
> Session Proposal:
> http://summit.openstack.org/cfp/details/228
>
> 9) Multiple Scheduler Policies:
> Session Proposal: NONE
>
>
> The proposal from last weeks meeting was to use the three slots for:
> - Instance Group Model and API   (1)
> - Smart Resource Placement (2)
> - Performance (6)
>
> However, at the moment there doesn't seem to be a session proposed to cover
> the performance work ?
>
> It also seems to me that the Group Model and Smart Placement are pretty
> closely linked along with (3) (which says it wants to combine 1 & 2 into the
> same topic) , so if we only have three slots available then these look like
> logical candidates for consolidating into a single session.That would
> free up a session to cover the generic metrics (4) and Ceilometer - where a
> lot of work in Havana stalled because we couldn't get a consensus on the way
> forward.  The third slot would be kept for performance - which based on the
> lively debate in the scheduler meetings I'm assuming will still be submitted
> as a session.Private Clouds isn't really a scheduler topic, so I suggest
> it takes its 

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Debojyoti Dutta]

Alex, agree with your comments. I think we need to think of both 1.
and 2. as the eventual outcome and the destination. IF we decide to
improve upon scheduling/polices at the heat level, that should be a
very nice and independent endeavor and we can all learn from it. I
dont think we can design this all upfront.

IMO the simple enough road is to do
1. simple resource group extension and show how it can be used to
specify groups of resources - no matter what you do on top of this,
you will need to specify groups of resources (e.g. API proposal from
Yathi/Garyk/Mike/Me)
2. have policies which can be simple scheduler hints for now
3. have some notion of intelligent scheduling (a simple example is a
solver scheduler)
4. have some notion of fast state management (like Boris' proposal)

Ref:
[api] 
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit
[overall] 
https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit

debo

Mike: I agree with most of your ideas of extensions to
heat/policies/ordering/dependencies etc but I wish we could start with
a simple API from the nova side that will grow into a cross services
thing while you could start from the Heat side and then eventually
come to teh same midpoint. I somehow feel we are almost there wrt the
1st cut of the API \cite{api}.

On Wed, Oct 9, 2013 at 11:11 PM, Alex Glikson  wrote:
> Thanks for the pointer -- was not able to attend that meeting,
> unfortunately. Couple of observations, based on what I've heard till now.
> 1. I think it is important not to restrict the discussion to Nova resources.
> So, I like the general direction in [1] to target a generic mechanism and
> API. However, once we start following that path, it becomes more challenging
> to figure out which component should manage those cross-resource constructs
> (Heat sounds like a reasonable candidate -- which seems consistent with the
> proposal at [2]), and what should be the API between it and the services
> deciding on the actual placement of individual resources (nova, cinder,
> neutron).
> 2. Moreover, we should take into account that we may need to take into
> consideration multiple sources of topology -- physical (maybe provided by
> Ironic, affecting availability -- hosts, racks, etc), virtual-compute
> (provided by Nova, affecting resource isolation -- mainly hosts),
> virtual-network (affecting connectivity and bandwidth/latency.. think of SDN
> policies enforcing routing and QoS almost orthogonally to physical
> topology), virtual-storage (affecting VM-to-volume connectivity and
> bandwidth/latency.. think of FC network implying topology different than the
> physical one and the IP network one).
>
> I wonder whether we will be able to come up with a simple-enough initial
> approach & implementation, which would not limit the ability to extend &
> customize it going forward to cover all the above.
>
> Regards,
> Alex
>
> [1]
> https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit
> [2] https://wiki.openstack.org/wiki/Heat/PolicyExtension
>
> 
> Alex Glikson
> Manager, Cloud Operating System Technologies, IBM Haifa Research Lab
> http://w3.haifa.ibm.com/dept/stt/cloud_sys.html |
> http://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml
> Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647
> | Fax: +972-4-8296112
>
>
>
>
> From:Mike Spreitzer 
> To:OpenStack Development Mailing List
> ,
> Date:10/10/2013 07:59 AM
> Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource
> Placement - Updated Instance Group Model and API extension model - WIP Draft
> 
>
>
>
> Yes, there is more than the northbound API to discuss.  Gary started us
> there in the Scheduler chat on Oct 1, when he broke the issues down like
> this:
>
> 11:12:22 AM garyk: 1. a user facing API
> 11:12:41 AM garyk: 2. understanding which resources need to be tracked
> 11:12:48 AM garyk: 3. backend implementation
>
> The full transcript is at
> http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-10-01-15.08.log.html
>
> Alex Glikson  wrote on 10/09/2013 02:14:03 AM:
>>
>> Good summary. I would also add that in A1 the schedulers (e.g., in
>> Nova and Cinder) could talk to each other to coordinate. Besides
>> defining the policy, and the user-facing APIs, I think we should
>> also outline those cross-component APIs (need to think whether they
>> have to be user-visible, or can be admin).
>>
>> Regards,
>> Alex ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.opens

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Debojyoti Dutta]

Mike, I agree we could have a cleaner API but I am not sure how
cleanly it will integrate with current nova which IMO should be test
we should pass (assuming we do cross services later)

On Tue, Oct 8, 2013 at 10:39 PM, Mike Spreitzer  wrote:
> Thanks for the clue about where the request/response bodies are documented.
> Is there any convenient way to view built documentation for Havana right
> now?
>
> You speak repeatedly of the desire for "clean" interfaces, and nobody could
> disagree with such words.  I characterize my desire that way too.  It might
> help me if you elaborate a little on what "clean" means to you.  To me it is
> about minimizing the number of interactions between different modules/agents
> and the amount of information in those interactions.  In short, it is about
> making narrow interfaces - a form of simplicity.
>

I think the word clean can be overloaded. For me a clean API is to use
minimal nouns and specify the policies, the resources we would like to
request and the extra metadata that we might want to pass. Thus the
three components.

> To me the most frustrating aspect of this challenge is the need for the
> client to directly mediate the dependencies between resources; this is
> really what is driving us to do ugly things.  As I mentioned before, I am
> coming from a setting that does not have this problem.  So I am thinking
> about two alternatives: (A1) how clean can we make a system in which the
> client continues to directly mediate dependencies between resources, and
> (A2) how easily and cleanly can we make that problem go away.

Am a little confused - How is the API dictating either A1 or A2? Isnt
that a function of the implementation of the API. For a moment let us
assume that the black box implementation will be awesome and address
your concerns. The question is this - does the current API help
specify what we  want assuming we will be able to extend the notion of
nodes, edges, policies and metadata?

debo

>
> For A1, we need the client to make a distinct activation call for each
> resource.  You have said that we should start the roadmap without joint
> scheduling; in this case, the scheduling can continue to be done
> independently for each resource and can be bundled with the activation call.
> That can be the call we know and love today, the one that creates a
> resource, except that it needs to be augmented to also carry some pointer
> that points into the policy data so that the relevant policy data can be
> taken into account when making the scheduling decision.  Ergo, the client
> needs to know this pointer value for each resource.  The simplest approach
> would be to let that pointer be the combination of (p1) a VRT's UUID and
> (p2) the local name for the resource within the VRT.  Other alternatives are
> possible, but require more bookkeeping by the client.
>
> I think that at the first step of the roadmap for A1, the client/service
> interaction for CREATE can be in just two phases.  In the first phase the
> client presents a topology (top-level InstanceGroup in your terminology),
> including resource definitions, to the new API for registration; the
> response is a UUID for that registered top-level group.  In the second phase
> the client "creates" the resources as is done today, except that each
> creation call is augmented to carry the aforementioned pointer into the
> policy information.  Each resource scheduler (just nova, at first) can use
> that pointer to access the relevant policy information and take it into
> account when scheduling.  The client/service interaction for UPDATE would be
> in the same two phases: first update the policy&resource definitions at the
> new API, then do the individual resource updates in dependency order.
>
> I suppose the second step in the roadmap is to have Nova do joint
> scheduling.  The client/service interaction pattern can stay the same.  The
> only difference is that Nova makes the scheduling decisions in the first
> phase rather than the second.  But that is not a detail exposed to the
> clients.
>
> Maybe the third step is to generalize beyond nova?
>
> For A2, the first question is how to remove "user-level" create-time
> dependencies between resources.  We are only concerned with the "user-level"
> create-time dependencies here because it is only they that drive intimate
> client interactions.  There are also create-time dependencies due to the
> nature of the resource APIs; for example, you can not attach a volume to a
> VM until after both have been created.  But handling those kinds of
> create-time dependencies does not require intimate interactions with the
> client.  I know of two software orchestration technologies developed in IBM,
> and both have the property that there are no "user-level" create-time
> dependencies between resources; rather, the startup code ("userdata") that
> each VM runs handles dependencies (using a library for cross-VM
> communication and synchronization).  This can even 

Re: [openstack-dev] Curvature and Donabe repos are now public!

[Debojyoti Dutta]

Hi Christopher,

Thanks for your comments!

We would definitely like to have some resources for maintaining it for
now. Also a lot depends on how the community reacts to this and what
features they demand.

Personally I hope its not just reference code :)

debo

On Thu, Oct 3, 2013 at 12:20 PM, Christopher Armstrong
 wrote:
> On Thu, Oct 3, 2013 at 1:43 PM, Debojyoti Dutta  wrote:
>>
>> Hi!
>>
>> We @Cisco just made the following repos public
>> https://github.com/CiscoSystems/donabe
>> https://github.com/CiscoSystems/curvature
>>
>> Donabe was pitched as a recursive container before Heat days.
>> Curvature is an alternative interactive GUI front end to openstack
>> that can handle virtual resources, templates and can instantiate
>> Donabe workloads. The D3 + JS stuff was incorporated into Horizon. A
>> short demo was shown last summit and can be found at
>>
>> http://www.openstack.org/summit/portland-2013/session-videos/presentation/interactive-visual-orchestration-with-curvature-and-donabe
>>
>> Congrats to the primary developers: @CaffeinatedBrad @John_R_Davidge
>> @Tehsmash_ @JackPeterFletch ... Special thanks to @lewtucker for
>> supporting this.
>>
>> Hope this leads to more cool stuff for the Openstack community!
>>
>
>
> Congrats! I'm glad you guys finally released the code :)
>
> Does Cisco (or anyone else) plan to continue to put development resources
> into these projects, or should we basically view them as reference code for
> solving particular problems?
>
> Thanks,
>
>
> --
> IRC: radix
> Christopher Armstrong
> Rackspace
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Curvature and Donabe repos are now public!

[Debojyoti Dutta]

Edgar

Thanks for your feedback during its early days ;)

debo

On Thu, Oct 3, 2013 at 12:34 PM, Edgar Magana  wrote:
> Debo,
>
> Congratulations on this move! The entire Cisco team is doing an awesome
> work around OpenStack.
>
> Cheers,
>
> Edgar
>
> On 10/3/13 11:43 AM, "Debojyoti Dutta"  wrote:
>
>>Hi!
>>
>>We @Cisco just made the following repos public
>>https://github.com/CiscoSystems/donabe
>>https://github.com/CiscoSystems/curvature
>>
>>Donabe was pitched as a recursive container before Heat days.
>>Curvature is an alternative interactive GUI front end to openstack
>>that can handle virtual resources, templates and can instantiate
>>Donabe workloads. The D3 + JS stuff was incorporated into Horizon. A
>>short demo was shown last summit and can be found at
>>http://www.openstack.org/summit/portland-2013/session-videos/presentation/
>>interactive-visual-orchestration-with-curvature-and-donabe
>>
>>Congrats to the primary developers: @CaffeinatedBrad @John_R_Davidge
>>@Tehsmash_ @JackPeterFletch ... Special thanks to @lewtucker for
>>supporting this.
>>
>>Hope this leads to more cool stuff for the Openstack community!
>>
>>--
>>-Debo~
>>
>>___
>>OpenStack-dev mailing list
>>OpenStack-dev@lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Curvature and Donabe repos are now public!

[Debojyoti Dutta]

And just for some history on Donabe: the idea was 1st presented at the
E summit along with @dendrobates at
http://www.slideshare.net/ddutta1/donabe-models-openstack-essex-summit

debo

On Thu, Oct 3, 2013 at 11:43 AM, Debojyoti Dutta  wrote:
> Hi!
>
> We @Cisco just made the following repos public
> https://github.com/CiscoSystems/donabe
> https://github.com/CiscoSystems/curvature
>
> Donabe was pitched as a recursive container before Heat days.
> Curvature is an alternative interactive GUI front end to openstack
> that can handle virtual resources, templates and can instantiate
> Donabe workloads. The D3 + JS stuff was incorporated into Horizon. A
> short demo was shown last summit and can be found at
> http://www.openstack.org/summit/portland-2013/session-videos/presentation/interactive-visual-orchestration-with-curvature-and-donabe
>
> Congrats to the primary developers: @CaffeinatedBrad @John_R_Davidge
> @Tehsmash_ @JackPeterFletch ... Special thanks to @lewtucker for
> supporting this.
>
> Hope this leads to more cool stuff for the Openstack community!
>
> --
> -Debo~



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Curvature and Donabe repos are now public!

[Debojyoti Dutta]

Hi!

We @Cisco just made the following repos public
https://github.com/CiscoSystems/donabe
https://github.com/CiscoSystems/curvature

Donabe was pitched as a recursive container before Heat days.
Curvature is an alternative interactive GUI front end to openstack
that can handle virtual resources, templates and can instantiate
Donabe workloads. The D3 + JS stuff was incorporated into Horizon. A
short demo was shown last summit and can be found at
http://www.openstack.org/summit/portland-2013/session-videos/presentation/interactive-visual-orchestration-with-curvature-and-donabe

Congrats to the primary developers: @CaffeinatedBrad @John_R_Davidge
@Tehsmash_ @JackPeterFletch ... Special thanks to @lewtucker for
supporting this.

Hope this leads to more cool stuff for the Openstack community!

-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Debojyoti Dutta]

Hi Mike

We understand your point, and, academically, we agree to a large
extent. We aware of the optimization results for holistic placement
etc (and how any realistic formulation is hard). When we look at the
scheduler today, it does one thing at a time because it was meant to
be pragmatic when it was built.

Where we want to go is to move gradually to a point where some entity
knows 1) what to place where and 2) what order to place holistically
so we are saying the same thing!

Now how do we go there 'gradually'? A few of us decided that we can
1st do scheduling for a group of instance and then gradually extend it
to the kind of things you are mentioning.

My $0.02: A possible next step is to 1st tackle the problem of
intelligent placement as in 1) and leave 2) for later or the heat guys
since 1) is independent of 1). Since Openstack is a framework, it
might be useful to define a simple extensible API (least common
denominator) since everyone will have their opinion on how to do
define this (and extend it). We can use the scheduler subgroup to do
this.

We would ideally like to define the API that is cross services
compliant but works with the current Nova so that we can incrementally
build this out. In the final version, the API needs to specify
compute, network, storage, policies etc. I think templates might be
more heat or heat-like service specific.

Debo


On Wed, Sep 25, 2013 at 12:57 PM, Mike Spreitzer  wrote:
> Debo, Yathi: I have read
> https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit?pli=1
> and most of the referenced materials, and I have a couple of big-picture
> questions.  That document talks about making Nova call out to something that
> makes the sort of smart decisions you and I favor.  As far as I know, Nova
> is still scheduling one thing at a time.  How does that smart decision maker
> get a look at the whole pattern/termplate/topology as soon as it is needed?
> I think you intend the smart guy gets it first, before Nova starts getting
> individual VM calls, right?  How does this picture grow to the point where
> the smart guy is making joint decisions about compute, storage, and network?
> I think the key idea has to be that the smart guy gets a look at the whole
> problem first, and makes its decisions, before any individual resources are
> requested from nova/cinder/neutron/etc.  I think your point about
> "non-disruptive, works with the current nova architecture" is about solving
> the problem of how the smart guy's decisions get into nova.  Presumably this
> problem will occur for cinder and so on, too.  Have I got this right?
>
> There is another way, right?  Today Nova accepts an 'availability zone'
> argument whose value can specify a particular host.  I am not sure about
> Cinder, but you can abuse volume types to get this job done.
>
> Thanks,
> Mike
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Debojyoti Dutta]

Joining the party late :)

I think there have been a lot of interesting ideas around wholistic
scheduling over the last few summits. However seems there is no clear
agreement on 1) where it should be specified and implemented 2) what
the specifications look like - VRT, policies, templates etc etc etc 3)
how to scale such implementations given the complexity.

In order to tackle the seemingly complex problem, why dont we get
together during the summit (and do homework beforehand) and converge
upon the above independent of where we implement. Maybe we implement
it in a separate scheduling layer which is independent of existing
services so that it touches less things.

Mike: agree that we should be specifying a VRT, constraints etc and
matching them via a constrained optimization ... all this is music to
my ears. However I feel we will just make it harder to get it done if
we dont simplify the problem without giving up room for where we want
to be. At the end of the day, in order to place resources efficiently
you need a demand vector/matrix etc and you match with the available
resource vector/matrix  so why not have an abstract resource
placement layer that hides details except for the quantities that are
really needed. That way it can be used inside Heat or it can be used
standalone 

To that effect - if you look at the BP
https://blueprints.launchpad.net/nova/+spec/solver-scheduler, and the
associated code (WIP), its an attempt to do the same within the
current Nova framework. One thing we could do is to have a layer that
abstracts the VRT and constraints so that a simple optimization
framework could then make the decisions instead of hand crafted
algorithms that are harder to extend (e.g. the entire suite of
scheduler filters that exist today).

debo

On Tue, Sep 24, 2013 at 7:01 AM, Zane Bitter  wrote:
> On 24/09/13 05:31, Mike Spreitzer wrote:
>>
>> I was not trying to raise issues of geographic dispersion and other
>> higher level structures, I think the issues I am trying to raise are
>> relevant even without them.  This is not to deny the importance, or
>> relevance, of higher levels of structure.  But I would like to first
>> respond to the discussion that I think is relevant even without them.
>>
>> I think it is valuable for OpenStack to have a place for holistic
>> infrastructure scheduling.  I am not the only one to argue for this, but
>> I will give some use cases.  Consider Hadoop, which stresses the path
>> between Compute and Block Storage.  In the usual way of deploying and
>> configuring Hadoop, you want each data node to be using directly
>> attached storage.  You could address this by scheduling one of those two
>> services first, and then the second with constraints from the first ---
>> but the decisions made by the first could paint the second into a
>> corner.  It is better to be able to schedule both jointly.  Also
>> consider another approach to Hadoop, in which the block storage is
>> provided by a bank of storage appliances that is equidistant (in
>> networking terms) from all the Compute.  In this case the Storage and
>> Compute scheduling decisions have no strong interaction --- but the
>> Compute scheduling can interact with the network (you do not want to
>> place Compute in a way that overloads part of the network).
>
>
> Thanks for writing this up, it's very helpful for figuring out what you mean
> by a 'holistic' scheduler.
>
> I don't yet see how this could be considered in-scope for the Orchestration
> program, which uses only the public APIs of other services.
>
> To take the first example, wouldn't your holistic scheduler effectively have
> to reserve a compute instance and some directly attached block storage prior
> to actually creating them? Have you considered Climate rather than Heat as
> an integration point?
>
>
>> Once a holistic infrastructure scheduler has made its decisions, there
>> is then a need for infrastructure orchestration.  The infrastructure
>> orchestration function is logically downstream from holistic scheduling.
>
>
> I agree that it's necessarily 'downstream' (in the sense of happening
> afterwards). I'd hesitate to use the word 'logically', since I think by it's
> very nature a holistic scheduler introduces dependencies between services
> that were intended to be _logically_ independent.
>
>
>>   I do not favor creating a new and alternate way of doing
>> infrastructure orchestration in this position.  Rather I think it makes
>> sense to use essentially today's heat engine.
>>
>> Today Heat is the only thing that takes a holistic view of
>> patterns/topologies/templates, and there are various pressures to expand
>> the mission of Heat.  A marquee expansion is to take on software
>> orchestration.  I think holistic infrastructure scheduling should be
>> downstream from the preparatory stage of software orchestration (the
>> other stage of software orchestration is the run-time action in and
>> supporting the resources themselves).  The

[openstack-dev] [Scheduler] API extension for instance groups

[Debojyoti Dutta]

Hi Folks

Sorry for missing today's meeting. Was reading the logs
 
http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-09-24-15.03.log.html

Regarding the API: the instance groups API extension could be used as
a starting point although it could be refined. Right now its quite
simple and abstract. For example one can pass arbitrary policy
objects.

Should we use the next meeting to discuss what we need in a placement
API. In my mind, we need the following:
* policy description to encapsulate the rules? you can pass a bunch of
policy objects and be done
* description of the resources to be placed. Possibly a virtual
topology? For now the instance group API extension can be passed a
topology instead of a list of instances ... voila!
* do we need patterns etc? That should be done by a higher level API like Heat

My thesis: if we improve the instance group API extension a bit, we
should be able to cover the cases discussed in today's meeting. Happy
to do a deep dive next week!

-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FFE Request: bp/instance-group-api-extension

[Debojyoti Dutta]

+1 Gary

If this was allowed in H2 and not stopped due to the mandatory
requirement to use API objects (then), this feature would have
definitely seen the light of day (and we would have improved it to use
the objects/v3api anyway). This API leads to a bunch of new features
that make Openstack even better in terms of resource allocation,
scheduling, policies etc. Hence a little sad.

Hopefully we will have better luck next time :) Thanks to all the
reviewers esp Chris Yeoh, Dan Smith, Alex Xu and Russell Bryant for
their time. Thanks Thierry for trying to help with the FFE!

debo

On Fri, Sep 6, 2013 at 9:09 AM, Gary Kotton  wrote:
>
>
> On 9/6/13 5:07 PM, "Russell Bryant"  wrote:
>
>>On 09/06/2013 05:06 AM, Thierry Carrez wrote:
>>> Debojyoti Dutta wrote:
>>>> As per my IRC chats with dansmith, russellb, this feature needs the
>>>> user auth checks (being taken care of in
>>>> https://bugs.launchpad.net/nova/+bug/1221396).
>>>>
>>>> Dan has some more comments 
>>>>
>>>> Could we please do a FFE for this one  has been waiting for a long
>>>> time and we have done all that we were asked relatively quickly 
>>>> in H2 it was gated by API object refactor. Most of the current
>>>> comments (pre last one by Dan) are due to
>>>> https://bugs.launchpad.net/nova/+bug/1221396
>>>
>>> This sounds relatively self-contained and ready. If you can get it
>>> merged (with the additional bugfix) before the release meeting Tuesday,
>>> personally I'm fine with it.
>>>
>>
>>I originally was fine with a FFE for this.  However, after some
>>discussion yesterday, it seems there are quite a few comments on the
>>review.  There's more iterating to do on this, so I don't have a high
>>confidence that we can merge it in time.
>>
>>I would hate to grant the FFE, have you work super hard to iterate on
>>this quickly, and it still not make it in time.  I think at this point
>>we should just defer to Icehouse to ensure that there is proper time to
>>do the updates and get good review on them.
>
> I am a little sad that this not been granted an FFE. We worked really hard
> on this. To be honest it was ready for review at the end of H2. I guess
> that our goal will now be to get this valuable feature in beginning of the
> Icehouse release.
>
> Thanks
> Gary
>
>>
>>NACK on the FFE.
>>
>>Thanks,
>>
>>--
>>Russell Bryant
>>
>>___
>>OpenStack-dev mailing list
>>OpenStack-dev@lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] FFE Request: bp/instance-group-api-extension

[Debojyoti Dutta]

Hi

As per my IRC chats with dansmith, russellb, this feature needs the
user auth checks (being taken care of in
https://bugs.launchpad.net/nova/+bug/1221396).

Dan has some more comments 

Could we please do a FFE for this one  has been waiting for a long
time and we have done all that we were asked relatively quickly 
in H2 it was gated by API object refactor. Most of the current
comments (pre last one by Dan) are due to
https://bugs.launchpad.net/nova/+bug/1221396

-- 
-Debo~

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev