Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey

[Mike Spreitzer]

Clint Byrum  wrote on 10/30/2013 01:42:53 PM:
> ...
> 
> The engine should store _all_ of its state in a distributed data store
> of some kind. Any engine should be aware of what is already happening
> with the stack from this state and act accordingly. That includes the
> engine currently working on actions. When viewed through this lense,
> to me, locking is a poor excuse for serializing the state of the engine
> scheduler.

I agree.  I reached a similar conclusion this spring when thinking through 
the multi-engine issue for my group's work.

> It feels like TaskFlow is the answer, with an eye for making sure
> TaskFlow can be made to work with distributed state. I am not well
> versed on TaskFlow's details though, so I may be wrong. It worries me
> that TaskFlow has existed a while and doesn't seem to be solving real
> problems, but maybe I'm wrong and it is actually in use already.

As Zane pointed out when I asked, there is a difference between 
orchestration and workflow: orchestration is about bringing two things 
into alignment, and can adapt to changes in one or both along the way; I 
think of a workflow as an ossified orchestration, it is not so inherently 
adaptable (unless you write a workflow that is really an orchestrator). 
Adaptability gets even more important when you add more agents driving 
change into the system.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

Alex Glikson  wrote on 10/30/2013 02:26:08 AM:

> Mike Spreitzer  wrote on 30/10/2013 06:11:04 AM:
> > Date: 30/10/2013 06:12 AM 
> > 
> > Alex also wrote: 
> > ``I wonder whether it is possible to find an approach that takes 
> > into account cross-resource placement considerations (VM-to-VM 
> > communicating over the application network, or VM-to-volume 
> > communicating over storage network), but does not require delivering
> > all the intimate details of the entire environment to a single place
> > -- which probably can not be either of Nova/Cinder/Neutron/etc.. but
> > can we still use the individual schedulers in each of them with 
> > partial view of the environment to drive a placement decision which 
> > is consistently better than random?'' 
> > 
> > I think you could create a cross-scheduler protocol that would 
> > accomplish joint placement decision making --- but would not want 
> > to.  It would involve a lot of communication, and the subject matter
> > of that communication would be most of what you need in a 
> > centralized placement solver anyway.  You do not need "all the 
> > intimate details", just the bits that are essential to making the 
> > placement decision. 
> 
> Amount of communication depends on the protocol, and what exactly 
> needs to be shared.. Maybe there is a range of options here that we 
> can potentially explore, between what exists today (Heat talking to 
> each of the components, retrieving local information about 
> availability zones, flavors and volume types, existing resources, 
> etc, and communicates back with scheduler hints), and having a 
> centralized DB that keeps the entire data model. 
> Also, maybe different points on the continuum between 'share few' 
> and 'share a lot' would be a good match for different kinds of 
> environments and different kinds of workload mix (for example, as 
> you pointed out, in an environment with flat network and centralized
> storage, the sharing can be rather minimal). 

I'm not going to claim the direction you're heading is impossible, I am 
not good at impossibility proofs.  But I do wonder about the why of it. 
This came up in the context of the issues around the fact that 
orchestration is downstream from joint decision making.  Even if that 
joint decision making is done in a distributed way, orchestration will 
still be downstream from it.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-dev][Nova][Discussion]Blueprint : Auto VM Discovery in OpenStack for existing workload

[Mike Spreitzer]

"Daniel P. Berrange"  wrote on 10/30/2013 12:38:21 
PM:
> ... This avoids any need to deal with the hard problems of adopting
> arbitrary VMs.

Sorry, my writing was unclear.  I did not mean another use case for 
adopting VMs; I mean another use case for VM discovery.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-dev][Nova][Discussion]Blueprint : Auto VM Discovery in OpenStack for existing workload

[Mike Spreitzer]

Swapnil Kulkarni  wrote on 10/30/2013 
02:36:37 AM:

> I had a discussion with russellb regarding this for yesterday, I 
> would like to discuss this with the team regarding the blueprint 
> mentioned in subject.
> 
> 
https://blueprints.launchpad.net/nova/+spec/auto-vm-discovery-on-hypervisor

> 
> Description: Organizations opting to use openstack can have varied 
> amount of workload that they would like to be available directly 
> with the use of some discovery workflows. One common usage of this 
> would be exising virtual machines present on the hypervisors. If 
> this instances can be disovered by the compute agent during 
> discovery, it would help to use Openstack to manage the existing 
> workload directly. Auto VM Discovery will enable this functionality 
> initially for KVM guests, the widely used hypervisor configuration 
> in OpenStack deployments and enhance it further for other hypervisors.

I agree with the doubts about asking Nova to adopt non-Nova VMs.  However, 
there is another use case that may make more sense.  I am told that 
sometimes a VM creation or deletion operation will fail partially, in such 
a way that the controller thinks the VM does not exist but the hypervisor 
is still reserving some resources for that VM.  If Nova had a way to check 
its facts against the hypervisors, confusion would not have to reign.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

Following is my reaction to the last few hours of discussion.

Russell Bryant wrote "Nova calling heat to orchestrate Nova seems 
fundamentally wrong".  I am not totally happy about this either, but would 
you be OK with Nova orchestrating Nova?  To me, that seems worse --- 
duplicating functionality we already have in Heat.  The way I see it, we 
have to decide how cope with the inescapable fact that orchestration is 
downstream from joint decision making.  I see no better choices than: (1) 
a 1-stage API in which the client presents the whole top-level group and 
is done, or (2) a 2-stage API in which the client first presents the whole 
top-level group and second proceeds to orchestrate the creations of the 
resources in that group.  BTW, when we go holistic, (1) will look less 
offensive: there will be a holistic infrastructure scheduler doing the 
joint decision making first, not one of the individual services, and that 
is followed by orchestration of the individual resources.  If we took Alex 
Glikson's suggestion and started holistic, we would not be so upset on 
this issue.

Alex also wrote:
``I wonder whether it is possible to find an approach that takes into 
account cross-resource placement considerations (VM-to-VM communicating 
over the application network, or VM-to-volume communicating over storage 
network), but does not require delivering all the intimate details of the 
entire environment to a single place -- which probably can not be either 
of Nova/Cinder/Neutron/etc.. but can we still use the individual 
schedulers in each of them with partial view of the environment to drive a 
placement decision which is consistently better than random?''

I think you could create a cross-scheduler protocol that would accomplish 
joint placement decision making --- but would not want to.  It would 
involve a lot of communication, and the subject matter of that 
communication would be most of what you need in a centralized placement 
solver anyway.  You do not need "all the intimate details", just the bits 
that are essential to making the placement decision.

Reacting to Andrew Lasky's note, Chris Friesen noted:
``As soon as we start trying to do placement logic outside of Nova it 
becomes trickier to deal with race conditions when competing against 
other API users trying to acquire resources at the same time.''

I have two reactions.  The simpler one is: we can avoid this problem if we 
simply route all placement problems (either all placement problems for 
Compute, or all placement problems for a larger set of services) though 
one thing that decides and commits allocations.  My other reaction is: we 
will probably want multi-engine.  That is, the option to run several 
placement solvers concurrently --- with optimistic concurrency control. 
That presents essentially the same problem as Chris noted.  As Yathi noted 
in one of his responses, this can be handled by appropriate implementation 
structure.  In the spring I worked out a multi-engine design for my 
group's old code.  The conclusion I reached is that after a placement 
engine finds a solution, you want an essentially ACID transaction that (1) 
checks that the solution is still valid and, if so, (2) makes the 
allocations in that solution.

Yathi wrote that the 2-stage API creates race conditions, but I do not see 
that.  As we are starting with Nova only, in the first of the two stages 
Nova can both decide and commit the allocations in one transaction; the 
second stage just picks up and uses the allocations made in the first 
stage.

Alex Glikson asked why not go directly to holistic if there is no value in 
doing Nova-only.  Yathi replied to that concern, and let me add some 
notes.  I think there *are* scenarios in which doing Nova-only joint 
policy-based scheduling is advantageous.  For example, if the storage is 
in SAN or NAS then you do not have a strong interaction between scheduling 
compute and storage so you do not need holistic scheduling to get good 
availability.  I know some organizations build their datacenters that way, 
with full cross-sectional bandwidth between the compute and storage, 
because (among other things) it makes that simplification.  Another thing 
that can be done with joint policy-based scheduling is minimize license 
costs for certain IBM software.  That software is licensed based on how 
many cores the software has access to, so in a situation with 
hyperthreading or overcommitment the license cost can depend on how the VM 
instances are arranged among hosts.

Yathi replied to Khanh-Toan's remark about edge policies, but I suspect 
there was a misunderstanding.  I think the critique concerns this part of 
the input:

  "policies" : [ {
"edge" : "http-app-edge-1",
"policy_uuid" : "some-policy-uuid-2",
"type" : "edge",
"policy_id" : 3
  } ],
  "edges" : [ {
"r_member" : "app-server-group-1",
"l_member" : "http-server-group-1",
"name" : "http-app-edge-1"
  } ],

That is, t

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

I should clarify my comment about invoking Heat to do the orchestration. I 
think we have a choice between designing a 1-stage API vs a 2-stage API. 
The 2-stage API goes like this: first the client defines the top-level 
group and everything inside it, then the client makes more calls to create 
the resources (with reference to the groups and policies).  In the 2-stage 
API, there is no need to use Heat for orchestration --- the client can 
orchestrate in any way it wants (BTW, we should eventually get around to 
talking about what happens if the client is the heat engine).  In the 
1-stage API, the client just makes one call to define the top-level group 
and everything inside it; the implementation takes care of all the rest; 
in this style of API, I think it would be natural for the implementation 
to call Heat to do the orchestration.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

John Garbutt  wrote on 10/29/2013 07:29:19 AM:
> ...
> Its looking good, but I was thinking about a slightly different 
approach:
> 
> * I would like to see instance groups be used to describe all
> scheduler hints (including, please run on cell X, or please run on
> hypervisor Y)

I think Yathi's proposal is open in the sense that any type of policy can 
appear (we only have to define the policy types :-).  Removing old 
features from the existing API is something that would have to be done 
over time, if at all.

> * passing old scheduler hints to the API will just create a new
> instance group to persist the request

Yes, implementation re-org is easier that retiring old API.

> * ensure live-migrate/migrate never lets you violate the rules in the
> user hints, at least don't allow it to happen by accident

Right, that's why we are persisting the policy information.

> * I was expecting to see hard and soft constraints/hints, like: try
> keep in same switch, but make sure on separate servers

Good point, I forgot to mention that in my earlier reviews of the model!

> * Would be nice to have admin defined global options, like: "ensure
> tenant does note have two servers on the same hypervisor" or soft

That's the second time I have seen that idea in a week, there might be 
something to it.

> * I expected to see the existing boot server command simply have the
> addition of a reference to a group, keeping the existing methods of
> specifying multiple instances

That was my expectation too, for how a 2-stage API would work.  (A 1-stage 
API would not have the client making distinct calls to create the 
instances.)

> * I aggree you can't change a group's spec once you have started some
> VMs in that group, but you could then simply launch more VMs keeping
> to the same policy

Not if a joint decision was already made based on the totality of the 
group.

> ...
> 
> * augment the server details (and group?) with more location
> information saying where the scheduler actually put things, obfuscated
> on per tenant basis. So imagine nova, cinder, neutron exposing ordered
> (arbitrary tagged) location metadata like nova: (("host_id", "foo"),
> ("switch_group_id": "bar"), ("power_group": "bas"))

+1

> * the above should help us define the "scope" of a constraint relative
> to either a nova, cinder or neutron resource.

I am lost.  What "above", what scope definition problem?

> * Consider a constraint that includes constraints about groups, like
> must be separate to group X, in the scope of the switch, or something
> like that

I think Yathi's proposal, with the policy types I suggested, already does 
a lot of stuff like that.  But I do not know what you mean by "in the 
scope of the switch".  I think you mean a location constraint, but am not 
sure which switch you have in mind.  I would approach this perhaps a 
little more abstractly, as a collocation constraint between two resources 
that are known to and meaningful to the client (yes, we are starting with 
Nova only in Icehouse, hope to go holistic later).

> * Need more thought on constraints between volumes, servers and
> networks, I don't think edges are the right way to state that, I think
> it would be better as a cross group constraint, where the scope of the
> constraint is related to neutron.

I need more explanation or concrete examples to understand what problem(s) 
you are thinking of.  We are explicitly limiting ourselves to Nova at 
first, later will add in other services.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

Khanh-Toan Tran  wrote on 10/29/2013 
09:10:00 AM:
> ...
> 1) Member of a group is recursive. A member can be group or an 
> instance. In this case there are two different declaration formats 
> for members, as with http-server-group-1 ("name, "policy", "edge") 
> and Http-Server-1 ("name", "request_spec", "type"). Would it be 
> better if group-typed member also have "type" field to better 
> interpret the member? Like policy which has "type" field to declare 
> that's a egde-typed policy or group-typed policy.

I have no strong opinion on this.

> 2) The "edge" is not clear to me. It seems to me that "edge" is just
> a place holder for the edge policy. Does it have some particular 
> configuration like group members (e.g. group-typed member is 
> described by its "member","edge" and "policy", while instance-typed 
> member is described by its "request_spec") ?

Yes, an edge is just a way to apply a policy to an ordered pair of groups.

> 3) Members & groups have policy declaration nested in them. Why is 
> edge-policy is declared outside of edge's declaration?

I agree, it would be more natural to write an edge's policy references 
inside the edge object itself.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

Alex Glikson  wrote on 10/29/2013 03:37:41 AM:

> 1. I assume that the motivation for rack-level anti-affinity is to 
> survive a rack failure. Is this indeed the case? 
> This is a very interesting and important scenario, but I am curious 
> about your assumptions regarding all the other OpenStack resources 
> and services in this respect.

Remember we are just starting on the roadmap.  Nova in Icehouse, holistic 
later.

> 2. What exactly do you mean by "network reachibility" between the 
> two groups? Remember that we are in Nova (at least for now), so we 
> don't have much visibility to the topology of the physical or 
> virtual networks. Do you have some concrete thoughts on how such 
> policy can be enforced, in presence of potentially complex 
> environment managed by Neutron?

I am aiming for the holistic future, and Yathi copied that from an example 
I drew with the holistic future in mind.  While we are only addressing 
Nova, I think a network reachability policy is inapproprite.

> 3. The JSON somewhat reminds me the interface of Heat, and I would 
> assume that certain capabilities that would be required to implement
> it would be similar too. What is the proposed approach to 
> 'harmonize' between the two, in environments that include Heat? What
> would be end-to-end flow? For example, who would do the 
> orchestration of individual provisioning steps? Would "create" 
> operation delegate back to Heat for that? Also, how other 
> relationships managed by Heat (e.g., links to storage and network) 
> would be incorporated in such an end-to-end scenario? 

You raised a few interesting issues.

1. Heat already has a way to specify resources, I do not see why we should 
invent another.

2. Should Nova call Heat to do the orchestration?  I would like to see an 
example where ordering is an issue.  IMHO, since OpenStack already has a 
solution for creating resources in the right order, I do not see why we 
should invent another.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][scheduler] Instance Group Model and APIs - Updated document with an example request payload

[Mike Spreitzer]

"Yathiraj Udupi (yudupi)"  wrote on 10/29/2013 02:46:30 
AM:

> The Instance Group API document is now updated with a simple example
> request payload of a nested group, and some description of how the 
> API implementation should handle the registration of the components 
> of a nested instance group. 
> https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit 

Thanks!  I tried viewing that JSON by copying into a file and using the 
JSONView extension to FireFox, but it complained about the JSON being 
malformed.  So I used Jackson to parse, and it found some odd closing 
quotes.  After fixing them I used Jackson to write out a normalized 
version (I find the indenting in the original odd).  I have attached the 
result (hope it makes it through the mailing list).



Thanks,
Mike

2tier-normalized.json
Description: Binary data
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config

[Mike Spreitzer]

Steve Baker  wrote on 10/28/2013 04:24:30 PM:

> On 10/29/2013 02:53 AM, Steven Hardy wrote:
> > ...
> > Can anyone provide me with a clear argument for what the "fundamental
> > differences" actually are?
> ...
> Since writing those proposals my thinking has evolved too. I'm currently
> thinking it would be best to implement software configuration resources
> rather than create a new component construct.

Please pardon the newbie question, but I do not understand.  A resource 
type is implemented in OpenStack code --- a part of Heat that calls a 
fixed service API that expects Keystone credentials.  A component is 
implemented by a bit of user code (and/or other sorts of instructions) 
embedded in or referenced by a template, with no fixed API and not invoked 
with Keystone credentials.  We desire the heat engine to invoke operations 
on resources; we do not desire the heat engine to invoke components (the 
VMs do that themselves, via whatever bootstrapping mechanism is used).  So 
yes, I do see fundamental differences.  What am I missing?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Network topologies [and more]

[Mike Spreitzer]

Zane Bitter  wrote on 10/28/2013 06:47:50 AM:
> On 27/10/13 16:37, Edgar Magana wrote:
> > Heat Developers,
> >
> > I am one of the core developers for Neutron who is lately working on 
the
> > concept of "Network Topologies". I want to discuss with you if the
> > following blueprint will make sense to have in heat or neutron code:
> > https://blueprints.launchpad.net/neutron/+spec/network-topologies-api
> >
> > ...
> 
> It sounds to me like the only thing there that Heat is not already doing 

> is to dump the existing network configuration. What if you were to 
> implement just that part and do it in the format of a Heat template? (An 

> independent tool to convert the JSON output to a Heat template would 
> also work, I guess.)
> 
> ...
> 
> It does sound very much like you're trying to solve the same problem as 
> Heat.
> 

In my templates I have more than a network topology.  How would I combine 
the extracted/shared network topology with the other stuff I want in my 
heat template?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

"Yathiraj Udupi (yudupi)"  wrote on 10/15/2013 03:08:32 
AM:

> I have made some edits to the document: https://docs.google.com/
> document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?pli=1# 
> ...

One other minor thing to discuss in the modeling is metadata.  I am not 
eager to totally gorp up the model, but shouldn't all sorts of things 
allow metadata?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

Steve Baker  wrote on 10/15/2013 06:48:53 PM:

> I've just written some proposals to address Heat's HOT software 
> configuration needs, and I'd like to use this thread to get some 
feedback:
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config
> 
https://wiki.openstack.org/wiki/Heat/Blueprints/native-tools-bootstrap-config

> 
> Please read the proposals and reply to the list with any comments or
> suggestions.

Can you confirm whether I have got the big picture right?  I think some of 
my earlier remarks were mistaken.

You propose to introduce the concept of component and recognize software 
configuration as a matter of invoking components --- with a DAG of data 
dependencies among the component invocations.  While this is similar to 
what today's heat engine does for resources, you do NOT propose that the 
heat engine will get in the business of invoking components.  Rather: each 
VM will run a series of component invocations, and in-VM mechanisms will 
handle the cross-component synchronization and data communication.  You 
propose to add a bit of sugaring for the wait condition&handle mechanism, 
and the heat engine will do the de-sugaring.  Each component is written in 
one of a few supported configuration management (CM) frameworks, and 
essentially all component invocations on a given VM invoke components of 
the same CM framework (with possible exceptions for one or two really 
basic ones).  The heat engine gains the additional responsibility of 
making sure that the appropriate CM framework(s) is(are) bootstrapped in 
each VM.  The heat engine gains no additional responsibilities.

Have I got that right?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

(I really do not understand how the archive is ordered.  In the by-thread 
view, in which all messages with this subject are equally indented, the 
last message listed is not the chronologically last.)

I see that components have parameters.  In some uses (invocations) of 
components, parameters are given actual values.  I find it surprising that 
in the component definitions there are no declarations of parameters. 
Wouldn't that be helpful or needed?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Change in openstack/heat[master]: Implement a Heat-native resource group

[Mike Spreitzer]

Clint Byrum  wrote on 10/17/2013 09:16:12 PM:

> Excerpts from Mike Spreitzer's message of 2013-10-17 17:19:58 -0700:
> > What is the rationale for this new feature?  Since there is already an 

> > autoscaling group implemented by Heat, what is the added benefit here? 
And 
> > why is it being done as another heat-native thing rather than as an 
> > independent service (e.g., as outlined in 
> > https://wiki.openstack.org/wiki/Heat/AutoScaling for an autoscaling 
group 
> > service)?
> 
> This supports that design quite well.
> 
> The point is to be able to group and clone any resource, not just
> server/instance. So autoscaling might be configured to manage a group
> of Trove database instances which are then fed as a list to a group of
> separately autoscaled webservers.

Thanks for the answer.  I'm just a newbie here, trying to understand 
what's going on.  I still don't quite follow.  
https://wiki.openstack.org/wiki/Heat/AutoScaling says that what's 
autoscaled is a set of resources, not just one.  Can there be dependencies 
among the resources in that set?  For example, is the intent that I could 
autoscale a pair of (DB server, web server) where the web server's 
properties depend on the DB server's attributes?  If so, would it be 
problematic to implement that in terms of a pair of Heat-native resource 
groups?

BTW, is there some place I could have read the answers to my questions 
about the design thinking here?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Change in openstack/heat[master]: Implement a Heat-native resource group

[Mike Spreitzer]

What is the rationale for this new feature?  Since there is already an 
autoscaling group implemented by Heat, what is the added benefit here? And 
why is it being done as another heat-native thing rather than as an 
independent service (e.g., as outlined in 
https://wiki.openstack.org/wiki/Heat/AutoScaling for an autoscaling group 
service)?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler meeting and Icehouse Summit

[Mike Spreitzer]

Mike Wilson  wrote on 10/16/2013 07:13:17 PM:

> I need to understand better what "holistic scheduling" means, ...

By "holistic" I simply mean making a joint decision all at once about a 
bunch of related resources of a variety of types.  For example, making a 
joint decision about where to place a set of VMs and the Cinder volumes 
that will be attached to the VMs.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

Zane Bitter  wrote on 10/16/2013 10:30:44 AM:

> On 16/10/13 15:58, Mike Spreitzer wrote:
> ...
> > Thanks for a great short sharp answer.  In that light, I see a 
concern.
> >   Once a workflow has been generated, the system has lost the ability 
to
> > adapt to changes in either model.  In a highly concurrent and dynamic
> > environment, that could be problematic.
> 
> I think you're referring to the fact if reality diverges from the model 
> we have no way to bring it back in line (and even when doing an update, 
> things can and usually will go wrong if Heat's idea of the existing 
> template does not reflect reality any more). If so, then I agree that we 

> are weak in this area. You're obviously aware of 
> http://summit.openstack.org/cfp/details/95 so it is definitely on the 
radar.

Actually, I am thinking of both of the two models you mentioned.  We are 
only in the midst of implementing an even newer design (heat based), but 
for my group's old code we have a revised design in which the 
infrastructure orchestrator can react to being overtaken by later updates 
to the model we call "target state" (origin source is client) as well as 
concurrent updates to the model we call "observed state" (origin source is 
hardware/hypervisor).  I haven't yet decided what to recommend to the heat 
community, so I'm just mentioning the issue as a possible concern.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

Zane Bitter  wrote on 10/16/2013 08:25:38 AM:

> To answer your question, the key thing that Heat does is take in two 
> declarative models and generate a workflow to transform one into the 
> other. (The general case of this is a stack update, where the two models 

> are defined in the previous and new templates. Stack create and delete 
> are special cases where one or the other of the models is empty.)
> 
> Workflows don't belong in HOT because they are a one-off thing. You need 

> a different one for every situation, and this is exactly why Heat exists 

> - to infer the correct workflow to reify a model in any given situation.

Thanks for a great short sharp answer.  In that light, I see a concern. 
Once a workflow has been generated, the system has lost the ability to 
adapt to changes in either model.  In a highly concurrent and dynamic 
environment, that could be problematic.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

Steven Hardy  wrote on 10/16/2013 04:11:40 AM:
> ...
> IMO we should be abstracting the software configuration complexity 
behind a
> Heat resource interface, not pushing it up to a pre-processor (which
> implies some horribly complex interfaces at the heat template level)

I am not sure I follow.  Can you please elaborate on the horrible 
implication?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

The threading in the archive includes this discussion under the "HOT 
Software orchestration proposal for workflows" heading, and the overall 
ordering in the archive looks very mixed up to me.  I am going to reply 
here, hoping that the new subject line will be subject to less strange 
ordering in the archive; this is really a continuation of the overall 
discussion, not just Steve Baker's proposal.

What is the difference between what today's heat engine does and a 
workflow?  I am interested to hear what you experts think, I hope it will 
be clarifying.  I presume the answers will touch on things like error 
handling, state tracking, and updates.

I see the essence of Steve Baker's proposal to be that of doing the 
minimal mods necessary to enable the heat engine to orchestrate software 
components.  The observation is that not much has to change, since the 
heat engine is already in the business of calling out to things and 
passing values around.  I see a little bit of a difference, maybe because 
I am too new to already know why it is not an issue.  In today's heat 
engine, the calls are made to fixed services to do CRUD operations on 
virtual resources in the cloud, using credentials managed implicitly; the 
services have fixed endpoints, even as the virtual resources come and go. 
Software components have no fixed service endpoints; the service endpoints 
come and go as the host Compute instances come and go; I did not notice a 
story about authorization for the software component calls.

Interestingly, Steve Baker's proposal reminds me a lot of Chef.  If you 
just rename Steve's "component" to "recipe", the alignment gets real 
obvious; I am sure it is no accident.  I am not saying it is isomorphic 
--- clearly Steve Baker's proposal has more going on, with its cross-VM 
data dependencies and synchronization.  But let me emphasize that we can 
start to see a different way of thinking here.  Rather than focusing on a 
centrally-run workflow, think of each VM as independently running its own 
series of recipes --- with the recipes invocations now able to communicate 
and synchronize between VMs as well as within VMs.

Steve Baker's proposal uses two forms of communication and synchronization 
between VMs: (1) get_attr and (2) wait conditions and handles (sugar 
coated or not).  The implementation of (1) is part of the way the heat 
engine invokes components, the implementation of (2) is independent of the 
heat engine.

Using the heat engine for orchestration is limited to the kinds of logic 
that the heat engine can run.  This may be one reason people are 
suggesting using a general workflow engine.  However, the recipes 
(components) running in the VMs can do general computation; if we allow 
general cross-VM communication and synchronization as part of those 
general computations, we clearly have a more expressive system than the 
heat engine.

Of course, a general distributed computation can get itself into trouble 
(e.g., deadlock, livelock).  If we structure that computation as a set of 
components (recipe invocations) with a DAG of dependencies then we avoid 
those troubles.  And the kind of orchestration that the heat engine does 
is sufficient to invoke such components.

Structuring software orchestration as a DAG of components also gives us a 
leg up on UPDATE.  Rather than asking the user to write a workflow for 
each different update, or a general meta-workflow that does introspection 
to decide what work needs to be done, we ask the thing that invokes the 
components to run through the components in the way that today's heat 
engine runs through resources for an UPDATE.

Lakshmi has been working on a software orchestration technique that is 
also centered on the idea of a DAG of components.  It was created before 
we got real interested in Heat.  It is implemented as a pre-processor that 
runs upstream of where today's heat engine goes, emitting fairly minimal 
userdata needed for bootstrapping.  The dependencies between recipe 
invocations are handled very smoothly in the recipes, which are written in 
Chef.  No hackery is needed in the recipe text at all (thanks to Ruby 
metaprogramming); what is needed is only an additional declaration of what 
are the cross-VM inputs and outputs of each recipe.  The propagation of 
data and synchronization between VMs is handled, under the covers, via 
simple usage of ZooKeeper (other implementations are reasonable too).  But 
the idea of heat-independent propagation of data and synchronization among 
a DAG of components is not limited to chef-based components, and can 
appear fairly smooth in any recipe language.

A value of making software orchestration independent of today's heat 
engine is that it enables the four-stage pipeline that I have sketched at 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and whose ordering of functionality has been experimentally vetted with 
some non-trivial examples.  The first big

Re: [openstack-dev] [Heat] HOT Software configuration proposal

[Mike Spreitzer]

Steve Baker  wrote on 10/15/2013 06:48:53 PM:

> From: Steve Baker 
> To: openstack-dev@lists.openstack.org, 
> Date: 10/15/2013 06:51 PM
> Subject: [openstack-dev] [Heat] HOT Software configuration proposal
> 
> I've just written some proposals to address Heat's HOT software 
> configuration needs, and I'd like to use this thread to get some 
feedback:
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config

In that proposal, each component can use a different configuration 
management tool.

> 
https://wiki.openstack.org/wiki/Heat/Blueprints/native-tools-bootstrap-config


In this proposal, I get the idea that it is intended that each Compute 
instance run only one configuration management tool.  At least, most of 
the text discusses the support (e.g., the idea that each CM tool supplies 
userdata to bootstrap itself) in terms appropriate for a single CM tool 
per instance; also, there is no discussion of combining userdata from 
several CM tools.

I agree with the separation of concerns issues that have been raised.  I 
think all this software config stuff can be handled by a pre-processor 
that takes an extended template in and outputs a plain template that can 
be consumed by today's heat engine (no extension to the heat engine 
necessary).

Regards,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] Policy Model

[Mike Spreitzer]

"Yathiraj Udupi (yudupi)"  wrote on 10/14/2013 11:43:34 
PM:

> ... 
> 
> For the policy model, you can expect rows in the DB each 
> representing different policy instances something like- 

>  {id: , uuid: "SOME-UUID-1", name: "anti-colocation-1",  type: 
> "anti-colocation", properties: {level: "rack"}}

>  {id: , uuid: "SOME-UUID-2", name: "anti-colocation-2",  type: 
> "anti-colocation", properties: {level: "PM"}}

>  {id: , uuid: "SOME-UUID-3", name: "network-reachabilty-1", 
> type: "network-reachability" properties: {}}
> 
> And for the InstanceGroupPolicy model, you can expect rows such as 

> {id: 5, policy: "SOME-UUID-1", type: "group", edge_id: "", 
> group_id: 12345}

> {id: 6, policy: "SOME-UUID-1", type: "group", edge_id: "", 
> group_id: 22334} 

Do you imagine just one policy object of a given contents, or many?  Put 
another way, would every InstanceGroupPolicy object that wants to apply a 
rack-level anti-collocation policy use SOME-UUID-1?

Who or what created the record with id ?  Who or what decides to 
delete it, and when and why?  What about dangling references?  It seems to 
me that needing to answer these questions simply imposes unnecessary 
burdens.  If the "type" and "properties" fields of record id  were 
merged inline (replacing the policy:SOME-UUID-1 field) into records id 
, , and the other uses, then there are no hard questions to 
answer; the group author knows what policies he wants to apply and where, 
and he simply writes them there.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler meeting and Icehouse Summit

[Mike Spreitzer]

Yes, Rethinking Scheduler Design  
http://summit.openstack.org/cfp/details/34 is not the same as the 
performance issue that Boris raised.  I think the former would be a 
natural consequence of moving to an optimization-based joint 
decision-making framework, because such a thing necessarily takes a "good 
enough" attitude.  The issue Boris raised is more efficient tracking of 
the true state of resources, and I am interested in that issue too.  A 
holistic scheduler needs such tracking, in addition to the needs of the 
individual services.  Having multiple consumers makes the issue more 
interesting :-)

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

That came through beautifully formatted to me, but it looks much worse in 
the archive.  I'm going to use crude email tech here, so that I know it 
won't lose anything in handling.

"Yathiraj Udupi (yudupi)"  wrote on 10/14/2013 01:17:47 
PM:

> I read your email where you expressed concerns regarding create-time
> dependencies, and I agree they are valid concerns to be addressed. 
> But like we all agree, as a starting point, we are just focusing on 
> the APIs for now, and will leave that aside as implementation 
> details to be addressed later. 

I am not sure I understand your language here.  To me, design decisions 
that affect what calls the clients make are not implementation details, 
they are part of the API design.

> Thanks for sharing your suggestions on how we can simplify the APIs.
> I think we are getting closer to finalizing this one. 
> 
> Let us start at the model proposed here - 
> [1] https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing 
> (Ignore the white diamonds - they will be black, when I edit the doc)
> 
> The InstanceGroup represents all the information necessary to 
> capture the group - nodes, edges, policies, and metadata
> 
> InstanceGroupMember - is a reference to an Instance, which is saved 
> separately, using the existing Instance Model in Nova.

I think you mean this is a reference to either a group or an individual 
Compute instance.

> 
> InstanceGroupMemberConnection - represents the edge
> 
> InstanceGroupPolicy is a reference to a Policy, which will also be 
> saved separately, (currently not existing in the model, but has to 
> be created). Here in the Policy model, I don't mind adding any 
> number of additional fields, and key-value pairs to be able to fully
> define a policy.  I guess a Policy-metadata dictionary is sufficient
> to capture all the required arguments. 
> The InstanceGroupPolicy will be associated to a group as a whole or an 
edge.

Like I said under separate cover, I think one of these is a policy *use* 
rather than a policy *definition*.  I go further and emphasize that the 
interesting out-of-scope definitions are of policy *types*.  A policy type 
takes parameters.  For example, policies of the anti-collocation (AKA 
anti-affinity) type have a parameter that specifies the level in the 
physical hierarchy where the location must differ (rack, host, ...).  Each 
policy type specifies a set of parameters, just like a procedure specifies 
parameters; each use of a policy type supplies values for the parameters, 
just like a procedure invocation supplies values for the procedure's 
parameters.  I suggest separating parameter values from metadata; the 
former are described by the policy type, while the latter are unknown to 
the policy type and are there for other needs of the client.

Yes, a use of a policy type is associated with a group or an edge.  In my 
own writing I have suggested a third possibility: that a policy use can be 
directly associated with an individual resource.  It just so happens that 
the code my group already has been running also has your restriction: it 
supports only policies associated with groups and relationships.  But I 
suggested allowing direct attachment to resources (as well as 
relationships also being able to directly reference resources instead of 
groups) because I think this restriction --- while it simplifies 
implementation --- makes templates more verbose; I felt the latter was a 
more important consideration than the former.  If you want to roadmap this 
--- restricted first, liberal later --- that's fine with me.

> 
> InstanceGroupMetadata - represents key-value dictionary for any 
> additional metadata for the instance group. 
> 
> I think this should fully support what we care about - nodes, edges,
> policies and metadata. 
> 
> Do we all agree ? 

Yes, with exceptions noted above.

> 
> Now going to the APIs, 
> 
> Register GROUP API (from my doc [1]): 
> 
> POST  /v3.0/{tenant_id}/groups --- Register a group

In such specs it would be good to be explicit about the request parameters 
and body.  If I follow correctly, 
https://review.openstack.org/#/c/30028/25/doc/api_samples/os-instance-groups/instance-groups-post-req.json
 
shows us that you intended (as of that patch) the body to carry a group 
definition.

> I think the confusion is only about when the member (all nested 
> members) and policy about when they are saved in the DB (registered,
> but not CREATED actually), such that we can associate a UUID.  This 
> led to my original thinking that it is a 3-phase operation where we 
> have to register (save in DB) the nested members first, then 
> register the group as a whole.  But this is not client friendly. 
> 
> Like I had suggested earlier, as an implementation detail of the 
> Group registration API (CREATE part 1 in your terminology), we can 
> support this: as part of the group registration transaction, 
> complete the registration of the nest

Re: [openstack-dev] [scheduler] Policy Model

[Mike Spreitzer]

Consider the example at 
https://docs.google.com/drawings/d/1nridrUUwNaDrHQoGwSJ_KXYC7ik09wUuV3vXw1MyvlY

We could indeed have distinct policy objects.  But I think they are policy 
*uses*, not policy *definitions* --- which is why is prefer to give them 
less prominent lifecycles.  In the example cited above, one policy use 
object might be: {id: , type: anti_collocation, properties: 
{level: rack}}, and there are four references to it; another policy use 
object might be {id: , type: network_reachability}, and there 
are three references to it.  What object should own the policy use 
objects?  You might answer that policy uses are owned by groups.  I do not 
think it makes sense to give them a more prominent lifecycle.  As I said, 
my preference would be to give them a less prominent lifecycle.  I would 
be happy to see each policy use owned by an InstanceGroupPolicy[Use] that 
references it and allow only one reference per policy use --- in other 
words, make the InstanceGroupPolicy[Use] class inherit from the Policy Use 
class.  And since I am not proposing that anything else inherit from the 
Policy Use class, I would even more prefer to see its contents simply 
merged inline into the InstanceGroupPolicy[Use] class.

Regards,
Mike



From:   "Yathiraj Udupi (yudupi)" 
To: Mike Spreitzer/Watson/IBM@IBMUS, 
Cc: OpenStack Development Mailing List 

Date:   10/14/2013 01:38 PM
Subject:Re: [scheduler] Policy Model



Mike, 

Like I proposed in my previous email about the model and the APIs, 

About the InstanceGroupPolicy, why not leave it as is, and introduce a new 
abstract model class called Policy. 
The InstanceGroupPolicy will be a reference to a Policy object saved 
separately. 
and the "policy" field will point to the saved Policy object's unique name 
or id. 

The new class Policy – can have the usual fields – id, name, uuid, and a 
dictionary of key-value pairs for any additional arguments about the 
policy. 

This is in alignment with the model for InstanceGroupMember, which is a 
reference to an actual Instance Object saved in the DB. 

I will color all the diamonds black to make it a composition I the UML 
diagram. 

Thanks,
Yathi. 







From: Mike Spreitzer 
Date: Monday, October 14, 2013 7:14 AM
To: Yathiraj Udupi 
Cc: OpenStack Development Mailing List 
Subject: [scheduler] Policy Model

Could we agree on the following small changes to the model you posted last 
week?

1.  Rename InstanceGroupPolicy to InstanceGroupPolicyUse

2.  In InstanceGroupPolicy[Use], rename the "policy" field to 
"policy_type"

3.  Add an InstanceGroupPolicyUseProperty table, holding key/value pairs 
(two strings) giving the properties of the policy uses

4.  Color all the diamonds black 

Thanks, 
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [scheduler] Policy Model

[Mike Spreitzer]

Could we agree on the following small changes to the model you posted last 
week?

1.  Rename InstanceGroupPolicy to InstanceGroupPolicyUse

2.  In InstanceGroupPolicy[Use], rename the "policy" field to 
"policy_type"

3.  Add an InstanceGroupPolicyUseProperty table, holding key/value pairs 
(two strings) giving the properties of the policy uses

4.  Color all the diamonds black

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

I'll be at the summit too.  Available Nov 4 if we want to do some prep 
then.  It will be my first summit, I am not sure how overbooked my summit 
time will be.

Regards,
Mike



From:   Sylvain Bauza 
To: OpenStack Development Mailing List 
, 
Cc: Mike Spreitzer/Watson/IBM@IBMUS
Date:   10/11/2013 08:19 AM
Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource 
Placement - Updated Instance Group Model and API extension model - WIP 
Draft



Long-story short, sounds like we do have the same concerns here in 
Climate.

I'll be present at the Summit, any chance to do an unconference meeting in

between all parties ?

Thanks,
-Sylvain

Le 11/10/2013 08:25, Mike Spreitzer a écrit :
Regarding Alex's question of which component does holistic infrastructure 
scheduling, I hesitate to simply answer "heat".  Heat is about 
orchestration, and infrastructure scheduling is another matter.  I have 
attempted to draw pictures to sort this out, see 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and 
https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH
_TONw6g 
.  In those you will see that I identify holistic infrastructure 
scheduling as separate functionality from infrastructure orchestration 
(the main job of today's heat engine) and also separate from software 
orchestration concerns.  However, I also see a close relationship between 
holistic infrastructure scheduling and heat, as should be evident in those

pictures too. 

Alex made a remark about the needed inputs, and I agree but would like to 
expand a little on the topic.  One thing any scheduler needs is knowledge 
of the amount, structure, and capacity of the hosting thingies (I wish I 
could say "resources", but that would be confusing) onto which the 
workload is to be scheduled.  Scheduling decisions are made against 
available capacity.  I think the most practical way to determine available

capacity is to separately track raw capacity and current (plus already 
planned!) allocations from that capacity, finally subtracting the latter 
from the former. 

In Nova, for example, sensing raw capacity is handled by the various 
nova-compute agents reporting that information.  I think a holistic 
infrastructure scheduler should get that information from the various 
individual services (Nova, Cinder, etc) that it is concerned with 
(presumably they have it anyway). 

A holistic infrastructure scheduler can keep track of the allocations it 
has planned (regardless of whether they have been executed yet).  However,

there may also be allocations that did not originate in the holistic 
infrastructure scheduler.  The individual underlying services should be 
able to report (to the holistic infrastructure scheduler, even if lowly 
users are not so authorized) all the allocations currently in effect.  An 
accurate union of the current and planned allocations is what we want to 
subtract from raw capacity to get available capacity. 

If there is a long delay between planning and executing an allocation, 
there can be nasty surprises from competitors --- if there are any 
competitors.  Actually, there can be nasty surprises anyway.  Any 
scheduler should be prepared for nasty surprises, and react by some 
sensible retrying.  If nasty surprises are rare, we are pretty much done. 
If nasty surprises due to the presence of competing managers are common, 
we may be able to combat the problem by changing the long delay to a short

one --- by moving the allocation execution earlier into a stage that is 
only about locking in allocations, leaving all the other work involved in 
creating virtual resources to later (perhaps Climate will be good for 
this).  If the delay between planning and executing an allocation is short

and there are many nasty surprises due to competing managers, then you 
have too much competition between managers --- don't do that. 

Debo wants a simpler nova-centric story.  OK, how about the following. 
This is for the first step in the roadmap, where scheduling decisions are 
still made independently for each VM instance.  For the client/service 
interface, I think we can do this with a simple clean two-phase interface 
when traditional software orchestration is in play, a one-phase interface 
when slick new software orchestration is used.  Let me outline the 
two-phase flow.  We extend the Nova API with CRUD operations on VRTs 
(top-level groups).  For example, the CREATE operation takes a definition 
of a top-level group and all its nested groups, definitions (excepting 
stuff like userdata) of all the resources (only VM instances, for now) 
contained in those groups, all the relationships among those 
groups/resources, and all the applications of policy to those groups, 
resources, and relationships.  This is a rest-style interface; the CREATE 
operation takes a definition of the thing (a top-level group and all that 
it 

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

Regarding Alex's question of which component does holistic infrastructure 
scheduling, I hesitate to simply answer "heat".  Heat is about 
orchestration, and infrastructure scheduling is another matter.  I have 
attempted to draw pictures to sort this out, see 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and 
https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g 
.  In those you will see that I identify holistic infrastructure 
scheduling as separate functionality from infrastructure orchestration 
(the main job of today's heat engine) and also separate from software 
orchestration concerns.  However, I also see a close relationship between 
holistic infrastructure scheduling and heat, as should be evident in those 
pictures too.

Alex made a remark about the needed inputs, and I agree but would like to 
expand a little on the topic.  One thing any scheduler needs is knowledge 
of the amount, structure, and capacity of the hosting thingies (I wish I 
could say "resources", but that would be confusing) onto which the 
workload is to be scheduled.  Scheduling decisions are made against 
available capacity.  I think the most practical way to determine available 
capacity is to separately track raw capacity and current (plus already 
planned!) allocations from that capacity, finally subtracting the latter 
from the former.

In Nova, for example, sensing raw capacity is handled by the various 
nova-compute agents reporting that information.  I think a holistic 
infrastructure scheduler should get that information from the various 
individual services (Nova, Cinder, etc) that it is concerned with 
(presumably they have it anyway).

A holistic infrastructure scheduler can keep track of the allocations it 
has planned (regardless of whether they have been executed yet).  However, 
there may also be allocations that did not originate in the holistic 
infrastructure scheduler.  The individual underlying services should be 
able to report (to the holistic infrastructure scheduler, even if lowly 
users are not so authorized) all the allocations currently in effect.  An 
accurate union of the current and planned allocations is what we want to 
subtract from raw capacity to get available capacity.

If there is a long delay between planning and executing an allocation, 
there can be nasty surprises from competitors --- if there are any 
competitors.  Actually, there can be nasty surprises anyway.  Any 
scheduler should be prepared for nasty surprises, and react by some 
sensible retrying.  If nasty surprises are rare, we are pretty much done. 
If nasty surprises due to the presence of competing managers are common, 
we may be able to combat the problem by changing the long delay to a short 
one --- by moving the allocation execution earlier into a stage that is 
only about locking in allocations, leaving all the other work involved in 
creating virtual resources to later (perhaps Climate will be good for 
this).  If the delay between planning and executing an allocation is short 
and there are many nasty surprises due to competing managers, then you 
have too much competition between managers --- don't do that.

Debo wants a simpler nova-centric story.  OK, how about the following. 
This is for the first step in the roadmap, where scheduling decisions are 
still made independently for each VM instance.  For the client/service 
interface, I think we can do this with a simple clean two-phase interface 
when traditional software orchestration is in play, a one-phase interface 
when slick new software orchestration is used.  Let me outline the 
two-phase flow.  We extend the Nova API with CRUD operations on VRTs 
(top-level groups).  For example, the CREATE operation takes a definition 
of a top-level group and all its nested groups, definitions (excepting 
stuff like userdata) of all the resources (only VM instances, for now) 
contained in those groups, all the relationships among those 
groups/resources, and all the applications of policy to those groups, 
resources, and relationships.  This is a rest-style interface; the CREATE 
operation takes a definition of the thing (a top-level group and all that 
it contains) being created; the UPDATE operation takes a revised 
definition of the whole thing.  Nova records the presented information; 
the familiar stuff is stored essentially as it is today (but marked as 
being in some new sort of tentative state), and the grouping, 
relationship, and policy stuff is stored according to a model like the one 
Debo&Yathi wrote.  The CREATE operation returns a UUID for the newly 
created top-level group.  The invocation of the top-level group CRUD is a 
single operation and it is the first of the two phases.  In the second 
phase of a CREATE flow, the client creates individual resources with the 
same calls as are used today, except that each VM instance create call is 
augmented with a pointer into the policy information.  T

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

Debojyoti Dutta  wrote on 10/09/2013 02:48:26 AM:
>
> Mike, I agree we could have a cleaner API but I am not sure how
> cleanly it will integrate with current nova which IMO should be test
> we should pass (assuming we do cross services later)

I think the cleaner APIs integrate with Nova as well as the three phase 
API you suggested.  Am I missing some obvious impediment?

> ...
> > To me the most frustrating aspect of this challenge is the need for 
the
> > client to directly mediate the dependencies between resources; this is
> > really what is driving us to do ugly things.  As I mentioned before, I 
am
> > coming from a setting that does not have this problem.  So I am 
thinking
> > about two alternatives: (A1) how clean can we make a system in which 
the
> > client continues to directly mediate dependencies between resources, 
and
> > (A2) how easily and cleanly can we make that problem go away.
> 
> Am a little confused - How is the API dictating either A1 or A2? Isnt
> that a function of the implementation of the API.  For a moment let us
> assume that the black box implementation will be awesome and address
> your concerns.

I am talking about how the client/service interface, it is not (just) a 
matter of service implementation.

My complaint is that the software orchestration technique commonly used 
prevents us from having a one-phase API for holistic infrastructure 
scheduling.  The commonly used software orchestration technique requires 
some serialization of the resource creation calls.  For example, if one VM 
instance runs a database and another VM instance runs a web server that 
needs to be configured with the private IP address of the database, the 
common technique is for the client to first create the database VM 
instance, then take the private IP address from that VM instance and use 
it to compose the userdata that is passed in the Nova call that creates 
the web server VM instance.  That client can not present all at once a 
fully concrete and literal specification of both VM instances, because the 
userdata for one is not knowable until the other has been created.  The 
client has to be able to make create-like calls in some particular order 
rather than ask for all creation at once.  If the client could ask for all 
creation at once then we could use a one-phase API: it simply takes a 
specification of the resources along with their policies and 
relationships.

Of course, there is another way out.  We do have in OpenStack a technology 
by which a client can present all at once a specification of many VM 
instances where the userdata of some depend on the results of creating 
others.  If we were willing to use this technology, we could follow A2. 
The CREATE flow would go like this: (i) the client presents the 
specification of resources (including the computations that link some), 
with grouping, relationships, and policies, to our new API; (ii) our new 
service registers the new topology and (once we advance this far on the 
development roadmap) does holistic scheduling; (iii) our new service 
updates the resource specifications to include pointers into the policy 
data; (iv) our new service passes the enhanced resource specifications to 
that other service that can do the creation calls linked by the prescribed 
computations; (v) that other service does its thing, causing a series 
(maybe with some allowed parallelism) of creation calls, each augmented by 
the relevant pointer into the policy information; (vi) the service 
implementing a creation call gets what it normally does plus the policy 
pointer, which it follows to get the relevant policy information (at the 
first step in the development roadmap) or the scheduling decision (in the 
second step of the development roadmap).  But I am getting ahead of myself 
here and discussing backend implementation; I think we are still working 
on the user-facing API.

> The question is this - does the current API help
> specify what we  want assuming we will be able to extend the notion of
> nodes, edges, policies and metadata?

I am not sure I understand that remark.  Of course the API you proposed is 
about enabling the client to express the policy information that we both 
advocate.  I am not sure I understand why you add the qualifier of 
"assuming we will be able to extend the notion of ...".  I do not think we 
(yet) have a policy type catalog set in stone, if that is the concern.  I 
think there is an interesting discussion to have about defining that 
catalog.

BTW, note that the class you called InstanceGroupPolicy is not just a 
reference to a policy, it also specifies one place where that policy is 
being applied.  That is really the class of policy applications (or 
"uses").

I think some types of policies have parameters.  A relationship policy 
about limiting the number of network hops takes a parameter that is the 
hop count limit.  A policy about anti-collocation takes a physical 
hierarchy level as a parameter, to p

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

Yes, there is more than the northbound API to discuss.  Gary started us 
there in the Scheduler chat on Oct 1, when he broke the issues down like 
this:

11:12:22 AM garyk: 1. a user facing API
11:12:41 AM garyk: 2. understanding which resources need to be tracked
11:12:48 AM garyk: 3. backend implementation

The full transcript is at 
http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-10-01-15.08.log.html

Alex Glikson  wrote on 10/09/2013 02:14:03 AM:
> 
> Good summary. I would also add that in A1 the schedulers (e.g., in 
> Nova and Cinder) could talk to each other to coordinate. Besides 
> defining the policy, and the user-facing APIs, I think we should 
> also outline those cross-component APIs (need to think whether they 
> have to be user-visible, or can be admin). 
> 
> Regards, 
> Alex 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Mike Spreitzer]

I favor separation of concerns.  I think (4), at least, has got nothing to 
do with infrastructure orchestration, the primary concern of today's heat 
engine.  I advocate (4), but as separate functionality.

Regards,
Mike

Alex Rudenko  wrote on 10/09/2013 12:59:22 PM:

> From: Alex Rudenko 
> To: OpenStack Development Mailing List 
, 
> Date: 10/09/2013 01:03 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration 
> proposal for workflows
> 
> Hi everyone,
> 
> I've read this thread and I'd like to share some thoughts. In my 
> opinion, workflows (which run on VMs) can be integrated with heat 
> templates as follows:
> 1. workflow definitions should be defined separately and processed 
> by stand-alone workflow engines (chef, puppet etc). 
> 2. the HOT resources should reference workflows which they require, 
> specifying a type of workflow and the way to access a workflow 
> definition. The workflow definition might be provided along with HOT.
> 3. Heat should treat the orchestration templates as transactions 
> (i.e. Heat should be able to rollback in two cases: 1) if something 
> goes wrong during processing of an orchestration workflow 2) when a 
> stand-alone workflow engine reports an error during processing of a 
> workflow associated with a resource)
> 4. Heat should expose an API which enables basic communication 
> between running workflows. Additionally, Heat should provide an API 
> to workflows that allows workflows to specify whether they completed
> successfully or not. The reference to these APIs should be passed to
> the workflow engine that is responsible for executing workflows on VMs.
> Pros of each point:
> 1 & 2 - keeps Heat simple and gives a possibility to choose the best
> workflows and engines among available ones.
> 3 - adds some kind of all-or-nothing semantics improving the control
> and awareness of what's going on inside VMs.
> 4 - allows workflow synchronization and communication through Heat 
> API. Provides the error reporting mechanism for workflows. If a 
> workflow does not need this functionality, it can ignore it.
> 
> Cons:
> - Changes to existing workflows making them aware of Heat existence 
> are required.
> 
> These thoughts might show some gaps in my understanding of how Heat 
> works, but I would like to share them anyway.
> 
> Best regards,
> Oleksii Rudenko
> 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

Thanks, that helps a lot.  At what point in the process is it decided (and 
by what) which host will run the VM instance?  Is that host picked from 
the same pool as is used for non-reserved instances?  How does the outline 
go if there is not enough capacity for the reserved instance?

Thanks,
Mike


Dina Belova  wrote on 10/09/2013 06:34:06 AM:

> Mike, I'll try to describe the reservation process for the virtual 
> reservations. I'll use Nova project as an example.
> 
> As I said, this Nova workflow is only the example that may and 
> certainly will be modified for other 'virtual' projects.
> 
> 1) User goes to Nova via CLI/Dashboard and commits all usual actions
> like he/she wants to boot instance. The only difference is that user
> passes reservation-connected hints to Nova. In the CLI this request 
> may look like the following:
> 
> nova  boot --flavor 1 --image bb3979c2-b2e1-4836-abbc-2ee510064718 
> --hint reserved=True --hint lease_params='{"name": "lease1", 
> "start": "now", "end": "2013-12-1 16:07"}' vm1
> 
> If scheduling process went OK, we'll see the following by 'nova list' 
command:
> 
> +--+--+--
> ++-+--+
> | ID   | Name | Status   | Task 
> State | Power State | Networks |
> +--+--+--
> ++-+--+
> | a7ac3b2e-dca5-4d21-ab37-cd019a813636 | vm1  | RESERVED | None 
>   | NOSTATE | private=10.0.0.3 |
> +--+--+--
> ++-+--+
> 
> 2) Request passes up to the Compute Manager, where scheduling 
> process is already done. If Manager finds reservation related hints 
> it uses Climate client to create lease using passed to Nova params 
> and id of the VM to be reserved. Also Nova changes status of VM in 
> its DB to 'RESERVED'. If there are no reservation related hints 
> filter properties, Nova just spawns instance as usual.
> 
> 3) Lease creation request goes to Climate Lease API via Climate 
> Client. Climate Lease API will be mostly used by other services 
> (like Nova in this example) and by admin users to manage leases as 
> 'contracts'.
> 
> 4) Climate Lease API passes lease creation request to Climate 
> Manager service via RPC. Climate Manager is the service that 
> communicates with all resource plugins and Climate DB. Climate 
> Manager creates lease record in DB, all reservation records (for the
> instance in this case) and all events records. Even if user passes 
> no additional events (like notifications in future), at least two 
> events for lease are created - 'start' and 'end' events.
> 
> 5) One more function that Manager does is periodical DB polling to 
> find out if there are any 'UNDONE' event to be processed. If there 
> is such event (for example, start event for the lease just saved in 
> DB), manager begins to process it. That means manager sets event 
> status to 'IN_PROGRESS' and for every reservation in lease commits 
> 'on_start' actions for this reservation. Now there is one-to-one 
> relationship between lease and reservation, but we suppose there may
> be cases for one-to-many relationship. 'On_start' actions are 
> defined in resource plugin responsible for this resource type 
> ("virtual:instance") in this example. Plugins are loaded using 
> stevedore and needed ones are defined in climate.conf file.
> 
> 6) "virtual:instance" plugin commits on_start actions. For VM it may
> be 'wake_up' action, that wakes reserved instance up through Nova 
> API. This may be implemented using Nova extensions mechanism. Wake 
> up action really spawns this instance.
> 
> 7) If everything is ok, Manager sets event status to 'DONE' or 
'COMPLETED'.
> 
> 8) Almost the same process is done when Manager gets 'end' event for
> the lease from DB. 
> 
> Thank you for the attention.
> 
> Dina
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

Thanks for the clue about where the request/response bodies are 
documented.  Is there any convenient way to view built documentation for 
Havana right now?

You speak repeatedly of the desire for "clean" interfaces, and nobody 
could disagree with such words.  I characterize my desire that way too. It 
might help me if you elaborate a little on what "clean" means to you.  To 
me it is about minimizing the number of interactions between different 
modules/agents and the amount of information in those interactions.  In 
short, it is about making narrow interfaces - a form of simplicity.

To me the most frustrating aspect of this challenge is the need for the 
client to directly mediate the dependencies between resources; this is 
really what is driving us to do ugly things.  As I mentioned before, I am 
coming from a setting that does not have this problem.  So I am thinking 
about two alternatives: (A1) how clean can we make a system in which the 
client continues to directly mediate dependencies between resources, and 
(A2) how easily and cleanly can we make that problem go away.

For A1, we need the client to make a distinct activation call for each 
resource.  You have said that we should start the roadmap without joint 
scheduling; in this case, the scheduling can continue to be done 
independently for each resource and can be bundled with the activation 
call.  That can be the call we know and love today, the one that creates a 
resource, except that it needs to be augmented to also carry some pointer 
that points into the policy data so that the relevant policy data can be 
taken into account when making the scheduling decision.  Ergo, the client 
needs to know this pointer value for each resource.  The simplest approach 
would be to let that pointer be the combination of (p1) a VRT's UUID and 
(p2) the local name for the resource within the VRT.  Other alternatives 
are possible, but require more bookkeeping by the client.

I think that at the first step of the roadmap for A1, the client/service 
interaction for CREATE can be in just two phases.  In the first phase the 
client presents a topology (top-level InstanceGroup in your terminology), 
including resource definitions, to the new API for registration; the 
response is a UUID for that registered top-level group.  In the second 
phase the client "creates" the resources as is done today, except that 
each creation call is augmented to carry the aforementioned pointer into 
the policy information.  Each resource scheduler (just nova, at first) can 
use that pointer to access the relevant policy information and take it 
into account when scheduling.  The client/service interaction for UPDATE 
would be in the same two phases: first update the policy&resource 
definitions at the new API, then do the individual resource updates in 
dependency order.

I suppose the second step in the roadmap is to have Nova do joint 
scheduling.  The client/service interaction pattern can stay the same. The 
only difference is that Nova makes the scheduling decisions in the first 
phase rather than the second.  But that is not a detail exposed to the 
clients.

Maybe the third step is to generalize beyond nova?

For A2, the first question is how to remove "user-level" create-time 
dependencies between resources.  We are only concerned with the 
"user-level" create-time dependencies here because it is only they that 
drive intimate client interactions.  There are also create-time 
dependencies due to the nature of the resource APIs; for example, you can 
not attach a volume to a VM until after both have been created.  But 
handling those kinds of create-time dependencies does not require intimate 
interactions with the client.  I know of two software orchestration 
technologies developed in IBM, and both have the property that there are 
no "user-level" create-time dependencies between resources; rather, the 
startup code ("userdata") that each VM runs handles dependencies (using a 
library for cross-VM communication and synchronization).  This can even be 
done in plain CFN, using wait conditions and handles (albeit somewhat 
clunkily), right?  So I think there are ways to get this nice property 
already.  The next question is how best to exploit it to make cleaner 
APIs.  I think we can have a one-step client/service interaction: the 
client presents a top-level group (including leaf resource definitions) to 
the new service, which registers it and proceeds to 
create/schedule/activate the resources.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

Sylvain: please do not interpret my questions as hostility.  I am only 
trying to understand your proposal, but I am still confused.  Can you 
please walk through a scenario involving Climate reservations on virtual 
resources?  I mean from start to finish, outlining which party makes which 
decision when, based on what.  I am trying to understand the relationship 
between the individual resource schedulers (such as nova, cinder) and 
climate --- they both seem to be about allocating the same resources.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

Yes, that helps.  Please, guys, do not interpret my questions as 
hostility, I really am just trying to understand.  I think there is some 
overlap between your concerns and mine, and I hope we can work together.

Sticking to the physical reservations for the moment, let me ask for a 
little more explicit details.  In your outline below, late in the game you 
write "the actual reservation is performed by the lease manager plugin". 
Is that the point in time when something (the lease manager plugin, in 
fact) decides which hosts will be used to satisfy the reservation?  Or is 
that decided up-front when the reservation is made?  I do not understand 
how the lease manager plugin can make this decision on its own, isn't the 
nova scheduler also deciding how to use hosts?  Why isn't there a problem 
due to two independent allocators making allocations of the same resources 
(the system's hosts)?

Thanks,
Mike

Patrick Petit  wrote on 10/07/2013 07:02:36 AM:

> Hi Mike,
> 
> There are actually more facets to this. Sorry if it's a little 
> confusing :-( Climate's original blueprint https://
> wiki.openstack.org/wiki/Blueprint-nova-planned-resource-reservation-api
> was about physical host reservation only. The typical use case 
> being: "I want to reserve x number of hosts that match the 
> capabilities expressed in the reservation request". The lease is 
> populated with reservations which at this point are only capacity 
> descriptors. The reservation becomes active only when the lease 
> starts at a specified time and for a specified duration. The lease 
> manager plugin in charge of the physical reservation has a planning 
> of reservations that allows Climate to grant a lease only if the 
> requested capacity is available at that time. Once the lease becomes
> active, the user can request instances to be created on the reserved
> hosts using a lease handle as a Nova's scheduler hint. That's 
> basically it. We do not assume or enforce how and by whom (Nova, 
> Heat ,...) a resource instantiation is performed. In other words, a 
> host reservation is like a whole host allocation https://
> wiki.openstack.org/wiki/WholeHostAllocation that is reserved ahead 
> of time by a tenant in anticipation of some workloads that is bound 
> to happen in the future. Note that while we are primarily targeting 
> hosts reservations the same service should be offered for storage. 
> Now, Mirantis brought in a slew of new use cases that are targeted 
> toward virtual resource reservation as explained earlier by Dina. 
> While architecturally both reservation schemes (physical vs virtual)
> leverage common components, it is important to understand that they 
> behave differently. For example, Climate exposes an API for the 
> physical resource reservation that the virtual resource reservation 
> doesn't. That's because virtual resources are supposed to be already
> reserved (through some yet to be created Nova, Heat, Cinder,... 
> extensions) when the lease is created. Things work differently for 
> the physical resource reservation in that the actual reservation is 
> performed by the lease manager plugin not before the lease is 
> created but when the lease becomes active (or some time before 
> depending on the provisioning lead time) and released when the lease 
ends.
> HTH clarifying things.
> BR,
> Patrick 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

On second thought, I should revise and extend my remarks.  This message 
supersedes my previous two replies.

Thanks.  I have a few questions.  First, I am a bit stymied by the style 
of API documentation used in that document and many others: it shows the 
first line of an HTTP request but says nothing about all the other 
details.  I am sure some of those requests must have interesting bodies, 
but I am not always sure which ones have a body at all, let alone what 
goes in it.  I suspect there may be some headers that are important too. 
Am I missing something? 

That draft says the VMs are created before the group.  Is there a way 
today to create a VM without scheduling it?  Is there a way to activate a 
resource that has already been scheduled but not activated? By "activate" 
I mean, for a VM instance for example, to start running it.  

As I understand your draft, it lays out a three phase process for a client 
to follow: create resources without scheduling or activating them, then 
present the groups and policies to the service for joint scheduling, then 
activate the resources.  With regard to a given resource, things must 
happen in that order; between resources there is a little more 
flexibility.  Activations are invoked by the client in an order that is 
consistent with (a) runtime dependencies that are mediated directly by the 
client (e.g., string slinging in the heat engine) and (b) the nature of 
the resources (for example, you  can not attach a volume to a VM instance 
until after both have been created).  Other than those considerations, the 
ordering and/or parallelism is a degree of freedom available to the 
client.  Have I got this right? 

Couldn't we simplify this into a two phase process: create groups and 
resources with scheduling, then activate the resources in an acceptable 
order? 

FYI: my group is using Weaver as the software orchestration technique, so 
there are no runtime dependencies that are mediated directly by the 
client.  The client sees a very simple API: the client presents a 
definition of all the groups and resources, and the service first 
schedules it all then activates in an acceptable order.  (We already have 
something in OpenStack that can do resource invocations in an acceptable 
order, right?)  Weaver is not the only software orchestration technique 
with this property.  The simplicity of this API is one reason I recommend 
software orchestration techniques that take dependency mediation out of 
the client's hands.  I hope that with coming work on HOT we can get 
OpenStack to this level of API simplicity.  But that struggle lies farther 
down the roadmap... 

I was wondering if you could explain why you included all those integer 
IDs; aren't the UUIDs sufficient?  Do you intend that clients will 
see/manipulate the integer IDs?

If I understand your UML correctly, an InstanceGroup owns its metadata but 
none of the other subsidiary objects introduced.  Why not?  If an 
InstanceGroup is deleted, shouldn't all those other subsidiary objects be 
deleted too?

Thanks, 
Mike 

"Yathiraj Udupi (yudupi)"  wrote on 10/07/2013 11:10:20 
PM:
> Hi, 
> 
> Based on the discussions we have had in the past few scheduler sub-
> team meetings,  I am sharing a document that proposes an updated 
> Instance Group Model and API extension model. 
> This is a work-in-progress draft version, but sharing it for early 
feedback. 
> 
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing
 

> 
> This model support generic instance types, where an instance can 
> represent a virtual node of any resource type.  But in the context 
> of Nova, an instance refers to the VM instance. 
> 
> This builds on the existing proposal for Instance Group Extension as
> documented here in this blueprint:  https://
> blueprints.launchpad.net/nova/+spec/instance-group-api-extension 
> 
> Thanks,
> Yathi. 
> 
> ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Scheduler] simpler policy example, with a picture

[Mike Spreitzer]

See the picture at 
https://docs.google.com/drawings/d/1nridrUUwNaDrHQoGwSJ_KXYC7ik09wUuV3vXw1MyvlY 
and the writeup at 
https://docs.google.com/document/d/1Ecr1zXpU4zMvEySmbD-wwxLPfinndb0eaKgMuALOb2I

This is not as small as possible, but not terribly large; there are only 8 
VM instances.  I wanted to give some idea of the use cases I have in mind. 
 It is a relatively simple 3-tier web app with some considerations for 
high availability.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

In addition to the other questions below, I was wondering if you could 
explain why you included all those integer IDs; aren't the UUIDs 
sufficient?

Thanks,
Mike



From:   Mike Spreitzer/Watson/IBM@IBMUS
To: "Yathiraj Udupi (yudupi)" , 
Cc: OpenStack Development Mailing List 

Date:   10/08/2013 12:41 AM
Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource 
Placement - Updated Instance Group Model and API extension model - WIP 
Draft



Thanks.  I have a few questions.  First, I am a bit stymied by the style 
of API documentation used in that document and many others: it shows the 
first line of an HTTP request but says nothing about all the other 
details.  I am sure some of those requests must have interesting bodies, 
but I am not always sure which ones have a body at all, let alone what 
goes in it.  I suspect there may be some headers that are important too. 
Am I missing something? 

That draft says the VMs are created before the group.  Is there a way 
today to create a VM without scheduling it? 

As I understand your draft, it lays out a three phase process for a client 
to follow: create resources without scheduling or activating them, then 
arrange them into groups, then schedule & activate them.  By "activate" I 
mean, for a VM instance, to start running it.  That ordering must hold 
independently for each resource.  Activations are invoked by the client in 
an order that is consistent with (a) runtime dependencies that are 
mediated directly by the client (e.g., string slinging in the heat engine) 
and (b) the nature of the resources (for example, you  can not attach a 
volume to a VM instance until after both have been created).  Other than 
those considerations, the ordering and/or parallelism is a degree of 
freedom available to the client.  Have I got this right? 

Couldn't we simplify this into a two phase process: create groups and 
resources with scheduling, then activate the resources in an acceptable 
order? 

FYI: my group is using Weaver as the software orchestration technique, so 
there are no runtime dependencies that are mediated directly by the 
client.  The client sees a very simple API: the client presents a 
definition of all the groups and resources, and the service first 
schedules it all then activates in an acceptable order.  (We already have 
something in OpenStack that can do activations in an acceptable order, 
right?)  Weaver is not the only software orchestration technique with this 
property.  The simplicity of this API is one reason I recommend software 
orchestration techniques that take dependency mediation out of the 
client's hands.  I hope that with coming work on HOT we can get OpenStack 
to this level of API simplicity.  But that struggle lies farther down the 
roadmap... 

Thanks, 
Mike 

"Yathiraj Udupi (yudupi)"  wrote on 10/07/2013 11:10:20 
PM: 
> 
> Hi, 
> 
> Based on the discussions we have had in the past few scheduler sub-
> team meetings,  I am sharing a document that proposes an updated 
> Instance Group Model and API extension model. 
> This is a work-in-progress draft version, but sharing it for early 
feedback. 
> https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing 
> 
> This model support generic instance types, where an instance can 
> represent a virtual node of any resource type.  But in the context 
> of Nova, an instance refers to the VM instance. 
> 
> This builds on the existing proposal for Instance Group Extension as
> documented here in this blueprint:  https://
> blueprints.launchpad.net/nova/+spec/instance-group-api-extension 
> 
> Thanks, 
> Yathi. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

In addition to the other questions below, I was wondering if you could 
explain why you included all those integer IDs; aren't the UUIDs 
sufficient?

Thanks,
Mike



From:   Mike Spreitzer/Watson/IBM@IBMUS
To: "Yathiraj Udupi (yudupi)" , 
Cc: OpenStack Development Mailing List 

Date:   10/08/2013 12:41 AM
Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource 
Placement - Updated Instance Group Model and API extension model - WIP 
Draft



Thanks.  I have a few questions.  First, I am a bit stymied by the style 
of API documentation used in that document and many others: it shows the 
first line of an HTTP request but says nothing about all the other 
details.  I am sure some of those requests must have interesting bodies, 
but I am not always sure which ones have a body at all, let alone what 
goes in it.  I suspect there may be some headers that are important too. 
Am I missing something? 

That draft says the VMs are created before the group.  Is there a way 
today to create a VM without scheduling it? 

As I understand your draft, it lays out a three phase process for a client 
to follow: create resources without scheduling or activating them, then 
arrange them into groups, then schedule & activate them.  By "activate" I 
mean, for a VM instance, to start running it.  That ordering must hold 
independently for each resource.  Activations are invoked by the client in 
an order that is consistent with (a) runtime dependencies that are 
mediated directly by the client (e.g., string slinging in the heat engine) 
and (b) the nature of the resources (for example, you  can not attach a 
volume to a VM instance until after both have been created).  Other than 
those considerations, the ordering and/or parallelism is a degree of 
freedom available to the client.  Have I got this right? 

Couldn't we simplify this into a two phase process: create groups and 
resources with scheduling, then activate the resources in an acceptable 
order? 

FYI: my group is using Weaver as the software orchestration technique, so 
there are no runtime dependencies that are mediated directly by the 
client.  The client sees a very simple API: the client presents a 
definition of all the groups and resources, and the service first 
schedules it all then activates in an acceptable order.  (We already have 
something in OpenStack that can do activations in an acceptable order, 
right?)  Weaver is not the only software orchestration technique with this 
property.  The simplicity of this API is one reason I recommend software 
orchestration techniques that take dependency mediation out of the 
client's hands.  I hope that with coming work on HOT we can get OpenStack 
to this level of API simplicity.  But that struggle lies farther down the 
roadmap... 

Thanks, 
Mike 

"Yathiraj Udupi (yudupi)"  wrote on 10/07/2013 11:10:20 
PM: 
> 
> Hi, 
> 
> Based on the discussions we have had in the past few scheduler sub-
> team meetings,  I am sharing a document that proposes an updated 
> Instance Group Model and API extension model. 
> This is a work-in-progress draft version, but sharing it for early 
feedback. 
> https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing 
> 
> This model support generic instance types, where an instance can 
> represent a virtual node of any resource type.  But in the context 
> of Nova, an instance refers to the VM instance. 
> 
> This builds on the existing proposal for Instance Group Extension as
> documented here in this blueprint:  https://
> blueprints.launchpad.net/nova/+spec/instance-group-api-extension 
> 
> Thanks, 
> Yathi. ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft

[Mike Spreitzer]

Thanks.  I have a few questions.  First, I am a bit stymied by the style 
of API documentation used in that document and many others: it shows the 
first line of an HTTP request but says nothing about all the other 
details.  I am sure some of those requests must have interesting bodies, 
but I am not always sure which ones have a body at all, let alone what 
goes in it.  I suspect there may be some headers that are important too. 
Am I missing something?

That draft says the VMs are created before the group.  Is there a way 
today to create a VM without scheduling it?

As I understand your draft, it lays out a three phase process for a client 
to follow: create resources without scheduling or activating them, then 
arrange them into groups, then schedule & activate them.  By "activate" I 
mean, for a VM instance, to start running it.  That ordering must hold 
independently for each resource.  Activations are invoked by the client in 
an order that is consistent with (a) runtime dependencies that are 
mediated directly by the client (e.g., string slinging in the heat engine) 
and (b) the nature of the resources (for example, you  can not attach a 
volume to a VM instance until after both have been created).  Other than 
those considerations, the ordering and/or parallelism is a degree of 
freedom available to the client.  Have I got this right?

Couldn't we simplify this into a two phase process: create groups and 
resources with scheduling, then activate the resources in an acceptable 
order?

FYI: my group is using Weaver as the software orchestration technique, so 
there are no runtime dependencies that are mediated directly by the 
client.  The client sees a very simple API: the client presents a 
definition of all the groups and resources, and the service first 
schedules it all then activates in an acceptable order.  (We already have 
something in OpenStack that can do activations in an acceptable order, 
right?)  Weaver is not the only software orchestration technique with this 
property.  The simplicity of this API is one reason I recommend software 
orchestration techniques that take dependency mediation out of the 
client's hands.  I hope that with coming work on HOT we can get OpenStack 
to this level of API simplicity.  But that struggle lies farther down the 
roadmap...

Thanks,
Mike

"Yathiraj Udupi (yudupi)"  wrote on 10/07/2013 11:10:20 
PM:
> 
> Hi, 
> 
> Based on the discussions we have had in the past few scheduler sub-
> team meetings,  I am sharing a document that proposes an updated 
> Instance Group Model and API extension model. 
> This is a work-in-progress draft version, but sharing it for early 
feedback. 
> https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing 
> 
> This model support generic instance types, where an instance can 
> represent a virtual node of any resource type.  But in the context 
> of Nova, an instance refers to the VM instance. 
> 
> This builds on the existing proposal for Instance Group Extension as
> documented here in this blueprint:  https://
> blueprints.launchpad.net/nova/+spec/instance-group-api-extension 
> 
> Thanks,
> Yathi. 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

Do not worry about what I want, right now I am just trying to understand 
the Climate proposal, wrt virtual resources (Patrick helped a lot on the 
physical side).  Can you please walk through a scenario involving Climate 
reservations on virtual resources?  I mean from start to finish, outlining

which party makes which decision, based on what.

Thanks,
Mike



From:   Sylvain Bauza 
To: OpenStack Development Mailing List 
, 
Cc: Mike Spreitzer/Watson/IBM@IBMUS
Date:   10/07/2013 05:07 AM
Subject:Re: [openstack-dev] [Climate] Questions and comments



Hi Mike,

Dina and you outlined some differences in terms of seeing what is 
dependent on what. 
As Dina explained, Climate plans to be integrated into Nova and Heat 
logics, where Heat and Nova would request Climate API by asking for a 
lease and would tag on their own the resources as 'RESERVED'.
On your point, and correct me if I'm wrong, you would rather see Climate 
on top of Heat and Nova, scheduling resources on its own, and only send 
creation requests to Heat and Nova. 

I'm happy to say both of you are right : Climate aims to be both called by

Nova and *also* calling Nova. That's just matter of what Climate *is*. And

here is the confusion.

That's why Climate is not only one API endpoint. It actually have two 
distinct endpoints : one called the Lease API endpoint, and one called the

Resource Reservation API endpoint.

As a Climate developer working on physical hosts reservations (and not 
Heat stacks), my concern is to be able to guarantee to a REST client 
(either a user or another service) that if this user wants to provision X 
hosts on a specific timeframe in the future (immediate or in 10 years), 
Climate will be able to provision them. By meaning "being able" and 
"guarantee", I do use strong words for stating that we engage ourselves to

be able to plan what will be resources capacity state in the future.

This decision-making process (ie. this "Climate scheduler") will be 
implemented as RPC Service for the Reservation API, and thus will needs to

keep its own persistence layer in Climate. Of course, it will request the 
Lease API for really creating the lease and managing lease start/end 
hooks, that's the Lease API job.


Provided you would want to use the Reservation API for "reserving" Heat 
stacks, you would have to implement it tho.


Thanks,
-Sylvain

Le 06/10/2013 20:41, Mike Spreitzer a écrit :
Thanks, Dina.  Yes, we do not understand each other; can I ask some more 
questions? 

You outlined a two-step reservation process ("We assume the following 
reservation process for the OpenStack services..."), and right after that 
talked about changing your mind to use Heat instead of individual 
services.  So I am confused, I am not sure which of your remarks reflect 
your current thinking and which reflect old thinking.  Can you just state 
your current thinking? 

On what basis would Climate decide to start or stop a lease?  What sort of

event notifications would Climate be sending, and when and why, and what 
would subscribers do upon receipt of such notifications? 

If the individual resource services continue to make independent 
scheduling decisions as they do today, what value does Climate add? 

Maybe a little more detailed outline of what happens in your current 
thinking, in support of an explicitly stated use case that shows the 
value, would help here. 

Thanks, 
Mike 

_
__
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

Thanks, Dina.  Yes, we do not understand each other; can I ask some more 
questions?

You outlined a two-step reservation process ("We assume the following 
reservation process for the OpenStack services..."), and right after that 
talked about changing your mind to use Heat instead of individual 
services.  So I am confused, I am not sure which of your remarks reflect 
your current thinking and which reflect old thinking.  Can you just state 
your current thinking?

On what basis would Climate decide to start or stop a lease?  What sort of 
event notifications would Climate be sending, and when and why, and what 
would subscribers do upon receipt of such notifications?

If the individual resource services continue to make independent 
scheduling decisions as they do today, what value does Climate add?

Maybe a little more detailed outline of what happens in your current 
thinking, in support of an explicitly stated use case that shows the 
value, would help here.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Climate] Questions and comments

[Mike Spreitzer]

I looked at the blueprint (
https://blueprints.launchpad.net/heat/+spec/stacks-reservation) and 
associated wiki page (https://wiki.openstack.org/wiki/Heat/Reservation), 
and I have a few comments/questions.

The wiki page has some remarks that are Nova-centric, and some other 
remarks that emphasize that Climate is not just about Nova, and I do not 
understand the relationship between these remarks.  Is this a roadmapping 
thought (start with just Nova, expand to other services later), or the 
inclusion of some details (related to Nova) and omission of other similar 
details (related to the other services), or what?

Will Climate be an independent service, or part of Nova, or what?

What will be the atomic operations?  I presume the primary interesting 
operation will be something like reserving a bag of resources, where that 
bag is allowed to contain any mixture of resources from any services. Have 
I got that right?

What exactly does reserving a resource mean?  Does this atomic reservation 
operation include some atomic cooperation from the resources' services' 
schedulers (e.g., Nova scheduler, Cinder scheduler, etc)?  Or is this 
reservation service logically independent of the resources' primary 
schedulers?  Overall I am getting the suggestion that reservation is an 
independent service.  The flow is something like first reserve a bag of 
resources, and then proceed to use them at your leisure.  But I also 
suppose that the important thing about a reservation is that it includes 
the result of scheduling (placement) --- the point of a reservation is 
that it is holding capacity to host the reserved resources.  You do not 
want an atomic operation to take a long time; do the scheduling decisions 
get made (tentatively, of course) before the real atomic section, with 
re-schedule and re-try on conflict detection, or is scheduling included in 
the atomic section?

For how long does a reservation last?  What sort of thing owns a 
reservation?  I suppose one important use case is that a heat engine, or 
the heat service (in the multi-engine world), could own a reservation; in 
this use case, the reservation would last until heat releases it. 
Hopefully heat will persist its reservation information, so that a process 
crash will not cause a dangling reservation; how will you enable complete 
avoidance of timing splinters (e.g., heat engine crashes after making 
reservation but before persisting information about it)?  Presumably other 
things besides heat could own reservations.

Is this primarily about planning for the distant future, or focused on the 
immediate future?  If a bag of resources (including their backing 
capacities) is reserved for a period that starts more than a little while 
in the future, what is done with that backing capacity in the meantime?

I see use cases for immediate future reservations; do you have use cases 
for more distant reservations?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse (now featuring software orchestration)

[Mike Spreitzer]

FYI, I have refined my pictures at 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and 
https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g 
to hopefully make it clearer that I agree with the sentiment that holistic 
infrastructure scheduling should not be part of heat but is closely 
related, and to make a graphical illustration of why I prefer the ordering 
of functionality that I do (the boundary between software and 
infrastructure issues gets less squiggly).

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] [heat] Policy specifics (for holistic infrastructure scheduling)

[Mike Spreitzer]

Clint Byrum  wrote on 10/01/2013 02:38:53 AM:

> From: Clint Byrum 
> To: openstack-dev , 
> Date: 10/01/2013 02:40 AM
> Subject: Re: [openstack-dev] [scheduler] [heat] Policy specifics 
> (for holistic infrastructure scheduling)
> 
> Mike, this has been really fun, but it is starting to feel like a
> rabbit hole.
> 
> The case for having one feels legitimate. However, at this point, I 
think
> someone will need to actually build it, or the idea is just a pipe 
dream.

Yes, Clint, I and colleagues are interested in building it.  I think Debo 
and Yathi are too.  And we are finding intersections with other projects. 
I am still new here and learning lots, so outputs have not come as fast as 
I had initially hoped.  But I appreciate being able to have discussions 
and draw on the wisdom of the group.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [scheduler] blueprint for host/hypervisor location information

[Mike Spreitzer]

Maybe the answer is hiding in plain sight: host aggregates.  This is a 
concept we already have, and it allows identification of arbitrary 
groupings for arbitrary purposes.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [scheduler] blueprint for host/hypervisor location information

[Mike Spreitzer]

Alex Glikson  wrote on 09/29/2013 03:30:35 PM:

> Mike Spreitzer  wrote on 29/09/2013 08:02:00 PM:
> 
> > Another reason to prefer host is that we have other resources to 
> > locate besides compute. 
> 
> Good point. Another approach (not necessarily contradicting) could 
> be to specify the location as a property of host aggregate rather 
> than individual hosts (and introduce similar notion in Cinder, and 
> maybe Neutron). This could be an evolution/generalization of the 
> existing 'availability zone' attribute, which would specify a more 
> fine-grained location path (e.g., 
> 'az_A:rack_R1:chassis_C2:node_N3'). We briefly discussed this 
> approach at the previous summit (see 'simple implementation' under 
> https://etherpad.openstack.org/HavanaTopologyAwarePlacement) -- but 
> unfortunately I don't think we made much progress with the actual 
> implementation in Havana (would be good to fix this in Icehouse). 

Thanks for the background.  I can still see the etherpad, but the old 
summit proposal to which it points is gone.

The etherpad proposes an API, and leaves open the question of whether it 
backs onto a common service.  I think that is a key question.  In my own 
group's work, this sort of information is maintained in a shared database. 
 I'm not sure what is the right approach for OpenStack.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] [heat] Policy specifics (for holistic infrastructure scheduling)

[Mike Spreitzer]

OK, let's take the holistic infrastructure scheduling out of Heat.  It 
really belongs at a lower level anyway.  Think of it as something you slap 
on top of Nova, Cinder, Neutron, etc. and everything that is going to use 
them goes first through the holistic scheduler, to give it a chance to 
make some joint decisions.  Zane has been worried about conflicting 
decisions being made, but if everything goes through the holistic 
infrastructure scheduling service then there does not need to be an issue 
with other parallel decision-making services (more on this below).  For a 
public cloud, think of this holistic infrastructure scheduling as part of 
the service that the cloud offers to the public; the public says what it 
wants, and the various levels of schedulers work on delivering it; the 
internals are not exposed to the public.  For example, a cloud user may 
say "spread my cluster across at least two racks, not too unevenly"; you 
do not want that public cloud customer to be in the business of knowing 
how many racks are in the cloud, knowing how much each one is currently 
being used, and picking which rack will contain which members of his 
cluster.  For a private cloud, the holistic infrastructure scheduler 
should have the same humility as the lower schedulers: offer enough 
visibility and control to the clients that they can make decisions if they 
want to (thus, nobody needs to "go around" the holistic infrastructure 
scheduler if they already know what they want).

You do not want to ask the holistic infrastructure scheduler to schedule 
resources one by one; you want to ask it to allocate a whole 
pattern/template/topology.  There is thus no need for infrastructure 
orchestration prior to holistic infrastructure scheduling.

Once the holistic infrastructure scheduler has done its job, there is a 
need for infrastructure orchestration.  What should we use for that?

OK, more on the business of conflicting decisions.  For the sake of 
scalability and modularity, the holistic infrastructure scheduler should 
delegate as much decision-making as it can to more specific services.  The 
job of the holistic infrastructure scheduler is to make joint decisions 
when there are strong interactions between services.  You can fudge this 
either way (have the holistic infrastructure scheduler make more or less 
decisions than ideal), but if you want the best then I think the principle 
I stated is what would guide.  So what if a delegated decision conflicts 
with a holistic decision?  Don't do that.  Divide the decision-making 
responsibilities into distinct domains, for example with the holistic 
scheduler making relatively big-picture decisions and individual resource 
services filling in the details.

That said, there can still be nasty surprises from lower layers.  Even if 
the design has carefully partitioned decision-making responsibilities, 
irregular things can still happen (e.g., authorized people can do 
something unexpected).  Even if nothing intentionally does anything 
irregular, there remains the possibility of bugs.  The holistic 
infrastructure scheduler should be prepared for nasty surprises, and 
getting information that is as authoritative as possible to begin with 
(promptness doesn't hurt either).

Then there is the question of the scalability of the holistic 
infrastructure scheduler.  One hard kernel of that is solving the 
optimization problem.  Nobody should expect the scheduler to find the 
truly optimal solution; this is an NP-hard problem.  However, there exist 
optimization algorithms that produce pretty good approximations in modest 
amounts of time.  Additionally: if the patterns are small relative to the 
size of the whole zone being scheduled then it should be possible to do 
concurrent decision-making with optimistic concurrency control (as Clint 
has mentioned).

You would not want one holistic infrastructure scheduler for a whole 
geographically distributed cloud.  You could use a hierarchical 
arrangement, with one top-level decision-maker dividing a pattern between 
availability zones (by which I mean the sort of large independent domains 
that are typically known by that term) and then a subsidiary scheduler for 
each availability zone.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [scheduler] blueprint for host/hypervisor location information

[Mike Spreitzer]

Monty Taylor  wrote on 09/29/2013 01:38:26 PM:

> On 09/29/2013 01:02 PM, Mike Spreitzer wrote:
> > Robert Collins  wrote on 09/29/2013 
02:21:28 AM:
> > 
> >> Host not hypervisor I think; consider nova baremetal, where 
hypervisor
> >> == machine that runs tftpd and makes IPMI calls, and host == place
> >> where the user workload will execute.
> > 
> > In nova baremetal, is there still a "hypervisor" in the picture, and 
is
> > it necessarily the same machine as the "host"?
> 
> There is one or more machiens where nova-compute runs. Those machines
> are necessarily _not_ the same machine as the "host".

So the host is the bare metal machine where the user's image is 
instantiated and run, and some other machine runs nova-compute to help set 
that up, right?

When I do a GET on http://novahost:port/v2/tenantid/servers/instanceid 
today, in a Grizzly installation running VM instances on KVM hypervisors, 
I get back a bunch of attributes, including hostId (whose value is a long 
hex string), OS-EXT-SRV-ATTR:host (whose value is a short name), and 
OS-EXT-SRV-ATTR:hypervisor_hostname (whose value is the same short name as 
OS-EXT-SRV-ATTR). Those short names are the same ones appearing in the 
reply to a GET of http://novahost:port/v2/tenantid/os-hosts and also in 
the reply to a GET of http://novahost:port/v2/tenantid/os-hypervisors.  In 
the case of baremetal, will a GET of 
http://novahost:port/v2/tenantid/os-hypervisors return things related to 
baremetal and, if so, which ones (the nova-compute machines or the hosts)?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [scheduler] blueprint for host/hypervisor location information

[Mike Spreitzer]

Robert Collins  wrote on 09/29/2013 02:21:28 
AM:

> Host not hypervisor I think; consider nova baremetal, where hypervisor
> == machine that runs tftpd and makes IPMI calls, and host == place
> where the user workload will execute.

In nova baremetal, is there still a "hypervisor" in the picture, and is it 
necessarily the same machine as the "host"?

Another reason to prefer host is that we have other resources to locate 
besides compute.

But the current API maps a host to a list of uniformly-shaped contents, it 
is not obvious to me what would be a good way to extend this.  Any ideas? 
Following is an example, it is the result of a GET on 
http://novahost:port/v2/tennantid/os-hosts/hostname


1.  {
2.  "host":
3.  [
4.  {
5.  "resource":
6.  {
7.  "project": "(total)",
8.  "memory_mb": 96661,
9.  "host": "x3630r7n8",
10. "cpu": 32,
11. "disk_gb": 2216
12. }
13. },
14. {
15. "resource":
16. {
17. "project": "(used_now)",
18. "memory_mb": 70144,
19. "host": "x3630r7n8",
20. "cpu": 34,
21. "disk_gb": 880
22. }
23. },
24. {
25. "resource":
26. {
27. "project": "(used_max)",
28. "memory_mb": 69632,
29. "host": "x3630r7n8",
30. "cpu": 34,
31. "disk_gb": 880
32. }
33. },
34. {
35. "resource":
36. {
37. "project": "5e5e2b0da114499b838c8d24c31bea08",
38. "memory_mb": 69632,
39. "host": "x3630r7n8",
40. "cpu": 34,
41. "disk_gb": 880
42. }
43. }
44. ]
45. }___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [scheduler] blueprint for host/hypervisor location information

[Mike Spreitzer]

I have begun drafting a blueprint about more detailed host/hypervisor 
location information, to support the sort of policy-informed placement 
decision-making that Debo, Yathi, and I have been talking about.  The 
blueprint is at 
https://blueprints.launchpad.net/nova/+spec/hypervisor-location-attribute 
and the details are at 
https://wiki.openstack.org/wiki/Nova/HypervisorExtendedAttributes

You see I am a little schizophrenic here about scope.  The blueprint is 
named quite narrowly, and the details page is named more broadly; this is 
because I am not sure what else you folks will chime in with.

I am not sure whether this information should really be attached to a 
hypervisor or to a host.  I proposed hypervisor because currently the 
details for a hypervisor are a map (easily extended) whereas the details 
for a host are currently a list of uniformly-typed contents (not so easily 
extended).  But host might actually be more appropriate.  I am looking for 
feedback on whether/how to go that way.  BTW, where would I find 
documentation on host details?  The current page on nova extensions (
http://docs.openstack.org/api/openstack-compute/2/content/ext-compute.html
) is lacking most of them.

You will see that I have proposed what the API looks like, but not the 
source of this additional information.  I will ask my colleagues who have 
something like this locally, how they got it done and what they would 
recommend to OpenStack.  Perhaps you good folks have some suggestions.  Is 
there obviously one way to do it?  Is it obvious that there can be no one 
way and so a plug point is required?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] [heat] Policy specifics

[Mike Spreitzer]

Zane also raised an important point about value.  Any scheduler is serving 
one master most directly, the cloud provider.  Any sane cloud provider has 
some interest in serving the interests of the cloud users, as well as 
having some concerns of its own.  The way my group has resolved this is in 
the translation from the incoming requests to the underlying optimization 
problem that is solved for placement; in that translation we fold in the 
cloud provider's interests as well as the cloud user's.  We currently have 
a fixed opinion of the cloud provider's interests; generalizing that is a 
possible direction for future progress.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] [heat] Policy specifics

[Mike Spreitzer]

Clint Byrum  wrote on 09/27/2013 11:58:16 AM:

> From: Clint Byrum 
> To: openstack-dev , 
> Date: 09/27/2013 12:01 PM
> Subject: Re: [openstack-dev] [scheduler] [heat] Policy specifics
> 
...
> > Mike,
> > These are not the kinds of specifics that are of any help at all in 
> > figuring out how (or, indeed, whether) to incorporate holistic 
> > scheduling into OpenStack.
> 
> I agree that the things in that page are a wet dream of logical 
deployment
> fun. However, I think one can target just a few of the basic ones,
> and see a real achievable case forming. I think I grasp Mike's ideas,
> so I'll respond to your concerns with what I think. Note that it is
> highly likely I've gotten some of this wrong.

It remains to be seen whether those things can be anything more than a wet 
dream for OpenStack, but they are running code elsewhere, so I have hope. 
What I wrote is pretty much a dump of what we have.  The exception is the 
network bandwidth stuff, which our holistic infrastructure scheduler 
currently ignores because we do not have a way to get the relevant 
capacity information from the physical infrastructure.  Part of the agenda 
here is to nudge Neutron to improve in that way.

> > - What would a holistic scheduling service look like? A standalone 
> > service? Part of heat-engine?
> 
> I see it as a preprocessor of sorts for the current infrastructure 
engine.
> It would take the logical expression of the cluster and either turn
> it into actual deployment instructions or respond to the user that it
> cannot succeed. Ideally it would just extend the same Heat API.

My own expectation is that it would be its own service, preceding 
infrastructure orchestration in the flow.  Alternatively, we could bundle 
holistic infrastructure scheduling, infrastructure orchestration, and 
software orchestration preparation together under one API but still 
maintained as fairly separate modules of functionality.  Or various in 
between ideas.  I do not yet have a strong reason for one choice over 
another.  I have been looking to gain cluefulness from discussion with you 
folks.

> > - How will the scheduling service reserve slots for resources in 
advance 
> > of them being created? How will those reservations be accounted for 
and 
> > billed?
> > - In the event that slots are reserved but those reservations are not 
> > taken up, what will happen?
> 
> I dont' see the word "reserve" in Mike's proposal, and I don't think 
this
> is necessary for the more basic models like Collocation and 
Anti-Collocation.
> 
> Reservations would of course make the scheduling decisions more likely 
to
> succeed, but it isn't necessary if we do things optimistically. If the
> stack create or update fails, we can retry with better parameters.

The raw truth of the matter is that even Nova has this problem already. 
The real ground truth of resource usage is in the hypervisor, not Nova. 
When Nova makes a decision, it really is provisional until confirmed by 
the hypervisor.  I have heard of cases, in different cloud software, where 
the thing making the placement decisions does not have a truly accurate 
picture of the resource usage.  These are typically caused by corner cases 
in failure scenarios, where the decision maker thinks something did not 
happen or was successfully deleted but in reality there is a zombie left 
over consuming some resources in the hypervisor.  There are probably cases 
where this can happen in OpenStack too, I am guessing.  Also, OpenStack 
does not prevent someone from going around Nova and directly asking a 
hypervisor to do something.

> > - Once scheduled, how will resources be created in their proper slots 
as 
> > part of a Heat template?
> 
> In goes a Heat template (sorry for not using HOT.. still learning it. ;)
> 
> Resources:
>   ServerTemplate:
> Type: Some::Defined::ProviderType
>   HAThing1:
> Type: OS::Heat::HACluster
> Properties:
>   ClusterSize: 3
>   MaxPerAZ: 1
>   PlacementStrategy: anti-collocation
>   Resources: [ ServerTemplate ]
> 
> And if we have at least 2 AZ's available, it feeds to the heat engine:
> 
> Resources:
>   HAThing1-0:
> Type: Some::Defined::ProviderType
>   Parameters:
> availability-zone: zone-A
>   HAThing1-1:
> Type: Some::Defined::ProviderType
>   Parameters:
> availability-zone: zone-B
>   HAThing1-2:
> Type: Some::Defined::ProviderType
>   Parameters:
> availability-zone: zone-A
> 
> If not, holistic scheduler says back "I don't have enough AZ's to
> satisfy MaxPerAZ".

Actually, I was thinking something even simpler (in the simple cases :-). 
By simple cases I mean where the holistic infrastructure scheduler makes 
all the placement decisions.  In that case, it only needs to get Nova to 
implement the decisions already made.  So the API call or template 
fragment for a VM instance would include an AZ parameter that specifies 
the particular host already chosen for that VM insta

Re: [openstack-dev] [scheduler] [heat] Policy specifics

[Mike Spreitzer]

Stephen Gran  wrote on 09/27/2013 10:46:09 
AM:

> If the admins of the openstack install wanted users to be able to select 

> placement by rack, surely the availability zones would be rack1 - rack5 
> ?  In this case, the user would write:
> 
>"Resources" : {
>  "MyASG" : {
>"Type" : "AWS::AutoScaling::AutoScalingGroup",
>"Properties" : {
>  "AvailabilityZones" : { "Fn::GetAZs" : ""},
>  "MinSize" : "2",
>"DesiredSize": "2",
>  "MaxSize" : "2",
>}
>  },
> 
> This should naturally implement placement as spread evenly across AZs.

You have added that "DesiredSize" property, to convey the idea of 
spreading across at least some number (2 in this case) of AZs, right? That 
is, it is not functionality in today's Nova, rather something we could 
add.

What if the cloud in question has several levels of structure available, 
and the admins want users to be able to spread at any of the available 
levels?

> I think maybe this is where I think my disagreement is.  Heat should be 
> able to express a user preference for placement, but only within the 
> bounds of the policy already created by the admins of the nova install. 
>   To have Heat have more knowledge than what is available via the nova 
> API seems overcomplicated and fragile to me.  If the nova API should 
> grow some extensions to make more complicated placement algorithms 
> available, then that's an argument that might have legs.

I am trying to find a way to introduce holistic infrastructure scheduling, 
whose purpose in life is to do what Nova, Cinder, etc can not do on their 
own.  (Making Nova, Cinder, and friends more capable on their own is also 
good, and complements what I am advocating.)  Yes, this requires some more 
visibility and control from those individual services.  Those needs have 
come up in vague ways in the discussion so far, and I plan to write 
something specific.

I am distinctly NOT advocating mixing holistic infrastructure scheduling 
up with infrastructure orchestration.  I think holistic infrastructure 
scheduling happens prior to infrastructure orchestration.  The more 
debatable question is the ordering with respect to the preparatory stage 
of software orchestration.  Another complicating factor is what happens 
when the infrastructure orchestration calls out to something that makes a 
nested stack.

Regards,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse (now featuring software orchestration)

[Mike Spreitzer]

Sorry, I was a bit too hasty in writing the last part of my last message; 
I forgot to qualify "software orchestration" to indicate I am speaking 
only of its preparatory phase.  I should have written:

Zane Bitter  wrote on 09/27/2013 08:24:49 AM:

...
> If I understood your remarks correctly, we agree that there is no 
> (known) reason that the scheduling has to occur in the middle of 
> orchestration (which would have implied that it needed to be 
> incorporated in some sense into Heat). 

If you agree that by orchestration you meant specifically infrastructure 
orchestration then we are agreed.  If software orchestration preparation 
is also in the picture then I also agree that holistic infrastructure 
scheduling does not *have to* go in between software orchestration 
preparation and infrastructure orchestration --- but I think that's a 
pretty good place for it.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse (now featuring software orchestration)

[Mike Spreitzer]

Zane Bitter  wrote on 09/27/2013 08:24:49 AM:

> Your diagrams clearly show scheduling happening in a separate stage to 
> (infrastructure) orchestration, which is to say that at the point where 
> resources are scheduled, their actual creation is in the *future*.
> 
> I am not a Climate expert, but it seems to me that they have a 
> near-identical problem to solve: how do they integrate with Heat such 
> that somebody who has reserved resources in the past can actually create 

> them (a) as part of a Heat stack or (b) as standalone resources, at the 
> user's option. IMO OpenStack should solve this problem only once.

If I understand correctly, what Climate adds to the party is planning 
allocations to happen at some specific time in the non-immediate future. A 
holistic infrastructure scheduler is planning allocations to happen just 
as soon as we can get the plans through the relevant code path, which is 
why I describe it as "now".


> If I understood your remarks correctly, we agree that there is no 
> (known) reason that the scheduling has to occur in the middle of 
> orchestration (which would have implied that it needed to be 
> incorporated in some sense into Heat).

If you agree that by orchestration you meant specifically infrastructure 
orchestration then we are agreed.  If software orchestration is also in 
the picture then I also agree that holistic infrastructure scheduling does 
not *have to* go in between software orchestration and infrastructure 
orchestration --- but I think that's a pretty good place for it.


> Right, so what I'm saying is that if all those things are _stated_ in 
> the input then there's no need to run the orchestration engine to find 
> out what they'll be; they're already stated.

Yep.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [scheduler] [heat] Policy specifics

[Mike Spreitzer]

Stephen Gran  wrote on 09/27/2013 04:26:37 
AM:

> Maybe I'm missing something obvious, but I'm not convinced all that 
> logic belongs in Heat.  I would expect nova and related components to 
> expose grouping information (availability zones in nova, networks in 
> quantum, etc) and for end users to supply the "group by" information.

Yes, this additional policy information is not intended to inform 
infrastructure orchestration.  It is intended to inform something that I 
have been calling holistic infrastructure scheduling and others have 
called things like unified resource placement and smart resource 
placement.  I frame it as an extension to Heat templates because this 
policy information needs to be added to a statement about a whole 
pattern/template/topology and Heat templates are the language we have for 
such things.  The idea is that holistic infrastructure scheduling comes 
before infrastructure orchestration; by the time infrastructure 
orchestration happens, the policy information has been handled and removed 
(or, possibly, encapsulated in some way for downstream processing --- but 
that's another story I am not trying to broach yet).

I have been discussing this outline here under the subject "Bringing 
things together for Icehouse" (
http://lists.openstack.org/pipermail/openstack-dev/2013-September/015118.html
), in the scheduler subgroup and heat weekly IRC meetings, and have a 
design summit proposal (http://summit.openstack.org/cfp/details/113).


> I think that your use case for anti-collocation (which is a very good 
> and important use case, don't get me wrong) is covered by using 
> availability zones/cells/regions and so on as they are, and doesn't 
> require much logic internal to Heat beyond obeying the constraint 
> specified by a user.

If there are five racks in the system and I want to say that two VMs 
should be placed on different racks, how do I say that with AZs without 
being overly specific?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [scheduler] [heat] Policy specifics

[Mike Spreitzer]

I have begun to draft some specifics about the sorts of policies that 
might be added to infrastructure to inform a smart unified placement 
engine.  These are cast as an extension to Heat templates.  See 
https://wiki.openstack.org/wiki/Heat/PolicyExtension .  Comments 
solicited.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

Debo, Yathi: I have read 
https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit?pli=1
 
and most of the referenced materials, and I have a couple of big-picture 
questions.  That document talks about making Nova call out to something 
that makes the sort of smart decisions you and I favor.  As far as I know, 
Nova is still scheduling one thing at a time.  How does that smart 
decision maker get a look at the whole pattern/termplate/topology as soon 
as it is needed?  I think you intend the smart guy gets it first, before 
Nova starts getting individual VM calls, right?  How does this picture 
grow to the point where the smart guy is making joint decisions about 
compute, storage, and network?  I think the key idea has to be that the 
smart guy gets a look at the whole problem first, and makes its decisions, 
before any individual resources are requested from 
nova/cinder/neutron/etc.  I think your point about "non-disruptive, works 
with the current nova architecture" is about solving the problem of how 
the smart guy's decisions get into nova.  Presumably this problem will 
occur for cinder and so on, too.  Have I got this right?

There is another way, right?  Today Nova accepts an 'availability zone' 
argument whose value can specify a particular host.  I am not sure about 
Cinder, but you can abuse volume types to get this job done.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse (now featuring software orchestration)

[Mike Spreitzer]

Clint wrote:

> There is a third stealth-objective that CFN has caused to linger in
> Heat. That is "packaging cloud applications". By allowing the 100%
> concrete CFN template to stand alone, users can "ship" the template.
> 
> IMO this marrying of software assembly, config, and orchestration is a
> concern unto itself, and best left outside of the core infrastructure
> orchestration system.

I favor separation of concerns.  I do not follow what you are suggesting 
about how to separate these particular concerns.  Can you elaborate?

Clint also wrote:

> A ruby DSL is not something I think is ever going to happen in
> OpenStack.

Ruby is particularly good when the runtime scripting is done through chef 
or puppet, which are based on Ruby.  For example, Weaver supports chef 
based scripting, and integrates in a convenient way.

A distributed system does not all have to be written in the same language.

Thomas wrote:

> I don't fully get this idea of HOT consuming a monolithic model produced 
by
> some compiler - be it Weaver or anything else.
> I thought the goal was to develop HOT in a way that users can actually
> write HOT, as opposed to having to use some "compiler" to produce some
> useful model.
> So wouldn't it make sense to make sure we add the right concepts to HOT 
to
> make sure we are able to express what we want to express and have things
> like composability, re-use, substitutability?

I am generally suspicious of analogies, but let me offer one here.  In the 
realm of programming languages, many have great features for modularity 
within one source file.  These features are greatly appreciated and used. 
But that does not stop people from wanting to maintain sources factored 
into multiple files.

Back to the world at hand, I do not see a conflict between (1) making a 
language for monoliths with sophisticated internal structure and (2) 
defining one or more languages for non-monolithic sources.

Thomas wrote:
> As said in my comment above, I would like to see us focusing on the
> agreement of one language - HOT - instead of yet another DSL.
> There are things out there that are well established (like chef or 
puppet),
> and HOT should be able to efficiently and intuitively use those things 
and
> orchestrate components built using those things.

Yes, it may be that our best tactic at this point is to allow multiple 
(2), some or all not defined through the OpenStack Foundation, while 
agreeing here on (1).

Thomas wrote:
> Anyway, this might be off the track that was originally discussed in 
this
> thread (i.e. holistic scheduling and so on) ...

We are engaged in a boundary-drawing and relationship-drawing exercise.  I 
brought up this idea of a software orchestration compiler to show why I 
think the software orchestration preparation stage is best done earlier 
rather than later.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Generalising racks :- modelling a datacentre

[Mike Spreitzer]

I agree that such a thing is useful for scheduling.  I see a bit of a 
tension here: for software engineering reasons we want some independence, 
but we also want to avoid wasteful duplication.

I think we are collectively backing into the problem of metamodeling for 
datacenters, and establishing one or more software thingies that will 
contain/communicate datacenter models.  A collection of "nodes" annotated 
with "tags" is a metamodel.  You could define a graph-based metamodel 
without mandating any particular graph shape.  You could be more 
prescriptive and mandate a tree shape as a good compromise between 
flexibility and making something that is reasonably easy to process.  We 
can debate what the metamodel should be, but that is different from 
debating whether there is a metamodel.

Regards,
Mike



From:   Tomas Sedovic 
To: openstack-dev@lists.openstack.org, 
Date:   09/25/2013 10:37 AM
Subject:Re: [openstack-dev] [TripleO] Generalising racks :- 
modelling a datacentre



On 09/25/2013 05:15 AM, Robert Collins wrote:
> One of the major things Tuskar does is model a datacenter - which is
> very useful for error correlation, capacity planning and scheduling.
>
> Long term I'd like this to be held somewhere where it is accessible
> for schedulers and ceilometer etc. E.g. network topology + switch
> information might be held by neutron where schedulers can rely on it
> being available, or possibly held by a unified topology db with
> scheduler glued into that, but updated by neutron / nova / cinder.
> Obviously this is a) non-trivial and b) not designed yet.
>
> However, the design of Tuskar today needs to accomodate a few things:
>   - multiple reference architectures for clouds (unless there really is
> one true design)
>   - the fact that today we don't have such an integrated vertical 
scheduler.
>
> So the current Tuskar model has three constructs that tie together to
> model the DC:
>   - nodes
>   - resource classes (grouping different types of nodes into service
> offerings - e.g. nodes that offer swift, or those that offer nova).
>   - 'racks'
>
> AIUI the initial concept of Rack was to map to a physical rack, but
> this rapidly got shifted to be 'Logical Rack' rather than physical
> rack, but I think of Rack as really just a special case of a general
> modelling problem..

Yeah. Eventually, we settled on Logical Rack meaning a set of nodes on 
the same L2 network (in a setup where you would group nodes into 
isolated L2 segments). Which kind of suggests we come up with a better 
name.

I agree there's a lot more useful stuff to model than just racks (or 
just L2 node groups).

>
>>From a deployment perspective, if you have two disconnected
> infrastructures, thats two AZ's, and two underclouds : so we know that
> any one undercloud is fully connected (possibly multiple subnets, but
> one infrastructure). When would we want to subdivide that?
>
> One case is quick fault aggregation: if a physical rack loses power,
> rather than having 16 NOC folk independently investigating the same 16
> down hypervisors, one would prefer to identify that the power to the
> rack has failed (for non-HA powered racks); likewise if a single
> switch fails (for non-HA network topologies) you want to identify that
> that switch is down rather than investigating all the cascaded errors
> independently.
>
> A second case is scheduling: you may want to put nova instances on the
> same switch as the cinder service delivering their block devices, when
> possible, or split VM's serving HA tasks apart. (We currently do this
> with host aggregates, but being able to do it directly would be much
> nicer).
>
> Lastly, if doing physical operations like power maintenance or moving
> racks around in a datacentre, being able to identify machines in the
> same rack can be super useful for planning, downtime announcements, 
orhttps://plus.google.com/hangouts/_/04919b4400b8c4c5ba706b752610cd433d9acbe1
> host evacuation, and being able to find a specific machine in a DC is
> also important (e.g. what shelf in the rack, what cartridge in a
> chassis).

I agree. However, we should take care not to commit ourselves to 
building a DCIM just yet.

>
> Back to 'Logical Rack' - you can see then that having a single
> construct to group machines together doesn't really support these use
> cases in a systematic fasion:- Physical rack modelling supports only a
> subset of the location/performance/failure use cases, and Logical rack
> doesn't support them at all: we're missing all the rich data we need
> to aggregate faults rapidly : power, network, air conditioning - and
> these things cover both single machine/groups of machines/racks/rows
> of racks scale (consider a networked PDU with 10 hosts on it - thats a
> fraction of a rack).
>
> So, what I'm suggesting is that we model the failure and performance
> domains directly, and include location (which is the incremental data
> racks add once failure and performance domain

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse (now featuring software orchestration)

[Mike Spreitzer]

Let me elaborate a little on my thoughts about software orchestration, and 
respond to the recent mails from Zane and Debo.  I have expanded my 
picture at 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
and added a companion picture at 
https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g 
that shows an alternative.

One of the things I see going on is discussion about better techniques for 
software orchestration than are supported in plain CFN.  Plain CFN allows 
any script you want in userdata, and prescription of certain additional 
setup elsewhere in cfn metadata.  But it is all mixed together and very 
concrete.  I think many contributors would like to see something with more 
abstraction boundaries, not only within one template but also the ability 
to have modular sources.

I work closely with some colleagues who have a particular software 
orchestration technology they call Weaver.  It takes as input for one 
deployment not a single monolithic template but rather a collection of 
modules.  Like higher level constructs in programming languages, these 
have some independence and can be re-used in various combinations and 
ways.  Weaver has a compiler that weaves together the given modules to 
form a monolithic model.  In fact, the input is a modular Ruby program, 
and the Weaver compiler is essentially running that Ruby program; this 
program produces the monolithic model as a side effect.  Ruby is a pretty 
good language in which to embed a domain-specific language, and my 
colleagues have done this.  The modular Weaver input mostly looks 
declarative, but you can use Ruby to reduce the verboseness of, e.g., 
repetitive stuff --- as well as plain old modularity with abstraction.  We 
think the modular Weaver input is much more compact and better for human 
reading and writing than plain old CFN.  This might not be obvious when 
you are doing the "hello world" example, but when you get to realistic 
examples it becomes clear.

The Weaver input discusses infrastructure issues, in the rich way Debo and 
I have been advocating, as well as software.  For this reason I describe 
it as an integrated model (integrating software and infrastructure 
issues).  I hope for HOT to evolve to be similarly expressive to the 
monolithic integrated model produced by the Weaver compiler.

In Weaver, as well as in some of the other software orchestration 
technologies being discussed, there is a need for some preparatory work 
before the infrastructure (e.g., VMs) is created.  This preparatory stage 
begins the implementation of the software orchestration abstractions. Here 
is the translation from something more abstract into flat userdata and 
other cfn metadata.  For Weaver, this stage also involves some 
stack-specific setup in a distinct coordination service.  When the VMs 
finally run their userdata, the Weaver-generated scripts there use that 
pre-configured part of the coordination service to interact properly with 
each other.

I think that, to a first-order approximation, the software orchestration 
preparatory stage commutes with holistic infrastructure scheduling.  They 
address independent issues, and can be done in either order.  That is why 
I have added a companion picture; the two pictures show the two orders.

My claim of commutativity is limited, as I and colleagues have 
demonstrated only one of the two orderings; the other is just a matter of 
recent thought.  There could be gotchas lurking in there.

Between the two orderings, I have a preference for the one I first 
mentioned and have experience with actually running.  It has the virtue of 
keeping related things closer together: the software orchestration 
compiler is next to the software orchestration preparatory stage, and the 
holistic infrastructure scheduling is next to the infrastructure 
orchestration.

In response to Debo's remark about flexibility: I am happy to see an 
architecture that allows either ordering if it turns out that they are 
both viable and the community really wants that flexibility.  I am not so 
sure we can totally give up on architecting where things go, but this 
level of flexibility I can understand and get behind (provided it works).

Just as a LP solver is a general utility whose uses do not require 
architecting, I can imagine a higher level utility that solves abstract 
placement problems.  Actually, this is not a matter of imagination.  My 
group has been evolving such a thing for years.  It is now based, as Debo 
recommends, on a very flexible and general optimization algorithm.  But 
the plumbing between it and the rest of the system is significant; I would 
not expect many users to take on that magnitude of task.

I do not really want to get into dogmatic fights over what gets labelled 
"heat".  I will leave the questions about which piece goes where in the 
OpenStack programs and projects to those more informed and anointed.  What 
I am trying t

[openstack-dev] Basic questions about climate

[Mike Spreitzer]

Climate is about reserving resources.  Are those physical resources or 
virtual ones?  Where was I supposed to read the answer to basic questions 
like that?

If climate is about reserving virtual resources, how is that different 
from "scheduling" them?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

Someone earlier asked for greater clarity about infrastructure 
orchestration, so here is my view.  I see two main issues: (1) deciding 
the order in which to do things, and (2) doing them in an acceptable 
order.  That's an oversimplified wording because, in general, some 
parallelism is possible.  In general, the set of things to do is 
constrained by a partial order --- and that partial order comes from two 
sources.  One is the nature of the downstream APIs.  For examples, you can 
not attach a volume or floating IP address to a VM until after both have 
been created.  The other source of ordering constraints is upstream 
decision makers.  Decisions made upstream are conveyed into today's heat 
engine by data dependencies between resources in a heat template.  The 
heat engine is not making those decisions.  It is not a source of 
important ordering constraints.  When the ordering constraints actually 
allow some parallelism --- they do not specify a total order --- the heat 
engine has freedom in which of that parallelism to exploit vs flatten into 
sequential ordering.  What today's heat engine does is make its available 
choices about that and issue the operations, keeping track of IDs and 
outcomes.  I have been using the term "infrastructure orchestration" to 
refer to this latter job (issuing infrastructure operations with 
acceptable ordering/parallelism), not the decision-making of upstream 
agents.  This might be confusing; I think the plain English meaning of 
"orchestration" suggests decision-making as well as execution.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

I was not trying to raise issues of geographic dispersion and other higher 
level structures, I think the issues I am trying to raise are relevant 
even without them.  This is not to deny the importance, or relevance, of 
higher levels of structure.  But I would like to first respond to the 
discussion that I think is relevant even without them.

I think it is valuable for OpenStack to have a place for holistic 
infrastructure scheduling.  I am not the only one to argue for this, but I 
will give some use cases.  Consider Hadoop, which stresses the path 
between Compute and Block Storage.  In the usual way of deploying and 
configuring Hadoop, you want each data node to be using directly attached 
storage.  You could address this by scheduling one of those two services 
first, and then the second with constraints from the first --- but the 
decisions made by the first could paint the second into a corner.  It is 
better to be able to schedule both jointly.  Also consider another 
approach to Hadoop, in which the block storage is provided by a bank of 
storage appliances that is equidistant (in networking terms) from all the 
Compute.  In this case the Storage and Compute scheduling decisions have 
no strong interaction --- but the Compute scheduling can interact with the 
network (you do not want to place Compute in a way that overloads part of 
the network).

Once a holistic infrastructure scheduler has made its decisions, there is 
then a need for infrastructure orchestration.  The infrastructure 
orchestration function is logically downstream from holistic scheduling. I 
do not favor creating a new and alternate way of doing infrastructure 
orchestration in this position.  Rather I think it makes sense to use 
essentially today's heat engine.

Today Heat is the only thing that takes a holistic view of 
patterns/topologies/templates, and there are various pressures to expand 
the mission of Heat.  A marquee expansion is to take on software 
orchestration.  I think holistic infrastructure scheduling should be 
downstream from the preparatory stage of software orchestration (the other 
stage of software orchestration is the run-time action in and supporting 
the resources themselves).  There are other pressures to expand the 
mission of Heat too.  This leads to conflicting usages for the word 
"heat": it can mean the infrastructure orchestration function that is the 
main job of today's heat engine, or it can mean the full expanded mission 
(whatever you think that should be).  I have been mainly using "heat" in 
that latter sense, but I do not really want to argue over naming of bits 
and assemblies of functionality.  Call them whatever you want.  I am more 
interested in getting a useful arrangement of functionality.  I have 
updated my picture at 
https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U 
--- do you agree that the arrangement of functionality makes sense?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [Openstack-devel] PGP key signing party during the HK summit

[Mike Spreitzer]

What's the threat model here?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

I have written a new outline of my thoughts, you can find it at 
https://docs.google.com/document/d/1RV_kN2Io4dotxZREGEks9DM0Ih_trFZ-PipVDdzxq_E

It is intended to stand up better to independent study.  However, it is 
still just an outline.  I am still learning about stuff going on in 
OpenStack, and am learning and thinking faster than I can write.  Trying 
to figure out how to cope.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Medium Availability VMs

[Mike Spreitzer]

> From: Tim Bell 
> ...
> Is this something that will be added into OpenStack or made 
> available as open source through something like stackforge ?

I and some others think that the OpenStack architecture should have a 
place for holistic infrastructure scheduling.  I also think this is an 
area where vendors will want to compete; I think my company has some 
pretty good technology for this and will want to sell it for money.  
https://wiki.openstack.org/wiki/Open requires that the free OpenStack 
includes a pretty good implementation of this function too, and I think 
others have some they want to contribute.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Medium Availability VMs

[Mike Spreitzer]

> From: Tim Bell 
> ...
> Discussing with various people in the community, there seems to be 
> interest in a way to
> 
> -  Identify when a hypervisor is being drained or is down 
> and inventory its VMs
> -  Find the best practise way of restarting that VM for 
> hypervisors still available
> o   Live migration
> o   Cold migration
> -  Defining policies for the remaining cases
> o   Restart from base image
> o   Suspend
> o   Delete
> 
> This touches multiple components from Nova/Cinder/Quantum (at minimum).
> 
> It also touches some cloud architecture questions if OpenStack can 
> start to move into the low hanging fruit parts of service consolidation.
> 
> I’d like to have some form of summit discussion in Hong Kong around 
> these topics but it is not clear where it fits.
> 
> Are there others who feel similarly ? How can we fit it in ?

When there are multiple viable choices, I think direction should be taken 
from higher layers.  The operation of draining a hypervisor can be 
parameterized, the VMs themselves can be tagged, by an indication of which 
to do.

I myself am working primarily on holistic infrastructure scheduling, which 
includes quiescing and draining hypervisors among the things it can do. 
Holistic scheduling works under the direction of a 
template/pattern/topology that describes a set of interacting resources 
and their relationships, and so is able to make a good decision about 
where VMs should move to.

Re-starting a VM can require software coordination.

I think holistic infrastructure scheduling is logically downstream from 
software coordination and upstream from infrastructure orchestration.  I 
think the ambitions for Heat are expanding to include the latter two, and 
so must also have something to do with holistic infrastructure scheduling.

Regards,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat

[Mike Spreitzer]

radix, thanks.  How exactly does the cooldown work?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat

[Mike Spreitzer]

I'd like to try to summarize this discussion, if nothing else than to see 
whether I have correctly understood it.  There is a lot of consensus, but 
I haven't heard from Adrian Otto since he wrote some objections.  I'll 
focus on trying to describe the consensus; Adrian's concerns are already 
collected in a single message.  Or maybe this is already written in some 
one place?

The consensus is that there should be an autoscaling (AS) service that is 
accessible via its own API.  This autoscaling service can scale anything 
describable by a snippet of Heat template (it's not clear to me exactly 
what sort of syntax this is; is it written up anywhere?).  The autoscaling 
service is stimulated into action by a webhook call.  The user has the 
freedom to arrange calls on that webhook in any way she wants.  It is 
anticipated that a common case will be alarms raised by Ceilometer.  For 
more specialized or complicated logic, the user is free to wire up 
anything she wants to call the webhook.

An instance of the autoscaling service maintains an integer variable, 
which is the current number of copies of the thing being autoscaled.  Does 
the webhook call provide a new number, or +1/-1 signal, or ...?

There was some discussion of a way to indicate which individuals to 
remove, in the case of decreasing the multiplier.  I suppose that would be 
an option in the webhook, and one that will not be exercised by Ceilometer 
alarms.

(It seems to me that there is not much "auto" in this autoscaling service 
--- it is really a scaling service driven by an external controller.  This 
is not a criticism, I think this is a good factoring --- but maybe not the 
best naming.)

The autoscaling service does its job by multiplying the heat template 
snippet (the thing to be autoscaled) by the current number of copies and 
passing this derived template to Heat to "make it so".  As the desired 
number of copies changes, the AS service changes the derived template that 
it hands to Heat.  Most commentators argue that the consistency and 
non-redundancy of making the AS service use Heat outweigh the extra 
path-length compared to a more direct solution.

Heat will have a resource type, analogous to 
AWS::AutoScaling::AutoScalingGroup, through which the template author can 
request usage of the AS service.

OpenStack in general, and Heat in particular, need to be much better at 
traceability and debuggability; the AS service should be good at these 
too.

Have I got this right?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] cross-stack references

[Mike Spreitzer]

My question is about stacks that are not nested.  Suppose, for example, 
that I create a stack that implements a shared service.  Later I create a 
separate stack that uses that shared service.  When creating that client 
stack, I would like to have a way of talking about its relationships with 
the service stack.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] cross-stack references

[Mike Spreitzer]

When we get into things like affinity concerns or managing network 
bandwidth, we see the need for cross-stack relationships.  You may want to 
place parts of a new stack near parts of an existing one, for example.  I 
see that in CFN you can make cross-references between different parts of a 
single stack using the resource names that appear in the original 
template.  Is there a way to refer to something that did not come from the 
same original template?  If not, won't we need such a thing to be 
introduced?  Any thoughts on how that would be done?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

Manuel, and others:
I am sorry, in the rush at the end of the scheduler meeting a critical 
fact flew from my mind: the material I distributed beforehand was intended 
as something I could reference during discussion in the meeting, I did not 
expect it to fully stand on its own.  Indeed, you have noticed that it 
does not.  It will take a little more time to write something that stands 
on its own.  I will try to get something out soon, including answers to 
your questions.

I should also make clear the overall sense of what I am doing.  I am in an 
in-between state.  My group has some running code on which I can report, 
but we are not satisfied with it for a few reasons.  One is that it is not 
integrated yet in any way with Heat, and I think the discussion we are 
having here overlaps with Heat.  Another is that it does not support very 
general changes, we have so far been solving initial deployment issues. We 
have been thinking about how to do better on these issues, and have an 
outline and are proceeding with the work; I can report on these too.  The 
things that concern me the most are issues of how to get architectural 
alignment with what the OpenStack community is doing.  So my main aim 
right now is to have a discussion of how the pieces fit together.  I am 
told that the OpenStack community likes to focus on small incremental 
changes, and that is a way to get things done, but I, at least, would like 
to get some sense of where this is going.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Change email address, or, why I can't use github and will I be able to submit patches?

[Mike Spreitzer]

> From: Anne Gentle 
> To: OpenStack Development Mailing List 
, 
> Date: 09/17/2013 05:51 PM
> Subject: Re: [openstack-dev] Change email address, or, why I can't 
> use github and will I be able to submit patches?
> 
> ...
> Github was experiencing issues earlier today. Nothing in our 
> GerritWorkflow requires ssh -T g...@github.com though. If you were 
> able to do a git clone, how did git review -s go for you? 

Both work.  So I guess I am in business.

Thanks!
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Change email address, or, why I can't use github and will I be able to submit patches?

[Mike Spreitzer]

I am working through the instructions at 
https://wiki.openstack.org/wiki/GerritWorkflow - and things are going OK, 
including installing ~/.ssh/id_rsa.pub at 
https://review.openstack.org/#/settings/ssh-keys, without any linebreaks 
in the middle nor at the end - except it fails at the point where I test 
my ability to use github:

mjs9:~ mspreitz$ git config --list
user.name=Mike
user.email=mspre...@us.ibm.com
core.editor=emacs

mjs9:~ mspreitz$ ssh -T g...@github.com
Warning: Permanently added the RSA host key for IP address 
'192.30.252.131' to the list of known hosts.
Permission denied (publickey).


What's going wrong here?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Change email address

[Mike Spreitzer]

Thanks Anne.  Since I have already signed the ICLA, my real question is 
about what has to be true on an on-going basis for me to do developer 
stuff like reviewing and submitting patches.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Change email address

[Mike Spreitzer]

Is it possible to change the email address I use in git and gerrit?  I 
think I started off with an inferior choice.  I have now taught LaunchPad 
and Gerrit that I have two email addresses.  The OpenStack Foundation 
appears a bit confused, but I'm hoping that's not critical.

I am stuck at the point on 
https://wiki.openstack.org/wiki/How_To_Contribute where it says, 
concerning signing the ICLA, "Your full name and E-mail address will be 
public (...) and the latter needs to match the user.email in your Git 
configuration".  Gerrit knows that I have signed the ICLA, and will not 
let me sign it again (I can not even try, it is grayed out).  Would it be 
correct to clarify the text I quoted above to say that one of your Gerrit 
email addresses has to match the one in your Git configuration?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

Fixed, sorry.




From:   Gary Kotton 
To: OpenStack Development Mailing List 
, 
Date:   09/17/2013 03:26 AM
Subject:Re: [openstack-dev] [heat] [scheduler] Bringing things 
together for Icehouse



Hi,
The document is locked.
Thanks
Gary

From: Mike Spreitzer 
Reply-To: OpenStack Development Mailing List <
openstack-dev@lists.openstack.org>
Date: Tuesday, September 17, 2013 8:00 AM
To: OpenStack Development Mailing List 
Subject: Re: [openstack-dev] [heat] [scheduler] Bringing things together 
for Icehouse

I have written a brief document, with pictures.  See 
https://docs.google.com/document/d/1hQQGHId-z1A5LOipnBXFhsU3VAMQdSe-UXvL4VPY4ps


Regards, 
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

I have written a brief document, with pictures.  See 
https://docs.google.com/document/d/1hQQGHId-z1A5LOipnBXFhsU3VAMQdSe-UXvL4VPY4ps

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tuskar] Tuskar Names Clarification & Unification

[Mike Spreitzer]

> From: Jaromir Coufal 
> To: openstack-dev@lists.openstack.org, 
> Date: 09/16/2013 11:51 AM
> Subject: Re: [openstack-dev] [Tuskar] Tuskar Names Clarification & 
Unification
> 
> Hi,
> 
> after few days of gathering information, it looks that no more new 
> ideas appear there, so let's take the last round of voting for names
> which you prefer. It's important for us to get on the same page.

I am concerned that the proposals around the term 'rack' do not recognize 
that there might be more than one layer in the organization.

Is it more important to get appropriately abstract and generic terms, or 
is the desire to match common concrete terms?

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] Program name and Mission statement

[Mike Spreitzer]

"data processing" is surely a superset of "big data".  Either, by itself, 
is way too vague.  But the wording that many people favor, which I will 
quote again, uses the vague term in a qualified way that makes it 
appropriately specific, IMHO.  Here is the wording again:

``To provide a simple, reliable and repeatable mechanism by which to 
deploy Hadoop and related Big Data projects, including management, 
monitoring and processing mechanisms driving further adoption of 
OpenStack.''

I think that saying "related Big Data projects" after "Hadoop" is fairly 
clear.  OTOH, I would not mind replacing "Hadoop and related Big Data 
projects" with "the Hadoop ecosystem".

Regards,
Mike

Matthew Farrellee  wrote on 09/16/2013 02:39:20 PM:

> From: Matthew Farrellee 
> To: OpenStack Development Mailing List 
, 
> Date: 09/16/2013 02:40 PM
> Subject: Re: [openstack-dev] [savanna] Program name and Mission 
statement
> 
> IMHO, Big Data is even more nebulous and currently being pulled in many 
> directions. Hadoop-as-a-Service may be too narrow. So, something in 
> between, such as Data Processing, is a good balance.
> 
> Best,
> 
> 
> matt
> 
> On 09/13/2013 08:37 AM, Abhishek Lahiri wrote:
> > IMHO data processing is too board , it makes more sense to clarify 
this
> > program as big data as a service or simply 
openstack-Hadoop-as-a-service.
> >
> > Thanks & Regards
> > Abhishek Lahiri
> >
> > On Sep 12, 2013, at 9:13 PM, Nirmal Ranganathan  > > wrote:
> >
> >>
> >>
> >>
> >> On Wed, Sep 11, 2013 at 8:39 AM, Erik Bergenholtz
> >> mailto:ebergenho...@hortonworks.com>>
> >> wrote:
> >>
> >>
> >> On Sep 10, 2013, at 8:50 PM, Jon Maron  >> > wrote:
> >>
> >>> Openstack Big Data Platform
> >>>
> >>>
> >>> On Sep 10, 2013, at 8:39 PM, David Scott
> >>>  >>> > wrote:
> >>>
>  I vote for 'Open Stack Data'
> 
> 
>  On Tue, Sep 10, 2013 at 5:30 PM, Zhongyue Luo
>  mailto:zhongyue@intel.com>> wrote:
> 
>  Why not "OpenStack MapReduce"? I think that pretty much 
says
>  it all?
> 
> 
>  On Wed, Sep 11, 2013 at 3:54 AM, Glen Campbell
>  mailto:g...@glenc.io>> wrote:
> 
>  "performant" isn't a word. Or, if it is, it means
>  "having performance." I think you mean 
"high-performance."
> 
> 
>  On Tue, Sep 10, 2013 at 8:47 AM, Matthew Farrellee
>  mailto:m...@redhat.com>> wrote:
> 
>  Rough cut -
> 
>  Program: OpenStack Data Processing
>  Mission: To provide the OpenStack community with an
>  open, cutting edge, performant and scalable data
>  processing stack and associated management 
interfaces.
> 
> >>
> >> Proposing a slightly different mission:
> >>
> >> To provide a simple, reliable and repeatable mechanism by which 
to
> >> deploy Hadoop and related Big Data projects, including 
management,
> >> monitoring and processing mechanisms driving further adoption of
> >> OpenStack.
> >>
> >>
> >>
> >> +1. I liked the data processing aspect as well, since EDP api 
directly
> >> relates to that, maybe a combination of both.
> >>
> 
> 
>  On 09/10/2013 09:26 AM, Sergey Lukjanov wrote:
> 
>  It sounds too broad IMO. Looks like we need to
>  define Mission Statement
>  first.
> 
>  Sincerely yours,
>  Sergey Lukjanov
>  Savanna Technical Lead
>  Mirantis Inc.
> 
>  On Sep 10, 2013, at 17:09, Alexander Kuznetsov
>    
>    >> wrote:
> 
>  My suggestion OpenStack Data Processing.
> 
> 
>  On Tue, Sep 10, 2013 at 4:15 PM, Sergey 
Lukjanov
>    
>    __>> wrote:
> 
>  Hi folks,
> 
>  due to the Incubator Application we
>  should prepare Program name
>  and Mission statement for Savanna, so, 
I
>  want to start mailing
>  thread about it.
> 
>  Please, provide any ideas here.
> 
>   

Re: [openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

> From: Gary Kotton 
> ...
> Can you please join us at the up and coming scheduler meeting. That 
> will give you a chance to bring up the idea's and discuss them with 
> a larger audience.

I will do so on Sep 17.  Later meetings still TBD.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [scheduler] (How to talk about) Bringing things together for Icehouse

[Mike Spreitzer]

As I mentioned the last time this was brought up, I already have a meeting 
series that conflicts with the scheduler group chats and will be hard to 
move; that is why I have been trying to participate asynchronously.  But 
since Gary asked again, I am seeing what I can do about that other meeting 
series.  Unless and until something gives, I will have to continue 
participating asynchronously.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] [scheduler] Bringing things together for Icehouse

[Mike Spreitzer]

I've read up on recent goings-on in the scheduler subgroup, and have some 
thoughts to contribute.

But first I must admit that I am still a newbie to OpenStack, and still am 
missing some important clues.  One thing that mystifies me is this: I see 
essentially the same thing, which I have generally taken to calling 
holistic scheduling, discussed in two mostly separate contexts: (1) the 
(nova) scheduler context, and (2) the ambitions for heat.  What am I 
missing?

I have read the Unified Resource Placement Module document (at 
https://docs.google.com/document/d/1cR3Fw9QPDVnqp4pMSusMwqNuB_6t-t_neFqgXA98-Ls/edit?pli=1#
) and NovaSchedulerPerspective document (at 
https://docs.google.com/document/d/1_DRv7it_mwalEZzLy5WO92TJcummpmWL4NWsWf0UWiQ/edit?pli=1#heading=h.6ixj0ctv4rwu
).  My group already has running code along these lines, and thoughts for 
future improvements, so I'll mention some salient characteristics.  I have 
read the etherpad at 
https://etherpad.openstack.org/IceHouse-Nova-Scheduler-Sessions - and I 
hope my remarks will help fit these topics together.

Our current code uses one long-lived process to make placement decisions. 
The information it needs to do this job is pro-actively maintained in its 
memory.  We are planning to try replacing this one process with a set of 
equivalent processes, not sure how well it will work out (we are a 
research group).

We make a distinction between desired state, target state, and observed 
state.  The desired state comes in through REST requests, each giving a 
full virtual resource topology (VRT).  A VRT includes constraints that 
affect placement, but does not include actual placement decisions.  Those 
are made by what we call the placement agent.  Yes, it is separate from 
orchestration (even in the first architecture figure in the u-rpm document 
the orchestration is separate --- the enclosing box does not abate the 
essential separateness).  In our architecture, orchestration is downstream 
from placement (as in u-rpm).  The placement agent produces target state, 
which is essentially desired state augmented by placement decisions. 
Observed state is what comes from the lower layers (Software Defined 
Compute, Storage, and Network).  We mainly use OpenStack APIs for the 
lower layers, and have added a few local extensions to make the whole 
story work.

The placement agent judges available capacity by subtracting current 
allocations from raw capacity.  The placement agent maintains in its 
memory a derived thing we call effective state; the allocations in 
effective state are the union of the allocations in target state and the 
allocations in observed state.  Since the orchestration is downstream, 
some of the planned allocations are not in observed state yet.  Since 
other actors can use the underlying cloud, and other weird sh*t happens, 
not all the allocations are in target state.  That's why placement is done 
against the union of the allocations.  This is somewhat conservative, but 
the alternatives are worse.

Note that placement is concerned with allocations rather than current 
usage.  Current usage fluctuates much faster than you would want placement 
to.  Placement needs to be done with a long-term perspective.  Of course, 
that perspective can be informed by usage information (as well as other 
sources) --- but it remains a distinct thing.

We consider all our copies of observed state to be soft --- they can be 
lost and reconstructed at any time, because the true source is the 
underlying cloud.  Which is not to say that reconstructing a copy is 
cheap.  We prefer making incremental updates as needed, rather than 
re-reading the whole thing.  One of our local extensions adds a mechanism 
by which a client can register to be notified of changes in the Software 
Defined Compute area.

The target state, on the other hand, is stored authoritatively by the 
placement agent in a database.

We pose placement as a constrained optimization problem, with a non-linear 
objective.  We approximate its solution with a very generic algorithm; it 
is easy to add new kinds of constraints and new contributions to the 
objective.

The core placement problem is about packing virtual resources into 
physical containers (e.g., VMs into hosts, volumes into Cinder backends). 
A virtual resource has a demand vector, and a corresponding container has 
a capacity vector of the same length.  For a given container, the sum of 
the demand vectors of the virtual resources in that container can not 
exceed the container's capacity vector in any dimension.  We can add 
dimensions as needed to handle the relevant host/guest characteristics.

We are just now working an example where a Cinder volume can be required 
to be the only one hosted on whatever Cinder backend hosts it.  This is 
exactly analogous to requiring that a VM (bare metal or otherwise) be the 
only one hosted by whatever PM hosts it.

We favor a fairly expressive language for stating desired poli

Re: [openstack-dev] [nova] [savanna] Host information for non admin users - from a holistic scheduler

[Mike Spreitzer]

Alex, my understanding is that the motivation for rack-awareness in Hadoop 
is optimizing availability rather than networking.  The good news, for 
those of us who favor a holistic scheduler, is that it can take both sorts 
of things into account when/where desired.

Yes, the case of a public cloud is more difficult than the case of a 
private cloud.  My understanding of Amazon's attitude, for example, is 
that they do not want to give out any bits of information about placement 
--- even though there are known techniques to reverse-engineer it, Amazon 
does not want to help that along at all.  Giving out obscured information 
--- some bits but not all --- is still disfavored.  Let me give a little 
background on how my group deals with placement for availability, then 
discuss options for the public cloud.

Our holistic scheduler takes as input something we call a virtual resource 
topology (VRT), other people use words like pattern, template, 
application, and cluster for such a thing.  It is an assembly of virtual 
resources that one tenant wants to instantiate.  In a VRT the resources 
are arranged into a tree of groups, the VRT itself is the root.  We use 
the groups for concise statements of various sorts, which I will omit here 
for the sake of simplicity.  As far as direct location constraints are 
concern, there is just one primitive thing: it is a relationship between 
two virtual resources and it is parameterized by a sense (positive or 
negative) and a level in the physical hierarchy (e.g., physical machine 
(PM), chassis, rack).  Thus: a negative relationship between VM1 and VM2 
at the rack level means that VM1 and VM2 must go on different racks; a 
positive relationship between VM3 and VM4 at the PM level means those two 
VMs must be on the same host.  Additionally, each constraint can be hard 
or soft: a hard constraint must be satisfied while a soft constraint is a 
preference.

Consider the example of six interchangeable VMs (say VM1, VM2, ... VM6) 
that should be spread across at least two racks with no more than half the 
VMs on any one rack.  How to say that with a collection of location 
primitives?  What we do is establish three rack-level anti-co-location 
constraints: one between VM1 and VM2, one between VM3 and VM4, and one 
between VM5 and VM6.  That is not the most obvious representation.  You 
might have expected this: nine rack-level anti-co-location constraints, 
one for every pair in the outer product between {VM1, VM2, VM3} and {VM4, 
VM5, VM6}.  Now consider what happens if the physical system has three 
racks and room for only two additional VMs on each rack.  With the latter 
set of constraints, there is no acceptable placement.  With the sparser 
set that we use, there are allowed placements.  In short, an obvious set 
of constraints may rule out otherwise acceptable placement.

I see two ways to give guaranteed-accurate rack awareness to Hadoop: 
constrain the placement so tightly that you know enough to configure 
Hadoop before the placement decision is made, or extract placement 
information after the placement decision is made.  The public cloud 
setting rules out the latter, leaving only the former.  This can be done, 
at a cost of suffering pattern rejections that would not occur if you did 
not have to over-constrain the placement.

One more option is to give up on guaranteed accuracy: prescribe a 
placement with sufficient precision to inform Hadoop, and so inform 
Hadoop, but make that prescription a preference rather than a hard 
constraint.  If the actual placement does not fully meet all the 
preferences, Hadoop is not informed of the differences and so will suffer 
in "non-functional" ways but still get the job done (modulo all those 
non-functional considerations, like tolerating a rack crash).  When your 
preferences are not met, it is because the system is very loaded and your 
only choice is between operating in some degraded way or not at all --- 
you might as well take the degraded operation.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [savanna] Host information for non admin users

[Mike Spreitzer]

> From: Nirmal Ranganathan 
> ...
> Well that's left upto the specific block placement policies in hdfs,
> all we are providing with the topology information is a hint on 
> node/rack placement.

Oh, you are looking at the placement of HDFS blocks within the fixed 
storage volumes, not choosing where to put the storage volumes.  In that 
case I understand and agree that simply providing identifiers from the 
infrastructure to the middleware (HDFS) will suffice.  Coincidentally my 
group is working on this very example right now in our own environment. We 
have a holistic scheduler that is given a whole template to place, and it 
returns placement information.  We imagine, as does Hadoop, a general 
hierarchy in the physical layout, and the holistic scheduler returns, for 
each VM, the path from the root to the VM's host.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [savanna] Host information for non admin users

[Mike Spreitzer]

> From: Nirmal Ranganathan 
> ...
> Not host capacity, just a opaque reference to distinguish a host is 
> enough. Hadoop can use that information to appropriately place block
> replicas. For example if the replication count is 3, and if a host/
> rack topology is provided to Hadoop, it will place each replica on a
> different host/rack granted one is available.

What if there are more than three racks, but some are better choices than 
others (perhaps even some are ruled out) due to considerations of various 
sorts of capacity and usage?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [savanna] Host information for non admin users

[Mike Spreitzer]

> From: Alexander Kuznetsov 
> ... 
> Currently Nova doesn’t provide information about the host of virtual
> machine for non admin users. Is it possible to change this 
> situation? This information is needed in Hadoop deployment case. 
> Because now Hadoop aware about virtual environment and this 
> knowledge help Hadoop to achieve a better performance and robustness.

How must host information are you looking for?  Do you want to know the 
capacities of the hosts?  Do you want to know about all the guests of the 
hosts?  If so, why is this only a Savanna issue?  If not, how much good is 
limited host information going to do Savanna?

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Comments/questions on the instance-group-api-extension blueprint

[Mike Spreitzer]

Gary Kotton  wrote on 09/12/2013 05:40:59 AM:

> From: Gary Kotton 
> To: OpenStack Development Mailing List 
, 
> Date: 09/12/2013 05:46 AM
> Subject: Re: [openstack-dev] [heat] Comments/questions on the 
> instance-group-api-extension blueprint
> 
> Hi,
> For some reason I am unable to access your proceed talk. I am not 
> 100% sure but I think that the voting may be closed. We have weekly 
> scheduling meetings (https://wiki.openstack.org/wiki/
> Meetings#Scheduler_Sub-group_meeting). It would be nice if you could
> attend and it will give you a platform to raise and share ideas with
> the rest of the guys in the community.
> At the moment the scheduling subgroup is working  on our ideas for 
> the design summit sessions. Please see https://
> etherpad.openstack.org/IceHouse-Nova-Scheduler-Sessions
> Thanks
> Gary

Worse yet, I know of no way to navigate to a list of design summit 
proposals.  What am I missing?

The scheduler group meeting conflicts with another meeting that I already 
have and will be difficult to move.  I will see what I can do 
asynchronously.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Comments/questions on the instance-group-api-extension blueprint

[Mike Spreitzer]

We are currently explicitly considering location and space.  For example, 
a template can require that a volume be in a disk that is directly 
attached to the machine hosting the VM to which the volume is attached. 
Spinning rust bandwidth is much trickier because it is not something you 
can simply add up when you combine workloads.  The IOPS, as well as the 
B/S, that a disk will deliver depends on the workload mix on that disk. 
While the disk may deliver X IOPS when serving only application A, and Y 
when serving only application B, you cannot conclude that it will serve 
(X+Y)/2 when serving (A+B)/2.  While we hope to do better in the future, 
we currently handle disk bandwidth in non-quantitative ways.  One is that 
a template may request that a volume be placed such that it does not 
compete with any other volume (i.e., is the only one on its disk). Another 
is that a template may specify a "type" for a volume, which effectively 
maps to a Cinder volume type that has been pre-defined to correspond to a 
QoS defined in an enterprise storage subsystem.

The choice between fast&expensive vs slow&cheap storage is currently left 
to higher layers.  That could be pushed down, supposing there is a 
suitably abstract yet accurate way of describing how the tradeoff choice 
should be made.

I think Savanna people are on this list too, so I presume it's a good 
place for this discussion.

Thanks,
Mike



From:   shalz 
To: OpenStack Development Mailing List 
, 
Date:   09/11/2013 09:55 PM
Subject:Re: [openstack-dev] [heat] Comments/questions on the 
instance-group-api-extension blueprint



Mike,

You mention  "We are now extending that example to include storage, and we 
are also working examples with Hadoop. "

In the context of your examples / scenarios, do these placement decisions 
consider storage performance and capacity on a physical node?

For example: Based on application needs, and IOPS, latency requirements - 
carving out a SSD storage or a traditional spinning disk block volume?  Or 
say for cost-efficiency reasons using SSD caching on Hadoop name nodes? 

I'm investigating  a) Per node PCIe SSD deployment need in Openstack 
environment /  Hadoop environment and ,b) selected node SSD caching, 
specifically for OpenStack Cinder.  Hope this is the right forum to ask 
this question.

rgds,
S

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Comments/questions on the instance-group-api-extension blueprint

[Mike Spreitzer]

Yes, I've seen that material.  In my group we have worked larger and more 
complex examples.  I have a proposed breakout session at the Hong Kong 
summit to talk about one, you might want to vote for it.  The URL is 
http://www.openstack.org/summit/openstack-summit-hong-kong-2013/become-a-speaker/TalkDetails/109
 
and the title is "Continuous Delivery of Lotus Connections on OpenStack". 
We used our own technology to do the scheduling (make placement decisions) 
and orchestration, calling Nova and Quantum to carry out the decisions our 
software made.  Above the OpenStack infrastructure we used two layers of 
our own software, one focused on infrastructure and one adding concerns 
for the software running on that infrastructure.  Each used its own 
language for a whole topology AKA pattern AKA application AKA cluster. For 
example, our pattern has 16 VMs running the WebSphere application server, 
organized into four homogenous groups (members are interchangeable) of 
four each.  For each group, we asked that it both (a) be spread across at 
least two racks, with no more than half the VMs on any one rack and (b) 
have no two VMs on the same hypervisor.  You can imagine how this would 
involve multiple levels of grouping and relationships between groups (and 
you will probably be surprised by the particulars).  We also included 
information on licensed products, so that the placement decision can 
optimize license cost (for the IBM "sub-capacity" licenses, placement of 
VMs can make a cost difference).  Thus, multiple policies per thing.  We 
are now extending that example to include storage, and we are also working 
examples with Hadoop.

Regards,
Mike



From:   Gary Kotton 
To: OpenStack Development Mailing List 
, 
Date:   09/11/2013 06:06 AM
Subject:Re: [openstack-dev] [heat] Comments/questions on the 
instance-group-api-extension blueprint





From: Mike Spreitzer 
Reply-To: OpenStack Development Mailing List <
openstack-dev@lists.openstack.org>
Date: Tuesday, September 10, 2013 11:58 PM
To: OpenStack Development Mailing List 
Subject: [openstack-dev] [heat] Comments/questions on the 
instance-group-api-extension blueprint

First, I'm a newbie here, wondering: is this the right place for 
comments/questions on blueprints?  Supposing it is...

[Gary Kotton] Yeah, as Russel said this is the correct place

I am referring to 
https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension

In my own research group we have experience with a few systems that do 
something like that, and more (as, indeed, that blueprint explicitly 
states that it is only the start of a longer roadmap).  I would like to 
highlight a couple of differences that alarm me.  One is the general 
overlap between groups.  I am not saying this is wrong, but as a matter of 
natural conservatism we have shied away from unnecessary complexities. The 
only overlap we have done so far is hierarchical nesting.  As the 
instance-group-api-extension explicitly contemplates groups of groups as a 
later development, this would cover the overlap that we have needed.  On 
the other hand, we already have multiple "policies" attached to a single 
group.  We have policies for a variety of concerns, so some can combine 
completely or somewhat independently.  We also have relationships (of 
various sorts) between groups (as well as between individuals, and between 
individuals and groups).  The policies and relationships, in general, are 
not simply names but also have parameters. 

[Gary Kotton] The instance groups was meant to be the first step towards 
what we had presented in Portland. Please look at the presentation that we 
gave an this may highlight what the aims were: 
https://docs.google.com/presentation/d/1oDXEab2mjxtY-cvufQ8f4cOHM0vIp4iMyfvZPqg8Ivc/edit?usp=sharing
. Sadly for this release we did not manage to get the instance groups 
through (it was an issue of timing and bad luck). We will hopefully get 
this though in the first stages of the I cycle and then carry on building 
on it as it has a huge amount of value for OpenStack. It will be great if 
you can also participate in the discussions.

Thanks, 
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] Program name and Mission statement

[Mike Spreitzer]

> To provide a simple, reliable and repeatable mechanism by which to 
> deploy Hadoop and related Big Data projects, including management, 
> monitoring and processing mechanisms driving further adoption of 
OpenStack.

That sounds like it is at about the right level of specificity.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] Program name and Mission statement

[Mike Spreitzer]

Jon Maron  wrote on 09/10/2013 08:50:23 PM:

> From: Jon Maron 
> To: OpenStack Development Mailing List 
, 
> Cc: OpenStack Development Mailing List 

> Date: 09/10/2013 08:55 PM
> Subject: Re: [openstack-dev] [savanna] Program name and Mission 
statement
> 
> Openstack Big Data Platform

Let's see if you mean that.  Does this project aim to cover big data 
things besides MapReduce?  Can you give examples of other things that are 
in scope?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] Comments/questions on the instance-group-api-extension blueprint

[Mike Spreitzer]

First, I'm a newbie here, wondering: is this the right place for 
comments/questions on blueprints?  Supposing it is...

I am referring to 
https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension

In my own research group we have experience with a few systems that do 
something like that, and more (as, indeed, that blueprint explicitly 
states that it is only the start of a longer roadmap).  I would like to 
highlight a couple of differences that alarm me.  One is the general 
overlap between groups.  I am not saying this is wrong, but as a matter of 
natural conservatism we have shied away from unnecessary complexities. The 
only overlap we have done so far is hierarchical nesting.  As the 
instance-group-api-extension explicitly contemplates groups of groups as a 
later development, this would cover the overlap that we have needed.  On 
the other hand, we already have multiple "policies" attached to a single 
group.  We have policies for a variety of concerns, so some can combine 
completely or somewhat independently.  We also have relationships (of 
various sorts) between groups (as well as between individuals, and between 
individuals and groups).  The policies and relationships, in general, are 
not simply names but also have parameters.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] Program name and Mission statement

[Mike Spreitzer]

A quick dictionary lookup of "data processing" yields the following.  I 
wonder if you mean something more specific.

data processing |ˈˌdædə ˈprɑsɛsɪŋ|
noun
a series of operations on data, esp. by a computer, to retrieve, 
transform, or classify information.



From:   Matthew Farrellee 
To: OpenStack Development Mailing List 
, 
Date:   09/10/2013 09:53 AM
Subject:Re: [openstack-dev] [savanna] Program name and Mission 
statement



Rough cut -

Program: OpenStack Data Processing
Mission: To provide the OpenStack community with an open, cutting edge, 
performant and scalable data processing stack and associated management 
interfaces.

On 09/10/2013 09:26 AM, Sergey Lukjanov wrote:
> It sounds too broad IMO. Looks like we need to define Mission Statement
> first.
>
> Sincerely yours,
> Sergey Lukjanov
> Savanna Technical Lead
> Mirantis Inc.
>
> On Sep 10, 2013, at 17:09, Alexander Kuznetsov  > wrote:
>
>> My suggestion OpenStack Data Processing.
>>
>>
>> On Tue, Sep 10, 2013 at 4:15 PM, Sergey Lukjanov
>> mailto:slukja...@mirantis.com>> wrote:
>>
>> Hi folks,
>>
>> due to the Incubator Application we should prepare Program name
>> and Mission statement for Savanna, so, I want to start mailing
>> thread about it.
>>
>> Please, provide any ideas here.
>>
>> P.S. List of existing programs:
>> https://wiki.openstack.org/wiki/Programs
>> P.P.S. https://wiki.openstack.org/wiki/Governance/NewPrograms
>>
>> Sincerely yours,
>> Sergey Lukjanov
>> Savanna Technical Lead
>> Mirantis Inc.
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Scaling of TripleO

[Mike Spreitzer]

Robert Collins  wrote on 09/06/2013 05:31:14 
PM:

> From: Robert Collins 
> To: OpenStack Development Mailing List 
, 
> Date: 09/06/2013 05:36 PM
> Subject: Re: [openstack-dev] [tripleo] Scaling of TripleO
> 
> ...
> My vision for TripleO/undercloud and scale in the long term is:
> - A fully redundant self-healing undercloud
>   - (implies self hosting)
...

Robert, what do you mean by "self hosting"?  If a cloud can self-host, why 
do we need two clouds (under and over)?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times

[Mike Spreitzer]

Joshua, I do not think such a strict and coarse scheduling is a practical 
way to manage developers, who have highly individualized talents, 
backgrounds, and interests.

Regards,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev