Re: [openstack-dev] [TripleO][Nova] Specs and approvals

[Jay Dobies]

I was on vacation last week and am late to the discussion, but I'm +1 
for the idea.


On 08/19/2014 02:08 PM, Joe Gordon wrote:




On Tue, Aug 19, 2014 at 8:23 AM, Russell Bryant mailto:rbry...@redhat.com>> wrote:

On 08/19/2014 05:31 AM, Robert Collins wrote:
 > Hey everybody - https://wiki.openstack.org/wiki/TripleO/SpecReviews
 > seems pretty sane as we discussed at the last TripleO IRC meeting.
 >
 > I'd like to propose that we adopt it with the following tweak:
 >
 > 19:46:34  so I propose that +2 on a spec is a commitment to
 > review it over-and-above the core review responsibilities
 > 19:47:05  if its not important enough for a reviewer to do
 > that thats a pretty strong signal
 > 19:47:06  lifeless: +1, I thought we already agreed to that
 > at the meetup
 > 19:47:17  yea, sounds fine to me
 > 19:47:20  +1
 > 19:47:30  dprince: it wasn't clear whether it was
 > part-of-responsibility, or additive, I'm proposing we make it clearly
 > additive
 > 19:47:52  and separately I think we need to make surfacing
 > reviews-for-themes a lot better
 >
 > That is - +1 on a spec review is 'sure, I like it', +2 is
specifically
 > "I will review this *over and above* my core commitment" - the goal
 > here is to have some very gentle choke on concurrent WIP without
 > needing the transition to a managed pull workflow that Nova are
 > discussing - which we didn't have much support for during the
meeting.
 >
 > Obviously, any core can -2 for any of the usual reasons - this motion
 > is about opening up +A to the whole Tripleo core team on specs.
 >
 > Reviewers, and other interested kibbitzers, please +1 / -1 as you
feel fit :)

+1

I really like this.  In fact, I like it a lot more than the current
proposal for Nova.  I think the Nova team should consider this, as well.


Nova and tripleo are at different points in there lifecycle just look at
tripleo-specs [0] vs nova-specs [1]. TripleO has 11 specs and nova has
80+, TripleO has 22 cores and nova has 21 cores.  AFAIK none of the
tripleo specs are vendor specific, while a good chunk of nova ones are.
I don't think there is a one size fits all solution here.


[0] http://specs.openstack.org/openstack/tripleo-specs/
[1] http://specs.openstack.org/openstack/nova-specs/


It still rate limits code reviews by making core reviewers explicitly
commit to reviewing things.  This is like our previous attempt at
sponsoring blueprints, but the use of gerrit I think would make it more
successful.

It also addresses my primary concerns with the tensions between "group
will" and small groups no longer being able to self organize and push
things to completion without having to haggle through yet another
process.

--
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Review metrics - what do we want to measure?

[Jay Dobies]

It can, by running your own... but again it seems far better for
core reviewers to decide if a change has potential or needs to be
abandoned--that way there's an accountable human making that
deliberate choice rather than the review team hiding behind an
automated process so that no one is to blame for hurt feelings
besides the infra operators who are enforcing this draconian measure
for you.


The thing is that it's also pushing more work onto already overloaded
core review teams.  Maybe submitters don't like auto-abandon, but I bet
they like having a core reviewer spending time cleaning up dead reviews
instead of reviewing their change even less.

TBH, if someone's offended by the bot then I can't imagine how incensed
they must be when a human does the same thing.  The bot clearly isn't
making it personal, and even if the human isn't either it's much easier
to have misunderstandings (see also every over-reaction to a -1 ever).

I suppose it makes it easier for cores to ignore reviews, but from the
other discussions I've read that hasn't gone away just because
auto-abandon did, so I'm not convinced that's a solution anyway.


+1, I don't think it'll come as much of a shock if a -1 review gets 
closed due to time without progress.



/2 cents




To make the whole process a little friendlier we could increase
the time frame from 1 week to 2.


How about just automatically abandon any new change as soon
as it's published, and if the contributor really feels it's
important they'll unabandon it.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Propose adding StevenK to core reviewers

[Jay Dobies]

+1

On 09/09/2014 02:32 PM, Gregory Haynes wrote:

Hello everyone!

I have been working on a meta-review of StevenK's reviews and I would
like to propose him as a new member of our core team.

As I'm sure many have noticed, he has been above our stats requirements
for several months now. More importantly, he has been reviewing a wide
breadth of topics and seems to have a strong understanding of our code
base. He also seems to be doing a great job at providing valuable
feedback and being attentive to responses on his reviews.

As such, I think he would make a great addition to our core team. Can
the other core team members please reply with your votes if you agree or
disagree.

Thanks!
Greg

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[Jay Dobies]

How many of the reviews that we WIP-1 will actually be revisited?

I'm sure there will be cases where a current developer forgetting they 
had started on something, seeing the e-mail about the WIP-1, and then 
abandoning the change.


But what about developers who have moved off the project entirely? Is 
this only masking the problem of stale reviews from our review stats and 
leaving the review queue to bloat?


I honestly don't know; those are real questions, not rhetorical ones 
trying to prove a point. I'd guess the longer-running OpenStack projects 
have had to deal with this as well, and perhaps I'm overestimating just 
how many of these perpetually in limbo reviews there are.



On 09/18/2014 03:26 AM, mar...@redhat.com wrote:

On 18/09/14 00:29, James Polley wrote:



On Wed, Sep 17, 2014 at 6:26 PM, mar...@redhat.com
 mailto:mandr...@redhat.com>> wrote:

 Hi,

 as part of general housekeeping on our reviews, it was discussed at last
 week's meeting [1] that we should set workflow -1 for stale reviews
 (like gerrit used to do when I were a lad).

 The specific criteria discussed was 'items that have a -1 from a core
 but no response from author for 14 days'. This topic came up again
 during today's meeting and it wasn't clear if the intention was for
 cores to start enforcing this? So:

 Do we start setting WIP/workflow -1 for those reviews that have a -1
 from a core but no response from author for 14 days


I'm in favour of doing this; as long as we make it clear that we're
doing it to help us focus review effort on things that are under active
development - it doesn't mean we think the patch shouldn't land, it just
means we know it's not ready yet so we don't want reviewers to be
looking at it until it moves forward.

For the sake of making sure new developers don't get put off, I'd like
to see us leaving a comment explaining why we're WIPing the change and
noting that uploading a new revision will remove the WIP automatically



+1 - indeed, I'd say as part of this discussion, or if/when it comes up
as a motion for a vote in the weekly meeting, we should also put out and
agree on the 'standard' text to be used for this and stick it on the
wiki (regardless of whether this is to be implemented manually at first
and perhaps automated later),

thanks, marios

"setting workflow -1 as this review has been inactive for two weeks
following a negative review. Please see the wiki @ foo for more
information. Note that once you upload a new revision the workflow is
expected to be reset (feel free to shout on freenode/#tripleo if it isn't)."





 thanks, marios

 [1]
 
http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-09-09-19.04.log.html

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] capturing build details in images

[Jay Dobies]

On 12/05/2013 08:38 AM, James Slagle wrote:

On Wed, Dec 4, 2013 at 5:19 PM, Robert Collins
 wrote:

This is a follow up to https://review.openstack.org/59621 to get
broader discussion..

So at the moment we capture a bunch of details in the image - what
parameters the image was built with and some environment variables.

Last week we were capturing everything, which there is broad consensus
was too much, but it seems to me that that is based on two things:
  - the security ramifications of unanticipated details being baked
into the image
  - many variables being irrelevant most of the time

I think those are both good points. But... the problem with diagnostic
information is you don't know that you need it until you don't have
it.

I'm particularly worried that things like bad http proxies, and third
party elements that need variables we don't know about will be
undiagnosable. Forcing everything through a DIB_FOO variable thunk
seems like just creating work for ourselves - I'd like to avoid that.

Further, some variables we should capture (like http_proxy) have
passwords embedded in them, so even whitelisting what variables to
capture doesn't solve the general problem.

So - what about us capturing this information outside the image: we
can create a uuid for the build, and write a file in the image with
that uuid, and outside the image we can write:
  - all variables (no security ramifications now as this file can be
kept by whomever built the image)
  - command line args
  - version information for the toolchain etc.


+1.  I like this idea a lot.

What about making the uuid file written outside of the image be in
json format so it's easily machine parseable?

Something like:

dib-.json would contain:

{
   "environment" : {
   "DIB_NO_TMPFS": "1",
   ...
},
   "dib" : {
  "command-line" : ,
  "version": .
   }
}

Could keep adding additional things like list of elements used, build time, etc.


+1 to having a machine parsable version. Is that going to be a standard 
schema for all images or will there be an open-ended section that 
contains key/value pairs that are contingent on the actual type of image 
being built?





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][TripleO] Nested resources

[Jay Dobies]

Along the same lines and while we're talking crazy ideas, one use case 
where a user might want to allocate entire nodes would be if TripleO 
were used to manage an ARM rack. The use cases aren't identical between 
cloud and ARM, but they are similar.


So for a rack of 1000 nodes, there is benefit in certain cases for a 
user not only taking an entire node, but a collection of nodes 
co-located in the same rack to take advantage of the rack fabric.


Again, crazy ideas and probably outside of the scope of things we want 
to bite off immediately. But as we're in the early stages of the Tuskar 
data and security models, it might make sense to at least keep in mind 
how we could play in this area as well.


On 12/05/2013 08:11 PM, Fox, Kevin M wrote:

I think the security issue can be handled by not actually giving the underlying 
resource to the user in the first place.
So, for example, if I wanted a bare metal node's worth of resource for my own 
containering, I'd ask for a bare metal node and use a "blessed" image that 
contains docker+nova bits that would hook back to the cloud. I wouldn't be able to login 
to it, but containers started on it would be able to access my tenant's networks. All 
access to it would have to be through nova suballocations. The bare resource would count 
against my quotas, but nothing run under it.

Come to think of it, this sounds somewhat similar to what is planned for 
Neutron service vm's. They count against the user's quota on one level but not 
all access is directly given to the user. Maybe some of the same implementation 
bits could be used.

Thanks,
Kevin

From: Mark McLoughlin [mar...@redhat.com]
Sent: Thursday, December 05, 2013 1:53 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova][TripleO] Nested resources

Hi Kevin,

On Mon, 2013-12-02 at 12:39 -0800, Fox, Kevin M wrote:

Hi all,

I just want to run a crazy idea up the flag pole. TripleO has the
concept of an under and over cloud. In starting to experiment with
Docker, I see a pattern start to emerge.

  * As a User, I may want to allocate a BareMetal node so that it is
entirely mine. I may want to run multiple VM's on it to reduce my own
cost. Now I have to manage the BareMetal nodes myself or nest
OpenStack into them.
  * As a User, I may want to allocate a VM. I then want to run multiple
Docker containers on it to use it more efficiently. Now I have to
manage the VM's myself or nest OpenStack into them.
  * As a User, I may want to allocate a BareMetal node so that it is
entirely mine. I then want to run multiple Docker containers on it to
use it more efficiently. Now I have to manage the BareMetal nodes
myself or nest OpenStack into them.

I think this can then be generalized to:
As a User, I would like to ask for resources of one type (One AZ?),
and be able to delegate resources back to Nova so that I can use Nova
to subdivide and give me access to my resources as a different type.
(As a different AZ?)

I think this could potentially cover some of the TripleO stuff without
needing an over/under cloud. For that use case, all the BareMetal
nodes could be added to Nova as such, allocated by the "services"
tenant as running a nested VM image type resource stack, and then made
available to all tenants. Sys admins then could dynamically shift
resources from VM providing nodes to BareMetal Nodes and back as
needed.

This allows a user to allocate some raw resources as a group, then
schedule higher level services to run only in that group, all with the
existing api.

Just how crazy an idea is this?


FWIW, I don't think it's a crazy idea at all - indeed I mumbled
something similar a few times in conversation with random people over
the past few months :)

With the increasing interest in containers, it makes a tonne of sense -
you provision a number of VMs and now you want to carve them up by
allocating containers on them. Right now, you'd need to run your own
instance of Nova for that ... which is far too heavyweight.

It is a little crazy in the sense that it's a tonne of work, though.
There's not a whole lot of point in discussing it too much until someone
shows signs of wanting to implement it :)

One problem is how the API would model this nesting, another problem is
making the scheduler aware that some nodes are only available to the
tenant which owns them but maybe a bigger problem is the security model
around allowing a node managed by an untrusted become a compute node.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@li

Re: [openstack-dev] [TripleO][Tuskar] Questions around Development Process

[Jay Dobies]

a) Because we're essentially doing a tear-down and re-build of the
whole architecture (a lot of the concepts in tuskar
will simply disappear), it's difficult to do small incremental patches
that support existing functionality.  Is it okay
to have patches that break functionality?  Are there good alternatives?


This is an incubating project, so there are no api stability promises.
If a patch breaks some functionality that we've decided to not support
going forward I don't see a problem with it.  That said, if a patch
breaks some functionality that we _do_ plan to keep, I'd prefer to see
it done as a series of dependent commits that end with the feature in a
working state again, even if some of the intermediate commits are not
fully functional.  Hopefully that will both keep the commit sizes down
and provide a definite path back to functionality.


Is there any sort of policy or convention of sending out a warning 
before that sort of thing is merged in so that people don't accidentally 
blindly pull master and break something they were using?



b) In the past, we allowed parallel development of the UI and API by
having well-documented expectations of what the API


Are these expectations documented yet? I'm new to the project and still 
finding my way around. I've seen the wireframes and am going through 
Chen's icehouse requirements, but I haven't stumbled on too much talk 
about the APIs specifically (not suggesting they don't exist, more 
likely that I haven't found them yet).



would provide.  We would then mock those calls in the UI, replacing
them with real API calls as they became available.  Is
this acceptable?


This sounds reasonable to me.



-Ben


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tripleo] Core reviewer update Dec

[Jay Dobies]

On 12/06/2013 12:26 PM, Clint Byrum wrote:

Excerpts from Robert Collins's message of 2013-12-03 23:12:39 -0800:

Hi,
 like most OpenStack projects we need to keep the core team up to
date: folk who are not regularly reviewing will lose context over
time, and new folk who have been reviewing regularly should be trusted
with -core responsibilities.

In this months review:
  - Ghe Rivero for -core


+1, We've been getting good reviews from Ghe for a while now. :)


  - Jan Provaznik for removal from -core
  - Jordan O'Mara for removal from -core
  - Martyn Taylor for removal from -core
  - Jiri Tomasek for removal from -core
  - Jamomir Coufal for removal from -core


I suggest we delay this removal for 30 days.


For what it's worth, keep in mind the holidays coming up at the end of 
December. I suspect that trying to reevaluate 30 days from now will be 
even trickier when you have to take into account vacation times.




I know it is easy to add
them back in, but I hesitate to disrupt the flow if these people all
are willing to pick up the pace again. They may not have _immediate_
code knowledge but they should have enough historical knowledge that
has not gone completely stale in just the last 30-60 days.

What I'm suggesting is that review velocity will benefit from core being
a little more sticky, especially for sustained contributors who have
just had their attention directed elsewhere briefly.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

Disclaimer: I'm very new to the project, so apologies if some of my 
questions have been already answered or flat out don't make sense.


As I proofread, some of my comments may drift a bit past basic 
requirements, so feel free to tell me to take certain questions out of 
this thread into specific discussion threads if I'm getting too detailed.





*** Requirements are assumed to be targeted for Icehouse, unless marked 
otherwise:
(M) - Maybe Icehouse, dependency on other in-development features
(F) - Future requirement, after Icehouse

* NODES
* Creation
   * Manual registration
  * hardware specs from Ironic based on mac address (M)
  * IP auto populated from Neutron (F)
   * Auto-discovery during undercloud install process (M)
* Monitoring
* assignment, availability, status
* capacity, historical statistics (M)
* Management node (where triple-o is installed)
* created as part of undercloud install process
* can create additional management nodes (F)
 * Resource nodes
 * searchable by status, name, cpu, memory, and all attributes from 
ironic
 * can be allocated as one of four node types


It's pretty clear by the current verbiage but I'm going to ask anyway: 
"one and only one"?



 * compute
 * controller
 * object storage
 * block storage
 * Resource class - allows for further categorization of a node type
 * each node type specifies a single default resource class
 * allow multiple resource classes per node type (M)


My gut reaction is that we want to bite this off sooner rather than 
later. This will have data model and API implications that, even if we 
don't commit to it for Icehouse, should still be in our minds during it, 
so it might make sense to make it a first class thing to just nail down now.



 * optional node profile for a resource class (M)
 * acts as filter for nodes that can be allocated to that class 
(M)


To my understanding, once this is in Icehouse, we'll have to support 
upgrades. If this filtering is pushed off, could we get into a situation 
where an allocation created in Icehouse would no longer be valid in 
Icehouse+1 once these filters are in place? If so, we might want to make 
it more of a priority to get them in place earlier and not eat the 
headache of addressing these sorts of integrity issues later.



 * nodes can be viewed by node types
 * additional group by status, hardware specification
 * controller node type
* each controller node will run all openstack services
   * allow each node to run specified service (F)
* breakdown by workload (percentage of cpu used per node) (M)
 * Unallocated nodes


Is there more still being flushed out here? Things like:
 * Listing unallocated nodes
 * Unallocating a previously allocated node (does this make it a 
vanilla resource or does it retain the resource type? is this the only 
way to change a node's resource type?)
 * Unregistering nodes from Tuskar's inventory (I put this under 
unallocated under the assumption that the workflow will be an explicit 
unallocate before unregister; I'm not sure if this is the same as 
"archive" below).



 * Archived nodes (F)


Can you elaborate a bit more on what this is?


 * Will be separate openstack service (F)

* DEPLOYMENT
* multiple deployments allowed (F)
  * initially just one
* deployment specifies a node distribution across node types
   * node distribution can be updated after creation
* deployment configuration, used for initial creation only
   * defaulted, with no option to change
  * allow modification (F)
* review distribution map (F)
* notification when a deployment is ready to go or whenever something 
changes

* DEPLOYMENT ACTION
* Heat template generated on the fly
   * hardcoded images
  * allow image selection (F)
   * pre-created template fragments for each node type
   * node type distribution affects generated template
* nova scheduler allocates nodes
   * filters based on resource class and node profile information (M)
* Deployment action can create or update
* status indicator to determine overall state of deployment
   * status indicator for nodes as well
   * status includes 'time left' (F)

* NETWORKS (F)
* IMAGES (F)
* LOGS (F)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

I believe we are still 'fighting' here with two approaches and I believe
we need both. We can't only provide a way 'give us resources we will do
a magic'. Yes this is preferred way - especially for large deployments,
but we also need a fallback so that user can say - no, this node doesn't
belong to the class, I don't want it there - unassign. Or I need to have
this node there - assign.


+1 to this. I think there are still a significant amount of admins out 
there that are really opposed to magic and want that fine-grained 
control. Even if they don't use it that frequently, in my experience 
they want to know it's there in the event they need it (and will often 
dream up a case that they'll need it).


I'm absolutely for pushing the magic approach as the preferred use. And 
in large deployments that's where people are going to see the biggest 
gain. The fine-grained approach can even be pushed off as a future 
feature. But I wouldn't be surprised to see people asking for it and I'd 
like to at least be able to say it's been talked about.



- As an infrastructure administrator, Anna wants to be able to view the history 
of nodes that have been in a deployment.

Why? This is super generic and could mean anything.

I believe this has something to do with 'archived nodes'. But correct me
if I am wrong.

-- Jarda


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

On 12/06/2013 09:39 PM, Tzu-Mainn Chen wrote:

Thanks for the comments and questions!  I fully expect that this list of 
requirements
will need to be fleshed out, refined, and heavily modified, so the more the 
merrier.

Comments inline:



*** Requirements are assumed to be targeted for Icehouse, unless marked
otherwise:
(M) - Maybe Icehouse, dependency on other in-development features
(F) - Future requirement, after Icehouse

* NODES


Note that everything in this section should be Ironic API calls.


* Creation
   * Manual registration
  * hardware specs from Ironic based on mac address (M)


Ironic today will want IPMI address + MAC for each NIC + disk/cpu/memory
stats


  * IP auto populated from Neutron (F)


Do you mean IPMI IP ? I'd say IPMI address managed by Neutron here.


   * Auto-discovery during undercloud install process (M)
* Monitoring
* assignment, availability, status
* capacity, historical statistics (M)


Why is this under 'nodes'? I challenge the idea that it should be
there. We will need to surface some stuff about nodes, but the
underlying idea is to take a cloud approach here - so we're monitoring
services, that happen to be on nodes. There is room to monitor nodes,
as an undercloud feature set, but lets be very very specific about
what is sitting at what layer.


That's a fair point.  At the same time, the UI does want to monitor both
services and the nodes that the services are running on, correct?  I would
think that a user would want this.

Would it be better to explicitly split this up into two separate requirements?


That was my understanding as well, that Tuskar would not only care about 
the services of the undercloud but the health of the actual hardware on 
which it's running. As I write that I think you're correct, two separate 
requirements feels much more explicit in how that's different from 
elsewhere in OpenStack.



* Management node (where triple-o is installed)


This should be plural :) - TripleO isn't a single service to be
installed - We've got Tuskar, Ironic, Nova, Glance, Keystone, Neutron,
etc.


I misspoke here - this should be "where the undercloud is installed".  My
current understanding is that our initial release will only support the 
undercloud
being installed onto a single node, but my understanding could very well be 
flawed.


* created as part of undercloud install process
* can create additional management nodes (F)
 * Resource nodes


 ^ nodes is again confusing layers - nodes are
what things are deployed to, but they aren't the entry point


 * searchable by status, name, cpu, memory, and all attributes from
 ironic
 * can be allocated as one of four node types


Not by users though. We need to stop thinking of this as 'what we do
to nodes' - Nova/Ironic operate on nodes, we operate on Heat
templates.


Right, I didn't mean to imply that users would be doing this allocation.  But 
once Nova
does this allocation, the UI does want to be aware of how the allocation is 
done, right?
That's what this requirement meant.


 * compute
 * controller
 * object storage
 * block storage
 * Resource class - allows for further categorization of a node type
 * each node type specifies a single default resource class
 * allow multiple resource classes per node type (M)


Whats a node type?


Compute/controller/object storage/block storage.  Is another term besides "node 
type"
more accurate?




 * optional node profile for a resource class (M)
 * acts as filter for nodes that can be allocated to that
 class (M)


I'm not clear on this - you can list the nodes that have had a
particular thing deployed on them; we probably can get a good answer
to being able to see what nodes a particular flavor can deploy to, but
we don't want to be second guessing the scheduler..


Correct; the goal here is to provide a way through the UI to send additional 
filtering
requirements that will eventually be passed into the scheduler, allowing the 
scheduler
to apply additional filters.


 * nodes can be viewed by node types
 * additional group by status, hardware specification


*Instances* - e.g. hypervisors, storage, block storage etc.


 * controller node type


Again, need to get away from node type here.


* each controller node will run all openstack services
   * allow each node to run specified service (F)
* breakdown by workload (percentage of cpu used per node) (M)
 * Unallocated nodes


This implies an 'allocation' step, that we don't have - how about
'Idle nodes' or something.


Is it imprecise to say that nodes are allocated by the scheduler?  Would 
something like
'active/idle' be better?


 * Archived nodes (F)
 * Will be se

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

So the question is are we looking at /nodes/ that have a /current
role/, or are we looking at /roles/ that have some /current nodes/.

My contention is that the role is the interesting thing, and the nodes
is the incidental thing. That is, as a sysadmin, my hierarchy of
concerns is something like:
  A: are all services running
  B: are any of them in a degraded state where I need to take prompt
action to prevent a service outage [might mean many things: - software
update/disk space criticals/a machine failed and we need to scale the
cluster back up/too much load]
  C: are there any planned changes I need to make [new software deploy,
feature request from user, replacing a faulty machine]
  D: are there long term issues sneaking up on me [capacity planning,
machine obsolescence]

If we take /nodes/ as the interesting thing, and what they are doing
right now as the incidental thing, it's much harder to map that onto
the sysadmin concerns. If we start with /roles/ then can answer:
  A: by showing the list of roles and the summary stats (how many
machines, service status aggregate), role level alerts (e.g. nova-api
is not responding)
  B: by showing the list of roles and more detailed stats (overall
load, response times of services, tickets against services
  and a list of in trouble instances in each role - instances with
alerts against them - low disk, overload, failed service,
early-detection alerts from hardware
  C: probably out of our remit for now in the general case, but we need
to enable some things here like replacing faulty machines
  D: by looking at trend graphs for roles (not machines), but also by
looking at the hardware in aggregate - breakdown by age of machines,
summary data for tickets filed against instances that were deployed to
a particular machine

C: and D: are (F) category work, but for all but the very last thing,
it seems clear how to approach this from a roles perspective.

I've tried to approach this using /nodes/ as the starting point, and
after two terrible drafts I've deleted the section. I'd love it if
someone could show me how it would work:)


 * Unallocated nodes

This implies an 'allocation' step, that we don't have - how about
'Idle nodes' or something.

It can be auto-allocation. I don't see problem with 'unallocated' term.


Ok, it's not a biggy. I do think it will frame things poorly and lead
to an expectation about how TripleO works that doesn't match how it
does, but we can change it later if I'm right, and if I'm wrong, well
it won't be the first time :).



I'm interested in what the distinction you're making here is.  I'd rather get 
things
defined correctly the first time, and it's very possible that I'm missing a 
fundamental
definition here.


So we have:
  - node - a physical general purpose machine capable of running in
many roles. Some nodes may have hardware layout that is particularly
useful for a given role.
  - role - a specific workload we want to map onto one or more nodes.
Examples include 'undercloud control plane', 'overcloud control
plane', 'overcloud storage', 'overcloud compute' etc.
  - instance - A role deployed on a node - this is where work actually happens.
  - scheduling - the process of deciding which role is deployed on which node.


This glossary is really handy to make sure we're all speaking the same 
language.



The way TripleO works is that we defined a Heat template that lays out
policy: 5 instances of 'overcloud control plane please', '20
hypervisors' etc. Heat passes that to Nova, which pulls the image for
the role out of Glance, picks a node, and deploys the image to the
node.

Note in particular the order: Heat -> Nova -> Scheduler -> Node chosen.

The user action is not 'allocate a Node to 'overcloud control plane',
it is 'size the control plane through heat'.

So when we talk about 'unallocated Nodes', the implication is that
users 'allocate Nodes', but they don't: they size roles, and after
doing all that there may be some Nodes that are - yes - unallocated,


I'm not sure if I should ask this here or to your point above, but what 
about multi-role nodes? Is there any piece in here that says "The policy 
wants 5 instances but I can fit two of them on this existing 
underutilized node and three of them on unallocated nodes" or since it's 
all at the image level you get just what's in the image and that's the 
finest-level of granularity?



or have nothing scheduled to them. So... I'm not debating that we
should have a list of free hardware - we totally should - I'm debating
how we frame it. 'Available Nodes' or 'Undeployed machines' or
whatever. I just want to get away from talking about something
([manual] allocation) that we don't offer.


My only concern here is that we're not talking about cloud users, we're 
talking about admins adminning (we'll pretend it's a word, come with me) 
a cloud. To a cloud user, "give me some power so I can do some stuff" is 
a safe use case if I trust the cloud I'm running on. I t

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

Thanks for the explanation!

I'm going to claim that the thread revolves around two main areas of 
disagreement.  Then I'm going
to propose a way through:

a) Manual Node Assignment

I think that everyone is agreed that automated node assignment through 
nova-scheduler is by
far the most ideal case; there's no disagreement there.

The disagreement comes from whether we need manual node assignment or not.  I 
would argue that we
need to step back and take a look at the real use case: heterogeneous nodes.  
If there are literally
no characteristics that differentiate nodes A and B, then why do we care which 
gets used for what?  Why
do we need to manually assign one?


This is a better way of verbalizing my concerns. I suspect there are 
going to be quite a few heterogeneous environments built from legacy 
pieces in the near term and fewer built from the ground up with all new 
matching hotness.


On the other side of it, instead of handling legacy hardware I was 
worried about the new hotness (not sure why I keep using that term) 
specialized for a purpose. This is exactly what Robert described in his 
GPU example. I think his explanation of how to use the scheduler to 
accommodate that makes a lot of sense, so I'm much less behind the idea 
of a strict manual assignment than I previously was.



If we can agree on that, then I think it would be sufficient to say that we 
want a mechanism to allow
UI users to deal with heterogeneous nodes, and that mechanism must use 
nova-scheduler.  In my mind,
that's what resource classes and node profiles are intended for.

One possible objection might be: nova scheduler doesn't have the appropriate 
filter that we need to
separate out two nodes.  In that case, I would say that needs to be taken up 
with nova developers.


b) Terminology

It feels a bit like some of the disagreement come from people using different 
words for the same thing.
For example, the wireframes already details a UI where Robert's roles come 
first, but I think that message
was confused because I mentioned "node types" in the requirements.

So could we come to some agreement on what the most exact terminology would be? 
 I've listed some examples below,
but I'm sure there are more.

node type | role
management node | ?
resource node | ?
unallocated | available | undeployed
create a node distribution | size the deployment
resource classes | ?
node profiles | ?

Mainn

- Original Message -

On 10 December 2013 09:55, Tzu-Mainn Chen  wrote:

* created as part of undercloud install process



By that note I meant, that Nodes are not resources, Resource instances
run on Nodes. Nodes are the generic pool of hardware we can deploy
things onto.


I don't think "resource nodes" is intended to imply that nodes are
resources; rather, it's supposed to
indicate that it's a node where a resource instance runs.  It's supposed to
separate it from "management node"
and "unallocated node".


So the question is are we looking at /nodes/ that have a /current
role/, or are we looking at /roles/ that have some /current nodes/.

My contention is that the role is the interesting thing, and the nodes
is the incidental thing. That is, as a sysadmin, my hierarchy of
concerns is something like:
  A: are all services running
  B: are any of them in a degraded state where I need to take prompt
action to prevent a service outage [might mean many things: - software
update/disk space criticals/a machine failed and we need to scale the
cluster back up/too much load]
  C: are there any planned changes I need to make [new software deploy,
feature request from user, replacing a faulty machine]
  D: are there long term issues sneaking up on me [capacity planning,
machine obsolescence]

If we take /nodes/ as the interesting thing, and what they are doing
right now as the incidental thing, it's much harder to map that onto
the sysadmin concerns. If we start with /roles/ then can answer:
  A: by showing the list of roles and the summary stats (how many
machines, service status aggregate), role level alerts (e.g. nova-api
is not responding)
  B: by showing the list of roles and more detailed stats (overall
load, response times of services, tickets against services
  and a list of in trouble instances in each role - instances with
alerts against them - low disk, overload, failed service,
early-detection alerts from hardware
  C: probably out of our remit for now in the general case, but we need
to enable some things here like replacing faulty machines
  D: by looking at trend graphs for roles (not machines), but also by
looking at the hardware in aggregate - breakdown by age of machines,
summary data for tickets filed against instances that were deployed to
a particular machine

C: and D: are (F) category work, but for all but the very last thing,
it seems clear how to approach this from a roles perspective.

I've tried to approach this using /nodes/ as the starting point, and
after two terrible drafts I've deleted th

Re: [openstack-dev] [TripleO] Tuskar CLI after architecture changes

[Jay Dobies]

Disclaimer: I swear I'll stop posting this sort of thing soon, but I'm 
new to the project. I only mention it again because it's relevant in 
that I missed any of the discussion on why proxying from tuskar API to 
other APIs is looked down upon. Jiri and I had been talking yesterday 
and he mentioned it to me when I started to ask these same sorts of 
questions.


On 12/11/2013 07:33 AM, Jiří Stránský wrote:

Hi all,

TL;DR: I believe that "As an infrastructure administrator, Anna wants a
CLI for managing the deployment providing the same fundamental features
as UI." With the planned architecture changes (making tuskar-api thinner
and getting rid of proxying to other services), there's not an obvious
way to achieve that. We need to figure this out. I present a few options
and look forward for feedback.

Previously, we had planned Tuskar arcitecture like this:

tuskar-ui <-> tuskarclient <-> tuskar-api <-> heat-api|ironic-api|etc.


My biggest concern was that having each client call out to the 
individual APIs directly put a lot of knowledge into the clients that 
had to be replicated across clients. At the best case, that's simply 
knowing where to look for data. But I suspect it's bigger than that and 
there are workflows that will be implemented for tuskar needs. If the 
tuskar API can't call out to other APIs, that workflow implementation 
needs to be done at a higher layer, which means in each client.


Something I'm going to talk about later in this e-mail but I'll mention 
here so that the diagrams sit side-by-side is the potential for a facade 
layer that hides away the multiple APIs. Lemme see if I can do this in 
ASCII:


tuskar-ui -+   +-tuskar-api
   |   |
   +-client-facade-+-nova-api
   |   |
tuskar-cli-+   +-heat-api

The facade layer runs client-side and contains the business logic that 
calls across APIs and adds in the tuskar magic. That keeps the tuskar 
API from calling into other APIs* but keeps all of the API call logic 
abstracted away from the UX pieces.


* Again, I'm not 100% up to speed with the API discussion, so I'm going 
off the assumption that we want to avoid API to API calls. If that isn't 
as strict of a design principle as I'm understanding it to be, then the 
above picture probably looks kinda silly, so keep in mind the context 
I'm going from.


For completeness, my gut reaction was expecting to see something like:

tuskar-ui -+
   |
   +-tuskar-api-+-nova-api
   ||
tuskar-cli-++-heat-api

Where a tuskar client talked to the tuskar API to do tuskar things. 
Whatever was needed to do anything tuskar-y was hidden away behind the 
tuskar API.



This meant that the "integration logic" of how to use heat, ironic and
other services to manage an OpenStack deployment lied within
*tuskar-api*. This gave us an easy way towards having a CLI - just build
tuskarclient to wrap abilities of tuskar-api.

Nowadays we talk about using heat and ironic (and neutron? nova?
ceilometer?) apis directly from the UI, similarly as Dashboard does.
But our approach cannot be exactly the same as in Dashboard's case.
Dashboard is quite a thin wrapper on top of python-...clients, which
means there's a natural parity between what the Dashboard and the CLIs
can do.


When you say python- clients, is there a distinction between the CLI and 
a bindings library that invokes the server-side APIs? In other words, 
the CLI is packaged as CLI+bindings and the UI as GUI+bindings?



We're not wrapping the APIs directly (if wrapping them directly would be
sufficient, we could just use Dashboard and not build Tuskar API at
all). We're building a separate UI because we need *additional logic* on
top of the APIs. E.g. instead of directly working with Heat templates
and Heat stacks to deploy overcloud, user will get to pick how many
control/compute/etc. nodes he wants to have, and we'll take care of Heat
things behind the scenes. This makes Tuskar UI significantly thicker
than Dashboard is, and the natural parity between CLI and UI vanishes.
By having this logic in UI, we're effectively preventing its use from
CLI. (If i were bold i'd also think about integrating Tuskar with other
software which would be prevented too if we keep the business logic in
UI, but i'm not absolutely positive about use cases here).


I see your point about preventing its use from the CLI, but more 
disconcerting IMO is that it just doesn't belong in the UI. That sort of 
logic, the "Heat things behind the scenes", sounds like the jurisdiction 
of the API (if I'm reading into what that entails correctly).



Now this raises a question - how do we get CLI reasonably on par with
abilities of the UI? (Or am i wrong that Anna the infrastructure
administrator would want that?)


To reiterate my point above, I see the idea of getting the CLI on par, 
but I also see it as striving for a cleaner design as well.



Here are some 

Re: [openstack-dev] [TripleO] Tuskar CLI after architecture changes

[Jay Dobies]

> I will take it little side ways. I think we should be asking why have 
> we needed the tuskar-api. It has done some more complex logic (e.g. > 
> building a heat template) or storing additional info, not supported > 
> by the services we use (like rack associations).

> That is a perfectly fine use-case of introducing tuskar-api.

> Although now, when everything is shifting to the services themselves, 
> we don't need tuskar-api for that kind of stuff. Can you please list 
> what complex operations are left, that should be done in tuskar? I > 
> think discussing concrete stuff would be best.


This is a good call to circle back on that I'm not sure of it either. 
The wireframes I've seen so far largely revolve around node listing and 
allocation, but I 100% know I'm oversimplifying it and missing something 
bigger there.



Also, as I have been talking with rdopieralsky, there has been some
problems in the past, with tuskar doing more steps in one. Like create a
rack and register new nodes in the same time. As those have been
separate API calls and there is no transaction handling, we should not
do this kind of things in the first place. If we have actions that
depends on each other, it should go from UI one by one. Otherwise we
will be showing messages like, "The rack has not been created, but 5
from 8 nodes has been added. We have tried to delete those added nodes,
but 2 of the 5 deletions has failed. Please figure this out, then you
can run this awesome action that calls multiple dependent APIs without
real rollback again." (or something like that, depending on what gets
created first)


This is what I expected to see as the primary argument against it, the 
lack of a good transactional model for calling the dependent APIs. And 
it's certainly valid.


But what you're describing is the exact same problem regardless if you 
go from the UI or from the Tuskar API. If we're going to do any sort of 
higher level automation of things for the user that spans APIs, we're 
going to run into it. The question is if the client(s) handle it or the 
API. The alternative is to not have the awesome action in the first 
place, in which case we're not really giving the user as much value as 
an application.



I am not saying we should not have tuskar-api. Just put there things
that belongs there, not proxy everything.

>

btw. the real path of the diagram is

tuskar-ui <-> tuskarclient <-> tuskar-api <-> heatclient <-> heat-api
.|ironic|etc.

My conclusion
--

I say if it can be tuskar-ui <-> heatclient <-> heat-api, lets keep it
that way.


I'm still fuzzy on what OpenStack means when it says *client. Is that 
just a bindings library that invokes a remote API or does it also 
contain the CLI bits?



If we realize we are putting some business logic to UI, that needs to be
done also to CLI, or we need to store some additional data, that doesn't
belong anywhere let's put it in Tuskar-API.

Kind Regards,
Ladislav


Thanks for the feedback  :)




On 12/11/2013 03:32 PM, Jay Dobies wrote:

Disclaimer: I swear I'll stop posting this sort of thing soon, but I'm
new to the project. I only mention it again because it's relevant in
that I missed any of the discussion on why proxying from tuskar API to
other APIs is looked down upon. Jiri and I had been talking yesterday
and he mentioned it to me when I started to ask these same sorts of
questions.

On 12/11/2013 07:33 AM, Jiří Stránský wrote:

Hi all,

TL;DR: I believe that "As an infrastructure administrator, Anna wants a
CLI for managing the deployment providing the same fundamental features
as UI." With the planned architecture changes (making tuskar-api thinner
and getting rid of proxying to other services), there's not an obvious
way to achieve that. We need to figure this out. I present a few options
and look forward for feedback.

Previously, we had planned Tuskar arcitecture like this:

tuskar-ui <-> tuskarclient <-> tuskar-api <-> heat-api|ironic-api|etc.


My biggest concern was that having each client call out to the
individual APIs directly put a lot of knowledge into the clients that
had to be replicated across clients. At the best case, that's simply
knowing where to look for data. But I suspect it's bigger than that
and there are workflows that will be implemented for tuskar needs. If
the tuskar API can't call out to other APIs, that workflow
implementation needs to be done at a higher layer, which means in each
client.

Something I'm going to talk about later in this e-mail but I'll
mention here so that the diagrams sit side-by-side is the potential
for a facade layer that hides away the multiple APIs. Lemme see if I
can do this in ASCII:

tuskar-ui -+   +-tuskar-api
   |   |
   +-clie

Re: [openstack-dev] [TripleO][Tuskar] Terminology

[Jay Dobies]

So glad we're hashing this out now. This will save a bunch of headaches 
in the future. Good call pushing this forward.


On 12/11/2013 02:15 PM, Tzu-Mainn Chen wrote:

Hi,

I'm trying to clarify the terminology being used for Tuskar, which may be 
helpful so that we're sure
that we're all talking about the same thing :)  I'm copying responses from the 
requirements thread
and combining them with current requirements to try and create a unified view.  
Hopefully, we can come
to a reasonably rapid consensus on any desired changes; once that's done, the 
requirements can be
updated.

* NODE a physical general purpose machine capable of running in many roles. 
Some nodes may have hardware layout that is particularly
useful for a given role.


Do we ever need to distinguish between undercloud and overcloud nodes?


  * REGISTRATION - the act of creating a node in Ironic


DISCOVERY - The act of having nodes found auto-magically and added to 
Ironic with minimal user intervention.




  * ROLE - a specific workload we want to map onto one or more nodes. 
Examples include 'undercloud control plane', 'overcloud control
plane', 'overcloud storage', 'overcloud compute' etc.

  * MANAGEMENT NODE - a node that has been mapped with an undercloud 
role
  * SERVICE NODE - a node that has been mapped with an overcloud role
 * COMPUTE NODE - a service node that has been mapped to an 
overcloud compute role
 * CONTROLLER NODE - a service node that has been mapped to an 
overcloud controller role
 * OBJECT STORAGE NODE - a service node that has been mapped to an 
overcloud object storage role
 * BLOCK STORAGE NODE - a service node that has been mapped to an 
overcloud block storage role

  * UNDEPLOYED NODE - a node that has not been mapped with a role
   * another option - UNALLOCATED NODE - a node that has not been 
allocated through nova scheduler (?)
- (after reading lifeless's explanation, I agree that 
"allocation" may be a
   misleading term under TripleO, so I 
personally vote for UNDEPLOYED)


Undeployed still sounds a bit odd to me when paired with the word role. 
I could see deploying a workload "bundle" or something, but a role 
doesn't feel like a tangible thing that is pushed out somewhere.


Unassigned? As in, it hasn't been assigned a role yet.


  * INSTANCE - A role deployed on a node - this is where work actually 
happens.


I'm fine with "instance", but the the phrasing "a role deployed on a 
node" feels odd to me in the same way "undeployed" does. Maybe a slight 
change to "A node that has been assigned a role", but that also may be 
me being entirely too nit-picky.


To put it in context, on a scale of 1-10, my objection to this and 
"undeployed" is around a 2, so don't let me come off as strenuously 
objecting.



* DEPLOYMENT

  * SIZE THE ROLES - the act of deciding how many nodes will need to be 
assigned to each role
* another option - DISTRIBUTE NODES (?)
  - (I think the former is more accurate, but 
perhaps there's a better way to say it?)

  * SCHEDULING - the process of deciding which role is deployed on which 
node


I know this derives from a Nova term, but to me, the idea of 
"scheduling" carries a time-in-the-future connotation to it. The 
interesting part of what goes on here is the assignment of which roles 
go to which instances.



  * SERVICE CLASS - a further categorization within a service role for a 
particular deployment.


I don't understand this one, can you add a few examples?


   * NODE PROFILE - a set of requirements that specify what attributes 
a node must have in order to be mapped to
a service class


Even without knowing what "service class" is, I like this one.  :)




Does this seem accurate?  All feedback is appreciated!

Mainn


Thanks again  :D

 ___

OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

Maybe this is a valid use case?

Cloud operator has several core service nodes of differing configuration
types.

[node1]  <-- balanced mix of disk/cpu/ram for general core services
[node2]  <-- lots of disks for Ceilometer data storage
[node3]  <-- low-end "appliance like" box for a specialized/custom core service
 (SIEM box for example)

All nodes[1,2,3] are in the same deployment grouping ("core services)".  As 
such,
this is a heterogenous deployment grouping.  Heterogeneity in this case defined 
by
differing roles and hardware configurations.

This is a real use case.

How do we handle this?


This is the sort of thing I had been concerned with, but I think this is 
just a variation on Robert's GPU example. Rather than butcher it by 
paraphrasing, I'll just include the relevant part:



"The basic stuff we're talking about so far is just about saying each
role can run on some set of undercloud flavors. If that new bit of kit
has the same coarse metadata as other kit, Nova can't tell it apart.
So the way to solve the problem is:
 - a) teach Ironic about the specialness of the node (e.g. a tag 'GPU')
 - b) teach Nova that there is a flavor that maps to the presence of
that specialness, and
   c) teach Nova that other flavors may not map to that specialness

then in Tuskar whatever Nova configuration is needed to use that GPU
is a special role ('GPU compute' for instance) and only that role
would be given that flavor to use. That special config is probably
being in a host aggregate, with an overcloud flavor that specifies
that aggregate, which means at the TripleO level we need to put the
aggregate in the config metadata for that role, and the admin does a
one-time setup in the Nova Horizon UI to configure their GPU compute
flavor."


You mention three specific nodes, but what you're describing is more 
likely three concepts:

- Balanced Nodes
- High Disk I/O Nodes
- Low-End Appliance Nodes

They may have one node in each, but I think your example of three nodes 
is potentially *too* simplified to be considered as proper sample size. 
I'd guess there are more than three in play commonly, in which case the 
concepts breakdown starts to be more appealing.


I think the disk flavor in particular has quite a few use cases, 
especially until SSDs are ubiquitous. I'd want to flag those (in Jay 
terminology, "the disk hotness") as hosting the data-intensive portions, 
but where I had previously been viewing that as manual allocation, it 
sounds like the approach is to properly categorize them for what they 
are and teach Nova how to use them.


Robert - Please correct me if I misread any of what your intention was, 
I don't want to drive people down the wrong path if I'm misinterpretting 
anything.






-k


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

On 12/12/2013 04:25 PM, Keith Basil wrote:

On Dec 12, 2013, at 4:05 PM, Jay Dobies wrote:


Maybe this is a valid use case?

Cloud operator has several core service nodes of differing configuration
types.

[node1]  <-- balanced mix of disk/cpu/ram for general core services
[node2]  <-- lots of disks for Ceilometer data storage
[node3]  <-- low-end "appliance like" box for a specialized/custom core service
 (SIEM box for example)

All nodes[1,2,3] are in the same deployment grouping ("core services)".  As 
such,
this is a heterogenous deployment grouping.  Heterogeneity in this case defined 
by
differing roles and hardware configurations.

This is a real use case.

How do we handle this?


This is the sort of thing I had been concerned with, but I think this is just a 
variation on Robert's GPU example. Rather than butcher it by paraphrasing, I'll 
just include the relevant part:


"The basic stuff we're talking about so far is just about saying each
role can run on some set of undercloud flavors. If that new bit of kit
has the same coarse metadata as other kit, Nova can't tell it apart.
So the way to solve the problem is:
- a) teach Ironic about the specialness of the node (e.g. a tag 'GPU')
- b) teach Nova that there is a flavor that maps to the presence of
that specialness, and
   c) teach Nova that other flavors may not map to that specialness

then in Tuskar whatever Nova configuration is needed to use that GPU
is a special role ('GPU compute' for instance) and only that role
would be given that flavor to use. That special config is probably
being in a host aggregate, with an overcloud flavor that specifies
that aggregate, which means at the TripleO level we need to put the
aggregate in the config metadata for that role, and the admin does a
one-time setup in the Nova Horizon UI to configure their GPU compute
flavor."



Yes, the core services example is a variation on the above.  The idea
of _undercloud_ flavor assignment (flavor to role mapping) escaped me
when I read that earlier.

It appears to be very elegant and provides another attribute for Tuskar's
notion of resource classes.  So +1 here.



You mention three specific nodes, but what you're describing is more likely 
three concepts:
- Balanced Nodes
- High Disk I/O Nodes
- Low-End Appliance Nodes

They may have one node in each, but I think your example of three nodes is 
potentially *too* simplified to be considered as proper sample size. I'd guess 
there are more than three in play commonly, in which case the concepts 
breakdown starts to be more appealing.


Correct - definitely more than three, I just wanted to illustrate the use case.


I not sure I explained what I was getting at properly. I wasn't implying 
you thought it was limited to just three. I do the same thing, simplify 
down for discussion purposes (I've done so in my head about this very 
topic).


But I think this may be a rare case where simplifying actually masks the 
concept rather than exposes it. Manual feels a bit more desirable in 
small sample groups but when looking at larger sets of nodes, the flavor 
concept feels less odd than it does when defining a flavor for a single 
machine.


That's all. :) Maybe that was clear already, but I wanted to make sure I 
didn't come off as attacking your example. It certainly wasn't my 
intention. The balanced v. disk machine thing is the sort of thing I'd 
been thinking for a while but hadn't found a good way to make concrete.



I think the disk flavor in particular has quite a few use cases, especially until SSDs 
are ubiquitous. I'd want to flag those (in Jay terminology, "the disk hotness") 
as hosting the data-intensive portions, but where I had previously been viewing that as 
manual allocation, it sounds like the approach is to properly categorize them for what 
they are and teach Nova how to use them.

Robert - Please correct me if I misread any of what your intention was, I don't 
want to drive people down the wrong path if I'm misinterpretting anything.


-k


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Icehouse Requirements

[Jay Dobies]

* ability to 'preview' changes going to the scheduler


What does this give you? How detailed a preview do you need? What
information is critical there? Have you seen the proposed designs for
a heat template preview feature - would that be sufficient?


Will will probably have a better answer to this, but I feel like at very 
least this goes back to the psychology point raised earlier (I think in 
this thread, but if not, definitely one of the TripleO ones).


A weird parallel is whenever I do a new install of Fedora. I never 
accept their default disk partitioning without electing to review/modify 
it. Even if I didn't expect to change anything, I want to see what they 
are going to give me. And then I compulsively review the summary of what 
actual changes will be applied in the follow up screen that's displayed 
after I say I'm happy with the layout.


Perhaps that's more a commentary on my own OCD and cynicism that I feel 
dirty accepting the magic defaults blindly. I love the idea of anaconda 
doing the heavy lifting of figuring out sane defaults for home/root/swap 
and so on (similarly, I love the idea of Nova scheduler rationing out 
where instances are deployed), but I at least want to know I've seen it 
before it happens.


I fully admit to not knowing how common that sort of thing is. I suspect 
I'm in the majority of geeks and tame by sys admin standards, but I 
honestly don't know. So I acknowledge that my entire argument for the 
preview here is based on my own personality.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] UI Wireframes for Resource Management - ready for implementation

[Jay Dobies]

On 12/13/2013 01:53 PM, Tzu-Mainn Chen wrote:

On 2013/13/12 11:20, Tzu-Mainn Chen wrote:

These look good!  Quick question - can you explain the purpose of Node
Tags?  Are they
an additional way to filter nodes through nova-scheduler (is that even
possible?), or
are they there solely for display in the UI?

Mainn


We start easy, so that's solely for UI needs of filtering and monitoring
(grouping of nodes). It is already in Ironic, so there is no reason why
not to take advantage of it.
-- Jarda


Okay, great.  Just for further clarification, are you expecting this UI 
filtering
to be present in release 0?  I don't think Ironic natively supports filtering
by node tag, so that would be further work that would have to be done.

Mainn


I might be getting ahead of things, but will the tags be free-form 
entered by the user, pre-entered in a separate settings and selectable 
at node register/update time, or locked into a select few that we specify?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tuskar CLI after architecture changes

[Jay Dobies]

On 12/20/2013 08:40 AM, Ladislav Smola wrote:

On 12/20/2013 02:06 PM, Radomir Dopieralski wrote:

On 20/12/13 13:04, Radomir Dopieralski wrote:

[snip]

I have just learned that tuskar-api stays, so my whole ranting is just a
waste of all our time. Sorry about that.



Hehe. :-)

Ok after the last meeting we are ready to say what goes to Tuskar-API.

Who wants to start that thread? :-)


I'm writing something up, but I won't have anything worth showing until 
after the New Year (sounds so far away when I say it that way; it's 
simply that I'm on vacation starting today until the 6th).





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Installing from packages in tripleo-image-elements

[Jay Dobies]

There were so many places in this thread that I wanted to jump in on as 
I caught up, it makes sense to just summarize things in once place 
instead of a half dozen quoted replies.


I agree with the sentiments about flexibility. Regardless of my personal 
preference on source v. packages, it's been my experience that the 
general mindset of production deployment is that new ideas move slowly. 
Admins are set in their ways and policies are in place on how things are 
consumed.


Maybe the newness of all things cloud-related and image-based management 
for scale is a good time to shift the mentality out of packages (again, 
I'm not suggesting whether or not it should be shifted). But I worry 
about adoption if we don't provide an option for people to use blessed 
distro packages, either because of company policy or years of habit and 
bias. If done correctly, there's no difference between a package and a 
particular tag in a source repository, but there is a psychological 
component there that I think we need to account for, assuming someone is 
willing to bite off the implementation costs (which is sounds like there 
is).



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][Tuskar] Domain Model Locations

[Jay Dobies]

I'm trying to hash out where data will live for Tuskar (both long term 
and for its Icehouse deliverables). Based on the expectations for 
Icehouse (a combination of the wireframes and what's in Tuskar client's 
api.py), we have the following concepts:



= Nodes =
A node is a baremetal machine on which the overcloud resources will be 
deployed. The ownership of this information lies with Ironic. The Tuskar 
UI will accept the needed information to create them and pass it to 
Ironic. Ironic is consulted directly when information on a specific node 
or the list of available nodes is needed.



= Resource Categories =
A specific type of "thing" that will be deployed into the overcloud. 
These are static definitions that describe the entities the user will 
want to add to the overcloud and are owned by Tuskar. For Icehouse, the 
categories themselves are added during installation for the four types 
listed in the wireframes.


Since this is a new model (as compared to other things that live in 
Ironic or Heat), I'll go into some more detail. Each Resource Category 
has the following information:


== Metadata ==
My intention here is that we do things in such a way that if we change 
one of the original 4 categories, or more importantly add more or allow 
users to add more, the information about the category is centralized and 
not reliant on the UI to provide the user information on what it is.


ID - Unique ID for the Resource Category.
Display Name - User-friendly name to display.
Description - Equally self-explanatory.

== Count ==
In the Tuskar UI, the user selects how many of each category is desired. 
This stored in Tuskar's domain model for the category and is used when 
generating the template to pass to Heat to make it happen.


These counts are what is displayed to the user in the Tuskar UI for each 
category. The staging concept has been removed for Icehouse. In other 
words, the wireframes that cover the "waiting to be deployed" aren't 
relevant for now.


== Image ==
For Icehouse, each category will have one image associated with it. Last 
I remember, there was discussion on whether or not we need to support 
multiple images for a category, but for Icehouse we'll limit it to 1 and 
deal with it later.


Metadata for each Resource Category is owned by the Tuskar API. The 
images themselves are managed by Glance, with each Resource Category 
keeping track of just the UUID for its image.



= Stack =
There is a single stack in Tuskar, the "overcloud". The Heat template 
for the stack is generated by the Tuskar API based on the Resource 
Category data (image, count, etc.). The template is handed to Heat to 
execute.


Heat owns information about running instances and is queried directly 
when the Tuskar UI needs to access that information.


--

Next steps for me are to start to work on the Tuskar APIs around 
Resource Category CRUD and their conversion into a Heat template. 
There's some discussion to be had there as well, but I don't want to put 
too much into one e-mail.



Thoughts?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Domain Model Locations

[Jay Dobies]

The UI will also need to be able to look at the Heat resources running
within the overcloud stack and classify them according to a resource
category.  How do you envision that working?


There's a way in a Heat template to specify arbitrary metadata on a 
resource. We can add flags in there and key off of those.



Next steps for me are to start to work on the Tuskar APIs around
Resource Category CRUD and their conversion into a Heat template.
There's some discussion to be had there as well, but I don't want to put
too much into one e-mail.



I'm looking forward to seeing the API specification, as Resource Category
CRUD is currently a big unknown in the tuskar-ui api.py file.


Mainn




Thoughts?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Domain Model Locations

[Jay Dobies]

Thanks for the feedback  :)


= Stack =
There is a single stack in Tuskar, the "overcloud".

A small nit here: in the long term Tuskar will support multiple overclouds.


Yes, absolutely. I should have added "For Icehouse" like I did in other 
places. Good catch.



There's few pieces of concepts which I think is missing from the list:
- overclouds: after Heat successfully created the stack, Tuskar needs to
keep track whether it applied the post configuration steps (Keystone
initialization, registering services, etc) or not. It also needs to know
the name of the stack (only 1 stack named 'overcloud' for Icehouse).


I assumed this sort of thing was captured by the resource status, though 
I'm far from a Heat expert. Is it not enough to assume that if the 
resource started successfully, all of that took place?



- service endpoints of an overcloud: eg. Tuskar-ui in the undercloud
will need the url of the overcloud Horizon. The overcloud Keystone owns
the information about this (after post configuration is done) and Heat
owns the information about the overcloud Keystone.



- user credentials for an overcloud: it will be used by Heat during
stack creation, by Tuskar during post configuration, by Tuskar-ui
querying various information (eg. running vms on a node) and finally by
the user logging in to the overcloud Horizon. Now it can be found in the
Tuskar-ui settings file [1].


Both of these are really good points that I haven't seen discussed yet. 
The wireframes cover the allocation of nodes and displaying basic 
details of what's created (even that is still placeholder) but not much 
beyond that.


I'd like to break that into a separate thread. I'm not saying it's 
unrelated, but since it's not even wireframed out I'd like to have a 
dedicated discussion about what it might look like. I'll start that 
thread up as soon as I collect my thoughts.



Imre

[1]
https://github.com/openstack/tuskar-ui/blob/master/local_settings.py.example#L351


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Domain Model Locations

[Jay Dobies]

As much as the Tuskar Chassis model is lacking compared to the Tuskar
Rack model, the opposite problem exists for each project's model of
Node. In Tuskar, the Node model is pretty bare and useless, whereas
Ironic's Node model is much richer.


Thanks for looking that deeply into it :)


So, it's not as simple as it may initially seem :)


Ah, I should have been clearer in my statement - my understanding is that
we're scrapping concepts like Rack entirely.


That was my understanding as well. The existing Tuskar domain model was 
largely placeholder/proof of concept and didn't necessarily reflect 
exactly what was desired/expected.



Mainn


Best,
-jay

[1]
https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/models.py
[2]
https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/models.py#L83



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] Deployment Management section - Wireframes

[Jay Dobies]

Thanks for recording this. A few questions:

- I'm guessing the capacity metrics will come from Ceilometer. Will 
Ceilometer provide the averages for the role or is that calculated by 
Tuskar?


- When on the change deployments screen, after making a change but not 
yet applying it, how are the projected capacity changes calculated?


- For editing a role, does it make a new image with the changes to what 
services are deployed each time it's saved?


- When a role is edited, if it has existing nodes deployed with the old 
version, are the automatically/immediately updated? If not, how do we 
reflect that there's a difference between how the role is currently 
configured and the nodes that were previously created from it?


- I don't see any indication that the role scaling process is taking 
place. That's a potentially medium/long running operation, we should 
have some sort of way to inform the user it's running and if any errors 
took place.


That last point is a bit of a concern for me. I like the simplicity of 
what the UI presents, but the nature of what we're doing doesn't really 
fit with that. I can click the count button to add 20 nodes in a few 
seconds, but the execution of that is a long running, asynchronous 
operation. We have no means of reflecting that it's running, nor finding 
any feedback on it as it runs or completes.


Related question. If I have 20 instances and I press the button to scale 
it out to 50, if I immediately return to the My Deployment screen what 
do I see? 20, 50, or the current count as they are stood up?


It could all be written off as a future feature, but I think we should 
at least start to account for it in the wireframes. The initial user 
experience could be off putting if it's hard to discern the difference 
between what I told the UI to do and when it's actually finished being done.


It's also likely to influence the ultimate design as we figure out who 
keeps track of the running operations and their results (for both simple 
display purposes to the user and auditing reasons).



On 01/10/2014 09:58 AM, Jaromir Coufal wrote:

Hi everybody,

there is first stab of Deployment Management section with future
direction (note that it was discussed as a scope for Icehouse).

I tried to add functionality in time and break it down to steps. This
will help us to focus on one functionality at a time and if we will be
in time pressure for Icehouse release, we can cut off last steps.

Wireframes:
http://people.redhat.com/~jcoufal/openstack/tripleo/2014-01-10_tripleo-ui_deployment-management.pdf


Recording of walkthrough:
https://www.youtube.com/watch?v=9ROxyc85IyE

We sare about to start with first step as soon as possible, so please
focus on our initial steps the most (which doesn't mean that we should
neglect the direction).

Every feedback is very welcome, thanks
-- Jarda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] Deployment Management section - Wireframes

[Jay Dobies]

Another question:

- A Role (sounds like we're moving away from that so I'll call it 
Resource Category) can have multiple Node Profiles defined (assuming I'm 
interpretting the + and the tabs in the Create a Role wireframe 
correctly). But I don't see anywhere where a profile is selected when 
scaling the Resource Category. Is the idea behind the profiles that you 
can select how much power you want to provide in addition to how many nodes?



On 01/10/2014 09:58 AM, Jaromir Coufal wrote:

Hi everybody,

there is first stab of Deployment Management section with future
direction (note that it was discussed as a scope for Icehouse).

I tried to add functionality in time and break it down to steps. This
will help us to focus on one functionality at a time and if we will be
in time pressure for Icehouse release, we can cut off last steps.

Wireframes:
http://people.redhat.com/~jcoufal/openstack/tripleo/2014-01-10_tripleo-ui_deployment-management.pdf


Recording of walkthrough:
https://www.youtube.com/watch?v=9ROxyc85IyE

We sare about to start with first step as soon as possible, so please
focus on our initial steps the most (which doesn't mean that we should
neglect the direction).

Every feedback is very welcome, thanks
-- Jarda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Domain Model Locations

[Jay Dobies]

Excellent write up Jay.

I don't actually know the answer. I'm not 100% bought into the idea that 
Tuskar isn't going to store any information about the deployment and 
will rely entirely on Heat/Ironic as the data store there. Losing this 
extra physical information may be a a strong reason why we need to store 
capture additional data beyond what is or will be utilized by Ironic.


For now, I think the answer is that this is the first pass for Icehouse. 
We're still a ways off from being able to do what you described 
regardless of where the model lives. There are ideas around how to 
partition things as you're suggesting (configuring profiles for the 
nodes; I forget the exact term but there was a big thread about manual 
v. automatic node allocation that had an idea) but there's nothing in 
the wireframes to account for it yet.


So not a very helpful reply on my part :) But your feedback was 
described well which will help keep those concerns in mind post-Icehouse.



Hmm, so this is a bit disappointing, though I may be less disappointed
if I knew that Ironic (or something else?) planned to account for
datacenter inventory in a more robust way than is currently modeled.

If Triple-O/Ironic/Tuskar are indeed meant to be the deployment tooling
that an enterprise would use to deploy bare-metal hardware in a
continuous fashion, then the modeling of racks, and the attributes of
those racks -- location, power supply, etc -- are a critical part of the
overall picture.

As an example of why something like power supply is important... inside
AT&T, we had both 8kW and 16kW power supplies in our datacenters. For a
42U or 44U rack, deployments would be limited to a certain number of
compute nodes, based on that power supply.

The average power draw for a particular vendor model of compute worker
would be used in determining the level of compute node packing that
could occur for that rack type within a particular datacenter. This was
a fundamental part of datacenter deployment and planning. If the tooling
intended to do bare-metal deployment of OpenStack in a continual manner
does not plan to account for these kinds of things, then the chances
that tooling will be used in enterprise deployments is diminished.

And, as we all know, when something isn't used, it withers. That's the
last thing I want to happen here. I want all of this to be the
bare-metal deployment tooling that is used *by default* in enterprise
OpenStack deployments, because the tooling "fits" the expectations of
datacenter deployers.

It doesn't have to be done tomorrow :) It just needs to be on the map
somewhere. I'm not sure if Ironic is the place to put this kind of
modeling -- I thought Tuskar was going to be that thing. But really,
IMO, it should be on the roadmap somewhere.

All the best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] Deployment Management section - Wireframes

[Jay Dobies]

On 01/13/2014 05:43 AM, Jaromir Coufal wrote:

On 2014/10/01 21:17, Jay Dobies wrote:

Another question:

- A Role (sounds like we're moving away from that so I'll call it
Resource Category) can have multiple Node Profiles defined (assuming I'm
interpretting the + and the tabs in the Create a Role wireframe
correctly). But I don't see anywhere where a profile is selected when
scaling the Resource Category. Is the idea behind the profiles that you
can select how much power you want to provide in addition to how many
nodes?


Yes, that is correct, Jay. I mentioned that in walkthrough and in
wireframes with the note "More views needed (for deploying, scaling,
managing roles)".

I would say there might be two approaches - one is to specify which node
profile you want to scale in order to select how much power you want to
add.

The other approach is just to scale the number of nodes in a role and
let system decide the best match (which node profile is chosen will be
decided on the best fit, probably).

I lean towards the first approach, where you specify what role and which
node profile you want to use for scaling. However this is just
introduction of the idea and I believe we can get answers until we get
to that step.

Any preferences for one of above mentioned approaches?


I lean towards the former as well. See the Domain Model Locations thread 
and Jay Pipes' response for an admin's use case that backs it up.


A few weeks ago, there was the giant thread that turned into manual v. 
automatic allocation[1]. The conversation used as an example a system 
that was heavily geared towards disk IO being specifically used for the 
storage-related roles.


Where I'm going with this is that I'm not sure it'll be enough to simply 
use some values for a node profile. I think we're going to need some way 
of identifying nodes as having a particular set of characteristics 
(totally running out of words here) and then saying that the new 
allocation should come from that type of node.


That's a long way of saying that I think an explicit step to say more 
about what we're adding is not only necessary, but potentially 
invalidates some of the wireframes as they exist today. I think over 
time, that is going to be much more complex than incrementing some numbers.


Don't get me wrong. I fully appreciate that we're still very early on 
and scoped to Icehouse for now. Need to start somewhere :)



[1] 
http://lists.openstack.org/pipermail/openstack-dev/2013-December/022163.html



-- Jarda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][Tuskar] Editing Nodes

[Jay Dobies]

I'm pulling this particular discussion point out of the Wireframes 
thread so it doesn't get lost in the replies.


= Background =

It started with my first bulletpoint:

- When a role is edited, if it has existing nodes deployed with the old 
version, are the automatically/immediately updated? If not, how do we 
reflect that there's a difference between how the role is currently 
configured and the nodes that were previously created from it?


Replies:

"I would expect any Role change to be applied immediately. If there is 
some change where I want to keep older nodes how they are set up and 
apply new settings only to new added nodes, I would create new Role then."


"We will have to store image metadata in tuskar probably, that would map 
to glance, once the image is generated. I would say we need to store the 
list of the elements and probably the commit hashes (because elements 
can change). Also it should be versioned, as the images in glance will 
be also versioned.
We can't probably store it in the Glance, cause we will first store the 
metadata, then generate image. Right?


Then we could see whether image was created from the metadata and 
whether that image was used in the heat-template. With versions we could 
also see what has changed.


But there was also idea that there will be some generic image, 
containing all services, we would just configure which services to 
start. In that case we would need to version also this.

"

= New Comments =

My comments on this train of thought:

- I'm afraid of the idea of applying changes immediately for the same 
reasons I'm worried about a few other things. Very little of what we do 
will actually finish executing immediately and will instead be long 
running operations. If I edit a few roles in a row, we're looking at a 
lot of outstanding operations executing against other OpenStack pieces 
(namely Heat).


The idea of immediately also suffers from a sort of "Oh shit, that's not 
what I meant" when hitting save. There's no way for the user to review 
what the larger picture is before deciding to make it so.


- Also falling into this category is the image creation. This is not 
something that finishes immediately, so there's a period between when 
the resource category is saved and the new image exists.


If the image is immediately created, what happens if the user tries to 
change the resource category counts while it's still being generated? 
That question applies both if we automatically update existing nodes as 
well as if we don't and the user is just quick moving around the UI.


What do we do with old images from previous configurations of the 
resource category? If we don't clean them up, they can grow out of hand. 
If we automatically delete them when the new one is generated, what 
happens if there is an existing deployment in process and the image is 
deleted while it runs?


We need some sort of task tracking that prevents overlapping operations 
from executing at the same time. Tuskar needs to know what's happening 
instead of simply having the UI fire off into other OpenStack components 
when the user presses a button.


To rehash an earlier argument, this is why I advocate for having the 
business logic in the API itself instead of at the UI. Even if it's just 
a queue to make sure they don't execute concurrently (that's not enough 
IMO, but for example), the server is where that sort of orchestration 
should take place and be able to understand the differences between the 
configured state in Tuskar and the actual deployed state.


I'm off topic a bit though. Rather than talk about how we pull it off, 
I'd like to come to an agreement on what the actual policy should be. My 
concerns focus around the time to create the image and get it into 
Glance where it's available to actually be deployed. When do we bite 
that time off and how do we let the user know it is or isn't ready yet?


- Editing a node is going to run us into versioning complications. So 
far, all we've entertained are ways to map a node back to the resource 
category it was created under. If the configuration of that category 
changes, we have no way of indicating that the node is out of sync.


We could store versioned resource categories in the Tuskar DB and have 
the version information also find its way to the nodes (note: the idea 
is to use the metadata field on a Heat resource to store the res-cat 
information, so including version is possible). I'm less concerned with 
eventual reaping of old versions here since it's just DB data, though we 
still hit the question of when to delete old images.


- For the comment on a generic image with service configuration, the 
first thing that came to mind was the thread on creating images from 
packages [1]. It's not the exact same problem, but see Clint Byrum's 
comments in there about drift. My gut feeling is that having specific 
images for each res-cat will be easier to manage than trying to edit 
what services are runni

Re: [openstack-dev] [TripleO] [Tuskar] Deployment Management section - Wireframes

[Jay Dobies]

I don't necessarily disagree with this assertion, but what this could
lead to is a proliferation of a bunch of very similar images.  Templatizing
some of the attributes (e.g., this package is enabled, that one isn't)
can reduce the potential explosion of images stored in glance.  If that's
a concern, then it needs to be addressed.  Note that this is true whether
tuskar does/helps with the image building or not.


We have quite a proliferation of services already:

http://docs.openstack.org/training-guides/content/figures/5/figures/image31.jpg

Realistically, the number of individual services on that diagram (I
rapidly counted 39.. I bet I'm off) are the maximum number of image
contents we should ever have in a given deployment. Well plus heat which
is two more. And maybe Trove.. and Designate.. Ok so lets say "50".

Of course, users might recombine every service with every other service
over the life-cycle of their deployment, but realistically, that might
lead to _100_ individual image definitions in service at one time while
new topologies are being rolled out.

I'm o-k with that kind of proliferation. It is measurable and
controllable. Also it is crazy. Realistically we're going to see maybe
10 definitions.. controllers.. computes.. blocks.. swifts.. and some
supporting things.

The positive trade there is that we don't have to wonder how a box has
changed since it was deployed. It is always running all of the software
we deployed to it, with the config we defined now, or it is in an
error state.


+1 to this last point. Customizing services outside of the image is 
going to complicate knowing what is running on each node. It's much 
easier to know the node was provisioned with image X and then knowing 
what's in X.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Editing Nodes

[Jay Dobies]

Thanks for the feedback, really good stuff in here :)

On 01/15/2014 08:07 AM, James Slagle wrote:

I'll start by laying out how I see editing or updating nodes working
in TripleO without Tuskar:

To do my initial deployment:
1.  I build a set of images for my deployment for different roles. The
images are different based on their role, and only contain the needed
software components to accomplish the role they intend to be deployed.
2.  I load the images into glance
3.  I create the Heat template for my deployment, likely from
fragments that are already avaiable. Set quantities, indicate which
images (via image uuid) are for which resources in heat.
4.  heat stack-create with my template to do the deployment

To update my deployment:
1.  If I need to edit a role (or create a new one), I create a new image.
2.  I load the new image(s) into glance
3.  I edit my Heat template, update any quantities, update any image uuids, etc.
4.  heat stack-update my deployment

In both cases above, I see the role of Tuskar being around steps 3 and 4.

I may be misinterpreting, but let me say that I don't think Tuskar
should be building images. There's been a fair amount of discussion
around a Nova native image building service [1][2]. I'm actually not
sure what the status/concensus on that is, but maybe longer term,
Tuskar might call an API to kick off an image build.


I didn't mean to imply that Tuskar would be building images, just 
kicking them off.


As for whether or not it should, that's an interesting question. You and 
I are both on the same page on not having a generic image and having the 
services be configured outside of that, so I'll ignore that idea for now.


I've always thought of Tuskar as providing the user with everything 
they'd need. My gut reaction is that I don't like the idea of saying 
they have to go through a separate step of creating the image and then 
configuring the resource category in Tuskar and attaching the image to it.


That said, I suspect my gut is wrong, or at very least not in line with 
the OpenStack way of thinking.



Ok, so given that frame of reference, I'll reply inline:

On Mon, Jan 13, 2014 at 11:18 AM, Jay Dobies  wrote:

I'm pulling this particular discussion point out of the Wireframes thread so
it doesn't get lost in the replies.

= Background =

It started with my first bulletpoint:

- When a role is edited, if it has existing nodes deployed with the old
version, are the automatically/immediately updated? If not, how do we
reflect that there's a difference between how the role is currently
configured and the nodes that were previously created from it?


I would think Roles need to be versioned, and the deployed version
recorded as Heat metadata/attribute. When you make a change to a Role,
it's a new version. That way you could easily see what's been
deployed, and if there's a newer version of the Role to deploy.


+1, the more I've been thinking about this, the more I like it. We can't 
assume changes will be immediately applied to all provisioned instances, 
so we need some sort of record of what an instance was actually built 
against.



Replies:


I know you quoted the below, but I'll reply here since we're in a new thread.


"I would expect any Role change to be applied immediately. If there is some
change where I want to keep older nodes how they are set up and apply new
settings only to new added nodes, I would create new Role then."


-1 to applying immediately.


Agreed. At large scales, there are a number of problems with this.


When you edit a Role, it gets a new version. But nodes that are
deployed with the older version are not automatically updated.


"We will have to store image metadata in tuskar probably, that would map to
glance, once the image is generated. I would say we need to store the list
of the elements and probably the commit hashes (because elements can
change). Also it should be versioned, as the images in glance will be also
versioned.


I'm not sure why this image metadata would be in Tuskar. I definitely
like the idea of knowing the versions/commit hashes of the software
components in your images, but that should probably be in Glance.


+1


We can't probably store it in the Glance, cause we will first store the
metadata, then generate image. Right?


I'm not sure I follow this point. But, mainly, I don't think Tuskar
should be automatically generating images.


Then we could see whether image was created from the metadata and whether
that image was used in the heat-template. With versions we could also see
what has changed.


We'll be able to tell what image was used in the heat template, and
thus the deployment,  based on it's UUID.

I love the idea of seeing differences between images, especially
installed software versions, but I'm not sure that belongs in Tuskar.
That sor

Re: [openstack-dev] [TripleO] [Tuskar] [UX] Infrastructure Management UI - Icehouse scoped wireframes

[Jay Dobies]

This is a really good evolution. I'm glad the wireframes are getting 
closer to what we're doing for Icehouse.


A few notes...

On page 6, what does the Provisioning Status chart reflect? The math 
doesn't add up if that's supposed to reflect the free v. deployed. That 
might just be a sample data thing, but the term "Provisioning Status" 
makes it sound like this could be tracking some sort of ongoing 
provisioning operation.


What's the distinction between the config shown on the first deployment 
page and the ones under "more options"? Is the idea that the ones on the 
overview page must be specified before the first deployment but the rest 
can be left to the defaults?


The Roles (Resource Category) subtab disappeared but the edit role 
dialog is still there. How do you get to it?


Super happy to see the progress stuff represented. I think it's a good 
first start towards handling the long running changes.


I like the addition of the Undeploy button, but since it's largely a dev 
utility it feels a bit weird being so prominent. Perhaps consider moving 
it under scale deployment; it's a variation of scaling, just scaling 
back to nothing  :)


You locked the controller count to 1 (good call for Icehouse) but still 
have incrementers on the scale page. That should also be disabled and 
hardcoded to 1, right?




On 01/16/2014 08:41 AM, Hugh O. Brock wrote:

On Thu, Jan 16, 2014 at 01:50:00AM +0100, Jaromir Coufal wrote:

Hi folks,

thanks everybody for feedback. Based on that I updated wireframes
and tried to provide a minimum scope for Icehouse timeframe.

http://people.redhat.com/~jcoufal/openstack/tripleo/2014-01-16_tripleo-ui-icehouse.pdf

Hopefully we are able to deliver described set of features. But if
you find something what is missing which is critical for the first
release (or that we are implementing a feature which should not have
such high priority), please speak up now.

The wireframes are very close to implementation. In time, there will
appear more views and we will see if we can get them in as well.

Thanks all for participation
-- Jarda



These look great Jarda, I feel like things are coming together here.

--Hugh



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Editing Nodes

[Jay Dobies]

On 01/17/2014 03:28 AM, mar...@redhat.com wrote:

On 16/01/14 00:28, Clint Byrum wrote:

Excerpts from James Slagle's message of 2014-01-15 05:07:08 -0800:

I'll start by laying out how I see editing or updating nodes working
in TripleO without Tuskar:

To do my initial deployment:
1.  I build a set of images for my deployment for different roles. The
images are different based on their role, and only contain the needed
software components to accomplish the role they intend to be deployed.
2.  I load the images into glance
3.  I create the Heat template for my deployment, likely from
fragments that are already avaiable. Set quantities, indicate which
images (via image uuid) are for which resources in heat.
4.  heat stack-create with my template to do the deployment

To update my deployment:
1.  If I need to edit a role (or create a new one), I create a new image.
2.  I load the new image(s) into glance
3.  I edit my Heat template, update any quantities, update any image uuids, etc.
4.  heat stack-update my deployment

In both cases above, I see the role of Tuskar being around steps 3 and 4.



Agreed!



+1 ...


review  /#/c/52045/ is about generating the overcloud template using
merge.py **. Having recently picked this up again and following latest
wireframes and UI design, it seems like most of current Tuskar code is
going away. After initial panick I saw Jay has actually already started
that with /#/c/66062/

Jay: I think at some point (asap) my /#/c/52045/ will be rebased on your
  /#/c/66062/. Currently my code creates templates from the Tuskar
representations, i.e. ResourceClasses. For now I will assume that I'll
be getting something along the lines of:

{
'resource_categories': { 'controller': 1, 'compute': 4, 'object': 1,
'block' 2}
}

i.e. just resource categories and number of instances for each (plus any
other user supplied config/auth info). Will there be controllers (do we
need them, apart from a way to create, update, delete)? Let's talk some
more on irc later. I'll update the commit message on my review to point
to yours for now,

thanks! marios

** merge.py is going to be binned but it is the best thing we've got
_today_ and within the Icehouse timeframe.


My stuff got merged in today. You should be able to use db's api.py to 
grab everything you need. Ping me (jdob) if you have any questions on it 
or need some different queries.





Steps 1 and 2 are really CI's responsibility in a CD cloud. The end of
the testing phase is "new images in glance!" For a stable release cloud,
a tool for pulling new released images from elsewhere into Glance would
be really useful, but worst case an admin downloads the new images and
loads them manually.


I may be misinterpreting, but let me say that I don't think Tuskar
should be building images. There's been a fair amount of discussion
around a Nova native image building service [1][2]. I'm actually not
sure what the status/concensus on that is, but maybe longer term,
Tuskar might call an API to kick off an image build.



Tuskar should just deploy what it has available. I definitely could
see value in having an image updating service separate from Tuskar,
but I think there are many different answers for "how do images arrive
in Glance?".


Ok, so given that frame of reference, I'll reply inline:

On Mon, Jan 13, 2014 at 11:18 AM, Jay Dobies  wrote:

I'm pulling this particular discussion point out of the Wireframes thread so
it doesn't get lost in the replies.

= Background =

It started with my first bulletpoint:

- When a role is edited, if it has existing nodes deployed with the old
version, are the automatically/immediately updated? If not, how do we
reflect that there's a difference between how the role is currently
configured and the nodes that were previously created from it?


I would think Roles need to be versioned, and the deployed version
recorded as Heat metadata/attribute. When you make a change to a Role,
it's a new version. That way you could easily see what's been
deployed, and if there's a newer version of the Role to deploy.



Could Tuskar version the whole deployment, but only when you want to
"make it so" ? If it gets too granular, it becomes pervasive to try and
keep track of or to roll back. I think that would work well with a goal
I've always hoped Tuskar would work toward which would be to mostly just
maintain deployment as a Heat stack that nests the real stack with the
parameters realized. With Glance growing Heat template storage capability,
you could just store these versions in Glance.


Replies:


I know you quoted the below, but I'll reply here since we're in a new thread.


"I would expect any Role change to be applied immediately. If there is some
change where I want to keep older nodes how they are set up and apply new
setti

Re: [openstack-dev] [TripleO] [Tuskar] Terminology Revival #1 - Roles

[Jay Dobies]

Thanks for starting this!  Comments in-line:


Hi folks,

when I was getting feedback on wireframes and we talked about Roles,
there were various objections and not much suggestions. I would love to
call for action and think a bit about the term for concept currently
known as Role (= Resource Category).

Definition:
Role is a representation of a group of nodes, with specific behavior.


I don't think this is technically correct; according to the wireframes, a 'Role'
is a representation of a group of Heat resources from an overcloud stack - a
Controller Resource, Compute Resource, etc.  Free nodes have no Role.


I'd further disagree that the resource category is a group of nodes. 
It's a *definition* of what can be put onto a node. Then there will be 
nodes who are deployed with that resource category. The nodes themselves 
aren't part of the category, they just, for lack of a better term, use it.



Each role contains (or will contain):
* one or more Node Profiles (which specify HW which is going in)
* association with image (which will be provisioned on new coming nodes)
* specific service settings


There's also likely a versioning component that we haven't fully figured 
out yet.



So far suggested terms:
* Role *
- short name - plus points
- quite overloaded term (user role, etc)


Role immediately conjures up auth-related concepts in my head, so I 
think it has too many existing connotations to be used here.



* Resource Category *
- pretty long (devs already shorten it - confusing)
- Heat specific term


That's why I suggested this term after others objected to Role; it seems to me
that whatever term we use should have the world 'Resource' in it, in order to
make the correspondence clear.


More than most people I've complained about having to type our "resource 
category" and my abbreviation of "res cat" is kinda lame. That said, I 
don't have any other ideas so by default I'm for sticking with what we 
have.  :D




Mainn


* Resource Class *
- older term

Are there any other suggestions (ideally something short and accurate)?
Or do you prefer any of already suggested terms?

Any ideas are welcome - we are not very good in finding the best match
for this particular term.

Thanks
-- Jarda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] [TripleO] adding process/service monitoring

[Jay Dobies]

On 01/27/2014 09:32 PM, Robert Collins wrote:

On 28 January 2014 14:59, Richard Su  wrote:

Hi,

I have been looking into how to add process/service monitoring to
tripleo. Here I want to be able to detect when an openstack dependent
component that is deployed on an instance has failed. And when a failure
has occurred I want to be notified and eventually see it in Tuskar.


+1 on the goal.

We have two basic routes here:
  - use existing things
  - build something new

Service monitoring is a rich field, and there is lots of opportunity
to do new and interesting things. However, it is also a wicked
problem, because - well see all the prior art.

Further to that, like with Chef/Puppet deployers will often have
existing infrastructure investment we should support.

My suggestion is that we take a modular approach - we define an
interface, supply glue code to glue e.g. the assimilation monitoring
project into the interface, and then build on the interface.

There are some important questions - how rich is the interface, what
lives in the glue code, how many backends we'll eventually support,
but the key thing for me is that we don't go down the path of
rewriting e.g. nagios because we're afraid of the dependency: it's
optional to bring in any specific backend, and we can always write our
own later.


+1 to modular. There is too much in this space to try to dictate a 
single blessed approach and still get adoption. I suggest starting by 
trying to nail down a common way of communicating what we're looking for 
without tying it to closely to a particular implementation.



Another point is the crawl/walk/run cycle - lets just get the ability
to click through to a native monitoring screen to start with. That
should be abou t a thousand times faster to ring together than a
complete custom everything.

-Rob



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] Terminology Revival #1 - Roles

[Jay Dobies]

On 01/28/2014 11:42 AM, Jay Pipes wrote:

On Tue, 2014-01-28 at 10:02 -0500, Tzu-Mainn Chen wrote:

Yep, although the reason why - that no end-user will know what these terms mean 
-
has never been entirely convincing to me.


Well, tenants would never see any of the Tuskar UI, so I don't think we
need worry about them. And if a deployer is enabling Tuskar -- and using
Tuskar/Triple-O for undercloud deployment -- then I would think that the
deployer would understand the concept/terminology of undercloud and
overcloud, since it's an essential concept in deploying with
Triple-O. :)

So, in short, I don't see a problem with using the terms undercloud and
overcloud.

Best,
-jay


+1, I was going to say the same thing. Someone installing and using 
Tuskar will have to be sold on the concept of it, and I'm not sure how 
we'd describe what it does without using those terms.








___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Ironic] Roadmap towards heterogenous hardware support

[Jay Dobies]

Wouldn't lying about the hardware specs when registering nodes be
problematic for upgrades?  Users would have
to re-register their nodes.


This was my first impression too, the line "basically lie about the 
hardware specs when enrolling them". It feels more wrong to have the 
user provide false data than it does to ignore that data for Icehouse. 
I'd rather have the data correct now and ignore it than tell users when 
they upgrade to Juno they have to re-enter all of their node data.


It's not heterogenous v. homogeneous support. It's whether or not we use 
the data. We can capture it now and not provide the user the ability to 
differentiate what something is deployed on. That's a heterogeneous 
enrivonment, but just a lack of fine-grained control over where the 
instances fall.


And all of this is simply for the time constraints of Icehouse's first 
pass. A known, temporary limitation.




One reason why a custom filter feels attractive is that it provides us
with a clear upgrade path:

Icehouse
   * nodes are registered with correct attributes
   * create a custom scheduler filter that allows any node to match
   * users are informed that for this release, Tuskar will not
differentiate between heterogeneous hardware

J-Release
   * implement the proper use of flavors within Tuskar, allowing Tuskar
to work with heterogeneous hardware
   * work with nova regarding scheduler filters (if needed)
   * remove the custom scheduler filter


Mainn



As far as nova-scheduler and Ironic go, I believe this is a solved
problem. Steps are:
- enroll hardware with proper specs (CPU, RAM, disk, etc)
- create flavors based on hardware specs
- scheduler filter matches requests exactly

There are, I suspect, three areas where this would fall short today:
- exposing to the user when certain flavors shouldn't be picked,
because there is no more hardware available which could match it
- ensuring that hardware is enrolled with the proper specs //
trouble-shooting when it is not
- a UI that does these well

If I understand your proposal correctly, you're suggesting that we
introduce non-deterministic behavior. If the scheduler filter falls
back to >$flavor when $flavor is not available, even if the search
is in ascending order and upper-bounded by some percentage, the user
is still likely to get something other than what they requested.
 From a utilization and inventory-management standpoint, this would
be a headache, and from a user standpoint, it would be awkward.
Also, your proposal is only addressing the case where hardware
variance is small; it doesn't include a solution for deployments
with substantially different hardware.

I don't think introducing a non-deterministic hack when the
underlying services already work, just to provide a temporary UI
solution, is appropriate. But that's just my opinion.

Here's an alternate proposal to support same-arch but different
cpu/ram/disk hardware environments:
- keep the scheduler filter doing an exact match
- have the UI only allow the user to define one flavor, and have
that be the lowest common denominator of available hardware
- assign that flavor's properties to all nodes -- basically lie
about the hardware specs when enrolling them
- inform the user that, if they have heterogeneous hardware, they
will get randomly chosen nodes from their pool, and that scheduling
on heterogeneous hardware will be added in a future UI release

This will allow folks who are using TripleO at the commandline to
take advantage of their heterogeneous hardware, instead of crippling
already-existing functionality, while also allowing users who have
slightly (or wildly) different hardware specs to still use the UI.


Regards,
Devananda



On Thu, Jan 30, 2014 at 7:14 AM, Tomas Sedovic mailto:tsedo...@redhat.com>> wrote:

On 30/01/14 15:53, Matt Wagner wrote:

On 1/30/14, 5:26 AM, Tomas Sedovic wrote:

Hi all,

I've seen some confusion regarding the homogenous
hardware support as
the first step for the tripleo UI. I think it's time to
make sure we're
all on the same page.

Here's what I think is not controversial:

1. Build the UI and everything underneath to work with
homogenous
hardware in the Icehouse timeframe
2. Figure out how to support heterogenous hardware and
do that (may or
may not happen within Icehouse)

The first option implies having a single nova flavour
that will match
all the boxes we want to work with. It may or may not be
surfaced in the
UI (I t

Re: [openstack-dev] [Heat] [TripleO] Rolling updates spec re-written. RFC

[Jay Dobies]

First, I don't think RollingUpdatePattern and CanaryUpdatePattern should be 2 
different entities. The second just looks like a parametrization of the first 
(growth_factor=1?).


Perhaps they can just be one. Until I find parameters which would need
to mean something different, I'll just use UpdatePattern.


I wondered about this too. Maybe I'm just not as familiar with the 
terminology, but since we're stopping on all failures both function as a 
canary in testing the waters before doing the update. The only 
difference is the potential for acceleration.


As for an example of an entirely different strategy, what about the idea 
of standing up new instances with the updates and then killing off the 
old ones? It may come down to me not fully understanding the scale of 
when you say updating configuration, but it may be desirable to not 
scale down your capacity while the update is executing and instead 
having a quick changeover (for instance, in the floating IPs or a load 
balancer).



I then feel that using (abusing?) depends_on for update pattern is a bit weird. 
Maybe I'm influenced by the CFN design, but the separate UpdatePolicy attribute 
feels better (although I would probably use a property). I guess my main 
question is around the meaning of using the update pattern on a server 
instance. I think I see what you want to do for the group, where child_updating 
would return a number, but I have no idea what it means for a single resource. 
Could you detail the operation a bit more in the document?



I would be o-k with adding another keyword. The idea in abusing depends_on
is that it changes the core language less. Properties is definitely out
for the reasons Christopher brought up, properties is really meant to
be for the resource's end target only.


I think depends_on would be a clever use of the existing language if we 
weren't in a position to influence it's evolution. A resource's update 
policy is a first-class concept IMO, so adding that notion directly into 
the definition feels cleaner.


[snip]

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] consistency vs packages in TripleO

[Jay Dobies]

On Fri, Feb 14, 2014 at 10:27:20AM +1300, Robert Collins wrote:

So progressing with the 'and folk that want to use packages can' arc,
we're running into some friction.

I've copied -operators in on this because its very relevant IMO to operators :)

So far:
  - some packages use different usernames
  - some put things in different places (and all of them use different
places to the bare metal ephemeral device layout which requires
/mnt/).
  - possibly more in future.

Now, obviously its a 'small matter of code' to deal with this, but the
impact on ops folk isn't so small. There are basically two routes that
I can see:

# A
  - we have a reference layout - install from OpenStack git / pypi
releases; this is what we will gate on, and can document.
  - and then each distro (both flavor of Linux and also possibly things
like Fuel that distribution OpenStack) is different - install on X,
get some delta vs reference.
  -> we need multiple manuals describing how to operate and diagnose
issues in such a deployment, which is a matrix that overlays platform
differences the user selects like 'Fedora' and 'Xen'.

I agree with what James already said here. It probably not TripleO's job to
document all that.  A good documentation for the reference layout should be the
goal.

And currently the differences aren't all that big I think. And for some of them
we already have good solutions (like e.g. the os-svc-* tools). There is room
for improvement in handling of the differences for usernames, though :)


# B
  - we have one layout, with one set of install paths, usernames
  - package installs vs source installs make no difference - we coerce
the package into reference upstream shape as part of installing it.

Unless I am completely missunderstanding your proposal I think this would void
many of the reasons why people would choose to install from packages in the
first place.


  - documentation is then identical for all TripleO installs, except
the platform differences (as above - systemd on Fedora, upstart on
Ubuntu, Xen vs KVM)

B seems much more useful to our ops users - less subtly wrong docs, we
avoid bugs where tools we write upstream make bad assumptions,
experience operating a TripleO deployed OpenStack is more widely
applicable (applies to all such installs, not just those that happened
to use the same package source).

I am propably repeating much of what James already said already. But I think an
operator that makes the decision to do a package base Triplo installation does
so e.g. because he is familiar with the tools and conventions of the specific
distro/provider of the packages he choose. And probably wants TripleO to be
consistent with that. And yes, with the decision for packages, he decides to
differ from the reference layout.


I agree with the notions that admins have come to expect differences 
from distro to distro. It's the case for any number of services.


I'd go beyond that and say you're going to have problems getting the 
packages accepted/certified if they break the typical distro 
conventions. There are guidelines that say where things like Python code 
must live and packages may not even be accepted if they violate those.


The same is likely for the admins themselves, taking issue if the 
packages don't match their expectation criteria for the distro.



I see this much like the way Nova abstracts out trivial Hypervisor
differences to let you 'nova boot' anywhere, that we should be hiding
these incidental (vs fundamental capability) differences.

What say ye all?

-Robv




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] [TripleO] [Tuskar] Thoughts on editing node profiles (aka flavors in Tuskar UI)

[Jay Dobies]

On 02/20/2014 06:40 AM, Dmitry Tantsur wrote:

Hi.

While implementing CRUD operations for node profiles in Tuskar (which
are essentially Nova flavors renamed) I encountered editing of flavors
and I have some doubts about it.

Editing of nova flavors in Horizon is implemented as
deleting-then-creating with a _new_ flavor ID.
For us it essentially means that all links to flavor/profile (e.g. from
overcloud role) will become broken. We had the following proposals:
- Update links automatically after editing by e.g. fetching all
overcloud roles and fixing flavor ID. Poses risk of race conditions with
concurrent editing of either node profiles or overcloud roles.
   Even worse, are we sure that user really wants overcloud roles to be
updated?


This is a big question. Editing has always been a complicated concept in 
Tuskar. How soon do you want the effects of the edit to be made live? 
Should it only apply to future creations or should it be applied to 
anything running off the old configuration? What's the policy on how to 
apply that (canary v. the-other-one-i-cant-remember-the-name-for v. 
something else)?



- The same as previous but with confirmation from user. Also risk of
race conditions.
- Do not update links. User may be confused: operation called "edit"
should not delete anything, nor is it supposed to invalidate links. One
of the ideas was to show also deleted flavors/profiles in a separate
table.
- Implement clone operation instead of editing. Shows user a creation
form with data prefilled from original profile. Original profile will
stay and should be deleted manually. All links also have to be updated
manually.
- Do not implement editing, only creating and deleting (that's what I
did for now in https://review.openstack.org/#/c/73576/ ).


I'm +1 on not implementing editing. It's why we wanted to standardize on 
a single flavor for Icehouse in the first place, the use cases around 
editing or multiple flavors are very complicated.



Any ideas on what to do?

Thanks in advance,
Dmitry Tantsur


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] Dealing with passwords in Tuskar-API

[Jay Dobies]

Just to throw this out there, is this something we need for Icehouse?

Yes, I fully acknowledge that it's an ugly security hole. But what's our 
story for how stable/clean Tuskar will be for Icehouse? I don't believe 
the intention is for people to use this in a production environment yet, 
so it will be people trying things out in a test environment. I don't 
think it's absurd to document that we haven't finished hardening the 
security yet and to not use super-sensitive passwords.


If there was a simple answer, I likely wouldn't even suggest this. But 
there's some real design and thought that needs to take place and, 
frankly, we're running out of time. Keeping in mind the intended usage 
of the Icehouse release of Tuskar, it might make sense to shelve this 
for now and file a big fat bug that we address in Juno.


On 02/20/2014 08:47 AM, Radomir Dopieralski wrote:

On 20/02/14 14:10, Jiří Stránský wrote:

On 20.2.2014 12:18, Radomir Dopieralski wrote:



Thinking about it some more, all the uses of the passwords come as a
result of an action initiated by the user either by tuskar-ui, or by
the tuskar command-line client. So maybe we could put the key in their
configuration and send it with the request to (re)deploy. Tuskar-API
would still need to keep it for the duration of deployment (to register
the services at the end), but that's it.


This would be possible, but it would damage the user experience quite a
bit. Afaik other deployment tools solve password storage the same way we
do now.


I don't think it would damage the user experience so much. All you need
is an additional configuration option in Tuskar-UI and Tuskar-client,
the encryption key.

That key would be used to encrypt the passwords when they are first sent
to Tuskar-API, and also added to the (re)deployment calls.

This way, if the database leaks due to a security hole in MySQL or bad
engineering practices administering the database, the passwords are
still inaccessible. To get them, the attacker would need to get
*both* the database and the config files from host on which Tuskar-UI runs.

With the tuskar-client it's a little bit more obnoxious, because you
would need to configure it on every host from which you want to use it,
but you already need to do some configuration to point it at the
tuskar-api and authenticate it, so it's not so bad.

I agree that this complicates the whole process a little, and adds
another potential failure point though.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Tuskar] JSON output values from Tuskar API

[Jay Dobies]

This is a new concept to me in JSON, I've never heard of a wrapper 
element like that being called a namespace.


My first impression is that is looks like cruft. If there's nothing else 
at the root of the JSON document besides the namespace, all it means is 
that every time I go to access relevant data I have an extra layer of 
indirection. Something like:


volume_wrapper = get_volume(url)
volume = volume_wrapper['volume']

or

volume = get_volume(url)
name = volume['volume']['name']

If we ever forsee an aggregate API, I can see some value in it. For 
instance, a single call that aggregates a volume with some relevant 
metrics from ceilometer. In that case, I could see leaving both distinct 
data sets separate at the root with some form of namespace rather than 
attempting to merge the data.


Even in that case, I think it'd be up to the aggregate API to introduce 
that.


Looking at api.openstack.org, there doesn't appear to be any high level 
resource get that would aggregate the different subcollections.


For instance, {tenant_id}/volumes stuffs everything inside of an element 
called "volumes". {tenant_id}/types stuffs everything inside of an 
element called volume_types. If a call to {tenant_id} aggregated both of 
those, then I can see leaving the namespace in on the single ID look ups 
for consistency (even if it's redundant). However, the API doesn't 
appear to support that, so just looking at the examples given it looks 
like an added layer of depth that carries no extra information and makes 
using the returned result a bit awkward IMO.



On 02/26/2014 01:38 PM, Petr Blaho wrote:

Hi,

I am wondering what is the OpenStack way of returning json from
apiclient.

I have got 2 different JSON response examples from http://api.openstack.org/:

json output with namespace:
{
   "volume":
   {
 "status":"available",
 "availability_zone":"nova",
 "id":"5aa119a8-d25b-45a7-8d1b-88e127885635",
 "name":"vol-002",
 "volume_type":"None",
 "metadata":{
   "contents":"not junk"
 }
   }
}
(example for GET 'v2/{tenant_id}/volumes/{volume_id}' of Block Storage API v2.0 
taken from
http://api.openstack.org/api-ref-blockstorage.html [most values ommited])

json output without namespace:
{
   "alarm_actions": [
   "http://site:8000/alarm";
 ],
 "alarm_id": null,
 "combination_rule": null,
 "description": "An alarm",
 "enabled": true,
 "type": "threshold",
 "user_id": "c96c887c216949acbdfbd8b494863567"
}
(example for GET 'v2/alarms/{alarm_id}' of Telemetry API v2.0 taken from
http://api.openstack.org/api-ref-telemetry.html [most values ommited])

Tuskar API now uses "without namespace" variant.

By looking at API docs at http://api.openstack.org/ I can say that
projects use both ways, altought what I would describe as "nicer API"
uses namespaced variant.

So, returning to my question, does OpenStack have some rules what
format of JSON (namespaced or not) should APIs return?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tuskar CLI UX

[Jay Dobies]

Hello,

i went through the CLI way of deploying overcloud, so if you're
interested what's the workflow, here it is:

https://gist.github.com/jistr/9228638


This is excellent to see it all laid out like this, thanks for writing 
it up.



I'd say it's still an open question whether we'll want to give better UX
than that ^^ and at what cost (this is very much tied to the benefits
and drawbacks of various solutions we discussed in December [1]). All in
all it's not as bad as i expected it to be back then [1]. The fact that
we keep Tuskar API as a layer in front of Heat means that CLI user
doesn't care about calling merge.py and creating Heat stack manually,
which is great.


I agree that it's great that Heat is abstracted away. I also agree that 
it's not as bad as I too expected it to be.


But generally speaking, I think it's not an ideal user experience. A few 
things jump out at me:


* We currently have glance, nova, and tuskar represented. We'll likely 
need something to ceilometer as well for gathering metrics and 
configuring notifications (I assume the notifications will fall under 
that, but come with me on it).


That's a lot for an end user to comprehend and remember, which concerns 
me for both adoption and long term usage. Even in the interim when a 
user remembers nova is related to node stuff, doing a --help on nova is 
huge.


That's going to put a lot of stress on our ability to document our 
prescribed path. It will be tricky for us to keep track of the relevant 
commands and still point to the other project client documentation so as 
to not duplicate it all.


* Even at this level, it exposes the underlying guts. There are calls to 
nova baremetal listed in there, but eventually those will turn into 
ironic calls. It doesn't give us a ton of flexibility in terms of 
underlying technology if that knowledge bubbles up to the end user that way.


* This is a good view into what third-party integrators are going to 
face if they choose to skip our UIs and go directly to the REST APIs.



I like the notion of OpenStackClient. I'll talk ideals for a second. If 
we had a standard framework and each project provided a command 
abstraction that plugged in, we could pick and choose what we included 
under the Tuskar umbrella. Advanced users with particular needs could go 
directly to the project clients if needed.


I think this could go beyond usefulness for Tuskar as well. On a 
previous project, I wrote a pluggable client framework, allowing the end 
user to add their own commands that put a custom spin on what data was 
returned or how it was rendered. That's a level between being locked 
into what we decide the UX should be and having to go directly to the 
REST APIs themselves.


That said, I know that's a huge undertaking to get OpenStack in general 
to buy into. I'll leave it more that I think it is a lesser UX (not even 
saying bad, just not great) to have so much for the end user to digest 
to attempt to even play with it. I'm more of the mentality of a unified 
TripleO CLI that would be catered towards handling TripleO stuffs. Short 
of OpenStackClient, I realize I'm not exactly in the majority here, but 
figured it didn't hurt to spell out my opinion  :)




In general the CLI workflow is on the same conceptual level as Tuskar
UI, so that's fine, we just need to use more commands than "tuskar".

There's one naming mismatch though -- Tuskar UI doesn't use Horizon's
Flavor management, but implements its own and calls it Node Profiles.
I'm a bit hesitant to do the same thing on CLI -- the most obvious
option would be to make python-tuskarclient depend on python-novaclient
and use a renamed Flavor management CLI. But that's wrong and high cost
given that it's only about naming :)

The above issue is once again a manifestation of the fact that Tuskar
UI, despite its name, is not a UI for Tuskar, it is UI for a bit more
services. If this becomes a greater problem, or if we want a top-notch
CLI experience despite reimplementing bits that can be already done
(just not in a super-friendly way), we could start thinking about
building something like OpenStackClient CLI [2], but directed
specifically at Undercloud/Tuskar needs and using undercloud naming.

Another option would be to get Tuskar UI a bit closer back to the fact
that Undercloud is OpenStack too, and keep the name "Flavors" instead of
changing it to "Node Profiles". I wonder if that would be unwelcome to
the Tuskar UI UX, though.


Jirka


[1]
http://lists.openstack.org/pipermail/openstack-dev/2013-December/021919.html

[2] https://wiki.openstack.org/wiki/OpenStackClient

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tuskar CLI UX

[Jay Dobies]

Yeah. This is a double bladed axe but i'm leaning towards naming flavors
consistently a bit more too. Here's an attempt at +/- summary:


"node profile"

+ a bit more descriptive for a newcomer imho

- CLI renaming/reimplementing mentioned before

- inconsistency dangers lurking in the deep - e.g. if an error message
bubbles up from Nova all the way to the user, it might mention flavors,
and if we talk 99% of time about node profiles, then user will not know
what is meant in the error message. I'm a bit worried that we'll keep
hitting things like this in the long run.


While I agree with all of your points, this is the one that resonates 
with me the most. We won't be able to be 100% consistent with a rename 
(exceptions are a great example). It's already irritating to the user to 
have to see an error, having to then see it in terms they aren't 
familiar with is an added headache.



- developers still often call them "flavors", because that's what Nova
calls them


"flavor"

+ fits with the rest, does not cause communication or development problems

- not so descriptive (but i agree with you - OpenStack admins will
already be familiar what "flavor" means in the overcloud, and i think
they'd be able to infer what it means in the undercloud)


I'm CCing Jarda as this affects his work quite a lot and i think he'll
have some insight+opinion (he's on PTO now so it might take some time
before he gets to this).





One other thing, I've looked at my own examples so far, so I didn't
really think about this but seeing it written down, I've realised the
way we specify the roles in the Tuskar CLI really bugs me.

  --roles 1=1 \
  --roles 2=1

I know what this means, but even reading it now I think: One equals
one? Two equals one? What? I think we should probably change the arg
name and also refer to roles by name.

  --role-count compute=10

and a shorter option

  -R compute=10


Yeah this is https://bugs.launchpad.net/tuskar/+bug/1281051

I agree with you on the solution (rename long option, support lookup by
names, add a short option).


Thanks

Jirka

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Reviewers] Please add openstack/os-cloud-config to your tripleo-repositories-to-review

[Jay Dobies]

I updated https://wiki.openstack.org/wiki/TripleO and the link at the 
All TripleO Reviews at the bottom to include it.


On 03/02/2014 12:07 AM, Robert Collins wrote:

This is a new repository to provide common code for tuskar and the
seed initialisation logic - the post heat completion initial
configuration of a cloud.

Cheers,
Rob



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Do we want to remove Nova-bm support?

[Jay Dobies]

+1, FWIW.


Alexis


+1

This is similar to the no merge.py discussion. If something isn't 
covered by CI, it's going to grow stale pretty quickly.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Meeting purpose

[Jay Dobies]

As an example of something that I think doesn't add much value in the
meeting - DerekH has already been giving semi-regular CI/CD status
reports via email. I'd like to make these weekly update emails
regular, and take the update off the meeting agenda. I'm offering to
share the load with him to make this easier to achieve.


The Tuskar item is the same way. Not sure how that was added as an 
explicit agenda item, but I don't see why we'd call out to one 
particular project within TripleO. Anything we'd need eyes on should be 
covered when we chime in about specs or reviews needing eyes.



Are there other things on our regular agenda that you feel aren't
offering much value?


I'd propose we axe the regular agenda entirely and let people promote
things in open discussion if they need to. In fact the regular agenda
often seems like a bunch of motions we go through... to the extent that
while the TripleO meeting was going on we've actually discussed what was
in my opinion the most important things in the normal #tripleo IRC
channel. Is getting through our review stats really that important!?


I think the review stats would be better handled in e-mail format like 
Derek's CI status e-mails. We don't want the reviews to get out of hand, 
but the time spent pasting in the links and everyone looking at the 
stats during the meeting itself are wasteful. I could see bringing it up 
if it's becoming a problem, but the number crunching doesn't need to be 
part of the meeting.



  Are there things you'd like to see moved onto, or off, the agenda?


Perhaps a streamlined agenda like this would work better:

  * Bugs


This one is valuable and I like the idea of keeping it.


  * Projects needing releases


Is this even needed as well? It feels like for months now the answer is 
always "Yes, release the world".


I think our cadence on those release can be slowed down as well (the 
last few releases I've done have had minimal churn at best), but I'm not 
trying to thread jack into that discussion. I bring it up because we 
could remove that from the meeting and do an entirely new model where we 
get the release volunteer through other means on a (potentially) less 
frequent release basis.



  * Open Discussion (including important SPECs, CI, or anything needing
attention). ** Leader might have to drive this **


I like the idea of a specific Specs/Reviews section. It should be quick, 
but a specific point in time where people can #info a review they need 
eyes on. I think it appeals to my OCD to have this more structured than 
interspersed with other topics in open discussion.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] nominating James Polley for tripleo-core

[Jay Dobies]

+1

On 01/14/2015 02:26 PM, Gregory Haynes wrote:

Excerpts from Clint Byrum's message of 2015-01-14 18:14:45 +:

Hello! It has been a while since we expanded our review team. The
numbers aren't easy to read with recent dips caused by the summit and
holidays. However, I believe James has demonstrated superb review skills
and a commitment to the project that shows broad awareness of the
project.

Below are the results of a meta-review I did, selecting recent reviews
by James with comments and a final score. I didn't find any reviews by
James that I objected to.

https://review.openstack.org/#/c/133554/ -- Took charge and provided
valuable feedback. +2
https://review.openstack.org/#/c/114360/ -- Good -1 asking for better
commit message and then timely follow-up +1 with positive comments for
more improvement. +2
https://review.openstack.org/#/c/138947/ -- Simpler review, +1'd on Dec.
19 and no follow-up since. Allowing 2 weeks for holiday vacation, this
is only really about 7 - 10 working days and acceptable. +2
https://review.openstack.org/#/c/146731/ -- Very thoughtful -1 review of
recent change with alternatives to the approach submitted as patches.
https://review.openstack.org/#/c/139876/ -- Simpler review, +1'd in
agreement with everyone else. +1
https://review.openstack.org/#/c/142621/ -- Thoughtful +1 with
consideration for other reviewers. +2
https://review.openstack.org/#/c/113983/ -- Thorough spec review with
grammar pedantry noted as something that would not prevent a positive
review score. +2

All current tripleo-core members are invited to vote at this time. Thank
you!



Definite +1.

-Greg

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] HAProxy and Keystone setup (in Overcloud)

[Jay Dobies]

We may want to consider making use of Heat outputs for this.


This was my first thought as well. stack-show returns a JSON document 
that would be easy enough to parse through instead of having it in two 
places.



Rather than assuming hard coding, create an output on the overcloud
template that is something like 'keystone_endpoint'. It would look
something like this:

Outputs:
   keystone_endpoint:
 Fn::Join:
   - ''
   - - "http://";
 - {Fn::GetAtt: [ haproxy_node, first_ip ]} # fn select and yada
 - ":"
 - {Ref: KeystoneEndpointPort} # thats a parameter
 - "/v2.0"


These are then made available via heatclient as stack.outputs in
'stack-show'.

That way as we evolve new stacks that have different ways of controlling
the endpoints (LBaaS anybody?) we won't have to change os-cloud-config
for each one.



2) do Keystone setup from inside Overcloud:
Extend keystone element, steps done in init-keystone script would be
done in keystone's os-refresh-config script. This script would have to
be called only on one of nodes in cluster and only once (though we
already do similar check for other services - mysql/rabbitmq, so I don't
think this is a problem). Then this script can easily get list of
haproxy ports from heat metadata. This looks like more attractive option
to me - it eliminates an extra post-create config step.


Things that can be done from outside the cloud, should be done from
outside the cloud. This helps encourage the separation of concerns and
also makes it simpler to reason about which code is driving the cloud
versus code that is creating the cloud.



Related to Keystone setup is also the plan around keys/cert setup
described here:
http://lists.openstack.org/pipermail/openstack-dev/2014-March/031045.html
But I think this plan would remain same no matter which of the options
above would be used.


What do you think?

Jan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][Summit] Tuskar Backend Changes for Juno

[Jay Dobies]

https://wiki.openstack.org/wiki/TripleO/TuskarJunoPlanning/TemplateBackend

One of the recurring thoughts at the mid-cycle meetup was removing the 
Tuskar database in favor of using Heat templates directly as the data 
store. The wiki above is intended as a starting point for discussions on 
the topic at the Atlanta Summit (I actually can't make it to Summit, but 
there is a Tuskar session scheduled).


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Spec Template Change Proposal

[Jay Dobies]

Currently, there is the following in the template:



Proposed change
===

[snip]

Alternatives


[snip]

Security impact
---



The unit tests assert the top and second level sections are standard, so 
if I add a section at the same level as Alternatives under Proposed 
Change, the tests will fail. If I add a third level section using ^, 
they pass.


The problem is that you can't add a ^ section under Proposed Change. 
Sphinx complains about a title level inconsistency since I'm skipping 
the second level and jumping to the third. But I can't add a 
second-level section directly under Proposed Change because it will 
break the unit tests that validate the structure.


The proposed change is going to be one of the beefier sections of a 
spec, so not being able to subdivide it is going to make the 
documentation messy and removes the ability to link directly to a 
portion of a proposed change.


I propose we add a section at the top of Proposed Change called Overview 
that will hold the change itself. That will allow us to use third level 
sections in the change itself while still having the first and section 
section structure validated by the tests.


I have no problem making the change to the templates, unit tests, and 
any existing specs (I don't think we have any yet), but before I go 
through that, I wanted to make sure there wasn't a major disagreement.


Thoughts?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Spec Template Change Proposal

[Jay Dobies]

Merging a few of the replies into a single response:

> I like all of this plan, except for the name "Overview". To me, 
"Overview" suggests a high-level summary rather than being "one of the 
beefier sections of a spec". Something like "Detail" or "Detailed 
overview" (because the low-level detail will come in the changes that 
implement the spec, not in the spec) seem like better descriptions of 
what we intend to have there.


I didn't put much thought into the name, so Overview, Summary, Detail, 
etc. doesn't matter to me. If we agree to go down the route of a holder 
section here (as compared to loosening the validation), I'll poll for a 
better name.



I'm a bit ambivalent to be honest, but adding a section for Overview
doesn't really do much IMO.  Just give an overview in the first couple
of sentences under "Proposed Change". If I go back and add an Overview
section to my spec in review, I'm just going to slap everything in
Proposed Change into one Overview section :).  To me, Work Items is
where more of the details goes (which does support aribtrary
subsections with ^^^).


That's actually my expectation, that everything currently in place gets 
slapped under Overview. The change is pretty much only to support being 
able to further break down that section while still leaving the existing 
level of validation in place. It's not so much organizational as it is 
to make sphinx happy.



In general though I think that the unit tests are too rigid and
pedantic. Plus, having to go back and update old specs when we make
changes to unit tests seems strange. No biggie right now, but we do
have a couple of specs in review. Unless we write the unit tests to be
backwards compatible. This just feels a bit like engineering just for
the sake of it.  Maybe we need a spec on it :).


I agree that it's possible I'll be back here in the next few days 
complaining that my problem description is too large and would benefit 
from subsections, which I couldn't currently add because they'd be 
second-level sections which are strictly enforced.



I was a bit surprised to see that we don't have the Data Model section
in our specs, and when I had one, unit tests failed. We actually do
have data model stuff in Tuskar and our json structures in tripleo.


You can blame me for that, when I created the repository I took the nova
template and removed the sections I thought we're not relevant perhaps I
was a little too aggressive. I got no problem if we want to add any of
them back in.

Looks like these are the sections I removed:
Data model impact
REST API impact
Notifications impact

I'd obviously forgotten about Tuskar, sorry.



> We just landed a change to permit the third level subsections, but the
intent AIUI of requiring exact titles to constrain the expression
space in the interests of clarity. We can (and should) add more
standard sections as needed.

I do like the idea of having these look consistent. I can work within 
the structure fine given that third-level subsections are permitted, but 
my issue is still that I have been treating the first section under 
"Proposed Change" as the meaty part of the change, which due to the lack 
of a second-level subsection doesn't let me add my own subsections.



Given the feedback, there are a few approaches we can take:

1. Add a second-level subsection at the start of Proposed Change. This 
subsection will be the description of the actual change and adding in 
this will allow custom subsections to be permitted by the existing unit 
tests.


2. Reduce the validation to only enforce required sections but not barf 
on the addition of new ones.



Somewhat tangential (but to address Slagle's concern) is the question of 
whether or not we need some sort of template version number to prevent 
having to update X many existing specs when changing the structure in 
the future. I feel like this is overkill and it's probably much simpler 
to settle on a Juno template in the very near future (selfishly, I say 
"near" to allow my own issue here to be addressed) and then only change 
the templates at new versions. Again, I'm probably overthinking things 
at this point, but just throwing it out there.



Personally, my vote is for #1. Existing specs are simple to update, just 
slap the existing change under the new subsection and move on. For the 
naming of it, I'm fine with James P's suggestion of "Detail".


Then for K, we make any changes to the template based on our usage of it 
in Juno. It's like a scrum post mortem task for a giant 6 month sprint :)


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [UX] [Ironic] [Ceilometer] [Horizon] [TripleO] Nodes Management UI - designs

[Jay Dobies]

Very nicely done, seeing this stuff laid out is really useful. A few 
comments:



= Page 3 =

* Nit: The rocker switch for power is a bit odd to me since it looks 
like it can be toggled.


* Can you show an example of a non-healthy node? Is it just an X instead 
of a check or are there different degrees/forms of unhealthy that can be 
discerned at this level?


* I didn't realize this until the next page and the nodes with bells on 
them, but there's no indication in this table of which node may have an 
alarm associated with it. Is there no way of viewing the node-alarm 
association from this view?



= Page 4 =

* I'm not trying to be a pain in the ass about the counts in the summary 
section, but they are kinda confusing me as I try to read this page 
without guidance.


** I see 26 nodes but it says 28. That's largely a test data nit that 
doesn't affect my understanding.


** It says 0 alarms, but I see three alarm bells. That one is a bit more 
than test data anal-retentiveness since it's making me wonder if I'm 
interpretting the bells correctly as alarms.


** It looks like this is a grid view, so I might be expecting too much, 
but is there any sorting available based on status? I'm guessing the 
columns in the previous view can be sorted (which will be very useful) 
but without something similar here, I wonder to its effectiveness if I 
can't couple the alarmed or non-running machines.



= Page 5 =

* I retract my previous statement about the sorting, the Group By 
example is what I was getting at. Can I drill into a particular group 
and see just those nodes?



= Page 6 =

* This is a cool idea, showing at the summary level why a node is 
unhealthy. What happens if it passes multiple thresholds? Do we just 
show one of the problematic values (assuming there's a priority to the 
metrics so we show the most important one)?



= Page 10 =

* Nit: The tags seem to take up prime screen real estate for something 
I'm not sure is terribly important on this page. Perhaps the intended 
use for them is more important than I'm giving credit.


* Is Flavors Consumption always displayed, or is that just the result of 
an the alarm? If it was unhealthy due to CPU usage, would that appear 
instead/in addition to?



= Page 11 =

* In this view, will we know about configured thresholds? I'm wondering 
if we can color or otherwise highlight more at-risk metrics to 
immediately grab the user's attention.



On 05/28/2014 05:18 PM, Jaromir Coufal wrote:

Hi All,

There is a lot of tags in the subject of this e-mail but believe me that
all listed projects (and even more) are relevant for the designs which I
am sending out.

Nodes management section in Horizon is being expected for a while and
finally I am sharing the results of designing around it.

http://people.redhat.com/~jcoufal/openstack/horizon/nodes/2014-05-28_nodes-ui.pdf


These views are based on modular approach and combination of multiple
services together; for example:
* Ironic - HW details and management
* Ceilometer - Monitoring graphs
* TripleO/Tuskar - Deployment Roles
etc.

Whenever some service is missing, that particular functionality should
be disabled and not displayed to a user.

I am sharing this without any bigger description so that I can get
feedback whether people can get oriented in the UI without hints. Of
course you cannot get each and every detail without exploring, having
tooltips, etc. But the goal for each view is to manage to express at
least the main purpose without explanation. If it does not, it needs to
be fixed.

Next week I will organize a recorded broadcast where I will walk you
through the designs, explain high-level vision, details and I will try
to answer questions if you have any. So feel free to comment anything or
ask whatever comes to your mind here in this thread, so that I can cover
your concerns. Any feedback is very welcome - positive so that I know
what you think that works, as well as negative so that we can improve
the result before implementation.

Thank you all
-- Jarda

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Horizon] Searching for a new name for Tuskar UI

[Jay Dobies]

It might be good to do a similar thing as Keystone does. We could keep
python-tuskarclient focused only on Python bindings for Tuskar (but keep
whatever CLI we already implemented there, for backwards compatibility),
and implement CLI as a plugin to OpenStackClient. E.g. when you want to
access Keystone v3 API features (e.g. domains resource), then
python-keystoneclient provides only Python bindings, it no longer
provides CLI.


+1

I've always liked the idea of separating out the bindings from the CLI 
itself.




I think this is a nice approach because it allows the python-*client to
stay thin for including within Python apps, and there's a common
pluggable CLI for all projects (one top level command for the user). At
the same time it would solve our naming problems (tuskarclient would
stay, because it would be focused on Tuskar only) and we could reuse the
already implemented other OpenStackClient plugins for anything on
undercloud.

We previously raised that OpenStackClient has more plugins (subcommands)
that we need on undercloud and that could confuse users, but i'd say it
might not be as troublesome to justify avoiding the OpenStackClient way.
(Even if we decide that this is a big problem after all and OSC plugin
is not enough, we should still probably aim for separating TripleO CLI
and Tuskarclient in the future.)

Jirka

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] [macaroni] Proposal: change name of Marconi to Macaroni

[Jay Dobies]

While I heartily approve of this renaming for all the reasons already
listed, and am clearly not a robot because I like cheese, you should
know that I will fork Macaroni to make a gluten-free version of the project.


I'll help out on this fork. My low carb diet would only let me use 
Macaroni once a week which would severely lower my productivity.



I hope that does not affect your decision to rename.

^o.0^
-Devananda


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [Murano] [Solum] applications in the cloud

[Jay Dobies]

One thing I'm not seeing handled is cross-app knowledge.

From an earlier post:

> 1) User selects an app which requires a DB
> 2) Murano sees this requirement for DB and do a search in the app
> catalog to find all apps which expose this functionality. Murano uses
> app package definitions for that.
> 3) User select in UI specific DB implementation he wants to use.

For example, take TripleO deploying OpenStack and just look at Nova and 
Glance. Presumably they are two different apps, allowing the user fine 
grained control over how many of each type to deploy and what not.


Both apps would indicate they need a message broker. But they need the 
same message broker, so it's invalid for QPID for one app and rabbit for 
another.


Perhaps I'm viewing it wrong, and instead of having different apps 
there's just one big "OpenStack" app, but that still has the same issue.


On 04/04/2014 05:54 AM, Stan Lagun wrote:

Hi Steve, Thomas

I'm glad the discussion is so constructive!

If we add type interfaces to HOT this may do the job.
Applications in AppCatalog need to be portable across OpenStack clouds.
Thus if we use some globally-unique type naming system applications
could identify their dependencies in unambiguous way.

We also would need to establish relations between between interfaces.
Suppose there is My::Something::Database interface and 7 compatible
materializations:
My::Something::TroveMySQL
My::Something::GaleraMySQL
My::Something::PostgreSQL
My::Something::OracleDB
My::Something::MariaDB
My::Something::MongoDB
My::Something::HBase

There are apps that (say SQLAlchemy-based apps) are fine with any
relational DB. In that case all templates except for MongoDB and HBase
should be matched. There are apps that design to work with MySQL only.
In that case only TroveMySQL, GaleraMySQL and MariaDB should be matched.
There are application who uses PL/SQL and thus require OracleDB (there
can be several Oracle implementations as well). There are also
applications (Marconi and Ceilometer are good example) that can use both
some SQL and NoSQL databases. So conformance to Database interface is
not enough and some sort of interface hierarchy required.


Along those lines, a variation is if a particular version is needed. 
It's an odd parallel, but think of the differences in running a Python 
2.4 app v. 2.6 or higher. There's a pretty clear line there of things 
that won't run on 2.4 and require higher.


Assuming a parallel in these resources, would we end up with multiple 
MongoDB templates with slightly different names? I think the interface 
hierarchy concept comes into play here because we may need to express 
something like "matches at least".



Another thing that we need to consider is that even compatible
implementations may have different set of parameters. For example
clustered-HA PostgreSQL implementation may require additional parameters
besides those needed for plain single instance variant. Template that
consumes *any* PostgreSQL will not be aware of those additional
parameters. Thus they need to be dynamically added to environment's
input parameters and resource consumer to be patched to pass those
parameters to actual implementation







On Fri, Apr 4, 2014 at 9:53 AM, Thomas Spatzier
mailto:thomas.spatz...@de.ibm.com>> wrote:

Hi Steve,

your indexing idea sounds interesting, but I am not sure it would work
reliably. The kind of matching based on names of parameters and
outputs and
internal get_attr uses has very strong assumptions and I think there
is a
not so low risk of false positives. What if the templates includes some
internal details that would not affect the matching but still change the
behavior in a way that would break the composition. Or what if a user by
chance built a template that by pure coincidence uses the same parameter
and output names as one of those abstract types that was mention,
but the
template is simply not built for composition?

I think it would be much cleaner to have an explicit attribute in the
template that says "this template can be uses as realization of type
My::SomeType" and use that for presenting the user choice and
building the
environment.

Regards,
Thomas

Steve Baker mailto:sba...@redhat.com>> wrote on
04/04/2014 06:12:38:
 > From: Steve Baker mailto:sba...@redhat.com>>
 > To: openstack-dev@lists.openstack.org

 > Date: 04/04/2014 06:14
 > Subject: Re: [openstack-dev] [Heat] [Murano] [Solum] applications
inthe
cloud
 >
 > On 03/04/14 13:04, Georgy Okrokvertskhov wrote:
 > Hi Steve,
 >
 > I think this is exactly the place where we have a boundary between
 > Murano catalog and HOT.
 >
 > In your example one can use abstract resource type and specify a
 > correct implementation via environment file. This is how it will be
 > done on the final stage in Murano 

Re: [openstack-dev] [TripleO] config options, defaults, oh my!

[Jay Dobies]

I'm very wary of trying to make the decision in TripleO of what should and 
shouldn't be configurable in some other project.For sure the number of 
config options in Nova is a problem, and one that's been discussed many times 
at summits.   However I think you could also make the case/assumption for any 
service that the debate about having a config option has already been held 
within that service as part of the review that merged that option in the code - 
re-running the debate about whether something should be configurable via 
TripleO feels like some sort of policing function on configurability above and 
beyond what the experts in that service have already considered, and that 
doesn't feel right to me.


My general feeling is that I agree with this sentiment. In my experience 
on management tools, there's always someone who wants to turn the one 
knob I forgot to expose. And that's been on significantly simpler 
projects than OpenStack; the complexity and scale of the features means 
there's potentially a ton of tweaking to be done.


More generally, this starts to drift into the bigger question of what 
TripleO is. The notion of defaults or limiting configuration exposure is 
for prescriptive purposes. "You can change X because we think it's going 
to have a major impact." If we don't expose Y, it's because we're 
driving the user to not want to change it.


I've always assumed TripleO is very low-level. Put another way, 
non-prescriptive. It's not going to push an agenda that says you should 
be doing things a certain way, but rather gives you more than enough 
rope to hang yourself (just makes it easier).


The question of how to make things easier to grok for a new user lies in 
a different area. Either documentation (basic v. advanced user guide 
sort of thing) or potentially in the Tuskar GUI. More configuration 
options means Tuskar's life is more difficult, but to me, that's where 
we add in the notion of "You almost definitely want to configure these 
things, but if you're really insane you can look at this other set of 
stuff to configure."


So I think we need to have a way of specifying everything. And we need 
to have that way not kill us in the process. I like the proposed idea of 
an open-ended config area. It's us acknowledging that we're sitting on 
top of a dozen other projects. Admittedly, I don't fully understand 
Slagle's proposal, but the idea of pulling in samples from other 
projects and not making us acknowledge every configuration option is 
also appealing.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] heat is not present in keystone service-list

[Jay Dobies]

For what it's worth, I have a fresh devstack installation from about a 
week ago and I have two Heat services registered without any extra steps.


On 04/08/2014 11:44 AM, Steven Dake wrote:

On 04/08/2014 07:00 AM, Peeyush Gupta wrote:

Hi all,

I have been trying to install heat with devstack. As shown here
http://docs.openstack.org/developer/heat/getting_started/on_devstack.html

I added the IMAGE_URLS to the locarc file. Then I ran unstack.sh and
then stack.sh. Now, when I run heat stack-list, I get the following error:

$ heat stack-list
publicURL endpoint for orchestration not found

I found that some people got this error because of wrong endpoint in
keystone service-list, but in my output there is no heat!


My guess is your devstack is older to the point of not having heat
enabled by default.  You can add the following to your localrc:

# heat
ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F18-x86_6
4-cfntools.qcow2,http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F18-i3
86-cfntools.qcow2,http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-i
386-cfntools.qcow2,http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-
x86_64-cfntools.qcow2,http://download.fedoraproject.org/pub/fedora/linux/release
s/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2"



$ keystone service-list
+--+--+---+---+
|id|   name   |  type   |
 description|
+--+--+---+---+
| 808b93d2008c48f69d42ae7555c27b6f |  cinder  | volume  |   Cinder
Volume Service   |
| f57c596db43443d7975d890d9f0f4941 | cinderv2 |  volumev2 |
 Cinder Volume Service V2 |
| d8567205287a4072a489a89959801629 |   ec2|  ec2|  EC2
Compatibility Layer  |
| 9064dc9d626045179887186d0b3647d0 |  glance  | image   |
 Glance Image Service   |
| 70cf29f8ceed48d0a39ba7e29481636d | keystone |  identity |
Keystone Identity Service |
| b6cca1393f814637bbb8f95f658ff70a |   nova   |  compute  |
 Nova Compute Service   |
| 0af6de1208a14d259006f86000d33f0d |  novav3  | computev3 |  Nova
Compute Service V3  |
| b170b6b212ae4843b3a6987c546bc640 |s3| s3|
  S3|
+--+--+---+---+

Please help me resolve this error.
Thanks,
~Peeyush Gupta


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] reviewer update march [additional cores]

[Jay Dobies]

On 04/07/2014 07:50 PM, Robert Collins wrote:

tl;dr: 3 more core members to propose:
bnemec
greghaynes
jdon


I'm comfortable with committing to at least 3 reviews a day and promise 
to wield the awesome power of +2 responsibly. I appreciate being 
nominated :)




On 4 April 2014 08:55, Chris Jones  wrote:

Hi

+1 for your proposed -core changes.

Re your question about whether we should retroactively apply the 3-a-day
rule to the 3 month review stats, my suggestion would be a qualified no.

I think we've established an agile approach to the member list of -core, so
if there are a one or two people who we would have added to -core before the
goalposts moved, I'd say look at their review quality. If they're showing
the right stuff, let's get them in and helping. If they don't feel our new
goalposts are achievable with their workload, they'll fall out again
naturally before long.


So I've actioned the prior vote.

I said: "Bnemec, jdob, greg etc - good stuff, I value your reviews
already, but..."

So... looking at a few things - long period of reviews:
60 days:
|greghaynes   | 1210  22  99   0   081.8% |
14 ( 11.6%)  |
|  bnemec | 1160  38  78   0   067.2% |
10 (  8.6%)  |
|   jdob  |  870  15  72   0   082.8% |
4 (  4.6%)  |

90 days:

|  bnemec | 1450  40 105   0   072.4% |
17 ( 11.7%)  |
|greghaynes   | 1420  23 119   0   083.8% |
22 ( 15.5%)  |
|   jdob  | 1060  17  89   0   084.0% |
7 (  6.6%)  |

Ben's reviews are thorough, he reviews across all contributors, he
shows good depth of knowledge and awareness across tripleo, and is
sensitive to the pragmatic balance between 'right' and 'good enough'.
I'm delighted to support him for core now.

Greg is very active, reviewing across all contributors with pretty
good knowledge and awareness. I'd like to see a little more contextual
awareness though - theres a few (but not many) reviews where looking
at how the big picture of things fitting together more would have been
beneficial. *however*, I think that's a room-to-improve issue vs
not-good-enough-for-core - to me it makes sense to propose him for
core too.

Jay's reviews are also very good and consistent, somewhere between
Greg and Ben in terms of bigger-context awareness - so another
definite +1 from me.

-Rob






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] [Tuskar] [Horizon] Icehouse Release of TripleO UI + Demo

[Jay Dobies]

On 04/10/2014 01:40 PM, Nachi Ueno wrote:

Hi Jarda

Congratulations
This release and the demo is super awesome!!
Do you have any instruction to install this one?


I'd like to see this too. I asked a few times and never got an answer on 
whether or not there was a documented way of demoing this without a ton 
of baremetal lying around.






2014-04-10 1:32 GMT-07:00 Jaromir Coufal :

Dear Stackers,

I am happy to announce that yesterday Tuskar UI (TripleO UI) has tagged
branch 0.1.0 for Icehouse release [0].

I put together a narrated demo of all included features [1].

You can find one manual part in the whole workflow - cloud initialization.
There is ongoing work on automatic os-cloud-config, but for the release we
had to include manual way. Automation should be added soon though.

I want to thank all contributors for hard work to make this happen. It has
been pleasure to cooperate with all of you guys and I am looking forward to
bringing new features [2] in.


-- Jarda


[0] 0.1.0 Icehouse Release of the UI:
https://github.com/openstack/tuskar-ui/releases/tag/0.1.0

[1] Narrated demo of TripleO UI 0.1.0:
https://www.youtube.com/watch?v=-6whFIqCqLU

[2] Juno Planning for Tuskar:
https://wiki.openstack.org/wiki/TripleO/TuskarJunoPlanning

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Default paths in os-*-config projects

[Jay Dobies]

On 04/14/2014 09:30 PM, Clint Byrum wrote:

Excerpts from Ben Nemec's message of 2014-04-14 15:41:23 -0700:

Right now the os-*-config projects default to looking for their files in
/opt/stack, with an override env var provided for other locations.  For
packaging purposes it would be nice if they defaulted to a more
FHS-compliant location like /var/lib.  For devtest we could either
override the env var or simply install the appropriate files to /var/lib.

This was discussed briefly in IRC and everyone seemed to be onboard with
the change, but Robert wanted to run it by the list before we make any
changes.  If anyone objects to changing the default, please reply here.
   I'll take silence as agreement with the move. :-)



+1 from me for doing FHS compliance. :)

/var/lib is not actually FHS compliant as it is for "Variable state
information". os-collect-config does have such things, and does use
/var/lib. But os-refresh-config reads executables and os-apply-config
reads templates, neither of which will ever be "variable state
information".

/usr/share would be the right place, as it is "Architecture independent
data". I suppose if somebody wants to compile a C program as an o-r-c
script we could rethink that, but I'd just suggest they drop it in a bin
dir and exec it from a one line shell script in the /usr/share.

So anyway, I suggest:

/usr/share/os-apply-config/templates
/usr/share/os-refresh-config/scripts


+1

This would have been my suggestion too if we were moving out of /opt. 
I've gotten yelled at in the past for not using this in these sorts of 
cases :)



With the usual hierarchy underneath.

We'll need to continue to support the non-FHS paths for at least a few
releases as well.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] /bin/bash vs. /bin/sh

[Jay Dobies]

+1 to using bash, the argument about not keeping POSIX compliance for 
the sake of it makes sense to me.


On 04/15/2014 07:31 AM, Ghe Rivero wrote:

+1 to use bash as the default shell. So far, all major distros use bash
as the default one (except Debian which uses dash).
An about rewriting the code in Python, I agree that shell is complicated
for large programs, but writing anything command oriented in other than
shell is a nightmare. But there are some parts that can benefit from that.

Ghe Rivero

On 04/15/2014 11:05 AM, Chris Jones wrote:

Hi

On 15 April 2014 09:14, Daniel P. Berrange mailto:berra...@redhat.com>> wrote:

I supose that rewriting the code to be in Python is out of the
question ?  IMHO shell is just a terrible language for doing any
program that is remotely complicated (ie longer than 10 lines of


I don't think it's out of the question - where something makes sense
to switch to Python, that would seem like a worthwhile thing to be
doing. I do think it's a different question though - we can quickly
flip things from /bin/sh to /bin/bash without affecting their
suitability for replacement with python.

--
Cheers,

Chris


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][design] review based conceptual design process

[Jay Dobies]

+1, I think it's a better medium for conversations than blueprints or wikis.

I'm also +1 to a tripleo-specs repo, but that's less me having a problem 
with using incubator and more my OCD.


On 04/15/2014 03:43 PM, Monty Taylor wrote:

On 04/15/2014 11:44 AM, Robert Collins wrote:

I've been watching the nova process, and I think its working out well
- it certainly addresses:
  - making design work visible
  - being able to tell who has had input
  - and providing clear feedback to the designers

I'd like to do the same thing for TripleO this cycle..


++


I'm thinking we can just add docs to incubator, since thats already a
repository separate to our production code - what do folk think?


In the current nova-specs thread on the ML, Tim Bell says:

"I think that there is also a need to verify the user story aspect. One
of the great things with the ability to subscribe to nova-specs is that
the community can give input early, when we can check on the need and
the approach. I know from the CERN team how the requirements need to be
reviewed early, not after the code has been written."

Which is great. I'm mentioning it because he calls out the ability to
subscribe to nova-specs.

I think if you put them in incubator, then people who are wanting to
fill a role like Tim - subscribing as an operator and validating user
stories - might be a bit muddied by patches to other thigns. (although
thanks for having a thought about less repos :) )

So I'd just vote, for whatever my vote is worth, for a tripleo-specs repo.

Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Set WIP for stale patches?

[Jay Dobies]

On 2014-09-18 15:21:20 -0400 (-0400), Jay Dobies wrote:

How many of the reviews that we WIP-1 will actually be revisited?

I'm sure there will be cases where a current developer forgetting
they had started on something, seeing the e-mail about the WIP-1,
and then abandoning the change.

But what about developers who have moved off the project entirely?
Is this only masking the problem of stale reviews from our review
stats and leaving the review queue to bloat?

[...]

What is "review queue bloat" in this scenario? How is a change
indefinitely left in Gerrit with "workflow -1" set any different
from a change indefinitely left in Gerrit with "abandoned" set? It's
not like we go through and purge changes from Gerrit based on these,
and they take up just as much space and other resources in either
state.


Ah, ok. I assumed the abandoned ones were reaped over time. Perhaps it's 
just a matter of me writing different searches when I want to ignore the 
workflow -1s.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Strategy for testing and merging the merge.py-free templates

[Jay Dobies]

On 10/14/2014 04:55 AM, Tomas Sedovic wrote:

Hi everyone,

As outlined in the "Remove merge.py"[1] spec, Peter Belanyi and I have
built the templates for controller, nova compute, swift and cinder nodes
that can be deploying directly to Heat (i.e. no merge.py pass is
necessary).

The patches:

https://review.openstack.org/#/c/123100/
https://review.openstack.org/#/c/123713/

I'd like to talk about testing and merging them.

Both Peter and myself have successfully run them through devtest
multiple times. The Tuskar and TripleO UI folks have managed to hook
them up to the UI and make things work, too.

That said, there is a number of limitations which don't warrant making
them the new default just yet:

* Juno Heat is required
* python-heatclient version 0.2.11 is required to talk to Heat
* There is currently no way in Heat to drop specific nodes from a
ResourceGroup (say because of a hardware failure) so the "ellision"
feature from merge.py is not supported yet
* I haven't looked into this yet, but I'm not very optimistic about an
upgrade path from the merge.py-based templates to the heat-native ones

On the other hand, it would be great if we could add them to devtest as
an alternative and to have them exercised by the CI. It would make it
easier to keep them in sync and to iron out any inconsistencies.


James Slagle proposed something like this when I talked to him on IRC:

1. teach devtest about the new templates, driven by a
OVERCLOUD_USE_MERGE_PY switch (defaulting to the merge.py-based templates)
2. Do a CI run of the new template patches, merge them
3. Add a (initially non-voting?) job to test the heat-only templates
4. When we've resolved all the issues stopping up from the switch, make
the native templates default, deprecate the merge.py ones


This makes sense to me. Any objections/ideas?


+1

I like the idea of getting them into CI as soon as possible without 
flipping the default and breaking everyone :)



Thanks,
Tomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tuskar][tripleo] Tuskar/TripleO on Devstack

[Jay Dobies]

5. API: You can't create or modify roles via the API, or even view the
content of the role after creating it


None of that is in place yet, mostly due to time. The tuskar-load-roles 
was a short-term solution to getting a base set of roles in. 
Conceptually you're on target with I want to see in the coming releases.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tuskar] Puppet module

[Jay Dobies]

Nope, there isn't a puppet module for deploying Tuskar, but starting one 
makes sense.


On 10/28/2014 06:04 PM, Emilien Macchi wrote:

Hi,

I was looking at deploying Tuskar API with Puppet and I was wondering if
you guys have already worked on a Puppet module.

If not, I think we could start something in stackforge like we already
did for other OpenStack components.

Thanks,



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] dib-utils Release Question

[Jay Dobies]

I finished the releases for all of our existing projects and after 
poking around tarballs.openstack.org and pypi, it looks like they built 
successfully. Yay me \o/


However, it doesn't look like dib-utils build worked. I don't see it 
listed on tarballs.openstack.org. It was the first release for that 
project, but I didn't take any extra steps (I just followed the 
instructions on the releases wiki and set it to version 0.0.1).


I saw the build for it appear in zuul but I'm not sure how to go back 
and view the results of a build once it disappears off the main page.


Can someone with experience releasing a new project offer me any insight?

Thanks  :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] dib-utils Release Question

[Jay Dobies]

Ahh, ok. I had just assumed it was a Python library, but I admittedly 
didn't look too closely at it. Thanks :)


On 06/23/2014 09:32 PM, Steve Kowalik wrote:

On 24/06/14 06:31, Jay Dobies wrote:

I finished the releases for all of our existing projects and after
poking around tarballs.openstack.org and pypi, it looks like they built
successfully. Yay me \o/

However, it doesn't look like dib-utils build worked. I don't see it
listed on tarballs.openstack.org. It was the first release for that
project, but I didn't take any extra steps (I just followed the
instructions on the releases wiki and set it to version 0.0.1).

I saw the build for it appear in zuul but I'm not sure how to go back
and view the results of a build once it disappears off the main page.

Can someone with experience releasing a new project offer me any insight?


\o/

I've been dealing with releases of new projects from the os-cloud-config
side recently, so let's see.

dib-utils has a post job of dib-utils-branch-tarball, so the job does
exist, as you pointed out, but it doesn't hurt to double check.

The object the tag points to is commit
45b7cf44bc939ef08afc6b1cb1d855e0a85710ad, so logs can be found at
http://logs.openstack.org/45/45b7cf44bc939ef08afc6b1cb1d855e0a85710ad

And from the log a few levels deep at the above URL, we see:

2014-06-16 07:17:13.122 | + tox -evenv python setup.py sdist
2014-06-16 07:17:13.199 | ERROR: toxini file 'tox.ini' not found
2014-06-16 07:17:13.503 | Build step 'Execute shell' marked build as failure

Since it's not a Python project, no tarball or pypi upload.

Cheers,



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Proposal to add Jon Paul Sullivan and Alexis Lee to core review team

[Jay Dobies]

FWIW, I'm a firm believer in progress over perfection and although I
comment on the form, I try to score on the function.


I really like this phrase, comment on the form, score on the function.

Lately I've been trying to be very specific about things I'm pointing 
out that are potentially a learning experience ("This could be shortened 
into self.foo = foo or None"), things that aren't a problem but the 
author might want to take into account ("Consider..."), or those that 
are actually problematic and would warrant a -1.


I've found in the past that it's good to step back every so often and 
reorient myself. Thanks Tomas for the write up.



I'll get better at
commenting to this effect, especially so if my nitpicking gains the
weight of core.

I love English and believe careful use is a great benefit, particularly
in dense technical documents. You're entirely correct that this
shouldn't be allowed to noticeably impede progress though.


Alexis



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Spec Minimum Review Proposal

[Jay Dobies]

At the meetup today, the topic of our spec process came up. The general 
sentiment is that the process is still young and the hiccups are 
expected, but we do need to get better about making sure we're staying 
on top of them.


As a first step, it was proposed to add 1 spec review a week to the 
existing 3 reviews per day requirement for cores.


Additionally, we're going to start to capture and review the metrics on 
spec patches specifically during the weekly meeting. That should help 
bring to light how long reviews are sitting in the queue without being 
touched.


What are everyone's feelings on adding a 1 spec review per week 
requirement for cores?


Not surprisingly, I'm +1 for it  :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] Shared code between server and client

[Jay Dobies]

I'm working on moving the functionality for merging environments from 
the client into the server [1]. I've only superficially looked at 
template_utils.py (in heatclient) but I'm guessing there is stuff in 
there I will want to use server-side.


The server has a requirement on heatclient, but I'm not sure what the 
convention is for using code in it. Can I directly call into a module in 
heatclient/common from the server or is the client dependency only 
intended to be used through the client-facing APIs?


[1] https://blueprints.launchpad.net/heat/+spec/multi-environments

Thanks :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] resource_registry base_url

[Jay Dobies]

In looking through the environment file loading code, I keep seeing a 
check for base_url in the resource registry. It looks like a way to have 
the registry entries only be the filenames (I suppose relative filenames 
work as well) instead of having to enter the full path every time. The 
base_url would be used as the root URL for those filenames when loading 
them.


Thing is, I can't find any reference to that in the docs. I did a search 
for resource_registry, but the only thing I can find is [1] which 
doesn't talk about base_url.


Is this something that's still supported or was it "turned off" (so to 
speak) by removing the docs about it so users didn't know to use it? Is 
the syntax to just sit it side by side with the resource definitions, 
similar to:


resource_registry:
  "base_url": /home/jdob/my_templates
  "OS::Nova::Server": my_nova.yaml

Or am I just totally missing where in the docs it's talked about (which 
is also terribly possible)?


[1] 
http://docs.openstack.org/developer/heat/template_guide/environment.html?highlight=resource_registry


Thanks :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Shared code between server and client

[Jay Dobies]

On 23/10/15 05:35, Robert Collins wrote:

My 2c - if its a stable API in the client, and can be kept stable,
there's no problem.


+1


Ok, forgive me for sounding dumb here (and changing the topic of the 
thread somewhat), but what do we consider a stable client API? Is it as 
broad as any methods in heatclient that aren't prefixed with _? Or do we 
scope it only to specific areas (e.g. common, RPC client) and anything 
else is considered "use at your own risk because we may need to change it"?


My guess is that it's the former given other sentiments I've heard 
around OpenStack, but I wanted to explicitly ask anyway.


Thanks :)



-Rob

On 23 October 2015 at 08:49, Jay Dobies  wrote:

I'm working on moving the functionality for merging environments from
the
client into the server [1]. I've only superficially looked at
template_utils.py (in heatclient) but I'm guessing there is stuff in
there I
will want to use server-side.

The server has a requirement on heatclient, but I'm not sure what the
convention is for using code in it. Can I directly call into a module in
heatclient/common from the server or is the client dependency only
intended
to be used through the client-facing APIs?

[1] https://blueprints.launchpad.net/heat/+spec/multi-environments

Thanks :)

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Shared code between server and client

[Jay Dobies]

On 10/27/2015 11:09 AM, Jay Dobies wrote:

On 23/10/15 05:35, Robert Collins wrote:

My 2c - if its a stable API in the client, and can be kept stable,
there's no problem.


+1


Ok, forgive me for sounding dumb here (and changing the topic of the
thread somewhat), but what do we consider a stable client API? Is it as
broad as any methods in heatclient that aren't prefixed with _? Or do we




scope it only to specific areas (e.g. common, RPC client)


Sorry, I merged two concepts in there. I was referring to common in the 
client and rpc_client in the server. It's possible the same answer 
doesn't apply to both; I'm not sure if we'd want to support someone 
using the server's RPC client code.



else is considered "use at your own risk because we may need to change it"?

My guess is that it's the former given other sentiments I've heard
around OpenStack, but I wanted to explicitly ask anyway.

Thanks :)



-Rob

On 23 October 2015 at 08:49, Jay Dobies  wrote:

I'm working on moving the functionality for merging environments from
the
client into the server [1]. I've only superficially looked at
template_utils.py (in heatclient) but I'm guessing there is stuff in
there I
will want to use server-side.

The server has a requirement on heatclient, but I'm not sure what the
convention is for using code in it. Can I directly call into a
module in
heatclient/common from the server or is the client dependency only
intended
to be used through the client-facing APIs?

[1] https://blueprints.launchpad.net/heat/+spec/multi-environments

Thanks :)

__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).

Some context:

- Historically TripleO has provided a fairly comprehensive "top level"
   parameters interface, where many per-role and common options are
   specified, then passed in to the respective ResourceGroups on deployment

https://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-without-mergepy.yaml#n14

The nice thing about this approach is it gives a consistent API to the
operator, e.g the parameters schema for the main overcloud template defines
most of the expected inputs to the deployment.

The main disadvantage is a degree of template bloat, where we wire dozens
of parameters into each ResourceGroup, and from there into whatever nested
templates consume them.

- When we started adding interfaces (such as all the OS::TripleO::*ExtraConfig*
   interfaces, there was a need to enable passing arbitrary additional
   values to nested templates, with no way of knowing what they are (e.g to
   enable wiring in third-party pieces we have no knowledge of or which
   require implementation-specific arguments which don't make sense for all
   deployments.

To do this, we made use of the heat parameter_defaults interface, which
(unlike normal parameters) have global scope (visible to all nested stacks,
without explicitly wiring in the values from the parent):

http://docs.openstack.org/developer/heat/template_guide/environment.html#define-defaults-to-parameters

The nice thing about this approach is its flexibility, any arbitrary
values can be provided without affecting the parent templates, and it can
allow for a terser implementation because you only specify the parameter
definition where it's actually used.

The main disadvantage of this approach is it becomes very much harder to
discover an API surface for the operator, e.g the parameters that must be
provided on deployment by any CLI/UI tools etc.  This has been partially
addressed by the new-for-liberty nested validation heat feature, but
there's still a bunch of unsolved complexity around how to actually consume
that data and build a coherent consolidated API for user interaction:

https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst

My question is, where do we draw the line on when to use each interface?

My position has always been that we should only use parameter_defaults for
the ExtraConfig interfaces, where we cannot know what reasonable parameters
are.  And for all other "core" functionality, we should accept the increased
template verbosity and wire arguments in from overcloud-without-mergepy.

However we've got some patches which fall into a grey area, e.g this SSL
enablement patch:

https://review.openstack.org/#/c/231930/46/overcloud-without-mergepy.yaml

Here we're actually removing some existing (non functional) top-level
parameters, and moving them to parameter_defaults.

I can see the logic behind it, it does make the templates a bit cleaner,
but at the expense of discoverablility of those (probably not
implementation dependent) parameters.

How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?

In particular I'm keen to hear from Mainn and others interested in building
UIs on top of TripleO as to which is best from that perspective, and how
such arguments may be handled relative to the capabilities mapping proposed
here:

https://review.openstack.org/#/c/242439/

Thanks!

Steve



(in re-reading this, I realize I'm not providing an answer to how I feel 
about the example as I am adding some more thoughts in general; 
apologies for that)


I see there being a few issues with the current approach:

- I'll get this one out of the way first, even though it's not the 
biggest issue. The name 'parameter_defaults' tends to confuse new 3rd 
party integrators since we're not using it as a default per se. I 
understand from Heat's point of view it's defaulting a parameter value, 
but from the user's standpoint, they are setting an actual value to be 
used. Perhaps this can be solved with better docs, and I largely mention 
it because I can see most of what you wrote in here turning into 
documentation, so it'll be good to have it mentioned.


- Back to the point of your e-mail, there are two ways to view it.

-- If you define too few parameters at the top level, you end up with a 
lot of comments like the following inside of nested templates:


ControlPlaneSubnetCidr: # Override this via parameter_defaults

or

# To be defined via a local or global environment in parameter_defaults
  rhel_reg_activation_key:
type: string

There are other examples too, but the thing to note is that we've so far 
been pretty good about adding those comments. It's not a programmatic 
marker, which may be a problem for a UX, but at least the inf

Re: [openstack-dev] [tripleo] Location of TripleO REST API

[Jay Dobies]

On 11/10/2015 10:08 AM, Tzu-Mainn Chen wrote:

Hi all,

At the last IRC meeting it was agreed that the new TripleO REST API
should forgo the Tuskar name, and simply be called... the TripleO
API.  There's one more point of discussion: where should the API
live?  There are two possibilities:

a) Put it in tripleo-common, where the business logic lives.  If we
do this, it would make sense to rename tripleo-common to simply
tripleo.


+1

If the exercise is to move the logic behind an API, let's not have two 
avenues into that logic.



b) Put it in its own repo, tripleo-api


The first option made a lot of sense to people on IRC, as the proposed
API is a very thin layer that's bound closely to the code in tripleo-
common.  The major objection is that renaming is not trivial; however
it was mentioned that renaming might not be *too* bad... as long as
it's done sooner rather than later.

What do people think?


Thanks,
Tzu-Mainn Chen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

My personal preference is to say:

1. Any templates which are included in the default environment (e.g
overcloud-resource-registry-puppet.yaml), must expose their parameters
via overcloud-without-mergepy.yaml

2. Any templates which are included in the default environment, but via a
"noop" implementation *may* expose their parameters provided they are
common and not implementation/vendor specific.


I think this makes sense. Two combination of these two represent what 
TripleO views as the common* API for deploying.


* "common" in the sense that you may not use all of them every time, but 
they are part of the expected deployments.


I still have concerns with us not treating this strongly enough as a 
versioned API, but we can discuss that in a different thread. That's 
more of a "when" we change the parameters v. the conventions on "how" we 
do it.



3. Any templates exposing vendor specific interfaces (e.g at least anything
related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any
parameters via the top level template.

How does this sound?

This does mean we suffer some template bloat from (1) and (2), but it makes
the job of any UI or other tool requiring user input much easier, I think?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

On 11/20/2015 07:05 PM, Ben Nemec wrote:

Thinking about this some more makes me wonder if we need a sample config
generator like oslo.config.  It would work off something similar to the
capabilities map, where you would say

SSL:
   templates:
 -puppet/extraconfig/tls/tls-cert-inject.yaml
   output:
 -environments/enable-ssl.yaml

And the tool would look at that, read all the params from
tls-cert-inject.yaml and generate the sample env file.  We'd have to be
able to do a few new things with the params in order for this to work:

-Need to specify whether a param is intended to be set as a top-level
param, parameter_defaults (which we informally do today with the Can be
overridden by parameter_defaults comment), or internal, to define params
that shouldn't be exposed in the sample config and are only intended as
an interface between templates.  There wouldn't be any enforcement of
the internal type, but Python relies on convention for its private
members so there's precedent. :-)


There is new functionality in Heat that will let you pass in a series of 
templates and environments and it will return:


- The list of top-level parameters, the same way template-validate 
always did

- A list of all nested parameters, keyed by resource.

Take a look at 
https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst 
for details and an example.


That's not entirely what you're getting at, I realize that. I'm glad to 
see you suggest a convention-based approach because I think that's the 
only way we're going to be able to convey some of this information.


I think at the same time we add a mechanism to distinguish between 
internal and external parameters, we need to add something to indicate 
required v. optional.


With a nested stack, anything that's not part of the top-level parameter 
contract is defaulted. The problem is that it loses information on what 
is a valid default v. what's simply defaulted to pass validation.


I've been noticing this more and more on the vendor integrations. They 
have parameters that are required (such as a username) and others that 
are less likely to be changed (I can't think of an example, but I think 
everyone can see where I'm going with this).


So I think there are two sorts of things (at least, I'm also thinking 
off the top of my head) we'd like this tool/sample file to convey:


- Parameters a user would want to change, as compared to those used for 
internal data shuffling
- Information on if the user must supply a value, as compared to 
parameters with an actual default


All that said, I dig this idea of a tool that would generate a skeleton 
environment file.



-There would have to be some way to pick out only certain params from a
template, since I think there are almost certainly features that are
configured using a subset of say puppet/controller.yaml which obviously
can't just take the params from an entire file.  Although maybe this is
an indication that we could/should refactor the templates to move some
of these optional params into their own separate files (at this point I
think I should take a moment to mention that this is somewhat of a brain
dump, so I haven't thought through all of the implications yet and I'm
not sure it all makes sense).



The nice thing about generating these programmatically is we would
formalize the interface of the templates somewhat, and it would be
easier to keep sample envs in sync with the actual implementation.


You could go so far as to put CI on top of it like we do with the oslo 
config stuff, which would be neat.



You'd never have to worry about someone adding a param to a file but
forgetting to update the env (or at least it would be easy to catch and
fix when they did, just run "tox -e genconfig").

I'm not saying this is a simple or short-term solution, but I'm curious
what people think about setting this as a longer-term goal, because as I
think our discussion in Tokyo exposed, we're probably going to have a
bit of an explosion of sample envs soon and we're going to need some way
to keep them sane.

Some more comments inline.

On 11/19/2015 10:16 AM, Steven Hardy wrote:

On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote:

On 11/16/2015 04:25 PM, Steven Hardy wrote:

Hi all,

I wanted to start some discussion re $subject, because it's been apparrent
that we have a lack of clarity on this issue (and have done ever since we
started using parameter_defaults).


[...]


How do people feel about this example, and others like it, where we're
enabling common, but not mandatory functionality?


At first I was thinking about something as simple as: "don't use top-level
params for resources which the registry doesn't enable by default".

It seems to be somewhat what we tried to do with the existing pluggable
resources.

Also, not to hijack the thread but I wanted to add another question related
to a similar issue:

   Is there a reason to prefer use of parameters: instead o

Re: [openstack-dev] [tripleo] When to use parameters vs parameter_defaults

[Jay Dobies]

I think at the same time we add a mechanism to distinguish between
internal and external parameters, we need to add something to indicate
required v. optional.

With a nested stack, anything that's not part of the top-level parameter
contract is defaulted. The problem is that it loses information on what
is a valid default v. what's simply defaulted to pass validation.


I thought the nested validation spec was supposed to handle that though?
  To me, required vs. optional should be as simple as "Does the parameter
definition have a 'default' key?  If yes, then it's optional, if no,
then it's required for the user to pass a value via a parameter or
parameter_default".  I realize we may not have been following that up to
now for various reasons, but it seems like Heat is already providing a
pretty explicit mechanism for marking params as required, so we ought to
use it.


Ya, I was mistaken here. Taking a look at the cinder-netapp.yaml, it 
looks like we're using this correctly:


...
  CinderNetappBackendName:
type: string
default: 'tripleo_netapp'
  CinderNetappLogin:
type: string
  CinderNetappPassword:
type: string
hidden: true
...


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] api-ref gate job is active

[Jay Dobies]

Just a quick note that there is a new job active called 
gate-heat-api-ref. Our API documentation has been pulled into our tree 
[1] and you can run it locally with `tox -e api-ref`.


For now, it's a direct port of our existing API docs, but I'm planning 
on taking a pass over them to double check that they are still valid. 
Feel free to ping me if you have any questions/issues.


[1] https://review.openstack.org/#/c/312712/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][TripleO] Adding interfaces to environment files?

[Jay Dobies]

All,

We've got some requirements around adding some interfaces to the heat
environment file format, for example:

1. Now that we support passing un-merged environment files to heat, it'd be
good to support an optional description key for environments,


I've never understood why the environment file doesn't have a 
description field itself. Templates have descriptions, and IMO it makes 
sense for an environment to describe what its particular additions to 
the parameters/registry do.


I'd be happy to write that patch, but I wanted to first double check 
that there wasn't a big philosophical reason why it shouldn't have a 
description.



such that we
could add an API (in addition to the one added by jdob to retrieve the
merged environment for a running stack) that can retrieve
all-the-environments and we can easily tell which one does what (e.g to
display in a UI perhaps)


I'm not sure I follow. Are you saying the API would return the list of 
descriptions, or the actual contents of each environment file that was 
passed in?


Currently, the environment is merged before we do anything with it. We'd 
have to change that to store... I'm not entirely sure. Multiple 
environments in the DB per stack? Is there a raw_environment in the DB 
that we would leverage?




2. We've got requirements around merge strategies for multiple environments
with potentially colliding keys.  Similar to the cloud-init merge
strategy[1] works.  Basically it should be possible to include multiple
environments then have heat e.g append to a list parameter_default instead
of just last-one-wins.

Both of these will likely require some optional additions to the
environment file format - can we handle them just like e.g event_sinks and
just add them?

Clearly since the environment format isn't versioned this poses a
compatibility problem if "new" environments are used on an old heat, but to
be fair we have done this before (with both parameter_defaults and
event_sinks)

What do folks think, can we add at least the description, and what
interface makes sense for the merge strategy (annotation in the environment
vs data passed to the API along with the environment files list?)

Any thoughts on the above would be great :)

Thanks,

Steve

[1] http://cloudinit.readthedocs.io/en/latest/topics/merging.html
[2] 
https://github.com/openstack/python-heatclient/blob/master/heatclient/common/environment_format.py#L22

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat]informal meetup during summit

[Jay Dobies]

On 4/20/16 1:00 PM, Rico Lin wrote:

Hi team
Let plan for more informal meetup(relax) time! Let all heaters and any
other projects can have fun and chance for technical discussions together.

After discuss in meeting, we will have a pre-meetup-meetup on Friday
morning to have a cup of cafe or some food. Would like to ask if anyone
knows any nice place for this meetup?:)

Also open for other chance for all can go out for a nice dinner and
beer. Right now seems maybe Monday or Friday night could be the best
candidate for this wonderful task, what all think about this? :)


I really like both of these ideas. I haven't met most of you and it'll 
be good to see everyone in a non-Heat light.


I'm available both Monday and Friday nights. I haven't looked at the 
schedule for Monday night to see what else is planned, but that's my 
vote since I suspect people may be leaving on Friday night.





--
May The Force of OpenStack Be With You,

*/Rico Lin
Chief OpenStack Technologist, inwinSTACK
/*irc: ricolin




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][heat] Summit session clashes

[Jay Dobies]

[snip]


I need to be at both of those Heat ones anyway, so this doesn't really
help me. I'd rather have the DLM session in this slot instead. (The only
sessions I can really skip are the Release Model, Functional Tests and
DLM.) That would give us:

   HeatTripleO

  Wed 3:30 Release Model
  Wed 4:30 HOT Parser
  Wed 5:20 Functional Tests

  Thu 1:30 DLM Upgrades
  Thu 2:20 Convergence switchover  Containers
  Thu 3:10 Convergence cleanup Composable Roles
  Thu 4:10 Performance API
  Thu 5:00 Validation  CI


+1 from me, this will let me bounce between the two as well.


I think that way Steve and I could probably both cover upgrades, and he
could cover the rest.

I'd like to get to the composable roles and containers sessions too, but
we'd have to rejig basically every Heat session and I think it's too
late to be doing that.

cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] Summit Friday Night Dinner

[Jay Dobies]

Torchy's Tacos
1311 S 1st St
Austin, TX 78704

It's about a 20 minute walk from the Radisson (pretty much straight down 
Congress Ave) and then over a block.


Meeting in the Radisson lobby at 7:10 to walk over. Apologies in advance 
if we see another snake and I jump on someone's shoulders (Zane, you're 
tall, so you're likely my target).


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat] Client checking of server version

[Jay Dobies]

I ran into an issue in a review about moving environment resolution from 
client to server [1]. It revolves around clients being able to access 
older versions of servers (that's a pretty simplistic description; see 
[2] for the spec).


Before the holiday, Steve Hardy and I were talking about the 
complications involved. In my case, there's no good way to differentiate 
an older server from a legitimate error.


Since the API isn't versioned to the extent that we can leverage that 
value, I was looking into using the template versions call. Something 
along the lines of:


  supported_versions = hc.template_versions.list()
  version_nums = [i.to_dict()['version'].split('.')[1] for i in 
supported_versions]

  mitaka_or_newer = [i for i in version_nums if i >= '2016-04-08']

Yes, I'm planning on cleaning that up before submitting it :)

What I'm wondering is if I should make this into some sort of 
generalized utility method in the client, under the assumption that 
we'll need this sort of check in the future for the same backward 
compatibility requirements.


So a few questions:

1. Does anyone strongly disagree to checking supported template versions 
as a way of determining the specifics of the server API.


2. Does anything like this already exist that I can use?

3. If not, any suggestions on where I should put it? I see a 
heat.common.utils module but I'm not sure if there is a convention 
against that module (or common in general) making live server calls.


Thanks :D


[1] https://review.openstack.org/#/c/239504/
[2] https://review.openstack.org/#/c/226157/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Client checking of server version

[Jay Dobies]

I ran into an issue in a review about moving environment resolution from
client to server [1]. It revolves around clients being able to access
older versions of servers (that's a pretty simplistic description; see
[2] for the spec).

Before the holiday, Steve Hardy and I were talking about the
complications involved. In my case, there's no good way to differentiate
an older server from a legitimate error.


Hmmm, it's true that you'll likely just get a 400 error, but I'd hope
that the error message is at least somewhat unique.


Unfortunately, it's not, but I don't think it's due to a Heat problem so 
much as just the nature of the issue. Here's what's happening.


New Client: doesn't do client-side environment resolution before sending 
it to the server.


Old Server: expects the environment to be fully populated and ignores 
the environment file(s) in the files dict.


The result is the server spits back an error saying that, in my 
scenario, there is no type mapping for jdob::Resource1.


The problem is, I get the exact same result for New Client + New Server 
+ incomplete environment files.


The reason I was looking for some sort of version checking is to avoid 
having logic that just says "Maybe it's because it's an old server, 
lemme resolve the environments and send the request again." It feels 
really wrong to trigger two create requests when it's the templates 
themselves that are wrong.



Since the API isn't versioned to the extent that we can leverage that


I mean... it totally is but so far we've chosen not to bump that
version. And we mostly got away with it because we were only adding
functionality. So far.


value, I was looking into using the template versions call. Something
along the lines of:

   supported_versions = hc.template_versions.list()
   version_nums = [i.to_dict()['version'].split('.')[1] for i in
supported_versions]
   mitaka_or_newer = [i for i in version_nums if i >= '2016-04-08']

Yes, I'm planning on cleaning that up before submitting it :)

What I'm wondering is if I should make this into some sort of
generalized utility method in the client, under the assumption that
we'll need this sort of check in the future for the same backward
compatibility requirements.

So a few questions:

1. Does anyone strongly disagree to checking supported template versions
as a way of determining the specifics of the server API.


Yes.

Template versions are supposed to be pluggable, and are explicitly under
control of the operator. We shouldn't be systematically inferring
anything about the server version based on this; in general there's no
causal relationship.


2. Does anything like this already exist that I can use?


Not really; there's the "heat build-info" command, but that is also
explicitly under the control of the operator (and is empty by default).


3. If not, any suggestions on where I should put it? I see a
heat.common.utils module but I'm not sure if there is a convention
against that module (or common in general) making live server calls.

Thanks :D


[1] https://review.openstack.org/#/c/239504/
[2] https://review.openstack.org/#/c/226157/

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Client checking of server version

[Jay Dobies]

On 01/05/2016 04:37 PM, Steven Hardy wrote:

On Mon, Jan 04, 2016 at 03:53:07PM -0500, Jay Dobies wrote:

I ran into an issue in a review about moving environment resolution from
client to server [1]. It revolves around clients being able to access older
versions of servers (that's a pretty simplistic description; see [2] for the
spec).

Before the holiday, Steve Hardy and I were talking about the complications
involved. In my case, there's no good way to differentiate an older server
from a legitimate error.

Since the API isn't versioned to the extent that we can leverage that value,
I was looking into using the template versions call. Something along the
lines of:

   supported_versions = hc.template_versions.list()
   version_nums = [i.to_dict()['version'].split('.')[1] for i in
supported_versions]
   mitaka_or_newer = [i for i in version_nums if i >= '2016-04-08']

Yes, I'm planning on cleaning that up before submitting it :)

What I'm wondering is if I should make this into some sort of generalized
utility method in the client, under the assumption that we'll need this sort
of check in the future for the same backward compatibility requirements.

So a few questions:

1. Does anyone strongly disagree to checking supported template versions as
a way of determining the specifics of the server API.


Ok, so some valid concerns have been raised over deriving things using the
HOT version (although I do still wonder if the environment itself should be
versioned, just like the templates, then we could rev the environment
verion and say it supports a list, vs changing anything in the API, but
that's probably a separate discussion).

Taking a step back for a moment, the original discussion was around
providing transparent access to the new interface via heatclient, but that
isn't actually a hard requirement - the old interface works fine for many
users, so we could just introduce a new interface (which would eventually
become the default, after all non-EOL heat versions released support the
new API argument):

Currently we do:

heat stack-create foo -f foo.yaml -e a.yaml -e b.yaml

And this implies some client-side resolution of the multiple -e arguments.

-e is short for "--environment-file", but we could introduce a new format,
e.g "-E", short for "--environment-files":

heat stack-create foo -f foo.yaml -E a.yaml -E b.yaml

This option would work the same way as the current interface, but it would
pass the files unmodified for resolution inside heat (by using the new API
format), and as it's opt-in, it's leaving all the current heatclient
interfaces alone without any internal fallback logic?


+1

My only concern is that the default isn't to exercise the "preferred" 
approach.


However, perhaps I'm viewing things wrong with that as being preferred 
instead of just an alternate for non-heatclient. IIRC, the code is 
largely the same, just being called from two separate places (client v. 
server), so it's not an issue of duplication or the actual logic growing 
stale. And it shouldn't really be an issue of the server-side path 
accidentally breaking since there is CI around it. So maybe my concerns 
are overblown.


It does feel weird to have to document something like that, trying to 
describe the differences between -e and -E, but I suppose if we mark -e 
as deprecated it should be understandable enough.


This also has the benefit of letting this code land without having to do 
a major implementation of micro-versions, so that's a plus :)




Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Client checking of server version

[Jay Dobies]

On 01/06/2016 12:05 PM, Zane Bitter wrote:

On 05/01/16 16:37, Steven Hardy wrote:

On Mon, Jan 04, 2016 at 03:53:07PM -0500, Jay Dobies wrote:

I ran into an issue in a review about moving environment resolution from
client to server [1]. It revolves around clients being able to access
older
versions of servers (that's a pretty simplistic description; see [2]
for the
spec).

Before the holiday, Steve Hardy and I were talking about the
complications
involved. In my case, there's no good way to differentiate an older
server
from a legitimate error.

Since the API isn't versioned to the extent that we can leverage that
value,
I was looking into using the template versions call. Something along the
lines of:

   supported_versions = hc.template_versions.list()
   version_nums = [i.to_dict()['version'].split('.')[1] for i in
supported_versions]
   mitaka_or_newer = [i for i in version_nums if i >= '2016-04-08']

Yes, I'm planning on cleaning that up before submitting it :)

What I'm wondering is if I should make this into some sort of
generalized
utility method in the client, under the assumption that we'll need
this sort
of check in the future for the same backward compatibility requirements.

So a few questions:

1. Does anyone strongly disagree to checking supported template
versions as
a way of determining the specifics of the server API.


Ok, so some valid concerns have been raised over deriving things using
the
HOT version (although I do still wonder if the environment itself
should be
versioned, just like the templates, then we could rev the environment
verion and say it supports a list, vs changing anything in the API, but
that's probably a separate discussion).

Taking a step back for a moment, the original discussion was around
providing transparent access to the new interface via heatclient, but
that
isn't actually a hard requirement - the old interface works fine for many
users, so we could just introduce a new interface (which would eventually
become the default, after all non-EOL heat versions released support the
new API argument):

Currently we do:

heat stack-create foo -f foo.yaml -e a.yaml -e b.yaml

And this implies some client-side resolution of the multiple -e
arguments.

-e is short for "--environment-file", but we could introduce a new
format,
e.g "-E", short for "--environment-files":

heat stack-create foo -f foo.yaml -E a.yaml -E b.yaml

This option would work the same way as the current interface, but it
would
pass the files unmodified for resolution inside heat (by using the new
API
format), and as it's opt-in, it's leaving all the current heatclient
interfaces alone without any internal fallback logic?


That would certainly work, but it sounds like a usability/support
nightmare :(

Is there a reason we wouldn't consider bumping the API version to 1.1
for this? We'll have to figure out how to do it some time.


I started to look into the Nova specs on how they handle micro versions. 
I have a few other things on my plate I want to finish up this week, but 
I should be able to take a stab at a POC for it.



cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] spec-lite for simple feature requests

[Jay Dobies]

On 01/20/2016 10:21 AM, Rabi Mishra wrote:

Hi All,

As discussed in the team meeting, below is the proposed spec-lite process for 
simple feature requests. This is already being used in Glance project. 
Feedback/comments/concerns are welcome, before we update the contributor docs 
with this:).


tl;dr - spec-lite is a simple feature request created as a bug with enough 
details and with a `spec-lite` tag. Once triaged with status 'Triaged' and 
importance changed to 'Whishlist', it's approved. Status 'Won’t fix' signifies 
the request is rejected and 'Invalid' means it would require a full spec.


Heat Spec Lite
--

Lite specs are small feature requests tracked as Launchpad bugs, with status 
'Wishlist' and tagged with 'spec-lite' tag. These allow for submission and 
review of these feature requests before code is submitted.

These can be used for simple features that don’t warrant a detailed spec to be 
proposed, evaluated, and worked on. The team evaluates these requests as it 
evaluates specs. Once a bug has been approved as a Request for Enhancement 
(RFE), it’ll be targeted for a release.


The workflow for the life of a spec-lite in Launchpad is as follows:

1. File a bug with a small summary of what the request change is and tag it as 
spec-lite.
2. The bug is triaged and importance changed to Wishlist.
3. The bug is evaluated and marked as Triaged to announce approval or to Won’t 
fix to announce rejection or Invalid to request a full spec.
4. The bug is moved to In Progress once the code is up and ready to review.
5. The bug is moved to Fix Committed once the patch lands.

In summary the states are:

New:This is where spec-lite starts, as filed by the community.
Triaged:Drivers - Move to this state to mean, “you can start working on 
it”
Won’t Fix:  Drivers - Move to this state to reject a lite-spec.
Invalid:Drivers - Move to this state to request a full spec for this 
request

Lite spec Submission Guidelines
---

When a bug is submitted, there are two fields that must be filled: ‘summary’ 
and ‘further information’. The ‘summary’ must be brief enough to fit in one 
line.

The ‘further information’ section must be a description of what you would like 
to see implemented in heat. The description should provide enough details for a 
knowledgeable developer to understand what is the existing problem and what’s 
the proposed solution.

Add spec-lite tag to the bug.


Thanks,
Rabi


I think the concept is a really good idea. I like the idea of a light 
weight verification that something makes sense before beginning coding.


One question, when are bugs triaged?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jay Dobies]

I fall very much in the same mentality as Ben. I'm +1 to all of his 
points, with a few comments inline.


On 01/22/2016 12:24 PM, Ben Nemec wrote:

So I haven't weighed in on this yet, in part because I was on vacation
when it was first proposed and missed a lot of the initial discussion,
and also because I wanted to take some time to order my thoughts on it.
  Also because my initial reaction...was not conducive to calm and
rational discussion. ;-)

The tldr is that I don't like it.  To explain why, I'm going to make a
list (everyone loves lists, right? Top $NUMBER reasons we should stop
expecting other people to write our API for us):

1) We've been down this road before.  Except last time it was with Heat.
  I'm being somewhat tongue-in-cheek here, but expecting a general
service to provide us a user-friendly API for our specific use case just
doesn't make sense to me.


I think it's important to think about outside integrations here. The 
current model is to tell other clients to manipulate Heat environments 
and understand how to parse/inspect templates*. Now it will be to 
understand/parse/manipulate Mistral workflows. Neither of those are 
conducive to the types of UI wireframes we've proposed in the past, much 
less friendly to completely outside integrators.


* I realize some of that inspection is moving into Heat, but it's still 
at the mechanical template level rather than providing insight into how 
to actually use them.



2) The TripleO API is not a workflow API.  I also largely missed this
discussion, but the TripleO API is a _Deployment_ API.  In some cases
there also happens to be a workflow going on behind the scenes, but
honestly that's not something I want our users to have to care about.


I'm glad Ben mentioned this, because I always viewed the workflow 
aspects as a subset of what actually needs to be done.



3) It ties us 100% to a given implementation.  If Mistral proves to be a
poor choice for some reason, or insufficient for a particular use case,
we have no alternative.  If we have an API and decide to change our
implementation, nobody has to know or care.  This is kind of the whole
point of having an API - it shields users from all the nasty
implementation details under the surface.


I strongly agree with this one. It's not even generic speculation; we've 
told people in the past to deal with Heat templates and now we're 
telling them to deal with workflows. We already have a history of the 
backend changing and an API would give us much more flexibility (and 
less annoyed users).



4) It raises the bar even further for both new deployers and developers.
  You already need to have a pretty firm grasp of Puppet and Heat
templates to understand how our stuff works, not to mention a decent
understanding of quite a number of OpenStack services.

This presents a big chicken and egg problem for people new to OpenStack.
  It's great that we're based on OpenStack and that allows people to peek
under the hood and do some tinkering, but it can't be required for
everyone.  A lot of our deployers are going to have little to no
OpenStack experience, and TripleO is already a daunting task for those
people (hell, it's daunting for people who _are_ experienced).

5) What does reimplementing all of our tested, well-understood Python
into a new YAML format gain us?  This is maybe the biggest thing I'm
missing from this whole discussion.  We lose a bunch of things (ease of
transition from other Python projects, excellent existing testing
framework, etc.), but what are we actually gaining other than the
ability to say that we use N + 1 OpenStack services?  Because we're way
past the point where "It's OpenStack deploying OpenStack" is sufficient
reason for people to pay attention to us.  We need less "Ooh, neat" and
more "Ooh, that's easy to use and works well."  It's still not clear to
me that Mistral helps in any way with the latter.

6) On the testing note, how do we test these workflows?  Do we know what
happens when step X fails?  How do we test that they handle it properly
in an automated and repeatable way?  In Python these are largely easy
questions to answer: unit tests.  How do you unit test YAML?  This is a
big reason I'm not even crazy about having Mistral on the back end of a
TripleO API.  We'd be going from code that we can test and prove works
in a variety of scenarios, to YAML that is tested and proven to work in
exactly the three scenarios we run in CI.  This is basically the same
situation we had with tripleo-incubator, and it was bad there too.

I dunno.  Maybe I'm too late to this party to have any impact on the
discussion, but I very much do not like the direction we're going and I
would be remiss if I didn't at least point out my concerns with it.

-Ben

On 01/13/2016 03:41 AM, Tzu-Mainn Chen wrote:

Hey all,

I realize now from the title of the other TripleO/Mistral thread [1] that
the discussion there may have gotten confused.  I think using Mistral for
TripleO processe

Re: [openstack-dev] [TripleO] Encapsulating logic and state in the client

[Jay Dobies]

Thinking about this further, the interesting question to me is how much
logic we aim to encapsulate behind an API. For example, one of the simpler
CLI commands we have in RDO-Manager (which is moving upstream[1]) is to
run introspection on all of the Ironic nodes. This involves a series of
commands that need to be run in order and it can take upwards of 20
minutes depending how many nodes you have. However, this does just
communicate with Ironic (and ironic inspector) so is it worth hiding
behind an API? I am inclined to say that it is so we can make the end
result as easy to consume as possible but I think it might be difficult
to draw the line in some cases.

The question then rises about what this API would look like? Generally
speaking I feel like it looks like a workflow API, it shouldn't offer
many (or any?) unique features, rather it manages the process of
performing a series of operations across multiple APIs. There have been
attempts at doing this within OpenStack before in a more general case,
I wonder what we can learn from those.


This is where my head is too. The OpenStack on OpenStack thing means we 
get to leverage the existing tools and users can leverage their existing 
knowledge of the products.


But what I think an API will provide is guidance on how to achieve that 
(the big argument there being if this should be done in an API or 
through documentation). It coaches new users and integrations on how to 
make all of the underlying pieces play together to accomplish certain 
things.


To your question on that ironic call, I'm split on how I feel.

On one hand, I really like the idea of the TripleO API being able to 
support an OpenStack deployment entirely on its own. You may want to go 
directly to some undercloud tools for certain edge cases, but for the 
most part you should be able to accomplish the goal of deploying 
OpenStack through the TripleO APIs.


But that's not necessarily what TripleO wants to be. I've seen the 
sentiment of it only being tools for deploying OpenStack, in which case 
a single API isn't really what it's looking to do. I still think we need 
some sort of documentation to guide integrators instead of saying "look 
at the REST API docs for these 5 projects", but that documentation is 
lighter weight than having pass through calls in a TripleO API.





Unfortunately, as undesirable as these are, they're sometimes necessary
in the world we currently live in. The only long-term solution to this
is to put all of the logic and state behind a ReST API where it can be
accessed from any language, and where any state can be stored
appropriately, possibly in a database. In principle that could be
accomplished either by creating a tripleo-specific ReST API, or by
finding native OpenStack undercloud APIs to do everything we need. My
guess is that we'll find a use for the former before everything is ready
for the latter, but that's a discussion for another day. We're not there
yet, but there are things we can do to keep our options open to make
that transition in the future, and this is where tripleo-common comes in.

I submit that anything that adds logic or state to the client should be
implemented in the tripleo-common library instead of the client plugin.
This offers a couple of advantages:

- It provides a defined boundary between code that is CLI-specific and
code that is shared between the CLI and GUI, which could become the
model for a future ReST API once it has stabilised and we're ready to
take that step.
- It allows for an orderly transition when that happens - we can have a
deprecation period during which the tripleo-common library is imported
into both the client and the (future, hypothetical) ReST API.

cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



[1]: https://review.openstack.org/#/c/215186/3/gerrit/projects.yaml,cm

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Plugin integration and environment file naming

[Jay Dobies]

I like where this is going. I've been asked a number of times where to 
put things and we never had a solid convention. I like the idea of 
having that docced somewhere.


I like either of the proposed solutions. My biggest concern is that they 
don't capture how you actually use them. I know that was the point of 
your e-mail; we don't yet have the Heat constructs in place for the 
templates to convey that information.


What about if we adopt the directory structure model and strongly 
request a README.md file in there? It's similar to the image elements 
model. We could offer a template to fill out or leave it open ended, but 
the purpose would be to specify:


- Installation instructions (e.g. "set the resource registry namespace 
for Blah to point to this file" or "use the corresponding environment 
file foo.yaml")
- Parameters that can/should be specified via parameter_defaults. I'm 
not saying we add a ton of documentation in there that would be 
duplicate of the actual parameter definitions, but perhaps just a list 
of the parameter names. That way, a user can have an idea of what 
specifically to look for in the template parameter list itself.


That should be all of the info that we'd like Heat to eventually provide 
and hold us over until those discussions are finished.


On 09/08/2015 08:20 AM, Jiří Stránský wrote:

On 8.9.2015 13:47, Jiří Stránský wrote:

Apart from "cinder" and "neutron-ml2" directories, we could also have a
"combined" (or sth similar) directory for env files which combine
multiple other env files. The use case which i see is for extra
pre-deployment configs which would be commonly used together. E.g.
combining Neutron and Horizon extensions of a single vendor [4].


Ah i mixed up two things in this paragraph -- env files vs. extraconfig
nested stacks. Not sure if we want to start namespacing the extraconfig
bits in a parallel manner. E.g.
"puppet/extraconfig/pre_deploy/controller/cinder",
"puppet/extraconfig/pre_deploy/controller/neutron-ml2". It would be
nice, especially if we're sort of able to map the extraconfig categories
to env file categories most of the time. OTOH the directory nesting is
getting quite deep there :)


That was my thought too, that the nesting is getting a bit deep. I also 
don't think we should enforce the role in the directory structure as 
we've already seen instances of things that have to happen on both 
controller and compute.




J.


[4]
https://review.openstack.org/#/c/213142/1/puppet/extraconfig/pre_deploy/controller/all-bigswitch.yaml




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Core reviewers for python-tripleoclient and tripleo-common

[Jay Dobies]

On 09/10/2015 10:06 AM, James Slagle wrote:

TripleO has added a few new repositories, one of which is
python-tripleoclient[1], the former python-rdomanager-oscplugin.

With the additional repositories, there is an additional review burden
on our core reviewers. There is also the fact that folks who have been
working on the client code for a while when it was only part of RDO
are not TripleO core reviewers.

I think we could help with the additional burden of reviews if we made
two of those people core on python-tripleoclient and tripleo-common
now.

Specifically, the folks I'm proposing are:
Brad P. Crochet 
Dougal Matthews 


+1 to both. I've seen a lot of Dougal's reviews and his Python knowledge 
is excellent.



The options I see are:
- keep just 1 tripleo acl, and add additional folks there, with a good
faith agreement not to +/-2,+A code that is not from the 2 client
repos.


+1 to this. I feel like it encourages cross pollination into other 
tripleo repos (we could use the eyes on THT) without having to jump 
through extra hoops as their involvement with them increases.



- create a new gerrit acl in project-config for just these 2 client
repos, and add folks there as needed. the new acl would also contain
the existing acl for tripleo core reviewers
- neither of the above options - don't add these individuals to any
TripleO core team at this time.

The first is what was more or less done when Tuskar was brought under
the TripleO umbrella to avoid splitting the core teams, and it's the
option I'd prefer.

TripleO cores, please reply here with your vote from the above
options. Or, if you have other ideas, you can share those as well :)

[1] https://review.openstack.org/#/c/215186/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] mox to mock migration

[Jay Dobies]

I forget where we left things at the last meeting with regard to whether 
or not there should be a blueprint on this. I was going to work on some 
during some downtime but I wanted to make sure I wasn't overlapping with 
what others may be converting (it's more time consuming than I anticipated).


Any thoughts on how to track it?

Thanks :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] mox to mock migration

[Jay Dobies]

This sounds good, I was hoping it'd be acceptable to use etherpad. I 
filed a blueprint [1] but I'm anticipating using the etherpad much more 
regularly to track which files are being worked or completed.


[1] https://blueprints.launchpad.net/heat/+spec/mox-to-mock-conversion
[2] https://etherpad.openstack.org/p/heat-mox-to-mock

Thanks for the guidance :)

On 10/09/2015 12:42 PM, Steven Hardy wrote:

On Fri, Oct 09, 2015 at 09:06:57AM -0400, Jay Dobies wrote:

I forget where we left things at the last meeting with regard to whether or
not there should be a blueprint on this. I was going to work on some during
some downtime but I wanted to make sure I wasn't overlapping with what
others may be converting (it's more time consuming than I anticipated).

Any thoughts on how to track it?


I'd probably suggest raising either a bug or a blueprint (not spec), then
link from that to an etherpad where you can track all the tests requiring
rework, and who's working on them.

"it's more time consuming than I anticipated" is pretty much my default
response for anything to do with heat unit tests btw, good luck! :)

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo]Recursive validation for easier composability

[Jay Dobies]

On 06/22/2015 12:19 PM, Steven Hardy wrote:

Hi all,

Lately I've been giving some thought to how we might enable easier
composability, and in particular how we can make it easier for folks to
plug in deeply nested optional extra logic, then pass data in via
parameter_defaults to that nested template.

Here's an example of the use-case I'm describing:

https://review.openstack.org/#/c/193143/5/environments/cinder-netapp-config.yaml

Here, we want to allow someone to easily turn on an optional configuration
or feature, in this case a netapp backend for cinder.


I think the actual desired goal is bigger than just optional 
configuration. I think it revolves more around choosing a nested stack 
implementation for a resource type and how to manage custom parameters 
for that implementation. We're getting into the territory here of having 
a parent stack defining an API that nested stacks can plug into. I'd 
like to have some sort of way of deriving that information instead of 
having it be completely relegated to outside documentation (but I'm 
getting off topic; at the end I mention how I want to do a better write 
up of the issues Tuskar has faced and I'll elaborate more there).



The parameters specific to this feature/configuration only exist in the
nested cinder-netapp-config.yaml template, then parameter_defaults are used
to wire in the implementation specific data without having to pass the
values through every parent template (potentially multiple layers of
nesting).

This approach is working out OK, but we're missing an interface which makes
the schema for parameters over the whole tree available.

>

This is obviously
a problem, particularly for UI's, where you really need a clearly defined
interface for what data is required, what type it is, and what valid values
may be chosen.


I think this is going to be an awesome addition to Heat. As you alluded 
to, we've struggled with this in TripleO. The parameter_defaults works 
to circumvent the parameter passing, but it's rough from a user 
experience point of view since getting the unified list of what's 
configurable is difficult.



I'm considering an optional additional flag to our template-validate API
which allows recursive validation of a tree of templates, with the data
returned on success to include a tree of parameters, e.g:

heat template-validate -f parent.yaml -e env.yaml --show-nested
{
   "Description": "The Parent",
   "Parameters": {
 "ParentConfig": {
   "Default": [],
   "Type": "Json",
   "NoEcho": "false",
   "Description": "",
   "Label": "ExtraConfig"
 },
 "ControllerFlavor": {
   "Type": "String",
   "NoEcho": "false",
   "Description": "",
   "Label": "ControllerFlavor"
 }
   }
  "NestedParameters": {
 "child.yaml": {
 "Parameters": {
   "ChildConfig": {
   "Default": [],
   "Type": "Json",
   "NoEcho": "false",
   "Description": "",
   "Label": "Child ExtraConfig"
   }
 }
  }
}


Are you intending on resolving parameters passed into a nested stack 
from the parent against what's defined in the nested stack's parameter 
list? I'd want NestedParameters to only list things that aren't already 
being specified to the parent.


Specifically with regard to the TripleO Heat templates, there is still a 
lot of logic that needs to be applied to properly divide out parameters. 
For example, there are some things passed in from the parents to the 
nested stacks that are kinda namespaced by convention, but its not a 
hard convention. So to try to group the parameters by service, we'd have 
to look at a particular NestedParameters section and then also add in 
anything from the parent that applies to that service. I don't believe 
we can use parameter groups to correlate them (we might be able to, or 
that might be its own improvement).


I realize that's less of a Heat issue and more of a THT issue, but I 
figured I'd bring it up anyway.



This implies that we also need to pass the "files" map to the validate API,
like we currently do for create (we already pass the environment, although
it's not really doing much beyond providing parameters for the parent stack
AFAICT, we completely skip validating TemplateResources because the files
aren't passed):

https://github.com/openstack/heat/blob/master/heat/engine/service.py#L873

Before I go ahead and spend time writing a spec/code for this, what do
folks think about enhancing validate like this?  Is there an alternative,
for example adding a parameters schema output to stack-preview?


For what it's worth, I'd rather see this as a spec before code. There 
are a lot of complications we hit in Tuskar in trying to make 
configuring the overcloud through THT user-friendly. This is one part of 
it, but there are others. I'd like to have them all talked out and see 
what the larger group of changes are.


For example, take the cinder-netapp-config example you ment

[openstack-dev] [TripleO][Heat] Tuskar v. Heat responsibilities

[Jay Dobies]

I didn't want to hijack Steve Hardy's thread about the recursive 
validation, but I wanted to summarize the needs that Tuskar and the UI 
have been trying to answer and some of the problems we ran into.


I think it's fairly common knowledge now that Tuskar and the THT 
templates diverged over the past few months, so I won't rehash it. If 
you need a summary of what happened, look here: 
https://jdob.fedorapeople.org/tuskar-heat.jpg


Below are some of the needs that the Tuskar UI in general has when 
working with the TripleO Heat Templates. I'm hoping we can come up with 
a decent list and use that to help drive what belongs in Heat v. what 
belongs elsewhere, and ultimately what that "elsewhere" actually is.



= Choosing Component Implementations =

== Background ==

I'm already off to a bad start, since the word "component" isn't 
actually a term in this context. What I'm referring to is the fact that 
we are starting to see what is almost a plugin model in the THT templates.


Previously, we had assumed that all of the overcloud configuration would 
be done through parameters. This is no longer the case as the 
resource_registry is used to add certain functionality.


For example, in overcloud-resource-registry-puppet.yaml, we see:

 # set to controller-config-pacemaker.yaml to enable pacemaker
 OS::TripleO::ControllerConfig: puppet/controller-config.yaml

That's a major overcloud configuration setting, but that choice isn't 
made through a parameter. It's in a different location and a different 
mechanism entirely.


Similarly, enabling a Netapp backend for Cinder is done by setting a 
resource_registry entry to change the CinderBackend template [1]. This 
is a slightly different case conceptually than HA since the original 
template being overridden is a noop [2], but the mechanics of how to set 
it are the same.


There are also a number of pre and post hooks that exist in the 
overcloud template that we are seeing more and more implementations of. 
RHEL registration is implemented as such a hook [3].


I'm drawing a difference here between fundamental configuration changes 
(HA v. non-HA) and optional additions (RHEL registration). Again, 
mechanically they are implemented as resource_registry substitutions, 
though from a UI standpoint we'd likely want to treat them differently. 
Whether or not that difference is actually captured by the templates 
themselves or is purely in the UI is open to debate.


== Usage in TripleO ==

All of the examples I mentioned above have landed upstream and the Heat 
features necessary to facilitate them all exist.


What doesn't exist is a way to manipulate the resource_registry. Tuskar 
doesn't have APIs for that level of changes; it assumed all 
configuration changes would be through parameters and hasn't yet had 
time to add in support for dorking with the registry in this fashion.


While, technically, all of the resource_registry entries can be 
overridden, there are only a few that would make sense for a user to 
want to configure (I'm not talking about advanced users writing their 
own templates).


On top of that, only certain templates can be used to fulfill certain 
resource types. For instance, you can't point CinderBackend to 
rhel-registration.yaml. That information isn't explicitly captured by 
Heat templates. I suppose you could inspect usages of a resource type in 
overcloud to determine the "api" of that type and then compare that to 
possible implementation templates' parameter lists to figure out what is 
compatible, but that seems like a heavy-weight approach.


I mention that because part of the user experience would be knowing 
which resource types can have a template substitution made and what 
possible templates can fulfill it.


== Responsibility ==

Where should that be implemented? That's a good question.

The idea of resolving resource type uses against candidate template 
parameter lists could fall under the model Steve Hardy is proposing of 
having Heat do it (he suggested the validate call, but this may be 
leading us more towards template inspection sorts of APIs supported by Heat.


It is also possibly an addition to HOT, to somehow convey an interface 
so that we can more easily programatically look at a series of templates 
and understand how they play together. We used to be able to use the 
resource_registry to understand those relationships, but that's not 
going to work if we're trying to find substitutions into the registry.


Alternatively, if Heat/HOT has no interest in any of this, this is 
something that Tuskar (or a Tuskar-like substitute) will need to solve 
going forward.



= Consolidated Parameter List =

== Background ==

This is what Steve was getting at in his e-mail. I'll rehash the issue 
briefly.


We used to be able to look at the parameters list in the overcloud 
template and know all of the parameters that need to be specified to 
configure the overcloud.


The parameter passing is pretty strict, so if overc